| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
Before touching CONFIG_LWIP_TCP_WND_DEFAULT, make the window question
decidable from the logs. New per-window metrics in the stream log,
/state, and stats struct:
- wire min/avg/max kbps: instantaneous throughput while each body
drained (jpeg_len/recv_us). Ceiling ~= TCP_WND/RTT, so it scales
with the window iff the window is the limiter.
- hdr_gap min/avg/max us: time blocked waiting for the next header
after finishing a body. Large = sender-paced; ~0 = receive path
is the bottleneck.
- pend_age min/avg/max us: publish->claim latency of painted frames.
Growing across windows = queue backlog building, the failure mode
that killed the previous WND=65535 attempt.
Together with recv/dec/paint the frame interval is now fully
decomposable: interval ~= hdr_gap + recv + pend_age + dec + paint.
Also stamp TCP_WND/TCP_MSS/RECVMBOX into the client-connect log line
so every capture is self-labeled with the config it ran under.
|
|
The panel used to hold its last framebuffer during the connect→first-frame gap,
flashing a stale old frame before video appeared — more visible now that the
Scrypted side no longer sends a bridging snapshot. The wake path already paints
the Loading screen, but state_machine_set(AWAKE) is a no-op when the device is
already awake (e.g. a stream restart), so the clear didn't happen there.
Tie the clear to the stream connection instead: recv-task paints
local_screens_show_loading() once per new conn_id, guarded by the decoder lock
so the RGB565 present can't race a trailing decode-task paint, and gated on
AWAKE so it never draws on a sleeping panel.
|
|
handle_client previously ran recv → decode → paint serially on one
FreeRTOS task. The kernel TCP buffer filled during decode+paint
(~6ms), and against the IDF-default 5760-byte window the sender
naturally stop-go-rate-limited to ~consumption. Raising the window
to 65535 (previous experiment) regressed g2g from ~100ms to 17s
growing unbounded — the sender pumped 45+ segments per round into
a kernel buffer the app couldn't drain in time, and there was no
way to skip-oldest on the kernel queue.
This commit decouples recv from decode+paint:
recv-task: owns the socket. Reads header + body into one of three
preallocated PSRAM body buffers. On body complete, swaps
the just-filled buffer into a 1-deep pending slot and
picks a free buffer for the next recv. If the slot
already held a frame (decode is slow), drops oldest in
place — mirror of the Scrypted-side skip-oldest from
e5acf93.
decode-task: waits on a binary semaphore. On signal, claims pending,
then decodes + paints without holding any shared lock.
Frees its prior buffer implicitly by overwriting
s_decode_idx on the next claim.
3 PSRAM body buffers (~3MB of 28MB free) ensure the invariant
{recv_idx, pending_idx, decode_idx} are pairwise distinct without
ever blocking recv. jpeg_decoder.c grew an alloc_input_buffer helper
+ jpeg_decoder_decode now takes an explicit input pointer so the
stream and http_api snapshot paths don't share scratch.
New stats:
- recv_dropped_oldest: per-window count of pending-slot overwrites
- decode_idle_min/avg/max_us: time decode-task spent waiting on signal
Measurement at IDF-default 5760 window, Unifi medium substream:
before split: recv_avg=32ms recv_max~44ms fps=22-26 (recv blocked
during 6ms decode+paint; chunk_max capped at 5760)
after split: recv_avg=17ms recv_max=18-37ms fps=21-29 steady,
decode_idle_avg=27-40ms (decode mostly waiting),
drop_oldest=0, painted at source rate
The bottleneck moved from 'decode+paint serializes recv' to the
wire's own send rate. Bigger windows are now safe (recv-task drains
continuously, can't bury us), but won't add fps until source rate
goes up — that's a separate conversation.
|
|
call/chunk stats, SO_RCVBUF probe)
Per-frame samples aggregated over the existing 30-frame window:
- queued_at_body_start (FIONREAD just before body recv loop): how much
of the frame the kernel already absorbed during the previous
decode+paint. Close to jpeg_len → wire delivered the full frame
while we were busy (we're decode/paint-bound). Much smaller →
wire is throttled (window or buffer too small to absorb a frame
in our paint window).
- recv_calls: number of recv() syscalls the body read needed per
frame. High → small chunks → window-throttled sender.
- recv_chunk min/avg/max: bytes returned per recv() return in the
window. Avg = window body bytes / total syscalls.
- SO_RCVBUF: one-shot getsockopt at accept, logged and stashed in
stats. Confirms whether sdkconfig values reached the build —
TCP_WND_DEFAULT discrepancies are otherwise invisible.
All surfaced in the windowed log and in /state JSON alongside the
existing recv/dec/paint/idle stats. No behavior change yet.
|
|
frame age
Previous Phase 4 implementation only published last_paint_event_us_low
into the windowed-stats snapshot at every 30-frame roll (~1.5s at
20fps). Combined with the script's 5s /state poll cadence, the
"freshest frame age" we could compute was up to ~6.5s stale before
any actual lag — meaningless as a perf signal.
Update s_stats.last_paint_event_us_low under portMUX on every
painted v1 frame. The other window stats (recv/dec/paint/idle
min/avg/max + fps/MBps) keep their roll cadence because they
genuinely need a full window to compute; this single u32 just gets
overwritten with the most recent value each paint.
portENTER_CRITICAL on the ESP32-P4 is ~half a microsecond per side
— at 20fps that's 20µs/s of overhead, immeasurable next to the
30-40ms recv per frame.
Expected: g2g during a saturated stream drops from the previous
2-6s reading to single-digit hundreds of ms or lower, reflecting
the actual emit→display lag.
|
|
stream_server
Two pieces of documentation, neither changes behavior:
1. TESTING.md gains a "Performance review playbook" section: per-
session capture commands for firmware serial + /state poll +
Scrypted console, an annotated walkthrough of what each log
shape means, an investigation-threshold table mapping
user-facing symptoms to likely causes and the first thing to
check, and a one-line tools list. Replaces the implicit "ask the
maintainer how to debug" loop with a reproducible workflow.
2. stream_server.c gains a "Why TCP and not UDP" header comment
documenting the analysis from the design phase. JPEGs are ~123
IP datagrams at 1500 MTU; on hardwired Gigabit LAN switch-fabric
loss is < 1e-9/packet → per-frame corruption ≈ 1.2e-7. UDP's
theoretical wins (no Nagle, latest-wins semantics) don't apply
because TCP_NODELAY is on, socket.write p50 < 1ms, and the
FIONREAD trick already implements latest-wins on the receive
side. UDP's costs (200-400 LOC of app-layer fragmentation, loss
of nc/curl debug, FIONREAD trick stops working under
fragmentation) are real. Documented as reference so a future
contributor doesn't re-derive the analysis from scratch.
|
|
Wire format change: stream frames now carry a 4-byte "VPRT" magic +
4-byte jpeg_len + 4-byte seq + 4-byte event_us_low. Total 16 bytes
(was 8). The firmware sniffs the first 4 bytes per frame: if they
spell VPRT it reads the remaining 12 bytes of v1 header; otherwise
it interprets bytes 0-3 as jpeg_len for the old v0 8-byte format and
reads 4 more for seq. Lets a v1 firmware accept a v0 (legacy)
Scrypted script during the rollout window. v0 will be removed once
all field deployments roll forward.
event_us_low is the low 32 bits of the Scrypted host's monotonic µs
at camera-event arrival. The firmware does NOT interpret it (the
clocks aren't sync'd); it just stamps it on every painted frame and
exposes the most recent value via /state. The script polls /state
every 5s during an active stream, reads last_paint_event_us_low,
and computes glass-to-glass = (now_us_low - last_paint_event_us_low)
with 32-bit wrap. 30s sanity ceiling on the wrap to discard event
timestamps from before the stream started.
Also expose the firmware's just-closed 30-frame window stats via
/state under the "stream" key — frames, bytes, window_us, plus
min/avg/max for recv/dec/paint/idle. Lets external tools (a curl
loop, the Scrypted plugin, etc) poll the firmware's view without
parsing serial logs.
Firmware:
- stream_server.h: 16-byte v1 wire spec, stream_server_stats_t
struct, stream_server_snapshot_stats(out) getter.
- stream_server.c: magic-detect header read path, last_event_us_low
capture into per-connection state, portMUX-protected window-stats
snapshot at every 30-frame roll.
- http_api.c: GET /state JSON gains a "stream" sub-object with the
full snapshot.
- viewport_state.h: VIEWPORT_VERSION 1.0.0 → 1.1.0 (new /state shape).
Scrypted:
- startStream captures eventUsLow = (tEvent * 1000) >>> 0.
- TCP demux loop writes the 16-byte v1 header with the VPRT magic.
- New fwPoller setInterval (5s) fetches /state, parses .stream,
computes g2g, emits one summary line per poll cycle.
|
|
Removes/rewrites every comment that referenced the dead HTTP-streaming
architecture so a reader of v1.0.0+ source isn't chasing a model
that's been gone for several releases. No code paths changed; this is
the safe pre-pass before Phase 2's actual deletions.
Firmware (main/http_api.c):
- s_last_painted_seq / s_last_post_us / X-Frame-Seq parse / stale-
frame guard / Server-Timing emission / TCP_NODELAY setsockopt /
max_open_sockets=4 — each block now leads with "(Legacy from the
HTTP-streaming era; removed in Phase 2)" so the reader knows the
block is doomed, not load-bearing.
- /frame dim-mismatch comment narrowed: panel-native is always 800x480
BGR888, no Scrypted-side variation expected; Scrypted does the
rotation+scale via sharp/mediaManager/ffmpeg cascade (snapshot) or
ffmpeg -vf (stream).
- "Single in-flight frame. Concurrent posts get 503" rewritten to
reflect: decoder mutex now mostly serves to fence /frame snapshots
against an active stream_server decode.
Firmware (main/stream_server.c:136-142):
- FIONREAD-skip rationale reduced from a 7-line paragraph to one
sentence; the savings/tradeoff math now lives in the plan, not the
per-line comment.
Script (scrypted/scrypted-viewport.ts):
- Top-of-file tuning constants block drops the "frame_interval_ms
removed", "fps filter", "in-flight back-to-back startStream"
rationale; one short line covers the model: "Stream rate is paced
by camera + TCP backpressure; no app-level fps cap."
- agentFor() rationale rewritten: this Agent is over-engineered for
control-plane traffic (~1 POST/min steady state) — a legacy of when
it backed per-frame /frame POSTs. Marked for Phase 2 retirement.
- noDelay on Agent: clarified it's now a no-op safety for control
plane (was load-bearing for live-stream pipelining).
- snapshot fire-and-forget comment: replaced X-Frame-Seq-race
rationale with the actual TCP-streaming truth (sharp/mediaManager/
ffmpeg cascade race against stream socket bring-up).
- writeLatencies probe: rewrote the "keep-alive socket" comment
(live stream uses raw net.Socket, not the http.Agent pool).
- socketBackpressured: explicit "diagnostic only, never gates writes"
comment added at declaration site.
- skipLogger header: rewrote the inFlight/MAX_INFLIGHT/fps-filter
rationale into a one-line description of what the log line actually
emits today.
- frameSeq map: now flagged as legacy of HTTP-streaming era, retired
in Phase 2.
|
|
backpressure-blind + lwIP TCP window bump
Three changes that work together to make the stream "always paint
what's freshest, never sit on a stale frame":
#1 — Firmware FIONREAD skip in stream_server
Right after the body of frame N comes off the wire (and before we
unlock the decoder + spend ~6ms on decode + paint), check the
kernel receive buffer with ioctl(FIONREAD). If at least one more
header (8 bytes) is queued, frame N is no longer the freshest
possible — skip its decode + paint and loop back to read frame
N+1. The TCP recv cost is unavoidable (bytes still have to cross
the wire) but the decoder + paint cost is saved on every superseded
frame. Glass-to-glass latency on the latest frame drops by however
many frames had backed up.
#2 — Scrypted: keep writing past kernel-buffer backpressure
Previously: when sock.write() returned false we dropped the next
ffmpeg frame at source. New: we keep writing through. Node buffers
internally; under our load (~16 MB/s ffmpeg → ~5-7 MB/s firmware)
the buffer rarely exceeds a frame or two. With the firmware now
silently skipping decode on backed-up frames (#1), excess frames
get shed for free on the device side. Scrypted's job is just to
hand the firmware the freshest bytes as fast as possible.
socketBackpressured is still tracked for the diagnostic log.
#3 — lwIP TCP window bump (revisiting earlier regression)
The previous attempt at LWIP_TCP_WND_DEFAULT=32k regressed under
HTTP because every /frame opened a fresh socket and we paid the
slow-start cost repeatedly. The streaming pivot eliminated that:
the socket is long-lived, slow-start runs exactly once, then we
ride the full window for the rest of the session. Bumping to
65535 (max for stock lwIP), SND_BUF to match, RECVMBOX to 16, SACK
on. Expected: recv throughput ceiling moves up from ~5.3 MB/s,
which directly raises the fps ceiling (recv is currently 37ms of
the 43ms per-frame total).
|
|
Three independent improvements landing together because they all
target the post-streaming-pivot "where do we spend the wall clock?"
question.
#1 — Snapshot: sharp → mediaManager native → ffmpeg cascade
pushSnapshot now tries three transforms in order of cost:
- sharp (~5-15ms, libvips bindings, handles resize + rotate)
- mediaManager.convertMediaObjectToBuffer with image/jpeg;width=W
;height=H mime hint (~10-30ms, Scrypted's native converter,
used only for landscape since rotation isn't standard)
- ffmpeg one-shot (~500-700ms cold start, the old slow path)
A `path=...` field in the snapshot log identifies which transform
actually ran so the user can confirm the fast paths are reachable
in their Scrypted runtime. takePicture, transform, and POST timings
are each broken out so we can see exactly where the snapshot wall
goes.
#2 — Firmware: windowed min/avg/max breakdown + idle-gap
Stream server replaces the per-frame single-sample log with a 30-
frame window summary:
N frames over Xs: Yfps Z MB/s avg-jpeg=KB |
lock min/avg/max | recv min/avg/max | dec min/avg/max |
paint min/avg/max | idle min/avg/max
- lock = mutex acquire time (sanity check; should be ~0us with one
client owning the decoder)
- recv = body bytes off the wire
- dec = HW JPEG decode
- paint = backbuffer flip + DMA queue
- idle = gap between previous paint completing and next header
landing (= upstream slack). Large idle means we're waiting on
ffmpeg/network; near-zero means we're the bottleneck.
#3 — Camera substream picker
"Stream-source choice drives end-to-end latency more than anything
else" — the existing hardcoded low-latency-first walk lands on the
camera's preview substream which is typically capped at 5-8 fps.
Adds a per-viewport setting under Display:
Camera substream: auto | low-resolution | medium-resolution
| local | remote | remote-recorder
auto keeps the current behavior; the pinned options let the user
force a higher-fps source when they want stream rate > preview
rate. The chosen destination is logged at stream start.
|
|
Replaces the per-frame HTTP POST loop with a single long-lived TCP
connection on port 81. The HTTP control plane (/state, /config,
/frame for snapshot) stays unchanged.
Wire protocol (big-endian, repeating until connection close):
[4 bytes jpeg_len][4 bytes seq][jpeg_len bytes JPEG]
Why
---
The HTTP path hit a measured floor of ~37ms p50 with intermittent
~230ms p95 spikes that survived every Nagle/keep-alive fix attempt.
Each frame paid: TCP setup (or pool churn), HTTP parsing, body recv,
decode, paint, response write, response ACK. Streaming removes
everything except recv + decode + paint.
Firmware (main/stream_server.[ch], new)
---------------------------------------
- TCP listen on configurable port (81) in its own FreeRTOS task,
one client at a time (matches the one-stream-per-device model).
- Per accepted socket: TCP_NODELAY on, then loop reading 8-byte
header → jpeg body → through the existing jpeg_decoder + display
paths. Same stale-seq guard as the HTTP /frame handler (reset per
connection so each session starts at seq 1).
- Frames received while asleep are still drained (to stay framed)
but not painted. The HTTP control-plane POST /state {wake}
resumes painting on the next frame.
- Every 30 painted frames a structured serial log shows per-stage
timing + sustained MB/s — replaces the cross-side Server-Timing
header (no HTTP response to attach it to anymore).
Scrypted side
-------------
- net.createConnection({ host, port: 81, noDelay: true }) opened
once per stream session. On disconnect/error/close we auto-
reconnect after 500ms.
- ffmpeg stdout demux writes [header][body] directly to the
socket. sock.write() returning false sets a backpressured flag
that drops incoming ffmpeg frames until 'drain' fires — natural
TCP backpressure handles "firmware can't keep up" without us
modeling it manually.
- Stripped the entire fetch-based timing infrastructure
(pushStreamFrame, fetchSamples, parseServerTiming, depth
histogram, Server-Timing parser). Replaced with one stream-shaped
log every 10s: fps + MB/s + socket.write p50/p95/max + drop count
+ backpressured flag.
- pushSnapshot (one-shot first-paint) still uses the HTTP /frame
endpoint — small and infrequent, not worth reworking.
- postJSON still uses node:http for /state and /config.
State machine status flags
--------------------------
Extended boot-time flags array from 6 → 7 slots so the stream
server's bring-up shows up alongside ETH/MDNS/HTTP. Layout is now
E M H S D J T (was E M H D J T).
|
