luke/esp32-poe-scrypted-viewport/main, branch v1.0.0

release: v1.0.0 — streaming pivot complete

2026-06-20T15:41:08+00:00

First milestone where the device runs the way it was intended end- to-end: - Raw-TCP streaming data plane at ~14-20 fps painted, sub-ms socket writes, no Nagle/ACK pathology. - HTTP control plane (/state, /config) plus HTTP /frame for the one-shot snapshot fast path on wake. Snapshot via sharp → ~700ms first paint from a cold trigger (down from ~1000ms+ via ffmpeg). - Firmware-side "always paint the latest" — FIONREAD skip drops superseded frames before decode, keeping glass-to-glass tight even when upstream produces faster than we can ingest. - HW JPEG decode + zero-copy paint into the panel back framebuffer. - 12 fps goal hit cleanly at q:v 1 (visually lossless). Source pinned to medium-resolution substream so the camera supplies enough frames to actually fill the pipe. - Scrypted side: per-host node:http keep-alive Agent with NODELAY, the cascading sharp → mediaManager → ffmpeg snapshot transform, wake-trigger picker on the new-device dialog, and the leaked- setInterval-across-script-reloads bug squashed. Bumping VIEWPORT_VERSION 0.1.0 → 1.0.0 so the boot log and info screen reflect it.

"paint the latest, drop the rest" — FIONREAD skip + Scrypted backpressure-blind + lwIP TCP window bump

2026-06-20T15:30:24+00:00

Three changes that work together to make the stream "always paint what's freshest, never sit on a stale frame": #1 — Firmware FIONREAD skip in stream_server Right after the body of frame N comes off the wire (and before we unlock the decoder + spend ~6ms on decode + paint), check the kernel receive buffer with ioctl(FIONREAD). If at least one more header (8 bytes) is queued, frame N is no longer the freshest possible — skip its decode + paint and loop back to read frame N+1. The TCP recv cost is unavoidable (bytes still have to cross the wire) but the decoder + paint cost is saved on every superseded frame. Glass-to-glass latency on the latest frame drops by however many frames had backed up. #2 — Scrypted: keep writing past kernel-buffer backpressure Previously: when sock.write() returned false we dropped the next ffmpeg frame at source. New: we keep writing through. Node buffers internally; under our load (~16 MB/s ffmpeg → ~5-7 MB/s firmware) the buffer rarely exceeds a frame or two. With the firmware now silently skipping decode on backed-up frames (#1), excess frames get shed for free on the device side. Scrypted's job is just to hand the firmware the freshest bytes as fast as possible. socketBackpressured is still tracked for the diagnostic log. #3 — lwIP TCP window bump (revisiting earlier regression) The previous attempt at LWIP_TCP_WND_DEFAULT=32k regressed under HTTP because every /frame opened a fresh socket and we paid the slow-start cost repeatedly. The streaming pivot eliminated that: the socket is long-lived, slow-start runs exactly once, then we ride the full window for the rest of the session. Bumping to 65535 (max for stock lwIP), SND_BUF to match, RECVMBOX to 16, SACK on. Expected: recv throughput ceiling moves up from ~5.3 MB/s, which directly raises the fps ceiling (recv is currently 37ms of the 43ms per-frame total).

faster snapshot + deeper firmware timing + camera substream UI control

2026-06-20T15:02:38+00:00

Three independent improvements landing together because they all target the post-streaming-pivot "where do we spend the wall clock?" question. #1 — Snapshot: sharp → mediaManager native → ffmpeg cascade pushSnapshot now tries three transforms in order of cost: - sharp (~5-15ms, libvips bindings, handles resize + rotate) - mediaManager.convertMediaObjectToBuffer with image/jpeg;width=W ;height=H mime hint (~10-30ms, Scrypted's native converter, used only for landscape since rotation isn't standard) - ffmpeg one-shot (~500-700ms cold start, the old slow path) A `path=...` field in the snapshot log identifies which transform actually ran so the user can confirm the fast paths are reachable in their Scrypted runtime. takePicture, transform, and POST timings are each broken out so we can see exactly where the snapshot wall goes. #2 — Firmware: windowed min/avg/max breakdown + idle-gap Stream server replaces the per-frame single-sample log with a 30- frame window summary: N frames over Xs: Yfps Z MB/s avg-jpeg=KB | lock min/avg/max | recv min/avg/max | dec min/avg/max | paint min/avg/max | idle min/avg/max - lock = mutex acquire time (sanity check; should be ~0us with one client owning the decoder) - recv = body bytes off the wire - dec = HW JPEG decode - paint = backbuffer flip + DMA queue - idle = gap between previous paint completing and next header landing (= upstream slack). Large idle means we're waiting on ffmpeg/network; near-zero means we're the bottleneck. #3 — Camera substream picker "Stream-source choice drives end-to-end latency more than anything else" — the existing hardcoded low-latency-first walk lands on the camera's preview substream which is typically capped at 5-8 fps. Adds a per-viewport setting under Display: Camera substream: auto | low-resolution | medium-resolution | local | remote | remote-recorder auto keeps the current behavior; the pinned options let the user force a higher-fps source when they want stream rate > preview rate. The chosen destination is logged at stream start.

streaming pivot: raw TCP data plane, drop per-frame HTTP entirely

2026-06-20T00:59:14+00:00

Replaces the per-frame HTTP POST loop with a single long-lived TCP connection on port 81. The HTTP control plane (/state, /config, /frame for snapshot) stays unchanged. Wire protocol (big-endian, repeating until connection close): [4 bytes jpeg_len][4 bytes seq][jpeg_len bytes JPEG] Why --- The HTTP path hit a measured floor of ~37ms p50 with intermittent ~230ms p95 spikes that survived every Nagle/keep-alive fix attempt. Each frame paid: TCP setup (or pool churn), HTTP parsing, body recv, decode, paint, response write, response ACK. Streaming removes everything except recv + decode + paint. Firmware (main/stream_server.[ch], new) --------------------------------------- - TCP listen on configurable port (81) in its own FreeRTOS task, one client at a time (matches the one-stream-per-device model). - Per accepted socket: TCP_NODELAY on, then loop reading 8-byte header → jpeg body → through the existing jpeg_decoder + display paths. Same stale-seq guard as the HTTP /frame handler (reset per connection so each session starts at seq 1). - Frames received while asleep are still drained (to stay framed) but not painted. The HTTP control-plane POST /state {wake} resumes painting on the next frame. - Every 30 painted frames a structured serial log shows per-stage timing + sustained MB/s — replaces the cross-side Server-Timing header (no HTTP response to attach it to anymore). Scrypted side ------------- - net.createConnection({ host, port: 81, noDelay: true }) opened once per stream session. On disconnect/error/close we auto- reconnect after 500ms. - ffmpeg stdout demux writes [header][body] directly to the socket. sock.write() returning false sets a backpressured flag that drops incoming ffmpeg frames until 'drain' fires — natural TCP backpressure handles "firmware can't keep up" without us modeling it manually. - Stripped the entire fetch-based timing infrastructure (pushStreamFrame, fetchSamples, parseServerTiming, depth histogram, Server-Timing parser). Replaced with one stream-shaped log every 10s: fps + MB/s + socket.write p50/p95/max + drop count + backpressured flag. - pushSnapshot (one-shot first-paint) still uses the HTTP /frame endpoint — small and infrequent, not worth reworking. - postJSON still uses node:http for /state and /config. State machine status flags -------------------------- Extended boot-time flags array from 6 → 7 slots so the stream server's bring-up shows up alongside ETH/MDNS/HTTP. Layout is now E M H S D J T (was E M H D J T).

firmware: build fixes — esp_app_format component + forward-declare s_last_painted_seq

2026-06-20T00:31:23+00:00

Two build breaks discovered when actually compiling the prior commits: - main/CMakeLists.txt missing esp_app_format requirement, so esp_app_desc.h couldn't be resolved when app_main.c included it for the boot-time git-hash log. - s_last_painted_seq is referenced inside state_post_handler (reset to 0 on /state wake) but the static was declared further down the file alongside the other /frame state. C requires declaration before use — forward declare it above state_post_handler and keep a stub comment at the original location.

disable Nagle on /frame socket + add min/max/worst-frame to script log

2026-06-20T00:15:28+00:00

#1: TCP_NODELAY for /frame ESP-IDF lwIP defaults Nagle ON. /frame is the worst Nagle workload: ~140KB body POST followed by an empty 204 response, no follow-up data either way. Both the final partial-MTU body packet and the response packet can sit in the kernel send buffer up to 40ms waiting for an ACK that the other side is delaying-ACKing — silently adding tens of ms to every frame's wall time and showing up as a fat net_up bucket on the Scrypted side. setsockopt(TCP_NODELAY) at handler entry on every /frame. Cheap and idempotent. Includes the /state and /config sockets too — those are infrequent but small responses also benefit. #2: min/max + worst-frame decomposition in the Scrypted log The p50/p95 line answered "what's typical" but not "what happened in the worst frame this window". Now every 10-fetch log adds: - min and max columns alongside p50/p95 per bucket - a worst-frame breakdown row: identifies the single slowest fetch in the window and decomposes its wall time across all stages. Answers "did the slow frame get gated by emit→post (ffmpeg backed up), net_up (TCP/handshake spike), or fw_dec (large JPEG)?" directly, instead of us inferring from percentiles. Together these are the prerequisites for evaluating whether further optimization (HTTP keep-alive, chunked streaming) is worth pursuing.

unified end-to-end per-frame instrumentation + version stamps

2026-06-20T00:07:55+00:00

Cross-side timing was opaque: script saw "req=70ms" but couldn't split TCP/dispatch overhead from firmware decode time, and firmware serial logs and Scrypted console logs couldn't be correlated. Firmware /frame handler now emits Server-Timing on every response with per-stage breakdown: Server-Timing: recv;dur=X, dec;dur=Y, paint;dur=Z, post;dur=W, handle;dur=total Script parses it, joins by X-Frame-Seq implicitly (one POST per seq), derives net_up = req − fw_total, and logs a multi-line p50/p95 breakdown every 10 fetches: fetch "kitchen" #10 (jpeg=137KB) wall p50=65ms p95=140ms emit→post p50=0ms p95=2ms (queue wait) req p50=62ms p95=135ms (fetch → Response headers) net_up p50=43ms p95=110ms (TCP + body wire + dispatch) fw_recv p50=12ms p95=18ms (body off the wire) fw_dec p50=6ms p95=8ms (hardware JPEG) fw_paint p50=0.1ms p95=0.2ms (backbuffer flip) fw_post p50=0.4ms p95=0.6ms body-read p50=1ms (drain — empty body) inflight d0=8 d1=2 d2=0 stale-drops=0 Plus version stamps on both sides for the "is the user on the right code" question: - CMakeLists: PROJECT_VER = git short hash + -dirty marker if dirty. esp_app_get_description()->version exposes it at runtime. Boot log: "Scrypted Viewport boot (v0.1.0 build=4cf36e2-dirty)". - TS: SCRIPT_VERSION const at the top, bumped per commit, logged at script-eval: "Scrypted Viewport up (script=4cf36e2). Callback ..."

add X-Frame-Seq monotonic ordering for pipelined /frame POSTs

2026-06-15T14:32:57+00:00

With two /frame POSTs in flight on separate sockets, the firmware's JPEG decoder mutex serialises decode but FreeRTOS semaphore acquisition is not FIFO — under jitter the later-arriving older frame can grab the lock first and paint over the newer one, producing brief "panel travels backward in time" glitches. Scrypted side: - Per-viewport monotonic frameSeq counter, sent as X-Frame-Seq on every /frame POST. Reset on stopStream so each new stream starts at 1. Firmware side: - s_last_painted_seq tracks the highest seq we've painted. Frame arrives, mutex acquired, body received — if seq <= s_last_painted_seq the frame is dropped (200 OK + X-Frame-Drop: stale-seq header) and the mutex released without touching the back buffer. - s_last_painted_seq resets to 0 on POST /state {wake} so the next stream's seq=1 isn't rejected because the previous session reached a higher counter. - Missing/zero X-Frame-Seq (legacy clients) skips the check entirely — preserves pre-pipelining behaviour.

pipeline two /frame POSTs + HTTP keep-alive on the Scrypted side

2026-06-15T14:29:52+00:00

Decouples network upload from firmware decode-and-paint. Before, the inFlight boolean guard meant Scrypted sent frame N, waited for the full ~80ms response (~40ms body upload + ~20ms decode + ~50µs paint + ack), THEN sent frame N+1. The next ffmpeg frame arriving during that window got dropped. After: - ScryptedViewportProvider keeps a per-host undici Agent with keepAliveTimeout 30s, connections:2, pipelining:0. Sockets stay open across /frame, /state, /config — saves the SYN+SYN-ACK+ACK round-trip every POST (small on LAN but real on Wi-Fi). - Stream loop's inFlight boolean is now a counter capped at 2 so frame N+1 can begin uploading on socket B while frame N is still being decoded on the device via socket A. Roughly doubles effective throughput when body upload time dominates. Firmware side: - esp_http_server max_open_sockets bumped 7→4 explicitly: 2 for pipelined /frame + 2 spare for concurrent /state and /config slots. The JPEG decoder mutex still serialises decode (only one /frame can be decoding at a time); this change only unblocks the network half of the pipeline.

instrumentation: firmware idle gap between frames + Scrypted per-fetch wall-clock

2026-06-15T13:25:02+00:00