src.nth.io/

summaryrefslogtreecommitdiff
path: root/README.md
diff options
context:
space:
mode:
authorLuke Hoersten <[email protected]>2026-06-15 07:41:45 -0500
committerLuke Hoersten <[email protected]>2026-06-15 07:41:45 -0500
commitb3217b9dbb731ec69cd1ebde201aa2178143d961 (patch)
tree99dc2c4443abeb8cd55e3c277252ace82a211d3c /README.md
parentdef48e1130a46d16b89794bafa78592dbe146184 (diff)
scrypted: don't reset idle timer on each painted frame + finer timing
Two fixes plus a README refresh: 1. scrypted: pushStreamFrame previously reset the per-stream idle timer on every successful /frame response. That made the timer anchored to "frames are flowing" rather than to "the camera event that triggered the stream", so a continuously-streaming source would never let the stream time out. Removed the reset. The startStream → stopStream(false) cancel-and-replace path on repeated events still keeps the stream alive while the event keeps firing; idle (no new events) now actually ends the stream at idle_timeout_ms. 2. firmware: break the previous coarse recv/dec/paint timing into lock : try_lock returned ttfb : first httpd_req_recv chunk landed body : remaining bytes received dec : hardware JPEG decode paint : esp_lcd_panel_draw_bitmap returned post : state-counter bookkeeping + unlock Logged every 10 frames at INFO. Splits the previously-fat recv bucket into TCP/HTTP handshake overhead (ttfb) vs wire-time (body), and surfaces any tail bookkeeping cost. 3. README: replace the stale "5 fps ceiling caused by CPU RGB conversion" guess with the actual measured per-phase budget and re-rank the backlog accordingly. Double-buffering the panel (paint 24 ms → ~2 ms) is now the highest-value next move; the previously-listed DMA-2D rewrite is moot because the CPU loop is already gone.
Diffstat (limited to 'README.md')
-rw-r--r--README.md40
1 files changed, 29 insertions, 11 deletions
diff --git a/README.md b/README.md
index 26e61c1..41cb4e1 100644
--- a/README.md
+++ b/README.md
@@ -490,17 +490,35 @@ Every endpoint is idempotent; every failure leaves the device in a sane state.
## What's next
-M1 – M8 are all ✅ on hardware (see [`TESTING.md`](TESTING.md) for verification details). End-to-end Scrypted streaming via ffmpeg is live and sustained at ~5 fps with high-quality JPEGs.
-
-Current backlog, in rough priority order:
-
-1. **Firmware fps optimisation via DMA-2D** — high impact. The 5-fps ceiling is `display_present_rgb565`: software RGB565→BGR888 conversion + portrait rotation on the CPU is ~1.15 MB of PSRAM-bound copy per frame. The ESP32-P4 DMA-2D engine can do both pixel-format conversion *and* rotation in hardware. Reconfigure the DPI panel for RGB565 input + `flags.use_dma2d = true`, hand the JPEG decoder's RGB565 output straight to the panel driver, let hardware do the rotation. Should ~double painted fps.
-2. **OTA firmware updates** — low effort, gateway to fleet ops. Partition table already has `ota_0` / `ota_1`. Add `esp_https_ota` (or a one-shot `POST /firmware` using `esp_ota_*`) so reflashing doesn't require USB. Critical once you have more than one viewport in production.
-3. **Task watchdog + crash counters** — low effort. Enable the ESP-IDF task watchdog, surface its bite count in `/state` alongside the existing `decode_errors` / `state_post_failures`. Good hygiene.
-4. **Multi-camera per viewport** — medium effort. Let one viewport listen to events from N cameras, picking which one to stream based on which fired. Useful for "show whichever doorbell rang" or zone monitoring.
-5. **MJPEG-over-WebSocket** instead of `POST /frame` per JPEG — medium effort, marginal win. Would shave per-frame HTTP overhead, but the firmware decode-paint mutex is the ceiling so the math doesn't change much in practice. Skip unless we find a use case the per-frame path can't cover.
-6. **Boot info-screen flash polish** — low effort. Keep the brief wake-on-boot (the FT5426 needs it to start reporting touches) but smoothen the ~600 ms flash so it doesn't visibly flicker on power-up.
-7. **Production sealing** — eventual. Configurable LAN scope (cross-VLAN, mDNS-via-Unicast), Scrypted-side mutual auth, replay protection for `/state` callbacks.
+M1 – M8 are all ✅ on hardware (see [`TESTING.md`](TESTING.md) for verification details). End-to-end Scrypted streaming via ffmpeg + the zero-copy `JPEG → BGR888 → DSI` hot path is live and sustains **~6.5 fps painted at the 100 ms target tick**, gated by per-frame work below.
+
+### Measured per-frame budget (target ≥ 22 fps requires shrinking these)
+
+The firmware logs a `frame N: lock=… ttfb=… body=… dec=… paint=… post=… total=…ms` line every 10 frames. Steady-state on the bench (Waveshare ESP32-P4-ETH + Hosyond 5" panel, ~210 KB JPEGs at `-q:v 2`):
+
+| Phase | Time | What it is |
+|---|---|---|
+| `lock` | ~tens of µs | `jpeg_decoder_try_lock` (mutex acquire) |
+| `ttfb` | ~1–2 ms | Time-to-first-byte after lock — TCP/HTTP handshake overhead |
+| `body` | ~38 ms | Wire time for the rest of the JPEG body (~210 KB at ~42 Mbit/s effective) |
+| `dec` | ~6 ms | Hardware JPEG decode → BGR888 (way faster than the textbook 50–80 ms guess) |
+| `paint` | ~24 ms | `esp_lcd_panel_draw_bitmap` + ~1.5 DSI refresh cycles at 60 Hz |
+| `post` | < 1 ms | State counter bookkeeping + unlock |
+| **total** | **~70 ms** | **Theoretical ceiling ~14 fps if Scrypted ticks faster than the 100 ms default** |
+
+So the firmware ceiling at full quality is ~14 fps, but Scrypted's `frame_interval_ms = 100` only fires 10 ticks/sec, leaving ~30 ms idle between frames → effective ~6.5 fps. Lower the per-viewport interval to ~75 ms to chase the ceiling.
+
+### Current backlog, in rough priority order
+
+1. **Double-buffer the panel (`num_fbs = 2`)** — *high impact*. Today `paint` is 24 ms because the synchronous `esp_lcd_panel_draw_bitmap` waits for a full DSI refresh cycle. With a second framebuffer the DSI engine streams from FB-A while the next frame fills FB-B; paint becomes a pointer swap (~1–2 ms). Total drops to ~46 ms, ceiling jumps to ~22 fps at full quality. Watch out for the TC358762 bridge's `non-burst with sync pulses` constraint — `flags.use_dma2d` and `num_fbs` interact non-obviously with the bridge wake sequence, so this needs careful re-bring-up.
+2. **HTTP persistent connection** — *medium impact, low effort*. Would shave the ~1–2 ms `ttfb` per frame by keeping the TCP connection open across frames (currently Scrypted's `fetch` opens fresh each POST). Real win is consistency, not absolute speed; useful once #1 lands.
+3. **OTA firmware updates** — *low effort, gateway to fleet ops*. Partition table already has `ota_0` / `ota_1`. Add `esp_https_ota` (or a one-shot `POST /firmware` using `esp_ota_*`) so reflashing doesn't require USB. Critical once you have more than one viewport in production.
+4. **Task watchdog + crash counters** — *low effort*. Enable the ESP-IDF task watchdog, surface its bite count in `/state` alongside the existing `decode_errors` / `state_post_failures`. Good hygiene.
+5. **Multi-camera per viewport** — *medium effort*. Let one viewport listen to events from N cameras, picking which one to stream based on which fired. Useful for "show whichever doorbell rang" or zone monitoring.
+6. **Per-viewport JPEG quality knob** — *trivial*. Add a `jpeg_quality` setting that drives ffmpeg's `-q:v` arg. Trade fidelity for fps if anyone wants. Only matters once we want >22 fps from a single viewport.
+7. **MJPEG-over-WebSocket** instead of `POST /frame` per JPEG — *medium effort, marginal win*. Would shave per-frame HTTP overhead but the firmware decode-paint mutex is the new ceiling. Skip unless we find a use case the per-frame path can't cover.
+8. **Boot info-screen flash polish** — *low effort*. Keep the brief wake-on-boot (the FT5426 needs it to start reporting touches) but smoothen the ~600 ms flash so it doesn't visibly flicker on power-up.
+9. **Production sealing** — *eventual*. Configurable LAN scope (cross-VLAN, mDNS-via-Unicast), Scrypted-side mutual auth, replay protection for `/state` callbacks.
## Philosophy