diff options
| author | Luke Hoersten <[email protected]> | 2026-06-20 11:40:26 -0500 |
|---|---|---|
| committer | Luke Hoersten <[email protected]> | 2026-06-20 11:40:26 -0500 |
| commit | 6e0e0270e2cb9c54b1b7e63d9e67d8ac16792cef (patch) | |
| tree | 3cc4d87048351ea8ebfb214e2d279238c5f4a916 | |
| parent | 431a3268802df15781a52ae47dcaf5081b3f98f4 (diff) | |
phase 6: performance review playbook in TESTING.md + UDP-vs-TCP rationale in stream_server
Two pieces of documentation, neither changes behavior:
1. TESTING.md gains a "Performance review playbook" section: per-
session capture commands for firmware serial + /state poll +
Scrypted console, an annotated walkthrough of what each log
shape means, an investigation-threshold table mapping
user-facing symptoms to likely causes and the first thing to
check, and a one-line tools list. Replaces the implicit "ask the
maintainer how to debug" loop with a reproducible workflow.
2. stream_server.c gains a "Why TCP and not UDP" header comment
documenting the analysis from the design phase. JPEGs are ~123
IP datagrams at 1500 MTU; on hardwired Gigabit LAN switch-fabric
loss is < 1e-9/packet → per-frame corruption ≈ 1.2e-7. UDP's
theoretical wins (no Nagle, latest-wins semantics) don't apply
because TCP_NODELAY is on, socket.write p50 < 1ms, and the
FIONREAD trick already implements latest-wins on the receive
side. UDP's costs (200-400 LOC of app-layer fragmentation, loss
of nc/curl debug, FIONREAD trick stops working under
fragmentation) are real. Documented as reference so a future
contributor doesn't re-derive the analysis from scratch.
| -rw-r--r-- | TESTING.md | 91 | ||||
| -rw-r--r-- | main/stream_server.c | 23 |
2 files changed, 114 insertions, 0 deletions
@@ -638,3 +638,94 @@ Run these once all milestones are implemented and individually verified. The poi ### G. Multi-viewport Once at least two physical units are configured: confirm each routes its callback to Scrypted with its own `viewport` name; confirm Scrypted's discovery map has both IPs; confirm a camera event on the wrong-named viewport is correctly ignored on the right one. + +## Performance review playbook + +Repeatable methodology for "is the device still hitting its baseline" and "where did the time go." Run after every commit that touches the data plane. + +### Per-session capture + +Three log streams, all timestamped: + +```sh +# Firmware serial — 30-frame window summaries +idf.py monitor | tee fw-$(date +%s).log & + +# /state poll — windowed stats + glass-to-glass anchor +while true; do + curl -s http://VIEWPORT_IP/state + echo + sleep 1 +done | tee state-$(date +%s).log & + +# Scrypted plugin console — copy/paste from the web UI when done. +``` + +Trigger a wake event (motion or "Wake now" button). Let it run two minutes. Stop captures. + +### What the firmware serial says + +Every 30 painted frames: + +``` +30 frames over 1.8s: 16.5fps 2.95MB/s avg-jpeg=180KB | + lock min/avg/max=8/9/12us | + recv min/avg/max=21000/35000/48000us | + dec min/avg/max=5100/5600/6200us | + paint min/avg/max=38/45/61us | + idle min/avg/max=120/8500/40000us +``` + +### What `/state` says (Phase 4 onward) + +```json +{ + "stream": { + "frames": 30, "bytes": 5400000, "window_us": 1800000, + "recv_min_us":21000,"recv_avg_us":35000,"recv_max_us":48000, + "dec_min_us":5100,"dec_avg_us":5600,"dec_max_us":6200, + "paint_min_us":38,"paint_avg_us":45,"paint_max_us":61, + "idle_min_us":120,"idle_avg_us":8500,"idle_max_us":40000, + "last_paint_event_us_low": 2148503712 + } +} +``` + +`last_paint_event_us_low` is the script's monotonic µs low 32 bits at the wake event that produced the most recent painted frame. Glass-to-glass is `(scripts_now_us_low - last_paint_event_us_low)` with 32-bit wrap. + +### What the Scrypted console says + +On wake: +``` +event MotionSensor -> "kitchen": fired at +0ms (wake) +stream "kitchen": start +0ms +stream "kitchen": socket connect requested +1ms +snapshot "kitchen": start +1ms +snapshot "kitchen": takePicture +320ms +stream "kitchen": socket connect open +24ms +snapshot "kitchen": transform +458ms via sharp (217KB) +snapshot "kitchen": post sent +459ms +snapshot "kitchen": post acked +551ms ← first user-visible paint +stream "kitchen": first ffmpeg frame +780ms (jpeg=210KB) +stream "kitchen": first socket.write +781ms +``` + +Every 5s during an active stream: +``` +firmware "kitchen": fps=16.7 2.99MB/s | recv=...us | dec=...us | paint=...us | idle=...us | g2g=140ms +``` + +### Investigation thresholds (alarms) + +| Symptom | Probable cause | First check | +|---|---|---| +| Painted fps sustained < 12 | Upstream starvation | `idle_avg_us` in firmware log — high = ffmpeg behind | +| `decode_p95 > 8ms` | HW decoder pathology | Recent `decode_errors` delta; PSRAM cache pressure | +| `paint_p95 > 200µs` | DPI or DMA stall | Look for `DSI` warnings in serial; PSRAM bandwidth | +| `idle_avg > 50ms` | ffmpeg falling behind | Check Scrypted CPU; try lower-rate camera substream | +| `g2g_p95 > 500ms` | User-facing regression | Bisect: is the gap in script-side or firmware-side? | +| Any non-zero `decode_errors` delta | JPEG corruption | TCP retransmit storm or memory corruption | + +### Tools + +`grep`, `awk`, `gnuplot`. Anything beyond that is over-engineering for a single device's logs. diff --git a/main/stream_server.c b/main/stream_server.c index c417e4a..1321f22 100644 --- a/main/stream_server.c +++ b/main/stream_server.c @@ -18,6 +18,29 @@ static const char *TAG = "stream"; +// Why TCP and not UDP. +// +// JPEGs at panel-native q:v 1 are ~180 KB → ~123 IP datagrams at 1500 +// MTU. On hardwired Gigabit LAN with a single managed switch, fabric +// loss is < 1e-9/packet, so per-frame corruption is ≈1.2e-7 (~1 bad +// frame every 95 days at 30 fps). UDP's theoretical wins don't apply +// to this deployment: +// +// - Nagle/SACK/window-scaling overhead: socket.write p50 < 1ms in +// steady state (see Scrypted-side log). TCP_NODELAY is on. No +// per-frame ACK round-trip exists in either direction. +// - "Latest wins" semantics: implemented for free above via the +// FIONREAD check below. UDP would have to reassemble fragments +// first before deciding to skip, defeating the latency win. +// +// UDP's costs would be real: 200-400 LOC of app-layer fragmentation +// + reassembly + partial-frame timeout, loss of `nc localhost 81 | +// xxd` for debug, and the FIONREAD-skip trick stops working because +// UDP recvmsg counts fragments not whole messages. +// +// Revisit only if a Wi-Fi-only variant ships (loss rate 1e-3..1e-5 +// would make TCP retransmits painful enough to consider the rewrite). + #define HEADER_V0_BYTES 8 // legacy: jpeg_len + seq #define HEADER_V1_BYTES 16 // current: magic + jpeg_len + seq + event_us_low #define HEADER_BYTES HEADER_V0_BYTES // used for FIONREAD threshold — |
