diff options
| author | Luke Hoersten <[email protected]> | 2026-06-20 11:40:26 -0500 |
|---|---|---|
| committer | Luke Hoersten <[email protected]> | 2026-06-20 11:40:26 -0500 |
| commit | 6e0e0270e2cb9c54b1b7e63d9e67d8ac16792cef (patch) | |
| tree | 3cc4d87048351ea8ebfb214e2d279238c5f4a916 /TESTING.md | |
| parent | 431a3268802df15781a52ae47dcaf5081b3f98f4 (diff) | |
phase 6: performance review playbook in TESTING.md + UDP-vs-TCP rationale in stream_server
Two pieces of documentation, neither changes behavior:
1. TESTING.md gains a "Performance review playbook" section: per-
session capture commands for firmware serial + /state poll +
Scrypted console, an annotated walkthrough of what each log
shape means, an investigation-threshold table mapping
user-facing symptoms to likely causes and the first thing to
check, and a one-line tools list. Replaces the implicit "ask the
maintainer how to debug" loop with a reproducible workflow.
2. stream_server.c gains a "Why TCP and not UDP" header comment
documenting the analysis from the design phase. JPEGs are ~123
IP datagrams at 1500 MTU; on hardwired Gigabit LAN switch-fabric
loss is < 1e-9/packet → per-frame corruption ≈ 1.2e-7. UDP's
theoretical wins (no Nagle, latest-wins semantics) don't apply
because TCP_NODELAY is on, socket.write p50 < 1ms, and the
FIONREAD trick already implements latest-wins on the receive
side. UDP's costs (200-400 LOC of app-layer fragmentation, loss
of nc/curl debug, FIONREAD trick stops working under
fragmentation) are real. Documented as reference so a future
contributor doesn't re-derive the analysis from scratch.
Diffstat (limited to 'TESTING.md')
| -rw-r--r-- | TESTING.md | 91 |
1 files changed, 91 insertions, 0 deletions
@@ -638,3 +638,94 @@ Run these once all milestones are implemented and individually verified. The poi ### G. Multi-viewport Once at least two physical units are configured: confirm each routes its callback to Scrypted with its own `viewport` name; confirm Scrypted's discovery map has both IPs; confirm a camera event on the wrong-named viewport is correctly ignored on the right one. + +## Performance review playbook + +Repeatable methodology for "is the device still hitting its baseline" and "where did the time go." Run after every commit that touches the data plane. + +### Per-session capture + +Three log streams, all timestamped: + +```sh +# Firmware serial — 30-frame window summaries +idf.py monitor | tee fw-$(date +%s).log & + +# /state poll — windowed stats + glass-to-glass anchor +while true; do + curl -s http://VIEWPORT_IP/state + echo + sleep 1 +done | tee state-$(date +%s).log & + +# Scrypted plugin console — copy/paste from the web UI when done. +``` + +Trigger a wake event (motion or "Wake now" button). Let it run two minutes. Stop captures. + +### What the firmware serial says + +Every 30 painted frames: + +``` +30 frames over 1.8s: 16.5fps 2.95MB/s avg-jpeg=180KB | + lock min/avg/max=8/9/12us | + recv min/avg/max=21000/35000/48000us | + dec min/avg/max=5100/5600/6200us | + paint min/avg/max=38/45/61us | + idle min/avg/max=120/8500/40000us +``` + +### What `/state` says (Phase 4 onward) + +```json +{ + "stream": { + "frames": 30, "bytes": 5400000, "window_us": 1800000, + "recv_min_us":21000,"recv_avg_us":35000,"recv_max_us":48000, + "dec_min_us":5100,"dec_avg_us":5600,"dec_max_us":6200, + "paint_min_us":38,"paint_avg_us":45,"paint_max_us":61, + "idle_min_us":120,"idle_avg_us":8500,"idle_max_us":40000, + "last_paint_event_us_low": 2148503712 + } +} +``` + +`last_paint_event_us_low` is the script's monotonic µs low 32 bits at the wake event that produced the most recent painted frame. Glass-to-glass is `(scripts_now_us_low - last_paint_event_us_low)` with 32-bit wrap. + +### What the Scrypted console says + +On wake: +``` +event MotionSensor -> "kitchen": fired at +0ms (wake) +stream "kitchen": start +0ms +stream "kitchen": socket connect requested +1ms +snapshot "kitchen": start +1ms +snapshot "kitchen": takePicture +320ms +stream "kitchen": socket connect open +24ms +snapshot "kitchen": transform +458ms via sharp (217KB) +snapshot "kitchen": post sent +459ms +snapshot "kitchen": post acked +551ms ← first user-visible paint +stream "kitchen": first ffmpeg frame +780ms (jpeg=210KB) +stream "kitchen": first socket.write +781ms +``` + +Every 5s during an active stream: +``` +firmware "kitchen": fps=16.7 2.99MB/s | recv=...us | dec=...us | paint=...us | idle=...us | g2g=140ms +``` + +### Investigation thresholds (alarms) + +| Symptom | Probable cause | First check | +|---|---|---| +| Painted fps sustained < 12 | Upstream starvation | `idle_avg_us` in firmware log — high = ffmpeg behind | +| `decode_p95 > 8ms` | HW decoder pathology | Recent `decode_errors` delta; PSRAM cache pressure | +| `paint_p95 > 200µs` | DPI or DMA stall | Look for `DSI` warnings in serial; PSRAM bandwidth | +| `idle_avg > 50ms` | ffmpeg falling behind | Check Scrypted CPU; try lower-rate camera substream | +| `g2g_p95 > 500ms` | User-facing regression | Bisect: is the gap in script-side or firmware-side? | +| Any non-zero `decode_errors` delta | JPEG corruption | TCP retransmit storm or memory corruption | + +### Tools + +`grep`, `awk`, `gnuplot`. Anything beyond that is over-engineering for a single device's logs. |
