src.nth.io/

summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorLuke Hoersten <[email protected]>2026-06-20 11:40:26 -0500
committerLuke Hoersten <[email protected]>2026-06-20 11:40:26 -0500
commit6e0e0270e2cb9c54b1b7e63d9e67d8ac16792cef (patch)
tree3cc4d87048351ea8ebfb214e2d279238c5f4a916
parent431a3268802df15781a52ae47dcaf5081b3f98f4 (diff)
phase 6: performance review playbook in TESTING.md + UDP-vs-TCP rationale in stream_server
Two pieces of documentation, neither changes behavior: 1. TESTING.md gains a "Performance review playbook" section: per- session capture commands for firmware serial + /state poll + Scrypted console, an annotated walkthrough of what each log shape means, an investigation-threshold table mapping user-facing symptoms to likely causes and the first thing to check, and a one-line tools list. Replaces the implicit "ask the maintainer how to debug" loop with a reproducible workflow. 2. stream_server.c gains a "Why TCP and not UDP" header comment documenting the analysis from the design phase. JPEGs are ~123 IP datagrams at 1500 MTU; on hardwired Gigabit LAN switch-fabric loss is < 1e-9/packet → per-frame corruption ≈ 1.2e-7. UDP's theoretical wins (no Nagle, latest-wins semantics) don't apply because TCP_NODELAY is on, socket.write p50 < 1ms, and the FIONREAD trick already implements latest-wins on the receive side. UDP's costs (200-400 LOC of app-layer fragmentation, loss of nc/curl debug, FIONREAD trick stops working under fragmentation) are real. Documented as reference so a future contributor doesn't re-derive the analysis from scratch.
-rw-r--r--TESTING.md91
-rw-r--r--main/stream_server.c23
2 files changed, 114 insertions, 0 deletions
diff --git a/TESTING.md b/TESTING.md
index bca9479..e3b738e 100644
--- a/TESTING.md
+++ b/TESTING.md
@@ -638,3 +638,94 @@ Run these once all milestones are implemented and individually verified. The poi
### G. Multi-viewport
Once at least two physical units are configured: confirm each routes its callback to Scrypted with its own `viewport` name; confirm Scrypted's discovery map has both IPs; confirm a camera event on the wrong-named viewport is correctly ignored on the right one.
+
+## Performance review playbook
+
+Repeatable methodology for "is the device still hitting its baseline" and "where did the time go." Run after every commit that touches the data plane.
+
+### Per-session capture
+
+Three log streams, all timestamped:
+
+```sh
+# Firmware serial — 30-frame window summaries
+idf.py monitor | tee fw-$(date +%s).log &
+
+# /state poll — windowed stats + glass-to-glass anchor
+while true; do
+ curl -s http://VIEWPORT_IP/state
+ echo
+ sleep 1
+done | tee state-$(date +%s).log &
+
+# Scrypted plugin console — copy/paste from the web UI when done.
+```
+
+Trigger a wake event (motion or "Wake now" button). Let it run two minutes. Stop captures.
+
+### What the firmware serial says
+
+Every 30 painted frames:
+
+```
+30 frames over 1.8s: 16.5fps 2.95MB/s avg-jpeg=180KB |
+ lock min/avg/max=8/9/12us |
+ recv min/avg/max=21000/35000/48000us |
+ dec min/avg/max=5100/5600/6200us |
+ paint min/avg/max=38/45/61us |
+ idle min/avg/max=120/8500/40000us
+```
+
+### What `/state` says (Phase 4 onward)
+
+```json
+{
+ "stream": {
+ "frames": 30, "bytes": 5400000, "window_us": 1800000,
+ "recv_min_us":21000,"recv_avg_us":35000,"recv_max_us":48000,
+ "dec_min_us":5100,"dec_avg_us":5600,"dec_max_us":6200,
+ "paint_min_us":38,"paint_avg_us":45,"paint_max_us":61,
+ "idle_min_us":120,"idle_avg_us":8500,"idle_max_us":40000,
+ "last_paint_event_us_low": 2148503712
+ }
+}
+```
+
+`last_paint_event_us_low` is the script's monotonic µs low 32 bits at the wake event that produced the most recent painted frame. Glass-to-glass is `(scripts_now_us_low - last_paint_event_us_low)` with 32-bit wrap.
+
+### What the Scrypted console says
+
+On wake:
+```
+event MotionSensor -> "kitchen": fired at +0ms (wake)
+stream "kitchen": start +0ms
+stream "kitchen": socket connect requested +1ms
+snapshot "kitchen": start +1ms
+snapshot "kitchen": takePicture +320ms
+stream "kitchen": socket connect open +24ms
+snapshot "kitchen": transform +458ms via sharp (217KB)
+snapshot "kitchen": post sent +459ms
+snapshot "kitchen": post acked +551ms ← first user-visible paint
+stream "kitchen": first ffmpeg frame +780ms (jpeg=210KB)
+stream "kitchen": first socket.write +781ms
+```
+
+Every 5s during an active stream:
+```
+firmware "kitchen": fps=16.7 2.99MB/s | recv=...us | dec=...us | paint=...us | idle=...us | g2g=140ms
+```
+
+### Investigation thresholds (alarms)
+
+| Symptom | Probable cause | First check |
+|---|---|---|
+| Painted fps sustained < 12 | Upstream starvation | `idle_avg_us` in firmware log — high = ffmpeg behind |
+| `decode_p95 > 8ms` | HW decoder pathology | Recent `decode_errors` delta; PSRAM cache pressure |
+| `paint_p95 > 200µs` | DPI or DMA stall | Look for `DSI` warnings in serial; PSRAM bandwidth |
+| `idle_avg > 50ms` | ffmpeg falling behind | Check Scrypted CPU; try lower-rate camera substream |
+| `g2g_p95 > 500ms` | User-facing regression | Bisect: is the gap in script-side or firmware-side? |
+| Any non-zero `decode_errors` delta | JPEG corruption | TCP retransmit storm or memory corruption |
+
+### Tools
+
+`grep`, `awk`, `gnuplot`. Anything beyond that is over-engineering for a single device's logs.
diff --git a/main/stream_server.c b/main/stream_server.c
index c417e4a..1321f22 100644
--- a/main/stream_server.c
+++ b/main/stream_server.c
@@ -18,6 +18,29 @@
static const char *TAG = "stream";
+// Why TCP and not UDP.
+//
+// JPEGs at panel-native q:v 1 are ~180 KB → ~123 IP datagrams at 1500
+// MTU. On hardwired Gigabit LAN with a single managed switch, fabric
+// loss is < 1e-9/packet, so per-frame corruption is ≈1.2e-7 (~1 bad
+// frame every 95 days at 30 fps). UDP's theoretical wins don't apply
+// to this deployment:
+//
+// - Nagle/SACK/window-scaling overhead: socket.write p50 < 1ms in
+// steady state (see Scrypted-side log). TCP_NODELAY is on. No
+// per-frame ACK round-trip exists in either direction.
+// - "Latest wins" semantics: implemented for free above via the
+// FIONREAD check below. UDP would have to reassemble fragments
+// first before deciding to skip, defeating the latency win.
+//
+// UDP's costs would be real: 200-400 LOC of app-layer fragmentation
+// + reassembly + partial-frame timeout, loss of `nc localhost 81 |
+// xxd` for debug, and the FIONREAD-skip trick stops working because
+// UDP recvmsg counts fragments not whole messages.
+//
+// Revisit only if a Wi-Fi-only variant ships (loss rate 1e-3..1e-5
+// would make TCP retransmits painful enough to consider the rewrite).
+
#define HEADER_V0_BYTES 8 // legacy: jpeg_len + seq
#define HEADER_V1_BYTES 16 // current: magic + jpeg_len + seq + event_us_low
#define HEADER_BYTES HEADER_V0_BYTES // used for FIONREAD threshold —