diff options
| -rw-r--r-- | README.md | 447 | ||||
| -rw-r--r-- | TESTING.md | 849 |
2 files changed, 477 insertions, 819 deletions
@@ -1,12 +1,19 @@ -# Scrypted Viewport v1 Technical Design Specification +# ESP32 PoE Scrypted Viewport -Version: 1.0 +A PoE-powered ambient camera display for [Scrypted](https://www.scrypted.app/): an ESP32-P4 driving a Raspberry Pi +7"-class DSI touch panel with live doorbell/camera video. Ring the doorbell and the panel is showing live video in ~0.7 +s, sustaining 24 fps with double-digit-millisecond glass-to-glass latency — then it sleeps itself until the next event +or tap. + +- **Write-up:** <https://nth.io/luke/projects/esp32-ethernet-display/> +- **Canonical source:** <https://src.nth.io/luke/esp32-poe-scrypted-viewport/> +- **Mirror:** <https://github.com/lukehoersten/esp32-poe-scrypted-viewport> ## Overview -Scrypted Viewport is an Ethernet-powered ambient display appliance optimized for Scrypted camera and doorbell -events. It's meant to be plug and play onto a trusted POE VLAN connection with Scrypted access so there's no -configuration done on the esp32 itself and instead is discoverd and configured within Scrypted. +Scrypted Viewport is an Ethernet-powered ambient display appliance optimized for Scrypted camera and doorbell events. +It's meant to be plug and play on a trusted PoE VLAN with Scrypted access: no configuration happens on the ESP32 itself +— devices are discovered over mDNS and configured entirely from within Scrypted. Design goals: - No Matter @@ -23,9 +30,13 @@ Scrypted Viewport owns Ethernet, JPEG decode, display, touch input and outbound ## Related docs -- [`TESTING.md`](TESTING.md) — **the self-contained verification reference.** Status snapshot for every milestone, hardware prerequisites + open unknowns, bench-order playbook, per-milestone `curl` recipes, integration test suite. Start here before flashing anything. -- [`scrypted/README.md`](scrypted/README.md) — Scrypted-side script install + per-viewport binding UI (camera picker, wake triggers, mDNS-discovered host dropdowns). -- [`scrypted/PLUGIN-CONVERSION.md`](scrypted/PLUGIN-CONVERSION.md) — TODO: plan for repackaging the script as a real installable Scrypted plugin (kills the Scripts-sandbox reload-leak machinery, makes deploys scriptable). +- [`TESTING.md`](TESTING.md) — **the verification reference.** New-unit bring-up playbook, regression `curl` recipes, + outstanding tests, and the performance-review methodology. (All bring-up milestones are ✅ verified; the historical + milestone log lives in git history.) +- [`scrypted/README.md`](scrypted/README.md) — Scrypted-side script install + per-viewport binding UI (camera picker, + wake triggers, mDNS-discovered host dropdowns). +- [`scrypted/PLUGIN-CONVERSION.md`](scrypted/PLUGIN-CONVERSION.md) — TODO: plan for repackaging the script as a real + installable Scrypted plugin (kills the Scripts-sandbox reload-leak machinery, makes deploys scriptable). ## Status @@ -33,25 +44,33 @@ Scrypted Viewport owns Ethernet, JPEG decode, display, touch input and outbound | --- | --- | --- | | Firmware (`main/`) | Full path ✅ on hardware: Ethernet, panel, streaming, OTA. Painted = sent = 24 fps at the Unifi medium substream, sub-50 ms glass-to-glass over a raw TCP data socket (:81). Binary ~900 KB, ESP-IDF 5.4 / `esp32p4`. See [What's next](#whats-next) for the measured budget. | Backlog only (task-watchdog counters, multi-camera per viewport, production sealing). | | Scrypted side (`scrypted/`) | v1.4 Script — DeviceProvider with per-viewport child devices, camera picker with trigger-scoped event subscriptions, live ffmpeg MJPEG streaming over the TCP data socket with prebuffer fast-start (~0.7 s wake-to-video), and built-in mDNS discovery (host dropdowns, auto-naming, auto-heal on DHCP renumber). Verified end-to-end. | v2: repackage the single-file script as an installable plugin — planned in [`scrypted/PLUGIN-CONVERSION.md`](scrypted/PLUGIN-CONVERSION.md). | -| Hardware | Ethernet pin map confirmed (Waveshare wiki + ESPHome). Hosyond panel architecture confirmed (Pi 7"-style, TC358762 bridge + ATTINY MCU at I²C `0x45`). Jumper wiring documented. | Schematic confirmation needed for DSI FPC pin count, I²C GPIO mapping, BOOT button GPIO, flash size. All gated by [`TESTING.md`'s Hardware prerequisites](TESTING.md#hardware-prerequisites). | +| Hardware | Ethernet pin map confirmed (Waveshare wiki + ESPHome). Hosyond panel architecture confirmed (Pi 7"-style, TC358762 bridge + ATTINY MCU at I²C `0x45`). All bring-up unknowns resolved — wiring + gotchas in [`TESTING.md`](TESTING.md#new-unit-bring-up). | Per-unit flash-size silkscreen check (16 vs 32 MB) when provisioning new boards. | -See [`TESTING.md`](TESTING.md) for the full milestone-by-milestone status and the bench-order playbook. +See [`TESTING.md`](TESTING.md) for the new-unit bring-up playbook, regression recipes, and the outstanding-test backlog. ## Hardware -### Controller -[Waveshare ESP32-P4-ETH](https://www.amazon.com/ESP32-P4-Ethernet-Development-MIPI-CSI-Microphone/dp/B0FN7JQ2V8/) — ESP32-P4 with IP101GRI PHY, 2-lane MIPI-DSI out, 32 MB PSRAM, 16/32 MB flash. PoE is an optional add-on module on the same SKU. +### Display driver +[Waveshare ESP32-P4-ETH](https://www.amazon.com/ESP32-P4-Ethernet-Development-MIPI-CSI-Microphone/dp/B0FN7JQ2V8/) — +ESP32-P4 with IP101GRI PHY, 2-lane MIPI-DSI out, 32 MB PSRAM, 16/32 MB flash. PoE is an optional add-on module on the +same SKU. (The *controller* of the system is Scrypted — the ESP32 is a thin display driver; see +[Philosophy](#philosophy).) ### Display -[Hosyond 5" 800x480 IPS Capacitive Touch MIPI DSI display](https://www.amazon.com/dp/B0CXTFN8K9) — Pi-compatible panel (TC358762 DSI-to-DPI bridge + ATTINY-class init MCU at I²C `0x45`, FT5426 touch at `0x38`). 15-pin Pi FPC for the DSI lanes + power; I²C runs as jumpers off the panel's auxiliary header (see [`TESTING.md`](TESTING.md) M3 for the wiring table). +[Hosyond 5" 800x480 IPS Capacitive Touch MIPI DSI display](https://www.amazon.com/dp/B0CXTFN8K9) — Pi-compatible panel +(TC358762 DSI-to-DPI bridge + ATTINY-class init MCU at I²C `0x45`, FT5426 touch at `0x38`). 15-pin Pi FPC for the DSI +lanes + power; I²C runs as jumpers off the panel's auxiliary header (see [`TESTING.md`](TESTING.md) M3 for the wiring +table). ## Boot +``` Power -> DHCP -> mDNS (_scrypted-viewport._tcp.local) --> If no `scrypted` URL set: backlight on, show info screen (persistent until `/config`) --> If configured: enter sleep state (backlight off, wait for `POST /state` or a tap) +-> If no `scrypted` URL set: backlight on, show info screen (persistent until /config) +-> If configured: enter sleep state (backlight off, wait for POST /state or a tap) +``` ## Resolution @@ -62,7 +81,8 @@ Effective resolution depends on `orientation` (set via `/config`): - `portrait` (default): 480x800 - `landscape`: 800x480 -Scrypted must render JPEGs at the **effective** resolution. The device does not scale or rotate JPEG content — orientation is applied during framebuffer-to-panel push. +Scrypted must render JPEGs at the **effective** resolution. The device does not scale or rotate JPEG content — +orientation is applied during framebuffer-to-panel push. ## Network @@ -72,11 +92,15 @@ Trust model: LAN-only, no auth, no TLS. Deploy on a trusted VLAN. ## Discovery -The ESP32 publishes itself via **mDNS-SD (service discovery)**. Scrypted discovers viewports by browsing the service; it does not need OS-level `.local` hostname resolution. +The ESP32 publishes itself via **mDNS-SD (service discovery)**. Scrypted discovers viewports by browsing the service; it +does not need OS-level `.local` hostname resolution. -The device runs an mDNS responder (ESP-IDF `mdns` component) that serves all of the following from itself — no external DNS server is involved: +The device runs an mDNS responder (ESP-IDF `mdns` component) that serves all of the following from itself — no external +DNS server is involved: -- **Hostname / A record**: `viewport-<name>.local`. The name always has a value — a MAC-derived default (colons stripped, e.g. `viewport-e8f60ae09094.local`) until `/config` sets a friendlier one (`viewport-mudroom.local`). The `viewport-` prefix is a namespace that avoids collisions with other LAN devices. +- **Hostname / A record**: `viewport-<name>.local`. The name always has a value — a MAC-derived default (colons + stripped, e.g. `viewport-e8f60ae09094.local`) until `/config` sets a friendlier one (`viewport-mudroom.local`). The + `viewport-` prefix is a namespace that avoids collisions with other LAN devices. - **Service advertisement**: `_scrypted-viewport._tcp.local` on port 80. - **SRV record**: hostname + port. - **TXT records**: @@ -86,14 +110,24 @@ The device runs an mDNS responder (ESP-IDF `mdns` component) that serves all of - `name=<viewport name>` (MAC-derived default until `/config`) - `mac=<aa:bb:cc:dd:ee:ff>` — stable identity for discovery; names are editable, the MAC is not -Scrypted-side discovery (implemented in the script — see `mdnsBrowse` in [`scrypted/scrypted-viewport.ts`](scrypted/scrypted-viewport.ts)): - -1. The Scripts sandbox has **no third-party mDNS libraries** (`bonjour-service`, `multicast-dns`, `mdns` all fail to resolve — verified), so the script browses with a plain `dgram` socket on an **ephemeral port**: per RFC 6762 §6.7 a query from a non-5353 source port is a *legacy unicast* query and responders reply unicast straight back to it. No `:5353` bind means no conflict with Scrypted's own HomeKit mDNS stack or a host `avahi-daemon`, and it works under Docker host-networking or a native install alike. -2. One PTR response packet carries PTR + SRV + TXT + A together (RFC 6763 §12.1), so parsing is per-packet with no follow-up queries. The **IP** from the A record is what's used for all subsequent calls. -3. Discovered viewports surface as dropdown choices on the host field (add-device form + each viewport's settings page), and a blank name on create inherits the discovered TXT `name`. -4. **Auto-heal instead of periodic re-browse**: when a `/config` registration fails (DHCP renumber), the script re-browses and matches by TXT `mac` first, then `name`; on a hit at a new address it rewrites the stored host and retries. This removes the need for a DHCP reservation. - -Manual entry (IP or hostname) always remains available on the host field — discovery is best-effort and degrades to typing. +Scrypted-side discovery (implemented in the script — see `mdnsBrowse` in +[`scrypted/scrypted-viewport.ts`](scrypted/scrypted-viewport.ts)): + +1. The Scripts sandbox has **no third-party mDNS libraries** (`bonjour-service`, `multicast-dns`, `mdns` all fail to + resolve — verified), so the script browses with a plain `dgram` socket on an **ephemeral port**: per RFC 6762 §6.7 a + query from a non-5353 source port is a *legacy unicast* query and responders reply unicast straight back to it. No + `:5353` bind means no conflict with Scrypted's own HomeKit mDNS stack or a host `avahi-daemon`, and it works under + Docker host-networking or a native install alike. +2. One PTR response packet carries PTR + SRV + TXT + A together (RFC 6763 §12.1), so parsing is per-packet with no + follow-up queries. The **IP** from the A record is what's used for all subsequent calls. +3. Discovered viewports surface as dropdown choices on the host field (add-device form + each viewport's settings page), + and a blank name on create inherits the discovered TXT `name`. +4. **Auto-heal instead of periodic re-browse**: when a `/config` registration fails (DHCP renumber), the script + re-browses and matches by TXT `mac` first, then `name`; on a hit at a new address it rewrites the stored host and + retries. This removes the need for a DHCP reservation. + +Manual entry (IP or hostname) always remains available on the host field — discovery is best-effort and degrades to +typing. ### Discover from the CLI @@ -110,11 +144,13 @@ dns-sd -G v4 viewport-kitchen.local. curl http://viewport-kitchen.local/state ``` -On Linux: `avahi-browse -rt _scrypted-viewport._tcp` for the same effect with addresses inline. The instance name (`viewport-kitchen`) **is** the hostname prefix — they're the same string. +On Linux: `avahi-browse -rt _scrypted-viewport._tcp` for the same effect with addresses inline. The instance name +(`viewport-kitchen`) **is** the hostname prefix — they're the same string. ## API -Four endpoints. `GET /state` and `GET /config` are the read surface; `POST /config`, `POST /state`, and `POST /frame` are the write surface. +Four endpoints. `GET /state` and `GET /config` are the read surface; `POST /config`, `POST /state`, and `POST /frame` +are the write surface. ### GET /state @@ -147,7 +183,14 @@ Returns `200 OK` with JSON: } ``` -`state` is `awake` or `asleep` (it reports the screen's current state only). `configured` reports whether a `scrypted` URL is registered (the name always has a MAC-derived default, so it doesn't factor in). `last_frame_ms_ago` is `null` if no frame has been received since boot. `ota_state` is the running image's OTA slot state (`pending-verify` right after an OTA until the 30 s healthy timer marks it `valid` — the rollback tell, see [POST /firmware](#post-firmware)). `temp_c` is the on-die junction temperature (~10–20 °C above ambient; omitted if the sensor is unavailable). `tear_guard_engaged` counts frames the triple-buffer guard saved from tearing. `stream` is the most recent 30-painted-frame window of data-plane stats (recv/decode/paint/idle min/avg/max, wire rate, header-gap and pending-age decomposition, drop counters — see `stream_server.h` for field semantics); all zeros before the first window rolls. +`state` is `awake` or `asleep` (it reports the screen's current state only). `configured` reports whether a `scrypted` +URL is registered (the name always has a MAC-derived default, so it doesn't factor in). `last_frame_ms_ago` is `null` if +no frame has been received since boot. `ota_state` is the running image's OTA slot state (`pending-verify` right after +an OTA until the 30 s healthy timer marks it `valid` — the rollback tell, see [POST /firmware](#post-firmware)). +`temp_c` is the on-die junction temperature (~10–20 °C above ambient; omitted if the sensor is unavailable). +`tear_guard_engaged` counts frames the triple-buffer guard saved from tearing. `stream` is the most recent +30-painted-frame window of data-plane stats (recv/decode/paint/idle min/avg/max, wire rate, header-gap and pending-age +decomposition, drop counters — see `stream_server.h` for field semantics); all zeros before the first window rolls. ### POST /state @@ -176,7 +219,8 @@ Returns the persisted config: } ``` -Before first `/config`: returns `200` with `scrypted` as `null`; `viewport` carries its MAC-derived default and the rest their first-boot defaults (`brightness: 80`, `orientation: "portrait"`, `idle_timeout_ms: 60000`). +Before first `/config`: returns `200` with `scrypted` as `null`; `viewport` carries its MAC-derived default and the rest +their first-boot defaults (`brightness: 80`, `orientation: "portrait"`, `idle_timeout_ms: 60000`). ### POST /config @@ -190,13 +234,21 @@ Before first `/config`: returns `200` with `scrypted` as `null`; `viewport` carr } ``` -- **Partial update**: only fields present in the body are changed; omitted fields keep their current values. The persisted config is replaced atomically with the merged result. +- **Partial update**: only fields present in the body are changed; omitted fields keep their current values. The + persisted config is replaced atomically with the merged result. - Persisted to NVS, survives reboot. - Idempotent; reposting the same body yields the same state. -- `viewport` must be non-empty and ≤ 54 chars (so the `viewport-<name>` mDNS hostname fits the 63-byte DNS label limit); `scrypted` must be `http://...` and is the Scrypted plugin's base URL. The device POSTs state changes to `<scrypted>/state`. -- `idle_timeout_ms`: `0` disables the idle timer; non-zero values must be ≥ `5000`. Otherwise `400`. Scrypted should use the same value for its own per-stream timeout so both ends agree, but they time independently — either can end the session. -- `orientation`: `portrait` (480x800) or `landscape` (800x480). Default `portrait` on first boot. Changing orientation takes effect immediately, including for the IP and Loading screens. Scrypted must send JPEGs at the new effective resolution after a change. -- `brightness`: integer `0`–`100`. Default `80` on first boot. Applied immediately if awake; takes effect on next wake if asleep. PWM is gamma-corrected so the scale is perceptual. +- `viewport` must be non-empty and ≤ 54 chars (so the `viewport-<name>` mDNS hostname fits the 63-byte DNS label limit); + `scrypted` must be `http://...` and is the Scrypted plugin's base URL. The device POSTs state changes to + `<scrypted>/state`. +- `idle_timeout_ms`: `0` disables the idle timer; non-zero values must be ≥ `5000`. Otherwise `400`. Scrypted should use + the same value for its own per-stream timeout so both ends agree, but they time independently — either can end the + session. +- `orientation`: `portrait` (480x800) or `landscape` (800x480). Default `portrait` on first boot. Changing orientation + takes effect immediately, including for the IP and Loading screens. Scrypted must send JPEGs at the new effective + resolution after a change. +- `brightness`: integer `0`–`100`. Default `80` on first boot. Applied immediately if awake; takes effect on next wake + if asleep. PWM is gamma-corrected so the scale is perceptual. - Response: `204 No Content`. Invalid body: `400`. To tweak only brightness: @@ -210,40 +262,37 @@ To tweak only brightness: Paints a frame. Does **not** change wake/sleep state. - `Content-Type: image/jpeg`, body is raw JPEG bytes. -- Image must match the effective resolution (480x800 portrait, 800x480 landscape) as a baseline JPEG. Device does not scale, rotate, or letterbox JPEG content. +- Image must match the effective resolution (480x800 portrait, 800x480 landscape) as a baseline JPEG. Device does not + scale, rotate, or letterbox JPEG content. - Max size: 1 MB. -- **Requires awake state.** While asleep, returns `409 Conflict` and does not paint. Scrypted must `POST /state {"state":"wake"}` first (or wait for a tap-driven `wake` POST from the device). +- **Requires awake state.** While asleep, returns `409 Conflict` and does not paint. Scrypted must `POST /state + {"state":"wake"}` first (or wait for a tap-driven `wake` POST from the device). - Resets the idle timer on success. - Single in-flight frame; concurrent posts may be rejected with `503`. - Returns `204` once decoded and pushed to the panel. -- `400` malformed JPEG, `409` device asleep, `413` over size, `500` decode/display failure. On failure the previous frame stays on screen. +- `400` malformed JPEG, `409` device asleep, `413` over size, `500` decode/display failure. On failure the previous + frame stays on screen. ### POST /firmware -Push a new application image and reboot into it. The device's HTTP OTA -endpoint — replaces USB reflash once the device is on the LAN. +Push a new application image and reboot into it. The device's HTTP OTA endpoint — replaces USB reflash once the device +is on the LAN. -- `Content-Type: application/octet-stream`, body is the raw built app - image (`build/scrypted-viewport.bin`, ~1.5 MB). `Content-Length` - required. +- `Content-Type: application/octet-stream`, body is the raw built app image (`build/scrypted-viewport.bin`, ~1.5 MB). + `Content-Length` required. - Single-shot: a second concurrent POST returns `409 Conflict`. -- Streams straight to the inactive OTA slot (no full-image RAM buffer), - validates header + checksum on `esp_ota_end`, flips `otadata` to point - at the new slot, replies `200` with - `{"status":"ok","previous":"<git>","next":"<git>","slot":"ota_1","reboot_in_ms":500}`, - then reboots ~500 ms later so the response can flush. -- On failure (`400` bad body / validate failed, `413` over partition size, - `500` flash error) the OTA handle is aborted and the running image stays - live. No partial-write half-state. -- **Rollback armed.** The new image boots `pending-verify`; firmware - flips it to `valid` after 30 s of healthy uptime - (`ota_arm_healthy_timer`). If the new image panics or the device is - power-cycled before the timer fires, the bootloader reverts to the - previous slot on next reset. `/state` reports the current state via - `ota_state`. - -The repo Makefile wraps the whole loop (recipes run in bash regardless -of your interactive shell; `idf.py` env is sourced per-recipe): +- Streams straight to the inactive OTA slot (no full-image RAM buffer), validates header + checksum on `esp_ota_end`, + flips `otadata` to point at the new slot, replies `200` with + `{"status":"ok","previous":"<git>","next":"<git>","slot":"ota_1","reboot_in_ms":500}`, then reboots ~500 ms later so + the response can flush. +- On failure (`400` bad body / validate failed, `413` over partition size, `500` flash error) the OTA handle is aborted + and the running image stays live. No partial-write half-state. +- **Rollback armed.** The new image boots `pending-verify`; firmware flips it to `valid` after 30 s of healthy uptime + (`ota_arm_healthy_timer`). If the new image panics or the device is power-cycled before the timer fires, the + bootloader reverts to the previous slot on next reset. `/state` reports the current state via `ota_state`. + +The repo Makefile wraps the whole loop (recipes run in bash regardless of your interactive shell; `idf.py` env is +sourced per-recipe): ```sh make ota # reconfigure (fresh git stamp) + build + push + verify @@ -254,14 +303,11 @@ make verify # post-push pending-verify -> valid check only make check # type-check the Scrypted plugin (tsc --noEmit) ``` -`make ota` (via `tools/ota.sh`) encodes the acceptance criterion from -the rollback bullet above: the fresh boot must report -`ota_state=pending-verify` before flipping to `valid` — a boot that -reads `valid` immediately means the bootloader silently reverted to the -old slot, and the script re-pushes once automatically (a known quirk of -first pushes). It also runs `idf.py reconfigure` first because the -embedded git hash is stamped at CMake configure time only; without it -the binary reports a stale SHA after new commits. +`make ota` (via `tools/ota.sh`) encodes the acceptance criterion from the rollback bullet above: the fresh boot must +report `ota_state=pending-verify` before flipping to `valid` — a boot that reads `valid` immediately means the +bootloader silently reverted to the old slot, and the script re-pushes once automatically (a known quirk of first +pushes). It also runs `idf.py reconfigure` first because the embedded git hash is stamped at CMake configure time only; +without it the binary reports a stale SHA after new commits. The raw mechanism underneath: @@ -276,12 +322,14 @@ curl -v --data-binary @build/scrypted-viewport.bin \ Wake and sleep couple the device backlight with Scrypted's frame stream. The device owns the state. -Both the device and Scrypted expose the same endpoint, `POST /state`, with the same body shape `{viewport, state}` — they're peers. Either side can push to the other to set state. Repeats are safe; both sides are idempotent. +Both the device and Scrypted expose the same endpoint, `POST /state`, with the same body shape `{viewport, state}` — +they're peers. Either side can push to the other to set state. Repeats are safe; both sides are idempotent. - `{"viewport": "<name>", "state": "wake"}` → "start streaming to this viewport now" - `{"viewport": "<name>", "state": "sleep"}` → "stop streaming to this viewport now" -Each request carries the device's `viewport` name as the routing key. Scrypted does not track per-viewport state across requests; it acts on each and forgets. +Each request carries the device's `viewport` name as the routing key. Scrypted does not track per-viewport state across +requests; it acts on each and forgets. Transitions: @@ -294,13 +342,20 @@ Transitions: | `POST /state {"state":"sleep"}` | Asleep | none | | `POST /frame` | (no state change) | — | -Frames never change state — neither the MJPEG stream on the data socket nor a `POST /frame`. Scrypted must `POST /state {"state":"wake"}` (or wait for a tap-driven `wake` POST from the device) before frames will paint. This makes the protocol race-free: once the device is asleep, frames arriving on the stream socket are discarded by the decode task and a `POST /frame` is rejected with `409` — neither silently re-wakes the panel. Scrypted learns the device slept from its `state=sleep` callback (and, as a backstop, its own per-stream safety timer). +Frames never change state — neither the MJPEG stream on the data socket nor a `POST /frame`. Scrypted must `POST /state +{"state":"wake"}` (or wait for a tap-driven `wake` POST from the device) before frames will paint. This makes the +protocol race-free: once the device is asleep, frames arriving on the stream socket are discarded by the decode task and +a `POST /frame` is rejected with `409` — neither silently re-wakes the panel. Scrypted learns the device slept from its +`state=sleep` callback (and, as a backstop, its own per-stream safety timer). -Only `tap` is detected on the touchscreen — long-press and swipes are out of scope for v1. `tap` itself is internal; what Scrypted sees is the resulting `state`. +The touchscreen gestures are deliberately minimal — short tap (wake/sleep toggle) and long-press (info overlay); no +swipes. `tap` itself is internal; what Scrypted sees is the resulting `state`. ### Device → Scrypted `POST /state` -When the device changes state on its own (a tap, the idle timer firing), it POSTs to Scrypted's `/state` endpoint. Same shape as Scrypted POSTing to the device's `/state`; no `event`, no `type`, no callback semantics. The two endpoints are peers, not request/response. +When the device changes state on its own (a tap, the idle timer firing), it POSTs to Scrypted's `/state` endpoint. Same +shape as Scrypted POSTing to the device's `/state`; no `event`, no `type`, no callback semantics. The two endpoints are +peers, not request/response. **Request** @@ -334,9 +389,13 @@ No other fields. No timestamp (no RTC, no SNTP); Scrypted timestamps on receipt. **No application-level ack** -HTTP 2xx is transport-level only. There is no application-level ack: the device does not retry, does not block subsequent state changes on the response, and does not treat a 5xx response as anything more than a counter increment. Idempotency + frames not painting while asleep + each side's independent idle timer recover every failure mode without an ack: +HTTP 2xx is transport-level only. There is no application-level ack: the device does not retry, does not block +subsequent state changes on the response, and does not treat a 5xx response as anything more than a counter increment. +Idempotency + frames not painting while asleep + each side's independent idle timer recover every failure mode without +an ack: -- Scrypted misses the device's `sleep`: its own per-stream safety timer eventually tears down the ffmpeg stream, and in the meantime any frames still on the wire are discarded by the (now-asleep) device, so nothing paints. +- Scrypted misses the device's `sleep`: its own per-stream safety timer eventually tears down the ffmpeg stream, and in + the meantime any frames still on the wire are discarded by the (now-asleep) device, so nothing paints. - Device misses Scrypted's `wake`/`sleep`: same — the next state change on either side syncs them. Don't design Scrypted-side logic that waits for the device to confirm a state change. There is no confirmation. @@ -345,7 +404,8 @@ Don't design Scrypted-side logic that waits for the device to confirm a state ch - Any 2xx is success. - Body is ignored. -- Anything else (non-2xx, connection refused, DNS failure, request timeout) increments `state_post_failures` and is otherwise ignored. +- Anything else (non-2xx, connection refused, DNS failure, request timeout) increments `state_post_failures` and is + otherwise ignored. **Timeouts** @@ -357,19 +417,23 @@ Don't design Scrypted-side logic that waits for the device to confirm a state ch - At most one outbound `/state` POST in flight at a time. - POSTs are delivered in the order state changes occur on the device. -- If a state change happens while a POST is in flight, it goes into a depth-1 queue. If the queue already holds a POST, the queued entry is **replaced** by the newer one. The in-flight POST is never cancelled. -- Replacement is safe because POSTs are imperatives: only the latest desired state matters to Scrypted. Intermediate flips between wake and sleep within the queue window collapse to the final state. +- If a state change happens while a POST is in flight, it goes into a depth-1 queue. If the queue already holds a POST, + the queued entry is **replaced** by the newer one. The in-flight POST is never cancelled. +- Replacement is safe because POSTs are imperatives: only the latest desired state matters to Scrypted. Intermediate + flips between wake and sleep within the queue window collapse to the final state. **Failure semantics** - The local state change always happens regardless of POST outcome. -- A dropped or failed POST is recovered by the next user tap, the device's idle timer firing `sleep`, or Scrypted's per-stream safety timer ending the stream. +- A dropped or failed POST is recovered by the next user tap, the device's idle timer firing `sleep`, or Scrypted's + per-stream safety timer ending the stream. - No retry queue. No backoff. No persistence across reboots. **When the device does NOT POST** - Before `/config` has registered a `scrypted` URL (boot state, NVS-erased). Silently dropped. -- For state changes Scrypted initiated (`POST /state`, `POST /frame` while asleep). Scrypted already knows; echoing would loop. +- For state changes Scrypted initiated (`POST /state`, `POST /frame` while asleep). Scrypted already knows; echoing + would loop. - Long-press info overlay (not a state change). **Trust model** @@ -378,24 +442,37 @@ Plain HTTP, no TLS, no auth — same as the inbound API. LAN-only. ### Race handling -Wake/sleep changes can race: a user taps the device while Scrypted is mid-flight with a `POST /state` from a stale camera-event timeout, or the idle timer fires at the same instant Scrypted POSTs a fresh `wake`. The protocol does **not** carry epochs, session IDs, or priorities. Race resolution is purely about each side serializing its own writes and trusting idempotency to converge. +Wake/sleep changes can race: a user taps the device while Scrypted is mid-flight with a `POST /state` from a stale +camera-event timeout, or the idle timer fires at the same instant Scrypted POSTs a fresh `wake`. The protocol does +**not** carry epochs, session IDs, or priorities. Race resolution is purely about each side serializing its own writes +and trusting idempotency to converge. Rules: -1. **Device-side serialization.** The device guards its state-mutation function with a mutex. Tap, idle timer, and `POST /state` all funnel through it. Whichever lands second wins. -2. **Scrypted-side serialization.** Scrypted handles inbound `POST /state` requests for a given `viewport` one at a time (in-process queue per viewport). Whichever lands second wins. -3. **Last write wins.** No priorities. No "the device is where the user is so it always trumps Scrypted." If a stale `sleep` lands after a fresh `wake`, the device sleeps; the user taps again and we're back. One extra tap is cheap. -4. **Scrypted must cancel its own pending operations on each inbound POST.** When a `wake` arrives, Scrypted cancels any pending per-viewport sleep timer before starting a fresh stream. Same in reverse. This makes "stale Scrypted timer fires after the user tapped to wake" impossible without needing protocol-level epochs. +1. **Device-side serialization.** The device guards its state-mutation function with a mutex. Tap, idle timer, and `POST + /state` all funnel through it. Whichever lands second wins. +2. **Scrypted-side serialization.** Scrypted handles inbound `POST /state` requests for a given `viewport` one at a time + (in-process queue per viewport). Whichever lands second wins. +3. **Last write wins.** No priorities. No "the device is where the user is so it always trumps Scrypted." If a stale + `sleep` lands after a fresh `wake`, the device sleeps; the user taps again and we're back. One extra tap is cheap. +4. **Scrypted must cancel its own pending operations on each inbound POST.** When a `wake` arrives, Scrypted cancels any + pending per-viewport sleep timer before starting a fresh stream. Same in reverse. This makes "stale Scrypted timer + fires after the user tapped to wake" impossible without needing protocol-level epochs. -Idempotency does the rest — `POST /state` on either side is a no-op when the recipient is already in the requested state, and frames (streamed on the data socket, or a `POST /frame`) never paint while the device is asleep, so a stale stream can't re-wake it. +Idempotency does the rest — `POST /state` on either side is a no-op when the recipient is already in the requested +state, and frames (streamed on the data socket, or a `POST /frame`) never paint while the device is asleep, so a stale +stream can't re-wake it. ## Idle -After `idle_timeout_ms` (default 60s) with no painted frame (a streamed frame over the data socket, or a `/frame` POST), the device sleeps and POSTs `state=sleep`. Scrypted should use the same timeout so its per-stream cutoff matches, but they run independently — either side can end the session, whichever notices first. +After `idle_timeout_ms` (default 60s) with no painted frame (a streamed frame over the data socket, or a `/frame` POST), +the device sleeps and POSTs `state=sleep`. Scrypted should use the same timeout so its per-stream cutoff matches, but +they run independently — either side can end the session, whichever notices first. ## Idempotency -All endpoints are safe to retry. Every state-change path converges to the same final state regardless of how many times it's repeated or in what order: +All endpoints are safe to retry. Every state-change path converges to the same final state regardless of how many times +it's repeated or in what order: | Endpoint | Idempotent? | Notes | | --- | --- | --- | @@ -406,7 +483,9 @@ All endpoints are safe to retry. Every state-change path converges to the same f | `POST /frame` | yes (within state) | paints if awake; `409` if asleep — no partial state | | Stream socket (`:81`) | yes (within state) | frames paint if awake, are discarded if asleep — no partial state; skip-oldest keeps only the freshest | -`POST /state` (in either direction) carries imperatives, not notifications. Both sides act and forget; neither expects an application-level ack. A dropped POST is recovered by the next user action or by Scrypted's own per-stream safety timer. +`POST /state` (in either direction) carries imperatives, not notifications. Both sides act and forget; neither expects +an application-level ack. A dropped POST is recovered by the next user action or by Scrypted's own per-stream safety +timer. Failure modes do not corrupt state: @@ -417,10 +496,14 @@ Failure modes do not corrupt state: ## Touch gestures -The board has no usable user button (GPIO 35 is owned by EMAC TXD1 at runtime), so both behaviours live on the touch panel: +The board has no usable user button (GPIO 35 is owned by EMAC TXD1 at runtime), so both behaviours live on the touch +panel: - **Short tap** (<500 ms): toggle wake / sleep. POSTs `/state` to Scrypted when configured. -- **Long-press** (≥1.5 s): overlay the info screen for 15 seconds, then return to the prior state. Useful for identifying or re-registering a device that's already configured. Wakes the backlight temporarily; does not change the wake/sleep state and does not POST to Scrypted. An incoming `/frame` while the overlay is showing is rejected with `409` (state is still "sleep" underneath). +- **Long-press** (≥1.5 s): overlay the info screen for 15 seconds, then return to the prior state. Useful for + identifying or re-registering a device that's already configured. Wakes the backlight temporarily; does not change the + wake/sleep state and does not POST to Scrypted. An incoming `/frame` while the overlay is showing is rejected with + `409` (state is still "sleep" underneath). There is no factory-reset gesture. To wipe NVS, plug USB and run `idf.py erase-flash` followed by a normal reflash. @@ -428,23 +511,44 @@ There is no factory-reset gesture. To wipe NVS, plug USB and run `idf.py erase-f The device renders exactly two things itself; everything else is a JPEG from Scrypted: -- **Info screen**: ~17 lines of `label value` pairs (white on black, auto-scaled) covering the full `GET /config` + `GET /state` dump — name, mac, host, ip, state, configured, scrypted, orientation, brightness, idle, firmware, uptime, frames, errors, free heap, free PSRAM, chip temperature. Shown on first boot until `/config`, on NVS erase, and as a 15 s overlay on a touch long-press. -- **Loading screen**: shown from a wake — and re-shown on each new stream connection — until the first frame paints, so a stale prior frame never flashes during the connect→first-frame gap. Plain "Loading…" text. Rendered in the current orientation. +- **Info screen**: ~17 lines of `label value` pairs (white on black, auto-scaled) covering the full `GET /config` + + `GET /state` dump — name, mac, host, ip, state, configured, scrypted, orientation, brightness, idle, firmware, uptime, + frames, errors, free heap, free PSRAM, chip temperature. Shown on first boot until `/config`, on NVS erase, and as a + 15 s overlay on a touch long-press. +- **Loading screen**: shown from a wake — and re-shown on each new stream connection — until the first frame paints, so + a stale prior frame never flashes during the connect→first-frame gap. Plain "Loading…" text. Rendered in the current + orientation. -Both use a small embedded bitmap font — full lowercase a–z, digits, period, colon, dash, slash, plus uppercase `L` for "Loading...". No LVGL, no general text engine. +Both use a small embedded bitmap font — full lowercase a–z, digits, period, colon, dash, slash, plus uppercase `L` for +"Loading...". No LVGL, no general text engine. ## Scrypted Integration -The Scrypted side is **code, not configuration** — Scrypted has no built-in concept of a network framebuffer. The code is small and lives inside Scrypted: - -- **v1** (in this repo at [`scrypted/scrypted-viewport.ts`](scrypted/scrypted-viewport.ts), install instructions in [`scrypted/README.md`](scrypted/README.md)): a Scrypted Script (in the Scripts plugin) — subscribes to the bound camera's events (only the interfaces the selected wake triggers need) and, on wake, spawns one `ffmpeg` child (via `MediaManager`) that pulls the camera's substream, scales/rotates to panel-native 800×480, and streams MJPEG frames to the firmware over a **raw TCP data socket (port 81)** at ~24 fps. It also exposes a `POST /state` handler at the plugin's endpoint root (e.g. `http://scrypted.local:11080/endpoint/scrypted-viewport/state`) via the EndpointManager. Single-file TypeScript, no package install. -- **Fast start via prebuffer**: the script requests the camera's prebuffered substream so ffmpeg opens on an already-buffered keyframe, cutting wake-to-first-frame from ~5–6 s to ~0.7 s. Requires a rebroadcast prebuffer on the streamed substream — see [`scrypted/README.md`](scrypted/README.md#fast-wake--camera-prebuffer-required). -- **Built-in discovery**: the script browses `_scrypted-viewport._tcp` itself (dependency-free legacy-unicast `dgram` query — see [Discovery](#discovery)), offers discovered viewports as host-field choices, names new viewports from the advertised TXT `name`, and auto-heals a viewport's stored host when a registration fails after a DHCP renumber. -- **Next (v2)**: repackage the single-file script as a proper installable plugin — planned in [`scrypted/PLUGIN-CONVERSION.md`](scrypted/PLUGIN-CONVERSION.md). The streaming path itself (ffmpeg → framed MJPEG over the TCP data socket) is already in place; the firmware's `POST /frame` remains for one-shot snapshots and debug. +The Scrypted side is **code, not configuration** — Scrypted has no built-in concept of a network framebuffer. The code +is small and lives inside Scrypted: + +- **The script** (in this repo at [`scrypted/scrypted-viewport.ts`](scrypted/scrypted-viewport.ts), install instructions + in [`scrypted/README.md`](scrypted/README.md)): a Scrypted Script (in the Scripts plugin) — subscribes to the bound + camera's events (only the interfaces the selected wake triggers need) and, on wake, spawns one `ffmpeg` child (via + `MediaManager`) that pulls the camera's substream, scales/rotates to panel-native 800×480, and streams MJPEG frames to + the firmware over a **raw TCP data socket (port 81)** at ~24 fps. It also exposes a `POST /state` handler at the + plugin's endpoint root (e.g. `http://scrypted.local:11080/endpoint/scrypted-viewport/state`) via the EndpointManager. + Single-file TypeScript, no package install. +- **Fast start via prebuffer**: the script requests the camera's prebuffered substream so ffmpeg opens on an + already-buffered keyframe, cutting wake-to-first-frame from ~5–6 s to ~0.7 s. Requires a rebroadcast prebuffer on the + streamed substream — see [`scrypted/README.md`](scrypted/README.md#fast-wake--camera-prebuffer-required). +- **Built-in discovery**: the script browses `_scrypted-viewport._tcp` itself (dependency-free legacy-unicast `dgram` + query — see [Discovery](#discovery)), offers discovered viewports as host-field choices, names new viewports from the + advertised TXT `name`, and auto-heals a viewport's stored host when a registration fails after a DHCP renumber. +- **Next (v2)**: repackage the single-file script as a proper installable plugin — planned in + [`scrypted/PLUGIN-CONVERSION.md`](scrypted/PLUGIN-CONVERSION.md). The streaming path itself (ffmpeg → framed MJPEG + over the TCP data socket) is already in place; the firmware's `POST /frame` remains for one-shot snapshots and debug. Either way, no Scrypted core changes and no external service. -Scrypted owns a static list of viewports, each **bound to one Scrypted camera device** in the script/plugin. The binding tells Scrypted which camera's events drive that viewport (doorbell press, person/motion detection, etc.) and which camera's frames to push to it. +Scrypted owns a static list of viewports, each **bound to one Scrypted camera device** in the script/plugin. The binding +tells Scrypted which camera's events drive that viewport (doorbell press, person/motion detection, etc.) and which +camera's frames to push to it. On startup, register every viewport: @@ -464,7 +568,8 @@ await fetch(`${v.url}/config`, { }); ``` -To start a session (camera event like doorbell, motion, person): POST `wake`, then open the TCP data socket and pipe MJPEG frames straight to it — no per-frame HTTP. +To start a session (camera event like doorbell, motion, person): POST `wake`, then open the TCP data socket and pipe +MJPEG frames straight to it — no per-frame HTTP. ```ts await fetch(`${v.url}/state`, { @@ -496,23 +601,35 @@ if (state === "sleep") stopStream(v); // idempotent res.status(204).end(); ``` -- `state=wake` → start streaming frames to the viewport's bound camera. You do not need to `POST /state` to the device first; the device is already awake when it sends this. +- `state=wake` → start streaming frames to the viewport's bound camera. You do not need to `POST /state` to the device + first; the device is already awake when it sends this. - `state=sleep` → stop streaming frames to that viewport. Both are idempotent. Don't track viewport state across requests; act on each and forget. -Scrypted should use the same `idle_timeout_ms` value it sent in `/config` as its own per-stream cutoff. The two timers run independently — either side can cut a session, whichever notices first. When the device idle-sleeps itself it POSTs `{state: "sleep"}` back, and Scrypted tears down the stream; if that callback is lost, the Scrypted-side safety timer ends the stream anyway. +Scrypted should use the same `idle_timeout_ms` value it sent in `/config` as its own per-stream cutoff. The two timers +run independently — either side can cut a session, whichever notices first. When the device idle-sleeps itself it POSTs +`{state: "sleep"}` back, and Scrypted tears down the stream; if that callback is lost, the Scrypted-side safety timer +ends the stream anyway. ## Ops -- Firmware updates: `POST /firmware` with the raw built `.bin` (see [POST /firmware](#post-firmware)). First flash of any new device still needs USB to install the bootloader + initial image; every update after that is over the LAN. Rollback is armed — a panicking new image reverts to the previous slot on next reset. -- Provisioning: flash the same firmware to every device. On first boot it advertises itself via mDNS and shows the info screen; in Scrypted, "+ Add Device" lists it in the host dropdown (name auto-fills from the advertisement) — no IP hunting needed. -- Viewport names must be unique across the LAN — mDNS hostnames are derived from `viewport` and two devices configured with the same name will collide. -- NVS wipe: plug USB and run `idf.py erase-flash` followed by `idf.py flash`. The device boots clean and shows the info screen until `/config` is POSTed. +- Firmware updates: `POST /firmware` with the raw built `.bin` (see [POST /firmware](#post-firmware)). First flash of + any new device still needs USB to install the bootloader + initial image; every update after that is over the LAN. + Rollback is armed — a panicking new image reverts to the previous slot on next reset. +- Provisioning: flash the same firmware to every device. On first boot it advertises itself via mDNS and shows the info + screen; in Scrypted, "+ Add Device" lists it in the host dropdown (name auto-fills from the advertisement) — no IP + hunting needed. +- Viewport names must be unique across the LAN — mDNS hostnames are derived from `viewport` and two devices configured + with the same name will collide. +- NVS wipe: plug USB and run `idf.py erase-flash` followed by `idf.py flash`. The device boots clean and shows the info + screen until `/config` is POSTed. - No DHCP lease: keep retrying; do not reboot. The info screen shows "ip no network" until a lease arrives. -- Ethernet disconnect: reconnect automatically. If Scrypted is unreachable, displays go stale — nothing the device can do about it. +- Ethernet disconnect: reconnect automatically. If Scrypted is unreachable, displays go stale — nothing the device can + do about it. - Watchdog: the ESP-IDF task watchdog reboots the device if a task hangs. Soft state is rebuilt from NVS on every boot. -- No usable on-board status LED on the Waveshare ESP32-P4-ETH. The info screen tells the boot story instead — short-tap to wake, long-press to overlay it. +- No usable on-board status LED on the Waveshare ESP32-P4-ETH. The info screen tells the boot story instead — short-tap + to wake, long-press to overlay it. ## Build @@ -526,9 +643,8 @@ idf.py build idf.py -p /dev/cu.usbmodem* flash monitor ``` -Every update after that is one command over the LAN — build, OTA push, -and rollback-checked verification (see [POST /firmware](#post-firmware) -for what it does and the underlying curl): +Every update after that is one command over the LAN — build, OTA push, and rollback-checked verification (see [POST +/firmware](#post-firmware) for what it does and the underlying curl): ```sh make ota [VIEWPORT=<host>] @@ -558,23 +674,43 @@ make ota [VIEWPORT=<host>] ### Memory strategy -The Waveshare board ships with 32 MB PSRAM. Everything large lives there: the stream body ring (3 × ~1 MB JPEG input buffers), the three `esp_lcd_dpi` BGR888 framebuffers (3 × 1.15 MB — the JPEG decoder writes directly into these, there is no separate decoder output buffer), and the local-screens scratch (~1.15 MB). Internal SRAM is reserved for FreeRTOS task stacks, EMAC DMA buffers, and small allocations — `CONFIG_SPIRAM_MALLOC_ALWAYSINTERNAL=16384` keeps allocations ≤ 16 KB in SRAM by default. +The Waveshare board ships with 32 MB PSRAM. Everything large lives there: the stream body ring (3 × ~1 MB JPEG input +buffers), the three `esp_lcd_dpi` BGR888 framebuffers (3 × 1.15 MB — the JPEG decoder writes directly into these, there +is no separate decoder output buffer), and the local-screens scratch (~1.15 MB). Internal SRAM is reserved for FreeRTOS +task stacks, EMAC DMA buffers, and small allocations — `CONFIG_SPIRAM_MALLOC_ALWAYSINTERNAL=16384` keeps allocations ≤ +16 KB in SRAM by default. ### Display strategy -JPEG → BGR888 → DSI panel, **triple-buffered**, zero-copy and tear-free. No LVGL, no general text engine. The only locally-drawn UI is the IP screen, the Loading screen, and the info overlay (all via the 8×8 bitmap font in `local_screens.c`). Brightness PWM is gamma-corrected — `duty = (level/100)^2.2 * 255` — so 0–100 maps to perceptual brightness instead of linear duty cycle. - -**Why three framebuffers.** The DPI driver owns the framebuffers (`num_fbs = 3`); the hardware JPEG decoder writes straight into one of them, so `esp_lcd_panel_draw_bitmap` takes the IDF fast path — a cache writeback plus an index swap (~40 µs), no memcpy anywhere. But that index swap is *deferred*: the DSI DMA only reloads the new index at the **end of the frame scan in progress** (~21 ms period at ~47 Hz). With two buffers, flipping and immediately decoding the next frame writes into the buffer the DMA is still scanning out — a torn frame. That regime is common once frames arrive back-to-back (measured: ~6% of painted frames at full stream rate). - -Waiting for vsync before decoding would fix it at the cost of up to one refresh period of latency per frame. Instead, the three buffers hold three roles — **scanning** (DMA is reading it), **pending** (flipped, displays at the next boundary), **free** — and `display_back_buffer()` always hands the decoder the free one. The scanning buffer is tracked from the driver's `on_refresh_done` ISR (fires exactly when the DMA reloads its index). Three buffers minus at most two excluded roles = always a safe decode target: tear-free by construction, with zero waiting. `/state` reports `tear_guard_engaged` — picks made while the previous buffer was still mid-scan, i.e. frames that would have torn under double buffering. Cost: one extra 1.15 MB PSRAM framebuffer. +JPEG → BGR888 → DSI panel, **triple-buffered**, zero-copy and tear-free. No LVGL, no general text engine. The only +locally-drawn UI is the IP screen, the Loading screen, and the info overlay (all via the 8×8 bitmap font in +`local_screens.c`). Brightness PWM is gamma-corrected — `duty = (level/100)^2.2 * 255` — so 0–100 maps to perceptual +brightness instead of linear duty cycle. + +**Why three framebuffers.** The DPI driver owns the framebuffers (`num_fbs = 3`); the hardware JPEG decoder writes +straight into one of them, so `esp_lcd_panel_draw_bitmap` takes the IDF fast path — a cache writeback plus an index swap +(~40 µs), no memcpy anywhere. But that index swap is *deferred*: the DSI DMA only reloads the new index at the **end of +the frame scan in progress** (~21 ms period at ~47 Hz). With two buffers, flipping and immediately decoding the next +frame writes into the buffer the DMA is still scanning out — a torn frame. That regime is common once frames arrive +back-to-back (measured: ~6% of painted frames at full stream rate). + +Waiting for vsync before decoding would fix it at the cost of up to one refresh period of latency per frame. Instead, +the three buffers hold three roles — **scanning** (DMA is reading it), **pending** (flipped, displays at the next +boundary), **free** — and `display_back_buffer()` always hands the decoder the free one. The scanning buffer is tracked +from the driver's `on_refresh_done` ISR (fires exactly when the DMA reloads its index). Three buffers minus at most two +excluded roles = always a safe decode target: tear-free by construction, with zero waiting. `/state` reports +`tear_guard_engaged` — picks made while the previous buffer was still mid-scan, i.e. frames that would have torn under +double buffering. Cost: one extra 1.15 MB PSRAM framebuffer. ### Error handling Every endpoint is idempotent; every failure leaves the device in a sane state. - JPEG decode fails → `decode_errors++`, return 400 or 500, keep previous frame on screen, wake/sleep state unchanged. -- Outbound `/state` POST fails → `state_post_failures++`, continue. Local state change still happened. No retry queue; Scrypted catches up via its own per-stream safety timer or the next event. -- Ethernet disconnects → driver reconnects automatically. No reboot loop. `GET /state` keeps serving over loopback for diagnostics. +- Outbound `/state` POST fails → `state_post_failures++`, continue. Local state change still happened. No retry queue; + Scrypted catches up via its own per-stream safety timer or the next event. +- Ethernet disconnects → driver reconnects automatically. No reboot loop. `GET /state` keeps serving over loopback for + diagnostics. - Display init fails → log loudly, keep serving the rest of the API. The protocol is still usable for re-registration. - Task hangs → ESP-IDF watchdog reboots. NVS rebuilds soft state on the next boot. @@ -589,11 +725,16 @@ Every endpoint is idempotent; every failure leaves the device in a sane state. ## What's next -M1 – M9 are all ✅ on hardware (see [`TESTING.md`](TESTING.md) for verification details). End-to-end Scrypted streaming via ffmpeg + the zero-copy `JPEG → BGR888 → DSI` hot path now sustains **painted = sent = 24 fps** at the Unifi medium substream rate, sub-50 ms glass-to-glass, no source-side backpressure. +M1 – M9 are all ✅ on hardware (see [`TESTING.md`](TESTING.md) for verification details). End-to-end Scrypted streaming +via ffmpeg + the zero-copy `JPEG → BGR888 → DSI` hot path now sustains **painted = sent = 24 fps** at the Unifi medium +substream rate, sub-50 ms glass-to-glass, no source-side backpressure. ### Measured per-frame budget -The firmware streams over a long-lived raw TCP socket on port 81 (replacing the per-frame HTTP `POST /frame` pattern from the early milestones). The stream server runs `recv` on its own FreeRTOS task and hands frames off to a separate decode/paint task through a 3-buffer PSRAM ping-pong ring. Steady-state on the bench (Waveshare ESP32-P4-ETH + Hosyond 5" panel, Unifi medium substream → ~80–130 KB JPEGs at ffmpeg `-q:v 1`): +The firmware streams over a long-lived raw TCP socket on port 81 (replacing the per-frame HTTP `POST /frame` pattern +from the early milestones). The stream server runs `recv` on its own FreeRTOS task and hands frames off to a separate +decode/paint task through a 3-buffer PSRAM ping-pong ring. Steady-state on the bench (Waveshare ESP32-P4-ETH + Hosyond +5" panel, Unifi medium substream → ~80–130 KB JPEGs at ffmpeg `-q:v 1`): | Phase | Time | Share | What it is | |---|---|---|---| @@ -602,34 +743,64 @@ The firmware streams over a long-lived raw TCP socket on port 81 (replacing the | `paint` | 16–60 µs | < 0.2% | `esp_lcd_panel_draw_bitmap` — cache writeback + index swap thanks to `num_fbs = 3` and zero-copy decode into the free fb (see *Display strategy* for the tear-free triple-buffer model) | | `decode_idle` | **27–40 ms** | n/a | Time decode-task spent waiting on the recv→decode signal. Means the decode/paint stage is *idle* most of the time at the source rate — the wire is the cap. | -Scrypted side: `sent=24.0fps painted=24.0fps backpressured=false`, `g2g=31–41 ms`. No `fw-skipped`, no `drops`, no `flushes`. +Scrypted side: `sent=24.0fps painted=24.0fps backpressured=false`, `g2g=31–41 ms`. No `fw-skipped`, no `drops`, no +`flushes`. ### The critical change that unlocked this -For a long stretch the device painted ~17 fps against a 24 fps source. The single-task `recv → decode → paint` loop in `stream_server.c` blocked the socket recv for ~6 ms every frame during decode+paint, which forced the sender into a tight stop-go cycle against the IDF-default 5760-byte TCP window. Raising the window to 65535 made it *worse* — the sender could pump 45+ segments before stopping but the lwIP RX path couldn't drain that into the single task, so the kernel buffer accumulated stale frames and `g2g` grew unbounded (one experiment hit 17 *seconds* before we reverted). +For a long stretch the device painted ~17 fps against a 24 fps source. The single-task `recv → decode → paint` loop in +`stream_server.c` blocked the socket recv for ~6 ms every frame during decode+paint, which forced the sender into a +tight stop-go cycle against the IDF-default 5760-byte TCP window. Raising the window to 65535 made it *worse* — the +sender could pump 45+ segments before stopping but the lwIP RX path couldn't drain that into the single task, so the +kernel buffer accumulated stale frames and `g2g` grew unbounded (one experiment hit 17 *seconds* before we reverted). -The fix wasn't a TCP knob, it was the task architecture. `d1c8d45` split `handle_client` into a **dedicated recv-task** (owns the socket, drains continuously) and a **decode-task** (waits on a binary semaphore), with a **3-buffer PSRAM ring**: recv-task fills one buffer, decode-task processes a second, the third is either free or holds a pending frame between them. The ring guarantees recv-task never blocks waiting on decode, and a 1-deep latest-frame slot lets the receiver skip-oldest if decode ever falls behind (mirror of the Scrypted-side skip-oldest in `e5acf93`). After the split, recv-task is busy ~30% of the time and decode-task is idle ~80% — the wire is now the only thing setting the rate, and at 24 fps source it is genuinely keeping up. +The fix wasn't a TCP knob, it was the task architecture. `d1c8d45` split `handle_client` into a **dedicated recv-task** +(owns the socket, drains continuously) and a **decode-task** (waits on a binary semaphore), with a **3-buffer PSRAM +ring**: recv-task fills one buffer, decode-task processes a second, the third is either free or holds a pending frame +between them. The ring guarantees recv-task never blocks waiting on decode, and a 1-deep latest-frame slot lets the +receiver skip-oldest if decode ever falls behind (mirror of the Scrypted-side skip-oldest in `e5acf93`). After the +split, recv-task is busy ~30% of the time and decode-task is idle ~80% — the wire is now the only thing setting the +rate, and at 24 fps source it is genuinely keeping up. -The full instrumentation that drove the diagnosis (`queued_at_body_start`, `recv_calls`, `recv_chunk_min/avg/max`, `recv_dropped_oldest`, `decode_idle_*`, `so_rcvbuf`) is still in `/state` and the windowed log. +The full instrumentation that drove the diagnosis (`queued_at_body_start`, `recv_calls`, `recv_chunk_min/avg/max`, +`recv_dropped_oldest`, `decode_idle_*`, `so_rcvbuf`) is still in `/state` and the windowed log. ### TCP window + EMAC tuning (the follow-up the task split unlocked) -With recv on its own task and a skip-oldest slot, the window became safe to raise — but "safe" had to be proven, so first the stream gained a decomposition that accounts for the whole frame interval: `interval ≈ hdr_gap (sender idle) + recv (wire) + pend_age (handoff wait) + dec + paint`, plus `wire_*_kbps` — the instantaneous throughput while a body drains, whose ceiling is `TCP_WND / RTT`, making it the definitive "is the window the limiter" metric. +With recv on its own task and a skip-oldest slot, the window became safe to raise — but "safe" had to be proven, so +first the stream gained a decomposition that accounts for the whole frame interval: `interval ≈ hdr_gap (sender idle) + +recv (wire) + pend_age (handoff wait) + dec + paint`, plus `wire_*_kbps` — the instantaneous throughput while a body +drains, whose ceiling is `TCP_WND / RTT`, making it the definitive "is the window the limiter" metric. -Measured at the default `TCP_WND=5760` (~190 KB frames): wire pinned at 53 Mbps vs ~94 Mbps line rate on the 10/100 PHY, `recv_chunk_max` at exactly 5760, sender backpressured on half its frames — window-bound, three ways. +Measured at the default `TCP_WND=5760` (~190 KB frames): wire pinned at 53 Mbps vs ~94 Mbps line rate on the 10/100 PHY, +`recv_chunk_max` at exactly 5760, sender backpressured on half its frames — window-bound, three ways. -Raising to `TCP_WND=23040` alone **regressed** (fps 20 → 14, 200–450 ms recv stalls): the EMAC RX DMA pool (20 × 512 B = 10 KB) was smaller than the in-flight window, so a full-window burst overran the RX descriptors, the burst tail dropped with no dup-ACKs behind it, and the sender waited out ~200 ms min-RTO recoveries. **Invariant: the EMAC RX pool must exceed `TCP_WND`.** With `ETH_DMA_BUFFER_SIZE=1600` (one MSS frame per buffer/descriptor) × 24 = 38.4 KB, the stall tail vanished: wire 74 avg / 84 max Mbps, recv 21 ms, painted fps +19%, g2g 73 → ~40–60 ms. +Raising to `TCP_WND=23040` alone **regressed** (fps 20 → 14, 200–450 ms recv stalls): the EMAC RX DMA pool (20 × 512 B = +10 KB) was smaller than the in-flight window, so a full-window burst overran the RX descriptors, the burst tail dropped +with no dup-ACKs behind it, and the sender waited out ~200 ms min-RTO recoveries. **Invariant: the EMAC RX pool must +exceed `TCP_WND`.** With `ETH_DMA_BUFFER_SIZE=1600` (one MSS frame per buffer/descriptor) × 24 = 38.4 KB, the stall tail +vanished: wire 74 avg / 84 max Mbps, recv 21 ms, painted fps +19%, g2g 73 → ~40–60 ms. -Stopped there deliberately: the interval is now ~half `hdr_gap` (sender has nothing ready), so bigger windows buy little until the source rate rises. `pend_age_*` remains the tripwire — it must stay in microseconds; growth means the kernel-queue latency backlog that killed the pre-split 65535 experiment is back. +Stopped there deliberately: the interval is now ~half `hdr_gap` (sender has nothing ready), so bigger windows buy little +until the source rate rises. `pend_age_*` remains the tripwire — it must stay in microseconds; growth means the +kernel-queue latency backlog that killed the pre-split 65535 experiment is back. ### Current backlog, in rough priority order -1. **(was) Network body shrink — done.** The receiver now drains near line rate without blocking decode: TCP window raised to 23040 with the EMAC RX pool sized above it (see *TCP window + EMAC tuning*), wire 74 Mbps avg. Source rate (Unifi substream `medium-resolution` ≈ 24 fps) is the cap. -2. **(was) OTA firmware updates — done.** `POST /firmware` ships in `175dd50`, ota_0/ota_1 alternation with `BOOTLOADER_APP_ROLLBACK_ENABLE` for first-boot revert. `curl --data-binary @build/scrypted-viewport.bin http://<viewport>/firmware` reboots into the new image. -3. **Task watchdog + crash counters** — *low effort*. Enable the ESP-IDF task watchdog, surface its bite count in `/state` alongside the existing `decode_errors` / `state_post_failures`. Good hygiene. -4. **Multi-camera per viewport** — *medium effort*. Let one viewport listen to events from N cameras, picking which one to stream based on which fired. Useful for "show whichever doorbell rang" or zone monitoring. -5. **Boot info-screen flash polish** — *low effort*. Keep the brief wake-on-boot (the FT5426 needs it to start reporting touches) but smoothen the ~600 ms flash so it doesn't visibly flicker on power-up. -6. **Production sealing** — *eventual*. Configurable LAN scope (cross-VLAN, mDNS-via-Unicast), Scrypted-side mutual auth, replay protection for `/state` callbacks. +1. **(was) Network body shrink — done.** The receiver now drains near line rate without blocking decode: TCP window + raised to 23040 with the EMAC RX pool sized above it (see *TCP window + EMAC tuning*), wire 74 Mbps avg. Source rate + (Unifi substream `medium-resolution` ≈ 24 fps) is the cap. +2. **(was) OTA firmware updates — done.** `POST /firmware` ships in `175dd50`, ota_0/ota_1 alternation with + `BOOTLOADER_APP_ROLLBACK_ENABLE` for first-boot revert. `curl --data-binary @build/scrypted-viewport.bin + http://<viewport>/firmware` reboots into the new image. +3. **Task watchdog + crash counters** — *low effort*. Enable the ESP-IDF task watchdog, surface its bite count in + `/state` alongside the existing `decode_errors` / `state_post_failures`. Good hygiene. +4. **Multi-camera per viewport** — *medium effort*. Let one viewport listen to events from N cameras, picking which one + to stream based on which fired. Useful for "show whichever doorbell rang" or zone monitoring. +5. **Boot info-screen flash polish** — *low effort*. Keep the brief wake-on-boot (the FT5426 needs it to start reporting + touches) but smoothen the ~600 ms flash so it doesn't visibly flicker on power-up. +6. **Production sealing** — *eventual*. Configurable LAN scope (cross-VLAN, mDNS-via-Unicast), Scrypted-side mutual + auth, replay protection for `/state` callbacks. ## Philosophy @@ -1,739 +1,226 @@ # Testing & Verification -This file is the self-contained reference for verifying the firmware on real hardware. It has three layers: - -1. **Status snapshot** — at-a-glance state of every milestone. -2. **Hardware prerequisites** — what to confirm or order before flashing anything. -3. **Recommended bench order** — the sequence to flip 🟡 → ✅, stage by stage. -4. **Per-milestone entries** (M1–M9) — acceptance criteria + exact commands/actions + status. -5. **Integration & system tests** — to run after every per-milestone test passes. - -Status legend: - -- ⬜ not verified — code not yet exercised at all. -- 🟡 partial — compiles cleanly, builds against ESP-IDF 5.4 for `esp32p4`, but not yet run on a board. -- ✅ verified — confirmed on hardware. Annotate with the date and board rev. +All bring-up milestones (M1–M9: Ethernet/DHCP, HTTP + mDNS, display, config +persistence, frame push, state + idle timer, touch + outbound POSTs, local +screens, live TCP stream) are ✅ **verified on hardware**. The +milestone-by-milestone log — acceptance runs, dates, and the fix notes from +each bring-up session — lived in this file through v1.4.0; see git history +(`git log --follow TESTING.md`, last full version at `8eebc0e`) if you need +it. + +What remains here is the forward-looking material: + +1. **New-unit bring-up** — provisioning another viewport from parts. +2. **Regression recipes** — quick `curl` checks for the API surface. +3. **Outstanding tests** — what has never been run. +4. **Performance review playbook** — the iterate-and-measure loop for + data-plane changes. > **Iterating on an already-flashed device:** `make ota [VIEWPORT=<host>]` > builds with a fresh git stamp, pushes over the LAN, and verifies the > `pending-verify -> valid` OTA sequence (auto-retrying the known -> first-push silent rollback). The USB commands below are for first -> flash / bring-up only. - ---- - -## Status snapshot - -| # | Milestone | Code | HW | -| --- | --- | --- | --- | -| M1 | Board Bring-Up — Ethernet + DHCP | ✅ | ✅ | -| M2 | HTTP + mDNS (`GET /state`) | ✅ | ✅ | -| M3 | Display Bring-Up (Hosyond panel) | ✅ | ✅ | -| M4 | Config Persistence (NVS, partial updates) | ✅ | ✅ | -| M5 | JPEG Frame Push (`POST /frame`) | ✅ | ✅ | -| M6 | State + Idle Timer (`POST /state`, 409 guard) | ✅ | ✅ | -| M7 | Touch + Outbound `/state` POST | ✅ | ✅ | -| M8 | Local Screens + touch long-press | ✅ | ✅ | -| M9 | Live Stream (raw TCP data socket `:81`) | ✅ | ✅ | - ---- - -## Hardware prerequisites - -Before flashing anything, confirm or acquire: - -| Item | Status as of this writing | How to resolve | -| --- | --- | --- | -| Waveshare ESP32-P4-ETH board | shipped | check silkscreen for flash size (16 vs 32 MB) — adjust `sdkconfig.defaults` if 32 | -| Hosyond 5" 800×480 panel | shipped | n/a | -| 15-pin Pi-FPC → Waveshare-side DSI adapter cable | **unknown — order before M3** | check Waveshare board DSI connector pin count, buy matching adapter | -| 4× jumper wires (5V / GND / SDA / SCL) | trivial | bench supply | -| Ethernet cable + DHCP-serving switch | trivial | LAN already in place | -| (optional) PoE injector | nice-to-have | for M1/M2 a USB-only power path is fine | - -Hardware unknowns — resolved: - -1. **DSI FPC pin count on the Waveshare board.** ✅ 22-pin DSI on board; bridge to the panel's 15-pin Pi FPC via a 15→22 adapter cable. -2. **I²C jumper destinations.** ✅ `PIN_I2C_SDA=7`, `PIN_I2C_SCL=8` confirmed by ATTINY ack at 0x45 + FT5426 ack at 0x38. -3. **BOOT button GPIO.** ✅ No usable GPIO — ESP32-P4 GPIO 35 strap pin is owned by EMAC TXD1 at runtime; no separate user button is wired on the Waveshare ESP32-P4-ETH. Info-overlay behaviour moved onto touch long-press (≥1.5 s). No factory-reset gesture — that path goes through USB reflash + `idf.py erase-flash`. -4. **Flash size silkscreen.** `sdkconfig.defaults` declares 16 MB. If your SKU shipped with 32 MB the build still works but you waste half the flash. - ---- - -## Recommended bench order - -Work through these stages in order. Each stage builds on the previous one's verified state. - -### Stage 1 — board comes alive (M1 + M2) - -No display, no panel, no jumpers. Just board on Ethernet over USB. - -```bash -source ~/Dev/code/git/esp32/env.sh -idf.py -p /dev/cu.usbmodem* flash monitor -``` - -Check serial for `net_eth: got ip` then `curl http://<ip>/state | jq .` and `dns-sd -B _scrypted-viewport._tcp local.`. If both pass, flip M1 + M2 to ✅. Stop here if you hit any issue — debug Ethernet before adding more variables. - -### Stage 2 — display panel (M3) ✅ - -Look for `panel MCU id 0xc3 — Pi v1.1 architecture ack'd`, `TC358762 bridge configured`, and `DSI up: 800x480 26 MHz, 1-lane 600 Mbps, non-burst` in the serial log. Info screen visible on tap-wake. - -### Stage 3 — protocol (M4 + M5 + M6) - -No new wiring. POST `/config`, send a JPEG, exercise wake/sleep, watch the idle timer fire. Run through the per-milestone curls below. - -### Stage 4 — touch (M7) - -Tap the screen. Confirm the FT5426 acks on I²C in the boot log. Stand up the flask receiver, configure the device to point at it, and watch outbound `/state` POSTs arrive. - -### Stage 5 — local screens + BOOT (M8) - -Visual: fresh boot shows IP screen; wake shows loading screen; tap-press of the BOOT button overlays IP. If BOOT GPIO is wrong, the screens still work — only the button feature stays cold. - -### Stage 6 — end-to-end with Scrypted - -Paste `scrypted/scrypted-viewport.ts` into Scrypted's Scripts plugin, add a viewport device with the binding (camera picker dropdown), trigger the bound camera, watch the panel light up. - -### Stage 7 — integration - -Run the suite at the bottom of this file (races, failures, longevity, etc.). - ---- - -## M1 — Board Bring-Up - -**Acceptance**: device gets a DHCP lease over Ethernet and logs its IP. - -**How to verify** - -```bash -source ~/Dev/code/git/esp32/env.sh -idf.py -p /dev/cu.usbmodem* flash monitor -``` - -Expected log within ~5–10s of boot: - -``` -I (xxx) viewport: Scrypted Viewport boot -I (xxx) net_eth: ethernet driver started, waiting for link + DHCP -I (xxx) net_eth: link up, mac xx:xx:xx:xx:xx:xx -I (xxx) net_eth: got ip 192.168.x.x gw 192.168.x.1 netmask 255.255.255.0 -I (xxx) viewport: online at 192.168.x.x -``` - -Cross-check from another host: - -```bash -ping 192.168.x.x -``` - -**Status**: ✅ verified 2026-06-14 on Waveshare ESP32-P4-ETH (Amazon B0FN7JQ2V8, 32 MB PSRAM, 32 MB flash silkscreen, IDF v5.4.1). DHCP lease landed at 10.0.13.83. Three fixes were needed during bring-up — see commit 220ee4c: -> 1. **RXD0 / RXD1 GPIO swap**: ESP32-P4 EMAC iomux fixes RXD0=29, RXD1=30. The original pin map had them transposed. -> 2. **mDNS hostname order**: `mdns_service_add()` returns INVALID_ARG if hostname isn't set yet. Now init → hostname → service_add → TXT. -> 3. **`app_main` best-effort everywhere**: any one missing subsystem (no LAN, no panel) used to abort the boot. Now each is independent; final log line summarizes which letters of `EMHDJTB` came up. - -**Link-loss & recovery also verified 2026-06-14** — important for PoE-only deployments where a switch reboot or briefly down VLAN won't kill power but will yank the link: - -``` -W (748316) net_eth: link down # cable pulled -I (916316) net_eth: link up, mac e8:f6:0a:e0:90:94 # cable plugged back in (3 min later) -I (924816) net_eth: got ip 10.0.13.83 gw 10.0.13.1 netmask 255.255.255.0 # DHCP renewed ~8.5s after link-up -``` - -Same boot throughout (uptime kept climbing past the disconnect, no reset). mDNS responder task and HTTP server stayed alive through the outage and resumed serving the moment DHCP came back. `free_heap` / `free_psram` identical to pre-disconnect — no leak from the cycle. - ---- - -## M2 — HTTP + mDNS - -**Acceptance**: `GET /state` returns JSON; mDNS service is discoverable. - -**How to verify** - -After flash, from a host on the same LAN: - -```bash -# Direct call by IP (always works): -curl http://<device-ip>/state | jq . - -# mDNS hostname (requires OS-level .local resolution: macOS Bonjour, Linux nss-mdns): -curl http://viewport.local/state | jq . - -# Service discovery browse (preferred — what Scrypted will do, no OS .local resolver needed): -dns-sd -B _scrypted-viewport._tcp local. # macOS -avahi-browse -r _scrypted-viewport._tcp # Linux -``` - -Expected `/state` body on a fresh device (no `/config` posted yet): - -```json -{ - "name": null, - "version": "0.1.0", - "configured": false, - "state": "asleep", - "uptime_ms": 12345, - "last_frame_ms_ago": null, - "frames_received": 0, - "decode_errors": 0, - "state_post_failures": 0, - "resolution": "480x800", - "ip": "192.168.x.x", - "free_heap": 200000, - "free_psram": 30000000 -} -``` - -Expected mDNS browse output should show a `_scrypted-viewport._tcp` instance with TXT records `version=`, `resolution=`, `orientation=`, `name=` (MAC-derived default until `/config` names it), and `mac=` (since v1.4.0 — the stable identity the Scrypted script's discovery/auto-heal matches on). - -**Status**: ✅ verified 2026-06-14 alongside M1. - -- `GET http://10.0.13.83/state` returned the full spec JSON: `name=null, configured=false, state=asleep, resolution=480x800, ip=10.0.13.83, free_psram=31730048, ...`. -- `dns-sd -B _scrypted-viewport._tcp local.` on macOS surfaced one instance named `viewport`. - -Bare-board behavior (no panel, no ethernet) was also verified — the device boots cleanly and prints a summary like `boot complete — subsystems [EMHdJ-B] ip=(no link)` (lowercase `d` = display down because no panel attached; `-` for touch because it shares the panel I²C bus). - ---- - -## M3 — Display Bring-Up - -**Acceptance**: panel powers on, MCU at I2C 0x45 responds, color-bar test pattern renders, brightness control works. - -**Wiring** (Hosyond 5" 800x480 panel ↔ Waveshare ESP32-P4-ETH) - -The Hosyond panel exposes only the 15-pin Pi DSI FPC — no auxiliary GPIO header. **Everything goes through the FPC**: DSI data lanes, panel power (3.3V / 5V / GND), and the I²C bus (SDA / SCL) shared by the on-panel ATTINY MCU at `0x45` and the FT5426 touch at `0x38`. No jumper wires required. - -Verified board-side mapping (Waveshare ESP32-P4-ETH wiki + ESPHome device page): - -| Signal | DSI FPC | ESP32-P4 GPIO | -| --- | --- | --- | -| DSI data lanes / clock | dedicated DSI PHY | (fixed silicon pins) | -| I²C SDA | FPC pin | **GPIO 7** | -| I²C SCL | FPC pin | **GPIO 8** | -| 3.3V / 5V / GND | FPC pins | board rails (no GPIO) | -| Backlight | I²C register `0x96` on the panel MCU | (no GPIO) | -| Panel reset / power-on | I²C `REG_POWERON` (`0x85`) on the panel MCU | (no GPIO) | - -`PIN_I2C_SDA=7` / `PIN_I2C_SCL=8` in `display.c` already match this. **No code changes needed before flashing.** - -DSI connector: 22-pin (Pi 5 / CM4 style). The 15-pin Pi DSI FPC on the panel plugs into a **15→22 pin adapter cable** that goes into the board. - -**Bring-up sequence** - -1. Power the ESP32-P4 board (USB is fine for bench; PoE/12V in production). -2. Plug the **15-pin** side of the DSI cable into the Hosyond panel. -3. Plug the **22-pin** side into the Waveshare DSI connector. (Pi FPCs are easy to install upside-down — make sure the contact side faces the connector lock.) -4. Reset the board (or `idf.py flash monitor` if you've changed firmware): - -```bash -idf.py -p /dev/cu.usbmodem* monitor -``` - -**Expected log** - -``` -I (xxx) display: panel MCU id 0xC3 — Pi 7" architecture ack'd -I (xxx) display: panel powered on -I (xxx) display: DSI up: 800x480 30 MHz, 2-lane 480 Mbps -I (xxx) viewport: display up — test pattern on screen -``` - -**Visual check**: 8 vertical color bars (white, yellow, cyan, green, magenta, red, blue, black) across the panel. Brightness should look perceptually mid-range (default 80/100 with gamma). - -**Failure-mode signals** - -- `panel MCU @0x45 unreachable` → either the FPC seating / orientation is wrong (most common — Pi FPCs are easy to install upside-down on either end), the adapter cable doesn't pass I²C straight through, or the panel is one of the rare Waveshare-bundled units with **GT911 touch at 0x5D/0x14** instead of the FT5426 Pi 7" architecture this driver targets (in which case the *panel MCU* check still fails because that variant has no ATTINY at all). No need to suspect DSI yet. -- I²C ack but no image → DSI cable orientation on the 22-pin side, or `LCD_COLOR_PIXEL_FORMAT_RGB565` byte order if there's something on the panel but garbled. -- Image but wrong colors → RGB565 byte order; flip to BGR variant. -- Image but vertical/horizontal sync issues → adjust the `PANEL_*SYNC_*` timings in `display.c`; Pi 7" canonical values are the defaults. -- Touch ack at `0x38` but no taps → polling task not running, or panel reset state wrong. - -**Status**: ✅ verified 2026-06-14 on Waveshare ESP32-P4-ETH + Hosyond 5" panel. ATTINY v1.1 firmware (ID 0xC3) acks; TC358762 bridge configured via 16 DSI Generic Long Writes; DSI 1-lane @ 600 Mbps non-burst; DPI 26 MHz RGB888; info screen renders correctly. Bring-up notes captured in commit 6c1a26b. - ---- - -## M4 — Config Persistence - -**Acceptance**: `POST /config` persists across reboot; partial-update semantics work. - -**How to verify** - -```bash -# Full config: -curl -X POST -H "Content-Type: application/json" \ - -d '{"viewport":"mudroom","scrypted":"http://host/endpoint/scrypted-viewport","orientation":"landscape"}' \ - http://<device-ip>/config - -# Read back: -curl http://<device-ip>/config | jq . - -# Partial update (only brightness): -curl -X POST -H "Content-Type: application/json" \ - -d '{"brightness":50}' \ - http://<device-ip>/config - -# Reboot, then re-read — values should survive: -curl http://<device-ip>/config | jq . -``` - -Also verify validation: - -```bash -# idle_timeout_ms below 5000 (and non-zero) — expect 400: -curl -i -X POST -H "Content-Type: application/json" \ - -d '{"idle_timeout_ms":1000}' http://<device-ip>/config - -# bogus orientation — expect 400: -curl -i -X POST -H "Content-Type: application/json" \ - -d '{"orientation":"sideways"}' http://<device-ip>/config - -# brightness out of range — expect 400: -curl -i -X POST -H "Content-Type: application/json" \ - -d '{"brightness":150}' http://<device-ip>/config - -# scrypted without http:// — expect 400: -curl -i -X POST -H "Content-Type: application/json" \ - -d '{"scrypted":"scrypted.local"}' http://<device-ip>/config - -# Garbage JSON — expect 400: -curl -i -X POST -H "Content-Type: application/json" \ - -d 'not json' http://<device-ip>/config -``` - -Idle-timer disable (`idle_timeout_ms: 0`) is intentionally allowed. - -Side-effects to confirm: -- After `POST /config` with `brightness`: panel brightness changes immediately (if display is up). -- After `POST /config` with `viewport` or `orientation`: mDNS TXT records update; `viewport-<name>.local` resolves; browse shows new TXT. -- After `POST /config` with both `viewport` and `scrypted` (any order, on any subsequent call): `GET /state` shows `configured: true`, `state: "asleep"`. - -**Status**: ✅ verified 2026-06-14. Full POST + partial POST + 5 validation failure modes all returned correctly. Config survives reboot (NVS persistence). `configured: true` was set automatically once both `viewport` and `scrypted` were supplied. Required raising httpd stack from 4 KiB to 8 KiB — the handler's 2 KiB body buffer plus locals were overflowing into the protect page. +> first-push silent rollback). USB is for first flash only. --- -## M5 — JPEG Frame Push - -**Acceptance**: `POST /frame` paints a 480x800 (portrait, default) or 800x480 (landscape) JPEG. Orientation determines the expected dimensions and the panel-side rotation. - -**How to verify** - -```bash -# Default orientation (portrait) — send 480x800: -curl -X POST -H "Content-Type: image/jpeg" \ - --data-binary @test-480x800.jpg \ - http://<device-ip>/frame - -# Then switch to landscape and re-test with an 800x480 JPEG: -curl -X POST -H "Content-Type: application/json" \ - -d '{"orientation":"landscape"}' http://<device-ip>/config - -curl -X POST -H "Content-Type: image/jpeg" \ - --data-binary @test-800x480.jpg \ - http://<device-ip>/frame -``` - -Visual: image fills the panel correctly oriented. After paint, `GET /state` shows `frames_received` incremented and `last_frame_ms_ago` populated. - -Generate test JPEGs with ImageMagick: - -```bash -# 8 vertical color bars at 480x800 portrait: -convert -size 60x800 gradient:white-black -duplicate 7 +append test-480x800.jpg -# or just any 480x800 JPEG: -convert -size 480x800 plasma: test-480x800.jpg -convert -size 800x480 plasma: test-800x480.jpg -``` - -Negative tests: - -```bash -# Wrong Content-Type — expect 400: -curl -i -X POST -H "Content-Type: image/png" --data-binary @anything \ - http://<device-ip>/frame - -# Oversize — expect 413: -dd if=/dev/urandom of=big.bin bs=1M count=2 -curl -i -X POST -H "Content-Type: image/jpeg" --data-binary @big.bin \ - http://<device-ip>/frame - -# Wrong dimensions — expect 400 with "expected WxH, got WxH": -convert -size 640x480 plasma: test-640x480.jpg -curl -i -X POST -H "Content-Type: image/jpeg" --data-binary @test-640x480.jpg \ - http://<device-ip>/frame - -# Concurrent posts — second gets 503: -curl -X POST -H "Content-Type: image/jpeg" --data-binary @test-480x800.jpg \ - http://<device-ip>/frame & -curl -i -X POST -H "Content-Type: image/jpeg" --data-binary @test-480x800.jpg \ - http://<device-ip>/frame # expect 503 if first is still in flight - -# Garbage bytes claiming to be JPEG — expect 400: -curl -i -X POST -H "Content-Type: image/jpeg" -d 'not a jpeg' \ - http://<device-ip>/frame -``` - -After every error: `decode_errors` in `GET /state` should increment. - -**Known gap (M6 closes this)**: M5 paints regardless of wake/sleep state. The `/frame` → `409` when asleep rule is added with `POST /state` in M6. Until then, a configured device boots ASLEEP per spec but `/frame` still paints because the state guard isn't wired yet. - -**Status**: ✅ verified 2026-06-14. Hardware JPEG decode + paint works end-to-end. Fixing it surfaced two stacked channel-swap bugs (commit 11ad249): the IDF JPEG decoder's `_RGB` element-order emits the RGB565 word in big-endian byte order (we now use `_BGR` for LE), and the ESP32-P4 DSI + TC358762 + Pi panel pipeline wants `[B, G, R]` in memory (we now pack the RGB888 framebuffer accordingly). Symmetric pixels like the white-on-black info screen masked both bugs; saturated solid colours exposed them. +## New-unit bring-up + +### Hardware prerequisites + +- **Waveshare ESP32-P4-ETH** — check the silkscreen for flash size; + `sdkconfig.defaults` declares 16 MB (a 32 MB SKU works but wastes half). +- **Hosyond 5" 800×480 Pi-architecture panel** — TC358762 bridge + ATTINY + MCU at I²C `0x45`, FT5426 touch at `0x38`. +- **15→22-pin DSI adapter cable** — the panel's 15-pin Pi FPC into the + board's 22-pin (Pi 5 / CM4 style) DSI connector. Everything goes through + the FPC: DSI lanes, panel power, and the I²C bus (board side: SDA=GPIO 7, + SCL=GPIO 8 — already what `display.c` uses). No jumper wires. +- Ethernet to a DHCP-serving switch; USB for the first flash; PoE optional. +- There is **no usable user button** (GPIO 35 is owned by EMAC TXD1); + everything is touch — no code or wiring expectations around BOOT. + +### Sequence + +1. **Board only** (no panel). First flash over USB: + + ```sh + source ~/Dev/code/git/esp32/env.sh + idf.py set-target esp32p4 + idf.py build + idf.py -p /dev/cu.usbmodem* flash monitor + ``` + + Expect `net_eth: got ip <addr>` within ~10 s of link-up. Then from + another host: `curl http://<ip>/state | jq .` and + `dns-sd -B _scrypted-viewport._tcp local.` (the TXT should carry + `version=`, `resolution=`, `orientation=`, `name=<mac-derived>`, + `mac=`). A bare board with no panel/Ethernet still boots — the flag + summary logs degraded subsystems in lowercase instead of panicking. + +2. **Attach the panel** (power off first). Pi FPCs install upside-down + easily — contact side faces the connector lock, both ends. On boot + expect `panel MCU id 0xc3 — Pi v1.1 architecture ack'd`, + `TC358762 bridge configured`, `DSI up: 800x480 26 MHz, 1-lane 600 Mbps, + non-burst`, and the info screen after a brief boot flash (the FT5426 + needs one lit scan cycle before it reports touches). + +3. **Touch**: short tap toggles wake/sleep; ≥1.5 s hold shows the info + overlay for 15 s. + +4. **Scrypted**: in the Scrypted Viewport script, "+ Add Device" — the new + unit appears in the host dropdown (name auto-fills from mDNS). Pick a + camera and triggers, save, tap the panel or ring the doorbell → live + video. See [`scrypted/README.md`](scrypted/README.md). + +### Failure-mode signals (panel bring-up) + +- `panel MCU @0x45 unreachable` → FPC seating/orientation (most common), + an adapter cable that doesn't pass I²C through, or the rare panel + variant with GT911 touch at `0x5D`/`0x14` instead of the Pi 7" + architecture this driver targets (that variant has no ATTINY at all). + Don't suspect DSI yet. +- I²C acks but no image → DSI cable orientation on the 22-pin side. +- Image but wrong colors → BGR/RGB element order (see `jpeg_decoder.c` / + `display.c` notes — the pipeline wants `[B, G, R]` in memory). +- Image with sync artifacts → `RPI_*` timing constants in `display.c` + (canonical Pi 7" values are the defaults). +- Touch acks at `0x38` but no taps → the panel hasn't had a lit scan + cycle yet (handled by the boot wake-flash) or reset-line state. --- -## M6 — State + Idle Timer - -**Acceptance**: `POST /state` toggles wake/sleep; `/frame` is 409 when asleep; idle timer fires after `idle_timeout_ms`. - -**How to verify** +## Regression recipes -First make sure the device is configured (M4 must be done): +Quick checks for the API surface after firmware changes. `V=http://<ip>`. -```bash -curl -X POST -H "Content-Type: application/json" \ - -d '{"viewport":"mudroom","scrypted":"http://host/cb","idle_timeout_ms":10000}' \ - http://<device-ip>/config -``` - -Wake / sleep: - -```bash -# Backlight off: -curl -i -X POST -H "Content-Type: application/json" \ - -d '{"state":"sleep"}' http://<device-ip>/state -# expect: 204 - -# Backlight on (loading-screen placeholder until M8): -curl -i -X POST -H "Content-Type: application/json" \ - -d '{"state":"wake"}' http://<device-ip>/state -# expect: 204 - -# Idempotency: repeating either is also 204 with no side effects: -curl -X POST -H "Content-Type: application/json" \ - -d '{"state":"wake"}' http://<device-ip>/state -curl -X POST -H "Content-Type: application/json" \ - -d '{"state":"wake"}' http://<device-ip>/state -# Both: 204 -``` +**Config validation** (each expects `400`): -`/frame` while asleep returns 409: - -```bash -curl -X POST -H "Content-Type: application/json" \ - -d '{"state":"sleep"}' http://<device-ip>/state -curl -i -X POST -H "Content-Type: image/jpeg" \ - --data-binary @test-480x800.jpg \ - http://<device-ip>/frame -# expect: HTTP/1.1 409 Conflict, body "device asleep — POST /state ..." -``` - -Idle timer: - -```bash -curl -X POST -H "Content-Type: application/json" \ - -d '{"state":"wake"}' http://<device-ip>/state -sleep 12 # idle_timeout_ms was set to 10000 above -curl http://<device-ip>/state | jq .state -# expect: "asleep" -``` - -When the timer fires, the serial log should print: - -``` -I (xxx) state: idle timer expired — sleeping -I (xxx) state: ASLEEP -``` - -Disabling the idle timer: - -```bash -curl -X POST -H "Content-Type: application/json" \ - -d '{"idle_timeout_ms":0}' http://<device-ip>/config -curl -X POST -H "Content-Type: application/json" \ - -d '{"state":"wake"}' http://<device-ip>/state -sleep 70 -curl http://<device-ip>/state | jq .state -# expect: still "awake" -``` - -Resetting on `/frame`: every successful paint restarts the idle timer. - -```bash -# Set short idle, then keep painting to keep awake: -curl -X POST -d '{"idle_timeout_ms":10000}' \ - -H "Content-Type: application/json" http://<device-ip>/config -curl -X POST -d '{"state":"wake"}' \ - -H "Content-Type: application/json" http://<device-ip>/state -for i in 1 2 3 4 5; do - sleep 8 - curl -X POST -H "Content-Type: image/jpeg" \ - --data-binary @test-480x800.jpg http://<device-ip>/frame -done -curl http://<device-ip>/state | jq .state # expect "awake" -``` - -`POST /state` always works regardless of whether `/config` has been posted — `state` is just the screen's awake/asleep, decoupled from the `configured` flag. Outbound POST-to-Scrypted is the part gated on a scrypted URL being present. - -Bad input: - -```bash -curl -i -X POST -d '{"state":"middle"}' \ - -H "Content-Type: application/json" http://<device-ip>/state # 400 -curl -i -X POST -d 'not json' \ - -H "Content-Type: application/json" http://<device-ip>/state # 400 +```sh +curl -i -X POST -d '{"idle_timeout_ms":1000}' -H 'Content-Type: application/json' $V/config +curl -i -X POST -d '{"orientation":"sideways"}' -H 'Content-Type: application/json' $V/config +curl -i -X POST -d '{"brightness":150}' -H 'Content-Type: application/json' $V/config +curl -i -X POST -d '{"scrypted":"no-scheme"}' -H 'Content-Type: application/json' $V/config +curl -i -X POST -d '{"viewport":"'$(python3 -c 'print("x"*60)')'"}' -H 'Content-Type: application/json' $V/config # >54 chars — mDNS label limit +curl -i -X POST -d 'not json' -H 'Content-Type: application/json' $V/config ``` -**Known gap (M7 closes this)**: when the idle timer fires the device transitions to ASLEEP locally but does NOT yet POST `{viewport,state:sleep}` to `<scrypted>/state`. That outbound POST lands with `state_client` in M7. +Valid partial updates merge atomically, persist across reboot (NVS), apply +brightness immediately when awake, and refresh the mDNS hostname + TXT on +name/orientation changes. `idle_timeout_ms: 0` is intentionally allowed +(never-sleep). -**Status**: ✅ verified 2026-06-14. Wake/sleep toggle 204, idempotent repeats 204, `/frame` while asleep returns 409 with expected body, bad inputs 400, idle timer fires after `idle_timeout_ms`, `idle_timeout_ms:0` correctly disables the timer. +**Frame push** (test images: `convert -size 480x800 plasma: t.jpg`): ---- - -## M7 — Touch + Outbound `/state` POST - -**Acceptance**: tap toggles wake/sleep locally; idle timer firing posts sleep; device POSTs `{viewport,state}` to `<scrypted>/state` for every local transition. - -**Test receiver** (run on the host you'll point `scrypted` at): - -```bash -# Single-file flask receiver that echoes every POST: -pip install flask -cat > /tmp/recv.py <<'EOF' -from flask import Flask, request -app = Flask(__name__) [email protected]("/state") -def state(): - print("RX", request.get_json()) - return "", 204 -app.run(host="0.0.0.0", port=11080) -EOF -python3 /tmp/recv.py -``` - -Configure the device: - -```bash -curl -X POST -H "Content-Type: application/json" \ - -d '{"viewport":"mudroom","scrypted":"http://<host>:11080","idle_timeout_ms":10000}' \ - http://<device-ip>/config +```sh +curl -i -X POST -H 'Content-Type: image/jpeg' --data-binary @t.jpg $V/frame # 204 awake, 409 asleep +curl -i -X POST -H 'Content-Type: image/png' --data-binary @t.jpg $V/frame # 400 +curl -i -X POST -H 'Content-Type: image/jpeg' -d 'not a jpeg' $V/frame # 400, decode_errors++ +# wrong dimensions -> 400 "expected WxH, got WxH"; >1MB -> 413; concurrent second POST -> 503 ``` -**Tap dispatch** +**State machine**: -- Tap while asleep → device wakes; receiver prints `RX {'viewport': 'mudroom', 'state': 'wake'}`. -- Tap while awake → device sleeps; receiver prints same with `state: 'sleep'`. -- Idle timer expires (10s here) → receiver prints `state: 'sleep'`. - -Each successful POST also logs on serial: - -``` -I (xxx) state_client: POST http://<host>:11080/state {state:wake} -> 204 +```sh +curl -i -X POST -d '{"state":"wake"}' -H 'Content-Type: application/json' $V/state # 204; repeat = 204 no-op +curl -i -X POST -d '{"state":"middle"}' -H 'Content-Type: application/json' $V/state # 400 +# idle timer: set idle_timeout_ms=10000, wake, wait 12s -> /state reports "asleep" +# and the Scrypted script logs the device's sleep callback ``` -**Failure path** - -- Stop the receiver, then tap. Serial: - - ``` - W (xxx) state_client: POST http://<host>:11080/state failed: err=ESP_OK status=-1 - ``` - - Then `GET /state` shows `state_post_failures` incremented. - -- Local state change still happens (backlight toggles). The POST is best-effort, not a precondition. - -**Depth-1 queue with replace-on-full** - -Tap rapidly (faster than the receiver can ack) and confirm the receiver only sees the most recent state transition between in-flight POSTs. Intermediate flips are coalesced. With a 1s receiver delay, only the final state of each ~1s window should hit the receiver. - -**No POST when Scrypted-initiated** - -- `curl -X POST -d '{"state":"sleep"}' /state` — receiver should print **nothing** (Scrypted already knows it initiated). -- Same for `/state {wake}` and `/frame` (asleep → 409, no callback either). - -**No POST when no scrypted URL is set** - -- Before `/config` provides a Scrypted URL, tapping the screen does nothing outbound (no Scrypted URL to call). No queue entries dropped to disk. - -**Status**: ✅ verified 2026-06-14. FT5426 acks at 0x38; short tap toggles awake/sleep, ≥1.5s hold opens the info overlay. Outbound `/state` POST gated on configured flag (untested without Scrypted). - ---- - -## M8 — Local Screens + touch long-press - -**Acceptance**: info screen on first boot; loading screen on every wake; long-press shows the info overlay. - -**Visual checks (panel-attached)** - -- Fresh flash → screen shows the info screen (~17 lines of `label value` pairs, white on black, auto-scaled). With no `/config` posted it reports the MAC-derived default name, `config no`, `state asleep`, `scrypt none`. -- `POST /config` with viewport + scrypted → device transitions to ASLEEP, backlight off. -- `POST /state {state:wake}` (or tap) → backlight on, `Loading...` centered until the first `/frame` lands. -- `POST /frame` while AWAKE → loading screen replaced by the JPEG. -- **Long-press (≥1.5s)** at any state → info overlay for 15 s with the full current config + state dump (name, host, ip, state, config, scrypt, orient, bright, idle, fw, up, frames, errs, heap, psram). -- `POST /state {state:sleep}` (or idle timeout, or tap-while-awake) → backlight off. - -**Touch gestures** (no hardware button — see Hardware note) - -- Short tap (<500 ms): toggle wake / sleep. -- Long-press (≥1.5 s): info overlay for 15 s. -- No factory-reset gesture. Use USB + `idf.py erase-flash` to clear NVS. - -**Hardware note**: ESP32-P4's strap pin GPIO35 is owned by EMAC TXD1 at runtime, and the Waveshare ESP32-P4-ETH doesn't expose any separate user button on a free GPIO. Both BOOT-button behaviours moved onto the touch panel (see M7). +**OTA**: `make ota` (full loop) or `make verify` after a manual push. The +acceptance criterion: a fresh boot must show `ota_state=pending-verify` +before flipping to `valid`; `valid` at low uptime = silent rollback to the +old slot. -**Negative / edge** - -- Info overlay during AWAKE: the overlay paints over the live frame. Scrypted's next `/frame` (within ~1s in a normal stream) overwrites it. Acceptable: the operator sees the info for ≤ 1 frame interval then the live view resumes. -- Info overlay during ASLEEP: backlight comes on for the overlay, then expires back to sleep when the 15 s timer fires. -- Font fallback: any character outside the supported set (digits, period, colon, dash, slash, `L`, lowercase a–z, space) renders as blank. Info-screen labels and IPv4 strings are all covered. - -**Status**: ✅ verified 2026-06-14. Info screen renders the full config + state dump; loading screen shown on wake when configured; info overlay returns to prior state when its 15 s timer expires. +**Device-initiated POSTs**: tap the panel — the script logs +`recv "<name>" -> wake (device-initiated)` and `state_post_failures` in +`/state` stays flat (each POST gets its 204 within the device's 1 s +timeout). Transitions Scrypted itself initiated produce no callback. --- -## M9 — Live Stream (raw TCP data socket `:81`) - -**Acceptance** (as designed: ≥ 10 fps): sustained live video from a real camera through Scrypted to the panel. - -**What shipped** (the design changed from the original `POST /stream` chunked-HTTP idea): a long-lived raw TCP connection to port 81 carrying length-prefixed JPEGs (16-byte `VPRT` v1 header — magic, length, seq, event-timestamp), received by a dedicated recv-task and handed to a decode/paint task through a 3-buffer PSRAM ring with drop-oldest semantics. See the README's *Measured per-frame budget* and *TCP window + EMAC tuning* sections for the architecture and numbers. - -**How to verify**: trigger a camera event (or tap the panel) with the Scrypted script installed; watch the script's 10 s stream-health log and `curl http://<ip>/state | jq .stream` for the firmware's 30-frame windows. - -**Status**: ✅ verified on hardware (stream server `d1c8d45` recv/decode task split; TCP window + EMAC RX pool tuning follow-up). Steady state at the Unifi medium substream: **painted = sent = 24 fps**, wire 74 Mbps avg, glass-to-glass ~40–100 ms, no drops/flushes, `decode_errors=0` across thousands of frames. Re-confirmed 2026-07-18 during the v1.4.0 verification (24 fps painted, g2g ~100 ms on the first windows of a cold start). +## Outstanding tests + +Never run — the remaining verification backlog: + +- **Mid-frame cable pull**: yank Ethernet during an active stream; expect + idle-sleep after `idle_timeout_ms`, clean recovery on reconnect. + (Idle-state link loss/recovery was verified during bring-up.) +- **24 h soak**: slow stream, verify `free_heap` / `free_psram` stable and + `decode_errors` flat. +- **Frame storm**: 30 min at max sustainable fps; no watchdog bites, no + resets. +- **Brown-out**: PoE at the edge of spec; device must reboot cleanly + without NVS corruption. +- **Multi-viewport**: needs a second physical unit — per-name callback + routing, discovery listing both, events on one camera not waking the + other viewport. +- **Depth-1 outbound queue coalescing** under the real script (verified + during bring-up against a test receiver only): rapid taps should + collapse to the final state per in-flight window. + +Race handling (concurrent tap vs Scrypted POST, mid-stream tap-to-sleep, +stale-timer-vs-fresh-wake) has been exercised continuously in real use +since bring-up; a DHCP-renumber recovery via the script's mDNS auto-heal +was explicitly verified 2026-07-18 (`host 10.0.13.99 -> 10.0.13.83 +(mdns auto-heal)`). --- -## Integration & system tests (post-M9) - -Run these once all milestones are implemented and individually verified. The point is to exercise races, edge cases, and longevity that single-milestone tests miss. - -### A. End-to-end with Scrypted - -- Real Scrypted-side Script (M7's scope): doorbell-event → `/state {wake}` → frame stream → idle timer fires → `/state {sleep}` callback → Scrypted stops. -- Bound camera switching: change the binding in Scrypted, confirm next wake routes to the new camera. - -### B. Race conditions - -- Concurrent tap-while-Scrypted-POSTs-`/state`: spam both, confirm the device converges (mutex serializes; last-write-wins). Use the wake/sleep counters in `/state` to track transitions. -- Mid-stream tap-to-sleep: while Scrypted is streaming frames, tap to sleep. Inflight `/frame` should return 409. Confirm no half-painted frames, no re-wake. -- Idle-timeout coincides with a fresh tap-wake: the second event wins. Verify by repeating with timing jitter. -- Stale Scrypted sleep timer races a fresh wake callback: Scrypted-side `cancelPendingSleep` must work — if a stale sleep lands after a wake, viewport sleeps; user taps again; recovery in one extra tap. - -### C. Failure modes - -- Cable pull mid-frame: device idle-sleeps after `idle_timeout_ms`. On reconnect, mDNS re-advertises; Scrypted re-finds and continues. (Link-loss/recovery part verified 2026-06-14 with the device idle — see M1 section. Mid-frame variant still pending.) -- Scrypted unreachable on tap: device still toggles backlight, `state_post_failures` increments. Recovery: Scrypted comes back, next tap syncs. -- DHCP lease change: device gets new IP, re-advertises via mDNS. The script's next 5-minute register fails against the stale host, which triggers its mDNS auto-heal — browse, match by `mac` TXT, rewrite host, retry. Verified 2026-07-18 by saving a wrong IP: `host 10.0.13.99 -> 10.0.13.83 (mdns auto-heal)`. -- `/state_post_failures` count should be observable via `GET /state`. - -### D. Longevity - -- 24h soak with a slow camera stream (~1 frame every 10s): verify no memory leaks (`free_heap` / `free_psram` stable in `/state`), no decode_errors growing. -- Frame storm: 30 minutes at max sustainable fps. Verify watchdog never fires, no resets. - -### E. Power & boot - -- PoE injector cycle: device boots clean, gets DHCP, re-registers via mDNS, no manual intervention needed. -- Brown-out (PoE on edge of spec): device should reboot cleanly, not corrupt NVS. - -### F. Negative protocol - -- Garbage JSON to `/config` → 400. -- `/frame` with `Content-Type: image/png` → 400. -- `/state` with `{"state":"middle"}` → 400. -- `/frame` with body > 1 MB → 413. -- Two concurrent `/frame` posts → one wins, the other 503. - -### G. Multi-viewport - -Once at least two physical units are configured: confirm each routes its callback to Scrypted with its own `viewport` name; confirm Scrypted's discovery map has both IPs; confirm a camera event on the wrong-named viewport is correctly ignored on the right one. - ## Performance review playbook -Repeatable methodology for "is the device still hitting its baseline" and "where did the time go." Run after every commit that touches the data plane. +Repeatable methodology for "is the device still hitting its baseline" and +"where did the time go." Run after every commit that touches the data +plane. Baseline (Unifi medium substream, `-q:v 1`): **painted = sent = +24 fps**, wire ~74 Mbps avg, g2g ~40–100 ms, zero drops/flushes. ### Per-session capture -Three log streams, all timestamped: - ```sh # Firmware serial — 30-frame window summaries idf.py monitor | tee fw-$(date +%s).log & # /state poll — windowed stats + glass-to-glass anchor -while true; do - curl -s http://VIEWPORT_IP/state - echo - sleep 1 -done | tee state-$(date +%s).log & +while true; do curl -s http://VIEWPORT_IP/state; echo; sleep 1; done \ + | tee state-$(date +%s).log & # Scrypted plugin console — copy/paste from the web UI when done. ``` -Trigger a wake event (motion or "Wake now" button). Let it run two minutes. Stop captures. +Trigger a wake (panel tap is the reliable manual trigger; camera events +otherwise). Let it run two minutes. First window after a stream starts is +transient — read steady state from the later windows. -### What the firmware serial says +### What the firmware serial says (every 30 painted frames) -Every 30 painted frames: - -``` -30 frames over 1.8s: 16.5fps 2.95MB/s avg-jpeg=180KB | - lock min/avg/max=8/9/12us | - recv min/avg/max=21000/35000/48000us | - dec min/avg/max=5100/5600/6200us | - paint min/avg/max=38/45/61us | - idle min/avg/max=120/8500/40000us -``` - -### What `/state` says (Phase 4 onward) - -```json -{ - "stream": { - "frames": 30, "bytes": 5400000, "window_us": 1800000, - "recv_min_us":21000,"recv_avg_us":35000,"recv_max_us":48000, - "dec_min_us":5100,"dec_avg_us":5600,"dec_max_us":6200, - "paint_min_us":38,"paint_avg_us":45,"paint_max_us":61, - "idle_min_us":120,"idle_avg_us":8500,"idle_max_us":40000, - "last_paint_event_us_low": 2148503712 - } -} -``` - -`last_paint_event_us_low` is the script's monotonic µs low 32 bits at the wake event that produced the most recent painted frame. Glass-to-glass is `(scripts_now_us_low - last_paint_event_us_low)` with 32-bit wrap. +One line per window with min/avg/max for every stage plus the TCP-window +decomposition — `wire` (throughput while a body drains; ceiling ≈ +`TCP_WND/RTT`), `hdr_gap` (sender idle), `pend_age` (handoff wait), `recv`, +`dec`, `paint`, `idle`, `queued`, `recv_calls`, `recv_chunk`, +`drop-oldest`. The same numbers land in `/state`'s `stream` object +(semantics documented in `stream_server.h`). ### What the Scrypted console says -On wake: -``` -event MotionSensor -> "kitchen": fired at +0ms (wake) -stream "kitchen": start +0ms -stream "kitchen": socket connect requested +1ms -snapshot "kitchen": start +1ms -snapshot "kitchen": takePicture +320ms -stream "kitchen": socket connect open +24ms -snapshot "kitchen": transform +458ms via sharp (217KB) -snapshot "kitchen": post sent +459ms -snapshot "kitchen": post acked +551ms ← first user-visible paint -stream "kitchen": first ffmpeg frame +780ms (jpeg=210KB) -stream "kitchen": first socket.write +781ms -``` - -Every 5s during an active stream: -``` -firmware "kitchen": fps=16.7 2.99MB/s | recv=...us | dec=...us | paint=...us | idle=...us | g2g=140ms -``` +Cold-start stamps on each wake (`socket connect open +Xms`, +`first ffmpeg frame +Xms` — ~0.7 s prebuffered, ~5–6 s cold), then a 10 s +stream-health line **only when noteworthy** (drops, a buffer-cap flush, +fw-skipped ≥ 2 fps, painted < 20 fps): sent/painted fps, drop counters, +socket-write latency percentiles, backpressure duty cycle, `node_buf` +depth, firmware stage timings, chip temp, and `g2g`. ### Investigation thresholds (alarms) | Symptom | Probable cause | First check | |---|---|---| -| Painted fps sustained < 12 | Upstream starvation | `idle_avg_us` in firmware log — high = ffmpeg behind | -| `decode_p95 > 8ms` | HW decoder pathology | Recent `decode_errors` delta; PSRAM cache pressure | -| `paint_p95 > 200µs` | DPI or DMA stall | Look for `DSI` warnings in serial; PSRAM bandwidth | -| `idle_avg > 50ms` | ffmpeg falling behind | Check Scrypted CPU; try lower-rate camera substream | -| `g2g_p95 > 500ms` | User-facing regression | Bisect: is the gap in script-side or firmware-side? | -| Any non-zero `decode_errors` delta | JPEG corruption | TCP retransmit storm or memory corruption | +| Painted fps sustained < 20 | Upstream starvation or wire regression | `hdr_gap` high = sender idle; `wire_avg` pinned = window-bound | +| `dec_avg > 8ms` | HW decoder pathology | `decode_errors` delta; PSRAM cache pressure | +| `paint_max > 200µs` | DPI or DMA stall | `DSI` warnings in serial; `tear_guard_engaged` growth rate | +| `pend_age` growing window-over-window | Kernel-queue latency backlog | The failure mode that killed the naive TCP-window raise — check EMAC RX pool ≥ `TCP_WND` | +| `g2g > 500ms` sustained | User-facing regression | Bisect script-side (`node_buf`, bp%) vs firmware-side (recv/dec) | +| Any `decode_errors` delta | JPEG corruption | TCP retransmit storm or memory corruption | ### Tools -`grep`, `awk`, `gnuplot`. Anything beyond that is over-engineering for a single device's logs. +`grep`, `awk`, `gnuplot`. Anything beyond that is over-engineering for a +single device's logs. |