| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
|
|
User reported "snapshot is still lower qual than the stream" even at
jpegQuality=1. Two compounding causes in the sharp transform path:
1. Quality mapping topped out at 97. Previous formula was
Math.max(50, 100 - v.jpegQuality * 3); at jpegQuality=1 that
produced quality=97 — versus the stream's ffmpeg mjpeg -q:v 1
which lands at sharp-equivalent ~99-100. Visible compression
artifacts on flat regions accounted for ~half the perceived
quality gap. New formula Math.min(100, 102 - v.jpegQuality * 2)
yields:
jpegQuality=1 → 100 (was 97)
jpegQuality=2 → 98 (was 94)
jpegQuality=5 → 92 (was 85)
jpegQuality=10 → 82 (was 70)
jpegQuality=31 → 40 (was 50, clamped)
2. Chroma subsampling was sharp's default 4:2:0 — half-rate chroma
in U+V planes. On colored edges (UI overlays, sharp camera
subjects against backgrounds, text) this smears and is the
dominant visible artifact at panel-native 800x480. ffmpeg
mjpeg's default behavior at -q:v 1 is closer to 4:2:2 or no
subsampling. Force 4:4:4 when jpegQuality <= 2 (the "I want max
quality" regime); keep 4:2:0 for jpegQuality >= 3 since the
point at that quality is smaller files.
Plus mozjpeg: true on the encoder. Same quality value, slightly
tighter file size — modest but free.
The Lanczos kernel was already lanczos3 — confirmed, no change.
Expected outcome: snapshot at jpegQuality=1 is visually
indistinguishable from a stream frame at the same scene.
|
|
|
|
Adds per-stage timing logs to both the stream and snapshot paths,
all relative to a single t=0 anchor: the moment the camera event
arrived in handleCameraEvent. Lets us see whether snapshot and
stream actually start in parallel (they should, within ~2ms) and
where wall-clock goes inside each path.
handleCameraEvent now captures tEvent = Date.now() and threads it
into startStream(v, tEvent). startStream threads it further into
pushSnapshot(v, cam, tEvent). Both helpers define a `since()` arrow
that computes the current offset.
New log lines on wake:
event MotionSensor -> "kitchen": fired at +0ms (wake)
stream "kitchen": start +0ms
stream "kitchen": socket connect requested +1ms
snapshot "kitchen": start +1ms
snapshot "kitchen": takePicture +320ms
stream "kitchen": socket connect open +24ms
snapshot "kitchen": transform +458ms via sharp (217KB)
snapshot "kitchen": post sent +459ms
snapshot "kitchen": post acked +551ms ← first user-visible paint
stream "kitchen": first ffmpeg frame +780ms (jpeg=210KB)
stream "kitchen": first socket.write +781ms
post_acked is the snapshot's true glass-to-glass: /frame returns
after display_flip_back_buffer, so by the time the response
resolves the firmware has the new pixels queued for the next DPI
scanout. No separate firmware-side wiring needed for the snapshot
path's g2g measurement.
The stream path's true g2g still needs the 16-byte header
extension in Phase 4 — `first socket.write` is just "Scrypted sent
the bytes," not "panel showed them."
|
|
|
|
Both sides simultaneously because the script's X-Frame-Seq sender and
the firmware's X-Frame-Seq receiver negotiated a contract that's now
retired entirely.
Firmware (main/http_api.c):
- Delete static uint32_t s_last_painted_seq (declaration + reset in
state_post_handler + the read in frame_post_handler + the
assignment after paint).
- Delete the X-Frame-Seq header read (httpd_req_get_hdr_value_str
+ strtoul block).
- Delete the X-Frame-Drop: stale-seq response path.
- Delete the entire Server-Timing httpd_resp_set_hdr block.
- Delete the setsockopt(TCP_NODELAY) at frame_post_handler entry.
/frame is a single body POST → empty 204; no second packet for
Nagle to coalesce with on the response side.
- Delete static int64_t s_last_post_us + the idle_us computation.
/frame fires at most once per wake; idle gap between wakes is
dominated by user/event timing, not anything firmware-controllable.
- Drop the per-10-frames condition on the timing log — /frame fires
rarely enough that one log per snapshot is the right cadence —
and rename "frame N: ..." → "snapshot: ..." to match.
- Drop cfg.max_open_sockets 4 → 2 (snapshot POST + concurrent /state
or /config).
- Drop #include "lwip/sockets.h" (orphaned with TCP_NODELAY).
Script (scrypted/scrypted-viewport.ts):
- Delete the agents Map + agentFor() method (per-host keepAlive
Agent pool; over-engineered for ~1 POST/min control plane).
- Delete httpRequest() helper (40 lines wrapping http.request to
surface tHeaders/tDone — no consumer reads those fields anymore).
- Rewrite postJSON() to a 10-line fetch() with AbortSignal.timeout.
- Delete the frameSeq Map + the seq counter + X-Frame-Seq header on
the snapshot fetch + the X-Frame-Drop response check + the
frameSeq.delete in stopStream.
- Extract buildVf(orientation, panelW, panelH) helper near top of
ScryptedViewportProvider — used by both startStream (live) and
pushSnapshot (one-shot ffmpeg fallback).
TCP_NODELAY references remain in the live-stream socket path
(scrypted-viewport.ts:773, 788). That's a different socket (raw
net.Socket on TCP/81) and noDelay there is what eliminates Nagle
stalls under the streaming-heavy live workload. Load-bearing.
|
|
|
|
Removes/rewrites every comment that referenced the dead HTTP-streaming
architecture so a reader of v1.0.0+ source isn't chasing a model
that's been gone for several releases. No code paths changed; this is
the safe pre-pass before Phase 2's actual deletions.
Firmware (main/http_api.c):
- s_last_painted_seq / s_last_post_us / X-Frame-Seq parse / stale-
frame guard / Server-Timing emission / TCP_NODELAY setsockopt /
max_open_sockets=4 — each block now leads with "(Legacy from the
HTTP-streaming era; removed in Phase 2)" so the reader knows the
block is doomed, not load-bearing.
- /frame dim-mismatch comment narrowed: panel-native is always 800x480
BGR888, no Scrypted-side variation expected; Scrypted does the
rotation+scale via sharp/mediaManager/ffmpeg cascade (snapshot) or
ffmpeg -vf (stream).
- "Single in-flight frame. Concurrent posts get 503" rewritten to
reflect: decoder mutex now mostly serves to fence /frame snapshots
against an active stream_server decode.
Firmware (main/stream_server.c:136-142):
- FIONREAD-skip rationale reduced from a 7-line paragraph to one
sentence; the savings/tradeoff math now lives in the plan, not the
per-line comment.
Script (scrypted/scrypted-viewport.ts):
- Top-of-file tuning constants block drops the "frame_interval_ms
removed", "fps filter", "in-flight back-to-back startStream"
rationale; one short line covers the model: "Stream rate is paced
by camera + TCP backpressure; no app-level fps cap."
- agentFor() rationale rewritten: this Agent is over-engineered for
control-plane traffic (~1 POST/min steady state) — a legacy of when
it backed per-frame /frame POSTs. Marked for Phase 2 retirement.
- noDelay on Agent: clarified it's now a no-op safety for control
plane (was load-bearing for live-stream pipelining).
- snapshot fire-and-forget comment: replaced X-Frame-Seq-race
rationale with the actual TCP-streaming truth (sharp/mediaManager/
ffmpeg cascade race against stream socket bring-up).
- writeLatencies probe: rewrote the "keep-alive socket" comment
(live stream uses raw net.Socket, not the http.Agent pool).
- socketBackpressured: explicit "diagnostic only, never gates writes"
comment added at declaration site.
- skipLogger header: rewrote the inFlight/MAX_INFLIGHT/fps-filter
rationale into a one-line description of what the log line actually
emits today.
- frameSeq map: now flagged as legacy of HTTP-streaming era, retired
in Phase 2.
|
|
First milestone where the device runs the way it was intended end-
to-end:
- Raw-TCP streaming data plane at ~14-20 fps painted, sub-ms socket
writes, no Nagle/ACK pathology.
- HTTP control plane (/state, /config) plus HTTP /frame for the
one-shot snapshot fast path on wake. Snapshot via sharp → ~700ms
first paint from a cold trigger (down from ~1000ms+ via ffmpeg).
- Firmware-side "always paint the latest" — FIONREAD skip drops
superseded frames before decode, keeping glass-to-glass tight
even when upstream produces faster than we can ingest.
- HW JPEG decode + zero-copy paint into the panel back framebuffer.
- 12 fps goal hit cleanly at q:v 1 (visually lossless). Source
pinned to medium-resolution substream so the camera supplies
enough frames to actually fill the pipe.
- Scrypted side: per-host node:http keep-alive Agent with NODELAY,
the cascading sharp → mediaManager → ffmpeg snapshot transform,
wake-trigger picker on the new-device dialog, and the leaked-
setInterval-across-script-reloads bug squashed.
Bumping VIEWPORT_VERSION 0.1.0 → 1.0.0 so the boot log and info
screen reflect it.
|
|
This reverts commit 6e36cbc14a06b8c0e15bfe78afb8adb43f8909ba.
|
|
Hypothesis confirmed by the symptoms: with WND_DEFAULT=65535,
SND_BUF=65535, RECVMBOX=16, the firmware boots fine and the stream
server logs "listening on tcp/:81", but Scrypted's first SYN never
produces a "client connected from ..." entry. The script-side
socket stays stuck in pending-connect — neither a connect nor an
error event fires — and every ffmpeg-emitted frame gets dropped.
Root cause is almost certainly lwIP's PBUF / MEMP pools not being
scaled to back the 64KB windows on multiple sockets (httpd has 4,
stream has 1, that's 5 × 64KB = 320KB of buffer commitment from a
heap that's also feeding PSRAM-backed framebuffers, JPEG scratch,
and Ethernet DMA). lwIP doesn't surface the allocation failure as
an accept error — it just refuses to complete the handshake.
Backing off the changes for now. The streaming path already hits
~14-20 fps at our measured 5.3 MB/s ingest ceiling under the lwIP
defaults — that's plenty above the 12-fps goal. Revisit window
tuning later by also bumping CONFIG_LWIP_PBUF_POOL_SIZE and
MEMP_NUM_TCP_PCB so the buffer commitment is actually allocatable.
Keeping the FIONREAD skip in stream_server (it's orthogonal to lwIP
buffer sizes and only helps).
|
|
|
|
backpressure-blind + lwIP TCP window bump
Three changes that work together to make the stream "always paint
what's freshest, never sit on a stale frame":
#1 — Firmware FIONREAD skip in stream_server
Right after the body of frame N comes off the wire (and before we
unlock the decoder + spend ~6ms on decode + paint), check the
kernel receive buffer with ioctl(FIONREAD). If at least one more
header (8 bytes) is queued, frame N is no longer the freshest
possible — skip its decode + paint and loop back to read frame
N+1. The TCP recv cost is unavoidable (bytes still have to cross
the wire) but the decoder + paint cost is saved on every superseded
frame. Glass-to-glass latency on the latest frame drops by however
many frames had backed up.
#2 — Scrypted: keep writing past kernel-buffer backpressure
Previously: when sock.write() returned false we dropped the next
ffmpeg frame at source. New: we keep writing through. Node buffers
internally; under our load (~16 MB/s ffmpeg → ~5-7 MB/s firmware)
the buffer rarely exceeds a frame or two. With the firmware now
silently skipping decode on backed-up frames (#1), excess frames
get shed for free on the device side. Scrypted's job is just to
hand the firmware the freshest bytes as fast as possible.
socketBackpressured is still tracked for the diagnostic log.
#3 — lwIP TCP window bump (revisiting earlier regression)
The previous attempt at LWIP_TCP_WND_DEFAULT=32k regressed under
HTTP because every /frame opened a fresh socket and we paid the
slow-start cost repeatedly. The streaming pivot eliminated that:
the socket is long-lived, slow-start runs exactly once, then we
ride the full window for the rest of the session. Bumping to
65535 (max for stock lwIP), SND_BUF to match, RECVMBOX to 16, SACK
on. Expected: recv throughput ceiling moves up from ~5.3 MB/s,
which directly raises the fps ceiling (recv is currently 37ms of
the 43ms per-frame total).
|
|
|
|
interval
#1 — Wake triggers in the new-device dialog
The "+ Add Device" form was missing the Wake triggers multi-select,
so a new viewport defaulted to the per-getter fallback of all three
(doorbell + motion + person) silently and the user only saw the
field after first edit. Now the dialog includes it up front with
all three pre-selected, and createDevice persists whatever the user
ticked into storage in the same JSON shape Viewport.putSetting uses
for subsequent edits.
#2 — Stop leaking the periodic re-register interval
Why the user was seeing the "registered ..." log line repeat 14
times in steady-state with nothing happening: Scrypted's Scripts
sandbox does NOT garbage-collect setInterval handles when a script
is re-pasted/reloaded. Every re-paste left an orphan interval
running against the previous Provider instance, accumulating one
extra timer per reload. Every 5 minutes (REREGISTER_INTERVAL_MS),
all N timers fired at once, producing N "registered ..." log lines
in rapid succession.
Fix: keep the timer handle on globalThis under a well-known key. At
script start, clearInterval the previous one (if any) before
arming the new one. Idempotent across reloads.
|
|
|
|
Reverts the per-viewport "Camera substream" UI added in 6d74d02 —
having a knob the user has to find and tune is the wrong shape. The
goal is "best quality, highest fps that's actually achievable" and
that's a deterministic walk, not a user choice.
New walk: medium-resolution → local → remote → camera-default.
Explicitly NOT in the list:
- low-resolution: the camera's preview substream, capped at the
~5-8 fps that's been bottlenecking us.
- remote-recorder: has the camera's ~10s prebuffer baked in (we'd
display the past rather than the present).
Wire cost is unchanged: every input resolution is re-encoded to
panel-native 800x480 mjpeg q:v 1 before going to the device. The
only tradeoff for picking the main stream is Scrypted-side ffmpeg
CPU, which is plentiful. Throughput to the firmware is the same;
upstream camera fps is what changes (5fps → 15-30fps typical).
|
|
|
|
Three independent improvements landing together because they all
target the post-streaming-pivot "where do we spend the wall clock?"
question.
#1 — Snapshot: sharp → mediaManager native → ffmpeg cascade
pushSnapshot now tries three transforms in order of cost:
- sharp (~5-15ms, libvips bindings, handles resize + rotate)
- mediaManager.convertMediaObjectToBuffer with image/jpeg;width=W
;height=H mime hint (~10-30ms, Scrypted's native converter,
used only for landscape since rotation isn't standard)
- ffmpeg one-shot (~500-700ms cold start, the old slow path)
A `path=...` field in the snapshot log identifies which transform
actually ran so the user can confirm the fast paths are reachable
in their Scrypted runtime. takePicture, transform, and POST timings
are each broken out so we can see exactly where the snapshot wall
goes.
#2 — Firmware: windowed min/avg/max breakdown + idle-gap
Stream server replaces the per-frame single-sample log with a 30-
frame window summary:
N frames over Xs: Yfps Z MB/s avg-jpeg=KB |
lock min/avg/max | recv min/avg/max | dec min/avg/max |
paint min/avg/max | idle min/avg/max
- lock = mutex acquire time (sanity check; should be ~0us with one
client owning the decoder)
- recv = body bytes off the wire
- dec = HW JPEG decode
- paint = backbuffer flip + DMA queue
- idle = gap between previous paint completing and next header
landing (= upstream slack). Large idle means we're waiting on
ffmpeg/network; near-zero means we're the bottleneck.
#3 — Camera substream picker
"Stream-source choice drives end-to-end latency more than anything
else" — the existing hardcoded low-latency-first walk lands on the
camera's preview substream which is typically capped at 5-8 fps.
Adds a per-viewport setting under Display:
Camera substream: auto | low-resolution | medium-resolution
| local | remote | remote-recorder
auto keeps the current behavior; the pinned options let the user
force a higher-fps source when they want stream rate > preview
rate. The chosen destination is logged at stream start.
|
|
|
|
Two settings cleanups now that streaming has landed:
- frame_interval_ms removed entirely. Under the TCP streaming data
plane ffmpeg emits at the camera's native rate and TCP backpressure
naturally caps us when the firmware can't keep up. The setting,
the getter, the UI field, and the -vf "fps=N" filter argument all
go away. Net: a few fewer questions to answer on every viewport,
and one fewer place for the user to misconfigure latency. Existing
storage values are silently ignored on next render.
- Default brightness 80 → 100. The panel is dim enough at 80 that
the change in ambient lighting can wash it out; 100 is a better
default. Users who want lower can still set it in the UI; this
only affects newly-created viewports (existing ones keep whatever
was last saved).
|
|
|
|
Replaces the per-frame HTTP POST loop with a single long-lived TCP
connection on port 81. The HTTP control plane (/state, /config,
/frame for snapshot) stays unchanged.
Wire protocol (big-endian, repeating until connection close):
[4 bytes jpeg_len][4 bytes seq][jpeg_len bytes JPEG]
Why
---
The HTTP path hit a measured floor of ~37ms p50 with intermittent
~230ms p95 spikes that survived every Nagle/keep-alive fix attempt.
Each frame paid: TCP setup (or pool churn), HTTP parsing, body recv,
decode, paint, response write, response ACK. Streaming removes
everything except recv + decode + paint.
Firmware (main/stream_server.[ch], new)
---------------------------------------
- TCP listen on configurable port (81) in its own FreeRTOS task,
one client at a time (matches the one-stream-per-device model).
- Per accepted socket: TCP_NODELAY on, then loop reading 8-byte
header → jpeg body → through the existing jpeg_decoder + display
paths. Same stale-seq guard as the HTTP /frame handler (reset per
connection so each session starts at seq 1).
- Frames received while asleep are still drained (to stay framed)
but not painted. The HTTP control-plane POST /state {wake}
resumes painting on the next frame.
- Every 30 painted frames a structured serial log shows per-stage
timing + sustained MB/s — replaces the cross-side Server-Timing
header (no HTTP response to attach it to anymore).
Scrypted side
-------------
- net.createConnection({ host, port: 81, noDelay: true }) opened
once per stream session. On disconnect/error/close we auto-
reconnect after 500ms.
- ffmpeg stdout demux writes [header][body] directly to the
socket. sock.write() returning false sets a backpressured flag
that drops incoming ffmpeg frames until 'drain' fires — natural
TCP backpressure handles "firmware can't keep up" without us
modeling it manually.
- Stripped the entire fetch-based timing infrastructure
(pushStreamFrame, fetchSamples, parseServerTiming, depth
histogram, Server-Timing parser). Replaced with one stream-shaped
log every 10s: fps + MB/s + socket.write p50/p95/max + drop count
+ backpressured flag.
- pushSnapshot (one-shot first-paint) still uses the HTTP /frame
endpoint — small and infrequent, not worth reworking.
- postJSON still uses node:http for /state and /config.
State machine status flags
--------------------------
Extended boot-time flags array from 6 → 7 slots so the stream
server's bring-up shows up alongside ETH/MDNS/HTTP. Layout is now
E M H S D J T (was E M H D J T).
|
|
|
|
After enabling keep-alive in bccf0fa, fw_recv p95 spiked from ~25ms
to ~230ms on a significant fraction of frames and net_up p95 went
from <15ms to 200+ms simultaneously. Worst frames hit 467ms wall.
Root cause: Node http.Agent's outgoing sockets default to noDelay
false (Nagle ON). When a ~128KB JPEG body doesn't end on an exact
MTU boundary, the sender holds the final partial packet for up to
200ms waiting for either more data or a peer ACK. Firmware-side
TCP_NODELAY only controls what the firmware *sends* (its ACKs and
response packets); it has no effect on what arrives at the firmware
from a Nagling sender.
Setting noDelay: true on the http.Agent makes new sockets in the pool
opt out of Nagle on creation. Plus a belt-and-suspenders setNoDelay
on the request's socket event in case the Agent-level option isn't
honored on this Node version.
The 230ms signature is the classic Nagle + delayed-ACK deadlock:
- Sender: "I have a partial packet; I'll wait for more data or an
ACK before sending."
- Receiver: "I'll delay this ACK up to 200ms in case I can piggyback
it on a response."
- Result: 200ms stall on every send that doesn't fill an MTU.
|
|
|
|
Adds a "JPEG quality" field under Display alongside frame_interval_ms,
brightness, etc. ffmpeg -q:v range 1..31 — 1 = highest quality + biggest
JPEG (~140KB at panel-native), higher numbers = smaller + lossier.
Plumbed through both the live-stream encoder and the snapshot one-shot
encoder. Reads from viewport storage; defaults to 1 if unset; clamps
to [1, 31] on read so an out-of-range input from the UI doesn't crash
ffmpeg.
Changing the value triggers the existing onBindingChanged debounce
which already restarts the live stream — picks up the new quality
within ~300ms of Save.
|
|
Two changes in one commit because the keep-alive refactor reshapes
the fetch surface and the quality bump piggybacks naturally on it.
#1 — Keep-alive
ScryptedViewportProvider now uses node:http (always available, no
require sandbox dance like undici) with a per-host Agent:
keepAlive: true, keepAliveMsecs: 30s, maxSockets: 2.
Pool size 2 matches the firmware's two-socket pipelining capacity
so frame N+1 can begin uploading on socket B while frame N is still
being decoded on socket A. Reusing the socket skips the SYN+SYN-ACK+
ACK round-trip on every POST — on a quiet LAN that's ~1ms saved per
frame, and far more on the p95 tail where TCP slow-start was driving
26-35ms net_up spikes in the measured data.
httpRequest() wraps http.request to expose tHeaders + tDone (so we
keep the existing per-stage script-side timing intact) and returns
{ status, headers, tHeaders, tDone }. Both pushStreamFrame and
postJSON migrated. The remaining fetch() calls (GET /state, GET
/config from the UI, and the one-shot snapshot POST) are non-hot
paths — left as fetch for now.
#2 — JPEG quality
ffmpeg -q:v 2 was the de facto "visually lossless" setting in earlier
notes; bumping to -q:v 1 squeezes one more notch of quality out of
the mjpeg encoder. JPEG size grows ~10-15%; with NODELAY landed and
fw_recv at ~22ms for 128KB we have plenty of headroom for the bigger
bodies. Applied to both the live-stream encoder and the one-shot
snapshot encoder.
|
|
s_last_painted_seq
Two build breaks discovered when actually compiling the prior commits:
- main/CMakeLists.txt missing esp_app_format requirement, so
esp_app_desc.h couldn't be resolved when app_main.c included it for
the boot-time git-hash log.
- s_last_painted_seq is referenced inside state_post_handler (reset
to 0 on /state wake) but the static was declared further down the
file alongside the other /frame state. C requires declaration
before use — forward declare it above state_post_handler and keep
a stub comment at the original location.
|
|
|
|
#1: TCP_NODELAY for /frame
ESP-IDF lwIP defaults Nagle ON. /frame is the worst Nagle workload:
~140KB body POST followed by an empty 204 response, no follow-up data
either way. Both the final partial-MTU body packet and the response
packet can sit in the kernel send buffer up to 40ms waiting for an
ACK that the other side is delaying-ACKing — silently adding tens of
ms to every frame's wall time and showing up as a fat net_up bucket
on the Scrypted side.
setsockopt(TCP_NODELAY) at handler entry on every /frame. Cheap and
idempotent. Includes the /state and /config sockets too — those are
infrequent but small responses also benefit.
#2: min/max + worst-frame decomposition in the Scrypted log
The p50/p95 line answered "what's typical" but not "what happened in
the worst frame this window". Now every 10-fetch log adds:
- min and max columns alongside p50/p95 per bucket
- a worst-frame breakdown row: identifies the single slowest fetch
in the window and decomposes its wall time across all stages.
Answers "did the slow frame get gated by emit→post (ffmpeg
backed up), net_up (TCP/handshake spike), or fw_dec (large
JPEG)?" directly, instead of us inferring from percentiles.
Together these are the prerequisites for evaluating whether further
optimization (HTTP keep-alive, chunked streaming) is worth pursuing.
|
|
|
|
Cross-side timing was opaque: script saw "req=70ms" but couldn't
split TCP/dispatch overhead from firmware decode time, and firmware
serial logs and Scrypted console logs couldn't be correlated.
Firmware /frame handler now emits Server-Timing on every response
with per-stage breakdown:
Server-Timing: recv;dur=X, dec;dur=Y, paint;dur=Z,
post;dur=W, handle;dur=total
Script parses it, joins by X-Frame-Seq implicitly (one POST per seq),
derives net_up = req − fw_total, and logs a multi-line p50/p95
breakdown every 10 fetches:
fetch "kitchen" #10 (jpeg=137KB)
wall p50=65ms p95=140ms
emit→post p50=0ms p95=2ms (queue wait)
req p50=62ms p95=135ms (fetch → Response headers)
net_up p50=43ms p95=110ms (TCP + body wire + dispatch)
fw_recv p50=12ms p95=18ms (body off the wire)
fw_dec p50=6ms p95=8ms (hardware JPEG)
fw_paint p50=0.1ms p95=0.2ms (backbuffer flip)
fw_post p50=0.4ms p95=0.6ms
body-read p50=1ms (drain — empty body)
inflight d0=8 d1=2 d2=0
stale-drops=0
Plus version stamps on both sides for the "is the user on the right
code" question:
- CMakeLists: PROJECT_VER = git short hash + -dirty marker if dirty.
esp_app_get_description()->version exposes it at runtime. Boot
log: "Scrypted Viewport boot (v0.1.0 build=4cf36e2-dirty)".
- TS: SCRIPT_VERSION const at the top, bumped per commit, logged at
script-eval: "Scrypted Viewport up (script=4cf36e2). Callback ..."
|
|
ffmpeg cold-start + RTSP connect + first-keyframe-wait puts 0.5–3
seconds of dead air between a tap/event and the first stream frame
landing on the panel. Most of that is unavoidable for the live
stream, but Scrypted can almost always produce a snapshot in
50–300ms via camera.takePicture() (often a cache hit).
New pushSnapshot path:
- Fires in parallel with the main stream spawn — does NOT delay the
stream path even if takePicture is slow.
- takePicture → quick one-shot ffmpeg with the same transpose+scale
filter chain → POST /frame, all under a 2s hard cap.
- Uses the shared X-Frame-Seq counter: if a stream frame beats the
snapshot to the firmware decoder (unlikely on cold start, possible
on warm reconnect), the firmware silently stale-drops the snapshot.
Logs "beaten by stream frame" when that happens so we can see it.
- Errors are swallowed silently — a camera that doesn't support
snapshots just falls through to the normal stream-only path.
Net effect: the panel shows the camera near-instantly on every wake,
then the snapshot gets replaced by the first ffmpeg frame whenever
it lands.
|
|
Scrypted's plugin sandbox doesn't always expose 'undici' as a
require()-able module — it's bundled into the runtime but only
reachable through globalThis.fetch internals, not as a CommonJS
module. require('undici') threw and broke registerViewport entirely.
Wrap the require in try/catch and cache the result. When undici is
available we still get keep-alive + 2 socket pool for pipelining;
when it isn't, dispatcherFor returns undefined and we fall back to
plain fetch with a fresh socket per POST. The X-Frame-Seq
deduplication on the firmware side still works regardless — even
without socket pooling, two fetch() calls can be in flight on two
separate ephemeral sockets concurrently.
|
|
Replaces the single-sample "fetch #N: Xms" log with an aggregated
window every 10 fetches showing where the wall-clock budget actually
goes:
fetch "kitchen" #10 (jpeg=147KB) wall p50=52ms p95=78ms |
emit→post p50=0ms p95=14ms | req p50=51ms p95=76ms |
body-read p50=0ms | inflight d0=4 d1=5 d2=1 | stale-drops=0
Buckets:
- emit→post: ffmpeg pushed JPEG to stdout → fetch() actually called.
Nonzero = inFlight queue was full, frame waited.
- req: fetch() → Response headers (body upload + firmware ttfb +
decode start + status line).
- body-read: Response → drained. Should be ~0 (empty body responses).
- wall: total of above three (matches the old single-sample number
but with p95 visible).
Plus inflight depth histogram (d0/d1/d2 = how many other POSTs were
in flight when this frame queued) — d0 dominating means pipelining
isn't doing anything; d1/d2 mass means real overlap. stale-drops
counts X-Frame-Drop responses so we can see how often the firmware
rejected an out-of-order pipelined frame.
|
|
With two /frame POSTs in flight on separate sockets, the firmware's
JPEG decoder mutex serialises decode but FreeRTOS semaphore
acquisition is not FIFO — under jitter the later-arriving older
frame can grab the lock first and paint over the newer one,
producing brief "panel travels backward in time" glitches.
Scrypted side:
- Per-viewport monotonic frameSeq counter, sent as X-Frame-Seq on
every /frame POST. Reset on stopStream so each new stream starts
at 1.
Firmware side:
- s_last_painted_seq tracks the highest seq we've painted. Frame
arrives, mutex acquired, body received — if seq <= s_last_painted_seq
the frame is dropped (200 OK + X-Frame-Drop: stale-seq header) and
the mutex released without touching the back buffer.
- s_last_painted_seq resets to 0 on POST /state {wake} so the next
stream's seq=1 isn't rejected because the previous session reached
a higher counter.
- Missing/zero X-Frame-Seq (legacy clients) skips the check entirely
— preserves pre-pipelining behaviour.
|
|
Decouples network upload from firmware decode-and-paint. Before, the
inFlight boolean guard meant Scrypted sent frame N, waited for the
full ~80ms response (~40ms body upload + ~20ms decode + ~50µs paint +
ack), THEN sent frame N+1. The next ffmpeg frame arriving during that
window got dropped.
After:
- ScryptedViewportProvider keeps a per-host undici Agent with
keepAliveTimeout 30s, connections:2, pipelining:0. Sockets stay
open across /frame, /state, /config — saves the SYN+SYN-ACK+ACK
round-trip every POST (small on LAN but real on Wi-Fi).
- Stream loop's inFlight boolean is now a counter capped at 2 so
frame N+1 can begin uploading on socket B while frame N is still
being decoded on the device via socket A. Roughly doubles effective
throughput when body upload time dominates.
Firmware side:
- esp_http_server max_open_sockets bumped 7→4 explicitly: 2 for
pipelined /frame + 2 spare for concurrent /state and /config slots.
The JPEG decoder mutex still serialises decode (only one /frame
can be decoding at a time); this change only unblocks the network
half of the pipeline.
|
|
The single warning conflated two unrelated symptoms:
- HTTP/POST backpressure (drops accumulating because the firmware or
network can't keep up) — fix is to raise frame_interval_ms.
- Source-limited rate (ffmpeg's fps filter or the camera substream
produces fewer frames than requested) — raising the interval makes
this worse, not better.
Now: drops > 25% of target prints "backpressure" advice; otherwise
delivered < 60% of target prints "source-limited" with the correct
hint that the camera/filter is the limit, not the firmware. The
600ms-interval case where we get 1.5 fps delivered with zero drops
no longer prints the (wrong) "raise interval" suggestion.
|
|
Previous threshold was "log if drops > 25% of target rate". At 500ms
interval (2 fps target) that fired at 0.5+ dropped fps — but a
displayed rate of 1.5 fps is fine, the panel still updates every
~666ms. The warning told the user to "raise frame_interval_ms" when
nothing was actually wrong.
Track sentFrames separately and only warn when delivered rate falls
under 75% of target. Drop rate is now informational context in the
log line, not the trigger.
|
|
ffmpeg's fps filter sometimes emits timestamp-clumped pairs at low
rates; the single-flight POST guard correctly drops the second one
and we logged a "raise frame_interval_ms" warning for every such
event. At 500ms interval (2 fps target) a single drop produced a
0.6 fps warning even though we were still painting the requested
rate.
Suppress the message unless drops exceed a quarter of the configured
rate. Above that threshold there's real backpressure worth knowing
about; below it's just decimation noise.
|
|
substream + dropping queued frames
Two compounding causes of the lag:
- destination:"remote-recorder" returned the high-bitrate main encoder
with the camera's own ~10s prebuffer baked in. Walk substreams
low-resolution → medium-resolution → local → remote → remote-recorder
and take the first that resolves. The low-res preview is what
Scrypted itself uses for grid views — it's near-realtime.
- ffmpeg's output queue had no drop policy, so whenever the encode +
push pipeline fell behind for a second the queue just kept growing,
and we'd play back a steadily-older view of the world. Added
-fps_mode drop so late frames hit the floor.
Plus a few input-side latency knobs (-probesize 32, -analyzeduration 0,
-avioflags direct, -fflags +discardcorrupt) so ffmpeg stops sitting
on the first chunk to probe the stream layout.
|
|
type:"button" Settings get silently dropped by Scrypted's renderer —
they were in the model but never appeared in the UI. Switch both to
type:"boolean" so they show as toggles. The toggle acts as a one-shot
trigger: flipping it on fires the wake/sleep action and the next
getSettings() call returns value:false so it's armed again.
|
|
Triage signal for the persistent "expected 800x480, got 480x800"
dimension mismatch. The error has survived two filter rewrites; logs
on the user's side still reference old script.js line numbers, which
suggests the Scrypted Script editor isn't picking up re-pasted code.
Emit the orientation + panel dims + actual `-vf` string once per
stream start. If the log shows up, we know the latest code is live
and the problem is the filter; if it doesn't, the script never
updated.
|
|
The portrait filter chain was scale=480:800,transpose=1. Output frame
dimensions came out 800x480 (correct) but the mjpeg encoder was
occasionally writing the pre-transpose 480x800 into the JPEG SOF
marker, so the firmware's dimension check rejected the frame:
expected 800x480, got 480x800
Reordering to transpose=1,scale=800:480,setsar=1 makes the post-filter
dims explicit and unambiguous — transpose first changes the frame to
800x480, scale re-asserts it, setsar clears any leftover non-1:1
aspect ratio metadata. The encoder now writes 800x480 into SOF every
frame.
|
|
Adds an "Actions" group with two buttons:
- Wake now: starts a stream for the bound camera right now, bypassing
trigger filters. Useful for verifying the panel + camera link
without having to walk in front of a motion sensor.
- Sleep now: tears down the active stream and POSTs /state {sleep}.
Drops the private modifier from streams/streamStarting/startStream/
stopStream so the child Viewport can drive them — same package, no
encapsulation lost.
|
|
onBindingChanged previously stopped the active stream and waited for
the next camera event to relaunch. Changing orientation, frame
interval, or camera meant the user saw no immediate effect — the
display sat dark until the next motion/doorbell trigger.
Now if a stream was live at change-time we relaunch it right after
registerViewport pushes the new /config. Guarded by streamStarting so
the relaunch doesn't race with a concurrent camera-event-triggered
start.
|
|
Two changes to handle a burst of motion events without piling up
startStream calls and tripping the 1s /state-POST timeout:
- HTTP_TIMEOUT_MS 1000 → 5000. The firmware's single httpd task
processes one TCP connection at a time; under heavy /frame streaming
a /state {wake} can queue behind 1–3 in-flight /frames before
landing. 1s was too tight.
- handleCameraEvent now ignores repeat triggers if a stream is already
live for that viewport (this.streams.has(name)) or already starting
(streamStarting set). Sustained motion fires MotionSensor every
~500ms; we used to launch a fresh startStream each time, racing
with the previous one's ffmpeg spawn + state POST.
|
|
wall-clock
|
|
once
|
