path: root/Scrypted-Viewport-v2-Claude-Code-Implementation-Guide.md

# Scrypted Viewport v2 Claude Code Implementation Guide

Version: 2.0  
Target: Claude Code / ESP-IDF firmware implementation  
Hardware: Waveshare ESP32-P4-ETH-POE + 5" 800×480 IPS Capacitive Touch MIPI DSI display  
Project name: Scrypted Viewport

---

## 1. Project Summary

Scrypted Viewport is a thin Ethernet-powered network framebuffer appliance for Scrypted.

It runs on a Waveshare ESP32-P4-ETH-POE board connected to a 5" 800×480 MIPI DSI capacitive-touch display.

The device does not render UI, calendars, camera labels, overlays, fonts, or business logic. It receives fully rendered JPEG frames from Scrypted, decodes them, displays them, and reports touch gestures back to Scrypted.

The goal is a reliable appliance, not a general-purpose smart display.

---

## 2. Non-Goals

Do not implement:

- Matter
- HomeKit
- Wi-Fi
- BLE
- polling
- authentication
- cloud connectivity
- web UI
- local calendar/weather/camera logic
- HTML rendering
- template rendering
- local camera streaming clients
- configuration portal
- multi-user support
- database
- display abstraction for other screens

If tempted to add application logic to the ESP, move that logic to Scrypted.

---

## 3. Hardware Target

### Controller

Waveshare ESP32-P4-ETH-POE

Expected capabilities:

- ESP32-P4
- Ethernet
- PoE
- PSRAM
- MIPI DSI display interface
- touch interface
- USB programming/debugging

### Display

5" 800×480 IPS capacitive-touch MIPI DSI display, Raspberry Pi-compatible Hosyond/Amazon model.

Native resolution:

```text
800x480
```

Firmware should target this exact hardware. No runtime display detection is required.

---

## 4. High-Level Runtime Behavior

Boot flow:

```text
power on
  -> initialize logging
  -> initialize NVS
  -> load saved config (display name, callback, brightness)
  -> initialize Ethernet
  -> DHCP
  -> initialize mDNS
  -> initialize display/backlight/touch
  -> start HTTP server
  -> advertise _scrypted-viewport._tcp.local
  -> if unconfigured: render IP + viewport.local on screen
  -> wait for API calls
```

Frame flow:

```text
Scrypted renders 800x480 JPEG
  -> HTTP POST /frame
  -> Scrypted Viewport receives raw JPEG
  -> decode JPEG into framebuffer/display buffer
  -> push to DSI display
  -> turn on backlight
  -> reset idle timer
```

Wake/sleep model:

Wake and sleep couple the backlight with Scrypted's frame stream. The two move together, never independently.

```text
wake  = backlight on  + Scrypted streaming frames
sleep = backlight off + Scrypted not streaming
```

Touch flow:

```text
tap while asleep:
  backlight on
  render loading screen
  POST {"event":"wake"} -> Scrypted starts streaming
  next /frame replaces loading screen

tap while awake:
  backlight off
  POST {"event":"sleep"} -> Scrypted stops streaming

idle timer fires (60s with no /frame):
  backlight off
  POST {"event":"sleep"} -> Scrypted stops streaming
```

The callback events are `wake` and `sleep`; `tap` itself is internal. The events are idempotent imperatives — "start streaming" / "stop streaming" — not state notifications. Scrypted does not track per-viewport state; it acts on the event and forgets.

Scrypted-initiated `/sleep` and `/frame` do not echo a callback.

Only `tap` is detected on the touchscreen. Long-press and swipes are out of scope for v1.

BOOT button:
- Short press: overlay IP screen for 15s, then return to prior state. Wakes backlight temporarily but does not change wake/sleep state and does not POST a callback.
- Hold ≥5s: clear NVS and reboot. Device comes back unconfigured.

---

## 5. API Contract

The firmware exposes a minimal HTTP API.

Default port:

```text
80
```

mDNS service:

```text
_scrypted-viewport._tcp.local
```

Example hostname:

```text
viewport.local
viewport-mudroom.local
```

### 5.1 GET /health

Return JSON status.

Example response:

```json
{
  "name": "mudroom",
  "version": "1.0.0",
  "configured": true,
  "uptime_ms": 12345678,
  "resolution": "800x480",
  "ip": "192.168.1.42",
  "free_heap": 123456,
  "free_psram": 12345678
}
```

Status code:

```text
200 OK
```

### 5.2 POST /config

Registers the controlling Scrypted callback.

Request body:

```json
{
  "display": "mudroom",
  "callback": "http://scrypted.local:11080/api/viewport/touch",
  "idle_timeout_ms": 60000
}
```

Behavior:

- Store display name.
- Store callback URL.
- Store idle timeout (optional in body, default 60000).
- Persist all to NVS so it survives reboot.
- May be called repeatedly.
- Replaces old config atomically.

Response:

```text
204 No Content
```

Validation:

- `display` must be non-empty.
- `callback` must be HTTP URL.
- HTTPS is not required for v1/v2.

### 5.3 POST /frame

Displays a full-frame JPEG.

Headers:

```text
Content-Type: image/jpeg
```

Body:

```text
raw JPEG bytes
```

Expected image:

```text
800x480 JPEG
```

Behavior:

- Accept raw JPEG body.
- Decode.
- Display.
- Turn on backlight.
- Reset idle timer.
- Return once frame is accepted/displayed.

Response:

```text
204 No Content
```

Failure responses:

```text
400 Bad Request        invalid content-type or malformed JPEG
413 Payload Too Large  JPEG exceeds configured maximum
500 Internal Error     decode/display failure
```

Recommended max JPEG size:

```text
1 MB
```

Implementation note:

Scrypted is responsible for scaling/cropping/composition. Scrypted Viewport should not scale images in v1/v2.

### 5.4 POST /sleep

Turns off the backlight.

Behavior:

- Backlight off.
- Preserve framebuffer.
- Device remains online and ready for `/frame`.

Response:

```text
204 No Content
```

### 5.5 POST /brightness

Sets backlight brightness.

Request:

```json
{
  "brightness": 75
}
```

Range:

```text
0-100
```

Behavior:

- Clamp or reject out-of-range values. Prefer rejecting invalid values with 400.
- Persist latest brightness to NVS.
- If display is awake, apply immediately.
- If asleep, store value for next wake.

Response:

```text
204 No Content
```

---

## 6. Touch Callback Contract

Scrypted Viewport initiates HTTP POSTs to the configured callback URL.

Example request body:

```json
{
  "display": "mudroom",
  "event": "tap",
  "timestamp": 1730000000
}
```

Supported event names:

```text
wake
sleep
```

`tap` is the input the touchscreen detects; the callback event is the resulting state (`wake` or `sleep`). Idle-timer-driven sleep also sends `sleep`.

Rules:

- The device owns wake/sleep state. Scrypted does not track it.
- Scrypted owns the viewport→camera binding and decides which stream goes to which viewport.
- `wake`/`sleep` callbacks are idempotent imperatives: "start streaming" / "stop streaming". Scrypted acts on receipt and forgets.
- Scrypted enforces its own per-stream timeout independently of the device's idle timer. Either can end a session, whichever notices first.
- Scrypted-initiated `/sleep` and `/frame` do not echo a callback.

Callback delivery:

- Best effort.
- Timeout quickly, e.g. 1 second.
- Log failures.
- Do not block display operation for long.
- No retry queue required for v1/v2.

---

## 7. Firmware Architecture

Recommended ESP-IDF project layout:

```text
scrypted-viewport/
  CMakeLists.txt
  sdkconfig.defaults
  README.md

  main/
    CMakeLists.txt
    app_main.c

    viewport_config.h
    viewport_state.h
    viewport_state.c

    net_eth.h
    net_eth.c

    mdns_service.h
    mdns_service.c

    http_api.h
    http_api.c

    display.h
    display.c

    jpeg_decoder.h
    jpeg_decoder.c

    touch.h
    touch.c

    callback_client.h
    callback_client.c

    idle_timer.h
    idle_timer.c

    local_screens.h
    local_screens.c

    nvs_config.h
    nvs_config.c
```

Keep modules boring and explicit.

---

## 8. Module Responsibilities

### 8.1 `app_main.c`

Responsibilities:

- Initialize NVS.
- Load saved config.
- Initialize Ethernet.
- Wait for IP address.
- Initialize display.
- Initialize touch.
- Start HTTP server.
- Start mDNS.
- Start idle timer.
- Start touch task.

Pseudo-flow:

```c
void app_main(void) {
    nvs_config_init();
    viewport_state_init();

    net_eth_init();
    net_eth_wait_for_ip();

    display_init();
    display_set_brightness(saved_brightness);
    display_sleep();

    touch_init();
    touch_start_task();

    http_api_start();
    mdns_service_start();

    idle_timer_start();
}
```

### 8.2 `net_eth.c`

Responsibilities:

- Initialize Ethernet driver for Waveshare ESP32-P4-ETH-POE.
- Use DHCP.
- Publish current IP to global state.

No Wi-Fi support.

### 8.3 `mdns_service.c`

Responsibilities:

- Set hostname.
- Advertise `_scrypted-viewport._tcp.local` on port 80.
- Include TXT records if easy:

```text
version=1.0.0
resolution=800x480
name=mudroom
```

### 8.4 `http_api.c`

Responsibilities:

- Register routes:
  - `GET /health`
  - `POST /config`
  - `POST /frame`
  - `POST /sleep`
  - `POST /brightness`
- Parse JSON for config/brightness.
- Stream JPEG request body into buffer.
- Call display/JPEG functions.
- Return simple status codes.

### 8.5 `display.c`

Responsibilities:

- Initialize MIPI DSI display.
- Initialize framebuffer or draw buffer.
- Control backlight.
- Push decoded frames to display.
- Sleep/wake backlight.

Do not draw application UI.

### 8.6 `jpeg_decoder.c`

Responsibilities:

- Decode JPEG into display-compatible pixel buffer.
- Prefer ESP-IDF JPEG/image decode support if available for ESP32-P4.
- Avoid repeated allocations where possible.
- Reuse a static PSRAM frame buffer.

Input:

```text
JPEG byte buffer
```

Output:

```text
800x480 framebuffer
```

### 8.7 `touch.c`

Responsibilities:

- Initialize capacitive touch controller.
- Detect `tap` only (touch-down + release within ~500ms, ignore movement). Long-press and swipes are out of scope.
- Debounce.
- On tap:
  - If asleep: transition to wake — backlight on, render loading screen, POST `{"event":"wake"}`.
  - If awake: transition to sleep — backlight off, POST `{"event":"sleep"}`.
- Also handle BOOT button:
  - Short press: ask `local_screens` to overlay the IP screen for 15s, then restore prior state. No callback.
  - Hold ≥5s: clear NVS via `nvs_config_reset()` and reboot.

### 8.8 `callback_client.c`

Responsibilities:

- POST touch JSON to configured callback URL.
- Fast timeout.
- Best effort.
- Do not retry indefinitely.

### 8.9 `idle_timer.c`

Responsibilities:

- Reset timer on `/frame` and on tap-driven wake.
- On expiry: transition to sleep (backlight off, POST `{"event":"sleep"}`).
- Idle timeout: read from NVS (`idle_timeout_ms`), default 60000 ms. Set via `/config`. Scrypted is expected to use the same value as its own per-stream cutoff, but timers run independently.

### 8.10 `local_screens.c`

The only application UI the firmware draws. Two screens, both via a small embedded bitmap font (no LVGL, no general text engine).

Responsibilities:

- `local_screens_show_ip(ip)` — centered IP address and `viewport.local`. Shown:
  - on boot when no callback URL is in NVS (persistent until `/config` arrives), and
  - as a 15s overlay on BOOT short-press (then restore prior state).
- `local_screens_show_loading()` — centered "Loading…" text. Shown on wake until the next `/frame` lands.
- Render into the same RGB565 framebuffer the JPEG decoder targets, then push to the panel.

Keep it small. Hard-code the font glyphs for digits, dots, colons, and a-z. No string formatting library — write tiny inline blitters.

### 8.11 `nvs_config.c`

Responsibilities:

Persist:

- display name
- callback URL
- brightness
- idle timeout (ms)

Do not persist frame data.

---

## 9. State Model

Global state should be small:

```c
typedef struct {
    char display_name[64];
    char callback_url[256];

    bool configured;
    bool display_awake;

    uint8_t brightness;      // 0-100
    uint32_t idle_timeout_ms;

    char ip_addr[64];

    uint64_t frames_received;
    uint64_t touch_events_sent;
    uint64_t decode_errors;
} viewport_state_t;
```

---

## 10. Memory Strategy

Use PSRAM for:

- JPEG request body buffer
- decode buffer
- framebuffer/draw buffer

Suggested constants:

```c
#define VIEWPORT_WIDTH 800
#define VIEWPORT_HEIGHT 480
#define VIEWPORT_MAX_JPEG_SIZE (1024 * 1024)
```

Target pixel format depends on display driver. Prefer RGB565 if supported:

```text
800 * 480 * 2 = 768,000 bytes
```

This fits comfortably in PSRAM.

---

## 11. Display Strategy

v1/v2 should use the simplest reliable display path.

Preferred:

```text
JPEG -> RGB565 framebuffer -> DSI display
```

Do not implement:

- double buffering unless needed
- animations
- transitions
- LVGL UI
- text rendering

If the Waveshare ESP32-P4 examples include a working DSI display demo, start from that display initialization code and strip it down.

Acceptance criterion:

- Boot shows a test color or test image.
- `/frame` updates the full screen.
- Backlight turns off/on reliably.

---

## 12. Scrypted Integration Assumptions

Scrypted owns a list of viewports, each bound to one camera device:

```ts
const viewports = [
  { url: "http://viewport-mudroom.local", display: "mudroom", camera: "frontDoor" },
  { url: "http://viewport-kitchen.local", display: "kitchen", camera: "driveway" },
];
```

The binding lives in Scrypted config. The device knows nothing about cameras.

Registration on startup:

```ts
for (const v of viewports) {
  await fetch(`${v.url}/config`, {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({
      display: v.display,
      callback: "http://scrypted.local:11080/api/viewport/touch",
      idle_timeout_ms: 60000,
    }),
  });
}
```

Stream a session of frames to a viewport — triggered by either a camera event (doorbell, person, motion) or a `wake` callback from the viewport:

```ts
await fetch(`${v.url}/frame`, {
  method: "POST",
  headers: { "Content-Type": "image/jpeg" },
  body: jpegBuffer,
});
```

Callback handler at `/api/viewport/touch`:

- `wake` → look up viewport→camera binding, start streaming.
- `sleep` → stop streaming to that viewport.

Both are idempotent. Don't track viewport state. Apply a Scrypted-side per-stream timeout using the same `idle_timeout_ms` value sent in `/config`, so streams end even if the `sleep` callback is dropped. Timers run independently on each side.

---

## 13. Development Milestones

### Milestone 1: Board Bring-Up

- Build ESP-IDF project.
- Flash ESP32-P4.
- Serial logging works.
- Ethernet DHCP works.
- Device prints IP.

Acceptance:

```text
Device gets DHCP lease over Ethernet.
```

### Milestone 2: HTTP + mDNS

- Start HTTP server.
- Implement `/health`.
- Advertise `_scrypted-viewport._tcp.local`.

Acceptance:

```bash
curl http://viewport.local/health
```

returns JSON.

### Milestone 3: Display Bring-Up

- Initialize DSI display.
- Turn backlight on/off.
- Show color bars/test pattern.

Acceptance:

```text
Screen displays deterministic test pattern.
```

### Milestone 4: JPEG Frame Push

- Implement `/frame`.
- Receive JPEG body.
- Decode into framebuffer.
- Display full frame.

Acceptance:

```bash
curl -X POST \
  -H "Content-Type: image/jpeg" \
  --data-binary @test-800x480.jpg \
  http://viewport.local/frame
```

updates screen.

### Milestone 5: Sleep/Brightness

- Implement `/sleep`.
- Implement `/brightness`.
- Add idle timer.

Acceptance:

```bash
curl -X POST http://viewport.local/sleep
```

turns off backlight.

### Milestone 6: Config Persistence

- Implement `/config`.
- Persist callback/name in NVS.
- Confirm survives reboot.

Acceptance:

```text
After reboot, /health shows configured=true and name preserved.
```

### Milestone 7: Touch Callback

- Initialize touch controller.
- Detect tap.
- POST callback JSON.

Acceptance:

```text
Tap generates callback to test HTTP endpoint.
```

### Milestone 8: Local Screens

- Embed minimal bitmap font.
- Render unconfigured screen (IP + viewport.local) on boot when NVS has no callback.
- Render loading screen on tap-wake; replaced by next `/frame`.

Acceptance:

```text
Fresh device boots to an IP-display screen.
Tap on a sleeping configured device shows "Loading…" until next frame.
```

---

## 14. Test Images

Generate test images at exactly 800×480:

- solid red
- solid green
- solid blue
- grayscale gradient
- camera-like JPEG
- text overlay rendered by host

Use these to validate color order, scaling, orientation, and full-screen update.

If colors are wrong, likely RGB/BGR or RGB565 byte-order issue.

---

## 15. Error Handling Philosophy

Keep failure behavior simple.

If frame decode fails:

- increment decode error counter
- return 400 or 500
- keep previous frame on screen

If callback fails:

- log
- increment counter
- continue

If Ethernet disconnects:

- keep display running
- reconnect automatically
- no reboot loop

If display init fails:

- log loudly
- continue serving `/health` if possible with display error status

---

## 16. Security Model

No authentication.

Assumption:

```text
Scrypted Viewport runs only on trusted private LAN/VLAN.
```

Do not add auth, pairing, TLS, users, or passwords.

If future security is needed, solve it at the network layer.

---

## 17. Coding Standards

- Prefer C with ESP-IDF APIs.
- Keep modules small.
- Avoid dynamic allocation in hot paths after startup where practical.
- Use PSRAM explicitly for large buffers.
- Log meaningful errors.
- Avoid clever abstractions.
- Avoid generalized display framework unless necessary.
- Keep all constants obvious and centralized.

---

## 18. Acceptance Criteria for v2

The project is successful when:

1. Device powers from PoE.
2. Device gets DHCP over Ethernet.
3. Device advertises mDNS.
4. `/health` works.
5. `/config` persists display name and callback URL.
6. `/frame` accepts an 800×480 JPEG and updates screen.
7. `/frame` implicitly wakes display.
8. Idle timeout turns backlight off.
9. `/sleep` works.
10. `/brightness` works.
11. Tap sends callback JSON to Scrypted/test server.
12. Unconfigured device shows its IP on screen so it can be registered from Scrypted.
13. Tap toggles the device between wake (backlight on + Scrypted streaming) and sleep (backlight off + Scrypted not streaming), via `wake`/`sleep` callback events.
14. Idle timer fires `sleep` after `idle_timeout_ms` with no `/frame` (default 60s, set by Scrypted via `/config`).
15. BOOT short-press overlays the IP screen for 15s; BOOT hold ≥5s factory-resets.
16. Firmware has no Matter, HomeKit, polling, business logic, or config portal. The only locally-rendered UI is the IP screen and the loading screen.

---

## 19. Guiding Rule

When deciding where to implement a feature:

```text
If it changes what should be shown, it belongs in Scrypted.
If it changes how pixels reach the screen, it belongs in Scrypted Viewport.
```

Scrypted Viewport receives pixels, displays pixels, and reports touch.

Everything else belongs outside the ESP.