docs: zones, transports, pattern ota, and submodule readmes

Made-with: Cursor

README.md

@@ -1,23 +1,26 @@
 # led-controller
 
-LED controller web app for managing profiles, tabs, presets, and colour palettes, and sending commands to LED devices over the serial -> ESP-NOW bridge.
+LED controller web app for managing profiles, **zones**, presets, and colour palettes, and sending commands to LED devices. Outbound paths include:
+
+- **Serial → ESP-NOW bridge**: JSON lines over UART to an ESP32 that forwards ESP-NOW frames (configure `serial_port` and baud in `settings.json` / Settings model).
+- **Wi-Fi LED drivers**: TCP JSON lines (default port **8765** on the Pi; drivers discover the controller via **UDP 8766** broadcast).
 
 ## Run
 
 - One-time setup for port 80 without root: `sudo scripts/setup-port80.sh`
-- Start app: `pipenv run run`
+- Start app: `pipenv run run` (override listen port with the **`PORT`** environment variable)
 - Dev watcher (auto-restart on `src/` changes): `pipenv run dev`
 - Regenerate **`docs/help.pdf`** from **`docs/help.md`**: `pipenv run help-pdf` (requires **pandoc** and **chromium** on the host)
 
 ## UI modes
 
-- **Run mode**: focused control view. Select tabs/presets and apply profiles. Editing actions are hidden.
+- **Run mode**: focused control view. Select zones/presets and apply profiles. Editing actions are hidden.
-- **Edit mode**: management view. Shows Tabs, Presets, Patterns, Colour Palette, and Send Presets controls, plus per-tile preset edit/remove and drag-reorder.
+- **Edit mode**: management view. Shows **Zones**, Presets, Patterns, Colour Palette, and Send Presets controls, plus per-tile preset edit/remove and drag-reorder.
 
 ## Profiles
 
 - Applying a profile updates session scope and refreshes the active zone content.
-- In **Run mode**, Profiles supports apply-only behavior (no create/clone/delete).
+- In **Run mode**, Profiles supports apply-only behaviour (no create/clone/delete).
 - In **Edit mode**, Profiles supports create/clone/delete.
 - Creating a profile always creates a populated `default` zone (starter presets).
 - Optional **DJ zone** seeding creates:
@@ -35,3 +38,6 @@ LED controller web app for managing profiles, tabs, presets, and colour palettes
 
 - Main API reference: `docs/API.md`
 
+## Driver pattern modules
+
+Pattern **`.py`** sources live under **`led-driver/src/patterns`**. The Pi app resolves that path via `util.driver_patterns.driver_patterns_dir()`. If you deploy without that tree next to the app, set **`LED_CONTROLLER_PATTERNS_DIR`** to the directory that contains those files.

docs/API.md

@@ -2,10 +2,12 @@
 
 This document covers:
 
-1. **HTTP and WebSocket** exposed by the Raspberry Pi app (`src/main.py`) — profiles, presets, transport send, and related resources.
+1. **HTTP and WebSocket** exposed by the Raspberry Pi app (`src/main.py`) — profiles, zones, presets, transport send, pattern OTA helpers, and related resources.
-2. **LED driver JSON** — the compact message format sent over the serial→ESP-NOW bridge to devices (same logical API as ESP-NOW payloads).
+2. **LED driver JSON** — the compact **v1** message format. It is sent over the **serial → ESP-NOW bridge** to ESP32 peers and as **newline-delimited JSON over TCP** to **Wi-Fi** drivers (same logical fields).
 
-Default listen address: `0.0.0.0`. Port defaults to **80**; override with the `PORT` environment variable (see `pipenv run run`).
+Default HTTP listen address: `0.0.0.0`. Port defaults to **80**; override with the **`PORT`** environment variable (see `pipenv run run`).
+
+**Serial:** UART path and baud come from settings (defaults include `serial_port` such as `/dev/ttyS0` and `serial_baudrate`). **Wi-Fi drivers:** the Pi accepts TCP connections on **`tcp_port`** in settings (default **8765**). **UDP discovery** listens on **8766** so drivers can find the controller IP on the LAN.
 
 All JSON APIs use `Content-Type: application/json` for bodies and responses unless noted.
 
@@ -16,7 +18,7 @@ All JSON APIs use `Content-Type: application/json` for bodies and responses unle
 The main UI has two modes controlled by the mode toggle:
 
 - **Run mode**: optimized for operation (zone/preset selection and profile apply).
-- **Edit mode**: shows editing/management controls (tabs, presets, patterns, colour palette, send presets, profile management actions, **Devices** registry for LED driver names/MACs, and related tools).
+- **Edit mode**: shows editing/management controls (zones, presets, patterns, colour palette, send presets, profile management actions, **Devices** registry for LED driver names/MACs, and related tools).
 
 Profiles are available in both modes, but behavior differs:
 
@@ -50,10 +52,12 @@ Profiles are selected with **`POST /profiles/<id>/apply`**, which sets `current_
 
 Connect to **`ws://<host>:<port>/ws`**.
 
-- Send **JSON**: the object is forwarded to the transport (serial bridge → ESP-NOW) as JSON. Optional key **`to`**: 12-character hex MAC address; if present it is removed from the object and the payload is sent to that peer; otherwise the default destination is used.
+- Send **JSON**: the object is forwarded through the **serial sender** (6-byte MAC prefix + payload to the ESP-NOW bridge). Optional key **`to`**: 12-character hex MAC address; if present it is removed from the object and the payload is sent to that peer; otherwise the default destination from settings is used.
 - Send **non-JSON text**: forwarded as raw bytes with the default address.
 - On send failure, the server may reply with `{"error": "Send failed"}`.
 
+Wi-Fi devices are not targeted by `/ws` directly; use **`POST /presets/send`**, device routes, or **`POST /patterns/<name>/send`** as appropriate.
+
 ---
 
 ## HTTP API by resource
@@ -77,11 +81,11 @@ Registry in `db/device.json`: storage key **`<id>`** (string, e.g. `"1"`) maps t
 | Field | Description |
 |-------|-------------|
 | **`id`** | Same as the storage key (stable handle for URLs). |
-| **`name`** | Shown in tabs and used in `select` keys. |
+| **`name`** | Shown in the UI and used in `select` keys. |
 | **`type`** | `led` (only value today; extensible). |
 | **`transport`** | `espnow` or `wifi`. |
 | **`address`** | For **`espnow`**: optional 12-character lowercase hex MAC. For **`wifi`**: optional IP or hostname string. |
-| **`default_pattern`**, **`tabs`** | Optional, as before. |
+| **`default_pattern`**, **`zones`** | Optional. Legacy **`tabs`** may still appear in old files and is migrated away on load. |
 
 Existing records without `type` / `transport` / `id` are backfilled on load (`led`, `espnow`, and `id` = key).
 
@@ -89,7 +93,7 @@ Existing records without `type` / `transport` / `id` are backfilled on load (`le
 |--------|------|-------------|
 | GET | `/devices` | Map of device id → device object. |
 | GET | `/devices/<id>` | One device, 404 if missing. |
-| POST | `/devices` | Create. Body: **`name`** (required), **`type`** (default `led`), **`transport`** (default `espnow`), optional **`address`**, **`default_pattern`**, **`tabs`**. Returns `{ "<id>": { ... } }`, 201. |
+| POST | `/devices` | Create. Body: **`name`** (required), **`type`** (default `led`), **`transport`** (default `espnow`), optional **`address`**, **`default_pattern`**, **`zones`**. Returns `{ "<id>": { ... } }`, 201. |
 | PUT | `/devices/<id>` | Partial update. **`name`** cannot be cleared. **`id`** in the body is ignored. **`type`** / **`transport`** validated; **`address`** normalised for the resulting transport. |
 | DELETE | `/devices/<id>` | Remove device. |
 
@@ -102,7 +106,7 @@ Existing records without `type` / `transport` / `id` are backfilled on load (`le
 | GET | `/profiles/<id>` | Single profile. If `<id>` is `current`, same as `/profiles/current`. |
 | POST | `/profiles` | Create profile. Body may include `name` and other fields. Optional `seed_dj_zone` (request-only) seeds a DJ zone + presets. New profiles always get a populated `default` zone. Returns `{ "<id>": { ... } }` with status 201. |
 | POST | `/profiles/<id>/apply` | Sets session current profile to `<id>`. |
-| POST | `/profiles/<id>/clone` | Clone profile (tabs, palettes, presets). Body may include `name`. |
+| POST | `/profiles/<id>/clone` | Clone profile (zones, palettes, presets). Body may include `name`. |
 | PUT | `/profiles/current` | Update the current profile (from session). |
 | PUT | `/profiles/<id>` | Update profile by id. |
 | DELETE | `/profiles/<id>` | Delete profile. |
@@ -143,11 +147,11 @@ Stored preset records can include:
 - `colors`: resolved hex colours for editor/display.
 - `palette_refs`: optional array of palette indexes parallel to `colors`. If a slot contains an integer index, the colour is linked to the current profile palette at that index.
 
-### Tabs — `/zones`
+### Zones — `/zones`
 
 | Method | Path | Description |
 |--------|------|-------------|
-| GET | `/zones` | `tabs`, `zone_order`, `current_zone_id`, `profile_id` for the session-backed profile. |
+| GET | `/zones` | `zones` (map of zone id → zone object), `zone_order`, `current_zone_id`, `profile_id` for the session-backed profile. |
 | GET | `/zones/current` | Current zone from cookie/session. |
 | POST | `/zones` | Create zone; optional JSON `name`, `names`, `presets`; can append to current profile’s zone list. |
 | GET | `/zones/<id>` | Zone JSON. |
@@ -198,20 +202,33 @@ Stored preset records can include:
 
 ### Patterns — `/patterns`
 
+Pattern metadata lives in **`db/pattern.json`**; driver source files live under **`led-driver/src/patterns/`**. Several routes expose a **runtime map** (metadata merged with on-disk `.py` names so new files appear in menus).
+
 | Method | Path | Description |
 |--------|------|-------------|
-| GET | `/patterns/definitions` | Contents of `pattern.json` (pattern metadata for the UI). |
+| GET | `/patterns` | Runtime pattern map (object keyed by pattern id). |
-| GET | `/patterns` | All pattern records. |
+| GET | `/patterns/definitions` | Same runtime map (intended for UI “definitions” clients). |
-| GET | `/patterns/<id>` | One pattern. |
+| GET | `/patterns/ota/manifest` | JSON `{"files":[{"name":"blink.py","url":"http://<Host>/patterns/ota/file/blink.py"},...]}` for OTA pulls. Requires **`Host`** header. |
+| GET | `/patterns/ota/file/<name>` | Raw **`.py`** source for one driver pattern (`name` must be a safe filename, e.g. `rainbow.py`). |
+| POST | `/patterns/<name>/send` | Push a **manifest** JSON line to **Wi-Fi** devices so they pull one pattern file over HTTP. Body may include **`device_id`** to target one device; otherwise all Wi-Fi devices with an **`address`** are tried. **`<name>`** may be with or without `.py`. |
+| POST | `/patterns/upload` | Body JSON: **`name`**, **`code`**, optional **`overwrite`** (default true). Writes **`led-driver/src/patterns/<name>.py`**. |
+| POST | `/patterns/driver` | Body JSON: **`name`** (identifier), **`code`**, optional metadata (`min_delay`, `max_delay`, `max_colors`, `n1`…`n8`, **`overwrite`**). Creates/updates both the **`.py`** file and **`db/pattern.json`** via the Pattern model. |
+| GET | `/patterns/<id>` | One pattern record from the Pattern model (metadata only). |
 | POST | `/patterns` | Create (`name`, optional `data`). |
 | PUT | `/patterns/<id>` | Update. |
 | DELETE | `/patterns/<id>` | Delete. |
 
+**Devices — pattern OTA push**
+
+| Method | Path | Description |
+|--------|------|-------------|
+| POST | `/devices/<id>/patterns/push` | Wi-Fi only. Asks the driver at **`address`** to pull pattern files from this server. Optional body **`manifest`**: either a **URL string** pointing at a manifest JSON document, or a **manifest object** (same shape as in driver messages). If omitted, a default manifest is built from the request **`Host`** header. |
+
 ---
 
-## LED driver message format (transport / ESP-NOW)
+## LED driver message format (transport / ESP-NOW / Wi-Fi)
 
-Messages are JSON objects. The Pi **`build_message()`** helper (`src/util/espnow_message.py`) produces the same shape sent over serial and forwarded by the ESP32 bridge.
+Messages are JSON objects. The Pi **`build_message()`** helper (`src/util/espnow_message.py`) produces the same shape sent over serial and forwarded by the ESP32 bridge, and the same logical object can be sent as a **single JSON line** to a Wi-Fi driver over TCP.
 
 ### Top-level fields
 

docs/SPECIFICATION.md

@@ -350,7 +350,7 @@ Manage connected devices and create/manage device groups.
 
 #### Layout
 - **Header:** Title with "Add Device" button
-- **Tabs:** Devices and Groups tabs
+- **Zones:** Devices and Groups zones (zone buttons / zone strip)
 - **Content Area:** Zone-specific content
 
 #### Devices Zone
@@ -1774,7 +1774,7 @@ peak_mem = usqlite.mem_peak()
 - Buttons respond to clicks
 - Sliders update values
 - Modals open/close
-- Tabs switch correctly
+- Zone buttons switch correctly
 - Preset selector works
 - Preset creation form validates input
 - Preset cards display correctly

docs/help.md

@@ -1,6 +1,6 @@
 # LED controller — user guide
 
-This page describes the **main web UI** served from the Raspberry Pi app: profiles, tabs, presets, colour palettes, and sending commands to LED devices over the serial → ESP-NOW bridge.
+This page describes the **main web UI** served from the Raspberry Pi app: profiles, **zones**, presets, colour palettes, and sending commands to LED devices. Traffic may go over the **serial → ESP-NOW bridge** or **Wi-Fi** (TCP to drivers on the LAN), depending on each device’s transport.
 
 For HTTP routes and the wire format the driver expects, see **[API.md](API.md)**. For running the app locally, see the project **README**.
 
@@ -12,24 +12,24 @@ Figures below are **schematic** (layout and ideas), not pixel-perfect screenshot
 
 The header has a mode toggle (desktop and mobile menu). The **label on the button is the mode you switch to** when you press it.
 
-![Schematic: zone buttons on the left; Profiles, Tabs, Presets, Patterns, and the mode toggle on the right (example shows Edit mode with “Run mode” on the button).](images/help/header-toolbar.svg)
+![Schematic: zone buttons on the left; Profiles, Zones, Presets, Patterns, and the mode toggle on the right (example shows Edit mode with “Run mode” on the button).](images/help/header-toolbar.svg)
 
 *The active zone is highlighted. Extra management buttons appear only in Edit mode.*
 
 | Mode | Purpose |
 |------|--------|
 | **Run mode** | Day-to-day control: choose a zone, tap presets, apply profiles. Management buttons are hidden. |
-| **Edit mode** | Full setup: tabs, presets, patterns, colour palette, **Send Presets**, profile create/clone/delete, preset reordering, and per-tile **Edit** on the strip. |
+| **Edit mode** | Full setup: zones, presets, patterns, colour palette, **Send Presets**, profile create/clone/delete, preset reordering, and per-tile **Edit** on the strip. |
 
 **Profiles** is available in both modes: in Run mode you can only **apply** a profile; in Edit mode you can also **create**, **clone**, and **delete** profiles.
 
 ---
 
-## Tabs
+## Zones
 
 - **Select a zone**: click its button in the top bar. The main area shows that zone’s preset strip and controls.
 - **Edit mode — open zone settings**: **right-click** a zone button to change its name, **device IDs** (comma-separated), and which presets appear on the zone. Device identifiers are matched to each device’s **name** when the app builds `select` messages for the driver.
-- **Tabs modal** (Edit mode): create new tabs from the header **Tabs** button. New tabs need a name and device ID list (defaults to `1` if you leave a simple placeholder).
+- **Zones modal** (Edit mode): create new zones from the header **Zones** button. New zones need a name and device ID list (defaults to `1` if you leave a simple placeholder).
 - **Brightness slider** (per zone): adjusts **global** brightness sent to devices (`b` in the driver message), with a short debounce so small drags do not flood the link.
 
 ---
@@ -68,7 +68,7 @@ The **Presets** header button (Edit mode) opens a **profile-wide** list: **Add**
 
 ## Profiles
 
-- **Apply**: sets the **current profile** in your session. Tabs and presets you see are scoped to that profile.
+- **Apply**: sets the **current profile** in your session. Zones and presets you see are scoped to that profile.
 - **Edit mode — Create**: new profiles always get a populated **default** zone. Optionally tick **DJ zone** to also create a `dj` zone (device name `dj`) with starter DJ-oriented presets.
 - **Clone** / **Delete**: available in Edit mode from the profile list.
 
@@ -82,7 +82,9 @@ The **Presets** header button (Edit mode) opens a **profile-wide** list: **Add**
 
 ## Patterns
 
-The **Patterns** dialog (Edit mode) is a **read-only reference**: pattern names and typical **delay** ranges from the pattern definitions. It does not change device behaviour by itself; patterns are chosen inside the preset editor.
+The **Patterns** dialog (Edit mode) lists pattern names and typical **delay** ranges from the pattern definitions. Choosing a pattern still happens inside the preset editor.
+
+**Wi-Fi drivers** can install new pattern modules over HTTP: the REST API exposes **`/patterns/ota/*`**, **`POST /patterns/<name>/send`**, **`POST /patterns/upload`**, and **`POST /patterns/driver`** (see [API.md](API.md)). ESP-NOW devices follow the bridge/serial path you configure for preset traffic.
 
 ---
 
@@ -98,7 +100,7 @@ The **Patterns** dialog (Edit mode) is a **read-only reference**: pattern names
 
 ## Mobile layout
 
-On narrow screens, use **Menu** to reach the same actions as the desktop header (Profiles, Tabs, Presets, Help, mode toggle, etc.).
+On narrow screens, use **Menu** to reach the same actions as the desktop header (Profiles, Zones, Presets, Help, mode toggle, etc.).
 
 ![Schematic: narrow layout with Menu and the same header actions in a dropdown.](images/help/mobile-menu.svg)
 
@@ -108,5 +110,5 @@ On narrow screens, use **Menu** to reach the same actions as the desktop header
 
 ## Further reading
 
-- **[API.md](API.md)** — REST routes, session scoping, WebSocket `/ws`, and LED driver JSON (`presets`, `select`, `save`, `default`, pattern keys).
+- **[API.md](API.md)** — REST routes, session scoping, WebSocket `/ws`, and LED driver JSON (`presets`, `select`, `save`, `default`, pattern keys, pattern **manifest**).
 - **README** — `pipenv run run`, port 80 setup, and high-level behaviour.

src/util/README.md

@@ -1,6 +1,6 @@
-# ESPNow Message Builder
+# Driver message builder (`espnow_message`)
 
-This utility module provides functions to build ESPNow messages according to the LED Driver API specification.
+This utility builds **v1** JSON payloads for LED drivers (serial/ESP-NOW bridge and Wi-Fi TCP). See **`docs/API.md`** for the full wire format.
 
 ## Usage
 
@@ -69,7 +69,7 @@ presets = build_presets_dict(presets_data)
 
 ## API Specification
 
-See `docs/API.md` for the complete ESPNow API specification.
+See **`docs/API.md`** for REST routes, session scoping, and the compact preset keys on the wire.
 
 ## Key Features
 

tests/README.md

@@ -1,79 +1,47 @@
 # Tests
 
-This directory contains tests for the LED Controller project.
+Tests for the LED Controller project live under **`tests/`** (pytest + legacy scripts).
 
-## Directory Structure
+## Layout
 
-- `test_endpoints.py` - HTTP endpoint tests that mimic web browser requests (runs against 192.168.4.1)
+| Path | Role |
-- `test_ws.py` - WebSocket tests
+|------|------|
-- `test_p2p.py` - ESP-NOW P2P tests
+| `test_endpoints.py` | HTTP endpoint checks (often against a running Pi; host configurable in file) |
-- `models/` - Model unit tests
+| `test_endpoints_pytest.py` | Pytest-style endpoint coverage |
-- `web.py` - Local development web server
+| `test_browser.py` | Selenium UI flows |
+| `test_pattern_ota_send.py` | Pattern OTA / Wi-Fi send helpers |
+| `tcp_test_server.py`, `async_tcp_server.py` | TCP test doubles for driver protocol |
+| `udp_server.py` | UDP discovery / hello test listener (port **8766**) |
+| `ws.py` | WebSocket client checks |
+| `p2p.py` | ESP-NOW–related helpers / experiments |
+| `web.py` | Local dev static server (not the main app) |
+| `conftest.py` | Pytest fixtures |
+| `models/` | Model unit tests (`run_all.py`, `test_zone.py`, …) |
 
-## Running Tests
+## Running tests
 
-### Browser Tests (Real Browser Automation)
+### Pytest (recommended)
 
-Tests the web interface in an actual browser using Selenium:
+From the project root (with dev dependencies installed):
+
+```bash
+pipenv run pytest tests/ -q
+```
+
+### Browser tests (real browser)
 
 ```bash
 python tests/test_browser.py
 ```
 
-These tests:
+Requires **Selenium**, Chrome/Chromium, and a matching **ChromeDriver**.
-- Open a real Chrome browser
-- Navigate to the device at 192.168.4.1
-- Interact with UI elements (buttons, forms, modals)
-- Test complete user workflows
-- Verify visual elements and interactions
 
-**Requirements:**
+### Model tests only
-```bash
-pip install selenium
-# Also need ChromeDriver installed and in PATH
-# Download from: https://chromedriver.chromium.org/
-```
-
-### Endpoint Tests (Browser-like HTTP)
-
-Tests HTTP endpoints by making requests to the device at 192.168.4.1:
-
-```bash
-python tests/test_endpoints.py
-```
-
-These tests:
-- Mimic web browser requests with proper headers
-- Handle cookies for session management
-- Test all CRUD operations (GET, POST, PUT, DELETE)
-- Verify responses and status codes
-
-**Requirements:**
-```bash
-pip install requests
-```
-
-### WebSocket Tests
-
-```bash
-python tests/test_ws.py
-```
-
-**Requirements:**
-```bash
-pip install websockets
-```
-
-### Model Tests
 
 ```bash
 python tests/models/run_all.py
 ```
 
-### Local Development Server
+### Local static server
 
-Run the local development server (port 5000):
+`tests/web.py` serves files for quick UI experiments; it is **not** the Microdot app. For the real server use **`pipenv run run`** from the repo root.
-
-```bash
-python tests/web.py
-```