diff --git a/README.md b/README.md index 0d08576..9ef167f 100644 --- a/README.md +++ b/README.md @@ -1,9 +1,11 @@ # led-controller -LED controller web app for managing profiles, **zones**, presets, and colour palettes, and sending commands to LED devices. Outbound paths include: +LED controller web app for managing profiles, **zones**, presets, and colour palettes, and sending commands to LED devices over **ESP-NOW** (binary wire format). -- **Serial → ESP-NOW bridge**: JSON lines over UART to an ESP32 that forwards ESP-NOW frames (configure `serial_port` and baud in `settings.json` / Settings model). -- **Wi-Fi LED drivers**: TCP JSON lines (default port **8765** on the Pi; drivers discover the controller via **UDP 8766** broadcast). +- **Bridge ESP32**: runs a WebSocket server; the Pi connects as client (`bridge_ws_url` in `settings.json`, e.g. `ws://192.168.4.1/ws`). +- **LED drivers**: announce on boot via ESP-NOW broadcast; the controller registers them and pushes group membership. +- Architecture (diagrams): [docs/espnow-architecture.md](docs/espnow-architecture.md) +- Wire format (byte layouts): [docs/espnow-binary-protocol.md](docs/espnow-binary-protocol.md) (≤250 bytes per frame, no JSON on the wire) ## Run diff --git a/docs/espnow-architecture.md b/docs/espnow-architecture.md new file mode 100644 index 0000000..cf7d62b --- /dev/null +++ b/docs/espnow-architecture.md @@ -0,0 +1,162 @@ +# ESP-NOW transport architecture + +This document describes how **led-controller**, the **bridge ESP32**, and **led-driver** devices work together. Wire-level byte layouts are in [espnow-binary-protocol.md](espnow-binary-protocol.md). + +**On the wire:** binary only (no JSON) for ESP-NOW and Pi↔bridge WebSocket. The Pi web UI and `db/*.json` still use JSON internally. + +## System overview + +![Three-node ESP-NOW architecture](images/espnow/system-overview.svg) + +| Component | Firmware / path | Role | +|-----------|-----------------|------| +| **led-controller** | Raspberry Pi, `src/` | Web app; WebSocket **client** to bridge; device registry; builds binary commands | +| **Bridge** | [`espnow-sender/`](../espnow-sender/) | WebSocket **server** `/ws`; relays binary ↔ ESP-NOW; max **20** peers (LRU) | +| **led-driver** | [`led-driver/`](../led-driver/) submodule | Boot **ANNOUNCE** broadcast; applies **GROUPS**, **CMD**, **GROUP_CMD** | + +Configure the Pi in `settings.json`: + +```json +{ + "bridge_ws_url": "ws://192.168.4.1/ws", + "wifi_channel": 6 +} +``` + +Connect the Pi to the **bridge access point** (SSID = bridge `name` in `/settings.json`, default IP **192.168.4.1**). All nodes must use the same 2.4 GHz **channel** (Pi sends `BRIDGE_CH` on connect; bridge updates AP + ESP-NOW STA). + +--- + +## Boot and registration + +![Boot and registration sequence](images/espnow/boot-sequence.svg) + +1. Driver powers on and sends **ANNOUNCE** to broadcast MAC `ff:ff:ff:ff:ff:ff`. +2. Bridge receives it and forwards a **WebSocket uplink** frame to the Pi (peer MAC + packet). +3. Pi **upserts** the device in `db/device.json` (key = 12-char hex MAC). +4. Pi scans `db/group.json` and builds a **GROUPS** packet. +5. Pi sends **GROUPS** unicast to that MAC via the bridge. +6. Driver stores group ids in RAM for **GROUP_CMD** filtering. + +If the Pi or bridge is not up yet, the driver re-sends **ANNOUNCE** periodically until **GROUPS** arrives. + +--- + +## Sending presets and commands + +![Command delivery flow](images/espnow/command-flow.svg) + +1. UI or API triggers a send (e.g. `POST /presets/send`). +2. Pi builds one or more **CMD** packets (v2 binary envelope, chunked to ≤250 bytes). +3. Each packet is wrapped in a **WebSocket downlink** frame (unicast MAC or broadcast). +4. Bridge forwards on ESP-NOW. +5. Driver parses and applies (presets, select, brightness, device_config, etc.). + +For a **group**, Pi may send **GROUP_CMD** on broadcast once per chunk; only drivers that belong to that group apply the payload. + +--- + +## Packet layers + +![Packet layer stack](images/espnow/packet-layers.svg) + +### Layer A — WebSocket frame (Pi ↔ bridge) + +| Offset | Size | Field | +|--------|------|--------| +| 0 | 1 | `flags` — bit0 = broadcast (`ff:ff:…`); peer ignored if set | +| 1 | 6 | `peer` — destination MAC (raw bytes) | +| 7 | … | Full ESP-NOW packet (layer B) | + +**Uplink** (bridge → Pi): same layout; `flags = 0`, `peer` = sender. + +**Ack** (bridge → Pi after downlink): 1 byte — `0x01` ok, `0x00` error. + +### Layer B — ESP-NOW packet (on air) + +| Offset | Size | Field | +|--------|------|--------| +| 0 | 1 | Magic `0x4C` (`'L'`) | +| 1 | 1 | Message type | +| 2 | … | Body (≤248 bytes so total ≤250) | + +![Message types](images/espnow/message-types.svg) + +| Type | Value | Direction | Purpose | +|------|-------|-------------|---------| +| ANNOUNCE | `0x01` | Driver → broadcast | Boot settings | +| GROUPS | `0x02` | Pi → driver | Group membership | +| CMD | `0x03` | Pi → driver | Command (v2 envelope) | +| GROUP_CMD | `0x04` | Pi → broadcast | Command scoped to one group | +| BRIDGE_CH | `0x10` | Pi → bridge | Set STA channel 1–11 | + +### Layer C — v2 command envelope (inside CMD / GROUP_CMD) + +Used for presets, select, default, brightness. **No JSON.** + +| Byte | Field | +|------|--------| +| 0 | Version `2` | +| 1 | Brightness wire 0–127 (→ 0–255); `128–255` = unchanged | +| 2 | `lp` — presets section length | +| 3 | `ls` — select section length | +| 4 | `ld` — default section length | +| 5… | Presets blob (`lp` bytes) | +| … | Select blob (`ls` bytes) | +| … | Default blob (`ld` bytes) | + +Optional trailing `0x01` after the envelope in **CMD** means `save` (persist to flash). + +Implementation: [`src/util/binary_envelope.py`](../src/util/binary_envelope.py), [`src/util/espnow_wire.py`](../src/util/espnow_wire.py). + +--- + +## Message body reference + +### ANNOUNCE (`0x01`) + +Sender MAC comes from ESP-NOW headers, not the body. + +``` +name_len (u8) | name (utf-8) | num_leds (u16 LE) | color_order (u8) | startup_mode (u8) | brightness (u8) | device_type (u8) +``` + +| `color_order` | `startup_mode` | +|---------------|----------------| +| 0=rgb, 1=rbg, 2=grb, 3=gbr, 4=brg, 5=bgr | 0=default, 1=last, 2=off | + +### GROUPS (`0x02`) + +``` +count (u8) | repeat: id_len (u8) | group_id (utf-8) +``` + +Group ids match keys in `db/group.json` (e.g. `"5"`, `"18"`). + +### GROUP_CMD (`0x04`) + +``` +group_id_len (u8) | group_id (utf-8) | v2 envelope | [optional 0x01 save] +``` + +Driver applies only if `group_id` is in its stored list. + +--- + +## Size limits and chunking + +- **250 bytes** max per ESP-NOW datagram. +- Large preset libraries → multiple **CMD** packets from the Pi. +- Bridge stores at most **20** peer MACs; oldest peer evicted (LRU) when full. + +--- + +## Related files + +| Topic | Location | +|-------|----------| +| Byte-level spec | [espnow-binary-protocol.md](espnow-binary-protocol.md) | +| Pi wire codec | [`src/util/espnow_wire.py`](../src/util/espnow_wire.py) | +| Pi bridge client | [`src/models/bridge_ws_client.py`](../src/models/bridge_ws_client.py) | +| Bridge firmware | [`espnow-sender/main.py`](../espnow-sender/main.py) | +| Driver ESP-NOW | [`led-driver/src/espnow_transport.py`](../led-driver/src/espnow_transport.py) | diff --git a/docs/espnow-binary-protocol.md b/docs/espnow-binary-protocol.md new file mode 100644 index 0000000..c1dd36a --- /dev/null +++ b/docs/espnow-binary-protocol.md @@ -0,0 +1,94 @@ +# ESP-NOW binary protocol + +**See also:** [espnow-architecture.md](espnow-architecture.md) (diagrams, flows, configuration). + +All ESP-NOW datagrams and Pi↔bridge WebSocket frames use **binary only** (no JSON on the wire). Maximum ESP-NOW payload length: **250 bytes**. + +## ESP-NOW packet + +| Offset | Field | +|--------|--------| +| 0 | Magic `0x4C` (`'L'`) | +| 1 | Message type | +| 2… | Type-specific body | + +### Message types + +| Value | Name | Direction | +|-------|------|-----------| +| `0x01` | `ANNOUNCE` | Driver → broadcast | +| `0x02` | `GROUPS` | Controller → driver | +| `0x03` | `CMD` | Controller → driver | +| `0x04` | `GROUP_CMD` | Controller → broadcast | +| `0x10` | `BRIDGE_CH` | Controller → broadcast | + +### ANNOUNCE (`0x01`) + +Driver settings at boot. Sender MAC is taken from the ESP-NOW peer address (not repeated in the body). + +| Field | Type | +|-------|------| +| name_len | u8 | +| name | UTF-8 | +| num_leds | u16 LE | +| color_order | u8 enum: 0=rgb, 1=rbg, 2=grb, 3=gbr, 4=brg, 5=bgr | +| startup_mode | u8: 0=default, 1=last, 2=off | +| brightness | u8 0–255 | +| device_type | u8: 0=led | + +### GROUPS (`0x02`) + +| Field | Type | +|-------|------| +| count | u8 | +| × count | u8 id_len + UTF-8 group id | + +### CMD (`0x03`) + +Bytes 2… are a **v2 binary envelope** (see `src/util/binary_envelope.py`): 5-byte header + presets/select/default blobs. Total packet ≤ 250 bytes. + +### GROUP_CMD (`0x04`) + +| Field | Type | +|-------|------| +| group_id_len | u8 | +| group_id | UTF-8 | +| cmd_envelope | v2 binary envelope | + +Drivers apply the nested envelope only if `group_id` is in their stored group list. + +### BRIDGE_CH (`0x10`) + +| Field | Type | +|-------|------| +| channel | u8 (1–11) | + +Sets the bridge ESP32 STA channel (not forwarded to LED drivers as a command). + +## Pi ↔ bridge WebSocket frame + +Binary WebSocket messages only. + +| Offset | Field | +|--------|--------| +| 0 | flags: bit0 = broadcast destination; bit1 reserved | +| 1–6 | peer MAC (6 bytes); ignored if broadcast | +| 7… | ESP-NOW packet (magic + type + body) | + +Broadcast destination uses peer `ff:ff:ff:ff:ff:ff`. + +The bridge maintains at most **20** ESP-NOW peers (LRU eviction). + +## v2 command envelope + +Native binary sections (no JSON). Header: + +| Byte | Meaning | +|------|---------| +| 0 | Version `2` | +| 1 | Brightness wire 0–127 (maps to 0–255); 128–255 = unchanged | +| 2 | Presets section length | +| 3 | Select section length | +| 4 | Default section length | + +See `binary_envelope.py` for blob layouts. diff --git a/docs/images/espnow/boot-sequence.svg b/docs/images/espnow/boot-sequence.svg new file mode 100644 index 0000000..dd49100 --- /dev/null +++ b/docs/images/espnow/boot-sequence.svg @@ -0,0 +1,57 @@ + + + + + + + + + Boot and registration sequence + + + + Driver + + + + Bridge + + + + led-controller + + + + + ESP-NOW broadcast ANNOUNCE + dest ff:ff:ff:ff:ff:ff + + + WS uplink: peer MAC + packet + + + upsert device in + db/device.json + + + WS downlink: GROUPS unicast + + + ESP-NOW unicast GROUPS + + + store group ids + in RAM + + Driver re-sends ANNOUNCE until GROUPS received if Pi/bridge late + ANNOUNCE body: name, num_leds, color_order, startup_mode, brightness + diff --git a/docs/images/espnow/command-flow.svg b/docs/images/espnow/command-flow.svg new file mode 100644 index 0000000..367d44e --- /dev/null +++ b/docs/images/espnow/command-flow.svg @@ -0,0 +1,53 @@ + + + + + + + + + Preset / command delivery + + + UI + + + + Pi + + + + Bridge + + + + Driver + + + + POST /presets/send (JSON) + + build v2 envelope + pack CMD (d250 B) + + + WS downlink + CMD + + + ESP-NOW unicast / broadcast + + parse CMD + apply presets / select + + + GROUP_CMD: one broadcast per group id  only members apply + Large libraries ’ multiple CMD chunks from Pi + Optional trailing 0x01 on CMD = save to flash + diff --git a/docs/images/espnow/message-types.svg b/docs/images/espnow/message-types.svg new file mode 100644 index 0000000..b6c3a2b --- /dev/null +++ b/docs/images/espnow/message-types.svg @@ -0,0 +1,42 @@ + + + ESP-NOW message types (byte 1 after 0x4C) + + + Value + Name + Direction + Purpose + + + 0x01 + ANNOUNCE + Driver ? broadcast + Boot settings + + + 0x02 + GROUPS + Pi ? driver + Group membership + + + 0x03 + CMD + Pi ? driver + v2 command envelope + + + 0x04 + GROUP_CMD + Pi ? broadcast + Filtered by group id + + + 0x10 + BRIDGE_CH + Pi ? bridge + Wi-Fi channel 1–11 + + Every packet: [0x4C magic][type][body…] total ? 250 bytes + diff --git a/docs/images/espnow/packet-layers.svg b/docs/images/espnow/packet-layers.svg new file mode 100644 index 0000000..86fd73f --- /dev/null +++ b/docs/images/espnow/packet-layers.svg @@ -0,0 +1,62 @@ + + + + + + Packet layers (outside ’ inside) + + + + WebSocket frame (Pi ” bridge) + + flags + + peer MAC ×6 + + ESP-NOW packet (below) + + + + ESP-NOW datagram (d250 bytes) + + 4C + + type + + body (ANNOUNCE / GROUPS / CMD / &) + + + + Inside CMD (0x03)  v2 command envelope + + 02 + + br + + lp + + ls + + ld + + presets + + select + + def + + save? + optional 0x01 after envelope + + + Pi REST/UI uses JSON · conversion to binary happens at bridge boundary + + diff --git a/docs/images/espnow/system-overview.svg b/docs/images/espnow/system-overview.svg new file mode 100644 index 0000000..5dd2fdd --- /dev/null +++ b/docs/images/espnow/system-overview.svg @@ -0,0 +1,65 @@ + + + + + + + + + ESP-NOW LED system  three nodes + + + + led-controller + Raspberry Pi + + Web UI / REST (JSON) + + db/device.json, groups + + espnow_wire + binary + + bridge_ws_client + WS client ’ bridge + + + + Bridge ESP32 + espnow-sender + + WebSocket server /ws + + ESP-NOW relay + max 20 peers (LRU) + + + + led-driver × N + ESP32 LED strips + + boot ANNOUNCE + + store GROUPS + + apply CMD / GROUP_CMD + binary only on air + + + + binary WS + + ESP-NOW + + broadcast + + ANNOUNCE + + d250 bytes per ESP-NOW frame · no JSON on wire + diff --git a/espnow-sender/README.md b/espnow-sender/README.md index bb2c0f7..f646220 100644 --- a/espnow-sender/README.md +++ b/espnow-sender/README.md @@ -1,7 +1,54 @@ -# espnow-sender +# espnow-sender (ESP-NOW bridge) -Minimal MicroPython project for receiving JSON over Microdot WebSocket. +ESP32 firmware that relays **binary** ESP-NOW packets to/from led-controller over WebSocket. -- WebSocket endpoint: `/ws` -- Entry point: `main.py` -- Message template: `msg.json` +Layout matches **led-driver** so you deploy with **led-tool** from this directory: + +``` +espnow-sender/ + src/ # uploaded to device root via --src + main.py + wifi_ap.py + util.py + espnow_wire.py + lib/ # uploaded to /lib via --lib + aioespnow.py + microdot/ +``` + +## Deploy with led-tool + +```bash +cd espnow-sender +python ../led-tool/cli.py -p /dev/ttyUSB0 --src --lib -r -f +``` + +| Flag | Effect | +|------|--------| +| `--src` | Upload `./src` → device `:/` (`main.py`, `util.py`, `espnow_wire.py`) | +| `--lib` | Upload `./lib` → device `/lib` (aioespnow, Microdot) | +| `-r` | Reset after upload | +| `-f` | Follow serial output | + +From **led-controller** root: + +```bash +python led-tool/cli.py -p /dev/ttyUSB0 --src --lib -r -f +``` + +(run with `cwd` = `espnow-sender`, or `cd espnow-sender` first) + +Optional: `--force-upload` to ignore `file_hashes.json` on the device. + +## Runtime + +- **Wi-Fi access point** (default IP **192.168.4.1**): connect the Pi to the bridge SSID (`name` in `/settings.json`, e.g. `bridge-aabbccddeeff`) +- WebSocket server: `/ws` on port **80** — set Pi `bridge_ws_url` to `ws://192.168.4.1/ws` (or the printed IP) +- Optional `ap_password` in `/settings.json` (empty = open network) +- Default Wi-Fi channel: **6** (Pi sends `BRIDGE_CH` on connect; updates AP + ESP-NOW STA) +- Max **20** ESP-NOW peers (LRU eviction) + +## Protocol + +- [docs/espnow-architecture.md](../docs/espnow-architecture.md) +- [docs/espnow-binary-protocol.md](../docs/espnow-binary-protocol.md) diff --git a/espnow-sender/lib/aioespnow.py b/espnow-sender/lib/aioespnow.py new file mode 100644 index 0000000..5d78075 --- /dev/null +++ b/espnow-sender/lib/aioespnow.py @@ -0,0 +1,28 @@ +# aioespnow module for MicroPython on ESP32 and ESP8266 +# MIT license; Copyright (c) 2022 Glenn Moloney @glenn20 +# Vendored from micropython-lib/micropython/aioespnow + +import asyncio +import espnow + + +class AIOESPNow(espnow.ESPNow): + async def arecv(self): + yield asyncio.core._io_queue.queue_read(self) + return self.recv(0) + + async def airecv(self): + yield asyncio.core._io_queue.queue_read(self) + return self.irecv(0) + + async def asend(self, mac, msg=None, sync=None): + if msg is None: + msg, mac = mac, None + yield asyncio.core._io_queue.queue_write(self) + return self.send(mac, msg, sync) + + def __aiter__(self): + return self + + async def __anext__(self): + return await self.airecv() diff --git a/espnow-sender/lib/microdot/__init__.py b/espnow-sender/lib/microdot/__init__.py new file mode 100644 index 0000000..68cb381 --- /dev/null +++ b/espnow-sender/lib/microdot/__init__.py @@ -0,0 +1,2 @@ +from microdot.microdot import Microdot, Request, Response, abort, redirect, \ + send_file # noqa: F401 \ No newline at end of file diff --git a/espnow-sender/lib/microdot/helpers.py b/espnow-sender/lib/microdot/helpers.py new file mode 100644 index 0000000..664e58c --- /dev/null +++ b/espnow-sender/lib/microdot/helpers.py @@ -0,0 +1,8 @@ +try: + from functools import wraps +except ImportError: # pragma: no cover + # MicroPython does not currently implement functools.wraps + def wraps(wrapped): + def _(wrapper): + return wrapper + return _ diff --git a/espnow-sender/lib/microdot/microdot.py b/espnow-sender/lib/microdot/microdot.py new file mode 100644 index 0000000..0513f21 --- /dev/null +++ b/espnow-sender/lib/microdot/microdot.py @@ -0,0 +1,1450 @@ +""" +microdot +-------- + +The ``microdot`` module defines a few classes that help implement HTTP-based +servers for MicroPython and standard Python. +""" +import asyncio +import io +import json +import time + +try: + from inspect import iscoroutinefunction, iscoroutine + from functools import partial + + async def invoke_handler(handler, *args, **kwargs): + """Invoke a handler and return the result. + + This method runs sync handlers in a thread pool executor. + """ + if iscoroutinefunction(handler): + ret = await handler(*args, **kwargs) + else: + ret = await asyncio.get_running_loop().run_in_executor( + None, partial(handler, *args, **kwargs)) + return ret +except ImportError: # pragma: no cover + def iscoroutine(coro): + return hasattr(coro, 'send') and hasattr(coro, 'throw') + + async def invoke_handler(handler, *args, **kwargs): + """Invoke a handler and return the result. + + This method runs sync handlers in the asyncio thread, which can + potentially cause blocking and performance issues. + """ + ret = handler(*args, **kwargs) + if iscoroutine(ret): + ret = await ret + return ret + +try: + from sys import print_exception +except ImportError: # pragma: no cover + import traceback + + def print_exception(exc): + traceback.print_exc() + +MUTED_SOCKET_ERRORS = [ + 32, # Broken pipe + 54, # Connection reset by peer + 104, # Connection reset by peer + 128, # Operation on closed socket +] + + +def urldecode_str(s): + s = s.replace('+', ' ') + parts = s.split('%') + if len(parts) == 1: + return s + result = [parts[0]] + for item in parts[1:]: + if item == '': + result.append('%') + else: + code = item[:2] + result.append(chr(int(code, 16))) + result.append(item[2:]) + return ''.join(result) + + +def urldecode_bytes(s): + s = s.replace(b'+', b' ') + parts = s.split(b'%') + if len(parts) == 1: + return s.decode() + result = [parts[0]] + for item in parts[1:]: + if item == b'': + result.append(b'%') + else: + code = item[:2] + result.append(bytes([int(code, 16)])) + result.append(item[2:]) + return b''.join(result).decode() + + +def urlencode(s): + return s.replace('+', '%2B').replace(' ', '+').replace( + '%', '%25').replace('?', '%3F').replace('#', '%23').replace( + '&', '%26').replace('=', '%3D') + + +class NoCaseDict(dict): + """A subclass of dictionary that holds case-insensitive keys. + + :param initial_dict: an initial dictionary of key/value pairs to + initialize this object with. + + Example:: + + >>> d = NoCaseDict() + >>> d['Content-Type'] = 'text/html' + >>> print(d['Content-Type']) + text/html + >>> print(d['content-type']) + text/html + >>> print(d['CONTENT-TYPE']) + text/html + >>> del d['cOnTeNt-TyPe'] + >>> print(d) + {} + """ + def __init__(self, initial_dict=None): + super().__init__(initial_dict or {}) + self.keymap = {k.lower(): k for k in self.keys() if k.lower() != k} + + def __setitem__(self, key, value): + kl = key.lower() + key = self.keymap.get(kl, key) + if kl != key: + self.keymap[kl] = key + super().__setitem__(key, value) + + def __getitem__(self, key): + kl = key.lower() + return super().__getitem__(self.keymap.get(kl, kl)) + + def __delitem__(self, key): + kl = key.lower() + super().__delitem__(self.keymap.get(kl, kl)) + + def __contains__(self, key): + kl = key.lower() + return self.keymap.get(kl, kl) in self.keys() + + def get(self, key, default=None): + kl = key.lower() + return super().get(self.keymap.get(kl, kl), default) + + def update(self, other_dict): + for key, value in other_dict.items(): + self[key] = value + + +def mro(cls): # pragma: no cover + """Return the method resolution order of a class. + + This is a helper function that returns the method resolution order of a + class. It is used by Microdot to find the best error handler to invoke for + the raised exception. + + In CPython, this function returns the ``__mro__`` attribute of the class. + In MicroPython, this function implements a recursive depth-first scanning + of the class hierarchy. + """ + if hasattr(cls, 'mro'): + return cls.__mro__ + + def _mro(cls): + m = [cls] + for base in cls.__bases__: + m += _mro(base) + return m + + mro_list = _mro(cls) + + # If a class appears multiple times (due to multiple inheritance) remove + # all but the last occurence. This matches the method resolution order + # of MicroPython, but not CPython. + mro_pruned = [] + for i in range(len(mro_list)): + base = mro_list.pop(0) + if base not in mro_list: + mro_pruned.append(base) + return mro_pruned + + +class MultiDict(dict): + """A subclass of dictionary that can hold multiple values for the same + key. It is used to hold key/value pairs decoded from query strings and + form submissions. + + :param initial_dict: an initial dictionary of key/value pairs to + initialize this object with. + + Example:: + + >>> d = MultiDict() + >>> d['sort'] = 'name' + >>> d['sort'] = 'email' + >>> print(d['sort']) + 'name' + >>> print(d.getlist('sort')) + ['name', 'email'] + """ + def __init__(self, initial_dict=None): + super().__init__() + if initial_dict: + for key, value in initial_dict.items(): + self[key] = value + + def __setitem__(self, key, value): + if key not in self: + super().__setitem__(key, []) + super().__getitem__(key).append(value) + + def __getitem__(self, key): + return super().__getitem__(key)[0] + + def get(self, key, default=None, type=None): + """Return the value for a given key. + + :param key: The key to retrieve. + :param default: A default value to use if the key does not exist. + :param type: A type conversion callable to apply to the value. + + If the multidict contains more than one value for the requested key, + this method returns the first value only. + + Example:: + + >>> d = MultiDict() + >>> d['age'] = '42' + >>> d.get('age') + '42' + >>> d.get('age', type=int) + 42 + >>> d.get('name', default='noname') + 'noname' + """ + if key not in self: + return default + value = self[key] + if type is not None: + value = type(value) + return value + + def getlist(self, key, type=None): + """Return all the values for a given key. + + :param key: The key to retrieve. + :param type: A type conversion callable to apply to the values. + + If the requested key does not exist in the dictionary, this method + returns an empty list. + + Example:: + + >>> d = MultiDict() + >>> d.getlist('items') + [] + >>> d['items'] = '3' + >>> d.getlist('items') + ['3'] + >>> d['items'] = '56' + >>> d.getlist('items') + ['3', '56'] + >>> d.getlist('items', type=int) + [3, 56] + """ + if key not in self: + return [] + values = super().__getitem__(key) + if type is not None: + values = [type(value) for value in values] + return values + + +class AsyncBytesIO: + """An async wrapper for BytesIO.""" + def __init__(self, data): + self.stream = io.BytesIO(data) + + async def read(self, n=-1): + return self.stream.read(n) + + async def readline(self): # pragma: no cover + return self.stream.readline() + + async def readexactly(self, n): # pragma: no cover + return self.stream.read(n) + + async def readuntil(self, separator=b'\n'): # pragma: no cover + return self.stream.readuntil(separator=separator) + + async def awrite(self, data): # pragma: no cover + return self.stream.write(data) + + async def aclose(self): # pragma: no cover + pass + + +class Request: + """An HTTP request.""" + #: Specify the maximum payload size that is accepted. Requests with larger + #: payloads will be rejected with a 413 status code. Applications can + #: change this maximum as necessary. + #: + #: Example:: + #: + #: Request.max_content_length = 1 * 1024 * 1024 # 1MB requests allowed + max_content_length = 16 * 1024 + + #: Specify the maximum payload size that can be stored in ``body``. + #: Requests with payloads that are larger than this size and up to + #: ``max_content_length`` bytes will be accepted, but the application will + #: only be able to access the body of the request by reading from + #: ``stream``. Set to 0 if you always access the body as a stream. + #: + #: Example:: + #: + #: Request.max_body_length = 4 * 1024 # up to 4KB bodies read + max_body_length = 16 * 1024 + + #: Specify the maximum length allowed for a line in the request. Requests + #: with longer lines will not be correctly interpreted. Applications can + #: change this maximum as necessary. + #: + #: Example:: + #: + #: Request.max_readline = 16 * 1024 # 16KB lines allowed + max_readline = 2 * 1024 + + class G: + pass + + def __init__(self, app, client_addr, method, url, http_version, headers, + body=None, stream=None, sock=None): + #: The application instance to which this request belongs. + self.app = app + #: The address of the client, as a tuple (host, port). + self.client_addr = client_addr + #: The HTTP method of the request. + self.method = method + #: The request URL, including the path and query string. + self.url = url + #: The path portion of the URL. + self.path = url + #: The query string portion of the URL. + self.query_string = None + #: The parsed query string, as a + #: :class:`MultiDict ` object. + self.args = {} + #: A dictionary with the headers included in the request. + self.headers = headers + #: A dictionary with the cookies included in the request. + self.cookies = {} + #: The parsed ``Content-Length`` header. + self.content_length = 0 + #: The parsed ``Content-Type`` header. + self.content_type = None + #: A general purpose container for applications to store data during + #: the life of the request. + self.g = Request.G() + + self.http_version = http_version + if '?' in self.path: + self.path, self.query_string = self.path.split('?', 1) + self.args = self._parse_urlencoded(self.query_string) + + if 'Content-Length' in self.headers: + self.content_length = int(self.headers['Content-Length']) + if 'Content-Type' in self.headers: + self.content_type = self.headers['Content-Type'] + if 'Cookie' in self.headers: + for cookie in self.headers['Cookie'].split(';'): + name, value = cookie.strip().split('=', 1) + self.cookies[name] = value + + self._body = body + self.body_used = False + self._stream = stream + self.sock = sock + self._json = None + self._form = None + self.after_request_handlers = [] + + @staticmethod + async def create(app, client_reader, client_writer, client_addr): + """Create a request object. + + :param app: The Microdot application instance. + :param client_reader: An input stream from where the request data can + be read. + :param client_writer: An output stream where the response data can be + written. + :param client_addr: The address of the client, as a tuple. + + This method is a coroutine. It returns a newly created ``Request`` + object. + """ + # request line + line = (await Request._safe_readline(client_reader)).strip().decode() + if not line: # pragma: no cover + return None + method, url, http_version = line.split() + http_version = http_version.split('/', 1)[1] + + # headers + headers = NoCaseDict() + content_length = 0 + while True: + line = (await Request._safe_readline( + client_reader)).strip().decode() + if line == '': + break + header, value = line.split(':', 1) + value = value.strip() + headers[header] = value + if header.lower() == 'content-length': + content_length = int(value) + + # body + body = b'' + if content_length and content_length <= Request.max_body_length: + body = await client_reader.readexactly(content_length) + stream = None + else: + body = b'' + stream = client_reader + + return Request(app, client_addr, method, url, http_version, headers, + body=body, stream=stream, + sock=(client_reader, client_writer)) + + def _parse_urlencoded(self, urlencoded): + data = MultiDict() + if len(urlencoded) > 0: # pragma: no branch + if isinstance(urlencoded, str): + for kv in [pair.split('=', 1) + for pair in urlencoded.split('&') if pair]: + data[urldecode_str(kv[0])] = urldecode_str(kv[1]) \ + if len(kv) > 1 else '' + elif isinstance(urlencoded, bytes): # pragma: no branch + for kv in [pair.split(b'=', 1) + for pair in urlencoded.split(b'&') if pair]: + data[urldecode_bytes(kv[0])] = urldecode_bytes(kv[1]) \ + if len(kv) > 1 else b'' + return data + + @property + def body(self): + """The body of the request, as bytes.""" + return self._body + + @property + def stream(self): + """The body of the request, as a bytes stream.""" + if self._stream is None: + self._stream = AsyncBytesIO(self._body) + return self._stream + + @property + def json(self): + """The parsed JSON body, or ``None`` if the request does not have a + JSON body.""" + if self._json is None: + if self.content_type is None: + return None + mime_type = self.content_type.split(';')[0] + if mime_type != 'application/json': + return None + self._json = json.loads(self.body.decode()) + return self._json + + @property + def form(self): + """The parsed form submission body, as a + :class:`MultiDict ` object, or ``None`` if the + request does not have a form submission.""" + if self._form is None: + if self.content_type is None: + return None + mime_type = self.content_type.split(';')[0] + if mime_type != 'application/x-www-form-urlencoded': + return None + self._form = self._parse_urlencoded(self.body) + return self._form + + def after_request(self, f): + """Register a request-specific function to run after the request is + handled. Request-specific after request handlers run at the very end, + after the application's own after request handlers. The function must + take two arguments, the request and response objects. The return value + of the function must be the updated response object. + + Example:: + + @app.route('/') + def index(request): + # register a request-specific after request handler + @req.after_request + def func(request, response): + # ... + return response + + return 'Hello, World!' + + Note that the function is not called if the request handler raises an + exception and an error response is returned instead. + """ + self.after_request_handlers.append(f) + return f + + @staticmethod + async def _safe_readline(stream): + line = (await stream.readline()) + if len(line) > Request.max_readline: + raise ValueError('line too long') + return line + + +class Response: + """An HTTP response class. + + :param body: The body of the response. If a dictionary or list is given, + a JSON formatter is used to generate the body. If a file-like + object or an async generator is given, a streaming response is + used. If a string is given, it is encoded from UTF-8. Else, + the body should be a byte sequence. + :param status_code: The numeric HTTP status code of the response. The + default is 200. + :param headers: A dictionary of headers to include in the response. + :param reason: A custom reason phrase to add after the status code. The + default is "OK" for responses with a 200 status code and + "N/A" for any other status codes. + """ + types_map = { + 'css': 'text/css', + 'gif': 'image/gif', + 'html': 'text/html', + 'jpg': 'image/jpeg', + 'js': 'application/javascript', + 'json': 'application/json', + 'png': 'image/png', + 'txt': 'text/plain', + } + + send_file_buffer_size = 1024 + + #: The content type to use for responses that do not explicitly define a + #: ``Content-Type`` header. + default_content_type = 'text/plain' + + #: The default cache control max age used by :meth:`send_file`. A value + #: of ``None`` means that no ``Cache-Control`` header is added. + default_send_file_max_age = None + + #: Special response used to signal that a response does not need to be + #: written to the client. Used to exit WebSocket connections cleanly. + already_handled = None + + def __init__(self, body='', status_code=200, headers=None, reason=None): + if body is None and status_code == 200: + body = '' + status_code = 204 + self.status_code = status_code + self.headers = NoCaseDict(headers or {}) + self.reason = reason + if isinstance(body, (dict, list)): + self.body = json.dumps(body).encode() + self.headers['Content-Type'] = 'application/json; charset=UTF-8' + elif isinstance(body, str): + self.body = body.encode() + else: + # this applies to bytes, file-like objects or generators + self.body = body + self.is_head = False + + def set_cookie(self, cookie, value, path=None, domain=None, expires=None, + max_age=None, secure=False, http_only=False, + partitioned=False): + """Add a cookie to the response. + + :param cookie: The cookie's name. + :param value: The cookie's value. + :param path: The cookie's path. + :param domain: The cookie's domain. + :param expires: The cookie expiration time, as a ``datetime`` object + or a correctly formatted string. + :param max_age: The cookie's ``Max-Age`` value. + :param secure: The cookie's ``secure`` flag. + :param http_only: The cookie's ``HttpOnly`` flag. + :param partitioned: Whether the cookie is partitioned. + """ + http_cookie = '{cookie}={value}'.format(cookie=cookie, value=value) + if path: + http_cookie += '; Path=' + path + if domain: + http_cookie += '; Domain=' + domain + if expires: + if isinstance(expires, str): + http_cookie += '; Expires=' + expires + else: # pragma: no cover + http_cookie += '; Expires=' + time.strftime( + '%a, %d %b %Y %H:%M:%S GMT', expires.timetuple()) + if max_age is not None: + http_cookie += '; Max-Age=' + str(max_age) + if secure: + http_cookie += '; Secure' + if http_only: + http_cookie += '; HttpOnly' + if partitioned: + http_cookie += '; Partitioned' + if 'Set-Cookie' in self.headers: + self.headers['Set-Cookie'].append(http_cookie) + else: + self.headers['Set-Cookie'] = [http_cookie] + + def delete_cookie(self, cookie, **kwargs): + """Delete a cookie. + + :param cookie: The cookie's name. + :param kwargs: Any cookie opens and flags supported by + ``set_cookie()`` except ``expires`` and ``max_age``. + """ + self.set_cookie(cookie, '', expires='Thu, 01 Jan 1970 00:00:01 GMT', + max_age=0, **kwargs) + + def complete(self): + if isinstance(self.body, bytes) and \ + 'Content-Length' not in self.headers: + self.headers['Content-Length'] = str(len(self.body)) + if 'Content-Type' not in self.headers: + self.headers['Content-Type'] = self.default_content_type + if 'charset=' not in self.headers['Content-Type']: + self.headers['Content-Type'] += '; charset=UTF-8' + + async def write(self, stream): + self.complete() + + try: + # status code + reason = self.reason if self.reason is not None else \ + ('OK' if self.status_code == 200 else 'N/A') + await stream.awrite('HTTP/1.0 {status_code} {reason}\r\n'.format( + status_code=self.status_code, reason=reason).encode()) + + # headers + for header, value in self.headers.items(): + values = value if isinstance(value, list) else [value] + for value in values: + await stream.awrite('{header}: {value}\r\n'.format( + header=header, value=value).encode()) + await stream.awrite(b'\r\n') + + # body + if not self.is_head: + iter = self.body_iter() + async for body in iter: + if isinstance(body, str): # pragma: no cover + body = body.encode() + try: + await stream.awrite(body) + except OSError as exc: # pragma: no cover + if exc.errno in MUTED_SOCKET_ERRORS or \ + exc.args[0] == 'Connection lost': + if hasattr(iter, 'aclose'): + await iter.aclose() + raise + if hasattr(iter, 'aclose'): # pragma: no branch + await iter.aclose() + + except OSError as exc: # pragma: no cover + if exc.errno in MUTED_SOCKET_ERRORS or \ + exc.args[0] == 'Connection lost': + pass + else: + raise + + def body_iter(self): + if hasattr(self.body, '__anext__'): + # response body is an async generator + return self.body + + response = self + + class iter: + ITER_UNKNOWN = 0 + ITER_SYNC_GEN = 1 + ITER_FILE_OBJ = 2 + ITER_NO_BODY = -1 + + def __aiter__(self): + if response.body: + self.i = self.ITER_UNKNOWN # need to determine type + else: + self.i = self.ITER_NO_BODY + return self + + async def __anext__(self): + if self.i == self.ITER_NO_BODY: + await self.aclose() + raise StopAsyncIteration + if self.i == self.ITER_UNKNOWN: + if hasattr(response.body, 'read'): + self.i = self.ITER_FILE_OBJ + elif hasattr(response.body, '__next__'): + self.i = self.ITER_SYNC_GEN + return next(response.body) + else: + self.i = self.ITER_NO_BODY + return response.body + elif self.i == self.ITER_SYNC_GEN: + try: + return next(response.body) + except StopIteration: + await self.aclose() + raise StopAsyncIteration + buf = response.body.read(response.send_file_buffer_size) + if iscoroutine(buf): # pragma: no cover + buf = await buf + if len(buf) < response.send_file_buffer_size: + self.i = self.ITER_NO_BODY + return buf + + async def aclose(self): + if hasattr(response.body, 'close'): + result = response.body.close() + if iscoroutine(result): # pragma: no cover + await result + + return iter() + + @classmethod + def redirect(cls, location, status_code=302): + """Return a redirect response. + + :param location: The URL to redirect to. + :param status_code: The 3xx status code to use for the redirect. The + default is 302. + """ + if '\x0d' in location or '\x0a' in location: + raise ValueError('invalid redirect URL') + return cls(status_code=status_code, headers={'Location': location}) + + @classmethod + def send_file(cls, filename, status_code=200, content_type=None, + stream=None, max_age=None, compressed=False, + file_extension=''): + """Send file contents in a response. + + :param filename: The filename of the file. + :param status_code: The 3xx status code to use for the redirect. The + default is 302. + :param content_type: The ``Content-Type`` header to use in the + response. If omitted, it is generated + automatically from the file extension of the + ``filename`` parameter. + :param stream: A file-like object to read the file contents from. If + a stream is given, the ``filename`` parameter is only + used when generating the ``Content-Type`` header. + :param max_age: The ``Cache-Control`` header's ``max-age`` value in + seconds. If omitted, the value of the + :attr:`Response.default_send_file_max_age` attribute is + used. + :param compressed: Whether the file is compressed. If ``True``, the + ``Content-Encoding`` header is set to ``gzip``. A + string with the header value can also be passed. + Note that when using this option the file must have + been compressed beforehand. This option only sets + the header. + :param file_extension: A file extension to append to the ``filename`` + parameter when opening the file, including the + dot. The extension given here is not considered + when generating the ``Content-Type`` header. + + Security note: The filename is assumed to be trusted. Never pass + filenames provided by the user without validating and sanitizing them + first. + """ + if content_type is None: + if compressed and filename.endswith('.gz'): + ext = filename[:-3].split('.')[-1] + else: + ext = filename.split('.')[-1] + if ext in Response.types_map: + content_type = Response.types_map[ext] + else: + content_type = 'application/octet-stream' + headers = {'Content-Type': content_type} + + if max_age is None: + max_age = cls.default_send_file_max_age + if max_age is not None: + headers['Cache-Control'] = 'max-age={}'.format(max_age) + + if compressed: + headers['Content-Encoding'] = compressed \ + if isinstance(compressed, str) else 'gzip' + + f = stream or open(filename + file_extension, 'rb') + return cls(body=f, status_code=status_code, headers=headers) + + +class URLPattern(): + def __init__(self, url_pattern): + self.url_pattern = url_pattern + self.segments = [] + self.regex = None + pattern = '' + use_regex = False + for segment in url_pattern.lstrip('/').split('/'): + if segment and segment[0] == '<': + if segment[-1] != '>': + raise ValueError('invalid URL pattern') + segment = segment[1:-1] + if ':' in segment: + type_, name = segment.rsplit(':', 1) + else: + type_ = 'string' + name = segment + parser = None + if type_ == 'string': + parser = self._string_segment + pattern += '/([^/]+)' + elif type_ == 'int': + parser = self._int_segment + pattern += '/(-?\\d+)' + elif type_ == 'path': + use_regex = True + pattern += '/(.+)' + elif type_.startswith('re:'): + use_regex = True + pattern += '/({pattern})'.format(pattern=type_[3:]) + else: + raise ValueError('invalid URL segment type') + self.segments.append({'parser': parser, 'name': name, + 'type': type_}) + else: + pattern += '/' + segment + self.segments.append({'parser': self._static_segment(segment)}) + if use_regex: + import re + self.regex = re.compile('^' + pattern + '$') + + def match(self, path): + args = {} + if self.regex: + g = self.regex.match(path) + if not g: + return + i = 1 + for segment in self.segments: + if 'name' not in segment: + continue + value = g.group(i) + if segment['type'] == 'int': + value = int(value) + args[segment['name']] = value + i += 1 + else: + if len(path) == 0 or path[0] != '/': + return + path = path[1:] + args = {} + for segment in self.segments: + if path is None: + return + arg, path = segment['parser'](path) + if arg is None: + return + if 'name' in segment: + args[segment['name']] = arg + if path is not None: + return + return args + + def _static_segment(self, segment): + def _static(value): + s = value.split('/', 1) + if s[0] == segment: + return '', s[1] if len(s) > 1 else None + return None, None + return _static + + def _string_segment(self, value): + s = value.split('/', 1) + if len(s[0]) == 0: + return None, None + return s[0], s[1] if len(s) > 1 else None + + def _int_segment(self, value): + s = value.split('/', 1) + try: + return int(s[0]), s[1] if len(s) > 1 else None + except ValueError: + return None, None + + +class HTTPException(Exception): + def __init__(self, status_code, reason=None): + self.status_code = status_code + self.reason = reason or str(status_code) + ' error' + + def __repr__(self): # pragma: no cover + return 'HTTPException: {}'.format(self.status_code) + + +class Microdot: + """An HTTP application class. + + This class implements an HTTP application instance and is heavily + influenced by the ``Flask`` class of the Flask framework. It is typically + declared near the start of the main application script. + + Example:: + + from microdot import Microdot + + app = Microdot() + """ + + def __init__(self): + self.url_map = [] + self.before_request_handlers = [] + self.after_request_handlers = [] + self.after_error_request_handlers = [] + self.error_handlers = {} + self.shutdown_requested = False + self.options_handler = self.default_options_handler + self.debug = False + self.server = None + + def route(self, url_pattern, methods=None): + """Decorator that is used to register a function as a request handler + for a given URL. + + :param url_pattern: The URL pattern that will be compared against + incoming requests. + :param methods: The list of HTTP methods to be handled by the + decorated function. If omitted, only ``GET`` requests + are handled. + + The URL pattern can be a static path (for example, ``/users`` or + ``/api/invoices/search``) or a path with dynamic components enclosed + in ``<`` and ``>`` (for example, ``/users/`` or + ``/invoices//products``). Dynamic path components can also + include a type prefix, separated from the name with a colon (for + example, ``/users/``). The type can be ``string`` (the + default), ``int``, ``path`` or ``re:[regular-expression]``. + + The first argument of the decorated function must be + the request object. Any path arguments that are specified in the URL + pattern are passed as keyword arguments. The return value of the + function must be a :class:`Response` instance, or the arguments to + be passed to this class. + + Example:: + + @app.route('/') + def index(request): + return 'Hello, world!' + """ + def decorated(f): + self.url_map.append( + ([m.upper() for m in (methods or ['GET'])], + URLPattern(url_pattern), f)) + return f + return decorated + + def get(self, url_pattern): + """Decorator that is used to register a function as a ``GET`` request + handler for a given URL. + + :param url_pattern: The URL pattern that will be compared against + incoming requests. + + This decorator can be used as an alias to the ``route`` decorator with + ``methods=['GET']``. + + Example:: + + @app.get('/users/') + def get_user(request, id): + # ... + """ + return self.route(url_pattern, methods=['GET']) + + def post(self, url_pattern): + """Decorator that is used to register a function as a ``POST`` request + handler for a given URL. + + :param url_pattern: The URL pattern that will be compared against + incoming requests. + + This decorator can be used as an alias to the``route`` decorator with + ``methods=['POST']``. + + Example:: + + @app.post('/users') + def create_user(request): + # ... + """ + return self.route(url_pattern, methods=['POST']) + + def put(self, url_pattern): + """Decorator that is used to register a function as a ``PUT`` request + handler for a given URL. + + :param url_pattern: The URL pattern that will be compared against + incoming requests. + + This decorator can be used as an alias to the ``route`` decorator with + ``methods=['PUT']``. + + Example:: + + @app.put('/users/') + def edit_user(request, id): + # ... + """ + return self.route(url_pattern, methods=['PUT']) + + def patch(self, url_pattern): + """Decorator that is used to register a function as a ``PATCH`` request + handler for a given URL. + + :param url_pattern: The URL pattern that will be compared against + incoming requests. + + This decorator can be used as an alias to the ``route`` decorator with + ``methods=['PATCH']``. + + Example:: + + @app.patch('/users/') + def edit_user(request, id): + # ... + """ + return self.route(url_pattern, methods=['PATCH']) + + def delete(self, url_pattern): + """Decorator that is used to register a function as a ``DELETE`` + request handler for a given URL. + + :param url_pattern: The URL pattern that will be compared against + incoming requests. + + This decorator can be used as an alias to the ``route`` decorator with + ``methods=['DELETE']``. + + Example:: + + @app.delete('/users/') + def delete_user(request, id): + # ... + """ + return self.route(url_pattern, methods=['DELETE']) + + def before_request(self, f): + """Decorator to register a function to run before each request is + handled. The decorated function must take a single argument, the + request object. + + Example:: + + @app.before_request + def func(request): + # ... + """ + self.before_request_handlers.append(f) + return f + + def after_request(self, f): + """Decorator to register a function to run after each request is + handled. The decorated function must take two arguments, the request + and response objects. The return value of the function must be an + updated response object. + + Example:: + + @app.after_request + def func(request, response): + # ... + return response + """ + self.after_request_handlers.append(f) + return f + + def after_error_request(self, f): + """Decorator to register a function to run after an error response is + generated. The decorated function must take two arguments, the request + and response objects. The return value of the function must be an + updated response object. The handler is invoked for error responses + generated by Microdot, as well as those returned by application-defined + error handlers. + + Example:: + + @app.after_error_request + def func(request, response): + # ... + return response + """ + self.after_error_request_handlers.append(f) + return f + + def errorhandler(self, status_code_or_exception_class): + """Decorator to register a function as an error handler. Error handler + functions for numeric HTTP status codes must accept a single argument, + the request object. Error handler functions for Python exceptions + must accept two arguments, the request object and the exception + object. + + :param status_code_or_exception_class: The numeric HTTP status code or + Python exception class to + handle. + + Examples:: + + @app.errorhandler(404) + def not_found(request): + return 'Not found' + + @app.errorhandler(RuntimeError) + def runtime_error(request, exception): + return 'Runtime error' + """ + def decorated(f): + self.error_handlers[status_code_or_exception_class] = f + return f + return decorated + + def mount(self, subapp, url_prefix=''): + """Mount a sub-application, optionally under the given URL prefix. + + :param subapp: The sub-application to mount. + :param url_prefix: The URL prefix to mount the application under. + """ + for methods, pattern, handler in subapp.url_map: + self.url_map.append( + (methods, URLPattern(url_prefix + pattern.url_pattern), + handler)) + for handler in subapp.before_request_handlers: + self.before_request_handlers.append(handler) + for handler in subapp.after_request_handlers: + self.after_request_handlers.append(handler) + for handler in subapp.after_error_request_handlers: + self.after_error_request_handlers.append(handler) + for status_code, handler in subapp.error_handlers.items(): + self.error_handlers[status_code] = handler + + @staticmethod + def abort(status_code, reason=None): + """Abort the current request and return an error response with the + given status code. + + :param status_code: The numeric status code of the response. + :param reason: The reason for the response, which is included in the + response body. + + Example:: + + from microdot import abort + + @app.route('/users/') + def get_user(id): + user = get_user_by_id(id) + if user is None: + abort(404) + return user.to_dict() + """ + raise HTTPException(status_code, reason) + + async def start_server(self, host='0.0.0.0', port=5000, debug=False, + ssl=None): + """Start the Microdot web server as a coroutine. This coroutine does + not normally return, as the server enters an endless listening loop. + The :func:`shutdown` function provides a method for terminating the + server gracefully. + + :param host: The hostname or IP address of the network interface that + will be listening for requests. A value of ``'0.0.0.0'`` + (the default) indicates that the server should listen for + requests on all the available interfaces, and a value of + ``127.0.0.1`` indicates that the server should listen + for requests only on the internal networking interface of + the host. + :param port: The port number to listen for requests. The default is + port 5000. + :param debug: If ``True``, the server logs debugging information. The + default is ``False``. + :param ssl: An ``SSLContext`` instance or ``None`` if the server should + not use TLS. The default is ``None``. + + This method is a coroutine. + + Example:: + + import asyncio + from microdot import Microdot + + app = Microdot() + + @app.route('/') + async def index(request): + return 'Hello, world!' + + async def main(): + await app.start_server(debug=True) + + asyncio.run(main()) + """ + self.debug = debug + + async def serve(reader, writer): + if not hasattr(writer, 'awrite'): # pragma: no cover + # CPython provides the awrite and aclose methods in 3.8+ + async def awrite(self, data): + self.write(data) + await self.drain() + + async def aclose(self): + self.close() + await self.wait_closed() + + from types import MethodType + writer.awrite = MethodType(awrite, writer) + writer.aclose = MethodType(aclose, writer) + + await self.handle_request(reader, writer) + + if self.debug: # pragma: no cover + print('Starting async server on {host}:{port}...'.format( + host=host, port=port)) + + try: + self.server = await asyncio.start_server(serve, host, port, + ssl=ssl) + except TypeError: # pragma: no cover + self.server = await asyncio.start_server(serve, host, port) + + while True: + try: + if hasattr(self.server, 'serve_forever'): # pragma: no cover + try: + await self.server.serve_forever() + except asyncio.CancelledError: + pass + await self.server.wait_closed() + break + except AttributeError: # pragma: no cover + # the task hasn't been initialized in the server object yet + # wait a bit and try again + await asyncio.sleep(0.1) + + def run(self, host='0.0.0.0', port=5000, debug=False, ssl=None): + """Start the web server. This function does not normally return, as + the server enters an endless listening loop. The :func:`shutdown` + function provides a method for terminating the server gracefully. + + :param host: The hostname or IP address of the network interface that + will be listening for requests. A value of ``'0.0.0.0'`` + (the default) indicates that the server should listen for + requests on all the available interfaces, and a value of + ``127.0.0.1`` indicates that the server should listen + for requests only on the internal networking interface of + the host. + :param port: The port number to listen for requests. The default is + port 5000. + :param debug: If ``True``, the server logs debugging information. The + default is ``False``. + :param ssl: An ``SSLContext`` instance or ``None`` if the server should + not use TLS. The default is ``None``. + + Example:: + + from microdot import Microdot + + app = Microdot() + + @app.route('/') + async def index(request): + return 'Hello, world!' + + app.run(debug=True) + """ + asyncio.run(self.start_server(host=host, port=port, debug=debug, + ssl=ssl)) # pragma: no cover + + def shutdown(self): + """Request a server shutdown. The server will then exit its request + listening loop and the :func:`run` function will return. This function + can be safely called from a route handler, as it only schedules the + server to terminate as soon as the request completes. + + Example:: + + @app.route('/shutdown') + def shutdown(request): + request.app.shutdown() + return 'The server is shutting down...' + """ + self.server.close() + + def find_route(self, req): + method = req.method.upper() + if method == 'OPTIONS' and self.options_handler: + return self.options_handler(req) + if method == 'HEAD': + method = 'GET' + f = 404 + for route_methods, route_pattern, route_handler in self.url_map: + req.url_args = route_pattern.match(req.path) + if req.url_args is not None: + if method in route_methods: + f = route_handler + break + else: + f = 405 + return f + + def default_options_handler(self, req): + allow = [] + for route_methods, route_pattern, route_handler in self.url_map: + if route_pattern.match(req.path) is not None: + allow.extend(route_methods) + if 'GET' in allow: + allow.append('HEAD') + allow.append('OPTIONS') + return {'Allow': ', '.join(allow)} + + async def handle_request(self, reader, writer): + req = None + try: + req = await Request.create(self, reader, writer, + writer.get_extra_info('peername')) + except Exception as exc: # pragma: no cover + print_exception(exc) + + res = await self.dispatch_request(req) + if res != Response.already_handled: # pragma: no branch + await res.write(writer) + try: + await writer.aclose() + except OSError as exc: # pragma: no cover + if exc.errno in MUTED_SOCKET_ERRORS: + pass + else: + raise + if self.debug and req: # pragma: no cover + print('{method} {path} {status_code}'.format( + method=req.method, path=req.path, + status_code=res.status_code)) + + async def dispatch_request(self, req): + after_request_handled = False + if req: + if req.content_length > req.max_content_length: + if 413 in self.error_handlers: + res = await invoke_handler(self.error_handlers[413], req) + else: + res = 'Payload too large', 413 + else: + f = self.find_route(req) + try: + res = None + if callable(f): + for handler in self.before_request_handlers: + res = await invoke_handler(handler, req) + if res: + break + if res is None: + res = await invoke_handler( + f, req, **req.url_args) + if isinstance(res, int): + res = '', res + if isinstance(res, tuple): + if isinstance(res[0], int): + res = ('', res[0], + res[1] if len(res) > 1 else {}) + body = res[0] + if isinstance(res[1], int): + status_code = res[1] + headers = res[2] if len(res) > 2 else {} + else: + status_code = 200 + headers = res[1] + res = Response(body, status_code, headers) + elif not isinstance(res, Response): + res = Response(res) + for handler in self.after_request_handlers: + res = await invoke_handler( + handler, req, res) or res + for handler in req.after_request_handlers: + res = await invoke_handler( + handler, req, res) or res + after_request_handled = True + elif isinstance(f, dict): + res = Response(headers=f) + elif f in self.error_handlers: + res = await invoke_handler(self.error_handlers[f], req) + else: + res = 'Not found', f + except HTTPException as exc: + if exc.status_code in self.error_handlers: + res = self.error_handlers[exc.status_code](req) + else: + res = exc.reason, exc.status_code + except Exception as exc: + print_exception(exc) + exc_class = None + res = None + if exc.__class__ in self.error_handlers: + exc_class = exc.__class__ + else: + for c in mro(exc.__class__)[1:]: + if c in self.error_handlers: + exc_class = c + break + if exc_class: + try: + res = await invoke_handler( + self.error_handlers[exc_class], req, exc) + except Exception as exc2: # pragma: no cover + print_exception(exc2) + if res is None: + if 500 in self.error_handlers: + res = await invoke_handler( + self.error_handlers[500], req) + else: + res = 'Internal server error', 500 + else: + if 400 in self.error_handlers: + res = await invoke_handler(self.error_handlers[400], req) + else: + res = 'Bad request', 400 + if isinstance(res, tuple): + res = Response(*res) + elif not isinstance(res, Response): + res = Response(res) + if not after_request_handled: + for handler in self.after_error_request_handlers: + res = await invoke_handler( + handler, req, res) or res + res.is_head = (req and req.method == 'HEAD') + return res + + +Response.already_handled = Response() + +abort = Microdot.abort +redirect = Response.redirect +send_file = Response.send_file \ No newline at end of file diff --git a/espnow-sender/lib/microdot/session.py b/espnow-sender/lib/microdot/session.py new file mode 100644 index 0000000..78ce2e6 --- /dev/null +++ b/espnow-sender/lib/microdot/session.py @@ -0,0 +1,225 @@ +try: + import jwt + HAS_JWT = True +except ImportError: + HAS_JWT = False + try: + import ubinascii + except ImportError: + import binascii as ubinascii + try: + import uhashlib as hashlib + except ImportError: + import hashlib + try: + import uhmac as hmac + except ImportError: + try: + import hmac + except ImportError: + hmac = None + import json + +from microdot.microdot import invoke_handler +from microdot.helpers import wraps + + +class SessionDict(dict): + """A session dictionary. + + The session dictionary is a standard Python dictionary that has been + extended with convenience ``save()`` and ``delete()`` methods. + """ + def __init__(self, request, session_dict): + super().__init__(session_dict) + self.request = request + + def save(self): + """Update the session cookie.""" + self.request.app._session.update(self.request, self) + + def delete(self): + """Delete the session cookie.""" + self.request.app._session.delete(self.request) + + +class Session: + """Session handling + + :param app: The application instance. + :param secret_key: The secret key, as a string or bytes object. + :param cookie_options: A dictionary with cookie options to pass as + arguments to :meth:`Response.set_cookie() + `. + """ + secret_key = None + + def __init__(self, app=None, secret_key=None, cookie_options=None): + self.secret_key = secret_key + self.cookie_options = cookie_options or {} + if app is not None: + self.initialize(app) + + def initialize(self, app, secret_key=None, cookie_options=None): + if secret_key is not None: + self.secret_key = secret_key + if cookie_options is not None: + self.cookie_options = cookie_options + if 'path' not in self.cookie_options: + self.cookie_options['path'] = '/' + if 'http_only' not in self.cookie_options: + self.cookie_options['http_only'] = True + app._session = self + + def get(self, request): + """Retrieve the user session. + + :param request: The client request. + + The return value is a session dictionary with the data stored in the + user's session, or ``{}`` if the session data is not available or + invalid. + """ + if not self.secret_key: + raise ValueError('The session secret key is not configured') + if hasattr(request.g, '_session'): + return request.g._session + session = request.cookies.get('session') + if session is None: + request.g._session = SessionDict(request, {}) + return request.g._session + request.g._session = SessionDict(request, self.decode(session)) + return request.g._session + + def update(self, request, session): + """Update the user session. + + :param request: The client request. + :param session: A dictionary with the update session data for the user. + + Applications would normally not call this method directly, instead they + would use the :meth:`SessionDict.save` method on the session + dictionary, which calls this method. For example:: + + @app.route('/') + @with_session + def index(request, session): + session['foo'] = 'bar' + session.save() + return 'Hello, World!' + + Calling this method adds a cookie with the updated session to the + request currently being processed. + """ + if not self.secret_key: + raise ValueError('The session secret key is not configured') + + encoded_session = self.encode(session) + + @request.after_request + def _update_session(request, response): + response.set_cookie('session', encoded_session, + **self.cookie_options) + return response + + def delete(self, request): + """Remove the user session. + + :param request: The client request. + + Applications would normally not call this method directly, instead they + would use the :meth:`SessionDict.delete` method on the session + dictionary, which calls this method. For example:: + + @app.route('/') + @with_session + def index(request, session): + session.delete() + return 'Hello, World!' + + Calling this method adds a cookie removal header to the request + currently being processed. + """ + @request.after_request + def _delete_session(request, response): + response.delete_cookie('session', **self.cookie_options) + return response + + def encode(self, payload, secret_key=None): + """Encode session data using JWT if available, otherwise use simple HMAC.""" + if HAS_JWT: + return jwt.encode(payload, secret_key or self.secret_key, + algorithm='HS256') + else: + # Simple encoding for MicroPython: base64(json) + HMAC signature + key = (secret_key or self.secret_key).encode() if isinstance(secret_key or self.secret_key, str) else (secret_key or self.secret_key) + payload_json = json.dumps(payload) + payload_b64 = ubinascii.b2a_base64(payload_json.encode()).decode().strip() + + # Create HMAC signature + if hmac: + # Use hmac module if available + h = hmac.new(key, payload_json.encode(), hashlib.sha256) + else: + # Fallback: simple SHA256(key + message) + h = hashlib.sha256(key + payload_json.encode()) + signature = ubinascii.b2a_base64(h.digest()).decode().strip() + + return f"{payload_b64}.{signature}" + + def decode(self, session, secret_key=None): + """Decode session data using JWT if available, otherwise use simple HMAC.""" + if HAS_JWT: + try: + payload = jwt.decode(session, secret_key or self.secret_key, + algorithms=['HS256']) + except jwt.exceptions.PyJWTError: # pragma: no cover + return {} + return payload + else: + try: + # Simple decoding for MicroPython + if '.' not in session: + return {} + + payload_b64, signature = session.rsplit('.', 1) + payload_json = ubinascii.a2b_base64(payload_b64).decode() + + # Verify HMAC signature + key = (secret_key or self.secret_key).encode() if isinstance(secret_key or self.secret_key, str) else (secret_key or self.secret_key) + if hmac: + # Use hmac module if available + h = hmac.new(key, payload_json.encode(), hashlib.sha256) + else: + # Fallback: simple SHA256(key + message) + h = hashlib.sha256(key + payload_json.encode()) + expected_signature = ubinascii.b2a_base64(h.digest()).decode().strip() + + if signature != expected_signature: + return {} + + return json.loads(payload_json) + except Exception: + return {} + + +def with_session(f): + """Decorator that passes the user session to the route handler. + + The session dictionary is passed to the decorated function as an argument + after the request object. Example:: + + @app.route('/') + @with_session + def index(request, session): + return 'Hello, World!' + + Note that the decorator does not save the session. To update the session, + call the :func:`session.save() ` method. + """ + @wraps(f) + async def wrapper(request, *args, **kwargs): + return await invoke_handler( + f, request, request.app._session.get(request), *args, **kwargs) + + return wrapper diff --git a/espnow-sender/lib/microdot/utemplate.py b/espnow-sender/lib/microdot/utemplate.py new file mode 100644 index 0000000..16d0398 --- /dev/null +++ b/espnow-sender/lib/microdot/utemplate.py @@ -0,0 +1,70 @@ +from utemplate import recompile + +_loader = None + + +class Template: + """A template object. + + :param template: The filename of the template to render, relative to the + configured template directory. + """ + @classmethod + def initialize(cls, template_dir='templates', + loader_class=recompile.Loader): + """Initialize the templating subsystem. + + :param template_dir: the directory where templates are stored. This + argument is optional. The default is to load + templates from a *templates* subdirectory. + :param loader_class: the ``utemplate.Loader`` class to use when loading + templates. This argument is optional. The default + is the ``recompile.Loader`` class, which + automatically recompiles templates when they + change. + """ + global _loader + _loader = loader_class(None, template_dir) + + def __init__(self, template): + if _loader is None: # pragma: no cover + self.initialize() + #: The name of the template + self.name = template + self.template = _loader.load(template) + + def generate(self, *args, **kwargs): + """Return a generator that renders the template in chunks, with the + given arguments.""" + return self.template(*args, **kwargs) + + def render(self, *args, **kwargs): + """Render the template with the given arguments and return it as a + string.""" + return ''.join(self.generate(*args, **kwargs)) + + def generate_async(self, *args, **kwargs): + """Return an asynchronous generator that renders the template in + chunks, using the given arguments.""" + class sync_to_async_iter(): + def __init__(self, iter): + self.iter = iter + + def __aiter__(self): + return self + + async def __anext__(self): + try: + return next(self.iter) + except StopIteration: + raise StopAsyncIteration + + return sync_to_async_iter(self.generate(*args, **kwargs)) + + async def render_async(self, *args, **kwargs): + """Render the template with the given arguments asynchronously and + return it as a string.""" + response = '' + async for chunk in self.generate_async(*args, **kwargs): + response += chunk + return response diff --git a/espnow-sender/lib/microdot/websocket.py b/espnow-sender/lib/microdot/websocket.py new file mode 100644 index 0000000..0fb6f7c --- /dev/null +++ b/espnow-sender/lib/microdot/websocket.py @@ -0,0 +1,231 @@ +import binascii +import hashlib +from microdot import Request, Response +from microdot.microdot import MUTED_SOCKET_ERRORS, print_exception +from microdot.helpers import wraps + + +class WebSocketError(Exception): + """Exception raised when an error occurs in a WebSocket connection.""" + pass + + +class WebSocket: + """A WebSocket connection object. + + An instance of this class is sent to handler functions to manage the + WebSocket connection. + """ + CONT = 0 + TEXT = 1 + BINARY = 2 + CLOSE = 8 + PING = 9 + PONG = 10 + + #: Specify the maximum message size that can be received when calling the + #: ``receive()`` method. Messages with payloads that are larger than this + #: size will be rejected and the connection closed. Set to 0 to disable + #: the size check (be aware of potential security issues if you do this), + #: or to -1 to use the value set in + #: ``Request.max_body_length``. The default is -1. + #: + #: Example:: + #: + #: WebSocket.max_message_length = 4 * 1024 # up to 4KB messages + max_message_length = -1 + + def __init__(self, request): + self.request = request + self.closed = False + + async def handshake(self): + response = self._handshake_response() + await self.request.sock[1].awrite( + b'HTTP/1.1 101 Switching Protocols\r\n') + await self.request.sock[1].awrite(b'Upgrade: websocket\r\n') + await self.request.sock[1].awrite(b'Connection: Upgrade\r\n') + await self.request.sock[1].awrite( + b'Sec-WebSocket-Accept: ' + response + b'\r\n\r\n') + + async def receive(self): + """Receive a message from the client.""" + while True: + opcode, payload = await self._read_frame() + send_opcode, data = self._process_websocket_frame(opcode, payload) + if send_opcode: # pragma: no cover + await self.send(data, send_opcode) + elif data: # pragma: no branch + return data + + async def send(self, data, opcode=None): + """Send a message to the client. + + :param data: the data to send, given as a string or bytes. + :param opcode: a custom frame opcode to use. If not given, the opcode + is ``TEXT`` or ``BINARY`` depending on the type of the + data. + """ + frame = self._encode_websocket_frame( + opcode or (self.TEXT if isinstance(data, str) else self.BINARY), + data) + await self.request.sock[1].awrite(frame) + + async def close(self): + """Close the websocket connection.""" + if not self.closed: # pragma: no cover + self.closed = True + await self.send(b'', self.CLOSE) + + def _handshake_response(self): + connection = False + upgrade = False + websocket_key = None + for header, value in self.request.headers.items(): + h = header.lower() + if h == 'connection': + connection = True + if 'upgrade' not in value.lower(): + return self.request.app.abort(400) + elif h == 'upgrade': + upgrade = True + if not value.lower() == 'websocket': + return self.request.app.abort(400) + elif h == 'sec-websocket-key': + websocket_key = value + if not connection or not upgrade or not websocket_key: + return self.request.app.abort(400) + d = hashlib.sha1(websocket_key.encode()) + d.update(b'258EAFA5-E914-47DA-95CA-C5AB0DC85B11') + return binascii.b2a_base64(d.digest())[:-1] + + @classmethod + def _parse_frame_header(cls, header): + fin = header[0] & 0x80 + opcode = header[0] & 0x0f + if fin == 0 or opcode == cls.CONT: # pragma: no cover + raise WebSocketError('Continuation frames not supported') + has_mask = header[1] & 0x80 + length = header[1] & 0x7f + if length == 126: + length = -2 + elif length == 127: + length = -8 + return fin, opcode, has_mask, length + + def _process_websocket_frame(self, opcode, payload): + if opcode == self.TEXT: + payload = payload.decode() + elif opcode == self.BINARY: + pass + elif opcode == self.CLOSE: + raise WebSocketError('Websocket connection closed') + elif opcode == self.PING: + return self.PONG, payload + elif opcode == self.PONG: # pragma: no branch + return None, None + return None, payload + + @classmethod + def _encode_websocket_frame(cls, opcode, payload): + frame = bytearray() + frame.append(0x80 | opcode) + if opcode == cls.TEXT: + payload = payload.encode() + if len(payload) < 126: + frame.append(len(payload)) + elif len(payload) < (1 << 16): + frame.append(126) + frame.extend(len(payload).to_bytes(2, 'big')) + else: + frame.append(127) + frame.extend(len(payload).to_bytes(8, 'big')) + frame.extend(payload) + return frame + + async def _read_frame(self): + header = await self.request.sock[0].read(2) + if len(header) != 2: # pragma: no cover + raise WebSocketError('Websocket connection closed') + fin, opcode, has_mask, length = self._parse_frame_header(header) + if length == -2: + length = await self.request.sock[0].read(2) + length = int.from_bytes(length, 'big') + elif length == -8: + length = await self.request.sock[0].read(8) + length = int.from_bytes(length, 'big') + max_allowed_length = Request.max_body_length \ + if self.max_message_length == -1 else self.max_message_length + if length > max_allowed_length: + raise WebSocketError('Message too large') + if has_mask: # pragma: no cover + mask = await self.request.sock[0].read(4) + payload = await self.request.sock[0].read(length) + if has_mask: # pragma: no cover + payload = bytes(x ^ mask[i % 4] for i, x in enumerate(payload)) + return opcode, payload + + +async def websocket_upgrade(request): + """Upgrade a request handler to a websocket connection. + + This function can be called directly inside a route function to process a + WebSocket upgrade handshake, for example after the user's credentials are + verified. The function returns the websocket object:: + + @app.route('/echo') + async def echo(request): + if not authenticate_user(request): + abort(401) + ws = await websocket_upgrade(request) + while True: + message = await ws.receive() + await ws.send(message) + """ + ws = WebSocket(request) + await ws.handshake() + + @request.after_request + async def after_request(request, response): + return Response.already_handled + + return ws + + +def websocket_wrapper(f, upgrade_function): + @wraps(f) + async def wrapper(request, *args, **kwargs): + ws = await upgrade_function(request) + try: + await f(request, ws, *args, **kwargs) + except OSError as exc: + if exc.errno not in MUTED_SOCKET_ERRORS: # pragma: no cover + raise + except WebSocketError: + pass + except Exception as exc: + print_exception(exc) + finally: # pragma: no cover + try: + await ws.close() + except Exception: + pass + return Response.already_handled + return wrapper + + +def with_websocket(f): + """Decorator to make a route a WebSocket endpoint. + + This decorator is used to define a route that accepts websocket + connections. The route then receives a websocket object as a second + argument that it can use to send and receive messages:: + + @app.route('/echo') + @with_websocket + async def echo(request, ws): + while True: + message = await ws.receive() + await ws.send(message) + """ + return websocket_wrapper(f, websocket_upgrade) diff --git a/espnow-sender/main.py b/espnow-sender/main.py deleted file mode 100644 index 2aa184b..0000000 --- a/espnow-sender/main.py +++ /dev/null @@ -1,120 +0,0 @@ -import asyncio -import json - -from microdot import Microdot -from microdot.websocket import WebSocketError, with_websocket - -import espnow -import network -from util import format_mac, parse_mac - - -app = Microdot() -_esp = None -_known_peers = set() -_ws_clients = set() - - -def _init_espnow(): - global _esp - sta = network.WLAN(network.STA_IF) - sta.active(True) - _esp = espnow.ESPNow() - _esp.active(True) - - -def _validate_envelope(obj): - if obj.get("v") != "1": - raise ValueError("message.v must be '1'") - devices = obj["devices"] - for address in devices.keys(): - parse_mac(address) - return obj - - -def _send_espnow(address, payload): - if _esp is None: - raise ValueError("espnow is not initialized") - mac = parse_mac(address) - msg = json.dumps(payload, separators=(",", ":")).encode("utf-8") - if mac not in _known_peers: - _esp.add_peer(mac) - _known_peers.add(mac) - _esp.send(mac, msg) - return mac, len(msg) - - -async def _broadcast_ws(obj): - text = json.dumps(obj) - dead = [] - for client in list(_ws_clients): - try: - await client.send(text) - except Exception: - dead.append(client) - for client in dead: - _ws_clients.discard(client) - - -async def _espnow_receive_loop(): - while True: - host, msg = _esp.recv(0) - if not host: - await asyncio.sleep(0.01) - continue - await _broadcast_ws( - { - "from": format_mac(host), - "payload": msg.decode("utf-8"), - } - ) - - -@app.route("/ws") -@with_websocket -async def ws(request, ws): - _ws_clients.add(ws) - while True: - try: - raw = await ws.receive() - except WebSocketError: - break - - if not raw: - break - - try: - parsed = json.loads(raw) - env = _validate_envelope(parsed) - sent = [] - for address, payload in env["devices"].items(): - mac, payload_size = _send_espnow(address, payload) - sent.append( - { - "address": format_mac(mac), - "bytes": payload_size, - } - ) - except (ValueError, TypeError) as e: - await ws.send(json.dumps({"ok": False, "error": str(e)})) - continue - - await ws.send( - json.dumps( - { - "ok": True, - "sent": sent, - } - ) - ) - _ws_clients.discard(ws) - - -async def main(port=80): - _init_espnow() - asyncio.create_task(_espnow_receive_loop()) - await app.start_server(host="0.0.0.0", port=port) - - -if __name__ == "__main__": - asyncio.run(main(port=80)) diff --git a/espnow-sender/src/espnow_wire.py b/espnow-sender/src/espnow_wire.py new file mode 100644 index 0000000..36b545a --- /dev/null +++ b/espnow-sender/src/espnow_wire.py @@ -0,0 +1,28 @@ +"""ESP-NOW / WebSocket framing (MicroPython). See docs/espnow-binary-protocol.md.""" + +WIRE_MAGIC = 0x4C +MSG_BRIDGE_CH = 0x10 +BROADCAST_MAC = b"\xff\xff\xff\xff\xff\xff" +WS_FLAG_BROADCAST = 0x01 +MAX_PEERS = 20 + + +def parse_ws_downlink(frame): + """Return (peer_bytes, espnow_packet, is_broadcast).""" + if not frame or len(frame) < 8: + raise ValueError("frame too short") + flags = frame[0] + peer = frame[1:7] + pkt = frame[7:] + broadcast = bool(flags & WS_FLAG_BROADCAST) or peer == BROADCAST_MAC + return peer, pkt, broadcast + + +def pack_ws_uplink(peer, espnow_packet): + return bytes([0]) + peer + espnow_packet + + +def parse_bridge_channel(pkt): + if len(pkt) >= 3 and pkt[0] == WIRE_MAGIC and pkt[1] == MSG_BRIDGE_CH: + return pkt[2] + return None diff --git a/espnow-sender/src/main.py b/espnow-sender/src/main.py new file mode 100644 index 0000000..ef5b7b4 --- /dev/null +++ b/espnow-sender/src/main.py @@ -0,0 +1,76 @@ +import asyncio +import time + +from microdot import Microdot +from microdot.websocket import WebSocketError, with_websocket + +import aioespnow +import machine +import network +from settings import Settings + + +wdt = machine.WDT(timeout=10000) +wdt.feed() +settings = Settings() +print(settings) + +app = Microdot() + +ap_if = network.WLAN(network.AP_IF) +ap_if.active(True) +ap_if.config(ssid=settings.get("name"), password=settings.get("ap_password")) +print(ap_if.ifconfig()) + +sta_if = network.WLAN(network.STA_IF) +sta_if.active(True) +print(sta_if.config("channel")) + +esp = aioespnow.AIOESPNow() +esp.active(True) +esp.add_peer(b"\xff\xff\xff\xff\xff\xff") + +clients = set() + +@app.route("/ws") +@with_websocket +async def ws(request, ws): + clients.add(ws) + while True: + + try: + raw = await ws.receive() + except WebSocketError as err: + print(err) + break + if not raw: + break + try: + await esp.asend(b"\xff\xff\xff\xff\xff\xff", raw) + print(raw) + except Exception as err: + print(err) + break + ws.close() + clients.discard(ws) + +async def _espnow_receive_loop(): + async for host, msg in esp.airecv(): + print(host, msg) + for client in clients: + await client.send(msg) + + +async def _wdt_feed_loop(): + while True: + await asyncio.sleep(1) + wdt.feed() + +async def main(): + asyncio.create_task(_wdt_feed_loop()) + asyncio.create_task(_espnow_receive_loop()) + await app.start_server(host="0.0.0.0", port=80) + + +if __name__ == "__main__": + asyncio.run(main()) \ No newline at end of file diff --git a/espnow-sender/src/settings.py b/espnow-sender/src/settings.py new file mode 100644 index 0000000..5f0e363 --- /dev/null +++ b/espnow-sender/src/settings.py @@ -0,0 +1,73 @@ +import json +import time +import ubinascii +import network + + +def _sta_mac_hex(): + """Read STA MAC without leaving the radio up (wifi_ap owns bring-up).""" + sta = network.WLAN(network.STA_IF) + was_on = False + try: + was_on = sta.active() + except Exception: + pass + if not was_on: + try: + sta.active(True) + time.sleep_ms(50) + except Exception: + pass + try: + mac = ubinascii.hexlify(sta.config("mac")).decode().lower() + except Exception: + mac = "000000000000" + if not was_on: + try: + sta.active(False) + except Exception: + pass + return mac + + +class Settings(dict): + SETTINGS_FILE = "/settings.json" + + def __init__(self): + super().__init__() + self.load() + + def set_defaults(self): + mac = _sta_mac_hex() + self["name"] = "bridge-" + mac + self["wifi_channel"] = 6 + self["ap_password"] = "" + self["ap_ip"] = "192.168.4.1" + self["ws_port"] = 80 + self["max_peers"] = 20 + + def save(self): + try: + with open(self.SETTINGS_FILE, "w") as file: + file.write(json.dumps(self)) + except Exception as e: + print("Error saving settings:", e) + + def load(self): + try: + with open(self.SETTINGS_FILE, "r") as file: + loaded = json.load(file) + if not isinstance(loaded, dict): + raise ValueError("settings.json is not an object") + except Exception: + print("Error loading settings") + self.clear() + self.set_defaults() + self.save() + return + self.clear() + self.set_defaults() + for k, v in loaded.items(): + self[k] = v + + \ No newline at end of file diff --git a/espnow-sender/src/util.py b/espnow-sender/src/util.py new file mode 100644 index 0000000..c65e8f4 --- /dev/null +++ b/espnow-sender/src/util.py @@ -0,0 +1,49 @@ +def parse_mac(value): + raw = value.strip().lower().replace(":", "").replace("-", "") + if len(raw) != 12: + raise ValueError("address must be 12 hex chars or aa:bb:cc:dd:ee:ff") + try: + return bytes.fromhex(raw) + except ValueError: + raise ValueError("address contains non-hex characters") + + +def format_mac(mac_bytes): + return ":".join("{:02x}".format(b) for b in mac_bytes) + + +def print_bridge_ip(ws_port=80): + import network + + try: + port = int(ws_port) + except (TypeError, ValueError): + port = 80 + + ips = [] + try: + sta = network.WLAN(network.STA_IF) + if sta.active(): + ip = sta.ifconfig()[0] + if ip and ip != "0.0.0.0": + ips.append(("STA", ip)) + except Exception: + pass + try: + ap = network.WLAN(network.AP_IF) + if ap.active(): + ip = ap.ifconfig()[0] + if ip and ip != "0.0.0.0": + ips.append(("AP", ip)) + except Exception: + pass + + if not ips: + print("bridge IP: (AP not up)") + return + + # Prefer AP address — Pi joins the bridge access point. + ips.sort(key=lambda x: 0 if x[0] == "AP" else 1) + label, ip = ips[0] + print("bridge IP (%s):" % label, ip) + print("bridge_ws_url: ws://%s:%s/ws" % (ip, port)) diff --git a/espnow-sender/src/wifi_ap.py b/espnow-sender/src/wifi_ap.py new file mode 100644 index 0000000..d8221f3 --- /dev/null +++ b/espnow-sender/src/wifi_ap.py @@ -0,0 +1,66 @@ +"""Bridge Wi-Fi: AP for Pi WebSocket client, STA for ESP-NOW (ESP32-C3: AP first).""" + +import time + +import network + + +def _wait_active(wlan, timeout_ms=1000): + for _ in range(timeout_ms // 20): + if wlan.active(): + return True + time.sleep_ms(20) + return bool(wlan.active()) + + +def _boot_channel(settings): + try: + return max(1, min(11, int(settings.get("wifi_channel", 6)))) + except (TypeError, ValueError): + return 6 + + +def init_bridge_network(settings): + """Bring up AP (Pi) then STA (ESP-NOW). Channel set on AP at boot only.""" + ch = _boot_channel(settings) + sta = network.WLAN(network.STA_IF) + ap = network.WLAN(network.AP_IF) + + try: + sta.active(False) + ap.active(False) + except Exception: + pass + time.sleep_ms(100) + + essid = settings.get("name") or "espnow-bridge" + password = settings.get("ap_password") or "" + + ap.active(True) + if not _wait_active(ap): + raise RuntimeError("AP did not become active") + + if password: + ap.config(essid=essid, password=password, channel=ch) + else: + ap.config(essid=essid, channel=ch) + + ap_ip = settings.get("ap_ip") or "192.168.4.1" + try: + ap.ifconfig((ap_ip, "255.255.255.0", ap_ip, "8.8.8.8")) + except Exception as e: + print("ap ifconfig:", e) + + sta.active(True) + if not _wait_active(sta): + raise RuntimeError("STA did not become active") + try: + sta.config(pm=network.WLAN.PM_NONE) + except Exception: + pass + + try: + actual = ap.config("channel") + except Exception: + actual = ch + print("bridge AP:", essid, "channel=", actual, "ip=", ap.ifconfig()[0]) diff --git a/espnow-sender/util.py b/espnow-sender/util.py deleted file mode 100644 index 65ac9a9..0000000 --- a/espnow-sender/util.py +++ /dev/null @@ -1,12 +0,0 @@ -def parse_mac(value): - raw = value.strip().lower().replace(":", "").replace("-", "") - if len(raw) != 12: - raise ValueError("address must be 12 hex chars or aa:bb:cc:dd:ee:ff") - try: - return bytes.fromhex(raw) - except ValueError: - raise ValueError("address contains non-hex characters") - - -def format_mac(mac_bytes): - return ":".join("{:02x}".format(b) for b in mac_bytes) diff --git a/led-driver b/led-driver index 85490a3..3e718f7 160000 --- a/led-driver +++ b/led-driver @@ -1 +1 @@ -Subproject commit 85490a3bd09f07e29744e098a560fe532fe7afb4 +Subproject commit 3e718f74322af2a0032a77b11e9f7333a34b2769 diff --git a/src/controllers/device.py b/src/controllers/device.py index 5163d9f..8f8674e 100644 --- a/src/controllers/device.py +++ b/src/controllers/device.py @@ -10,12 +10,8 @@ from models.group import Group from models.transport import get_current_sender from settings import get_settings from util.brightness_combine import effective_brightness_for_mac -from models.wifi_ws_clients import ( - normalize_tcp_peer_ip, - send_json_line_to_ip, - tcp_client_connected, -) from util.driver_patterns import driver_patterns_dir +from util.binary_driver_messages import v1_dict_to_cmd_packet from util.espnow_message import build_message import asyncio import json @@ -81,17 +77,8 @@ _pi_settings = get_settings() def _device_live_connected(dev_dict): - """ - Wi-Fi: whether the controller has an outbound WebSocket to this device's IP. - ESP-NOW: None (no Wi-Fi session on the Pi for that transport). - """ - tr = (dev_dict.get("transport") or "espnow").strip().lower() - if tr != "wifi": - return None - ip = normalize_tcp_peer_ip(dev_dict.get("address") or "") - if not ip: - return False - return tcp_client_connected(ip) + """ESP-NOW has no live session flag on the Pi.""" + return None def _device_json_with_live_status(dev_dict): @@ -155,14 +142,13 @@ def _http_post_pattern_source(ip, filename, code_text, reload_patterns=True, tim return b" 2" in first_line -async def _identify_send_off_after_delay(sender, transport, wifi_ip, dev_id, name): +async def _identify_send_off_after_delay(sender, dev_id, name): try: await asyncio.sleep(IDENTIFY_OFF_DELAY_S) - off_msg = build_message(select={name: ["off"]}) - if transport == "wifi": - await send_json_line_to_ip(wifi_ip, off_msg) - else: - await sender.send(off_msg, addr=dev_id) + pkt = v1_dict_to_cmd_packet( + {"v": "1", "select": {name: ["off"]}}, + ) + await sender.send(pkt, addr=dev_id) except Exception: pass @@ -184,27 +170,20 @@ async def send_identify_to_device(dev_id: str) -> tuple[int, str]: if not name: return 400, "Device must have a name to identify" - transport = dev.get("transport") or "espnow" - wifi_ip = None - if transport == "wifi": - wifi_ip = dev.get("address") - if not wifi_ip: - return 400, "Device has no IP address" - try: - msg = _compact_v1_json( - presets={_IDENTIFY_PRESET_KEY: dict(_IDENTIFY_DRIVER_PRESET)}, - select={name: [_IDENTIFY_PRESET_KEY]}, + pkt = v1_dict_to_cmd_packet( + { + "v": "1", + "presets": {_IDENTIFY_PRESET_KEY: dict(_IDENTIFY_DRIVER_PRESET)}, + "select": {name: [_IDENTIFY_PRESET_KEY]}, + } ) - if transport == "wifi": - ok = await send_json_line_to_ip(wifi_ip, msg) - if not ok: - return 503, "Wi-Fi driver not connected" - else: - await sender.send(msg, addr=dev_id) + ok = await sender.send(pkt, addr=dev_id) + if not ok: + return 503, "Send failed" asyncio.create_task( - _identify_send_off_after_delay(sender, transport, wifi_ip, dev_id, name) + _identify_send_off_after_delay(sender, dev_id, name) ) except Exception as e: return 503, str(e) @@ -236,11 +215,6 @@ async def send_identify_to_group_devices(macs: list[str]) -> tuple[int, list[dic if not name: errors.append({"mac": dev_id, "error": "Device must have a name to identify"}) continue - transport = (dev.get("transport") or "espnow").strip().lower() - if transport == "wifi": - if not dev.get("address"): - errors.append({"mac": dev_id, "error": "Device has no IP address"}) - continue merged_select[name] = [_IDENTIFY_PRESET_KEY] valid_macs.append(dev_id) @@ -259,10 +233,8 @@ async def send_identify_to_group_devices(macs: list[str]) -> tuple[int, list[dic for dev_id in valid_macs: dev = devices.read(dev_id) or {} name = str(dev.get("name") or "").strip() - transport = (dev.get("transport") or "espnow").strip().lower() - wifi_ip = dev.get("address") if transport == "wifi" else None asyncio.create_task( - _identify_send_off_after_delay(sender, transport, wifi_ip, dev_id, name) + _identify_send_off_after_delay(sender, dev_id, name) ) return len(valid_macs), errors @@ -476,30 +448,20 @@ async def push_device_output_brightness(request, id): zone_brightness=zb, ) - msg = _brightness_save_message_json(b_val) - transport = (dev.get("transport") or "espnow").strip().lower() - - if transport == "wifi": - ip = normalize_tcp_peer_ip(str(dev.get("address") or "")) - if not ip: - return json.dumps({"error": "Device has no IP address"}), 400, { - "Content-Type": "application/json", - } - ok = await send_json_line_to_ip(ip, msg) + pkt = v1_dict_to_cmd_packet({"v": "1", "b": b_val, "save": True}) + sender = get_current_sender() + if not sender: + return json.dumps({"error": "Transport not configured"}), 503, { + "Content-Type": "application/json", + } + try: + ok = await sender.send(pkt, addr=id) if not ok: - return json.dumps({"error": "Wi-Fi driver not connected"}), 503, { + return json.dumps({"error": "Send failed"}), 503, { "Content-Type": "application/json", } - else: - sender = get_current_sender() - if not sender: - return json.dumps({"error": "Transport not configured"}), 503, { - "Content-Type": "application/json", - } - try: - await sender.send(msg, addr=id) - except Exception as e: - return json.dumps({"error": str(e)}), 503, {"Content-Type": "application/json"} + except Exception as e: + return json.dumps({"error": str(e)}), 503, {"Content-Type": "application/json"} return json.dumps({"message": "brightness sent", "brightness": b_val}), 200, { "Content-Type": "application/json", @@ -509,7 +471,7 @@ async def push_device_output_brightness(request, id): @controller.post("//driver-config") async def push_driver_config(request, id): """ - Push ``device_config`` to a Wi‑Fi LED driver over WebSocket. + Push ``device_config`` to an ESP-NOW LED driver. Body JSON: optional ``name``, ``num_leds``, ``color_order``, ``startup_mode`` (default|last|off). """ dev = devices.read(id) @@ -517,13 +479,9 @@ async def push_driver_config(request, id): return json.dumps({"error": "Device not found"}), 404, { "Content-Type": "application/json", } - if (dev.get("transport") or "").lower() != "wifi": - return json.dumps({"error": "driver-config is only for Wi-Fi devices"}), 400, { - "Content-Type": "application/json", - } - wifi_ip = str(dev.get("address") or "").strip() - if not wifi_ip: - return json.dumps({"error": "Device has no IP address"}), 400, { + sender = get_current_sender() + if not sender: + return json.dumps({"error": "Transport not configured"}), 503, { "Content-Type": "application/json", } body = request.json or {} @@ -551,12 +509,10 @@ async def push_driver_config(request, id): "error": "Provide at least one of name, num_leds, color_order, startup_mode" } ), 400, {"Content-Type": "application/json"} - msg = json.dumps( - {"v": "1", "device_config": dc, "save": True}, separators=(",", ":") - ) - ok = await send_json_line_to_ip(wifi_ip, msg) + pkt = v1_dict_to_cmd_packet({"v": "1", "device_config": dc, "save": True}) + ok = await sender.send(pkt, addr=id) if not ok: - return json.dumps({"error": "Wi-Fi driver not connected"}), 503, { + return json.dumps({"error": "Send failed"}), 503, { "Content-Type": "application/json", } return json.dumps({"message": "driver-config sent"}), 200, { @@ -567,71 +523,13 @@ async def push_driver_config(request, id): @controller.post("//patterns/push") async def push_patterns_ota(request, id): """ - Push all local pattern files directly to a Wi-Fi LED driver over HTTP upload. + Pattern OTA over HTTP is not available for ESP-NOW drivers. """ dev = devices.read(id) if not dev: return json.dumps({"error": "Device not found"}), 404, { "Content-Type": "application/json", } - if (dev.get("transport") or "").lower() != "wifi": - return json.dumps({"error": "Pattern OTA push is only supported for Wi-Fi devices"}), 400, { - "Content-Type": "application/json", - } - wifi_ip = str(dev.get("address") or "").strip() - if not wifi_ip: - return json.dumps({"error": "Device has no IP address"}), 400, { - "Content-Type": "application/json", - } - - base_dir = driver_patterns_dir() - try: - names = sorted(os.listdir(base_dir)) - except OSError as e: - return json.dumps({"error": str(e)}), 500, { - "Content-Type": "application/json", - } - - files = [n for n in names if _safe_pattern_filename(n) and n != "__init__.py"] - if not files: - return json.dumps({"error": "No pattern files found"}), 404, { - "Content-Type": "application/json", - } - - sent = [] - failed = [] - total = len(files) - for idx, filename in enumerate(files): - path = os.path.join(base_dir, filename) - try: - with open(path, "r") as f: - code = f.read() - except OSError: - failed.append(filename) - continue - reload_patterns = idx == (total - 1) - ok = _http_post_pattern_source( - wifi_ip, - filename, - code, - reload_patterns=reload_patterns, - timeout_s=10.0, - ) - if ok: - sent.append(filename) - else: - failed.append(filename) - - if not sent: - return json.dumps({"error": "Wi-Fi driver did not accept pattern uploads", "failed": failed}), 503, { - "Content-Type": "application/json", - } - - return json.dumps({ - "message": "Pattern files uploaded", - "sent_count": len(sent), - "sent": sent, - "failed": failed, - }), 200, { - "Content-Type": "application/json", - } + return json.dumps( + {"error": "Pattern OTA push is not supported for ESP-NOW devices"} + ), 400, {"Content-Type": "application/json"} diff --git a/src/controllers/group.py b/src/controllers/group.py index 697afc6..9198e4d 100644 --- a/src/controllers/group.py +++ b/src/controllers/group.py @@ -4,7 +4,8 @@ import asyncio from models.group import Group from models.device import Device from models.transport import get_current_sender -from models.wifi_ws_clients import normalize_tcp_peer_ip, send_json_line_to_ip +from util.binary_driver_messages import v1_dict_to_cmd_packet +from util.espnow_registry import push_groups_for_group_devices from settings import get_settings from util.brightness_combine import effective_brightness_for_mac import json @@ -101,6 +102,9 @@ async def create_group(request, session): cur = get_current_profile_id(session) if cur: groups.update(group_id, {"profile_id": str(cur)}) + g = groups.read(group_id) + if g: + await push_groups_for_group_devices(g) return json.dumps(groups.read(group_id)), 201, {"Content-Type": "application/json"} except Exception as e: return json.dumps({"error": str(e)}), 400 @@ -119,6 +123,7 @@ async def update_group(request, session, id): if groups.update(id, data): g = groups.read(id) if g: + await push_groups_for_group_devices(g) return json.dumps(g), 200, {"Content-Type": "application/json"} return json.dumps({"error": "Group not found"}), 404 except Exception as e: @@ -135,7 +140,9 @@ async def delete_group(request, session, id): if not _group_doc_visible_for_profile(g, get_current_profile_id(session)): return json.dumps({"error": "Group not found"}), 404 + macs = list(g.get("devices") or []) if isinstance(g, dict) else [] if groups.delete(id): + await push_groups_for_group_devices({"devices": macs}) return json.dumps({"message": "Group deleted successfully"}), 200 return json.dumps({"error": "Group not found"}), 404 @@ -184,7 +191,7 @@ def _read_group_for_session(session, id): @with_session async def push_group_driver_config(request, session, id): """ - Push group Wi‑Fi defaults to every Wi‑Fi device listed in the group (TCP WebSocket). + Push group driver defaults to every ESP-NOW device listed in the group. Uses stored ``wifi_*`` fields on the group; optional JSON body may override for this send only. """ gdoc = _read_group_for_session(session, id) @@ -211,11 +218,10 @@ async def push_group_driver_config(request, session, id): mac_list = gdoc.get("devices") if isinstance(gdoc.get("devices"), list) else [] sent = 0 errors = [] - msg = json.dumps( - {"v": "1", "device_config": dc, "save": True}, separators=(",", ":") - ) - tasks = [] - meta_macs = [] + sender = get_current_sender() + if not sender: + return json.dumps({"error": "Transport not configured"}), 503 + pkt = v1_dict_to_cmd_packet({"v": "1", "device_config": dc, "save": True}) for mac in mac_list: m = str(mac).strip().lower().replace(":", "").replace("-", "") if len(m) != 12: @@ -224,23 +230,13 @@ async def push_group_driver_config(request, session, id): if not dev: errors.append({"mac": m, "error": "not in registry"}) continue - if (dev.get("transport") or "").lower() != "wifi": - continue - ip = normalize_tcp_peer_ip(str(dev.get("address") or "")) - if not ip: - errors.append({"mac": m, "error": "no IP"}) - continue - tasks.append(send_json_line_to_ip(ip, msg)) - meta_macs.append(m) - if tasks: - results = await asyncio.gather(*tasks, return_exceptions=True) - for m, r in zip(meta_macs, results): - if r is True: + try: + if await sender.send(pkt, addr=m): sent += 1 - elif isinstance(r, Exception): - errors.append({"mac": m, "error": str(r)}) else: - errors.append({"mac": m, "error": "driver not connected"}) + errors.append({"mac": m, "error": "send failed"}) + except Exception as e: + errors.append({"mac": m, "error": str(e)}) return json.dumps( {"message": "driver-config sent", "sent": sent, "errors": errors} @@ -275,19 +271,14 @@ async def push_group_output_brightness(request, session, id): m, zone_brightness=None, ) - msg = _brightness_save_message_json(b_val) - transport = (dev.get("transport") or "espnow").strip().lower() - if transport == "wifi": - ip = normalize_tcp_peer_ip(str(dev.get("address") or "")) - if not ip: - return m, False, "no IP" - ok = await send_json_line_to_ip(ip, msg) - return m, bool(ok), None if ok else "driver not connected" + pkt = v1_dict_to_cmd_packet( + {"v": "1", "b": b_val, "save": True}, + ) if not sender: return m, False, "transport not configured" try: - await sender.send(msg, addr=m) - return m, True, None + ok = await sender.send(pkt, addr=m) + return m, bool(ok), None if ok else "send failed" except Exception as e: return m, False, str(e) diff --git a/src/controllers/preset.py b/src/controllers/preset.py index ef2c34e..39861ee 100644 --- a/src/controllers/preset.py +++ b/src/controllers/preset.py @@ -7,6 +7,7 @@ from models.device import Device, normalize_mac from models.transport import get_current_sender from util.driver_delivery import deliver_json_messages, deliver_preset_broadcast_then_per_device from util.espnow_message import build_message, build_preset_dict +from util.binary_driver_messages import build_preset_cmd_chunks from util.profile_bundle import export_preset_bundle, import_preset_bundle import json @@ -225,39 +226,13 @@ async def send_presets(request, session): if not sender: return json.dumps({"error": "Transport not configured"}), 503, {'Content-Type': 'application/json'} - MAX_BYTES = 240 send_delay_s = 0.1 - entries = list(presets_by_name.items()) - total_presets = len(entries) - - batch = {} - chunk_messages = [] - for name, preset_obj in entries: - test_batch = dict(batch) - test_batch[name] = preset_obj - test_msg = build_message(presets=test_batch, save=save_flag, default=default_id) - size = len(test_msg) - - if size <= MAX_BYTES or not batch: - batch = test_batch - else: - chunk_messages.append( - build_message( - presets=dict(batch), - save=False, - default=None, - ) - ) - batch = {name: preset_obj} - - if batch: - chunk_messages.append( - build_message( - presets=dict(batch), - save=save_flag, - default=default_id, - ) - ) + total_presets = len(presets_by_name) + chunk_messages = build_preset_cmd_chunks( + presets_by_name, + save=save_flag, + default=str(default_id) if default_id is not None else None, + ) target_list = None raw_targets = data.get("targets") diff --git a/src/controllers/settings.py b/src/controllers/settings.py index 7e10093..40da5b0 100644 --- a/src/controllers/settings.py +++ b/src/controllers/settings.py @@ -3,7 +3,6 @@ import json from microdot import Microdot, send_file -from models import wifi_ws_clients from settings import get_settings controller = Microdot() @@ -108,13 +107,6 @@ async def update_settings(request): else: settings[key] = value settings.save() - if global_brightness_changed: - try: - asyncio.get_running_loop().create_task( - wifi_ws_clients.broadcast_global_brightness_to_tcp_drivers() - ) - except RuntimeError: - pass return json.dumps({"message": "Settings updated successfully"}), 200, {'Content-Type': 'application/json'} except ValueError as e: return json.dumps({"error": str(e)}), 400 diff --git a/src/main.py b/src/main.py index f97edc5..279d0de 100644 --- a/src/main.py +++ b/src/main.py @@ -4,9 +4,6 @@ import json import os import secrets import signal -import socket -import threading -import traceback from microdot import Microdot, send_file from microdot.websocket import with_websocket from microdot.session import Session @@ -24,171 +21,36 @@ import controllers.settings as settings_controller import controllers.device as device_controller import controllers.led_tool as led_tool_controller from models.transport import get_sender, set_sender, get_current_sender -from models.device import Device, normalize_mac -from models import wifi_ws_clients as tcp_client_registry -from util.device_status_broadcaster import ( - broadcast_device_tcp_snapshot_to, - broadcast_device_tcp_status, - register_device_status_ws, - unregister_device_status_ws, -) +from models.device import Device +from models.bridge_ws_client import init_bridge_client +from util.espnow_registry import handle_espnow_announce +from util.binary_driver_messages import v1_dict_to_cmd_packet from util.audio_detector import AudioBeatDetector -_tcp_device_lock = threading.Lock() - -DISCOVERY_UDP_PORT = 8766 - def _live_reload_enabled() -> bool: v = os.environ.get("LED_CONTROLLER_LIVE_RELOAD", "").strip().lower() return v not in ("", "0", "false", "no") -def _register_udp_device_sync( - device_name: str, peer_ip: str, mac, device_type=None -) -> None: - with _tcp_device_lock: - try: - d = Device() - did, persisted = d.upsert_wifi_tcp_client( - device_name, peer_ip, mac, device_type=device_type - ) - if did and persisted: - print( - f"UDP device registered: mac={did} name={device_name!r} ip={peer_ip!r}" - ) - except Exception as e: - print(f"UDP device registry failed: {e}") - traceback.print_exception(type(e), e, e.__traceback__) - - -async def _handle_udp_discovery(sock, udp_holder=None) -> None: - while True: - try: - data, addr = await asyncio.get_running_loop().sock_recvfrom(sock, 2048) - except asyncio.CancelledError: - raise - except OSError as e: - if udp_holder and udp_holder.get("closing"): - break - print(f"[UDP] recv failed: {e!r}") - continue - except Exception as e: - print(f"[UDP] recv failed: {e!r}") - continue - peer_ip = addr[0] if addr else "" - line = data.split(b"\n", 1)[0].strip() - if line: - try: - parsed = json.loads(line.decode("utf-8")) - if isinstance(parsed, dict): - dns = str(parsed.get("device_name") or "").strip() - mac = parsed.get("mac") or parsed.get("device_mac") or parsed.get( - "sta_mac" - ) - device_type = parsed.get("type") or parsed.get("device_type") - if dns and normalize_mac(mac): - _register_udp_device_sync(dns, peer_ip, mac, device_type) - if str(parsed.get("v") or "") == "1": - tcp_client_registry.ensure_driver_connection(peer_ip) - except (UnicodeError, ValueError, TypeError): - pass - try: - await asyncio.get_running_loop().sock_sendto(sock, data, addr) - except Exception as e: - print(f"[UDP] echo send failed: {e!r}") - - -def _prime_wifi_outbound_driver_connections() -> None: - """On boot, dial each registered Wi-Fi driver (same 4-attempt limit as UDP hello).""" - n = 0 - try: - dev = Device() - for mac_key, doc in list(dev.items()): - if not isinstance(doc, dict): - continue - if doc.get("transport") != "wifi": - continue - ip = _ipv4_address(str(doc.get("address") or "")) - if not ip: - continue - tcp_client_registry.ensure_driver_connection(ip) - n += 1 - except Exception as e: - print(f"[startup] Wi-Fi driver connection prime failed: {e!r}") - traceback.print_exception(type(e), e, e.__traceback__) - return - if n: - print(f"[startup] primed outbound WebSocket for {n} Wi-Fi driver(s)") - - -def _ipv4_address(addr: str) -> str | None: - """Return dotted IPv4 string or None (hostnames skipped for UDP nudge).""" - s = (addr or "").strip() - if not s: - return None - parts = s.split(".") - if len(parts) != 4: - return None - try: - nums = [int(p) for p in parts] - except ValueError: - return None - if not all(0 <= n <= 255 for n in nums): - return None - return s - - -async def _run_udp_discovery_server(udp_holder=None) -> None: - sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) - sock.setblocking(False) - try: - sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) - except (AttributeError, OSError): - pass - try: - sock.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1) - except (AttributeError, OSError): - pass - sock.bind(("0.0.0.0", DISCOVERY_UDP_PORT)) - if udp_holder is not None: - udp_holder["sock"] = sock - print(f"UDP discovery listening on 0.0.0.0:{DISCOVERY_UDP_PORT}") - try: - await _handle_udp_discovery(sock, udp_holder) - finally: - if udp_holder is not None: - udp_holder.pop("sock", None) - try: - sock.close() - except Exception: - pass - - -async def _send_bridge_wifi_channel(settings, sender): - """Tell the serial ESP32 bridge to set STA channel (settings wifi_channel); not forwarded as ESP-NOW.""" - try: - ch = int(settings.get("wifi_channel", 6)) - except (TypeError, ValueError): - ch = 6 - ch = max(1, min(11, ch)) - payload = json.dumps({"m": "bridge", "ch": ch}, separators=(",", ":")) - try: - await sender.send(payload, addr="ffffffffffff") - print(f"[startup] bridge Wi-Fi channel -> {ch}") - except Exception as e: - print(f"[startup] bridge channel message failed: {e}") - - async def main(port=80): settings = get_settings() print(settings) print("Starting") - # Initialize transport (serial to ESP32 bridge) sender = get_sender(settings) set_sender(sender) + bridge_url = str(settings.get("bridge_ws_url") or "").strip() + if bridge_url: + try: + ch = int(settings.get("wifi_channel", 6)) + except (TypeError, ValueError): + ch = 6 + bridge = init_bridge_client(bridge_url, wifi_channel=ch) + bridge.set_uplink_handler(handle_espnow_announce) + bridge.start() + app = Microdot() audio_detector = AudioBeatDetector() try: @@ -243,9 +105,6 @@ async def main(port=80): app.mount(device_controller.controller, '/devices') app.mount(led_tool_controller.controller, '/led-tool') - tcp_client_registry.set_settings(settings) - tcp_client_registry.set_tcp_status_broadcaster(broadcast_device_tcp_status) - live_reload = _live_reload_enabled() dev_build_id = secrets.token_hex(12) if live_reload else None if live_reload: @@ -408,56 +267,35 @@ async def main(port=80): @app.route('/ws') @with_websocket async def ws(request, ws): - await register_device_status_ws(ws) - await broadcast_device_tcp_snapshot_to(ws) try: while True: data = await ws.receive() - print(data) - if data: - try: - parsed = json.loads(data) - print("WS received JSON:", parsed) - # Optional "to": 12-char hex MAC; rest is payload (sent with that address). - addr = parsed.pop("to", None) - payload = json.dumps(parsed) if parsed else data - await sender.send(payload, addr=addr) - except json.JSONDecodeError: - # Not JSON: send raw with default address - try: - await sender.send(data) - except Exception: - try: - await ws.send(json.dumps({"error": "Send failed"})) - except Exception: - pass - except Exception: - try: - await ws.send(json.dumps({"error": "Send failed"})) - except Exception: - pass - else: + if not data: break - finally: - await unregister_device_status_ws(ws) + try: + if isinstance(data, (bytes, bytearray)): + await sender.send(bytes(data)) + continue + parsed = json.loads(data) + addr = parsed.pop("to", None) + pkt = v1_dict_to_cmd_packet(parsed) + await sender.send(pkt, addr=addr) + except json.JSONDecodeError: + pass + except Exception: + try: + await ws.send(json.dumps({"error": "Send failed"})) + except Exception: + pass + except Exception: + pass - - - # Touch Device singleton early so db/device.json exists before first UDP hello. Device() - await _send_bridge_wifi_channel(settings, sender) - _prime_wifi_outbound_driver_connections() - - udp_holder = {"closing": False, "shutting_down": False} loop = asyncio.get_running_loop() server_tasks: list[asyncio.Task] = [] def _graceful_shutdown(*_args): - if udp_holder.get("shutting_down"): - raise SystemExit(0) - udp_holder["shutting_down"] = True print("[server] shutting down...") - udp_holder["closing"] = True try: audio_detector.stop() except Exception: @@ -472,13 +310,6 @@ async def main(port=80): t.cancel() except Exception: pass - u = udp_holder.get("sock") - if u is not None: - try: - u.close() - except OSError: - pass - tcp_client_registry.cancel_all_driver_tasks() if getattr(app, "server", None) is not None: try: app.shutdown() @@ -497,15 +328,11 @@ async def main(port=80): except (NotImplementedError, RuntimeError): pass - # Await HTTP + UDP discovery; bind failures (e.g. port 80 in use) surface here. try: server_tasks[:] = [ asyncio.create_task( app.start_server(host="0.0.0.0", port=port), name="http" ), - asyncio.create_task( - _run_udp_discovery_server(udp_holder), name="udp" - ), ] await asyncio.gather(*server_tasks) except asyncio.CancelledError: @@ -534,7 +361,6 @@ async def main(port=80): app.server = None except Exception: pass - udp_holder["closing"] = True for t in list(server_tasks): if not t.done(): t.cancel() diff --git a/src/models/bridge_ws_client.py b/src/models/bridge_ws_client.py new file mode 100644 index 0000000..cc920e4 --- /dev/null +++ b/src/models/bridge_ws_client.py @@ -0,0 +1,142 @@ +"""Persistent WebSocket client to the ESP-NOW bridge (binary frames).""" + +from __future__ import annotations + +import asyncio +from typing import Awaitable, Callable, Optional + +import websockets +from websockets.exceptions import ConnectionClosed + +from util.espnow_wire import ( + MSG_ANNOUNCE, + WIRE_MAGIC, + pack_bridge_channel, + pack_ws_downlink, + parse_ws_frame, + wire_msg_type, +) + +UplinkHandler = Callable[[bytes, bytes], Awaitable[None]] + + +class BridgeWsClient: + def __init__(self, url: str, *, wifi_channel: int = 6): + self._url = url.strip() + self._wifi_channel = wifi_channel + self._ws: Optional[websockets.WebSocketClientProtocol] = None + self._send_lock = asyncio.Lock() + self._uplink_handler: Optional[UplinkHandler] = None + self._task: Optional[asyncio.Task] = None + self._connected = asyncio.Event() + self._ack_waiter: Optional[asyncio.Future] = None + + def set_uplink_handler(self, handler: Optional[UplinkHandler]) -> None: + self._uplink_handler = handler + + async def run_forever(self) -> None: + while True: + try: + await self._connect_once() + except asyncio.CancelledError: + raise + except Exception as e: + print(f"[bridge] connection error: {e!r}") + self._connected.clear() + self._ws = None + await asyncio.sleep(2.0) + + async def _reader_loop(self) -> None: + ws = self._ws + if ws is None: + return + try: + async for message in ws: + if isinstance(message, str): + continue + if len(message) == 1: + fut = self._ack_waiter + if fut is not None and not fut.done(): + fut.set_result(message[0] == 0x01) + continue + try: + peer, pkt, _bcast = parse_ws_frame(message) + except ValueError: + continue + if wire_msg_type(pkt) == MSG_ANNOUNCE and self._uplink_handler: + await self._uplink_handler(peer, pkt) + except ConnectionClosed: + pass + + async def _connect_once(self) -> None: + print(f"[bridge] connecting to {self._url}") + async with websockets.connect(self._url, ping_interval=20, ping_timeout=20) as ws: + self._ws = ws + ch_pkt = pack_bridge_channel(self._wifi_channel) + await ws.send(pack_ws_downlink(ch_pkt, broadcast=True)) + self._connected.set() + print("[bridge] connected") + reader = asyncio.create_task(self._reader_loop()) + try: + while True: + await asyncio.sleep(3600) + finally: + reader.cancel() + try: + await reader + except asyncio.CancelledError: + pass + + async def wait_connected(self, timeout: float = 30.0) -> bool: + try: + await asyncio.wait_for(self._connected.wait(), timeout=timeout) + return True + except asyncio.TimeoutError: + return False + + async def send_frame(self, frame: bytes) -> bool: + await self._connected.wait() + ws = self._ws + if ws is None: + return False + async with self._send_lock: + loop = asyncio.get_running_loop() + self._ack_waiter = loop.create_future() + try: + await ws.send(frame) + return bool(await asyncio.wait_for(self._ack_waiter, timeout=5.0)) + except (ConnectionClosed, asyncio.TimeoutError, OSError) as e: + print(f"[bridge] send failed: {e!r}") + return False + finally: + self._ack_waiter = None + + async def send_espnow( + self, + packet: bytes, + *, + peer_mac: Optional[str] = None, + broadcast: bool = False, + ) -> bool: + if not packet or packet[0] != WIRE_MAGIC: + raise ValueError("packet must be espnow wire format") + frame = pack_ws_downlink(packet, peer_mac=peer_mac, broadcast=broadcast) + return await self.send_frame(frame) + + def start(self) -> asyncio.Task: + if self._task is None or self._task.done(): + self._task = asyncio.create_task(self.run_forever()) + return self._task + + +_client: Optional[BridgeWsClient] = None + + +def get_bridge_client() -> Optional[BridgeWsClient]: + return _client + + +def init_bridge_client(url: str, *, wifi_channel: int = 6) -> BridgeWsClient: + global _client + _client = BridgeWsClient(url, wifi_channel=wifi_channel) + return _client diff --git a/src/models/device.py b/src/models/device.py index 81abe14..4442540 100644 --- a/src/models/device.py +++ b/src/models/device.py @@ -256,6 +256,68 @@ class Device(Model): def list(self): return list(self.keys()) + def upsert_espnow_announced( + self, + mac, + device_name, + *, + device_type="led", + num_leds=None, + color_order=None, + startup_mode=None, + brightness=None, + ): + """ + Register or update an ESP-NOW device from a binary ANNOUNCE. + + Returns ``(mac_hex | None, persisted)``. + """ + mac_hex = normalize_mac(mac) + if not mac_hex: + return None, False + name = (device_name or "").strip() + if not name: + return None, False + resolved_type = validate_device_type(device_type) + meta = {} + if num_leds is not None: + meta["num_leds"] = int(num_leds) + if color_order is not None: + meta["color_order"] = str(color_order) + if startup_mode is not None: + meta["startup_mode"] = str(startup_mode) + if brightness is not None: + meta["brightness"] = int(brightness) + + if mac_hex in self: + prev = self[mac_hex] + merged = dict(prev) + merged["name"] = name + merged["type"] = resolved_type + merged["transport"] = "espnow" + merged["address"] = mac_hex + merged["id"] = mac_hex + merged.update({k: v for k, v in meta.items() if v is not None}) + if merged == prev: + return mac_hex, False + self[mac_hex] = merged + self.save() + return mac_hex, True + + row = { + "id": mac_hex, + "name": name, + "type": resolved_type, + "transport": "espnow", + "address": mac_hex, + "default_pattern": None, + "zones": [], + } + row.update({k: v for k, v in meta.items() if v is not None}) + self[mac_hex] = row + self.save() + return mac_hex, True + def upsert_wifi_tcp_client(self, device_name, peer_ip, mac, device_type=None): """ Register or update a Wi-Fi client by **MAC** (storage id). Updates **name**, diff --git a/src/models/transport.py b/src/models/transport.py index 236bee6..fbb4668 100644 --- a/src/models/transport.py +++ b/src/models/transport.py @@ -1,59 +1,53 @@ +"""Transport to LED drivers via ESP-NOW bridge WebSocket.""" + import asyncio -import json +from typing import Optional, Union + +from models.bridge_ws_client import get_bridge_client +from util.espnow_wire import WIRE_MAGIC, pack_ws_downlink + +BROADCAST_MAC_HEX = "ffffffffffff" -# Default: broadcast (6 bytes). Pi always sends 6-byte address + payload to ESP32. -BROADCAST_MAC = bytes.fromhex("ffffffffffff") - - -def _encode_payload(data): - if isinstance(data, str): - return data.encode() - if isinstance(data, dict): - return json.dumps(data).encode() - return data - - -def _parse_mac(addr): - """Convert 12-char hex string or 6-byte bytes to 6-byte MAC.""" - if addr is None or addr == b"": - return BROADCAST_MAC +def _parse_mac(addr) -> Optional[bytes]: + if addr is None or addr == "": + return None if isinstance(addr, bytes) and len(addr) == 6: return addr - if isinstance(addr, str) and len(addr) == 12: - return bytes.fromhex(addr) - return BROADCAST_MAC - - -async def _to_thread(func, *args): - to_thread = getattr(asyncio, "to_thread", None) - if to_thread: - return await to_thread(func, *args) - loop = asyncio.get_event_loop() - return await loop.run_in_executor(None, func, *args) + if isinstance(addr, str): + s = addr.strip().lower().replace(":", "").replace("-", "") + if len(s) == 12: + return bytes.fromhex(s) + return None class NullSender: - """Used when no ESP-NOW UART bridge is configured or the port cannot be opened.""" + """No bridge configured.""" async def send(self, data, addr=None): return True -class SerialSender: - def __init__(self, port, baudrate, default_addr=None): - import serial +class BridgeWsSender: + """Send binary ESP-NOW packets via bridge WebSocket client.""" - self._serial = serial.Serial(port, baudrate=baudrate, timeout=1) - self._default_addr = _parse_mac(default_addr) - self._write_lock = asyncio.Lock() - - async def send(self, data, addr=None): - mac = _parse_mac(addr) if addr is not None else self._default_addr - payload = _encode_payload(data) - async with self._write_lock: - await _to_thread(self._serial.write, mac + payload) - return True + async def send(self, data: Union[bytes, str, dict], addr=None) -> bool: + client = get_bridge_client() + if client is None: + return False + if isinstance(data, (bytes, bytearray)): + packet = bytes(data) + else: + return False + if not packet or packet[0] != WIRE_MAGIC: + return False + peer = _parse_mac(addr) + broadcast = peer is None or addr == BROADCAST_MAC_HEX + return await client.send_espnow( + packet, + peer_mac=peer, + broadcast=broadcast, + ) _current_sender = None @@ -69,22 +63,11 @@ def get_current_sender(): def get_sender(settings): - # Serial ESP-NOW bridge is opt-in (serial_enabled true); default off for dev / Wi-Fi-only. - if not settings.get("serial_enabled"): - print("[startup] serial bridge disabled (set serial_enabled true in settings.json to enable)") - return NullSender() - port = settings.get("serial_port", "/dev/ttyS0") - raw_port = str(port).strip() if port is not None else "" - if not raw_port or raw_port.lower() in ("none", "off"): - print("[startup] serial bridge disabled (empty serial_port)") - return NullSender() - baudrate = settings.get("serial_baudrate", 912000) - default_addr = settings.get("serial_destination_mac", "ffffffffffff") - try: - return SerialSender(raw_port, baudrate, default_addr=default_addr) - except Exception as e: + url = str(settings.get("bridge_ws_url") or "").strip() + if not url: print( - f"[startup] serial open failed ({raw_port!r}): {e}; " - "continuing without ESP-NOW bridge (Wi-Fi drivers unchanged)" + "[startup] bridge disabled (set bridge_ws_url in settings.json, e.g. ws://192.168.4.1/ws)" ) return NullSender() + print(f"[startup] ESP-NOW via bridge WebSocket {url!r}") + return BridgeWsSender() diff --git a/src/settings.py b/src/settings.py index 60882cc..890d304 100644 --- a/src/settings.py +++ b/src/settings.py @@ -52,31 +52,9 @@ class Settings(dict): # ESP-NOW STA channel (2.4 GHz) for LED drivers / bridge alignment; 1–11 if 'wifi_channel' not in self: self['wifi_channel'] = 6 - # Wi-Fi LED drivers: controller opens WebSocket to device (firmware serves /ws) - if 'wifi_driver_ws_port' not in self: - self['wifi_driver_ws_port'] = 80 - if 'wifi_driver_ws_path' not in self: - self['wifi_driver_ws_path'] = '/ws' - # Legacy (unused): periodic UDP nudges removed; connect only on driver hello. - if 'wifi_driver_hello_interval_s' not in self: - self['wifi_driver_hello_interval_s'] = 0 - if 'wifi_driver_connect_retry_window_s' not in self: - self['wifi_driver_connect_retry_window_s'] = 120.0 - # Spread outbound dials 0..N s by device IP so six+ drivers do not all hit the AP at once. - if 'wifi_driver_connect_stagger_max_s' not in self: - self['wifi_driver_connect_stagger_max_s'] = 2.5 - # TCP/WebSocket open timeout per attempt (seconds). - if 'wifi_driver_ws_open_timeout' not in self: - self['wifi_driver_ws_open_timeout'] = 45.0 - # Pause between outbound WebSocket dial attempts (seconds). - if 'wifi_driver_connect_retry_interval_s' not in self: - self['wifi_driver_connect_retry_interval_s'] = 2.0 - # Outbound WebSocket dial attempts per driver UDP hello (then wait for next hello). - if 'wifi_driver_initial_connect_attempts' not in self: - self['wifi_driver_initial_connect_attempts'] = 4 - # UART to ESP32 ESP-NOW bridge; default off (Wi-Fi drivers need no serial). - if 'serial_enabled' not in self: - self['serial_enabled'] = False + # WebSocket URL of ESP-NOW bridge (Pi is client), e.g. ws://192.168.4.1/ws + if 'bridge_ws_url' not in self: + self['bridge_ws_url'] = '' # Zone UI global brightness (0–255); shared across browsers/devices. if 'global_brightness' not in self: self['global_brightness'] = 255 @@ -91,9 +69,10 @@ class Settings(dict): def save(self): try: - j = json.dumps(self) + j = json.dumps(self, indent=2, sort_keys=True) with open(self.SETTINGS_FILE, 'w') as file: file.write(j) + file.write("\n") if not getattr(self, "_quiet", False): print("Settings saved successfully.") except Exception as e: diff --git a/src/util/binary_driver_messages.py b/src/util/binary_driver_messages.py new file mode 100644 index 0000000..90b5de9 --- /dev/null +++ b/src/util/binary_driver_messages.py @@ -0,0 +1,62 @@ +"""Build binary ESP-NOW CMD / GROUP_CMD packets from preset/select data.""" + +from __future__ import annotations + +from typing import Any, Dict, List, Optional + +from util.binary_envelope import pack_binary_envelope_v2 +from util.espnow_wire import MAX_ESPNOW_PAYLOAD, pack_cmd, pack_group_cmd + + +def v1_dict_to_cmd_packet(body: Dict[str, Any]) -> bytes: + save = bool(body.get("save")) + kw: Dict[str, Any] = {} + if "presets" in body: + kw["presets"] = body["presets"] + if "select" in body: + kw["select"] = body["select"] + if "default" in body: + kw["default"] = body["default"] + kw["default_targets"] = body.get("targets") + if "b" in body: + kw["brightness_0_255"] = int(body["b"]) + return pack_cmd(pack_binary_envelope_v2(**kw), save=save) + + +def build_preset_cmd_chunks( + presets_by_name: Dict[str, Any], + *, + save: bool = False, + default: Optional[str] = None, + max_payload: int = MAX_ESPNOW_PAYLOAD, +) -> List[bytes]: + """Chunk presets into CMD packets each ≤ max_payload bytes.""" + entries = list(presets_by_name.items()) + chunks: List[bytes] = [] + batch: Dict[str, Any] = {} + + def _packet_for(presets_map: Dict[str, Any], *, final_save: bool, def_id: Optional[str]): + kw: Dict[str, Any] = {"presets": presets_map} + if def_id is not None: + kw["default"] = def_id + return pack_cmd(pack_binary_envelope_v2(**kw), save=final_save) + + for name, preset_obj in entries: + trial = dict(batch) + trial[name] = preset_obj + try: + pkt = _packet_for(trial, final_save=False, def_id=None) + except ValueError: + pkt = b"\xff\xff" + if len(pkt) <= max_payload or not batch: + batch = trial + else: + chunks.append(_packet_for(batch, final_save=False, def_id=None)) + batch = {name: preset_obj} + + if batch: + chunks.append( + _packet_for(batch, final_save=save, def_id=str(default) if default else None), + ) + + return [c for c in chunks if c and c[0] == 0x4C] diff --git a/src/util/device_status_broadcaster.py b/src/util/device_status_broadcaster.py index b6d5d4a..58561ae 100644 --- a/src/util/device_status_broadcaster.py +++ b/src/util/device_status_broadcaster.py @@ -1,52 +1,22 @@ -"""Push Wi-Fi driver connect/disconnect updates to browser WebSocket clients.""" +"""Device status WebSocket broadcasts (ESP-NOW has no live TCP session).""" +import asyncio import json -import threading -from typing import Any, Set -# Threading lock: safe across asyncio tasks and avoids binding asyncio.Lock to the wrong loop. -_clients_lock = threading.Lock() -_clients: Set[Any] = set() +_ws_clients: set = set() -async def register_device_status_ws(ws: Any) -> None: - with _clients_lock: - _clients.add(ws) +async def register_device_status_ws(ws): + _ws_clients.add(ws) -async def unregister_device_status_ws(ws: Any) -> None: - with _clients_lock: - _clients.discard(ws) +async def unregister_device_status_ws(ws): + _ws_clients.discard(ws) -async def broadcast_device_tcp_status(ip: str, connected: bool) -> None: - from models.wifi_ws_clients import normalize_tcp_peer_ip - - ip = normalize_tcp_peer_ip(ip) - if not ip: - return - msg = json.dumps({"type": "device_tcp", "ip": ip, "connected": bool(connected)}) - with _clients_lock: - targets = list(_clients) - dead = [] - for ws in targets: - try: - await ws.send(msg) - except Exception as exc: - dead.append(ws) - print(f"[device_status_broadcaster] ws.send failed: {exc!r}") - if dead: - with _clients_lock: - for ws in dead: - _clients.discard(ws) +async def broadcast_device_tcp_snapshot_to(ws): + await ws.send(json.dumps({"type": "device_tcp_snapshot", "devices": {}})) -async def broadcast_device_tcp_snapshot_to(ws: Any) -> None: - from models import wifi_ws_clients as tcp - - ips = tcp.list_connected_ips() - msg = json.dumps({"type": "device_tcp_snapshot", "connected_ips": ips}) - try: - await ws.send(msg) - except Exception as exc: - print(f"[device_status_broadcaster] snapshot send failed: {exc!r}") +async def broadcast_device_tcp_status(mac: str, connected: bool): + pass diff --git a/src/util/driver_delivery.py b/src/util/driver_delivery.py index a02eb64..19cc6e3 100644 --- a/src/util/driver_delivery.py +++ b/src/util/driver_delivery.py @@ -1,70 +1,73 @@ -"""Deliver driver JSON messages over serial (ESP-NOW) and/or WebSocket (Wi-Fi drivers).""" +"""Deliver binary ESP-NOW messages via bridge WebSocket.""" import asyncio -import json +from typing import List, Optional, Union from models.device import normalize_mac -from models.wifi_ws_clients import send_json_line_to_ip +from util.binary_driver_messages import build_preset_cmd_chunks, v1_dict_to_cmd_packet +from util.espnow_wire import BROADCAST_MAC, pack_group_cmd -# Serial bridge (ESP32): broadcast MAC + this envelope → firmware unicasts ``body`` to each peer. -_SPLIT_MODE = "split" -_BROADCAST_MAC_HEX = "ffffffffffff" +_BROADCAST_HEX = "ffffffffffff" -def _split_serial_envelope(inner_json_str, peer_hex_list): - """One UART frame: broadcast dest + JSON {m:split, peers:[hex,...], body:}.""" - body = json.loads(inner_json_str) - env = {"m": _SPLIT_MODE, "peers": list(peer_hex_list), "body": body} - return json.dumps(env, separators=(",", ":")) +async def deliver_binary_packets( + sender, + packets: List[bytes], + target_macs: Optional[List[str]] = None, + *, + delay_s: float = 0.1, +) -> int: + """Send binary CMD packets unicast per MAC or broadcast when no targets.""" + if not packets: + return 0 + deliveries = 0 + if not target_macs: + for pkt in packets: + if await sender.send(pkt, addr=_BROADCAST_HEX): + deliveries += 1 + await asyncio.sleep(delay_s) + return deliveries + + seen = set() + ordered: List[str] = [] + for raw in target_macs: + m = normalize_mac(str(raw)) if raw else None + if not m or m in seen: + continue + seen.add(m) + ordered.append(m) + + for pkt in packets: + for mac in ordered: + if await sender.send(pkt, addr=mac): + deliveries += 1 + await asyncio.sleep(delay_s) + return deliveries -def _wifi_message_for_device(msg, device_name): - """ - For Wi-Fi WebSocket fanout, narrow a v1 select map to a single device name. - Returns the original message when no narrowing applies. - """ - if not device_name: - return msg - try: - body = json.loads(msg) - except Exception: - return msg - if not isinstance(body, dict): - return msg - select = body.get("select") - if not isinstance(select, dict): - return msg - if device_name not in select: - return msg - body["select"] = {device_name: select[device_name]} - return json.dumps(body, separators=(",", ":")) +async def deliver_group_binary_packets( + sender, + group_id: str, + packets: List[bytes], + *, + delay_s: float = 0.1, +) -> int: + """Broadcast GROUP_CMD packets (one ESP-NOW send per packet).""" + from util.espnow_wire import parse_cmd - -def _combine_preset_chunks_for_wifi(chunk_messages): - """Merge chunked v1 preset messages into one v1 JSON string for Wi-Fi.""" - merged_presets = {} - save_flag = False - default_id = None - for msg in chunk_messages: + deliveries = 0 + for pkt in packets: + env, save = parse_cmd(pkt) + if env is None: + continue try: - body = json.loads(msg) - except Exception: + g_pkt = pack_group_cmd(str(group_id), env, save=save) + except ValueError: continue - if not isinstance(body, dict): - continue - presets = body.get("presets") - if isinstance(presets, dict): - merged_presets.update(presets) - if body.get("save"): - save_flag = True - if body.get("default") is not None: - default_id = body.get("default") - out = {"v": "1", "presets": merged_presets} - if save_flag: - out["save"] = True - if default_id is not None: - out["default"] = default_id - return json.dumps(out, separators=(",", ":")) + if await sender.send(g_pkt, addr=_BROADCAST_HEX): + deliveries += 1 + await asyncio.sleep(delay_s) + return deliveries async def deliver_preset_broadcast_then_per_device( @@ -76,11 +79,24 @@ async def deliver_preset_broadcast_then_per_device( delay_s=0.1, ): """ - Send preset definition chunks: ESP-NOW broadcast once per chunk; same chunk to each - Wi-Fi driver over WebSocket. If default_id is set, send a per-target default message - (unicast serial or WebSocket) with targets=[device name] for each registry entry. + chunk_messages: list of v1 JSON strings OR binary CMD bytes. + Converts JSON strings to binary when needed. """ - if not chunk_messages: + packets: List[bytes] = [] + for msg in chunk_messages: + if isinstance(msg, (bytes, bytearray)): + packets.append(bytes(msg)) + else: + import json + + try: + body = json.loads(msg) + except Exception: + continue + if isinstance(body, dict): + packets.append(v1_dict_to_cmd_packet(body)) + + if not packets: return 0 seen = set() @@ -92,30 +108,9 @@ async def deliver_preset_broadcast_then_per_device( seen.add(m) ordered.append(m) - wifi_ips = [] - for mac in ordered: - doc = devices_model.read(mac) - if doc and doc.get("transport") == "wifi" and doc.get("address"): - wifi_ips.append(str(doc["address"]).strip()) - - deliveries = 0 - wifi_combined_msg = _combine_preset_chunks_for_wifi(chunk_messages) - for msg in chunk_messages: - tasks = [sender.send(msg, addr=_BROADCAST_MAC_HEX)] - results = await asyncio.gather(*tasks, return_exceptions=True) - if results and results[0] is True: - deliveries += 1 - await asyncio.sleep(delay_s) - - for ip in wifi_ips: - if not ip: - continue - try: - if await send_json_line_to_ip(ip, wifi_combined_msg): - deliveries += 1 - except Exception as e: - print(f"[driver_delivery] Wi-Fi preset send failed: {e!r}") - await asyncio.sleep(delay_s) + deliveries = await deliver_binary_packets( + sender, packets, ordered, delay_s=delay_s + ) if default_id: did = str(default_id) @@ -123,20 +118,9 @@ async def deliver_preset_broadcast_then_per_device( doc = devices_model.read(mac) or {} name = str(doc.get("name") or "").strip() or mac body = {"v": "1", "default": did, "save": True, "targets": [name]} - out = json.dumps(body, separators=(",", ":")) - if doc.get("transport") == "wifi" and doc.get("address"): - ip = str(doc["address"]).strip() - try: - if await send_json_line_to_ip(ip, out): - deliveries += 1 - except Exception as e: - print(f"[driver_delivery] default Wi-Fi send failed: {e!r}") - else: - try: - await sender.send(out, addr=mac) - deliveries += 1 - except Exception as e: - print(f"[driver_delivery] default serial failed: {e!r}") + pkt = v1_dict_to_cmd_packet(body) + if await sender.send(pkt, addr=mac): + deliveries += 1 await asyncio.sleep(delay_s) return deliveries @@ -144,26 +128,29 @@ async def deliver_preset_broadcast_then_per_device( async def deliver_json_messages(sender, messages, target_macs, devices_model, delay_s=0.1): """ - Send each message string to the bridge and/or Wi-Fi WebSocket clients. - - If target_macs is None or empty: one serial send per message (default/broadcast address). - Otherwise: Wi-Fi uses WebSocket in parallel. Multiple ESP-NOW peers are sent in **one** serial - write to the ESP32 (broadcast + split envelope); the bridge unicasts ``body`` to each - peer. A single ESP-NOW peer still uses one unicast serial frame. Wi-Fi and serial - tasks run together in one asyncio.gather. - - Returns (delivery_count, chunk_count) where chunk_count is len(messages). + Convert v1 JSON message strings to binary CMD packets and deliver. + Returns (delivery_count, chunk_count). """ - if not messages: + packets: List[bytes] = [] + import json + + for msg in messages: + if isinstance(msg, (bytes, bytearray)): + packets.append(bytes(msg)) + continue + try: + body = json.loads(msg) + except Exception: + continue + if isinstance(body, dict): + packets.append(v1_dict_to_cmd_packet(body)) + + if not packets: return 0, 0 if not target_macs: - deliveries = 0 - for msg in messages: - await sender.send(msg) - deliveries += 1 - await asyncio.sleep(delay_s) - return deliveries, len(messages) + n = await deliver_binary_packets(sender, packets, None, delay_s=delay_s) + return n, len(packets) seen = set() ordered_macs = [] @@ -174,51 +161,5 @@ async def deliver_json_messages(sender, messages, target_macs, devices_model, de seen.add(m) ordered_macs.append(m) - deliveries = 0 - for msg in messages: - wifi_tasks = [] - espnow_hex = [] - for mac in ordered_macs: - doc = devices_model.read(mac) - if doc and doc.get("transport") == "wifi": - ip = doc.get("address") - if ip: - name = str(doc.get("name") or "").strip() - wifi_msg = _wifi_message_for_device(msg, name) - wifi_tasks.append(send_json_line_to_ip(ip, wifi_msg)) - else: - espnow_hex.append(mac) - - tasks = [] - espnow_peer_count = 0 - if len(espnow_hex) > 1: - tasks.append( - sender.send( - _split_serial_envelope(msg, espnow_hex), - addr=_BROADCAST_MAC_HEX, - ) - ) - espnow_peer_count = len(espnow_hex) - elif len(espnow_hex) == 1: - tasks.append(sender.send(msg, addr=espnow_hex[0])) - espnow_peer_count = 1 - - tasks.extend(wifi_tasks) - - if tasks: - results = await asyncio.gather(*tasks, return_exceptions=True) - n_serial = len(tasks) - len(wifi_tasks) - for i, r in enumerate(results): - if i < n_serial: - if r is True: - deliveries += espnow_peer_count - elif isinstance(r, Exception): - print(f"[driver_delivery] serial delivery failed: {r!r}") - else: - if r is True: - deliveries += 1 - elif isinstance(r, Exception): - print(f"[driver_delivery] Wi-Fi delivery failed: {r!r}") - - await asyncio.sleep(delay_s) - return deliveries, len(messages) + n = await deliver_binary_packets(sender, packets, ordered_macs, delay_s=delay_s) + return n, len(packets) diff --git a/src/util/espnow_registry.py b/src/util/espnow_registry.py new file mode 100644 index 0000000..93abc86 --- /dev/null +++ b/src/util/espnow_registry.py @@ -0,0 +1,70 @@ +"""Handle ESP-NOW ANNOUNCE uplink and push GROUPS to drivers.""" + +from __future__ import annotations + +from models.device import Device, normalize_mac # noqa: F401 — re-export for callers +from models.group import Group +from models.bridge_ws_client import get_bridge_client +from util.espnow_wire import mac_bytes_to_hex, pack_groups, parse_announce +from util.groups_for_device import groups_for_mac + + +async def handle_espnow_announce(peer_mac: bytes, packet: bytes) -> None: + info = parse_announce(packet) + if not info: + return + mac_hex = mac_bytes_to_hex(peer_mac) + if not mac_hex: + return + + devices = Device() + did, persisted = devices.upsert_espnow_announced( + mac_hex, + info["name"], + device_type=info.get("device_type", "led"), + num_leds=info.get("num_leds"), + color_order=info.get("color_order"), + startup_mode=info.get("startup_mode"), + brightness=info.get("brightness"), + ) + if not did: + return + if persisted: + print(f"[espnow] registered mac={did} name={info['name']!r}") + + groups = Group() + gids = groups_for_mac(did, groups) + groups_pkt = pack_groups(gids) + + client = get_bridge_client() + if client is None: + print("[espnow] bridge client not configured; groups not sent") + return + ok = await client.send_espnow(groups_pkt, peer_mac=peer_mac) + if ok: + print(f"[espnow] groups -> {did}: {gids}") + else: + print(f"[espnow] groups send failed for {did}") + + +async def push_groups_for_group_devices(gdoc: dict) -> None: + """Refresh GROUPS on every MAC listed on a group document.""" + mac_list = gdoc.get("devices") if isinstance(gdoc.get("devices"), list) else [] + for mac in mac_list: + m = normalize_mac(str(mac)) + if m: + await push_groups_to_mac(m) + + +async def push_groups_to_mac(mac_hex: str) -> bool: + """Re-send GROUPS packet to one device (after group membership change).""" + mac = normalize_mac(mac_hex) + if not mac: + return False + client = get_bridge_client() + if client is None: + return False + groups = Group() + gids = groups_for_mac(mac, groups) + pkt = pack_groups(gids) + return await client.send_espnow(pkt, peer_mac=bytes.fromhex(mac)) diff --git a/src/util/espnow_wire.py b/src/util/espnow_wire.py new file mode 100644 index 0000000..e6436ad --- /dev/null +++ b/src/util/espnow_wire.py @@ -0,0 +1,291 @@ +""" +ESP-NOW wire format: magic header + message types, Pi↔bridge WebSocket framing. + +See docs/espnow-binary-protocol.md. +""" + +from __future__ import annotations + +import struct +from typing import Any, Dict, List, Optional, Tuple + +from util.binary_envelope import ( + BINARY_ENVELOPE_VERSION_2, + pack_binary_envelope_v2, + parse_binary_envelope_v2, +) + +WIRE_MAGIC = 0x4C +MAX_ESPNOW_PAYLOAD = 250 + +MSG_ANNOUNCE = 0x01 +MSG_GROUPS = 0x02 +MSG_CMD = 0x03 +MSG_GROUP_CMD = 0x04 +MSG_BRIDGE_CH = 0x10 + +BROADCAST_MAC = bytes.fromhex("ffffffffffff") + +WS_FLAG_BROADCAST = 0x01 + +COLOR_ORDER_TO_ENUM = { + "rgb": 0, + "rbg": 1, + "grb": 2, + "gbr": 3, + "brg": 4, + "bgr": 5, +} +ENUM_TO_COLOR_ORDER = {v: k for k, v in COLOR_ORDER_TO_ENUM.items()} + +STARTUP_MODE_TO_ENUM = {"default": 0, "last": 1, "off": 2} +ENUM_TO_STARTUP_MODE = {v: k for k, v in STARTUP_MODE_TO_ENUM.items()} + + +def normalize_mac_bytes(mac: Any) -> Optional[bytes]: + if mac is None: + return None + if isinstance(mac, (bytes, bytearray)) and len(mac) == 6: + return bytes(mac) + s = str(mac).strip().lower().replace(":", "").replace("-", "") + if len(s) == 12 and all(c in "0123456789abcdef" for c in s): + return bytes.fromhex(s) + return None + + +def mac_bytes_to_hex(mac: bytes) -> str: + return mac.hex() + + +def _pack_header(msg_type: int, body: bytes) -> bytes: + pkt = bytes([WIRE_MAGIC, msg_type]) + body + if len(pkt) > MAX_ESPNOW_PAYLOAD: + raise ValueError(f"ESP-NOW packet {len(pkt)} exceeds {MAX_ESPNOW_PAYLOAD}") + return pkt + + +def pack_announce( + *, + name: str, + num_leds: int, + color_order: str = "rgb", + startup_mode: str = "default", + brightness: int = 32, + device_type: int = 0, +) -> bytes: + name_b = name.encode("utf-8") + if len(name_b) > 250: + raise ValueError("name too long") + co = COLOR_ORDER_TO_ENUM.get(str(color_order).lower(), 0) + sm = STARTUP_MODE_TO_ENUM.get(str(startup_mode).lower(), 0) + body = ( + bytes([len(name_b)]) + + name_b + + struct.pack(" Optional[Dict[str, Any]]: + """Parse full ESP-NOW packet or body-only after type byte.""" + if len(payload) >= 2 and payload[0] == WIRE_MAGIC: + if payload[1] != MSG_ANNOUNCE: + return None + body = payload[2:] + else: + body = payload + off = 0 + if off + 1 > len(body): + return None + nl = body[off] + off += 1 + if off + nl + 5 > len(body): + return None + name = body[off : off + nl].decode("utf-8") + off += nl + num_leds = struct.unpack_from(" bytes: + parts = [bytes([min(255, len(group_ids))])] + for gid in group_ids[:255]: + gb = str(gid).encode("utf-8") + if len(gb) > 250: + raise ValueError("group id too long") + parts.append(bytes([len(gb)])) + parts.append(gb) + return _pack_header(MSG_GROUPS, b"".join(parts)) + + +def parse_groups(payload: bytes) -> Optional[List[str]]: + if len(payload) >= 2 and payload[0] == WIRE_MAGIC: + if payload[1] != MSG_GROUPS: + return None + body = payload[2:] + else: + body = payload + if not body: + return [] + off = 0 + if off + 1 > len(body): + return None + count = body[off] + off += 1 + out: List[str] = [] + for _ in range(count): + if off + 1 > len(body): + return None + gl = body[off] + off += 1 + if off + gl > len(body): + return None + out.append(body[off : off + gl].decode("utf-8")) + off += gl + return out + + +def cmd_envelope_size(envelope: bytes) -> int: + from util.binary_envelope import HEADER_LEN + + if len(envelope) < HEADER_LEN: + return len(envelope) + lp, ls, ld = envelope[2], envelope[3], envelope[4] + return HEADER_LEN + lp + ls + ld + + +def pack_cmd(envelope: bytes, *, save: bool = False) -> bytes: + if envelope and envelope[0] != BINARY_ENVELOPE_VERSION_2: + raise ValueError("CMD envelope must be v2 binary") + need = cmd_envelope_size(envelope) + body = envelope[:need] + if save: + body = body + bytes([1]) + if len(body) + 2 > MAX_ESPNOW_PAYLOAD: + raise ValueError("CMD envelope too large for ESP-NOW") + return _pack_header(MSG_CMD, body) + + +def pack_cmd_from_kwargs(*, save: bool = False, **kwargs: Any) -> bytes: + return pack_cmd(pack_binary_envelope_v2(**kwargs), save=save) + + +def parse_cmd(payload: bytes) -> Tuple[Optional[bytes], bool]: + """Return (v2 envelope bytes, save_flag) inside CMD packet.""" + if len(payload) < 2 or payload[0] != WIRE_MAGIC or payload[1] != MSG_CMD: + return None, False + env = payload[2:] + if not env: + return None, False + need = cmd_envelope_size(env) + if need > len(env): + return None, False + save = len(env) > need and env[need] == 1 + return bytes(env[:need]), save + + +def parse_cmd_as_v1_dict(payload: bytes) -> Optional[Dict[str, Any]]: + env, save = parse_cmd(payload) + if env is None: + return None + data = parse_binary_envelope_v2(env) + if data is None: + return None + if save: + data["save"] = True + return data + + +def pack_group_cmd(group_id: str, envelope: bytes, *, save: bool = False) -> bytes: + if envelope and envelope[0] != BINARY_ENVELOPE_VERSION_2: + raise ValueError("GROUP_CMD envelope must be v2 binary") + gid_b = str(group_id).encode("utf-8") + if len(gid_b) > 250: + raise ValueError("group id too long") + need = cmd_envelope_size(envelope) + env = envelope[:need] + if save: + env = env + bytes([1]) + body = bytes([len(gid_b)]) + gid_b + env + return _pack_header(MSG_GROUP_CMD, body) + + +def pack_group_cmd_from_kwargs(group_id: str, **kwargs: Any) -> bytes: + return pack_group_cmd(group_id, pack_binary_envelope_v2(**kwargs)) + + +def parse_group_cmd(payload: bytes) -> Optional[Tuple[str, bytes]]: + if len(payload) < 2 or payload[0] != WIRE_MAGIC or payload[1] != MSG_GROUP_CMD: + return None + body = payload[2:] + if not body: + return None + gl = body[0] + if 1 + gl > len(body): + return None + gid = body[1 : 1 + gl].decode("utf-8") + env = body[1 + gl :] + return gid, bytes(env) + + +def pack_bridge_channel(channel: int) -> bytes: + ch = max(1, min(11, int(channel))) + return _pack_header(MSG_BRIDGE_CH, bytes([ch])) + + +def parse_bridge_channel(payload: bytes) -> Optional[int]: + if len(payload) < 3 or payload[0] != WIRE_MAGIC or payload[1] != MSG_BRIDGE_CH: + return None + return int(payload[2]) + + +def wire_msg_type(payload: bytes) -> Optional[int]: + if len(payload) >= 2 and payload[0] == WIRE_MAGIC: + return int(payload[1]) + return None + + +def pack_ws_downlink( + espnow_packet: bytes, + *, + peer_mac: Optional[Any] = None, + broadcast: bool = False, +) -> bytes: + flags = WS_FLAG_BROADCAST if broadcast else 0 + if broadcast: + peer = BROADCAST_MAC + else: + peer = normalize_mac_bytes(peer_mac) + if peer is None: + raise ValueError("peer MAC required for unicast downlink") + return bytes([flags]) + peer + espnow_packet + + +def pack_ws_uplink(peer_mac: bytes, espnow_packet: bytes) -> bytes: + peer = normalize_mac_bytes(peer_mac) + if peer is None: + raise ValueError("invalid peer MAC") + return bytes([0]) + peer + espnow_packet + + +def parse_ws_frame(frame: bytes) -> Tuple[bytes, bytes, bool]: + """ + Returns (peer_mac_6bytes, espnow_packet, is_broadcast_dest). + """ + if len(frame) < 8: + raise ValueError("WS frame too short") + flags = frame[0] + peer = frame[1:7] + pkt = frame[7:] + broadcast = bool(flags & WS_FLAG_BROADCAST) or peer == BROADCAST_MAC + return peer, pkt, broadcast diff --git a/src/util/groups_for_device.py b/src/util/groups_for_device.py new file mode 100644 index 0000000..787c2ab --- /dev/null +++ b/src/util/groups_for_device.py @@ -0,0 +1,23 @@ +"""Resolve group membership for a device MAC.""" + +from models.device import normalize_mac + + +def groups_for_mac(mac_hex: str, groups_model) -> list[str]: + """Return group ids (string keys) that list this device MAC.""" + mac = normalize_mac(mac_hex) + if not mac: + return [] + out: list[str] = [] + for gid in groups_model.list(): + doc = groups_model.read(gid) + if not isinstance(doc, dict): + continue + devs = doc.get("devices") + if not isinstance(devs, list): + continue + for d in devs: + if normalize_mac(str(d)) == mac: + out.append(str(gid)) + break + return out diff --git a/tests/bridge_broadcast_test.py b/tests/bridge_broadcast_test.py new file mode 100644 index 0000000..7c48799 --- /dev/null +++ b/tests/bridge_broadcast_test.py @@ -0,0 +1,224 @@ +#!/usr/bin/env python3 +"""Send binary ESP-NOW packets via the bridge (broadcast passthrough). + +The simplified ``espnow-sender`` forwards each WebSocket **binary** message +unchanged to ESP-NOW ``ff:ff:ff:ff:ff:ff``. No ``pack_ws_downlink`` wrapper +and no 1-byte ack — raw wire packets only (see ``docs/espnow-binary-protocol.md``). + +Group membership is expected to be configured on each **led-driver**; this +script only broadcasts **CMD** (and optional **GROUPS** / **GROUP_CMD** for +manual testing). + +Examples:: + + pipenv run python tests/bridge_broadcast_test.py + + pipenv run python tests/bridge_broadcast_test.py --url ws://192.168.4.1/ws + + pipenv run python tests/bridge_broadcast_test.py --brightness 200 + + pipenv run python tests/bridge_broadcast_test.py --select led-abc --state on + + pipenv run python tests/bridge_broadcast_test.py --groups 5,18 --group-cmd 18 --brightness 64 +""" + +from __future__ import annotations + +import argparse +import asyncio +import json +import sys +from pathlib import Path + +PROJECT_ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(PROJECT_ROOT / "src")) + +from util.espnow_wire import ( # noqa: E402 + pack_cmd_from_kwargs, + pack_group_cmd_from_kwargs, + pack_groups, + wire_msg_type, +) + +MSG_TYPE_NAMES = { + 0x01: "ANNOUNCE", + 0x02: "GROUPS", + 0x03: "CMD", + 0x04: "GROUP_CMD", + 0x10: "BRIDGE_CH", +} + + +def _load_bridge_url(explicit: str | None) -> str: + if explicit and explicit.strip(): + return explicit.strip() + path = PROJECT_ROOT / "settings.json" + if path.is_file(): + try: + data = json.loads(path.read_text(encoding="utf-8")) + url = str(data.get("bridge_ws_url") or "").strip() + if url: + return url + except (OSError, json.JSONDecodeError, TypeError): + pass + return "ws://192.168.4.1/ws" + + +def _describe_packet(pkt: bytes) -> str: + if len(pkt) < 2: + return f"{len(pkt)} B" + name = MSG_TYPE_NAMES.get(pkt[1], f"0x{pkt[1]:02x}") + return f"{name} {len(pkt)} B" + + +async def _send_packets(url: str, packets: list[bytes], delay_s: float) -> None: + import websockets + + print(f"connecting to {url}") + async with websockets.connect(url, ping_interval=20, ping_timeout=20) as ws: + print("connected (broadcast passthrough)") + for i, pkt in enumerate(packets): + print(f" send [{i + 1}/{len(packets)}] {_describe_packet(pkt)}") + await ws.send(pkt) + if delay_s > 0 and i + 1 < len(packets): + await asyncio.sleep(delay_s) + print("done") + + +def _build_packets(args: argparse.Namespace) -> list[bytes]: + packets: list[bytes] = [] + + if args.groups: + gids = [g.strip() for g in args.groups.split(",") if g.strip()] + if gids: + packets.append(pack_groups(gids)) + + if args.group_cmd: + packets.append( + pack_group_cmd_from_kwargs( + args.group_cmd, + brightness_0_255=args.brightness, + select={args.select: [args.state]} if args.select else None, + save=args.save, + ) + ) + + if args.brightness is not None and not args.group_cmd: + packets.append( + pack_cmd_from_kwargs(brightness_0_255=args.brightness, save=args.save) + ) + + if args.select: + packets.append( + pack_cmd_from_kwargs( + select={args.select: [args.state]}, + save=args.save, + ) + ) + + if args.off: + if args.select: + packets.append( + pack_cmd_from_kwargs(select={args.select: ["off"]}, save=args.save) + ) + else: + packets.append(pack_cmd_from_kwargs(select={"all": ["off"]}, save=args.save)) + + if not packets: + packets.append(pack_cmd_from_kwargs(brightness_0_255=128)) + packets.append(pack_cmd_from_kwargs(select={"all": ["on"]})) + packets.append(pack_cmd_from_kwargs(select={"all": ["off"]})) + + for pkt in packets: + if wire_msg_type(pkt) is None: + raise ValueError("built packet is not valid wire format") + return packets + + +def main() -> int: + parser = argparse.ArgumentParser( + description="Broadcast binary ESP-NOW packets through the bridge WebSocket.", + ) + parser.add_argument( + "--url", + default=None, + help="Bridge WebSocket URL (default: settings.json bridge_ws_url or ws://192.168.4.1/ws)", + ) + parser.add_argument( + "--delay", + type=float, + default=0.5, + help="Seconds between packets (default: 0.5)", + ) + parser.add_argument( + "--brightness", + "-b", + type=int, + default=None, + metavar="0-255", + help="Broadcast CMD: global brightness", + ) + parser.add_argument( + "--select", + metavar="DEVICE_NAME", + help="Broadcast CMD: device name in select map (must match driver settings name)", + ) + parser.add_argument( + "--state", + default="on", + help="Pattern/state for --select (default: on)", + ) + parser.add_argument( + "--off", + action="store_true", + help="After other commands, send select off (all devices if --select omitted)", + ) + parser.add_argument( + "--groups", + metavar="ID,ID", + help="Optional GROUPS broadcast (normally configured on device instead)", + ) + parser.add_argument( + "--group-cmd", + metavar="GROUP_ID", + help="Optional GROUP_CMD broadcast (driver must list this group locally)", + ) + parser.add_argument( + "--save", + action="store_true", + help="Set save flag on CMD / GROUP_CMD envelopes", + ) + parser.add_argument( + "--dry-run", + action="store_true", + help="Print packets only; do not connect", + ) + args = parser.parse_args() + + url = _load_bridge_url(args.url) + try: + packets = _build_packets(args) + except ValueError as e: + print(f"error: {e}", file=sys.stderr) + return 1 + + print(f"url={url!r} packets={len(packets)}") + for pkt in packets: + print(f" {_describe_packet(pkt)} hex={pkt.hex()}") + + if args.dry_run: + return 0 + + try: + asyncio.run(_send_packets(url, packets, args.delay)) + except KeyboardInterrupt: + print("interrupted") + return 130 + except Exception as e: + print(f"failed: {e!r}", file=sys.stderr) + return 1 + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tests/models/test_device.py b/tests/models/test_device.py index b3aadbd..c8d3fb1 100644 --- a/tests/models/test_device.py +++ b/tests/models/test_device.py @@ -150,6 +150,27 @@ def test_device_duplicate_names_allowed(): assert devices.read(a1)["name"] == devices.read(a2)["name"] == "alpha" +def test_upsert_espnow_announced(): + devices = _fresh_device() + m = "e8f60a16dad0" + i1, p1 = devices.upsert_espnow_announced( + m, + "led-test", + num_leds=120, + color_order="grb", + startup_mode="last", + brightness=70, + ) + assert i1 == m and p1 is True + d = devices.read(m) + assert d["transport"] == "espnow" + assert d["address"] == m + assert d["name"] == "led-test" + assert d["num_leds"] == 120 + i2, p2 = devices.upsert_espnow_announced(m, "led-test") + assert i2 == m and p2 is False + + def test_device_duplicate_mac_rejected(): devices = _fresh_device() devices.create("one", address="aa:bb:cc:dd:ee:ff") @@ -163,6 +184,7 @@ def test_device_duplicate_mac_rejected(): if __name__ == "__main__": test_device() test_upsert_wifi_tcp_client() + test_upsert_espnow_announced() test_device_can_change_address() test_device_duplicate_names_allowed() test_device_duplicate_mac_rejected() diff --git a/tests/test_espnow_wire.py b/tests/test_espnow_wire.py new file mode 100644 index 0000000..e841dbd --- /dev/null +++ b/tests/test_espnow_wire.py @@ -0,0 +1,110 @@ +"""Tests for ESP-NOW binary wire format.""" + +import sys +from pathlib import Path + +PROJECT_ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(PROJECT_ROOT / "src")) + +from util.binary_envelope import pack_binary_envelope_v2 # noqa: E402 +from util.espnow_wire import ( # noqa: E402 + BROADCAST_MAC, + MAX_ESPNOW_PAYLOAD, + MSG_ANNOUNCE, + MSG_CMD, + MSG_GROUPS, + WIRE_MAGIC, + pack_announce, + pack_bridge_channel, + pack_cmd, + pack_cmd_from_kwargs, + pack_group_cmd_from_kwargs, + pack_groups, + pack_ws_downlink, + pack_ws_uplink, + parse_announce, + parse_cmd_as_v1_dict, + parse_group_cmd, + parse_groups, + parse_ws_frame, + wire_msg_type, +) + + +def test_announce_round_trip(): + raw = pack_announce( + name="led-abc123", + num_leds=119, + color_order="grb", + startup_mode="last", + brightness=70, + ) + assert len(raw) <= MAX_ESPNOW_PAYLOAD + assert raw[0] == WIRE_MAGIC + assert raw[1] == MSG_ANNOUNCE + d = parse_announce(raw) + assert d["name"] == "led-abc123" + assert d["num_leds"] == 119 + assert d["color_order"] == "grb" + assert d["startup_mode"] == "last" + assert d["brightness"] == 70 + + +def test_groups_round_trip(): + raw = pack_groups(["5", "18", "test"]) + assert wire_msg_type(raw) == MSG_GROUPS + assert parse_groups(raw) == ["5", "18", "test"] + + +def test_cmd_envelope_round_trip(): + env = pack_binary_envelope_v2(brightness_0_255=128) + raw = pack_cmd(env, save=True) + assert wire_msg_type(raw) == MSG_CMD + assert len(raw) <= MAX_ESPNOW_PAYLOAD + d = parse_cmd_as_v1_dict(raw) + assert d == {"v": "1", "b": 128, "save": True} + + +def test_cmd_from_kwargs(): + raw = pack_cmd_from_kwargs( + select={"dev1": ["on"]}, + brightness_0_255=64, + ) + d = parse_cmd_as_v1_dict(raw) + assert d["select"]["dev1"] == ["on"] + assert d["b"] == 64 + + +def test_group_cmd_round_trip(): + raw = pack_group_cmd_from_kwargs("18", brightness_0_255=32) + gid, env = parse_group_cmd(raw) + assert gid == "18" + d = parse_cmd_as_v1_dict(bytes([WIRE_MAGIC, MSG_CMD]) + env) + assert d["b"] == 32 + + +def test_ws_frame_round_trip(): + pkt = pack_announce(name="led-x", num_leds=10) + peer = bytes.fromhex("e8f60a16dad0") + up = pack_ws_uplink(peer, pkt) + p2, pkt2, bcast = parse_ws_frame(up) + assert p2 == peer + assert pkt2 == pkt + assert not bcast + + down = pack_ws_downlink(pkt, peer_mac=peer) + p3, pkt3, bcast3 = parse_ws_frame(down) + assert p3 == peer + assert pkt3 == pkt + assert not bcast3 + + bdown = pack_ws_downlink(pkt, broadcast=True) + _, pkt4, bcast4 = parse_ws_frame(bdown) + assert pkt4 == pkt + assert bcast4 + + +def test_bridge_channel(): + raw = pack_bridge_channel(6) + assert len(raw) == 3 + assert raw[1] == 0x10