Files
led-controller/docs/BENCH_TEST.md
jimmyandClaude Sonnet 5.5 6e2d5ff80c chore: build with espressif/idf v6.1
Makefile and the bench-test doc use the v6.1 image. dependencies.lock came out of the build as idf 5.5.5 /
littlefs 2.0.0, which does not match the image: regenerate it with a v6.1 build.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-12 00:46:11 +13:00

296 lines
22 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Bench test: one S3 controller and one LED driver
This walks one ESP32-S3 controller (W5500 Ethernet, INMP441 mic) and one `led-driver` unit through flashing, a smoke test, a lossy-link test and a beat latency measurement. Nothing here has been run on hardware yet, so treat the expected numbers as targets to check, not results.
## 0. What you need
- Controller: XIAO ESP32-S3 with a W5500 module wired as in `main/Kconfig.projbuild` (SCLK 7, MISO 8, MOSI 9, CS 2, INT 3, RST 1), Ethernet cable to a DHCP network. The mic is optional for this test; the bench generator makes its own beats.
- Driver: one `led-driver` unit (MicroPython, the `p2p` ESP-NOW branch) with a few LEDs. It needs no Wi-Fi network.
- A PC on the same network. Docker with `espressif/idf:v6.1` for building.
- For the true latency measurement: a logic analyzer or scope with two channels and a common ground. Optional: the software echo numbers need nothing extra.
**Channel rule.** ESP-NOW only works when both radios use the same channel. The p2p driver uses its `wifi_channel` setting (default 5); the controller's default is now 5 as well (`CONFIG_ESPNOW_CHANNEL` in `sdkconfig.defaults`). If your driver uses another channel, change the controller before building:
```bash
docker run --rm -v "$PWD":/project -w /project espressif/idf:v6.1 idf.py menuconfig
```
then *ESP-NOW config → Wi-Fi channel*. Alternatively set `CONFIG_ESPNOW_CHANNEL=<n>` in `sdkconfig.defaults` (delete `sdkconfig` after, so it is regenerated). To use the beat marker pin, also set `CONFIG_BEAT_TEST_GPIO` (for example 44) in the same menu.
## 1. Build and flash the controller
The repository root has a Makefile that wraps all of this: `make build`, `make flash`, `make monitor` (Ctrl-] to quit), `make log` (console for 10 s, no terminal needed), `make test`, and `make help` for the rest; `PORT=/dev/ttyACM1` selects another port. The manual commands below do the same thing.
```bash
tools/build_web.sh
docker run --rm -v "$PWD":/project -w /project espressif/idf:v6.1 idf.py build
```
Expect `Project build complete`, the app well under its 2 MB slot, with the web UI embedded (`www_dist/www.pack`). Fix ownership if Docker left it root-owned: `docker run --rm -v "$PWD":/project espressif/idf:v6.1 chown -R 0:0 /project`.
Flash with the S3 on USB (check which port it is; `ls /dev/ttyACM*`):
```bash
docker run --rm --device=/dev/ttyACM0 -v "$PWD":/project -w /project espressif/idf:v6.1 idf.py -p /dev/ttyACM0 flash
```
This writes the bootloader, partition table and app (the web UI is part of the app). Watch the log:
```bash
docker run --rm -it --device=/dev/ttyACM0 -v "$PWD":/project -w /project espressif/idf:v6.1 idf.py -p /dev/ttyACM0 monitor
```
Look for, in order: `store: ... records` lines, `espnow_tx: ESP-NOW up on channel N`, `rel: reliable link up`, `Ethernet Link Up`, `Got IP: x.x.x.x`, `HTTP server started`.
## 2. First checks (replace `$S3` with the IP)
```bash
export S3=192.168.1.50
curl -s http://$S3/status
```
Expect JSON with `uptime_s`, `reset_reason` (why the board last restarted, such as `panic`, `task_watchdog` or `brownout`; opening the USB serial port resets the board and replaces it with `usb`, so read it first), `bpm`, an `espnow` block and a `link` block (`ver` nonzero, `digests` increasing by one about every 1.5 s). If the mic hears music, `bpm` and `beats_sent` move too, which means beats are already being broadcast. Open `http://$S3/` in a browser: the UI should load, and the dev tools network tab should show `Content-Encoding: gzip` on `/static/*.js`.
```bash
curl -s -D - -o /dev/null -H 'Accept-Encoding: gzip' http://$S3/static/presets.js | grep -i -E 'content-encoding|content-type'
```
**Audio and beats.** `http://$S3/audio` is the mic page: live levels, tempo and bar position, a tap button to correct beat 1, and a Listen button for the 16 kHz mono stream. The underlying endpoints are `/audio.wav` (one listener at a time), `/beats` (server-sent events: `beat`, `onset`, `tempo`), `POST /tap` and `/time`. `/status` carries the mic levels (`mic_rms_dbfs`, `mic_spl_db`) and the beat state (`bpm`, `beat_locked`, `meter`, `bar_beat`, `bar_sure`).
## 3. Load data and add a test zone
Import the Pi database (the files hold your real zones and presets, so use a copy if you want a smaller test):
```bash
for n in device zone preset profile palette pattern sequence scene; do
curl -s -X PUT --data-binary @../db/$n.json http://$S3/db/$n; echo " $n"
done
```
The driver registers itself: it broadcasts a hello at boot, the controller adds it to the device list under its MAC and sends it its group list. Check with `curl -s http://$S3/devices/<mac>` once the driver is running (step 4). Zones are the drivers' groups: a zone's `names` must contain the driver's **name** exactly (`led-<mac>` by default, see its `settings.json`).
```bash
curl -s -X POST -d '{"name":"bench","names":["led-aabbccddeeff"]}' http://$S3/zones
curl -s -X POST -d '{"name":"bench red","pattern":"on","colors":["#ff0000"],"delay":100,"brightness":255,"auto":true}' http://$S3/presets
```
Note the returned ids (`<zone>` and `<preset>` below). Creating or changing a zone re-sends group lists to every registered ESP-NOW driver.
## 4. Prepare the driver
The driver runs the `p2p` branch of `led-driver` plus these files from this work: `src/beat_sync.py`, `src/espnow_rx.py`, and changed `src/main.py` and `src/presets.py`. Back up the device's own copies first (`mpremote fs cp :main.py ./main.py.bak`), then copy the new ones (`mpremote fs cp src/main.py :main.py` and so on) and reset.
Optional keys in the driver's `settings.json` for the later tests:
| key | value | meaning |
|---|---|---|
| `bench` | `true` | echo every fired beat to the controller and print stats every 50 beats |
| `beat_pin` | e.g. `5` | GPIO toggled on every fired beat, for the logic analyzer |
| `espnow_loss` | `0` | percent of received frames the driver discards (set later) |
After boot, the controller's `/devices` list should contain the driver's MAC and `/status` should show `link.acked` rising (the group list was acknowledged). `mpremote` interrupts the running program, so reset the driver after using it.
## 5. Smoke test
```bash
curl -s -X POST -d '{"preset_id":"<preset>"}' http://$S3/zones/<zone>/play
```
The driver's console should log the preset and select it received (once each, although the controller sends every state frame twice), and the LEDs should go red within a fraction of a second. Then:
```bash
curl -s -X POST http://$S3/zones/<zone>/stop # LEDs off
curl -s http://$S3/devices/status # last_ack_ms_ago / failed_messages per registered driver
```
Zone play is a group broadcast, so `devices/status` only changes for acked (unicast) traffic. Trigger one with a targeted push:
```bash
curl -s -X POST -d '{"sequence":[{"v":"1","select":{"led-aabbccddeeff":["<preset>"]}}],"targets":["aabbccddeeff"]}' http://$S3/presets/push
curl -s http://$S3/devices/status
```
Expected: `last_ack_ms_ago` is small (tens of ms) and `failed_messages` is 0. `/status` shows `link.acked` increased.
## 6. Lossy-link test
The loss mode discards a percentage of frames before the radio on the controller (all outgoing frames, and incoming acks and echoes). The driver can discard received frames too. This makes the retry, fragment and resync layers do real work without moving antennas.
```bash
# controller: 40% loss, beats included
curl -s -X POST -d '{"percent":40,"beats":true}' http://$S3/test/loss
curl -s http://$S3/status # sim_dropped counts dropped frames
```
Run these and compare with the table:
| test | how | expected |
|---|---|---|
| Acked unicast under loss | repeat the targeted push from step 5 twenty times at 40% (give each a different select step so you can tell them apart on the driver console) | all reach the driver; `link.retransmits` rises. A message needs its data and its ack to both survive, and there are 5 sends in total, so about 1 in 10 is reported as timed out even though the driver usually has it |
| Give up and report | set `percent` to 100, push once, wait 2 s | `link.ack_timeouts` +1 and `devices/status` shows `failed_messages: 1` |
| Lost broadcast, digest recovery | `percent` 100, play a different preset on the zone, then set `percent` 0 | the LEDs do not change at first, then within about 1.5 to 5 s the driver sends a resync and shows the new preset; `link.resyncs` +1 |
| Driver-side loss | set `"espnow_loss": 40` in the driver's `settings.json` (restore it afterwards), controller loss 0, push twenty times | every push is processed once on the driver (retransmissions are de-duplicated); `link.retransmits` rises. Lost acks are covered by the controller-side loss rows above, because that setting also drops incoming acks |
| Multi-message send | `POST /presets/send` with 10+ preset ids, `save: false` and `targets` | the controller packs the presets into a few binary messages (14 presets took 2); all arrive under loss, with retransmits. `save: true` makes the driver write its `presets.json`, so restore that file afterwards. Frame-level fragmentation exists in the link but nothing in the current API produces a message over one frame, so it is only covered by the driver's unit tests |
Set loss back to 0 when done: `curl -s -X POST -d '{"percent":0}' http://$S3/test/loss`.
Reading the numbers: the controller drops a reply with the same probability as a request, so under loss an unicast message needs both the data and the ack to survive. Resync requests are also subject to loss and to the driver's 3 s minimum gap, so recovery may take a few digests.
### Results of the first run (S3 to one p2p driver, channel 5)
| test | result |
|---|---|
| 20 acked pushes, controller loss 40% | 20 of 20 reached the driver; 19 acked, 1 reported timed out (its ack was lost); 19 retransmits; 34 frames dropped by the test mode |
| 1 push, loss 100% | gave up after 2.0 s with 4 retransmits; `failed_messages` 1; driver heard nothing |
| zone play during a 100% outage, then loss back to 0 | driver was untouched during the outage; 1.5 s after the link returned it received its groups, the preset and the select, one resync, and the new state |
| 20 acked pushes, driver-side loss 40% | 20 of 20 processed exactly once; 14 retransmits; 0 timeouts |
| `/presets/send` of 14 presets at driver loss 40% | 2 messages (209 and 217 bytes), both acked, 1 retransmit |
## 7. Beat latency and delivery
### Software numbers (no equipment)
Start synthetic beats with the driver in `bench` mode:
```bash
curl -s -X POST -d '{"bpm":120,"count":200}' http://$S3/test/beats
sleep 105
curl -s http://$S3/test/beats
```
Results include:
- `beats_sent`, `echoes` and `hit_pct_total`: the share of beats each driver fired on and reported. With no loss expect 100%.
- `rtt_minus_lead_min/avg/max_us`: time from the first beat copy going out to the driver's echo coming back, minus the 30 ms the driver deliberately waits (`CONFIG_BEAT_LEAD_MS`). This is out-airtime plus the driver's MicroPython work plus return airtime, so it overstates one-way delay. Expect roughly 2 to 15 ms; the maximum shows the worst case you will see on the LEDs.
- `spread_avg_us` and `spread_max_us`: only meaningful with two or more drivers; shows how far apart they fire for the same beat.
On the driver's serial console every 50 beats you get the true driver-side counts, for example `beat {'fires': 100, 'dups': 100, 'late': 0, 'freerun': 0}`:
- `fires`: beats it acted on.
- `dups`: second copies dropped (with the double send this should be close to `fires`).
- `freerun`: beats it generated itself because both copies were missed. Zero without loss.
- `late`: frames rejected as too soon or old.
### Delivery under loss
Repeat the run with loss on:
```bash
curl -s -X POST -d '{"percent":30,"beats":true}' http://$S3/test/loss
curl -s -X POST -d '{"bpm":120,"count":200}' http://$S3/test/beats
```
Judge delivery from the driver's own `fires` and `freerun` counts: with two copies the chance of missing both is about 9% at 30% loss, and the free-run covers those, so `fires` should stay near `beats_sent` with `freerun` around 9% of it. The controller's `hit_pct_total` is lower because echoes are dropped by the same loss setting; it is a lower bound, roughly `(1 - loss) × delivery`.
### Latency over USB (no extra equipment)
Both boards are on USB, so the PC can timestamp one event line from each and compare them. Settings already applied for this: the driver has `"usb_log": true` in `settings.json`. The controller prints `BT <beat> <microseconds>` for every synthetic beat; the driver prints `FIRE <beat> <ticks_us>` when it fires.
```bash
python3 tools/usb_latency.py --ctrl-port /dev/ttyACM0 --drv-port /dev/ttyACM1 --url http://$S3 --beats 200
```
The script starts the generator, reads both ports, and maps each board's clock onto the PC's by using the fastest USB deliveries per 5 s window (this also removes crystal drift). USB serial delivers in bursts, so single lines arrive up to about 50 ms late; the fastest ones are close to on time, which is what the mapping relies on. The remaining error is the difference between the two ports' fastest delivery, roughly 1 ms, so read results to about a millisecond. The script prints the delivery jitter so you can see how noisy the ports were. The driver's `FIRE` time is when the beat triggers the pattern step; the LED data follows within the same loop pass, and a 119-LED strip takes about 3.6 ms to clock out.
Close any serial monitor first (only one program can hold a port). Do not run long `mpremote exec` commands on the driver during the capture (see Troubleshooting).
### True latency with a logic analyzer
For a measurement that does not depend on USB. Marker pins are set up already: the controller toggles **GPIO44 (D7)** when the first copy of each beat goes out (`CONFIG_BEAT_TEST_GPIO=44`, in `sdkconfig.defaults`) and the driver toggles **GPIO44 (D7)** when it fires (`"beat_pin": 44` in its `settings.json`). Both are XIAO ESP32S3 boards, where D7 is free.
Wiring (3.3 V logic, any 2 or 3 channel analyzer):
| analyzer | connect to |
|---|---|
| GND | GND on the controller **and** GND on the driver (one common ground) |
| channel 0 | controller **D7** |
| channel 1 | driver **D7** |
| channel 2 (optional) | driver **D10** (GPIO9, the LED data line) to see when the LEDs really update |
Steps:
1. Make sure the driver runs a preset that changes on every beat if you use channel 2, for example a `chase` preset with `auto` off, selected on the zone. With the plain `on` preset the LED data line stays quiet.
2. Start a capture: sample rate at least 1 MS/s, length at least 60 s (edges only; it stays small), trigger on any edge of channel 0.
3. Start the beats: `curl -s -X POST -d '{"bpm":120,"count":100}' http://$S3/test/beats`. Both markers toggle on every beat, so you will see edges 500 ms apart.
4. Stop the capture when the generator finishes (about 52 s) and export the digital data as CSV (time in seconds, one column per channel).
5. Analyze: `python3 tools/analyze_edges.py capture.csv --ctrl 0 --drv 1 --time-unit s`. Use `--lead-ms` if you changed `CONFIG_BEAT_LEAD_MS`. It pairs each controller edge with the next driver edge and prints the delay beyond the 30 ms lead, with min, mean, median, p95, max and jitter, plus the count of unmatched beats. On the scope, channel 2 shows the true light delay: the first data burst after the driver edge.
Targets from the plan: under about 10 ms from the controller edge to the driver edge (after subtracting the lead).
Return to normal use by deleting `beat_pin` and `usb_log` from the driver's `settings.json` and setting `CONFIG_BEAT_TEST_GPIO=-1` (the marker pin otherwise toggles on every beat, including live mic beats).
### Results of the first run (software numbers, one driver, channel 5)
Synthetic beats at 120 BPM, 160 beats per run, mic beats muted while the generator runs, driver in `bench` mode.
| | no loss | 30% loss (beats included) |
|---|---|---|
| beats the driver acted on, per beat sent | 1.00 | 1.00 |
| beats the driver invented itself (free-run) | 0% | 11% (theory for two independent copies: 9%) |
| second copies dropped as duplicates | 100% | 47% (theory: 49%) |
| echoes the controller received | 162 of 162 | 96 of 161 (echoes are lost at 30% too, and invented beats do not echo) |
| round trip minus the 30 ms lead, min / avg / max | 9.8 / 15.0 / 25.7 ms | 3.4 / 16.2 / 26.0 ms |
Reading it: the double send plus free-run gave the driver a beat every time, even with 30% of frames lost. The round-trip figure includes the driver's loop and the echo's trip back, so it is an upper bound on the delay the LEDs see; it is not the controller-to-LED latency. The USB and logic analyzer measurements below give the real figure.
Latency over USB (200 beats at 120 BPM, one driver, 119 LEDs): delay beyond the 30 ms lead min 0.49, mean 2.07, median 1.77, p95 4.41, max 7.95 ms (std 1.3 ms; accuracy about 1 ms). 197 of 200 beats matched. Both clocks drifted by 60 to 80 ppm against the PC, and individual USB lines arrived up to 54 ms late, which is why the script uses the fastest deliveries. This is the time from the controller sending a beat to the driver firing, beyond the built-in lead. The logic analyzer run has not been done.
### ESP32-C3 driver compared with the S3 driver
Driver: XIAO ESP32-C3 (MAC 58:8c:81:a3:74:58), MicroPython v1.22.1 stock (no ws2812-dma patch), 160 MHz, about 128 KB free heap, 119 LEDs on pin 10. Same tests as above, same controller.
| | S3 driver, no loss | C3 driver, no loss |
|---|---|---|
| echoes the controller received | 162 of 162 | 160 of 160 |
| round trip minus the 30 ms lead, min / avg / max | 9.8 / 15.0 / 25.7 ms | 2.6 / 4.5 / 17.8 ms |
C3 with 30% loss (160 beats): every beat produced exactly one pattern step, 15 of the 160 (9%, theory 9%) came from free-run, and 68 second copies were dropped as duplicates; the controller got 99 of 160 echoes. That matches the S3 result in the table above.
Reading the free-run counter: after the generator stops, the driver keeps stepping for up to 4 more beats on its last period. A stats window that runs past the end of the test therefore shows 4 extra "free-run" fires even with no loss. They are not missed beats. (My first read of the C3's no-loss run said otherwise; that was wrong.)
Latency over USB, same settings on both boards (`usb_log` only), 200 beats at 120 BPM, delay beyond the 30 ms lead, accuracy about 1 ms:
| | matched | min | mean | median | p95 | max | std |
|---|---|---|---|---|---|---|---|
| S3 driver (240 MHz) | 200 / 200 | -2.4 | 2.15 | 2.03 | 6.03 | 12.3 | 2.5 ms |
| C3 driver (160 MHz) | 200 / 200 | 0.2 | 3.36 | 3.06 | 7.47 | 19.7 | 2.3 ms |
The C3 is about 1.2 ms slower on average and has a longer tail (maximum 19.7 ms against 12.3 ms). With the method's 1 ms accuracy, the mean difference is small and the tail difference is the more telling one. Both are well inside a 120 BPM beat. The echo round trip disagrees with this (S3 15 ms mean, C3 4.5 ms). The S3 echo runs had `usb_log` and a marker pin switched on and the C3 runs did not, and I did not rerun the S3 echo test with identical settings, so treat the echo figures as unexplained and prefer the USB ones.
Pattern frame rate (strip writes per second, 119 LEDs, delay 1 ms, 2 s each; only patterns that wrote anything are shown):
| pattern | C3 (160 MHz, 115 KB free) | S3 (240 MHz, 8 MB free) |
|---|---|---|
| flame | 22 | 33 |
| palette_morph | 21 | 32 |
| circle | 150 | 170 |
The CPU-heavy patterns run about 1.5 times faster on the S3, which is the clock ratio; a light pattern is limited by the strip write and differs by about 13%. `aurora` and `chase` wrote nothing with the parameters I used, so they are not measured.
Groups: the controller sends a driver its group list when the driver announces itself, but now only if the driver is in at least one zone. A driver in no zone keeps the groups it already has (an empty list from the controller used to wipe them). An explicit zone or device edit still re-sends every list, including empty ones.
**Wi-Fi start-up order on the C3.** With the extra link code loaded before Wi-Fi start-up, the C3 failed with `Wifi Unknown Error 0x0101` on about every other cold boot (2 of 4). Starting ESP-NOW before the pattern library and the LED driver claim memory fixed it (6 of 6 boots). `led-driver/src/main.py` now does this.
## 8. Troubleshooting
- **A board on a USB-serial/JTAG port (C3, and the S3 controller) resets whenever a program opens its port,** and a C3 can land in ROM download mode (`wait usb download`). The old apt `mpremote` 0.4.0 hangs on this. Use the newer one in `~/projects/micropython/micropython/tools/mpremote/mpremote.py`, or `led-tool`, and note that `fs cp -r` fails on drivers that print timestamps (copy named files instead).
- **Only one program can use a board's serial port at a time.** A serial logger left running while `led-cli` uploads corrupts the transfer silently: `led-cli` reports the files uploaded, but the old ones stay on the driver.
- **Running a long `mpremote exec` on the driver resets it:** the driver's watchdog (10 s) keeps running after `mpremote` interrupts `main.py`, so keep REPL experiments under about 8 s and reset the driver afterwards.
- **Driver never reacts, no acks:** channel mismatch. Check `CONFIG_ESPNOW_CHANNEL` against the driver's `wifi_channel`. Neighbouring channels can still work at bench range, so a working link does not prove they match.
- **Zone play logs nothing on the driver:** the driver is not in the zone's group (silent). Check the zone's `names` against the driver's `name`.
- **Devices show `last_ack_ms_ago: -1`:** the device record has no usable 12-character MAC in `address` or as its key.
- **`beats_sent` rises but `echoes` stays 0:** the driver is not in `bench` mode, or it has not yet heard a controller frame (it learns the controller's MAC from the first data or digest frame; wait a few seconds).
- **Resync never triggers:** the driver ignores a digest equal to its own version; confirm with `/status` that `link.ver` changed after the missed play.
- **Web UI is blank:** the web UI was not packed before the build; run `tools/build_web.sh` and build again (`make build` does this).
## 9. Cleanup
```bash
curl -s -X POST -d '{"percent":0}' http://$S3/test/loss
curl -s -X POST http://$S3/test/beats/stop
```
Set the driver's `bench` and `espnow_loss` back to off for normal use.