Files
led-controller/v2/docs/BENCH_TEST.md
T
jimmyandClaude Sonnet 5.5 a94d6a7de5 feat(v2): ESP32-S3 controller with Ethernet, I2S mic beat detection and ESP-NOW
New ESP-IDF project in v2/ that replaces the Pi and the UART bridge:

- W5500 Ethernet, INMP441 beat/tempo detector (from esp32-eth), ESP-NOW master
- LittleFS data store and a JSON API over the Pi's db/*.json collections,
  with /db import and export; web UI minified and gzipped at build time
- Zone play/stop as 0x4C GROUP_CMD, preset push/send as CMD (binary v2
  envelope, byte-identical to src/util/binary_envelope.py), sequencer and
  scenes, driver registration from the boot hello
- Reliable bulk link: acks with retransmit, fragments, state-version digest
  and resync; beat frames sent twice with a lead so drivers fire together
- Bench tooling: lossy-link mode, synthetic beats with echo measurement,
  USB and logic-analyzer latency scripts, and docs/BENCH_TEST.md

Needs the matching led-driver changes (p2p branch) to receive the link and
beat frames.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-27 17:15:28 +13:00

17 KiB
Raw Blame History

v2 bench test: one S3 controller and one LED driver

This walks one ESP32-S3 controller (W5500 Ethernet, INMP441 mic) and one led-driver unit through flashing, a smoke test, a lossy-link test and a beat latency measurement. Nothing here has been run on hardware yet, so treat the expected numbers as targets to check, not results.

0. What you need

  • Controller: XIAO ESP32-S3 with a W5500 module wired as in main/Kconfig.projbuild (SCLK 7, MISO 8, MOSI 9, CS 2, INT 3, RST 1), Ethernet cable to a DHCP network. The mic is optional for this test; the bench generator makes its own beats.
  • Driver: one led-driver unit (MicroPython, the p2p ESP-NOW branch) with a few LEDs. It needs no Wi-Fi network.
  • A PC on the same network. Docker with espressif/idf:v5.5.5 for building.
  • For the true latency measurement: a logic analyzer or scope with two channels and a common ground. Optional: the software echo numbers need nothing extra.

Channel rule. ESP-NOW only works when both radios use the same channel. The p2p driver uses its wifi_channel setting (default 5); the controller's default is now 5 as well (CONFIG_ESPNOW_CHANNEL in sdkconfig.defaults). If your driver uses another channel, change the controller before building:

cd v2 && docker run --rm -v "$PWD":/project -w /project espressif/idf:v5.5.5 idf.py menuconfig

then ESP-NOW config → Wi-Fi channel. Alternatively set CONFIG_ESPNOW_CHANNEL=<n> in sdkconfig.defaults (delete sdkconfig after, so it is regenerated). To use the beat marker pin, also set CONFIG_BEAT_TEST_GPIO (for example 4) in the same menu.

1. Build and flash the controller

cd v2
tools/build_web.sh
docker run --rm -v "$PWD":/project -w /project espressif/idf:v5.5.5 idf.py build

Expect Project build complete, the app well under its 2 MB slot, and build/www.bin present. Fix ownership if Docker left it root-owned: docker run --rm -v "$PWD":/project espressif/idf:v5.5.5 chown -R 0:0 /project.

Flash with the S3 on USB (check which port it is; ls /dev/ttyACM*):

docker run --rm --device=/dev/ttyACM0 -v "$PWD":/project -w /project espressif/idf:v5.5.5 idf.py -p /dev/ttyACM0 flash

This writes the bootloader, partition table, app and the web UI image. Watch the log:

docker run --rm -it --device=/dev/ttyACM0 -v "$PWD":/project -w /project espressif/idf:v5.5.5 idf.py -p /dev/ttyACM0 monitor

Look for, in order: store: ... records lines, espnow_tx: ESP-NOW up on channel N, rel: reliable link up, Ethernet Link Up, Got IP: x.x.x.x, HTTP server started.

2. First checks (replace $S3 with the IP)

export S3=192.168.1.50
curl -s http://$S3/status

Expect JSON with uptime_s, bpm, an espnow block and a link block (ver nonzero, digests increasing by one about every 1.5 s). If the mic hears music, bpm and beats_sent move too, which means beats are already being broadcast. Open http://$S3/ in a browser: the UI should load, and the dev tools network tab should show Content-Encoding: gzip on /static/*.js.

curl -s -D - -o /dev/null -H 'Accept-Encoding: gzip' http://$S3/static/presets.js | grep -i -E 'content-encoding|content-type'

3. Load data and add a test zone

Import the Pi database (the files hold your real zones and presets, so use a copy if you want a smaller test):

for n in device zone preset profile palette pattern sequence scene; do
  curl -s -X PUT --data-binary @../db/$n.json http://$S3/db/$n; echo " $n"
done

The driver registers itself: it broadcasts a hello at boot, the controller adds it to the device list under its MAC and sends it its group list. Check with curl -s http://$S3/devices/<mac> once the driver is running (step 4). Zones are the drivers' groups: a zone's names must contain the driver's name exactly (led-<mac> by default, see its settings.json).

curl -s -X POST -d '{"name":"bench","names":["led-aabbccddeeff"]}' http://$S3/zones
curl -s -X POST -d '{"name":"bench red","pattern":"on","colors":["#ff0000"],"delay":100,"brightness":255,"auto":true}' http://$S3/presets

Note the returned ids (<zone> and <preset> below). Creating or changing a zone re-sends group lists to every registered ESP-NOW driver.

4. Prepare the driver

The driver runs the p2p branch of led-driver plus these files from this work: src/beat_sync.py, src/espnow_rx.py, and changed src/main.py and src/presets.py. Back up the device's own copies first (mpremote fs cp :main.py ./main.py.bak), then copy the new ones (mpremote fs cp src/main.py :main.py and so on) and reset.

Optional keys in the driver's settings.json for the later tests:

key value meaning
bench true echo every fired beat to the controller and print stats every 50 beats
beat_pin e.g. 5 GPIO toggled on every fired beat, for the logic analyzer
espnow_loss 0 percent of received frames the driver discards (set later)

After boot, the controller's /devices list should contain the driver's MAC and /status should show link.acked rising (the group list was acknowledged). mpremote interrupts the running program, so reset the driver after using it.

5. Smoke test

curl -s -X POST -d '{"preset_id":"<preset>"}' http://$S3/zones/<zone>/play

The driver's console should log the preset and select it received (once each, although the controller sends every state frame twice), and the LEDs should go red within a fraction of a second. Then:

curl -s -X POST http://$S3/zones/<zone>/stop     # LEDs off
curl -s http://$S3/devices/status                # last_ack_ms_ago / failed_messages per registered driver

Zone play is a group broadcast, so devices/status only changes for acked (unicast) traffic. Trigger one with a targeted push:

curl -s -X POST -d '{"sequence":[{"v":"1","select":{"led-aabbccddeeff":["<preset>"]}}],"targets":["aabbccddeeff"]}' http://$S3/presets/push
curl -s http://$S3/devices/status

Expected: last_ack_ms_ago is small (tens of ms) and failed_messages is 0. /status shows link.acked increased.

The loss mode discards a percentage of frames before the radio on the controller (all outgoing frames, and incoming acks and echoes). The driver can discard received frames too. This makes the retry, fragment and resync layers do real work without moving antennas.

# controller: 40% loss, beats included
curl -s -X POST -d '{"percent":40,"beats":true}' http://$S3/test/loss
curl -s http://$S3/status        # sim_dropped counts dropped frames

Run these and compare with the table:

test how expected
Acked unicast under loss repeat the targeted push from step 5 twenty times at 40% (give each a different select step so you can tell them apart on the driver console) all reach the driver; link.retransmits rises. A message needs its data and its ack to both survive, and there are 5 sends in total, so about 1 in 10 is reported as timed out even though the driver usually has it
Give up and report set percent to 100, push once, wait 2 s link.ack_timeouts +1 and devices/status shows failed_messages: 1
Lost broadcast, digest recovery percent 100, play a different preset on the zone, then set percent 0 the LEDs do not change at first, then within about 1.5 to 5 s the driver sends a resync and shows the new preset; link.resyncs +1
Driver-side loss set "espnow_loss": 40 in the driver's settings.json (restore it afterwards), controller loss 0, push twenty times every push is processed once on the driver (retransmissions are de-duplicated); link.retransmits rises. Lost acks are covered by the controller-side loss rows above, because that setting also drops incoming acks
Multi-message send POST /presets/send with 10+ preset ids, save: false and targets the controller packs the presets into a few binary messages (14 presets took 2); all arrive under loss, with retransmits. save: true makes the driver write its presets.json, so restore that file afterwards. Frame-level fragmentation exists in the link but nothing in the current API produces a message over one frame, so it is only covered by the driver's unit tests

Set loss back to 0 when done: curl -s -X POST -d '{"percent":0}' http://$S3/test/loss.

Reading the numbers: the controller drops a reply with the same probability as a request, so under loss an unicast message needs both the data and the ack to survive. Resync requests are also subject to loss and to the driver's 3 s minimum gap, so recovery may take a few digests.

Results of the first run (S3 to one p2p driver, channel 5)

test result
20 acked pushes, controller loss 40% 20 of 20 reached the driver; 19 acked, 1 reported timed out (its ack was lost); 19 retransmits; 34 frames dropped by the test mode
1 push, loss 100% gave up after 2.0 s with 4 retransmits; failed_messages 1; driver heard nothing
zone play during a 100% outage, then loss back to 0 driver was untouched during the outage; 1.5 s after the link returned it received its groups, the preset and the select, one resync, and the new state
20 acked pushes, driver-side loss 40% 20 of 20 processed exactly once; 14 retransmits; 0 timeouts
/presets/send of 14 presets at driver loss 40% 2 messages (209 and 217 bytes), both acked, 1 retransmit

7. Beat latency and delivery

Software numbers (no equipment)

Start synthetic beats with the driver in bench mode:

curl -s -X POST -d '{"bpm":120,"count":200}' http://$S3/test/beats
sleep 105
curl -s http://$S3/test/beats

Results include:

  • beats_sent, echoes and hit_pct_total: the share of beats each driver fired on and reported. With no loss expect 100%.
  • rtt_minus_lead_min/avg/max_us: time from the first beat copy going out to the driver's echo coming back, minus the 30 ms the driver deliberately waits (CONFIG_BEAT_LEAD_MS). This is out-airtime plus the driver's MicroPython work plus return airtime, so it overstates one-way delay. Expect roughly 2 to 15 ms; the maximum shows the worst case you will see on the LEDs.
  • spread_avg_us and spread_max_us: only meaningful with two or more drivers; shows how far apart they fire for the same beat.

On the driver's serial console every 50 beats you get the true driver-side counts, for example beat {'fires': 100, 'dups': 100, 'late': 0, 'freerun': 0}:

  • fires: beats it acted on.
  • dups: second copies dropped (with the double send this should be close to fires).
  • freerun: beats it generated itself because both copies were missed. Zero without loss.
  • late: frames rejected as too soon or old.

Delivery under loss

Repeat the run with loss on:

curl -s -X POST -d '{"percent":30,"beats":true}' http://$S3/test/loss
curl -s -X POST -d '{"bpm":120,"count":200}' http://$S3/test/beats

Judge delivery from the driver's own fires and freerun counts: with two copies the chance of missing both is about 9% at 30% loss, and the free-run covers those, so fires should stay near beats_sent with freerun around 9% of it. The controller's hit_pct_total is lower because echoes are dropped by the same loss setting; it is a lower bound, roughly (1 - loss) × delivery.

Latency over USB (no extra equipment)

Both boards are on USB, so the PC can timestamp one event line from each and compare them. Settings already applied for this: the driver has "usb_log": true in settings.json. The controller prints BT <beat> <microseconds> for every synthetic beat; the driver prints FIRE <beat> <ticks_us> when it fires.

python3 tools/usb_latency.py --ctrl-port /dev/ttyACM0 --drv-port /dev/ttyACM1 --url http://$S3 --beats 200

The script starts the generator, reads both ports, and maps each board's clock onto the PC's by using the fastest USB deliveries per 5 s window (this also removes crystal drift). USB serial delivers in bursts, so single lines arrive up to about 50 ms late; the fastest ones are close to on time, which is what the mapping relies on. The remaining error is the difference between the two ports' fastest delivery, roughly 1 ms, so read results to about a millisecond. The script prints the delivery jitter so you can see how noisy the ports were. The driver's FIRE time is when the beat triggers the pattern step; the LED data follows within the same loop pass, and a 119-LED strip takes about 3.6 ms to clock out.

Close any serial monitor first (only one program can hold a port). Do not run long mpremote exec commands on the driver during the capture (see Troubleshooting).

True latency with a logic analyzer

For a measurement that does not depend on USB. Marker pins are set up already: the controller toggles GPIO4 (D3) when the first copy of each beat goes out (CONFIG_BEAT_TEST_GPIO=4, in sdkconfig.defaults) and the driver toggles GPIO4 (D3) when it fires ("beat_pin": 4 in its settings.json). Both are XIAO ESP32S3 boards, where D3 is free.

Wiring (3.3 V logic, any 2 or 3 channel analyzer):

analyzer connect to
GND GND on the controller and GND on the driver (one common ground)
channel 0 controller D3
channel 1 driver D3
channel 2 (optional) driver D10 (GPIO9, the LED data line) to see when the LEDs really update

Steps:

  1. Make sure the driver runs a preset that changes on every beat if you use channel 2, for example a chase preset with auto off, selected on the zone. With the plain on preset the LED data line stays quiet.
  2. Start a capture: sample rate at least 1 MS/s, length at least 60 s (edges only; it stays small), trigger on any edge of channel 0.
  3. Start the beats: curl -s -X POST -d '{"bpm":120,"count":100}' http://$S3/test/beats. Both markers toggle on every beat, so you will see edges 500 ms apart.
  4. Stop the capture when the generator finishes (about 52 s) and export the digital data as CSV (time in seconds, one column per channel).
  5. Analyze: python3 tools/analyze_edges.py capture.csv --ctrl 0 --drv 1 --time-unit s. Use --lead-ms if you changed CONFIG_BEAT_LEAD_MS. It pairs each controller edge with the next driver edge and prints the delay beyond the 30 ms lead, with min, mean, median, p95, max and jitter, plus the count of unmatched beats. On the scope, channel 2 shows the true light delay: the first data burst after the driver edge.

Targets from the plan: under about 10 ms from the controller edge to the driver edge (after subtracting the lead).

Return to normal use by deleting beat_pin and usb_log from the driver's settings.json and setting CONFIG_BEAT_TEST_GPIO=-1 (the marker pin otherwise toggles on every beat, including live mic beats).

Results of the first run (software numbers, one driver, channel 5)

Synthetic beats at 120 BPM, 160 beats per run, mic beats muted while the generator runs, driver in bench mode.

no loss 30% loss (beats included)
beats the driver acted on, per beat sent 1.00 1.00
beats the driver invented itself (free-run) 0% 11% (theory for two independent copies: 9%)
second copies dropped as duplicates 100% 47% (theory: 49%)
echoes the controller received 162 of 162 96 of 161 (echoes are lost at 30% too, and invented beats do not echo)
round trip minus the 30 ms lead, min / avg / max 9.8 / 15.0 / 25.7 ms 3.4 / 16.2 / 26.0 ms

Reading it: the double send plus free-run gave the driver a beat every time, even with 30% of frames lost. The round-trip figure includes the driver's loop and the echo's trip back, so it is an upper bound on the delay the LEDs see; it is not the controller-to-LED latency. The USB and logic analyzer measurements below give the real figure.

Latency over USB (200 beats at 120 BPM, one driver, 119 LEDs): delay beyond the 30 ms lead min 0.49, mean 2.07, median 1.77, p95 4.41, max 7.95 ms (std 1.3 ms; accuracy about 1 ms). 197 of 200 beats matched. Both clocks drifted by 60 to 80 ppm against the PC, and individual USB lines arrived up to 54 ms late, which is why the script uses the fastest deliveries. This is the time from the controller sending a beat to the driver firing, beyond the built-in lead. The logic analyzer run has not been done.

8. Troubleshooting

  • Running a long mpremote exec on the driver resets it: the driver's watchdog (10 s) keeps running after mpremote interrupts main.py, so keep REPL experiments under about 8 s and reset the driver afterwards.

  • Driver never reacts, no acks: channel mismatch. Check CONFIG_ESPNOW_CHANNEL against the driver's wifi_channel. Neighbouring channels can still work at bench range, so a working link does not prove they match.

  • Zone play logs nothing on the driver: the driver is not in the zone's group (silent). Check the zone's names against the driver's name.

  • Devices show last_ack_ms_ago: -1: the device record has no usable 12-character MAC in address or as its key.

  • beats_sent rises but echoes stays 0: the driver is not in bench mode, or it has not yet heard a controller frame (it learns the controller's MAC from the first data or digest frame; wait a few seconds).

  • Resync never triggers: the driver ignores a digest equal to its own version; confirm with /status that link.ver changed after the missed play.

  • Web UI is blank: the www image was not built before flashing; run tools/build_web.sh and flash again.

9. Cleanup

curl -s -X POST -d '{"percent":0}' http://$S3/test/loss
curl -s -X POST http://$S3/test/beats/stop

Set the driver's bench and espnow_loss back to off for normal use.