tools/driver_ota.py (make ota-drivers) pushes the driver's .py files through the controller, one file per POST /drivers/ota/file. Drivers skip files they already have, and the rest are committed together. - one target: acked unicast, two chunks in flight - several targets: each file is broadcast once, then each driver is asked for its missing chunks and repaired by unicast - rel: rel_send_seq, rel_wait and rel_send_wait, plus a callback for the drivers' 0x26 status replies - commands are retried, because status replies are not acked - broadcast chunks are paced 50 ms apart: faster broadcast left C3 drivers unable to receive until reset - bench results in docs/BENCH_TEST.md, endpoints in docs/API.md Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
25 KiB
Bench test: one S3 controller and one LED driver
This walks one ESP32-S3 controller (W5500 Ethernet, INMP441 mic) and one led-driver unit through flashing, a smoke test, a lossy-link test and a beat latency measurement. Nothing here has been run on hardware yet, so treat the expected numbers as targets to check, not results.
0. What you need
- Controller: XIAO ESP32-S3 with a W5500 module wired as in
main/Kconfig.projbuild(SCLK 7, MISO 8, MOSI 9, CS 2, INT 3, RST 1), Ethernet cable to a DHCP network. The mic is optional for this test; the bench generator makes its own beats. - Driver: one
led-driverunit (MicroPython, thep2pESP-NOW branch) with a few LEDs. It needs no Wi-Fi network. - A PC on the same network. Docker with
espressif/idf:v5.5.5for building. - For the true latency measurement: a logic analyzer or scope with two channels and a common ground. Optional: the software echo numbers need nothing extra.
Channel rule. ESP-NOW only works when both radios use the same channel. The p2p driver uses its wifi_channel setting (default 5); the controller's default is now 5 as well (CONFIG_ESPNOW_CHANNEL in sdkconfig.defaults). If your driver uses another channel, change the controller before building:
docker run --rm -v "$PWD":/project -w /project espressif/idf:v5.5.5 idf.py menuconfig
then ESP-NOW config → Wi-Fi channel. Alternatively set CONFIG_ESPNOW_CHANNEL=<n> in sdkconfig.defaults (delete sdkconfig after, so it is regenerated). To use the beat marker pin, also set CONFIG_BEAT_TEST_GPIO (for example 4) in the same menu.
1. Build and flash the controller
The repository root has a Makefile that wraps all of this: make build, make flash, make monitor (Ctrl-] to quit), make log (console for 10 s, no terminal needed), make test, and make help for the rest; PORT=/dev/ttyACM1 selects another port. The manual commands below do the same thing.
tools/build_web.sh
docker run --rm -v "$PWD":/project -w /project espressif/idf:v5.5.5 idf.py build
Expect Project build complete, the app well under its 2 MB slot, with the web UI embedded (www_dist/www.pack). Fix ownership if Docker left it root-owned: docker run --rm -v "$PWD":/project espressif/idf:v5.5.5 chown -R 0:0 /project.
Flash with the S3 on USB (check which port it is; ls /dev/ttyACM*):
docker run --rm --device=/dev/ttyACM0 -v "$PWD":/project -w /project espressif/idf:v5.5.5 idf.py -p /dev/ttyACM0 flash
This writes the bootloader, partition table and app (the web UI is part of the app). Watch the log:
docker run --rm -it --device=/dev/ttyACM0 -v "$PWD":/project -w /project espressif/idf:v5.5.5 idf.py -p /dev/ttyACM0 monitor
Look for, in order: store: ... records lines, espnow_tx: ESP-NOW up on channel N, rel: reliable link up, Ethernet Link Up, Got IP: x.x.x.x, HTTP server started.
2. First checks (replace $S3 with the IP)
export S3=192.168.1.50
curl -s http://$S3/status
Expect JSON with uptime_s, bpm, an espnow block and a link block (ver nonzero, digests increasing by one about every 1.5 s). If the mic hears music, bpm and beats_sent move too, which means beats are already being broadcast. Open http://$S3/ in a browser: the UI should load, and the dev tools network tab should show Content-Encoding: gzip on /static/*.js.
curl -s -D - -o /dev/null -H 'Accept-Encoding: gzip' http://$S3/static/presets.js | grep -i -E 'content-encoding|content-type'
Audio and beats. http://$S3/audio is the mic page: live levels, tempo and bar position, a tap button to correct beat 1, and a Listen button for the 16 kHz mono stream. The underlying endpoints are /audio.wav (one listener at a time), /beats (server-sent events: beat, onset, tempo), POST /tap and /time. /status carries the mic levels (mic_rms_dbfs, mic_spl_db) and the beat state (bpm, beat_locked, meter, bar_beat, bar_sure).
3. Load data and add a test zone
Import the Pi database (the files hold your real zones and presets, so use a copy if you want a smaller test):
for n in device zone preset profile palette pattern sequence scene; do
curl -s -X PUT --data-binary @../db/$n.json http://$S3/db/$n; echo " $n"
done
The driver registers itself: it broadcasts a hello at boot, the controller adds it to the device list under its MAC and sends it its group list. Check with curl -s http://$S3/devices/<mac> once the driver is running (step 4). Zones are the drivers' groups: a zone's names must contain the driver's name exactly (led-<mac> by default, see its settings.json).
curl -s -X POST -d '{"name":"bench","names":["led-aabbccddeeff"]}' http://$S3/zones
curl -s -X POST -d '{"name":"bench red","pattern":"on","colors":["#ff0000"],"delay":100,"brightness":255,"auto":true}' http://$S3/presets
Note the returned ids (<zone> and <preset> below). Creating or changing a zone re-sends group lists to every registered ESP-NOW driver.
4. Prepare the driver
The driver runs the p2p branch of led-driver plus these files from this work: src/beat_sync.py, src/espnow_rx.py, and changed src/main.py and src/presets.py. Back up the device's own copies first (mpremote fs cp :main.py ./main.py.bak), then copy the new ones (mpremote fs cp src/main.py :main.py and so on) and reset.
Optional keys in the driver's settings.json for the later tests:
| key | value | meaning |
|---|---|---|
bench |
true |
echo every fired beat to the controller and print stats every 50 beats |
beat_pin |
e.g. 5 |
GPIO toggled on every fired beat, for the logic analyzer |
espnow_loss |
0 |
percent of received frames the driver discards (set later) |
After boot, the controller's /devices list should contain the driver's MAC and /status should show link.acked rising (the group list was acknowledged). mpremote interrupts the running program, so reset the driver after using it.
5. Smoke test
curl -s -X POST -d '{"preset_id":"<preset>"}' http://$S3/zones/<zone>/play
The driver's console should log the preset and select it received (once each, although the controller sends every state frame twice), and the LEDs should go red within a fraction of a second. Then:
curl -s -X POST http://$S3/zones/<zone>/stop # LEDs off
curl -s http://$S3/devices/status # last_ack_ms_ago / failed_messages per registered driver
Zone play is a group broadcast, so devices/status only changes for acked (unicast) traffic. Trigger one with a targeted push:
curl -s -X POST -d '{"sequence":[{"v":"1","select":{"led-aabbccddeeff":["<preset>"]}}],"targets":["aabbccddeeff"]}' http://$S3/presets/push
curl -s http://$S3/devices/status
Expected: last_ack_ms_ago is small (tens of ms) and failed_messages is 0. /status shows link.acked increased.
6. Lossy-link test
The loss mode discards a percentage of frames before the radio on the controller (all outgoing frames, and incoming acks and echoes). The driver can discard received frames too. This makes the retry, fragment and resync layers do real work without moving antennas.
# controller: 40% loss, beats included
curl -s -X POST -d '{"percent":40,"beats":true}' http://$S3/test/loss
curl -s http://$S3/status # sim_dropped counts dropped frames
Run these and compare with the table:
| test | how | expected |
|---|---|---|
| Acked unicast under loss | repeat the targeted push from step 5 twenty times at 40% (give each a different select step so you can tell them apart on the driver console) | all reach the driver; link.retransmits rises. A message needs its data and its ack to both survive, and there are 5 sends in total, so about 1 in 10 is reported as timed out even though the driver usually has it |
| Give up and report | set percent to 100, push once, wait 2 s |
link.ack_timeouts +1 and devices/status shows failed_messages: 1 |
| Lost broadcast, digest recovery | percent 100, play a different preset on the zone, then set percent 0 |
the LEDs do not change at first, then within about 1.5 to 5 s the driver sends a resync and shows the new preset; link.resyncs +1 |
| Driver-side loss | set "espnow_loss": 40 in the driver's settings.json (restore it afterwards), controller loss 0, push twenty times |
every push is processed once on the driver (retransmissions are de-duplicated); link.retransmits rises. Lost acks are covered by the controller-side loss rows above, because that setting also drops incoming acks |
| Multi-message send | POST /presets/send with 10+ preset ids, save: false and targets |
the controller packs the presets into a few binary messages (14 presets took 2); all arrive under loss, with retransmits. save: true makes the driver write its presets.json, so restore that file afterwards. Frame-level fragmentation exists in the link but nothing in the current API produces a message over one frame, so it is only covered by the driver's unit tests |
Set loss back to 0 when done: curl -s -X POST -d '{"percent":0}' http://$S3/test/loss.
Reading the numbers: the controller drops a reply with the same probability as a request, so under loss an unicast message needs both the data and the ack to survive. Resync requests are also subject to loss and to the driver's 3 s minimum gap, so recovery may take a few digests.
Results of the first run (S3 to one p2p driver, channel 5)
| test | result |
|---|---|
| 20 acked pushes, controller loss 40% | 20 of 20 reached the driver; 19 acked, 1 reported timed out (its ack was lost); 19 retransmits; 34 frames dropped by the test mode |
| 1 push, loss 100% | gave up after 2.0 s with 4 retransmits; failed_messages 1; driver heard nothing |
| zone play during a 100% outage, then loss back to 0 | driver was untouched during the outage; 1.5 s after the link returned it received its groups, the preset and the select, one resync, and the new state |
| 20 acked pushes, driver-side loss 40% | 20 of 20 processed exactly once; 14 retransmits; 0 timeouts |
/presets/send of 14 presets at driver loss 40% |
2 messages (209 and 217 bytes), both acked, 1 retransmit |
7. Beat latency and delivery
Software numbers (no equipment)
Start synthetic beats with the driver in bench mode:
curl -s -X POST -d '{"bpm":120,"count":200}' http://$S3/test/beats
sleep 105
curl -s http://$S3/test/beats
Results include:
beats_sent,echoesandhit_pct_total: the share of beats each driver fired on and reported. With no loss expect 100%.rtt_minus_lead_min/avg/max_us: time from the first beat copy going out to the driver's echo coming back, minus the 30 ms the driver deliberately waits (CONFIG_BEAT_LEAD_MS). This is out-airtime plus the driver's MicroPython work plus return airtime, so it overstates one-way delay. Expect roughly 2 to 15 ms; the maximum shows the worst case you will see on the LEDs.spread_avg_usandspread_max_us: only meaningful with two or more drivers; shows how far apart they fire for the same beat.
On the driver's serial console every 50 beats you get the true driver-side counts, for example beat {'fires': 100, 'dups': 100, 'late': 0, 'freerun': 0}:
fires: beats it acted on.dups: second copies dropped (with the double send this should be close tofires).freerun: beats it generated itself because both copies were missed. Zero without loss.late: frames rejected as too soon or old.
Delivery under loss
Repeat the run with loss on:
curl -s -X POST -d '{"percent":30,"beats":true}' http://$S3/test/loss
curl -s -X POST -d '{"bpm":120,"count":200}' http://$S3/test/beats
Judge delivery from the driver's own fires and freerun counts: with two copies the chance of missing both is about 9% at 30% loss, and the free-run covers those, so fires should stay near beats_sent with freerun around 9% of it. The controller's hit_pct_total is lower because echoes are dropped by the same loss setting; it is a lower bound, roughly (1 - loss) × delivery.
Latency over USB (no extra equipment)
Both boards are on USB, so the PC can timestamp one event line from each and compare them. Settings already applied for this: the driver has "usb_log": true in settings.json. The controller prints BT <beat> <microseconds> for every synthetic beat; the driver prints FIRE <beat> <ticks_us> when it fires.
python3 tools/usb_latency.py --ctrl-port /dev/ttyACM0 --drv-port /dev/ttyACM1 --url http://$S3 --beats 200
The script starts the generator, reads both ports, and maps each board's clock onto the PC's by using the fastest USB deliveries per 5 s window (this also removes crystal drift). USB serial delivers in bursts, so single lines arrive up to about 50 ms late; the fastest ones are close to on time, which is what the mapping relies on. The remaining error is the difference between the two ports' fastest delivery, roughly 1 ms, so read results to about a millisecond. The script prints the delivery jitter so you can see how noisy the ports were. The driver's FIRE time is when the beat triggers the pattern step; the LED data follows within the same loop pass, and a 119-LED strip takes about 3.6 ms to clock out.
Close any serial monitor first (only one program can hold a port). Do not run long mpremote exec commands on the driver during the capture (see Troubleshooting).
True latency with a logic analyzer
For a measurement that does not depend on USB. Marker pins are set up already: the controller toggles GPIO4 (D3) when the first copy of each beat goes out (CONFIG_BEAT_TEST_GPIO=4, in sdkconfig.defaults) and the driver toggles GPIO4 (D3) when it fires ("beat_pin": 4 in its settings.json). Both are XIAO ESP32S3 boards, where D3 is free.
Wiring (3.3 V logic, any 2 or 3 channel analyzer):
| analyzer | connect to |
|---|---|
| GND | GND on the controller and GND on the driver (one common ground) |
| channel 0 | controller D3 |
| channel 1 | driver D3 |
| channel 2 (optional) | driver D10 (GPIO9, the LED data line) to see when the LEDs really update |
Steps:
- Make sure the driver runs a preset that changes on every beat if you use channel 2, for example a
chasepreset withautooff, selected on the zone. With the plainonpreset the LED data line stays quiet. - Start a capture: sample rate at least 1 MS/s, length at least 60 s (edges only; it stays small), trigger on any edge of channel 0.
- Start the beats:
curl -s -X POST -d '{"bpm":120,"count":100}' http://$S3/test/beats. Both markers toggle on every beat, so you will see edges 500 ms apart. - Stop the capture when the generator finishes (about 52 s) and export the digital data as CSV (time in seconds, one column per channel).
- Analyze:
python3 tools/analyze_edges.py capture.csv --ctrl 0 --drv 1 --time-unit s. Use--lead-msif you changedCONFIG_BEAT_LEAD_MS. It pairs each controller edge with the next driver edge and prints the delay beyond the 30 ms lead, with min, mean, median, p95, max and jitter, plus the count of unmatched beats. On the scope, channel 2 shows the true light delay: the first data burst after the driver edge.
Targets from the plan: under about 10 ms from the controller edge to the driver edge (after subtracting the lead).
Return to normal use by deleting beat_pin and usb_log from the driver's settings.json and setting CONFIG_BEAT_TEST_GPIO=-1 (the marker pin otherwise toggles on every beat, including live mic beats).
Results of the first run (software numbers, one driver, channel 5)
Synthetic beats at 120 BPM, 160 beats per run, mic beats muted while the generator runs, driver in bench mode.
| no loss | 30% loss (beats included) | |
|---|---|---|
| beats the driver acted on, per beat sent | 1.00 | 1.00 |
| beats the driver invented itself (free-run) | 0% | 11% (theory for two independent copies: 9%) |
| second copies dropped as duplicates | 100% | 47% (theory: 49%) |
| echoes the controller received | 162 of 162 | 96 of 161 (echoes are lost at 30% too, and invented beats do not echo) |
| round trip minus the 30 ms lead, min / avg / max | 9.8 / 15.0 / 25.7 ms | 3.4 / 16.2 / 26.0 ms |
Reading it: the double send plus free-run gave the driver a beat every time, even with 30% of frames lost. The round-trip figure includes the driver's loop and the echo's trip back, so it is an upper bound on the delay the LEDs see; it is not the controller-to-LED latency. The USB and logic analyzer measurements below give the real figure.
Latency over USB (200 beats at 120 BPM, one driver, 119 LEDs): delay beyond the 30 ms lead min 0.49, mean 2.07, median 1.77, p95 4.41, max 7.95 ms (std 1.3 ms; accuracy about 1 ms). 197 of 200 beats matched. Both clocks drifted by 60 to 80 ppm against the PC, and individual USB lines arrived up to 54 ms late, which is why the script uses the fastest deliveries. This is the time from the controller sending a beat to the driver firing, beyond the built-in lead. The logic analyzer run has not been done.
ESP32-C3 driver compared with the S3 driver
Driver: XIAO ESP32-C3 (MAC 58:8c:81:a3:74:58), MicroPython v1.22.1 stock (no ws2812-dma patch), 160 MHz, about 128 KB free heap, 119 LEDs on pin 10. Same tests as above, same controller.
| S3 driver, no loss | C3 driver, no loss | |
|---|---|---|
| echoes the controller received | 162 of 162 | 160 of 160 |
| round trip minus the 30 ms lead, min / avg / max | 9.8 / 15.0 / 25.7 ms | 2.6 / 4.5 / 17.8 ms |
C3 with 30% loss (160 beats): every beat produced exactly one pattern step, 15 of the 160 (9%, theory 9%) came from free-run, and 68 second copies were dropped as duplicates; the controller got 99 of 160 echoes. That matches the S3 result in the table above.
Reading the free-run counter: after the generator stops, the driver keeps stepping for up to 4 more beats on its last period. A stats window that runs past the end of the test therefore shows 4 extra "free-run" fires even with no loss. They are not missed beats. (My first read of the C3's no-loss run said otherwise; that was wrong.)
Latency over USB, same settings on both boards (usb_log only), 200 beats at 120 BPM, delay beyond the 30 ms lead, accuracy about 1 ms:
| matched | min | mean | median | p95 | max | std | |
|---|---|---|---|---|---|---|---|
| S3 driver (240 MHz) | 200 / 200 | -2.4 | 2.15 | 2.03 | 6.03 | 12.3 | 2.5 ms |
| C3 driver (160 MHz) | 200 / 200 | 0.2 | 3.36 | 3.06 | 7.47 | 19.7 | 2.3 ms |
The C3 is about 1.2 ms slower on average and has a longer tail (maximum 19.7 ms against 12.3 ms). With the method's 1 ms accuracy, the mean difference is small and the tail difference is the more telling one. Both are well inside a 120 BPM beat. The echo round trip disagrees with this (S3 15 ms mean, C3 4.5 ms). The S3 echo runs had usb_log and a marker pin switched on and the C3 runs did not, and I did not rerun the S3 echo test with identical settings, so treat the echo figures as unexplained and prefer the USB ones.
Pattern frame rate (strip writes per second, 119 LEDs, delay 1 ms, 2 s each; only patterns that wrote anything are shown):
| pattern | C3 (160 MHz, 115 KB free) | S3 (240 MHz, 8 MB free) |
|---|---|---|
| flame | 22 | 33 |
| palette_morph | 21 | 32 |
| circle | 150 | 170 |
The CPU-heavy patterns run about 1.5 times faster on the S3, which is the clock ratio; a light pattern is limited by the strip write and differs by about 13%. aurora and chase wrote nothing with the parameters I used, so they are not measured.
Groups: the controller sends a driver its group list when the driver announces itself, but now only if the driver is in at least one zone. A driver in no zone keeps the groups it already has (an empty list from the controller used to wipe them). An explicit zone or device edit still re-sends every list, including empty ones.
Wi-Fi start-up order on the C3. With the extra link code loaded before Wi-Fi start-up, the C3 failed with Wifi Unknown Error 0x0101 on about every other cold boot (2 of 4). Starting ESP-NOW before the pattern library and the LED driver claim memory fixed it (6 of 6 boots). led-driver/src/main.py now does this.
8. Driver app update over ESP-NOW
Pushes the led-driver .py files from the PC, through the controller, to the drivers. The drivers need espnow_ota.py first, so install the code once over USB (led-cli -p <port> --all). After that:
python3 tools/driver_ota.py $S3 --mac <mac> --dry-run # list the files and drivers, send nothing
python3 tools/driver_ota.py $S3 --mac <mac> # one driver
make ota-drivers HOST=$S3 BROADCAST=1 # every ESP-NOW driver, one broadcast and repairs
Each file prints written, or nothing when the driver already has it. Drivers that received files reset after the commit, and the script waits for them to ack the link again. --bad-sha <device path> sends a wrong hash for one file, to test the abort path.
| Check | Expected |
|---|---|
| Push with nothing changed | every file skipped, no reset |
| Edit one pattern, push | only that file written; the driver resets, comes back and runs the new code; file_hashes.json has the new hash |
--bad-sha patterns/<file>.py |
ERROR badhash and aborted:; no .new files left on the driver and the old file unchanged |
Controller loss at 20% (beats: false) |
still completes, slower, with +n repaired on some files |
--broadcast with two or more drivers |
each changed file is sent once to all of them; afterwards every driver still acks and fires beats |
Results of the first run (S3 controller, two C3 drivers with 119 LEDs, channel 5)
| result | |
|---|---|
| 62 files, nothing changed | all skipped in about 7 s (about 110 ms per file, mostly the driver hashing its copy), no reset |
| One changed pattern (7.3 KB) | written and committed; driver reset and acked again; content and manifest checked over USB |
| 60 KB file, unicast | 4.6 s, 12.8 KB/s including both hash checks |
| 60 KB file, unicast, 20% controller loss | 4 of 4 runs completed, 3.8 to 8.3 KB/s |
| Wrong hash | badhash, aborted, nothing staged or committed |
| 60 KB file, broadcast to both | 4.2 s, about 14 KB/s per driver (29 KB/s in total), usually no repairs (once 3) |
| 60 KB file, broadcast to both, 20% loss | 2 of 2 completed in about 33 s; 33 to 58 of the 65 chunks needed a repair (each chunk is 4 frames with no radio retries) |
Beat echo round trip after the change to esp.irq(), 177 beats, two drivers |
2.4 / 5.5 / 23.5 ms (min / avg / max) beyond the 30 ms lead, against 9.8 / 15.0 / 25.7 ms when the driver polled; 176 of 177 echoed |
Broadcast pacing. With 12 ms between broadcast chunks (about 30 KB/s) the drivers stopped receiving ESP-NOW frames altogether. Their esp.stats() receive count froze, with no drops counted, while the controller's unicasts were still acked by the drivers' radios. It lasted until a reset, more than 5 minutes once, and beats stopped too. The likely cause is the C3 writing to flash while the frames keep coming. At 50 ms (BCAST_GAP_MS in main/drv_ota.c) it did not happen in 5 runs.
9. Troubleshooting
-
A board on a USB-serial/JTAG port (C3, and the S3 controller) resets whenever a program opens its port, and a C3 can land in ROM download mode (
wait usb download). The old aptmpremote0.4.0 hangs on this. Use the newer one in~/projects/micropython/micropython/tools/mpremote/mpremote.py, orled-tool, and note thatfs cp -rfails on drivers that print timestamps (copy named files instead). -
Only one program can use a board's serial port at a time. A serial logger left running while
led-cliuploads corrupts the transfer silently:led-clireports the files uploaded, but the old ones stay on the driver. -
Running a long
mpremote execon the driver resets it: the driver's watchdog (10 s) keeps running aftermpremoteinterruptsmain.py, so keep REPL experiments under about 8 s and reset the driver afterwards. -
Driver never reacts, no acks: channel mismatch. Check
CONFIG_ESPNOW_CHANNELagainst the driver'swifi_channel. Neighbouring channels can still work at bench range, so a working link does not prove they match. -
Zone play logs nothing on the driver: the driver is not in the zone's group (silent). Check the zone's
namesagainst the driver'sname. -
Devices show
last_ack_ms_ago: -1: the device record has no usable 12-character MAC inaddressor as its key. -
beats_sentrises butechoesstays 0: the driver is not inbenchmode, or it has not yet heard a controller frame (it learns the controller's MAC from the first data or digest frame; wait a few seconds). -
Resync never triggers: the driver ignores a digest equal to its own version; confirm with
/statusthatlink.verchanged after the missed play. -
Web UI is blank: the web UI was not packed before the build; run
tools/build_web.shand build again (make builddoes this).
10. Cleanup
curl -s -X POST -d '{"percent":0}' http://$S3/test/loss
curl -s -X POST http://$S3/test/beats/stop
Set the driver's bench and espnow_loss back to off for normal use.