Files
jimmyandClaude Sonnet 5.5 6e2d5ff80c chore: build with espressif/idf v6.1
Makefile and the bench-test doc use the v6.1 image. dependencies.lock came out of the build as idf 5.5.5 /
littlefs 2.0.0, which does not match the image: regenerate it with a v6.1 build.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-12 00:46:11 +13:00

22 KiB
Raw Permalink Blame History

Bench test: one S3 controller and one LED driver

This walks one ESP32-S3 controller (W5500 Ethernet, INMP441 mic) and one led-driver unit through flashing, a smoke test, a lossy-link test and a beat latency measurement. Nothing here has been run on hardware yet, so treat the expected numbers as targets to check, not results.

0. What you need

  • Controller: XIAO ESP32-S3 with a W5500 module wired as in main/Kconfig.projbuild (SCLK 7, MISO 8, MOSI 9, CS 2, INT 3, RST 1), Ethernet cable to a DHCP network. The mic is optional for this test; the bench generator makes its own beats.
  • Driver: one led-driver unit (MicroPython, the p2p ESP-NOW branch) with a few LEDs. It needs no Wi-Fi network.
  • A PC on the same network. Docker with espressif/idf:v6.1 for building.
  • For the true latency measurement: a logic analyzer or scope with two channels and a common ground. Optional: the software echo numbers need nothing extra.

Channel rule. ESP-NOW only works when both radios use the same channel. The p2p driver uses its wifi_channel setting (default 5); the controller's default is now 5 as well (CONFIG_ESPNOW_CHANNEL in sdkconfig.defaults). If your driver uses another channel, change the controller before building:

docker run --rm -v "$PWD":/project -w /project espressif/idf:v6.1 idf.py menuconfig

then ESP-NOW config → Wi-Fi channel. Alternatively set CONFIG_ESPNOW_CHANNEL=<n> in sdkconfig.defaults (delete sdkconfig after, so it is regenerated). To use the beat marker pin, also set CONFIG_BEAT_TEST_GPIO (for example 44) in the same menu.

1. Build and flash the controller

The repository root has a Makefile that wraps all of this: make build, make flash, make monitor (Ctrl-] to quit), make log (console for 10 s, no terminal needed), make test, and make help for the rest; PORT=/dev/ttyACM1 selects another port. The manual commands below do the same thing.


tools/build_web.sh
docker run --rm -v "$PWD":/project -w /project espressif/idf:v6.1 idf.py build

Expect Project build complete, the app well under its 2 MB slot, with the web UI embedded (www_dist/www.pack). Fix ownership if Docker left it root-owned: docker run --rm -v "$PWD":/project espressif/idf:v6.1 chown -R 0:0 /project.

Flash with the S3 on USB (check which port it is; ls /dev/ttyACM*):

docker run --rm --device=/dev/ttyACM0 -v "$PWD":/project -w /project espressif/idf:v6.1 idf.py -p /dev/ttyACM0 flash

This writes the bootloader, partition table and app (the web UI is part of the app). Watch the log:

docker run --rm -it --device=/dev/ttyACM0 -v "$PWD":/project -w /project espressif/idf:v6.1 idf.py -p /dev/ttyACM0 monitor

Look for, in order: store: ... records lines, espnow_tx: ESP-NOW up on channel N, rel: reliable link up, Ethernet Link Up, Got IP: x.x.x.x, HTTP server started.

2. First checks (replace $S3 with the IP)

export S3=192.168.1.50
curl -s http://$S3/status

Expect JSON with uptime_s, reset_reason (why the board last restarted, such as panic, task_watchdog or brownout; opening the USB serial port resets the board and replaces it with usb, so read it first), bpm, an espnow block and a link block (ver nonzero, digests increasing by one about every 1.5 s). If the mic hears music, bpm and beats_sent move too, which means beats are already being broadcast. Open http://$S3/ in a browser: the UI should load, and the dev tools network tab should show Content-Encoding: gzip on /static/*.js.

curl -s -D - -o /dev/null -H 'Accept-Encoding: gzip' http://$S3/static/presets.js | grep -i -E 'content-encoding|content-type'

Audio and beats. http://$S3/audio is the mic page: live levels, tempo and bar position, a tap button to correct beat 1, and a Listen button for the 16 kHz mono stream. The underlying endpoints are /audio.wav (one listener at a time), /beats (server-sent events: beat, onset, tempo), POST /tap and /time. /status carries the mic levels (mic_rms_dbfs, mic_spl_db) and the beat state (bpm, beat_locked, meter, bar_beat, bar_sure).

3. Load data and add a test zone

Import the Pi database (the files hold your real zones and presets, so use a copy if you want a smaller test):

for n in device zone preset profile palette pattern sequence scene; do
  curl -s -X PUT --data-binary @../db/$n.json http://$S3/db/$n; echo " $n"
done

The driver registers itself: it broadcasts a hello at boot, the controller adds it to the device list under its MAC and sends it its group list. Check with curl -s http://$S3/devices/<mac> once the driver is running (step 4). Zones are the drivers' groups: a zone's names must contain the driver's name exactly (led-<mac> by default, see its settings.json).

curl -s -X POST -d '{"name":"bench","names":["led-aabbccddeeff"]}' http://$S3/zones
curl -s -X POST -d '{"name":"bench red","pattern":"on","colors":["#ff0000"],"delay":100,"brightness":255,"auto":true}' http://$S3/presets

Note the returned ids (<zone> and <preset> below). Creating or changing a zone re-sends group lists to every registered ESP-NOW driver.

4. Prepare the driver

The driver runs the p2p branch of led-driver plus these files from this work: src/beat_sync.py, src/espnow_rx.py, and changed src/main.py and src/presets.py. Back up the device's own copies first (mpremote fs cp :main.py ./main.py.bak), then copy the new ones (mpremote fs cp src/main.py :main.py and so on) and reset.

Optional keys in the driver's settings.json for the later tests:

key value meaning
bench true echo every fired beat to the controller and print stats every 50 beats
beat_pin e.g. 5 GPIO toggled on every fired beat, for the logic analyzer
espnow_loss 0 percent of received frames the driver discards (set later)

After boot, the controller's /devices list should contain the driver's MAC and /status should show link.acked rising (the group list was acknowledged). mpremote interrupts the running program, so reset the driver after using it.

5. Smoke test

curl -s -X POST -d '{"preset_id":"<preset>"}' http://$S3/zones/<zone>/play

The driver's console should log the preset and select it received (once each, although the controller sends every state frame twice), and the LEDs should go red within a fraction of a second. Then:

curl -s -X POST http://$S3/zones/<zone>/stop     # LEDs off
curl -s http://$S3/devices/status                # last_ack_ms_ago / failed_messages per registered driver

Zone play is a group broadcast, so devices/status only changes for acked (unicast) traffic. Trigger one with a targeted push:

curl -s -X POST -d '{"sequence":[{"v":"1","select":{"led-aabbccddeeff":["<preset>"]}}],"targets":["aabbccddeeff"]}' http://$S3/presets/push
curl -s http://$S3/devices/status

Expected: last_ack_ms_ago is small (tens of ms) and failed_messages is 0. /status shows link.acked increased.

The loss mode discards a percentage of frames before the radio on the controller (all outgoing frames, and incoming acks and echoes). The driver can discard received frames too. This makes the retry, fragment and resync layers do real work without moving antennas.

# controller: 40% loss, beats included
curl -s -X POST -d '{"percent":40,"beats":true}' http://$S3/test/loss
curl -s http://$S3/status        # sim_dropped counts dropped frames

Run these and compare with the table:

test how expected
Acked unicast under loss repeat the targeted push from step 5 twenty times at 40% (give each a different select step so you can tell them apart on the driver console) all reach the driver; link.retransmits rises. A message needs its data and its ack to both survive, and there are 5 sends in total, so about 1 in 10 is reported as timed out even though the driver usually has it
Give up and report set percent to 100, push once, wait 2 s link.ack_timeouts +1 and devices/status shows failed_messages: 1
Lost broadcast, digest recovery percent 100, play a different preset on the zone, then set percent 0 the LEDs do not change at first, then within about 1.5 to 5 s the driver sends a resync and shows the new preset; link.resyncs +1
Driver-side loss set "espnow_loss": 40 in the driver's settings.json (restore it afterwards), controller loss 0, push twenty times every push is processed once on the driver (retransmissions are de-duplicated); link.retransmits rises. Lost acks are covered by the controller-side loss rows above, because that setting also drops incoming acks
Multi-message send POST /presets/send with 10+ preset ids, save: false and targets the controller packs the presets into a few binary messages (14 presets took 2); all arrive under loss, with retransmits. save: true makes the driver write its presets.json, so restore that file afterwards. Frame-level fragmentation exists in the link but nothing in the current API produces a message over one frame, so it is only covered by the driver's unit tests

Set loss back to 0 when done: curl -s -X POST -d '{"percent":0}' http://$S3/test/loss.

Reading the numbers: the controller drops a reply with the same probability as a request, so under loss an unicast message needs both the data and the ack to survive. Resync requests are also subject to loss and to the driver's 3 s minimum gap, so recovery may take a few digests.

Results of the first run (S3 to one p2p driver, channel 5)

test result
20 acked pushes, controller loss 40% 20 of 20 reached the driver; 19 acked, 1 reported timed out (its ack was lost); 19 retransmits; 34 frames dropped by the test mode
1 push, loss 100% gave up after 2.0 s with 4 retransmits; failed_messages 1; driver heard nothing
zone play during a 100% outage, then loss back to 0 driver was untouched during the outage; 1.5 s after the link returned it received its groups, the preset and the select, one resync, and the new state
20 acked pushes, driver-side loss 40% 20 of 20 processed exactly once; 14 retransmits; 0 timeouts
/presets/send of 14 presets at driver loss 40% 2 messages (209 and 217 bytes), both acked, 1 retransmit

7. Beat latency and delivery

Software numbers (no equipment)

Start synthetic beats with the driver in bench mode:

curl -s -X POST -d '{"bpm":120,"count":200}' http://$S3/test/beats
sleep 105
curl -s http://$S3/test/beats

Results include:

  • beats_sent, echoes and hit_pct_total: the share of beats each driver fired on and reported. With no loss expect 100%.
  • rtt_minus_lead_min/avg/max_us: time from the first beat copy going out to the driver's echo coming back, minus the 30 ms the driver deliberately waits (CONFIG_BEAT_LEAD_MS). This is out-airtime plus the driver's MicroPython work plus return airtime, so it overstates one-way delay. Expect roughly 2 to 15 ms; the maximum shows the worst case you will see on the LEDs.
  • spread_avg_us and spread_max_us: only meaningful with two or more drivers; shows how far apart they fire for the same beat.

On the driver's serial console every 50 beats you get the true driver-side counts, for example beat {'fires': 100, 'dups': 100, 'late': 0, 'freerun': 0}:

  • fires: beats it acted on.
  • dups: second copies dropped (with the double send this should be close to fires).
  • freerun: beats it generated itself because both copies were missed. Zero without loss.
  • late: frames rejected as too soon or old.

Delivery under loss

Repeat the run with loss on:

curl -s -X POST -d '{"percent":30,"beats":true}' http://$S3/test/loss
curl -s -X POST -d '{"bpm":120,"count":200}' http://$S3/test/beats

Judge delivery from the driver's own fires and freerun counts: with two copies the chance of missing both is about 9% at 30% loss, and the free-run covers those, so fires should stay near beats_sent with freerun around 9% of it. The controller's hit_pct_total is lower because echoes are dropped by the same loss setting; it is a lower bound, roughly (1 - loss) × delivery.

Latency over USB (no extra equipment)

Both boards are on USB, so the PC can timestamp one event line from each and compare them. Settings already applied for this: the driver has "usb_log": true in settings.json. The controller prints BT <beat> <microseconds> for every synthetic beat; the driver prints FIRE <beat> <ticks_us> when it fires.

python3 tools/usb_latency.py --ctrl-port /dev/ttyACM0 --drv-port /dev/ttyACM1 --url http://$S3 --beats 200

The script starts the generator, reads both ports, and maps each board's clock onto the PC's by using the fastest USB deliveries per 5 s window (this also removes crystal drift). USB serial delivers in bursts, so single lines arrive up to about 50 ms late; the fastest ones are close to on time, which is what the mapping relies on. The remaining error is the difference between the two ports' fastest delivery, roughly 1 ms, so read results to about a millisecond. The script prints the delivery jitter so you can see how noisy the ports were. The driver's FIRE time is when the beat triggers the pattern step; the LED data follows within the same loop pass, and a 119-LED strip takes about 3.6 ms to clock out.

Close any serial monitor first (only one program can hold a port). Do not run long mpremote exec commands on the driver during the capture (see Troubleshooting).

True latency with a logic analyzer

For a measurement that does not depend on USB. Marker pins are set up already: the controller toggles GPIO44 (D7) when the first copy of each beat goes out (CONFIG_BEAT_TEST_GPIO=44, in sdkconfig.defaults) and the driver toggles GPIO44 (D7) when it fires ("beat_pin": 44 in its settings.json). Both are XIAO ESP32S3 boards, where D7 is free.

Wiring (3.3 V logic, any 2 or 3 channel analyzer):

analyzer connect to
GND GND on the controller and GND on the driver (one common ground)
channel 0 controller D7
channel 1 driver D7
channel 2 (optional) driver D10 (GPIO9, the LED data line) to see when the LEDs really update

Steps:

  1. Make sure the driver runs a preset that changes on every beat if you use channel 2, for example a chase preset with auto off, selected on the zone. With the plain on preset the LED data line stays quiet.
  2. Start a capture: sample rate at least 1 MS/s, length at least 60 s (edges only; it stays small), trigger on any edge of channel 0.
  3. Start the beats: curl -s -X POST -d '{"bpm":120,"count":100}' http://$S3/test/beats. Both markers toggle on every beat, so you will see edges 500 ms apart.
  4. Stop the capture when the generator finishes (about 52 s) and export the digital data as CSV (time in seconds, one column per channel).
  5. Analyze: python3 tools/analyze_edges.py capture.csv --ctrl 0 --drv 1 --time-unit s. Use --lead-ms if you changed CONFIG_BEAT_LEAD_MS. It pairs each controller edge with the next driver edge and prints the delay beyond the 30 ms lead, with min, mean, median, p95, max and jitter, plus the count of unmatched beats. On the scope, channel 2 shows the true light delay: the first data burst after the driver edge.

Targets from the plan: under about 10 ms from the controller edge to the driver edge (after subtracting the lead).

Return to normal use by deleting beat_pin and usb_log from the driver's settings.json and setting CONFIG_BEAT_TEST_GPIO=-1 (the marker pin otherwise toggles on every beat, including live mic beats).

Results of the first run (software numbers, one driver, channel 5)

Synthetic beats at 120 BPM, 160 beats per run, mic beats muted while the generator runs, driver in bench mode.

no loss 30% loss (beats included)
beats the driver acted on, per beat sent 1.00 1.00
beats the driver invented itself (free-run) 0% 11% (theory for two independent copies: 9%)
second copies dropped as duplicates 100% 47% (theory: 49%)
echoes the controller received 162 of 162 96 of 161 (echoes are lost at 30% too, and invented beats do not echo)
round trip minus the 30 ms lead, min / avg / max 9.8 / 15.0 / 25.7 ms 3.4 / 16.2 / 26.0 ms

Reading it: the double send plus free-run gave the driver a beat every time, even with 30% of frames lost. The round-trip figure includes the driver's loop and the echo's trip back, so it is an upper bound on the delay the LEDs see; it is not the controller-to-LED latency. The USB and logic analyzer measurements below give the real figure.

Latency over USB (200 beats at 120 BPM, one driver, 119 LEDs): delay beyond the 30 ms lead min 0.49, mean 2.07, median 1.77, p95 4.41, max 7.95 ms (std 1.3 ms; accuracy about 1 ms). 197 of 200 beats matched. Both clocks drifted by 60 to 80 ppm against the PC, and individual USB lines arrived up to 54 ms late, which is why the script uses the fastest deliveries. This is the time from the controller sending a beat to the driver firing, beyond the built-in lead. The logic analyzer run has not been done.

ESP32-C3 driver compared with the S3 driver

Driver: XIAO ESP32-C3 (MAC 58:8c:81:a3:74:58), MicroPython v1.22.1 stock (no ws2812-dma patch), 160 MHz, about 128 KB free heap, 119 LEDs on pin 10. Same tests as above, same controller.

S3 driver, no loss C3 driver, no loss
echoes the controller received 162 of 162 160 of 160
round trip minus the 30 ms lead, min / avg / max 9.8 / 15.0 / 25.7 ms 2.6 / 4.5 / 17.8 ms

C3 with 30% loss (160 beats): every beat produced exactly one pattern step, 15 of the 160 (9%, theory 9%) came from free-run, and 68 second copies were dropped as duplicates; the controller got 99 of 160 echoes. That matches the S3 result in the table above.

Reading the free-run counter: after the generator stops, the driver keeps stepping for up to 4 more beats on its last period. A stats window that runs past the end of the test therefore shows 4 extra "free-run" fires even with no loss. They are not missed beats. (My first read of the C3's no-loss run said otherwise; that was wrong.)

Latency over USB, same settings on both boards (usb_log only), 200 beats at 120 BPM, delay beyond the 30 ms lead, accuracy about 1 ms:

matched min mean median p95 max std
S3 driver (240 MHz) 200 / 200 -2.4 2.15 2.03 6.03 12.3 2.5 ms
C3 driver (160 MHz) 200 / 200 0.2 3.36 3.06 7.47 19.7 2.3 ms

The C3 is about 1.2 ms slower on average and has a longer tail (maximum 19.7 ms against 12.3 ms). With the method's 1 ms accuracy, the mean difference is small and the tail difference is the more telling one. Both are well inside a 120 BPM beat. The echo round trip disagrees with this (S3 15 ms mean, C3 4.5 ms). The S3 echo runs had usb_log and a marker pin switched on and the C3 runs did not, and I did not rerun the S3 echo test with identical settings, so treat the echo figures as unexplained and prefer the USB ones.

Pattern frame rate (strip writes per second, 119 LEDs, delay 1 ms, 2 s each; only patterns that wrote anything are shown):

pattern C3 (160 MHz, 115 KB free) S3 (240 MHz, 8 MB free)
flame 22 33
palette_morph 21 32
circle 150 170

The CPU-heavy patterns run about 1.5 times faster on the S3, which is the clock ratio; a light pattern is limited by the strip write and differs by about 13%. aurora and chase wrote nothing with the parameters I used, so they are not measured.

Groups: the controller sends a driver its group list when the driver announces itself, but now only if the driver is in at least one zone. A driver in no zone keeps the groups it already has (an empty list from the controller used to wipe them). An explicit zone or device edit still re-sends every list, including empty ones.

Wi-Fi start-up order on the C3. With the extra link code loaded before Wi-Fi start-up, the C3 failed with Wifi Unknown Error 0x0101 on about every other cold boot (2 of 4). Starting ESP-NOW before the pattern library and the LED driver claim memory fixed it (6 of 6 boots). led-driver/src/main.py now does this.

8. Troubleshooting

  • A board on a USB-serial/JTAG port (C3, and the S3 controller) resets whenever a program opens its port, and a C3 can land in ROM download mode (wait usb download). The old apt mpremote 0.4.0 hangs on this. Use the newer one in ~/projects/micropython/micropython/tools/mpremote/mpremote.py, or led-tool, and note that fs cp -r fails on drivers that print timestamps (copy named files instead).

  • Only one program can use a board's serial port at a time. A serial logger left running while led-cli uploads corrupts the transfer silently: led-cli reports the files uploaded, but the old ones stay on the driver.

  • Running a long mpremote exec on the driver resets it: the driver's watchdog (10 s) keeps running after mpremote interrupts main.py, so keep REPL experiments under about 8 s and reset the driver afterwards.

  • Driver never reacts, no acks: channel mismatch. Check CONFIG_ESPNOW_CHANNEL against the driver's wifi_channel. Neighbouring channels can still work at bench range, so a working link does not prove they match.

  • Zone play logs nothing on the driver: the driver is not in the zone's group (silent). Check the zone's names against the driver's name.

  • Devices show last_ack_ms_ago: -1: the device record has no usable 12-character MAC in address or as its key.

  • beats_sent rises but echoes stays 0: the driver is not in bench mode, or it has not yet heard a controller frame (it learns the controller's MAC from the first data or digest frame; wait a few seconds).

  • Resync never triggers: the driver ignores a digest equal to its own version; confirm with /status that link.ver changed after the missed play.

  • Web UI is blank: the web UI was not packed before the build; run tools/build_web.sh and build again (make build does this).

9. Cleanup

curl -s -X POST -d '{"percent":0}' http://$S3/test/loss
curl -s -X POST http://$S3/test/beats/stop

Set the driver's bench and espnow_loss back to off for normal use.