[JensOgorek]

Motivation

When two tasks of the same ESP32 write checked lines at the same time, a line can go out with the other task's checksum or with the other task's content, and the receiver loses it. Serial::write_checked_line formats the checksum trailer into a static buffer, and SerialBus keeps the frame it reads, the frame it writes and the frame it copies out of a telemetry slot in static buffers that all buses share. The main task (core 0) writes the lines of an expander, and every bus writes its frames from its own communication task (core 1). So the trailer race hits every ESP32 with an expander next to a serial bus today, and both races hit every ESP32 with two serial buses, as with port expanders as bus peers (#348).

On a bench pair of two ESP32s, main loses lines in both setups within 2 minutes: with an expander and a bus 58 of 24 000, with two buses 504 of 12 000 messages of one bus. The receiver drops such a line with a checksum warning or, if the other bus replaced the whole frame, ignores it without any warning. With this PR, none was lost in 10 minutes of each setup. It was found on F30, whose P0 ran as a second bus peer in a test setup and received garbled polls of the other bus every 5 to 10 minutes, and none in 40 minutes with this change.

Implementation

• Serial::write_checked_line builds the checksum trailer on the stack.
• SerialBus keeps its read, write and slot buffers as members, one set per bus; each set is used only by the communication task of its bus. The write buffer is sized by the largest frame send_message can build (16 + PAYLOAD_CAPACITY), the read buffer keeps 512 B for any line a peer may send.
• After the review (1e46171): handle_incoming_message prints an echo from the payload itself instead of copying it into the last static buffer of the file, and the dead check for a negative result of csprintf in send_message is gone.

Cost: −160 B flash on the ESP32; static RAM −1552 B. Each bus takes 1040 B of heap for its frame buffers, so the RAM use only grows from the second bus on.

Progress

• The implementation is complete.
• Documentation has been updated (or is not necessary).
• Tested on hardware (or is not necessary).

Bench test 1: an expander and a serial bus on the same core

Two ESP32s wired like a Robot Brain's core and P0: core GPIO27 → target RX (GPIO3), target TX (GPIO1) → core GPIO26, and core GPIO25 → target GPIO0, which is free for a second line once the expander runs without boot and enable pins. The target ran main ec5d268 throughout; only the core's firmware changed. Both were built with ESP-IDF 5.3.1.

Core:

ps = Serial(26, 27, 115200, 1)
p0 = Expander(ps)
bs = Serial(32, 25, 115200, 2)
bus = SerialBus(bs, 1)
bus.make_coordinator(2)
int np = 0
int nb = 0
bool go = false
when go then p0.run("n = n + 1"); np = np + 1; bus.send(2, "m = m + 1"); nb = nb + 1 end

Target, sent with p0.run(…):

int n = 0
int m = 0
zs = Serial(0, 33, 115200, 2)
zbus = SerialBus(zs, 2)

The target's GPIO33 is not connected, so peer 2 never answers: the coordinator's poll times out every 250 ms, and then it sends the queued messages in a burst. Each main loop (10 ms) writes one expander line from the main task and queues one bus message for the bus task. go was true for 120 s or 600 s; afterwards the core printed np, nb and the target n, m.

Core firmware	Duration	Expander lines sent / executed	Bus messages sent / executed	Checksum warnings of the target (expander / bus)
main ec5d268	120 s	12 001 / 11 993	12 001 / 11 951	8 / 50
this PR ba49826	120 s	12 000 / 12 000	12 000 / 12 000	0 / 0
this PR ba49826	600 s	60 000 / 60 000	60 000 / 60 000	0 / 0

On main, every lost line has a warning. The bus frames came with their own content but with the trailer of the expander line written at the same moment, e.g. serial bus zbus checksum mismatch: $$1:2$$m = m + 1@27: the frame's checksum is 1e, and 27 is the checksum of n = n + 1. Once it was @53, the checksum of the expander's ping core.print('__PONG__').

Bench test 2: two serial buses on the same core

Same pair and wiring, no expander. Bus O runs on UART2 to the target's GPIO0; bus X runs on UART1 with its TX on the unconnected GPIO4 and has no peer. The target ran main ec5d268 with this startup and printed its counter every second:

int m = 0
int t = 0
zs = Serial(0, 33, 115200, 2)
zbus = SerialBus(zs, 2)
when core.millis > t then core.print("count", m); t = core.millis + 1000 end

Core:

xser = Serial(33, 4, 115200, 1)
xbus = SerialBus(xser, 1)
xbus.make_coordinator(3)
oser = Serial(32, 25, 115200, 2)
obus = SerialBus(oser, 1)
obus.make_coordinator(2)
int cx = 0
int co = 0
bool go = false
when go then xbus.send(3, "k = k + 1"); cx = cx + 1; obus.send(2, "m = m + 1                                                                                "); co = co + 1 end

The message on bus O ends with 80 spaces, so that the 25 messages of one burst (100 bytes each) overflow the 2 KB TX ring and bus O's task blocks in the middle of a line, while bus X's task sends its poll or its own burst.

Core firmware	Duration	Bus O messages sent / executed	Checksum warnings of the target	of them with bus X's content
main ec5d268	120 s	12 000 / 11 496	358	346
this PR ba49826	120 s	12 000 / 12 000	0	0
this PR ba49826	600 s	60 000 / 60 000	0	0
this PR 32b2197	120 s	12 000 / 12 000	0	0
this PR 1e46171	120 s	12 000 / 12 000	0	0

343 of the 346 warnings with bus X's content read $$1:3$$__POLL__1⎵⎵⎵…@27: the first 15 bytes are bus X's poll of its peer 3, the 1 and the spaces are the rest of bus O's message, and 27 is the checksum of the poll; in the other three, bus X's k = k + 1 stood in front. The remaining 12 warnings had bus O's content with bus X's trailer (1f of $$1:3$$k = k + 1, 27 of the poll). The remaining 146 lost messages raised no warning; presumably bus X replaced the whole frame, which then is a valid frame for node 3 that the target ignores.

Bench check of 1e46171: echoes and the longest frames

Bench test 2's peer never answers, so it does not reach the echo path. Here the PC played peer 202 on the target's console wires (core GPIO27 and GPIO26) while the target was held in reset, and the core ran 1e46171 as coordinator 201, so that every header it built was the longest one, $$201:202$$:

ser = Serial(27, 26, 115200, 1)
bus = SerialBus(ser, 201)
bus.make_coordinator(202)

• Echoes from the peer with payloads of 14, 255, 13 and 9 bytes, in this order, printed exactly as sent, e.g. bus[202]: Ready. and bus[202]: short right after the 255-byte one. A payload of 256 bytes was rejected with could not parse message, and the echo after it printed exactly again.
• bus.send(202, …) with 254 and 255 bytes arrived complete with a valid checksum; 256 bytes were rejected with payload is too large for serial bus.
• A core.print of 240 characters sent by the peer came back as a relayed echo of 250 bytes with a valid checksum.
• All 286 frames of the core had a valid checksum.

※

[falkoschindler]

@JensOgorek the analysis holds: every caller of process_uart, send_message and send_frame_slots sits in the communication task of its own bus, the main task reaches a bus only through its queues, and the remaining static buffers in the repository (expander, echo, get_output, the echo copy in handle_incoming_message) run on the main task only. The two bench tests cover both races. I fixed three small things on the branch in 1e46171; nothing is open for you.

Fixed in 1e46171

• tx_frame was twice its largest frame. send_message builds at most the header $$255:255$$ (11 bytes) and a payload below PAYLOAD_CAPACITY, so of the 512 B inherited from the shared FRAME_BUFFER_SIZE every bus paid 240 B of heap for nothing once the buffer became a member. tx_frame now takes 16 + PAYLOAD_CAPACITY with a static_assert for the longest header; rx_frame keeps its 512 B for any line a peer may send.
• The last static buffer of SerialBus. handle_incoming_message copied an echo into a static char buffer[PAYLOAD_CAPACITY] before printing it, although process_uart already terminates the payload. It runs on the main task only, so it was no race, but 256 B of static RAM for nothing; it now prints and compares the payload itself.
• Dead check in send_message. The header_len < 0 branch after csprintf could never run: csprintf throws on an encoding error or a too-small buffer, and both land in the catch of communication_loop.

Against main ec5d268 the branch now takes 160 B less flash on the ESP32 and 1552 B less static RAM; each bus takes 1040 B of heap instead of 1280 B. The cost line of the description is updated accordingly.

※

[JensOgorek]

Retest of 1e46171 passed — merged as d4e07ce

@falkoschindler thanks for the three fixes. Bench test 2 again delivered 12 000 / 12 000 without a checksum warning, and since its peer never answers, a second check with the PC as bus peer covered your echo change and the 255/256-byte limits of the smaller tx_frame (12/12 passed); both are in the description.

※