Files
MicrOBU/05-obu-bench-test-2026-08-25.md
Ashin Walpola 04b0076b8b Move the RX capture buffer off the WiFi driver's callback stack
rx_item_t is ~800 bytes at RX_FRAME_MAX_LEN, and wifi_promisc_rx_cb declared one
as a local. That callback runs on the WiFi driver's own task, already several
frames deep in the driver's call chain, on a stack of roughly 3.5 KB
(CONFIG_ESP_WIFI_TASK_STACK_SIZE, left at its default). Putting a fifth of that
stack into a single local is a stack-overflow risk that only appears under real
traffic - in front of an RSU rather than on the bench - and would present as a
random panic rather than anything pointing at its cause.

Both instances are now static: one in the callback, one in rx_forward_task. Safe
because each is touched by exactly one task, so there is no re-entrancy to guard
against; the same reasoning serial_link.c already uses for its static send
buffers. xQueueSend copies the struct out before returning, so reusing the
callback's buffer on the next frame is fine.

Firmware-only, no protocol change, so it does not require a matching app install.

Re-verified against live traffic after flashing: 1094 frames over 125 s with zero
decode failures, USB errors, detaches, crashes or mutex timeouts. SPATEM capture
rose from 3.20/s to 3.98/s against a theoretical maximum of 4.00/s, which is the
direction relieving stack pressure would produce, though RF geometry moves
between runs and this is not proof.

Report updated with T9, the accepted 512-byte ceiling, and the decision to drop
Phase B: the intersection use case is CAM-driven and needs none of it.
2026-08-25 15:13:23 +02:00

291 lines
14 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# ESP32-C5 OBU firmware — bench test against live ITS-G5 traffic
**Date:** 2026-08-25
**Firmware:** `obu-firmware` @ commit `b2b57fa`, flashed to COM3 (CH343 UART bridge, 921600 baud)
**App:** `app-debug.apk`, installed 14:33:45, Pixel 9 Pro on the C5's native USB-C port
**Verdict:** the receive path works against all three live message types with zero failures.
Two blocking items remain before real-world use, both known and both outside what this bench can
exercise. See [Readiness](#readiness).
## Test environment
| role | device | notes |
|---|---|---|
| Device under test | ESP32-C5 OBU, COM3 | our firmware; phone attached to its native USB port |
| Traffic source | RSU, broker `192.168.3.202` | transmits SPATEM + CAM |
| Traffic source | CiT One OBU, broker `192.168.3.201` | transmits DENM + CAM |
| Independent witness | both brokers' `v2x/rx/*` topics | each hears the *other* device |
Both brokers publish raw UPER (protobuf-wrapped, field 3), so their counts are directly comparable
with what the OBU forwarded. That is what makes this a cross-check rather than a self-report: the
OBU's output is measured against two receivers that share none of its code.
Station IDs observed today (`3983312873` RSU, `3257224191` CiT One) differ from those seen this
morning (`968482441`, `2880458775`). **Station IDs rotate**, so nothing may treat one as a durable
identity for a physical unit.
## T1 — Message type coverage and rate
305-second continuous capture, phone logcat.
| type | frames | rate | stations |
|---|---|---|---|
| CAM | 1244 | 4.08/s | 2 |
| SPATEM | 978 | 3.20/s | 1 |
| DENM | 646 | 2.12/s | 3 |
| **total** | **2868** | **9.40/s** | |
Per station, with received signal strength:
| type | station | frames | RSSI min/median/max |
|---|---|---|---|
| SPATEM | 3983312873 | 978 | −65 / −60 / −48 dBm |
| CAM | 3257224191 | 633 | −58 / −52 / −48 dBm |
| DENM | 3257224191 | 616 | −65 / −58 / −48 dBm |
| CAM | 3983312873 | 611 | −65 / −62 / −60 dBm |
| DENM | 2908440021 | 20 | −64 / −61 / −53 dBm |
| DENM | 1220851972 | 10 | −63 / −52 / −49 dBm |
All three types decoded concurrently, from both transmitters plus two additional DENM sources that
happened to be on air. **PASS.**
## T2 — Decode integrity
Over the same window:
| check | result |
|---|---|
| Decode failures (CAM / DENM / SPATEM) | **0** |
| Unexpected BTP ports from firmware | **0** |
| USB I/O errors | **0** |
| Device detach events | **0** |
| Fatal exceptions | **0** |
Zero decode failures across 2868 frames. Since the decoders return null rather than guessing
whenever an extension bit or unsupported optional appears, a zero here means every frame matched
the bit layouts exactly — not that failures were being swallowed. **PASS.**
## T3 — Cross-check against independent receivers
60-second simultaneous capture from both brokers, counting the same UPER the OBU sees.
| stream | independent witness | our OBU |
|---|---|---|
| RSU SPATEM (3983312873) | 2.00/s | 3.20/s |
| RSU CAM (3983312873) | 2.00/s | 2.00/s |
| CiT One CAM (3257224191) | 2.05/s | 2.08/s |
| CiT One DENM (3257224191) | 1.00/s | 2.02/s |
CAM matches on both transmitters. SPATEM and DENM read high — explained in T4, not a defect.
## T4 — Duplicate transmission (finding, not a fault)
The DENM and SPATEM discrepancies above are **real duplicate transmissions**, not double-counting
in our firmware. Inter-arrival analysis of the captured stream:
| type | gaps < 150 ms | median of those | identical content, different RSSI |
|---|---|---|---|
| SPATEM | 50% (485/977) | 7 ms | **480 / 485** |
| DENM | 50% (308/615) | 5 ms | **282 / 308** |
| CAM | 2% (11/632) | 100 ms | 1 / 11 |
Each SPATEM and DENM goes out **twice, ~5–7 ms apart, with different RSSI** — two antennas. CAM is
sent once. This also reconciles the broker figures: the CiT One reports one copy on `v2x/rx/*` and
the other on `v2x/rx-red/*` ("red" = redundant), and only the primary was counted.
Our firmware is a promiscuous receiver, so forwarding both copies is correct behaviour. The app
deduplicates downstream — DENM on ETSI actionID, SPATEM on intersection key — so the UI shows one
entry per event. The cost is serial bandwidth: **38% of the bytes carried are duplicate copies.**
## T5 — Serial link load
Measured over the 305 s window, using UPER sizes taken from the brokers:
- **9.40 frames/s, ~1550 B/s (12.4 kbit/s)**
- Largest frame: DENM at 402 B UPER → 416 B payload (81% of the 512 B cap, 96 B headroom)
- On the wire that frame is 423 B, 41% of the 1024 B TX ring
- Suppressing duplicate copies would cut this to ~961 B/s (7.7 kbit/s)
No frame in this session exceeded the payload cap.
## T6 — Firmware drop counters
Read directly off the app at 14:45, after the capture:
```
ESP32: tx fail 0 · oversize 0 · crc err 0
```
These are free-running totals **since firmware boot**, so all three being zero covers the whole
session, not just the test window — no oversize drops, no `esp_wifi_80211_tx` failures, no CRC
errors at any point since the C5 was flashed. **PASS.**
The app also now logs these counters whenever one changes (added for this test; they previously
reached only the UI), so a future bench run captured through logcat records drops as they happen.
## T7 — End-to-end UI verification
Screenshot at 14:45 confirms the full chain reaches the display:
| element | shown |
|---|---|
| Link state | CONNECTED |
| Hazard (DENM) | `stationaryVehicle · station 3257224191`, 30 m, 1000 m radius, −63 dBm |
| Signals (SPATEM) | `Intersection -1/23 · station 3983312873`, SG1 red / SG2 amber, −63 dBm |
| Station (CAM) | `Station 3257224191 · Car`, 1.4 km/h, heading 19°, 25 m, −50 dBm |
Signal-group colouring, the DENM relevance radius from the GeoNetworking header, and per-station
RSSI all render correctly.
### Finding: the RSU's CAM decodes but is never displayed
The station list reads **"1 station(s) in range"** — only the CiT One (`3257224191`). The RSU
(`3983312873`) is absent, despite **611 of its CAMs decoding successfully** during the capture.
Cause: RSU CAMs are deliberately excluded from `UseCaseDetectionEngine` (a permanently stationary
station at a fixed point otherwise trips the stopped-vehicle use case continuously) — but
`remoteCamPositions`, which feeds both the station list and the map, is populated *by that engine*.
So the exclusion removes them from the display as well as from detection.
The RSU is not entirely invisible: it appears in the SPAT section as the intersection's station. But
its CAM-reported position is dropped on the floor. This is a defect introduced with the RSU CAM
decode fix earlier today, not a firmware problem — the firmware forwarded all 611 correctly.
**Fixed and re-verified the same session.** RSU CAMs are now tracked in a separate
`rsuStations` flow in the repository, merged with the engine's road users for display only, with a
15 s staleness window and a clear on link-down. Screenshot at 15:01 confirms:
```
2 station(s) in range - latest CAM per station
Station 2199514753 · Car 0.8 km/h · heading 192° 28 m -52 dBm
Station 440624502 · Roadside unit roadside unit - no kinematics reported
46 m -61 dBm
```
The kinematics line is suppressed for RSUs: their CAM carries none, so the zeroes in the model are
placeholders and printing "0.0 km/h · heading 0" would assert a stationary vehicle facing north.
The same station also drives the SPAT row, so the two views agree.
## T8 — Station ID rotation
Station IDs rotated **twice within one session**:
| time | RSU | CiT One |
|---|---|---|
| ~09:00 | 968482441 | 2880458775 |
| 14:45 | 3983312873 | 3257224191 |
| 15:01 | 440624502 | 2199514753 |
That is a rotation inside 16 minutes. Consequences for the app, none of them currently handled:
- **DENM dedup keys on ETSI actionID**, which contains the originating station ID. A hazard that
outlives a rotation will appear as a second, independent pin rather than an update of the first.
Both then persist until the 60 s TTL expires them.
- **The station list and map key on station ID**, so a rotation shows the same physical vehicle
twice for up to the 15 s window.
- **SPATEM is unaffected**, because it dedups on the intersection reference (`region/id`), which is
a property of the junction rather than the sender. That is the right key and it survives rotation.
Nothing here is a firmware issue, and pseudonym rotation is the intended privacy behaviour of the
transmitters. But any future logic that assumes a station ID identifies a physical unit over time
will be wrong.
## T9 — Stack fix and re-verification
`rx_item_t` was moved off both task stacks (`static` in the promiscuous callback and in
`rx_forward_task`), firmware reflashed, and the campaign re-run:
| | before fix (305 s) | after fix (125 s) |
|---|---|---|
| Total | 9.40/s | 8.73/s |
| CAM | 4.08/s | 4.11/s |
| SPATEM | 3.20/s | **3.98/s** |
| DENM | 2.12/s | 0.64/s |
| Decode failures / IO errors / crashes | 0 | 0 |
| Mutex timeouts | — | 0 |
SPATEM capture rose from ~80% to ~100% of the theoretical 4.00/s (2 Hz × two antennas). Not
attributable to the fix with confidence — RF geometry moves between runs — but it is the direction
stack pressure relief would produce, and worth re-checking on the next run. The DENM drop is the
CiT One's trigger being intermittent, not a receive problem.
### Finding: no automatic reconnect after re-enumeration
Reflashing resets the C5, which re-enumerates its USB device. The app did **not** recover: it went
to `Connection error - check the cable and native USB-C port, then try again` and stayed there until
Connect was tapped manually, followed by a fresh USB permission grant.
This matters more for the intersection use case than SPATEM does. On a bike, a jostled cable that
re-enumerates leaves the link dead until the rider notices and taps a button — a silent loss of the
CAM stream the use case runs on. The permission grant is a genuine one-time consent and cannot be
automated, but retrying automatically when a matching device is already attached would cover the
common case.
## Readiness
### Working
- All three received message types decode correctly from live over-the-air traffic
- Concurrent multi-station, multi-type reception with no interference between streams
- Sustained 5-minute run with no link drop, no I/O error, no crash
- RSSI plausible and discriminating between transmitters (−48 to −65 dBm at bench distance)
- No frame exceeded the serial payload cap under this traffic mix
### Scope decision (2026-08-25): Phase B dropped
Raising the payload cap was considered and **deliberately rejected**. The project goal is V2X
communication with at least one white-paper use case — incoming car at an intersection — working on
the ESP32. That use case is `IMA-B`/`IMA-S`, which `UseCaseDetectionEngine` drives entirely from CAM
kinematics; the engine contains **zero references to SPATEM or MAPEM**. Everything the goal needs
fits the current cap with margin: CAM 26–211 B, DENM 402 B, bench SPATEM 58 B, against a 498 B
budget.
Phase B would buy only road-RSU SPATEM/MAPEM — the add-on, not the goal — while putting a measured,
zero-failure chain at risk. The one component of it that *reduces* risk, moving `rx_item_t` off the
WiFi callback stack, was done separately (T9).
### Known ceiling, accepted
1. **Serial payload cap (512 B).** The bench RSU sends 58-byte SPATEMs, but the 2026-03-18 drive
measured real road RSUs at 555 B median and 1243 B max — **roughly 70% would be dropped as
oversize**. Raising `SERIAL_LINK_MAX_PAYLOAD` and `RX_FRAME_MAX_LEN` to ~1536 is required, and
forces item 2.
2. ~~**`rx_item_t` on the WiFi driver's callback stack**~~ — **fixed 2026-08-25**, see T9.
3. **DENM headroom is 96 B.** DENM matters to this project in a way SPATEM does not, and at 402 B
it is the closest message to the cap. A DENM carrying more optional containers than the CiT One's
HLN-SV currently sends would be silently dropped and counted as oversize. The `oversize` counter
on the CAM Pinger card is the thing to check if hazards ever stop appearing.
### Defect found and fixed during this test
- **RSU CAM positions were decoded but never displayed** (see T7). App-side; the firmware forwarded
all 611 correctly. Fixed and re-verified in the same session.
### Open, found by this test
- **Station ID rotation** (see T8) fragments DENM and CAM identity across a rotation. Not yet
handled.
### Untested here
- ~~**Link recovery**~~ — exercised by the reflash in T9: it does **not** auto-recover. See T9.
- **Sustained load at road rates.** This bench ran at 9.4 frames/s. The drive data implies 24–32
frames/s with frames 3× larger, where the TX-mutex interaction (400 ms worst-case hold vs the
1 Hz heartbeat and the phone's 3-beat dead-link timeout) becomes the thing to watch.
- **The link's actual ceiling**, which has never been saturated and so is unmeasured.
- **MAPEM** — nothing on air is transmitting it; no decoder written.
- **Secured messages** — 75 frames with GN `NextHeader=2` appeared in earlier pcaps; these are
rejected by design. The bench runs with `ItsGnSecurity = 0`.
### Recommendation
Ready for continued bench and short-range field work as it stands. **Not ready for a road drive
past real RSUs** until items 1 and 2 land, because the failure there is silent: oversize SPATEMs are
counted and dropped, so the symptom is "the intersection never appears" rather than an error.
## Reproducing
Capture: `adb logcat -d` filtered on `CamUseCaseRepo` while subscribed to `v2x/rx/#` on both
brokers. Decode cross-checks use `asn1tools` with the modules in `asn1/` — see `asn1/README.md`.