MicrOBU/TODO.md

TODO

Engineering to-do list. The reviewer-facing open items live in docs/01-requirements-traceability.md ("Open items"); this file is the working list behind them.

Waiting on hardware

Signed-TX firmware (vanetza-idf port), VAM and BLE: first on-air checks (added 2026-09-23)

obu-firmware is now a port of the colleague's microbu-esp32c5 station (vanetza-idf, TS 103 097 signing, station-link protocol, BLE GATT), built with ESP-IDF 6.0.2. See obu-firmware/NOTES.md. The previous firmware is backed up in firmware-backups/ (restore command in its README.txt). The app speaks the new protocol over USB or BLE and still falls back to the old frames against the old firmware.

Done without hardware: IDF 6.0.2 build clean (39 % app partition free); host suite (make in obu-firmware/test/host) passes unchanged; app unit tests 103/103, including the VAM encoder against asn1tools, the station-link codec against the colleague's Python messages.py, and the VAM generation rules. Flashed to COM3 2026-09-23 (hash verified); boot log: IDF v6.0.2, BLE advertising started as 'micrOBU-4AFA', station task ready. The radio stays off until the app configures the station. New app build installed on the Pixel 9 Pro (adb, install -r).

First phone session (user, 2026-09-23): BLE works and CAMs go out. Three faults, fixed and reflashed/reinstalled the same day:

1. No RX until the CAM pinger ran, with ~177 RX-queue drops: the colleague's Station::tick() returned before draining the radio until the first PoTi had set the clock. Now drained always.
2. "refused a request: time_regression": the loops re-send the latest fix every tick; a stale fix timestamp read as the clock going back > 1 s, and each time the board rebuilt its stack. Esp32Link now sends a PoTi only for a newer fix (or a >= 60 s real clock correction).
3. BLE reconnect loop: GATT operations with a 5 s timeout ran during Android's pairing, cut it off and restarted it on every attempt. Encryption/pairing is now settled first (60 s), retries back off to 30 s, and every failure reason is logged and shown; the board logs encryption changes.

Second session (user, 2026-09-23): USB, RX and signing work; BLE still prompted every time and never connected; time_regression every ~8 s. Found and fixed, reflashed (full flash, NVS erased): 4. The board never stored a bond: NVS (24 KB, the colleague's 4 MB-board layout) was full, mostly Wi-Fi settings the previous firmware left behind, and NimBLE's bond write failed. NVS is now 80 KB (app moved to 0x20000) and was erased; boot logs N bonded phone(s) in NVS. 5. The station loop waited pdMS_TO_TICKS(5) = 0 ticks at 100 Hz, so it spun on the single core (task watchdog: IDLE starved). Now waits at least one tick. 6. The phone clock is ~14 min fast; GnssTimeSource fell back to it whenever GNSS time blinked out indoors, so every transmitted timestamp (CAM generationDeltaTime too) jumped 14 min back and forth. It now keeps the last measured error.

Watch COM3 (idf.py -p COM3 monitor, or readlog.py-style with DTR/RTS low) during these; it only resets the board, the phone is on the other port.

• USB session. Settings > Connection > ESP32-C5: link USB-C, transmit CAM, signing on. Phone on the native port, Connect. Expected: the card shows "Provisioning the demo credentials" once, then Connected and Signing on · tickets 1 · signed N with N rising while recording. COM3: radio on channel 180, transmit and receive, tx power: … dBm, credentials provisioned: 1 roots, 1 authorities, 1 tickets, and no radio refused a frame. Confirmed by the user 2026-09-23: connects, signing works.
• Reception intact. Same session, sim car (COM8) beaconing: its CAMs (station 195936478) on the V2X map at ~3 Hz as before. Then put a DENM and a SPATEM on air: both show up (they come through the raw V2X_RX path; the vanetza stack drops them because they are not demo-signed). Confirmed 2026-09-23: sim car and the RSU's CAM/SPATEM/MAPEM arrive; DENM not yet re-tested.
• Signed CAM on air (2026-09-23, CAM pinger over BLE, signing on). Recorded 25 s through the V2X2MAP bridge's /api/record (micrOBU_workspace/v2x-obu-esp32c5/signed-cam-check.pcap) and checked with the new obu-firmware/test/verify_signed_pcap.py (asn1tools + OpenSSL, no vanetza code): 12/12 secured CAMs, station 999999, psid 36, signer = full certificate of the demo AT B80B49387A4C12EB, all signatures valid, COER canonical, and the bundle's chain (AT <- AA <- root) verifies. The CiT One (192.168.40.201) also receives them (~1 Hz on v2x/rx/cam), i.e. a third-party stack unwraps our 1609.2 envelope; its MQTT API exposes no security fields, and it forwards unsigned and unknown-root messages alike, so it cannot say whether it verified them. The same capture showed generationTime wobbling by seconds, with time_regression still firing: fixed in Esp32Link (the PoTi never moves the micrOBU's clock back except for a >= 60 s correction). Re-check: no "restarted its stack" lines in logcat.
• Unsigned toggle. Signing off: the same capture shows next header 1 (common header), as the previous firmware sent. The card's signed count stops rising. Confirmed 2026-09-23: unsigned pinger CAMs show on V2X2MAP as unsigned.
• Signed VAM on V2X2MAP. Since 2026-09-23 17:16 the COM10 bridge is the colleague's v2x2map-0.3.0 from source with a new verify.py and --trust demo-chain.vcr (launcher: micrOBU_workspace/v2x-obu-esp32c5/start-v2x2map-signed.bat, replacing its-g5-bridge.exe). Signed CAMs from the pinger already show "signature verified" live. Switch to VAM with signing on: the VAM must be decoded (cyclist, position) and show "signature verified" too. Confirmed by the user 2026-09-23: signed VAMs decode and verify.
• VAM. Transmit VAM: BTP port 2018, psid 638; with tools/wireshark/psid-vru.lua from the colleague's microbu-esp32c5 repository Wireshark decodes the VAM (stationType cyclist, bicyclist profile in every ~2 s VAM). Rate: ≥1 per 5 s standing still, about one per GNSS fix while riding. Covered by the V2X2MAP check above (decoded VAM, psid 638); Wireshark not needed.
• RX without recording. Connect only (no recording, no pinger): sim-car CAMs appear and the RX-queue drop counter stays at 0 or near it. Confirmed 2026-09-23: messages come in on connect alone.
• No time_regression. Record for a few minutes standing still indoors: no "refused" line on the card, and COM3 never logs ITS time moved back.
• BLE after the NVS fix. First forget micrOBU-4AFA in Android's Bluetooth settings (the phone still holds the bond the board lost). Then Connect, passkey 123456 once; a second Connect after an app restart must not prompt again, and COM3's next boot must say 1 bonded phone(s) in NVS. Confirmed 2026-09-23: pairs once, reconnects after an app restart.
• BLE. Link Bluetooth, unplug USB, Connect. Android asks to pair with micrOBU-4AFA: passkey 123456. Expected: Connected, CAMs keep going, sim-car CAMs keep arriving. While USB is plugged in and in use, the phone's BLE scan must not see micrOBU-4AFA (COM3: USB link in use: BLE advertising paused). If it loops again, the card now says why; "refused this phone's stored pairing" means forget micrOBU-4AFA in Android and pair again. COM3 shows encryption change status=... for the board's side. Confirmed 2026-09-23: CAMs and VAMs out, reception in, over BLE.
• BLE/ITS-G5 coexistence (the unmeasured risk from the hardware review). With BLE connected, count the sim car's CAMs received per minute and ours at the sniffer; compare with the same over USB. A clear drop, or reception stopping altogether, means the coex arbiter takes the radio off 5900 MHz (our channel is set behind the driver's back with phy_change_channel). Then BLE cannot be used while receiving, or needs a longer connection interval.
• Board reset recovery over BLE. Press RST mid-session: the app reconnects by itself and reconfigures on the first STATUS saying not configured, with no manual Connect. (Over USB a reset re-enumerates the port and needs a manual Connect, as before.)

Confirm the RX queue drop counter explains the bench-session frame drops / map flicker (added 2026-09-22)

Investigated the user's report of "OBU mode keeps dropping a few frames" and "v2x screen comes and goes" while bench-testing against obu-cam-transmistter. Found a real, previously invisible drop path: obu-firmware/main/main.c's wifi_promisc_rx_cb() calls xQueueSend(s_rx_queue, ..., 0) (queue depth 8) without checking the return value, so a burst of promiscuously-captured frames arriving faster than rx_forward_task can drain them (each drain can legitimately block up to ~400ms under USB/UART contention) silently vanishes. None of the existing EspLinkStatus counters (oversizeDrops/txFailures/rxCrcErrors) caught this class of drop.

This plausibly also explains the map symptom: UseCaseDetectionEngine.pruneStale() drops a remote station's marker after staleRemoteMs (3 s) with no CAM update. Measured 2026-09-22 via tools/cit_one_rx_watch.py --host 192.168.40.201 against obu-cam-transmistter's bench beacon (stationID 195936478 / 0x0BADC0DE): 75 CAMs in 25 s, ~3 Hz, not the 1 Hz this note assumed earlier — faster than assumed means more promiscuous captures per second and a shorter fuse on staleRemoteMs, both of which make the queue-overflow theory more likely, not less.

Fixed to be visible, not yet fixed to not drop: added a rxQueueDrops counter, checked xQueueSend's return value (main.c), wired it through the STATUS heartbeat as a new trailing uint16 field (serial_link.c/.h, SerialFrame.kt's EspLinkStatus), and surfaced it on the CAM Pinger card (MqttTopicViewerScreen.kt, string mqtt_cam_pinger_fw_counters). Host build untouched (serial_link.c/main.c aren't in the host test's standard-headers-only set); IDF build verification is the remaining pre-flash check. Deliberately did NOT bump s_rx_queue's depth from 8 — no real burst-size data yet, and guessing a bigger number against an unmeasured memory budget is exactly the kind of assumption microbu-hw-review flags as needing verification first, not capacity that's cheap to reason your way into.

Needs: a phone attached to the production OBU's native USB port, watching the CAM Pinger card, while obu-cam-transmistter (or real traffic) beacons.

• idf.py build succeeds (obu-firmware, IDF 6.1) — clean, both changed files compiled with no warnings, 17% flash free.
• Reflashed the production OBU on COM3 2026-09-22 (hash verified). Boot log confirms the new build (21e0149-dirty, compiled Sep 22 2026 14:14:09), clean boot, OCB @ 5900 MHz TX/RX armed, serial_link up ... 1 Hz heartbeat, no panic. Incidentally answers part of the "measure the OBU's actual transmit power" item below: this boot logged tx power: 72 quarter-dBm = 18.00 dBm (20.00 requested) — the driver is clamping below the requested 20 dBm at 5900 MHz, as that item suspected but had not measured.
• 25 s of steady-state console (no phone attached, obu-cam-transmistter beaconing nearby): silent — no crash, no oversize/rx queue full/crc warnings. Inconclusive on its own (successful forwards aren't logged, and nothing was attached to trigger the ~400 ms UART stalls the theory needs), but at least rules out a crash-on-boot regression.
• Confirmed the wider bench RF path independently via the CiT One OBU broker (py -3.11 tools/cit_one_rx_watch.py --host 192.168.40.201): heard obu-cam-transmistter's beacon cleanly, 75/25 s, GN source 14:00:02:00:00:00:00:01, position in the expected St. Georg route area. This is a different receiver from the production OBU though — it shows the beacon is genuinely on air, not that COM3 forwards every one of it without drops.
• Confirmed on real hardware, 2026-09-22. Installed the updated debug APK (previous build on the phone was from 2026-09-15, predating this fix entirely) on the Pixel 9 Pro (adb over Wi-Fi), relaunched against the freshly-reflashed COM3, and read rx queue drop via adb logcat -s UsbSerialTransport. The counter mechanism works end-to-end and the bug is real: rxQueueDrops was 0 at the last flash (14:22), read as 89 at first reconnect (14:48, ~26 min later), and 90 at a second reconnect (14:52). No oversizeDrops, txFailures, or rxCrcErrors moved at all, and zero decode FAILED lines — this queue is the only place frames are going missing. Nuance: over a clean ~4.5 min window in between (14:48→14:52) with obu-cam-transmistter actively beaconing at a measured ~3.33 Hz (matches the CiT One's 75/25 s independently) and 490+ CAMs decoding cleanly with steady cadence and no gaps, the counter did not move — it only ticked at connect/reconnect moments. So this is a low-rate, bursty drop (matches the user's own "a few frames" framing), not a continuous overflow under steady single-station traffic; it may be specific to WiFi/PHY activity around association or reconnect rather than raw beacon rate. Worth a longer, quieter-boot capture before sizing a s_rx_queue bump. Did not independently confirm the map-flicker connection this session — that needs eyes on the app's V2X screen while watching this same counter live, not just logcat.

On-device check of the full-screen V2X live map (added 2026-09-15)

The live map moved out of the V2X Monitor's view-mode row into its own full-screen destination (V2xMapScreen, route v2x_map), reached from the map button in that screen's header. Markers are now cached and reused across updates instead of being rebuilt on every incoming message, and SPATEM intersections are drawn as traffic lights at the position of the RSU's own CAM. All of that compiles and the unit tests pass, but none of it has been seen with live traffic.

Needs: the phone with the app, plus a CAM/DENM/SPATEM source - either the CiT One, or the OBU ESP32-C5 with a second board or a real RSU transmitting.

• Both hardware modes: tap the map button, confirm the map fills the screen (no status bar, no bottom nav) and the back button returns to the V2X Monitor.
• Panning stays smooth while CAMs are arriving - this is what the marker reuse is for. Compare against the old behaviour if it still judders.
• Touching the map stops it recentring; the location FAB resumes follow and lights up.
• A DENM shows the warning triangle, and a SPATEM intersection shows a traffic light with the lamp matching the Dashboard's SignalCard for the same intersection.
• Near a real RSU: confirm the RSU is drawn once, as a traffic light, not as a CAM pin with a light on top of it. If the RSU sends SPATEM but no CAM, the "signals not shown" note should appear instead - worth knowing which of the two the HAW RSUs actually do.

Over-the-air check of the GN lifetime fix (added 2026-09-11)

geonet.c now writes GN lifetime 0x05 (1 s) instead of 0x83, which decoded to 3200 s. Changed in both obu-firmware and obu-cam-transmistter. Both still build (IDF 6.1 / 5.5.4), and the compiled geonet_wrap_shb stores the new byte, but it has not been seen on air yet. Nothing else reads this byte (gn_unwrap.c ignores it, the app never sees GN headers), so the app does not need updating alongside the firmware.

Needs: the phone with the app, the OBU ESP32-C5, and a second ESP32-C5 running its-g5-receiver-firmware to capture with.

• Flash obu-firmware (see obu-firmware/FLASHING.md).
• Connect the phone, let it send CAMs, and confirm the CAM Pinger's tx fail counter stays 0.
• Capture with the receiver into its-g5-receiver-firmware/recordings/.
• Run python obu-firmware/test/pcap_gn_tally.py its-g5-receiver-firmware/recordings/<capture>.pcap. The rows for the phone's pseudonym MACs must show SHB, port 2001, lifetime 0x05, exactly like every other station's CAMs.
• While the phone is connected: real-station CAMs/DENMs still reach the app (RX path unchanged).

Partial check possible with one board and no phone: flash it, idf.py -p COMx monitor, and look for OCB @ 5900 MHz - TX/RX armed. That proves the new build boots and brings the radio up, not that it transmits correctly.

Measure the OBU's actual transmit power (added 2026-09-14)

Nothing in this project has ever measured it. main.c asks for 20 dBm (esp_wifi_set_max_tx_power(80), 0.25 dBm units) and the build's ceiling is the same (CONFIG_ESP_PHY_MAX_TX_POWER=20), but a request is a ceiling, not a guarantee: the driver clamps it to its own calibrated table, and 5900 MHz is above the range this chip is rated for, so the table actually in use is channel 177's. The firmware now reads the value back and logs it at boot, which records what the driver admits to, not what leaves the antenna.

• Flash and idf.py -p COM3 monitor, then note the tx power: line. A value below 80 means the driver clamped the request, which the code alone cannot tell you.
• Relative check with the second ESP32-C5 on its-g5-receiver-firmware: capture at a measured distance in a straight line, read the RSSI the receive path already reports, and record distance and RSSI together. This gives a comparable number between builds and antennas, which is what matters for range work, without any lab equipment.
• Only a spectrum analyser or a calibrated reference receiver gives real radiated power. Worth it only if the range result looks wrong, or if the thesis needs an absolute figure.

For context: ETSI allows up to 33 dBm EIRP on the ITS band, and production OBUs sit around 20 to 23 dBm, so the requested figure is in the right region if the PA really keys it there.

obu-cam-transmistter yawRateConfidence fix (added 2026-09-11)

Its cam.c (compiled into that firmware) wrote yawRateConfidence as 3 bits / 7 instead of 4 bits / unavailable(8), the bug the app fixed on 2026-08-20. Fixed in it and in obu-firmware's reference copy; asn1tools now decodes the CAM and re-encodes it byte-identically, and it builds on IDF 5.5.4. No board runs this firmware right now (the production OBU runs obu-firmware), so this only matters if it is flashed again:

• After flashing it: capture, run pcap_gn_tally.py, and decode the CAM payload with asn1tools (py -3.11, modules in asn1/).

Signed-message reception and exact payloads (added 2026-09-11)

obu-firmware's gn_unwrap.c now unwraps TS 103 097 signed packets (signature not verified, reported as V2X_RX flags bit1) and cuts every message to the length its header declares, dropping the 8 bytes the chip's RX appends to each frame, which were forwarded to the phone until now. Verified on the host (obu-firmware/test/host: chain, replay of all recordings against asn1tools, 50M-iteration fuzz) and built on IDF 6.1, but not flashed: the production OBU still runs the 2026-09-10 build. Needs the OBU with this build, the phone, and signed traffic - real vehicles or RSUs, since the bench CiT One sends unsigned. A second ESP32 running the receiver firmware is optional, but shows what was on air at the time.

• Flash obu-firmware (this also carries the GN lifetime fix above).
• Near signed traffic: signed CAMs/DENMs appear in the app, and a simultaneous capture shows them on air (pcap_gn_tally.py lists them as secured).
• Unsigned bench traffic still decodes in the app as before (messages now arrive 8 bytes shorter).
• The heartbeat's oversize counter still counts over-long messages (e.g. road SPATEMs).

CiT One custom CAM injection over v2x/tx/v2/cam (added 2026-09-14)

The haw-002 unit now runs the special firmware: Cohda's own CAM transmission disabled, and a V2X-Gateway build that accepts a SendV2XMessage (schemas.consider-innovation.de/its-s/ v2x_interface.proto) carrying a UPER CAM on v2x/tx/v2/cam. tools/cit_one_cam_tx.py builds and publishes those from a PC; its --self-test passes offline, proving only that the bytes match CamEncodeGoldenTest.kt and that the protobuf wrapper round-trips. Nothing about what the OBU does with them is established.

Reach the broker over Wi-Fi or Ethernet for now - the USB-peripheral-mode link needs the phone to be USB host on a 172.25.1.0/24 interface with no DHCP server, which Android cannot configure from inside an app.

Needs: the CiT One haw-002 on the same network as a PC, and a second ESP32-C5 running its-g5-receiver-firmware sniffing G5CC (-c 5900) to capture with.

Bench run 2026-09-14, PC -> haw-002 (192.168.3.201), captured on the RSU (192.168.3.202, not .2.202 - that address does not route). tools/cit_one_rx_watch.py decodes what a unit hears. Result: the injection path works end to end, with one blocker found.

• Publishes without the broker refusing the topic. 1.00 Hz, confirmed by subscribing to v2x/tx/v2/cam on the OBU itself.

• The RSU hears our CAMs on air, 1.00 Hz, matching what we publish.

• ItsPduHeader is expected in the payload - we send it included and it decodes.

• BTP destination port 2001. GN source address 08:00:26:93:92:01:91:dc, the OBU's.

• Our CAM content goes out intact: position, speed (417), heading (639), width (7) and length (18) arrive byte-exact. The gateway does not touch the content.

• The gateway overwrites stationID with the OBU's own (999999 -> 4033890855, which matches own_info.stationID on v2x/rx/obu_gnss). This is what the "OBU owns identity" decision wants, so --follow-obu-identity is not needed on this unit.

• BLOCKER: Cohda's own CAM is still transmitting. Fixed 2026-09-14 by disabling CAM in a second conf file: the RSU now hears only our stream, 0 CAMs with the stack's unavailable dimensions over 30 s. Note the stack restart gave the unit a new identity (stationID 4033890855 -> 2553426533, GN source 08:00:26:... -> 08:00:a2:...), which is expected under ItsGnLocalAddrConfMethod = 2 (anonymous, random at boot).

• Re-checked: 41 published / 41 heard over 40 s, 1.02 Hz both ends, inter-arrival a steady 1.0 s. 100% delivery, no gateway rate limiting. An earlier 0.40 Hz sample was the stack still settling after the restart and did not persist. Two topics the v6 API does not document, found by subscribing to # on haw-002:

• v2x/loopback/cam - a RecvV2XMessage (btpHeader.type=2) carrying each CAM the unit transmits, 1:1 with what we publish and after the gateway's stationID rewrite. This is the TX confirmation we were going to ask consider it for: it makes "did my CAM go out, and under which identity" answerable on the transmitting unit alone, without an RSU or a second ESP32.

• v2x/rx/obuinfo at 10 Hz - the protobuf OwnStationInfo (binary twin of obu_gnss; field 2 decodes to the same stationID, field 10 to the same heading). Output only, so it is not the content-feed input we speculated about.

• Wire v2x/loopback/cam into cit_one_rx_watch.py as a local TX check.

• Sanity-check the rate: --rate 4 should produce 4 CAMs/s on air, since ItsDCCEnabled = 0 on this unit.

Set up host testing

• Install MSYS2 UCRT64 gcc (done 2026-09-11: gcc 16.2.0, GNU Make 4.4.1; chosen over WSL, vanetza is not going to be built). Setup and the PATH gotcha: obu-firmware/test/host/README.md.
• Host round-trip test obu-firmware/test/host/test_chain.c (geonet_wrap_shb -> dot11p_build_frame -> gn_unwrap_its, byte-checked against the standard). Done 2026-09-11: 491 checks, 0 failed. Run make in that folder before flashing any firmware fix.
• Replay of the recorded captures (test_replay.c + check_replay.py, independent asn1tools check). Done 2026-09-11: C and Python agree on all 15 145 records.
• Mutation fuzzer fuzz_gn_unwrap.c, inputs against a no-access guard page. Done 2026-09-11: 50 000 000 iterations, no crash. make runs a 2 000 000-iteration pass every time.

Firmware ideas from the vanetza review (2026-09-11, not started)

Suggested order after the host tests exist:

• Read secured packets (GN NextHeader=2) without verifying them. Done 2026-09-11 in gn_unwrap.c, host-verified; flagged to the phone as V2X_RX flags bit1. On-air check under "Waiting on hardware".
• Forward the full GeoBroadcast area: shape (circle/rectangle/ellipse), DistanceB, angle, appended to the V2X_RX prefix behind a capability bit. Port vanetza's geonet/areas.cpp inside_or_at_border to the app, which currently treats every area as a circle.
• RX filtering before the serial link: duplicate detection for GBC (last 8 sequence numbers per source, as vanetza does), drop our own frames, reject GN version != 1.
• Read the DCC-MCO field (the 4 "reserved" bytes of an SHB header): neighbours' channel busy ratio for free.
• Minimum TX gap in firmware as a DCC safety net (vanetza reactive table: 60 ms relaxed ... 460 ms restrictive), with a CBR estimate in the heartbeat.
• Generic V2X_TX message (BTP port, SHB/GBC, traffic class, lifetime, area) so the phone can send DENM and VAM without reflashing. Consider QoS Data frames: vanetza's Cohda receive path drops non-QoS ones.

Dropped: building vanetza as a GN/BTP oracle. Real captures (pcap_gn_tally.py), the host round-trip test and asn1tools for UPER cover what it would have checked.

Follow-ups found 2026-09-11

• App: show the signed flag. V2xRxFrame.parse in SerialFrame.kt only reads bit0 of the flags byte; read bit1 (signed, not verified) and show it where messages are listed.
• Messages that do not decode with asn1tools. In the recordings, 56 from the CiT One (aa:f8:76:7d:bd:ad: 54 CAMs of 245 bytes, 2 DENMs of 402 bytes) and one 218-byte CAM from 6e:94:03:1b:05:26 fail against cam_1_4_1/denm_1_3_1 + cdd_1_3_1_1, with or without the old trailing bytes. A newer module version on the sender, or a sender bug; check what the app's decoders make of them (check_replay.py lists the records).