a08494b56a7e2c50e561b5b3a6db757f08e714d9
12
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Ashin Walpola
|
d2fd222a62 |
Sign ITS messages on the ESP32-C5 with vanetza-idf, over USB or BLE
obu-firmware is now a port of the colleague's standalone VRU station (microbu-esp32c5/firmware, kept beside this repository and gitignored): the vanetza-idf C-ITS stack with the TS 103 097 security entity, credentials in NVS, the station-link v1 protocol over the native USB port (frame type 0x10 in the existing 0xAA55 framing) and over a BLE GATT peripheral, and its ITS-G5 radio adapter. The phone still builds CAM and VAM; the board adds GeoNetworking/BTP and signs with the provisioned authorization ticket. The private key never leaves the board. Builds with ESP-IDF 6.0.2 only, which vanetza-idf pins for the radio's private driver ABI. The previous C firmware stays on disk unbuilt; a full-flash backup of the bench board is kept in firmware-backups/ (gitignored). Changed against the colleague's firmware, marked MicrOBU: in the sources: - Reception unchanged for the app. vanetza-idf drops what it cannot verify (unsigned traffic, every RSU), so each captured frame also goes through the previous gn_unwrap.c and reaches the phone as link opcode V2X_RX (0x85), whose body is the old SERIAL_MSG_V2X_RX payload. - Unsigned transmission still possible, with the previous geonet.c header; the phone chooses per message. - Console on UART0 (CH343 port); the native USB port carries only link frames. - BLE advertising pauses while the USB link is in use: BLE and ITS-G5 share one RF front end. - NVS 80 KB (app at 0x20000). At 24 KB, with Wi-Fi settings the previous firmware left behind, the BLE bond could not be stored and the phone had to pair on every connection. - Bench fixes: the radio queue is drained before the first PoTi (no RX and ~177 queue drops before); the station loop waited pdMS_TO_TICKS(5) = 0 ticks at 100 Hz and starved the idle task; the 2.4 KB RX capture buffer is off the Wi-Fi task stack; BLE notifications longer than the MTU are dropped instead of cut short, MTU 517; serial writes are skipped with no USB host. - Manual country policy and TX-power read-back from the previous radio setup; logs for BLE encryption changes and the number of stored bonds. Verified on the bench board (COM3) with the phone over USB and BLE: CAM and VAM, signed and unsigned, go out; reception of the sim car and the RSU's CAM/SPATEM/MAPEM continues; the board survives app restarts and reconnects. See docs/06-signed-its-vam-ble.md. |
||
|
Ashin Walpola
|
7285fa19b7 |
Count and surface RX-queue drops on the ESP32-C5's promiscuous path
wifi_promisc_rx_cb() fed s_rx_queue with a 0-timeout xQueueSend() and never checked whether it succeeded, so a burst of captured frames arriving faster than rx_forward_task could drain them vanished with no counter anywhere - none of oversizeDrops/txFailures/rxCrcErrors caught it. Added a rxQueueDrops counter, threaded it through the STATUS heartbeat as a new trailing uint16 (old firmware/app on either side still parse fine), and surfaced it on the CAM Pinger card. Confirmed on the bench: flashed to the production OBU (COM3) and installed the matching app build on the phone, then watched the counter over logcat against obu-cam-transmistter's ~3.3 Hz beacon - it is real (0 -> 89 -> 90 across two sessions) but bursty around connect/reconnect rather than a continuous overflow under steady single-station traffic. |
||
|
Ashin Walpola
|
75d6d3b85c |
Receive signed ITS messages and forward each at its declared length
Signed packets. A GeoNetworking Basic Header NextHeader of 2 means a TS 103 097 (IEEE 1609.2) envelope follows, with the Common Header inside it. gn_unwrap_its rejected all of these, and most real traffic is signed: the 2026-08-17 capture holds 157 signed frames from 15 source MACs against 2 unsecured stations. It now opens a COER-encoded signedData, or a bare unsecuredData, and parses the inner packet as before. The inner packet comes first inside tbsData, so the certificate and signature are never parsed, and the signature is not verified - the firmware has no trust store. Such messages reach the phone with the new V2X_RX flags bit1, signed but not verified. The app reads only bit0 and is unaffected until it learns the flag. Encrypted payloads, nested signing and the legacy v1.2.1 envelope are still rejected. All 157 recorded signed frames have the layout this reads, in all three COER length forms, and asn1tools decodes every envelope to the same inner packet. Payload bounds. Every frame recorded through the ESP32-C5's promiscuous RX, about 15 000 of them, ends in 8 bytes that are not part of the 802.11 frame and not a valid FCS. obu-firmware reads frames through the same API and took the rest of the frame as the message, so it forwarded those 8 bytes to the phone after every message. UPER decoders stop where the message ends, so nothing visibly broke, but the bytes cost serial bandwidth and 8 bytes of the DENM's headroom, and they stayed attached wherever raw payloads were stored or passed on. The payload is now exactly what the Common Header's payload-length field declares, which is also what separates a signed message from its signature. A frame longer than main.c's 800-byte capture buffer is now reported as truncated instead of being forwarded cut off, and counted as an oversize drop through the new serial_link_note_oversize_drop, as it was when the cut-off frame failed serial_link's size check. Host tests in obu-firmware/test/host build the firmware sources unmodified with MSYS2 gcc; `make` runs all three. - test_chain: frames from the firmware's TX code checked byte by byte against EN 302 636-4-1 and parsed back, including hand-built signed frames, the payload-length rule, the RX trailer, and every truncation length against a no-access guard page. 1731 checks, 0 failures. - test_replay and check_replay.py: all 15 145 recorded frames through gn_unwrap_its, cut to 800 bytes as on the board, and re-derived independently in Python with the envelope decoded by asn1tools. They agree on every record; 15 131 accepted, 157 of them signed. 11 043 of the 11 106 distinct messages re-encode byte-identically. The other 63 fail the same way with the old 8 bytes put back, so the boundary is not the cause: 5 are our own CAMs from before the 2026-08-20 yawRateConfidence fix, and the rest, from other stations, are a follow-up in TODO.md. - fuzz_gn_unwrap: random edits of every recorded frame, each run against the guard page. 50 000 000 iterations, no crash. obu-firmware/test/pcap_gn_tally.py tallies GeoNetworking header fields per station over captures; it is how the other stations' lifetimes were measured. TODO.md collects what is still open, including the on-air check for this change: it builds on IDF 6.1 but has not been flashed. |
||
|
Ashin Walpola
|
1baae2c5f6 |
Encode yawRateConfidence in 4 bits in the firmware CAM encoders
YawRateConfidence has nine enumerands, degSec-000-01(0) to unavailable(8) (cdd_1_3_1_1.asn), so UPER needs 4 bits and "unavailable" is 8. Both firmware copies of cam.c wrote 3 bits with value 7, which is also the wrong symbol (outOfRange), and every field after it shifted by one bit. The app's CamUperCodec fixed the same line on 2026-08-20; these two copies were missed. obu-cam-transmistter compiles its cam.c, so a board running that bench beacon sent CAMs no standards-compliant station could decode. obu-firmware's copy is reference only - it is not in SRCS, since the phone encodes the CAM - and is kept in step because the app's encoder was ported from it. The production OBU runs obu-firmware and was never affected. Every other field width was compared against CamUperCodec.kt and matches. Checked with asn1tools against asn1/cam_1_4_1.asn and cdd_1_3_1_1.asn: the CAM both fixed copies emit decodes with every expected value and re-encodes byte-identically, while the version before this change fails on yawRateConfidence. obu-cam-transmistter builds on IDF 5.5.4. |
||
|
Ashin Walpola
|
8871708a98 |
Send the GeoNetworking lifetime as 1 s, not 3200 s
geonet_wrap_shb wrote lifetime 0x83, commented as about 60 s. The field holds the multiplier in its upper six bits and the base in the lower two (50 ms, 1 s, 10 s, 100 s), so 0x83 is 32 x 100 s = 3200 s. That is over the 600 s itsGnMaxPacketLifetime a sender may use at all; vanetza refuses to send such a packet. The byte arrived with the Phase 03 commit as a placeholder and was never checked against the encoding. For a single-hop CAM this is non-compliance rather than a functional fault: nothing stores or forwards an SHB packet, so no receiver acts on the value, and no dropped CAM was ever traced to it. 0x05 (1 x 1 s) is what every other station in its-g5-receiver-firmware/recordings sends its CAMs with; the recorded GeoBroadcast DENMs use 0x79 (30 s). Changed in obu-cam-transmistter's copy as well. Both firmwares build (IDF 6.1 and 5.5.4), and the disassembled geonet_wrap_shb of each stores 0x05. Not yet seen on air: the production OBU still runs the 2026-09-10 build, and the on-air check is listed in TODO.md. |
||
|
Ashin Walpola
|
3eeccfb268 |
Send CAMs under the phone's position vector, not bench placeholders
Every field of the GeoNetworking Source Position Vector this firmware sent was a compile-time constant: the bench coordinates, speed 0, heading 0, TST 0, station type passengerCar and one fixed MAC. The CAM inside described a moving cyclist while the GN header around it described a car parked at the bench. SERIAL_MSG_CAM_TX_PV (0x05) puts a 24-byte prefix ahead of the CAM UPER: MAC, station type, PAI, TST, latitude, longitude, speed and heading, all values the phone already has when it builds the CAM and none of which this chip can know. geonet_wrap_shb now takes them as a gn_lpv_t, and tx_radio_task hands the same MAC to dot11p_build_frame, so the 802.11 source address and the GN_ADDR MID stay one address across a pseudonym change. Speed is clamped rather than masked, since an overflowing 15-bit value flips its sign bit and reads as travelling backwards. This reverses the Phase 03 decision that the firmware owns the pseudonym. A pseudonym only protects anyone if the MAC, the GN_ADDR and the CAM's stationID change together, and the phone owns the stationID. The heartbeat gains a capability byte (payload[7], bit0 = CAM_TX_PV), appended so an app reading the first 7 bytes is unaffected. The app sends 0x05 only once it sees that bit, so app and firmware can be updated in either order. CAM_TX (0x01) is still handled and falls back to the bench values, with the station type corrected to cyclist to match the CAM. Verified on air from the COM10 test board, decoded independently by the CiT One's gnHeader: 24 of 24 CAM_TX_PV frames matched the sent position vector field by field, and so did the CAM station ID. The legacy path delivered 23 of 24 frames with no field mismatches. Flashed on the COM3 OBU and its boot log is clean. Also corrects the SERIAL_LINK_MAX_PAYLOAD comment, which still named the 400-byte receive capture buffer as the ceiling on the RX path. That buffer is 800 bytes now, so the serial link is the ceiling, and larger payloads are dropped and counted there. |
||
|
Ashin Walpola
|
04b0076b8b |
Move the RX capture buffer off the WiFi driver's callback stack
rx_item_t is ~800 bytes at RX_FRAME_MAX_LEN, and wifi_promisc_rx_cb declared one as a local. That callback runs on the WiFi driver's own task, already several frames deep in the driver's call chain, on a stack of roughly 3.5 KB (CONFIG_ESP_WIFI_TASK_STACK_SIZE, left at its default). Putting a fifth of that stack into a single local is a stack-overflow risk that only appears under real traffic - in front of an RSU rather than on the bench - and would present as a random panic rather than anything pointing at its cause. Both instances are now static: one in the callback, one in rx_forward_task. Safe because each is touched by exactly one task, so there is no re-entrancy to guard against; the same reasoning serial_link.c already uses for its static send buffers. xQueueSend copies the struct out before returning, so reusing the callback's buffer on the next frame is fine. Firmware-only, no protocol change, so it does not require a matching app install. Re-verified against live traffic after flashing: 1094 frames over 125 s with zero decode failures, USB errors, detaches, crashes or mutex timeouts. SPATEM capture rose from 3.20/s to 3.98/s against a theoretical maximum of 4.00/s, which is the direction relieving stack pressure would produce, though RF geometry moves between runs and this is not proof. Report updated with T9, the accepted 512-byte ceiling, and the decision to drop Phase B: the intersection use case is CAM-driven and needs none of it. |
||
|
Ashin Walpola
|
a5ad3dcc5d |
SPATEM receive, RSU CAM decode, and two ASN.1 encoding fixes
SPATEM over the air - gn_unwrap.c accepts BTP-B port 2004 alongside 2001/2002. The serial protocol already carries the port in its V2X_RX prefix, so nothing else changed there. Note the crossover that makes this easy to get wrong: SPATEM is port 2004 but messageID 4, while MAPEM is port 2003 and messageID 5. - SpatemUperCodec decodes SPAT down to per-signal-group phase and timing. The bit layout was validated by replaying 79,042 real SPATEMs - the whole 2026-03-18 drive across 7+ RSUs plus the bench trigger - against asn1tools using the ETSI modules. All 79,042 matched on every field, none hit an unsupported branch. Two traps are pinned by tests: TimeChangeDetails is the one SEQUENCE here that is NOT extensible (5 optional bits, no extension bit), and maneuverAssistList cannot be skipped when present - it is variable-length, so it has to be walked to find where the next movement starts. - The V2X list shows one row per intersection with each signal group coloured by phase and a countdown where the RSU supplies timing. TimeMark wraps hourly, so the countdown corrects for it; without that it reads hugely negative once an hour, precisely when someone is watching it. - Entries expire after 15 s, much shorter than DENM's window: a traffic light that stopped updating is not "still green". Size caveat, deliberately deferred: SERIAL_LINK_MAX_PAYLOAD is still 512, so a SPATEM over ~498 bytes is counted as an oversize drop. The bench RSU sends 58 bytes and is unaffected, but real road RSUs measured 555 median / 1243 max, so roughly 70% would not arrive. Raising the cap also requires enlarging RX_FRAME_MAX_LEN and moving rx_item_t off the WiFi driver's callback stack, where it would otherwise overflow. RSU CAM decode - HighFrequencyContainer is a CHOICE, and a roadside unit picks rsuContainerHighFrequency, which carries no kinematics at all. The decoder bailed on that branch, so every RSU CAM was dropped - including the bench RSU, which sends CAM and SPATEM from the same station id. It now decodes for position and stationType. - RSU CAMs are kept out of UseCaseDetectionEngine. They arrive as a permanently stationary station at a fixed point, which is exactly the shape the stopped-vehicle and intersection-movement use cases match, and would raise a standing false alert for as long as the RSU was in range. CAM transmit: yawRateConfidence - YawRateConfidence has nine enumerands (0..8), so UPER needs 4 bits and "unavailable" is 8. The encoder wrote 3 bits with value 7 - one bit short and the wrong symbol - shifting every field after yawRate for any standards-strict receiver. The decoder read 3 bits too, so phone and ESP32 agreed with each other and with nothing else. - This is the third instance of that exact failure mode in this project, after CurvatureCalculationMode and the GeoNetworking reserved bytes. A round-trip test through our own decoder structurally cannot catch it, so CamEncodeGolden Test asserts the bytes asn1tools produces instead: it decoded this encoder's output and re-encoded it byte-identically. Confirmed on air afterwards - 26 of our own CAMs captured back off the OBU's receiver, all 26 accepted, where the same decoder rejected them before. DENM - Hazards now expire 60 s after their last repetition. This needs a clock, not just a filter: both source flows only emit when a DENM arrives, so a sender that drives away or loses power would never trigger a recompute and its hazard would stay on screen indefinitely. - The MQTT path was dropping every DENM for two independent reasons, both found by checking the payload against CI-CiT-MQTT_API_Documentation-v6 listing 2.6 rather than guessing: the station id key is originatingStationId, and eventPosition IS a GeoJSON Point rather than an object containing one. Also parses termination (presence is the signal), sequenceNumber, stationType and the RFC3339 detectionTime. Note roadSideUnit is 15, not 12 - the enumeration has a gap after tram(11). V2X screen - The decoded CAM/DENM list now renders on the CiT One path too; it was gated to the ESP32-C5 path and CiT One fell through to the raw MQTT topic list. Those topics move to their own tab, hidden on the ESP32-C5 path where there is no broker. Testing - Adds org.json as a test-only dependency: the android.jar stub throws "not mocked" on every JSONObject call, which made the MQTT payload parsers untestable off-device. - 23 V2X tests pass. EventDetectorTest's 4 failures are pre-existing and untouched by this change. |
||
|
Ashin Walpola
|
0ccb867228 |
DENM over-the-air receive on the ESP32-C5 path
The firmware forwarded CAM only: gn_unwrap_cam accepted single-hop broadcast (HT=5) and BTP port 2001, so every DENM was dropped before it reached the phone. Real OBUs disseminate DENM by GeoBroadcast (HT=4), whose 44-byte extended header also carries the hazard's relevance area - materially more useful on a map than the sender's own position, since a sender may be relaying for someone else. Firmware - gn_unwrap_cam -> gn_unwrap_its: accepts GeoBroadcast alongside TSB/SHB, and BTP ports 2001 and 2002, extracting the GeoBroadcast destination area. Both extended-header lengths were measured against live air capture rather than read off a spec table. Secured packets (Basic Header NextHeader=2) are rejected rather than misparsed. - SERIAL_MSG_CAM_RX (0x02) superseded by SERIAL_MSG_V2X_RX (0x04): a 14-byte prefix carrying BTP port, RSSI and the destination area. Adding MAPEM later needs a decoder on the phone but no protocol change. 0x02 stays reserved so the numbering is not silently reused. - Promiscuous RX capture buffer 400 -> 800 bytes. A real GeoBroadcast DENM is around 500 bytes on air and was being truncated mid-payload, which no amount of correct unwrapping downstream could have recovered from. - geonet_wrap_shb, both firmwares: the SHB extended header is 28 bytes, not 24. The Source Position Vector is followed by a 4-byte reserved field; without it a standards-strict receiver reads the CAM payload's first two bytes as the BTP destination port. App - DenmUperCodec: UPER decoder for the ManagementContainer and the SituationContainer's eventType. ValidityDuration is 17 bits, not 16, and ManagementContainer, SituationContainer and CauseCode each carry their own extension bit - a single wrong bit made a real frame read causeCode 47 instead of 94. - DenmEvent gains actionID (originatingStationID + sequenceNumber), stationType, termination, detectionTime, relevance radius and RSSI. Dedup keys on actionID where available, so a termination lands on the event it ends instead of creating a second pin. - denmEvents merges the MQTT and over-the-air sources and drops terminated events. The V2X list view now shows hazards above the CAM stations; it previously took no DENM parameter at all, so hazards reached the map but never the list. - DenmParser: the Use Case API sends causeCode as a string enum, so reading it as an Int always yielded null. Testing - DenmAirReceiveTest covers the V2X_RX prefix and the decoder using real frames from a live capture as fixtures. Expected values were cross-checked against the ETSI ASN.1 modules via asn1tools, which agreed on all 1885 decodable DENMs across the capture set, every field including detectionTime. - Verified on hardware: a CiT One HLN-SV DENM decodes as cause 94/0 with a 1000 m relevance radius at 1 Hz alongside CAM, with no decode failures and no unexpected BTP ports. Also replaces em dashes with hyphens throughout the user-facing strings, including the German translation. |
||
|
Ashin Walpola
|
f507a8a9fd |
Fix UPER encoding of CurvatureCalculationMode; verified on hardware
CurvatureCalculationMode is the one extensible ENUMERATED in CAM:
ENUMERATED {yawRateUsed(0), yawRateNotUsed(1), unavailable(2), ...}
UPER encodes an extensible ENUMERATED as an extension bit followed by the root
index - 1 + 2 = 3 bits. All three of our encoders wrote only the 2-bit index,
shifting yawRate and the entire low-frequency container one bit early for any
standards-compliant receiver.
It went unnoticed because every end of this project shared the mistake: the
Kotlin codec was ported bit-for-bit from cam.c, so phone and ESP32 agreed
perfectly with each other and with nothing else. Confirmed against the ETSI
ASN.1 in the C-ITS-Parser checkout, where rasn marks this type - and only this
type - #[non_exhaustive].
Fixed in all three copies of the encoder (app CamUperCodec.kt,
obu-firmware/main/cam.c, obu-cam-transmistter/main/cam.c) plus the decoder,
which now rejects rather than misreads a set extension bit. Frame size is
unchanged at 43 bytes. Transmitter reflashed and the phone decodes its CAMs.
Also in this change:
- serial_link: skip send_frame entirely when no USB host is attached, and raise
the tx mutex timeout above the worst-case hold. With the phone unplugged every
write blocked its full timeout while holding the lock, so forwarded CAM_RX
traffic starved the 1 Hz heartbeat - observed as "tx mutex timeout, dropping
frame" on the console, and it would have tripped the phone's link watchdog.
Verified gone on hardware.
- Log decoded and failed CAMs in CamUseCaseRepository. "The app shows nothing"
had two indistinguishable causes; a silent `?: return` made this bug much
harder to find than it needed to be.
- Remove the ESP32 send-only/send-and-receive toggle. Reception can't be
disabled in firmware (raw TX only works while promiscuous), so it was an
app-side filter pretending to be a radio control.
- V2X monitor follows the serial link state on the ESP32 path instead of MQTT,
which is permanently disconnected there; CAM intake is gated on the link being
up, and engine state is cleared when it drops.
- About screen: 0.5.0, Phase 03.
- Track obu-cam-transmistter, the bench CAM transmitter. Its cam.c is compiled
(unlike obu-firmware's reference copy) and must stay bit-identical to the other
two - this commit is what that coupling costs when it's broken.
- Document the two-toolchain split: this project builds on IDF 5.5.4, obu-firmware
on the pinned 6.1. Exporting both in one shell fails confusingly.
|
||
|
Ashin Walpola
|
64fb78590e |
Phase 03: ESP32-C5 serial link hardening + bench diagnostics
Enumeration: - Merge the library's stock probe table instead of replacing it, so adding Espressif 0x303A/0x1001 doesn't drop every other supported device - Select the ESP32-C5 by VID/PID rather than list position - Log USB interface descriptors to distinguish CDC data from the JTAG interface Lifecycle: - Don't close the shared port in MqttViewModel.onCleared() - the foreground recording service outlives the ViewModel and would beacon into a dead port - Handle ACTION_USB_DEVICE_DETACHED so the UI stops reporting a stale link - Implement the STATUS heartbeat on both sides (1 Hz) plus a phone-side watchdog - Surface write failures and firmware drop counters on the CAM Pinger card Protocol: - Assert DTR/RTS on open (unverified on hardware - see FLASHING.md step 5) - Raise SERIAL_LINK_MAX_PAYLOAD 160 -> 512 on both sides; real third-party CAMs exceed 160 and were being silently dropped at the resync branch - Move the enlarged buffers off task stacks; serialize send_frame with a mutex Firmware and app must be updated together - a 512/160 mismatch fails silently. |
||
|
Ashin Walpola
|
33c4ec5998 |
Phase 03: real CAM UPER codec + ESP32-C5 TX/RX serial link
- Firmware: rewrite obu-firmware TX loop to be serial-driven (no on-chip timer), add promiscuous RX + GeoNetworking/BTP unwrap (gn_unwrap.c), add binary UART framing to the phone (serial_link.c/.h). Drop local cam_encode() - CAM is now built on the phone. - Kotlin: byte-exact UPER CAM encoder/decoder ported from cam.c (BitWriter/BitReader/CamUperCodec), matching SerialFrame codec, real UsbSerialTransport (usb-serial-for-android), CamTransmitLoop (1Hz base rate, event/geofence boost, ESP32-C5-only), wired into CamUseCaseRepository for RX and TripRecordingService for TX. - Add V2X message retention: persist all CAM (own+remote) to Room while recording, drop otherwise (DB v2 -> v3 migration). - Add jitpack repo + usb-serial-for-android dependency. Fixes: UsbSerialTransport now uses SerialInputOutputManager.start()/stop() (this lib version manages its own thread internally) instead of manual Runnable/Thread submission, which didn't compile. |