diff --git a/.gitignore b/.gitignore index 4fe1f84..bed65e0 100644 --- a/.gitignore +++ b/.gitignore @@ -46,3 +46,6 @@ sdkconfig.old # Office lock files. Word/Excel create these beside a document while it is open # and remove them on close, so they are transient and machine-local. ~$* + +# Captures are large data files, not source (see capture/README.md). +/capture/recordings/ diff --git a/TODO.md b/TODO.md index a2c01c4..8ae371b 100644 --- a/TODO.md +++ b/TODO.md @@ -103,14 +103,14 @@ round-trip test and `asn1tools` for UPER cover what it would have checked. ## Follow-ups found 2026-09-14 -- [ ] **Commit the `live_capture.py` fix in the receiver repo.** Its console inserts a CR before - every LF, which also hits every 0x0a byte of the binary pcap stream, shifting pcap record - headers and frames. A 787 KB capture parsed cleanly for only 82 of ~2000 records, and DENMs - showed up on nonsense BTP ports. `undo_crlf()` now reverses it on the raw stream before - framing; afterwards a capture parsed to EOF and DENMs read as port 2002. This is the - "occasional byte inserted mid-frame" in the older recordings, so **every capture taken before - 2026-09-14 is truncated at its first corrupted record** - re-measure anything derived from - them. The change is in the `its-g5-receiver-firmware` repo, uncommitted. +- [x] **Capture tooling moved into this repo** (`capture/`), with the CR-insertion fix. The + sniffer's console inserts a CR before every LF, which also hits every 0x0a byte of the binary + pcap stream, shifting pcap record headers and frames. A 787 KB capture parsed cleanly for + only 82 of ~2000 records, and DENMs showed up on nonsense BTP ports. `undo_crlf()` reverses + it on the raw stream before framing; afterwards a capture parsed to EOF and DENMs read as + port 2002. **Every capture taken before 2026-09-14 is truncated at its first corrupted + record** - re-measure anything derived from them. +- [ ] `capture/dump_pcap.py` reads the same console and still needs the same treatment. - [ ] **Do not open COM3's console while the phone is attached.** Opening it toggles DTR/RTS on the CH343 and resets the OBU, which drops the phone's USB link and needs a manual Connect. diff --git a/capture/README.md b/capture/README.md new file mode 100644 index 0000000..19679a7 --- /dev/null +++ b/capture/README.md @@ -0,0 +1,102 @@ +# Sniffer board and capture tooling + +How to put an ESP32-C5 on the ITS-G5 channel as a passive sniffer, pull its captures onto this +PC, and check what is on air. The sniffer firmware itself is the third-party +`its-g5-receiver-firmware` checkout beside this repo; only the tooling and these notes are ours. + +| File | What it does | +|---|---| +| `live_capture.py` | Streams the device's captures into a growing `.pcap` while it runs. The usual choice. | +| `dump_pcap.py` | Pulls one capture out of the device's in-memory buffer after the fact. | +| `../obu-firmware/test/pcap_gn_tally.py` | Tallies GeoNetworking headers per station over a `.pcap`. | + +One-time: `pip install pyserial` (present in Python 3.11 on the bench PC, so `py -3.11` works). + +## Which port + +The sniffer firmware's console, and with it the pcap stream, goes out **UART0** - the board's +USB-bridge port (a CH343, its own COM number), not the native USB-C port. A board with only one +USB-C port cannot be used as a sniffer for this reason. On the bench this has been COM5 and, after +a re-enumeration, COM8. + +## Live capture (preferred) + +```powershell +cd capture +py -3.11 live_capture.py COM8 +``` + +It writes `recordings/capture_.pcap` next to itself, flushing after every packet, so +the file can be read while it grows. Stop it with Ctrl+C. Use `-o ` to write elsewhere; +`recordings/` is gitignored, since captures are large and are data rather than source. Captures +taken before 2026-09-14 are still in `its-g5-receiver-firmware/recordings/`; the host tests read +both directories. + +### The CR insertion, and why captures used to be corrupt + +ESP-IDF's newlib console converts LF to CRLF on its way out, and that applies to every `0x0a` byte +of the **binary** pcap stream, not only to log text. Each inserted CR shifts everything after it, +so pcap record headers and captured frames alike come out corrupt, and the file stops being +parseable at the first occurrence. + +Measured on 2026-09-14: a 787 KB capture parsed cleanly for only 82 of about 2000 records, and +DENMs appeared on nonsense BTP ports because their payloads contain `0x0a` often. `undo_crlf()` in +`live_capture.py` reverses it on the raw stream before any framing, which is exact; afterwards a +capture parsed to EOF and DENMs read as port 2002 again. + +**Captures taken before 2026-09-14 are truncated at their first corrupted record.** Anything +measured from them is worth re-checking. `dump_pcap.py` reads the same console and has not been +given the same treatment yet. + +## Checking a capture + +```powershell +py -3.11 ..\obu-firmware\test\pcap_gn_tally.py recordings\capture_.pcap +``` + +One row per station, packet type, BTP port and GN lifetime. For the messages themselves, decode +the payloads with `asn1tools` against the modules in `../asn1/` and re-encode them: identical bytes +mean the message was read exactly, wrong bytes mean it was not. `obu-firmware/test/check_replay.py` +does this over a whole capture. + +## Flashing the sniffer firmware + +From the receiver checkout, with its **pinned** ESP-IDF (not the global 5.5.4 install): + +```powershell +cd its-g5-receiver-firmware +git submodule update --init --recursive +.\esp-idf\install.bat +Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass +.\esp-idf\export.ps1 +idf.py set-target esp32c5 +idf.py -p COM8 -b 921600 flash +``` + +Its `sdkconfig` for a bare board (nothing wired) needs SPI Ethernet off, and the pcap destination +set to Memory, both under `idf.py menuconfig`. Wired variants have ready-made configs in that +checkout: `sdkconfig.proto-w5500`, `sdkconfig.proto-enc28j60`, `sdkconfig.proto-spi-eppp`. + +To reflash a board that already has a built image, without a toolchain terminal: + +```powershell +cd its-g5-receiver-firmware\build +C:\Espressif\python_env\idf5.5_py3.11_env\Scripts\python.exe -m esptool --chip esp32c5 -p COM8 -b 921600 write_flash --flash_mode dio --flash_freq 80m --flash_size 16MB 0x2000 bootloader/bootloader.bin 0x8000 partition_table/partition-table.bin 0x1e000 ota_data_initial.bin 0x20000 its-g5-receiver-firmware.bin +``` + +## Pulling a capture after the fact + +Only for the Memory destination, and the buffer is small (`SNIFFER_PCAP_MEMORY_SIZE`, 4096 bytes +by default) - a smoke test, not a session. In the device console (`idf.py -p COM8 monitor`, exit +with Ctrl+T then Ctrl+X): + +``` +sniffer -P +sniffer --stop +``` + +Then, with the port free: + +```powershell +py -3.11 dump_pcap.py COM8 +``` diff --git a/capture/dump_pcap.py b/capture/dump_pcap.py new file mode 100644 index 0000000..b5d9a43 --- /dev/null +++ b/capture/dump_pcap.py @@ -0,0 +1,101 @@ +#!/usr/bin/env python3 +""" +Pulls a capture off the ITS-G5 receiver's in-memory pcap buffer over the existing USB serial +connection and saves it as a real .pcap file on this machine. + +Requires the firmware to be built with: + Example Configuration -> Select destination to store pcap file -> Memory + +Usage (typical): + 1. Close idf.py monitor (only one program can hold the COM port at a time). + 2. Run a capture on the device: `sniffer -P` ... let it run ... `sniffer --stop` + 3. python dump_pcap.py COM5 + +The device has no access to this computer's filesystem, so it can't write here directly. Instead, +`pcap --dump` streams the raw pcap bytes back over the same serial link, wrapped in plain-text +markers ("===PCAP-DUMP-START:===" ... raw bytes ... "===PCAP-DUMP-END==="). This script finds +those markers and writes just the raw bytes out as a .pcap file. + +Install dependency once: pip install pyserial +""" + +import argparse +import datetime +import re +import sys + +try: + import serial +except ImportError: + print("Missing dependency. Install it with: pip install pyserial", file=sys.stderr) + sys.exit(1) + +START_RE = re.compile(rb"===PCAP-DUMP-START:(\d+)===\n") +END_MARKER = b"\n===PCAP-DUMP-END===\n" + + +def main(): + parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + parser.add_argument("port", help="Serial port the device is on, e.g. COM5") + parser.add_argument("-b", "--baud", type=int, default=115200, help="Baud rate (default: 115200)") + parser.add_argument("-o", "--outdir", default="recordings", help="Output directory (default: ./recordings)") + parser.add_argument("-t", "--timeout", type=float, default=15.0, help="Seconds to wait for the dump to start") + args = parser.parse_args() + + import os + os.makedirs(args.outdir, exist_ok=True) + + print(f"Opening {args.port} @ {args.baud}...") + with serial.Serial(args.port, args.baud, timeout=1) as ser: + # Nudge the console in case there's stale input, then request the dump. + ser.reset_input_buffer() + ser.write(b"\r\n") + ser.write(b"pcap -f dump --dump\r\n") + + print("Waiting for dump to start...") + buf = b"" + match = None + deadline = datetime.datetime.now() + datetime.timedelta(seconds=args.timeout) + while datetime.datetime.now() < deadline: + chunk = ser.read(256) + if chunk: + buf += chunk + match = START_RE.search(buf) + if match: + break + if not match: + print("Timed out waiting for '===PCAP-DUMP-START:...===' marker.\n" + "Check that: the firmware is built with the Memory pcap destination, a capture was\n" + "actually taken ('sniffer -P' then 'sniffer --stop'), and no other program (like\n" + "idf.py monitor) is holding the serial port open.", file=sys.stderr) + sys.exit(1) + + length = int(match.group(1)) + print(f"Dump starting, {length} bytes expected.") + + # Anything after the marker in our buffer is already part of the payload. + payload = buf[match.end():] + remaining = length - len(payload) + while remaining > 0: + chunk = ser.read(min(remaining, 4096)) + if not chunk: + print(f"Serial read timed out with {remaining} bytes still missing.", file=sys.stderr) + sys.exit(1) + payload += chunk + remaining -= len(chunk) + + # Drain (and sanity-check) the trailing end marker, but don't fail hard if it's not exact. + tail = ser.read(len(END_MARKER)) + if tail != END_MARKER: + print("Warning: end marker didn't match exactly - payload may still be fine.", file=sys.stderr) + + timestamp = datetime.datetime.now().strftime("%Y%m%d_%H%M%S") + outpath = os.path.join(args.outdir, f"capture_{timestamp}.pcap") + with open(outpath, "wb") as f: + f.write(payload) + + print(f"Saved {len(payload)} bytes to {outpath}") + + +if __name__ == "__main__": + main() diff --git a/capture/live_capture.py b/capture/live_capture.py new file mode 100644 index 0000000..8604d7a --- /dev/null +++ b/capture/live_capture.py @@ -0,0 +1,226 @@ +#!/usr/bin/env python3 +""" +Continuously listens on the ITS-G5 receiver's serial console and writes every captured packet into a +live-growing .pcap file, with no console commands needed on the device side. + +The firmware streams every packet it captures out over the same serial connection the console runs on, +automatically, as soon as the sniffer is running (which happens on boot by default). Each packet is framed +with plain-text markers so this script can pull the binary pcap bytes out of the stream even though +regular log lines are interleaved with it: + + ===PCAP-LIVE-HEADER:===\\n<24 raw bytes>\\n (sent once, the pcap global header) + ===PCAP-LIVE-PKT:===\\n\\n (sent once per captured packet) + +Usage: + python live_capture.py COM5 + +Runs until you press Ctrl+C. Writes to recordings/capture_.pcap, flushing after every packet +so you can open the file in Wireshark while it's still being written (use "File > Open" again, or +Wireshark's own "Follow" won't auto-refresh but re-opening will show the latest packets). + +Install dependency once: pip install pyserial +""" + +import argparse +import datetime +import os +import re +import sys +import time + +try: + import serial +except ImportError: + print("Missing dependency. Install it with: pip install pyserial", file=sys.stderr) + sys.exit(1) + +# \r? because the ESP console emits CRLF: on Windows the markers arrive as +# "===PCAP-LIVE-PKT:310===\r\n", which never matched a bare \n and left the capture silently +# empty while the device was streaming perfectly well. +HEADER_RE = re.compile(rb"===PCAP-LIVE-HEADER:(\d+)===\r?\n") +PKT_RE = re.compile(rb"===PCAP-LIVE-PKT:(\d+)===\r?\n") + +LINKTYPE_ETHERNET = 1 +LINKTYPE_IEEE802_11_RADIOTAP = 127 + + +def mac_str(b): + return ":".join(f"{x:02x}" for x in b) + + +def build_default_pcap_header(link_type): + """Synthesizes the same 24-byte global pcap header the firmware would have sent, for when we + connect after the device's one-time header already went out (see the race note in main()).""" + header = bytearray(24) + header[0:4] = bytes([0xD4, 0xC3, 0xB2, 0xA1]) # magic (LE bytes of 0xA1B2C3D4) + header[4:6] = (2).to_bytes(2, "little") # major version + header[6:8] = (4).to_bytes(2, "little") # minor version + header[16:20] = (0x40000).to_bytes(4, "little") # snaplen + header[20:24] = link_type.to_bytes(4, "little") + return bytes(header) + + +def summarize_packet(link_type, record_bytes, index): + """Best-effort human-readable one-line summary of a captured packet, for live feedback. + record_bytes is the raw 16-byte pcap record header followed by the captured frame.""" + seconds = int.from_bytes(record_bytes[0:4], "little") + microseconds = int.from_bytes(record_bytes[4:8], "little") + cap_len = int.from_bytes(record_bytes[8:12], "little") + frame = record_bytes[16:] + ts = f"{seconds}.{microseconds:06d}" + + if link_type == LINKTYPE_IEEE802_11_RADIOTAP and len(frame) >= 24: + radiotap_len = int.from_bytes(frame[2:4], "little") + rssi = frame[8] - 256 if frame[8] >= 128 else frame[8] + station_id = int.from_bytes(frame[16:24], "little") + mac_frame = frame[radiotap_len:] + if len(mac_frame) >= 16: + dst = mac_str(mac_frame[4:10]) + src = mac_str(mac_frame[10:16]) + else: + dst = src = "?" + station = f"{station_id:012x}" if station_id else "unknown" + return (f"#{index:<5} [{ts}] len={cap_len:<5} rssi={rssi:>4}dBm " + f"station={station} {src} -> {dst}") + + if link_type == LINKTYPE_ETHERNET and len(frame) >= 14: + dst = mac_str(frame[0:6]) + src = mac_str(frame[6:12]) + ethertype = int.from_bytes(frame[12:14], "big") + return f"#{index:<5} [{ts}] len={cap_len:<5} eth {src} -> {dst} type=0x{ethertype:04x}" + + return f"#{index:<5} [{ts}] len={cap_len:<5} (unrecognized frame format)" + + +def undo_crlf(chunk, state): + """Undo the CR the device console inserts before every LF. + + ESP-IDF's newlib console converts LF to CRLF on its way out, and that happens to every 0x0A + byte of the binary pcap stream too, not only to log text. Each inserted CR shifts everything + after it, so pcap record headers and captured frames alike come out corrupt. This is the + "byte inserted mid-frame" seen in older recordings; with DENM traffic on air it wrecks most + of a capture (measured 2026-09-14: a 787 KB file parsed cleanly for only 82 records). + + Dropping one CR immediately before each LF undoes it exactly, provided it is done on the raw + stream before any framing and a trailing CR is carried across read boundaries. CR and LF are + written as byte values here so the transformation cannot be confused with an escape. + """ + CR, LF = bytes([13]), bytes([10]) + if state["pending_cr"]: + chunk = CR + chunk + state["pending_cr"] = False + if chunk.endswith(CR): + chunk = chunk[:-1] + state["pending_cr"] = True + return chunk.replace(CR + LF, LF) + + +def main(): + parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + parser.add_argument("port", help="Serial port the device is on, e.g. COM5") + parser.add_argument("-b", "--baud", type=int, default=115200, help="Baud rate (default: 115200)") + parser.add_argument("-o", "--outdir", default="recordings", help="Output directory (default: ./recordings)") + args = parser.parse_args() + + os.makedirs(args.outdir, exist_ok=True) + timestamp = datetime.datetime.now().strftime("%Y%m%d_%H%M%S") + outpath = os.path.join(args.outdir, f"capture_{timestamp}.pcap") + + print(f"Opening {args.port} @ {args.baud}...") + print(f"Writing live capture to {outpath}") + print("Press Ctrl+C to stop.") + + header_written = False + link_type = None + packet_count = 0 + buf = b"" + total_bytes = 0 + last_status = time.monotonic() + printed_raw_preview = False + + crlf_state = {"pending_cr": False} + + with serial.Serial(args.port, args.baud, timeout=1) as ser, open(outpath, "wb") as outfile: + def read_bytes(n): + return undo_crlf(ser.read(n), crlf_state) + + try: + while True: + chunk = read_bytes(256) + if chunk: + buf += chunk + total_bytes += len(chunk) + + now = time.monotonic() + if now - last_status >= 2: + last_status = now + print(f"[diagnostic] {total_bytes} raw bytes received so far, " + f"{packet_count} packets recognized, header_written={header_written}") + if total_bytes > 0 and not printed_raw_preview and not header_written and not PKT_RE.search(buf): + # We're getting bytes but none of them look like our markers - show a preview + # so we can tell whether this is plain log text (markers just haven't shown up + # yet), garbage (baud/port mismatch), or something else entirely. + preview = buf[:200] + print(f"[diagnostic] no markers matched yet - raw preview: {preview!r}") + printed_raw_preview = True + elif total_bytes == 0: + print("[diagnostic] zero bytes received from the port at all - this points at " + "the wrong COM port, another program holding the port, or a port that " + "isn't actually wired to the console/sniffer output.") + + # The device only sends the global header once, right when the sniffer first starts + # (typically within a second or two of boot). If this script connects even slightly + # late - very likely right after a fresh flash, since esptool itself resets the board - + # that header is already gone before we ever see it. Rather than blocking forever + # waiting for a header that's never coming, look for whichever marker shows up first. + header_match = None if header_written else HEADER_RE.search(buf) + pkt_match = PKT_RE.search(buf) + + if header_match and (not pkt_match or header_match.start() < pkt_match.start()): + length = int(header_match.group(1)) + buf = buf[header_match.end():] + while len(buf) < length: + buf += read_bytes(length - len(buf)) + header_bytes = buf[:length] + outfile.write(header_bytes) + outfile.flush() + buf = buf[length:] + header_written = True + if length >= 24: + link_type = int.from_bytes(header_bytes[20:24], "little") + print(f"Got pcap global header (link type {link_type}) - device is streaming.\n") + continue + + if not header_written and pkt_match: + link_type = LINKTYPE_IEEE802_11_RADIOTAP + outfile.write(build_default_pcap_header(link_type)) + outfile.flush() + header_written = True + print("Note: missed the device's one-time pcap header (it was likely sent before " + "this script connected, e.g. right after a flash/reset) - assuming WLAN " + "radiotap capture and writing a default header instead.\n") + # fall through and process pkt_match below, don't discard this packet + + if not pkt_match: + # Keep the buffer from growing unbounded while waiting for a marker, but don't + # discard anything - a marker could be split across reads. + if len(buf) > 65536: + buf = buf[-4096:] + continue + + length = int(pkt_match.group(1)) + buf = buf[pkt_match.end():] + while len(buf) < length: + buf += read_bytes(length - len(buf)) + record_bytes = buf[:length] + outfile.write(record_bytes) + outfile.flush() + buf = buf[length:] + packet_count += 1 + print(summarize_packet(link_type, record_bytes, packet_count)) + except KeyboardInterrupt: + print(f"\nStopped. {packet_count} packets saved to {outpath}") + + +if __name__ == "__main__": + main() \ No newline at end of file diff --git a/obu-cam-transmistter/NOTES.md b/obu-cam-transmistter/NOTES.md index 8ed701b..ed18867 100644 --- a/obu-cam-transmistter/NOTES.md +++ b/obu-cam-transmistter/NOTES.md @@ -88,7 +88,7 @@ source address. To see what it is sending, capture on the sniffer board and decode: ```powershell -cd its-g5-receiver-firmware +cd capture py -3.11 live_capture.py COM8 py -3.11 ..\obu-firmware\test\pcap_gn_tally.py recordings\capture_.pcap ``` diff --git a/obu-firmware/test/host/Makefile b/obu-firmware/test/host/Makefile index a9c6e91..7330e67 100644 --- a/obu-firmware/test/host/Makefile +++ b/obu-firmware/test/host/Makefile @@ -14,7 +14,9 @@ PYTHON_ASN1 = py -3.11 FW = ../../main BUILD = build EXE = $(if $(filter Windows_NT,$(OS)),.exe,) -RECORDINGS = $(wildcard ../../../its-g5-receiver-firmware/recordings/*.pcap) +# Captures live in capture/recordings/ since 2026-09-14; older ones are still in the +# receiver checkout beside this repo. +RECORDINGS = $(wildcard ../../../capture/recordings/*.pcap ../../../its-g5-receiver-firmware/recordings/*.pcap) FUZZ_ITER = 2000000 FUZZ_SEED = 1