ashin-walpola/MicrOBU
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Move the capture tooling into the repo

Browse Source 
live_capture.py and dump_pcap.py were untracked files inside the third-party
its-g5-receiver-firmware checkout, so the tools every on-air measurement depends
on were versioned nowhere and would vanish with a fresh clone of that project.
They are ours rather than that project's, so they now live in capture/, with the
setup notes rewritten as its README: which port the sniffer speaks on and why,
how to capture, how to check a capture, and how to flash the sniffer board.

live_capture.py carries the fix made on 2026-09-14. The sniffer's console
converts LF to CRLF on its way out, and that applies to every 0x0a byte of the
binary pcap stream, not only to log text, so every inserted CR shifts the rest
of the stream. A 787 KB capture parsed cleanly for only 82 of about 2000
records, and DENMs turned up on nonsense BTP ports because their payloads carry
0x0a often. undo_crlf() reverses it on the raw stream before any framing, which
is exact; afterwards a capture parsed to EOF and DENMs read as port 2002.
Captures taken before that date are truncated at their first corrupted record,
so anything measured from them is worth re-checking.

capture/recordings/ is gitignored, since captures are data rather than source.
The host tests now read both that directory and the older one in the receiver
checkout, so no capture has to be moved while it is being written.

dump_pcap.py reads the same console and still needs the same treatment; that is
recorded in TODO.md.
This commit is contained in:
Ashin Walpola
2026-09-14 13:38:38 +02:00
parent 277c6b20f4
commit 73d4477f1e
7 changed files with 444 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitignore
+3
View File
@@ -46,3 +46,6 @@ sdkconfig.old
# Office lock files. Word/Excel create these beside a document while it is open
# and remove them on close, so they are transient and machine-local.
~$*
# Captures are large data files, not source (see capture/README.md).
/capture/recordings/
TODO.md
+8 -8
View File
@@ -103,14 +103,14 @@ round-trip test and `asn1tools` for UPER cover what it would have checked.
## Follow-ups found 2026-09-14
- [ ] **Commit the `live_capture.py` fix in the receiver repo.** Its console inserts a CR before
every LF, which also hits every 0x0a byte of the binary pcap stream, shifting pcap record
headers and frames. A 787 KB capture parsed cleanly for only 82 of ~2000 records, and DENMs
showed up on nonsense BTP ports. `undo_crlf()` now reverses it on the raw stream before
framing; afterwards a capture parsed to EOF and DENMs read as port 2002. This is the
"occasional byte inserted mid-frame" in the older recordings, so **every capture taken before
2026-09-14 is truncated at its first corrupted record** - re-measure anything derived from
them. The change is in the `its-g5-receiver-firmware` repo, uncommitted.
- [x] **Capture tooling moved into this repo** (`capture/`), with the CR-insertion fix. The
sniffer's console inserts a CR before every LF, which also hits every 0x0a byte of the binary
pcap stream, shifting pcap record headers and frames. A 787 KB capture parsed cleanly for
only 82 of ~2000 records, and DENMs showed up on nonsense BTP ports. `undo_crlf()` reverses
it on the raw stream before framing; afterwards a capture parsed to EOF and DENMs read as
port 2002. **Every capture taken before 2026-09-14 is truncated at its first corrupted
record** - re-measure anything derived from them.
- [ ] `capture/dump_pcap.py` reads the same console and still needs the same treatment.
- [ ] **Do not open COM3's console while the phone is attached.** Opening it toggles DTR/RTS on the
CH343 and resets the OBU, which drops the phone's USB link and needs a manual Connect.
capture/README.md
+102
View File
@@ -0,0 +1,102 @@
# Sniffer board and capture tooling
How to put an ESP32-C5 on the ITS-G5 channel as a passive sniffer, pull its captures onto this
PC, and check what is on air. The sniffer firmware itself is the third-party
`its-g5-receiver-firmware` checkout beside this repo; only the tooling and these notes are ours.
| File | What it does |
|---|---|
| `live_capture.py` | Streams the device's captures into a growing `.pcap` while it runs. The usual choice. |
| `dump_pcap.py` | Pulls one capture out of the device's in-memory buffer after the fact. |
| `../obu-firmware/test/pcap_gn_tally.py` | Tallies GeoNetworking headers per station over a `.pcap`. |
One-time: `pip install pyserial` (present in Python 3.11 on the bench PC, so `py -3.11` works).
## Which port
The sniffer firmware's console, and with it the pcap stream, goes out **UART0** - the board's
USB-bridge port (a CH343, its own COM number), not the native USB-C port. A board with only one
USB-C port cannot be used as a sniffer for this reason. On the bench this has been COM5 and, after
a re-enumeration, COM8.
## Live capture (preferred)
```powershell
cd capture
py -3.11 live_capture.py COM8
```
It writes `recordings/capture_<timestamp>.pcap` next to itself, flushing after every packet, so
the file can be read while it grows. Stop it with Ctrl+C. Use `-o <dir>` to write elsewhere;
`recordings/` is gitignored, since captures are large and are data rather than source. Captures
taken before 2026-09-14 are still in `its-g5-receiver-firmware/recordings/`; the host tests read
both directories.
### The CR insertion, and why captures used to be corrupt
ESP-IDF's newlib console converts LF to CRLF on its way out, and that applies to every `0x0a` byte
of the **binary** pcap stream, not only to log text. Each inserted CR shifts everything after it,
so pcap record headers and captured frames alike come out corrupt, and the file stops being
parseable at the first occurrence.
Measured on 2026-09-14: a 787 KB capture parsed cleanly for only 82 of about 2000 records, and
DENMs appeared on nonsense BTP ports because their payloads contain `0x0a` often. `undo_crlf()` in
`live_capture.py` reverses it on the raw stream before any framing, which is exact; afterwards a
capture parsed to EOF and DENMs read as port 2002 again.
**Captures taken before 2026-09-14 are truncated at their first corrupted record.** Anything
measured from them is worth re-checking. `dump_pcap.py` reads the same console and has not been
given the same treatment yet.
## Checking a capture
```powershell
py -3.11 ..\obu-firmware\test\pcap_gn_tally.py recordings\capture_<timestamp>.pcap
```
One row per station, packet type, BTP port and GN lifetime. For the messages themselves, decode
the payloads with `asn1tools` against the modules in `../asn1/` and re-encode them: identical bytes
mean the message was read exactly, wrong bytes mean it was not. `obu-firmware/test/check_replay.py`
does this over a whole capture.
## Flashing the sniffer firmware
From the receiver checkout, with its **pinned** ESP-IDF (not the global 5.5.4 install):
```powershell
cd its-g5-receiver-firmware
git submodule update --init --recursive
.\esp-idf\install.bat
Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass
.\esp-idf\export.ps1
idf.py set-target esp32c5
idf.py -p COM8 -b 921600 flash
```
Its `sdkconfig` for a bare board (nothing wired) needs SPI Ethernet off, and the pcap destination
set to Memory, both under `idf.py menuconfig`. Wired variants have ready-made configs in that
checkout: `sdkconfig.proto-w5500`, `sdkconfig.proto-enc28j60`, `sdkconfig.proto-spi-eppp`.
To reflash a board that already has a built image, without a toolchain terminal:
```powershell
cd its-g5-receiver-firmware\build
C:\Espressif\python_env\idf5.5_py3.11_env\Scripts\python.exe -m esptool --chip esp32c5 -p COM8 -b 921600 write_flash --flash_mode dio --flash_freq 80m --flash_size 16MB 0x2000 bootloader/bootloader.bin 0x8000 partition_table/partition-table.bin 0x1e000 ota_data_initial.bin 0x20000 its-g5-receiver-firmware.bin
```
## Pulling a capture after the fact
Only for the Memory destination, and the buffer is small (`SNIFFER_PCAP_MEMORY_SIZE`, 4096 bytes
by default) - a smoke test, not a session. In the device console (`idf.py -p COM8 monitor`, exit
with Ctrl+T then Ctrl+X):
```
sniffer -P
sniffer --stop
```
Then, with the port free:
```powershell
py -3.11 dump_pcap.py COM8
```
capture/dump_pcap.py
+101
View File
@@ -0,0 +1,101 @@
#!/usr/bin/env python3
"""
Pulls a capture off the ITS-G5 receiver's in-memory pcap buffer over the existing USB serial
connection and saves it as a real .pcap file on this machine.
Requires the firmware to be built with:
Example Configuration -> Select destination to store pcap file -> Memory
Usage (typical):
1. Close idf.py monitor (only one program can hold the COM port at a time).
2. Run a capture on the device: `sniffer -P` ... let it run ... `sniffer --stop`
3. python dump_pcap.py COM5
The device has no access to this computer's filesystem, so it can't write here directly. Instead,
`pcap --dump` streams the raw pcap bytes back over the same serial link, wrapped in plain-text
markers ("===PCAP-DUMP-START:<len>===" ... raw bytes ... "===PCAP-DUMP-END==="). This script finds
those markers and writes just the raw bytes out as a .pcap file.
Install dependency once: pip install pyserial
"""
import argparse
import datetime
import re
import sys
try:
import serial
except ImportError:
print("Missing dependency. Install it with: pip install pyserial", file=sys.stderr)
sys.exit(1)
START_RE = re.compile(rb"===PCAP-DUMP-START:(\d+)===\n")
END_MARKER = b"\n===PCAP-DUMP-END===\n"
def main():
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
parser.add_argument("port", help="Serial port the device is on, e.g. COM5")
parser.add_argument("-b", "--baud", type=int, default=115200, help="Baud rate (default: 115200)")
parser.add_argument("-o", "--outdir", default="recordings", help="Output directory (default: ./recordings)")
parser.add_argument("-t", "--timeout", type=float, default=15.0, help="Seconds to wait for the dump to start")
args = parser.parse_args()
import os
os.makedirs(args.outdir, exist_ok=True)
print(f"Opening {args.port} @ {args.baud}...")
with serial.Serial(args.port, args.baud, timeout=1) as ser:
# Nudge the console in case there's stale input, then request the dump.
ser.reset_input_buffer()
ser.write(b"\r\n")
ser.write(b"pcap -f dump --dump\r\n")
print("Waiting for dump to start...")
buf = b""
match = None
deadline = datetime.datetime.now() + datetime.timedelta(seconds=args.timeout)
while datetime.datetime.now() < deadline:
chunk = ser.read(256)
if chunk:
buf += chunk
match = START_RE.search(buf)
if match:
break
if not match:
print("Timed out waiting for '===PCAP-DUMP-START:...===' marker.\n"
"Check that: the firmware is built with the Memory pcap destination, a capture was\n"
"actually taken ('sniffer -P' then 'sniffer --stop'), and no other program (like\n"
"idf.py monitor) is holding the serial port open.", file=sys.stderr)
sys.exit(1)
length = int(match.group(1))
print(f"Dump starting, {length} bytes expected.")
# Anything after the marker in our buffer is already part of the payload.
payload = buf[match.end():]
remaining = length - len(payload)
while remaining > 0:
chunk = ser.read(min(remaining, 4096))
if not chunk:
print(f"Serial read timed out with {remaining} bytes still missing.", file=sys.stderr)
sys.exit(1)
payload += chunk
remaining -= len(chunk)
# Drain (and sanity-check) the trailing end marker, but don't fail hard if it's not exact.
tail = ser.read(len(END_MARKER))
if tail != END_MARKER:
print("Warning: end marker didn't match exactly - payload may still be fine.", file=sys.stderr)
timestamp = datetime.datetime.now().strftime("%Y%m%d_%H%M%S")
outpath = os.path.join(args.outdir, f"capture_{timestamp}.pcap")
with open(outpath, "wb") as f:
f.write(payload)
print(f"Saved {len(payload)} bytes to {outpath}")
if __name__ == "__main__":
main()
capture/live_capture.py
+226
View File
@@ -0,0 +1,226 @@
#!/usr/bin/env python3
"""
Continuously listens on the ITS-G5 receiver's serial console and writes every captured packet into a
live-growing .pcap file, with no console commands needed on the device side.
The firmware streams every packet it captures out over the same serial connection the console runs on,
automatically, as soon as the sniffer is running (which happens on boot by default). Each packet is framed
with plain-text markers so this script can pull the binary pcap bytes out of the stream even though
regular log lines are interleaved with it:
===PCAP-LIVE-HEADER:<len>===\\n<24 raw bytes>\\n (sent once, the pcap global header)
===PCAP-LIVE-PKT:<len>===\\n<raw bytes>\\n (sent once per captured packet)
Usage:
python live_capture.py COM5
Runs until you press Ctrl+C. Writes to recordings/capture_<timestamp>.pcap, flushing after every packet
so you can open the file in Wireshark while it's still being written (use "File > Open" again, or
Wireshark's own "Follow" won't auto-refresh but re-opening will show the latest packets).
Install dependency once: pip install pyserial
"""
import argparse
import datetime
import os
import re
import sys
import time
try:
import serial
except ImportError:
print("Missing dependency. Install it with: pip install pyserial", file=sys.stderr)
sys.exit(1)
# \r? because the ESP console emits CRLF: on Windows the markers arrive as
# "===PCAP-LIVE-PKT:310===\r\n", which never matched a bare \n and left the capture silently
# empty while the device was streaming perfectly well.
HEADER_RE = re.compile(rb"===PCAP-LIVE-HEADER:(\d+)===\r?\n")
PKT_RE = re.compile(rb"===PCAP-LIVE-PKT:(\d+)===\r?\n")
LINKTYPE_ETHERNET = 1
LINKTYPE_IEEE802_11_RADIOTAP = 127
def mac_str(b):
return ":".join(f"{x:02x}" for x in b)
def build_default_pcap_header(link_type):
"""Synthesizes the same 24-byte global pcap header the firmware would have sent, for when we
connect after the device's one-time header already went out (see the race note in main())."""
header = bytearray(24)
header[0:4] = bytes([0xD4, 0xC3, 0xB2, 0xA1]) # magic (LE bytes of 0xA1B2C3D4)
header[4:6] = (2).to_bytes(2, "little") # major version
header[6:8] = (4).to_bytes(2, "little") # minor version
header[16:20] = (0x40000).to_bytes(4, "little") # snaplen
header[20:24] = link_type.to_bytes(4, "little")
return bytes(header)
def summarize_packet(link_type, record_bytes, index):
"""Best-effort human-readable one-line summary of a captured packet, for live feedback.
record_bytes is the raw 16-byte pcap record header followed by the captured frame."""
seconds = int.from_bytes(record_bytes[0:4], "little")
microseconds = int.from_bytes(record_bytes[4:8], "little")
cap_len = int.from_bytes(record_bytes[8:12], "little")
frame = record_bytes[16:]
ts = f"{seconds}.{microseconds:06d}"
if link_type == LINKTYPE_IEEE802_11_RADIOTAP and len(frame) >= 24:
radiotap_len = int.from_bytes(frame[2:4], "little")
rssi = frame[8] - 256 if frame[8] >= 128 else frame[8]
station_id = int.from_bytes(frame[16:24], "little")
mac_frame = frame[radiotap_len:]
if len(mac_frame) >= 16:
dst = mac_str(mac_frame[4:10])
src = mac_str(mac_frame[10:16])
else:
dst = src = "?"
station = f"{station_id:012x}" if station_id else "unknown"
return (f"#{index:<5} [{ts}] len={cap_len:<5} rssi={rssi:>4}dBm "
f"station={station} {src} -> {dst}")
if link_type == LINKTYPE_ETHERNET and len(frame) >= 14:
dst = mac_str(frame[0:6])
src = mac_str(frame[6:12])
ethertype = int.from_bytes(frame[12:14], "big")
return f"#{index:<5} [{ts}] len={cap_len:<5} eth {src} -> {dst} type=0x{ethertype:04x}"
return f"#{index:<5} [{ts}] len={cap_len:<5} (unrecognized frame format)"
def undo_crlf(chunk, state):
"""Undo the CR the device console inserts before every LF.
ESP-IDF's newlib console converts LF to CRLF on its way out, and that happens to every 0x0A
byte of the binary pcap stream too, not only to log text. Each inserted CR shifts everything
after it, so pcap record headers and captured frames alike come out corrupt. This is the
"byte inserted mid-frame" seen in older recordings; with DENM traffic on air it wrecks most
of a capture (measured 2026-09-14: a 787 KB file parsed cleanly for only 82 records).
Dropping one CR immediately before each LF undoes it exactly, provided it is done on the raw
stream before any framing and a trailing CR is carried across read boundaries. CR and LF are
written as byte values here so the transformation cannot be confused with an escape.
"""
CR, LF = bytes([13]), bytes([10])
if state["pending_cr"]:
chunk = CR + chunk
state["pending_cr"] = False
if chunk.endswith(CR):
chunk = chunk[:-1]
state["pending_cr"] = True
return chunk.replace(CR + LF, LF)
def main():
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
parser.add_argument("port", help="Serial port the device is on, e.g. COM5")
parser.add_argument("-b", "--baud", type=int, default=115200, help="Baud rate (default: 115200)")
parser.add_argument("-o", "--outdir", default="recordings", help="Output directory (default: ./recordings)")
args = parser.parse_args()
os.makedirs(args.outdir, exist_ok=True)
timestamp = datetime.datetime.now().strftime("%Y%m%d_%H%M%S")
outpath = os.path.join(args.outdir, f"capture_{timestamp}.pcap")
print(f"Opening {args.port} @ {args.baud}...")
print(f"Writing live capture to {outpath}")
print("Press Ctrl+C to stop.")
header_written = False
link_type = None
packet_count = 0
buf = b""
total_bytes = 0
last_status = time.monotonic()
printed_raw_preview = False
crlf_state = {"pending_cr": False}
with serial.Serial(args.port, args.baud, timeout=1) as ser, open(outpath, "wb") as outfile:
def read_bytes(n):
return undo_crlf(ser.read(n), crlf_state)
try:
while True:
chunk = read_bytes(256)
if chunk:
buf += chunk
total_bytes += len(chunk)
now = time.monotonic()
if now - last_status >= 2:
last_status = now
print(f"[diagnostic] {total_bytes} raw bytes received so far, "
f"{packet_count} packets recognized, header_written={header_written}")
if total_bytes > 0 and not printed_raw_preview and not header_written and not PKT_RE.search(buf):
# We're getting bytes but none of them look like our markers - show a preview
# so we can tell whether this is plain log text (markers just haven't shown up
# yet), garbage (baud/port mismatch), or something else entirely.
preview = buf[:200]
print(f"[diagnostic] no markers matched yet - raw preview: {preview!r}")
printed_raw_preview = True
elif total_bytes == 0:
print("[diagnostic] zero bytes received from the port at all - this points at "
"the wrong COM port, another program holding the port, or a port that "
"isn't actually wired to the console/sniffer output.")
# The device only sends the global header once, right when the sniffer first starts
# (typically within a second or two of boot). If this script connects even slightly
# late - very likely right after a fresh flash, since esptool itself resets the board -
# that header is already gone before we ever see it. Rather than blocking forever
# waiting for a header that's never coming, look for whichever marker shows up first.
header_match = None if header_written else HEADER_RE.search(buf)
pkt_match = PKT_RE.search(buf)
if header_match and (not pkt_match or header_match.start() < pkt_match.start()):
length = int(header_match.group(1))
buf = buf[header_match.end():]
while len(buf) < length:
buf += read_bytes(length - len(buf))
header_bytes = buf[:length]
outfile.write(header_bytes)
outfile.flush()
buf = buf[length:]
header_written = True
if length >= 24:
link_type = int.from_bytes(header_bytes[20:24], "little")
print(f"Got pcap global header (link type {link_type}) - device is streaming.\n")
continue
if not header_written and pkt_match:
link_type = LINKTYPE_IEEE802_11_RADIOTAP
outfile.write(build_default_pcap_header(link_type))
outfile.flush()
header_written = True
print("Note: missed the device's one-time pcap header (it was likely sent before "
"this script connected, e.g. right after a flash/reset) - assuming WLAN "
"radiotap capture and writing a default header instead.\n")
# fall through and process pkt_match below, don't discard this packet
if not pkt_match:
# Keep the buffer from growing unbounded while waiting for a marker, but don't
# discard anything - a marker could be split across reads.
if len(buf) > 65536:
buf = buf[-4096:]
continue
length = int(pkt_match.group(1))
buf = buf[pkt_match.end():]
while len(buf) < length:
buf += read_bytes(length - len(buf))
record_bytes = buf[:length]
outfile.write(record_bytes)
outfile.flush()
buf = buf[length:]
packet_count += 1
print(summarize_packet(link_type, record_bytes, packet_count))
except KeyboardInterrupt:
print(f"\nStopped. {packet_count} packets saved to {outpath}")
if __name__ == "__main__":
main()
obu-cam-transmistter/NOTES.md
+1 -1
View File
@@ -88,7 +88,7 @@ source address.
To see what it is sending, capture on the sniffer board and decode:
```powershell
cd its-g5-receiver-firmware
cd capture
py -3.11 live_capture.py COM8
py -3.11 ..\obu-firmware\test\pcap_gn_tally.py recordings\capture_<timestamp>.pcap
```
obu-firmware/test/host/Makefile
+3 -1
View File
@@ -14,7 +14,9 @@ PYTHON_ASN1 = py -3.11
FW = ../../main
BUILD = build
EXE = $(if $(filter Windows_NT,$(OS)),.exe,)
RECORDINGS = $(wildcard ../../../its-g5-receiver-firmware/recordings/*.pcap)
# Captures live in capture/recordings/ since 2026-09-14; older ones are still in the
# receiver checkout beside this repo.
RECORDINGS = $(wildcard ../../../capture/recordings/*.pcap ../../../its-g5-receiver-firmware/recordings/*.pcap)
FUZZ_ITER = 2000000
FUZZ_SEED = 1