An orchestrator keeps the tracked set equal to Gamma's live markets (a scan at
startup and every hour, new and closed markets every minute, reaching back 15
minutes for Gamma's indexing delay) and spreads them over lanes, each a
connection and a book thread, all writing through the one writer. Lanes report
each market's load in bytes per second; the orchestrator puts new markets on
the least-loaded lane and moves the busiest markets off lanes over a 300 KB/s
budget, busiest lane first. Differing books our own subscribe explains are
subscription gaps, not losses, and every first book after a move is logged,
so reconcile's mismatches are all accounted for in the log. A 75-minute live
run tracked ~195k markets (~390k symbols) in 89 lanes.
Collects the order books of Polymarket's 20 busiest markets (40 symbols)
into the template's SBE format: a Delta per level change, plus Snapshots on
subscribe, every minute, and whenever a fresh book corrects ours. Every
change is checked against Polymarket's own book hash; a failed check
resyncs the symbol on the open socket, and the log counts losses apart from
the few ms our own resync hides. Runs until SIGINT, which closes the socket
and flushes the output; reconnects when the server drops it. A `reconcile`
binary checks an output file against itself.
Replaces the template's demo main.rs with a spike that connects to
Polymarket's public market channel, subscribes to one market, and logs every
frame with its receive time. Adds scripts/ci.sh (fmt, clippy, tests), the
project CLAUDE.md, ADR 0001 (OS threads, shared-nothing), and the market
channel's message shapes in docs/notes.md: no sequence number, a book hash on
every change, mirrored Yes/No books.
Order-book collector for prediction-market data (Polymarket, then Kalshi). Central bet is the concurrency pipeline; correctness/completeness first, speed later. Roadmap covers base challenge plus all four extensions.