# Roadmap Goal: the order-book collector defined in [project.md](project.md) — a correct, complete, time-accurate feed for prediction-market order books, covering all four challenge extensions across Polymarket and Kalshi. Ordering principle: **prove the central bet (the pipeline) and one exchange first, so base + Polymarket is already a complete, submittable deliverable.** Everything after is additive; nothing risky sits at the end. The generic abstraction is *extracted* once a second exchange exists, not designed up front. ## Milestones 1. **Skeleton + connectivity** — *done (2026-11-04)* Project scaffolding inside the template: module layout (replace `main.rs`), `scripts/ci.sh` (fmt + clippy `-D warnings` + tests), project CLAUDE.md, the websocket library (`tungstenite` + `rustls`). Runtime settled: thread-only V1 (async deferred to M4; see [adr/0001-concurrency-model.md](adr/0001-concurrency-model.md)). Output destination is a non-issue (an output path; `/dev/stdout` for stdout). Connect to Polymarket and log raw messages. *Ends:* CI green; raw Polymarket messages flowing. 2. **Single-source book pipeline (Polymarket, subset of symbols)** — *done (2026-11-04)* The central bet in its first real form: parse → per-symbol book (fold absolute deltas) → loss-detect → resync → SBE output (deltas + snapshot every minute, on subscribe, after a resync) → receive-time stamping → clean SIGINT, with the `PING` keepalive every 10 s. Loss detection: Polymarket's book hash, reproduced after every change. One socket, a fixed subset of symbols. Deterministic replay tests + snapshot↔delta reconciliation. Sprint: [../roadmap/02. book pipeline.md](../roadmap/02.%20book%20pipeline.md). *Ends:* correct SBE output for a symbol subset; tests green. 3. **Full coverage: discovery + lanes (base challenge complete)** — *done (2026-11-11)* Every live symbol (~476k tokens, 2026-11-04) spread over lanes, each a connection and a book thread of its own; an orchestrator keeps the tracked set equal to Gamma's live set (a scan at startup and every hour, new and closed markets every minute) through subscribe / unsubscribe on open connections. Sprint: [../roadmap/03. full coverage.md](../roadmap/03.%20full%20coverage.md). *Ends:* every live Polymarket symbol tracked — base challenge satisfied for one exchange. 4. **Extension 1 — redundant websockets** — *not started* Multiple connections per symbol (several connections in a lane, its book thread choosing among them, the orchestrator coordinating); choose a trusted source and switch on loss/quiet/delay; emit a fresh snapshot on every source switch. Recovery from errors, with the lane budget scaling itself down when connections still get 1013 closes (no rate is safe: [notes.md](notes.md), so a budget set too high must correct itself). **Async re-evaluation checkpoint:** before building this, reassess whether the socket-management complexity (reconnect/timeout/select, redundancy) warrants swapping the thread-only runtime for tokio (see ADR 0001). The lane that holds markets busy together (the 5- and 15-minute crypto markets: closed 16 times in 75 minutes, in bursts a minute's load average doesn't show) is the case to beat. Also: surviving a hard crash without losing the writer's in-memory chunk, a move's Remove reaching the old lane before the new lane's first Snapshot, the memory growth seen in sprint 03 (~90 MB every 10 minutes over 75 minutes), and closing the subscription gap (the few ms of a symbol's changes the server doesn't send between our unsubscribe and subscribe); the crash chunk and the gap were accepted as ceilings in M2. *Ends:* redundancy demonstrably reduces data loss; a Snapshot on every source switch. 5. **Extension 2 — subscription monitoring** — *not started* Per-symbol activity monitoring (data points per second/minute); unsubscribe + resubscribe symbols that go too quiet; periodic full-socket recycle (disconnect → reconnect → resubscribe) to shed cumulative exchange-side bugs. *Ends:* monitoring + recycling running. 6. **Extension 3 — generic core + Kalshi adapter** — *not started* Extract the exchange-agnostic core (the `NormalizedEvent` seam) from the working Polymarket code, then add a Kalshi adapter so both exchanges run through the same core. The abstraction is extracted from two concretes here, not before. Requires a Kalshi API key. *Ends:* both Polymarket and Kalshi work through one generic core. 7. **Extension 4 — latency experiments + writeup** — *not started* Experiment with faster connections (first-after-maintenance, connection racing, source-port tricks) on both exchanges; document findings in a markdown file, with the local clock's offset measured alongside (the dev machine ran ~30 ms slow; see notes.md). *Ends:* experiments documented. 8. **Polish + submission** — *not started* Final review, README for the solution (the template's README still says the helpers are "demonstrated in `main.rs`", whose demo M1 replaced), packaging/handoff for the challenge. *Ends:* ready to submit. ## Why this order - Milestones 1–3 prove the pipeline (the central bet) and finish the base challenge on one exchange — the riskiest and most valuable work, up front. - Extensions 1 and 2 harden that single exchange before we generalize, so the generic core (6) is extracted from code that already does the hard things. - The generic abstraction waits for the second exchange (Kalshi), so it's extracted from two real implementations rather than guessed from one. - Latency (4) is experimentation, least coupled to correctness, so it's last before polish. ## Possible features (beyond v1) - More exchanges beyond Polymarket and Kalshi. - MVE / multi-outcome market support. - Non-websocket exchange support. - Persistence / replay tooling beyond the raw SBE output. - Observability: metrics, structured logging, dashboards. - Config-driven symbol filtering and per-exchange tuning. - Containerized packaging. - Counting a reconnect's differing books as losses rather than subscription gaps when the symbol was resyncing or not yet verified since its opening book (a reconnect marker through the frames channel; they are labelled gaps). - A `reconcile` that reads the collector's log, to tell subscription gaps from losses in its own report. - Polymarket's `new_market` websocket event as a faster discovery signal (it is broadcast, but lacks `orderMinSize` / `negRisk`, so Gamma is still needed; `market_resolved` was never seen). - Closing lanes that drain. - Fewer Snapshots for unchanged books (the brief asks for every symbol every minute: ~130 MB a minute at full coverage). - Re-reading a market's `orderMinSize` / `negRisk` if they can change.