The exchange can fail without telling us, and first one wins hides a subscription gone quiet on one of a symbol's two connections. - A connection that receives nothing for 30 s, not even the keepalive's answer, reconnects. - Each book thread counts a symbol's changes per connection every minute: a subscription with none against three or more on the other is resubscribed, and moved to another connection if it stays quiet. - The orchestrator recycles one connection at a time, each about hourly, and never while another connection is down or just back. - A reconnect's opening book of a symbol unchanged for 10 s is judged like a resync's answer: one newer than ours that differs shows a symbol quiet on both connections, written as a Snapshot and logged as a loss.
7.2 KiB
Roadmap
Goal: the order-book collector defined in project.md — a correct, complete, time-accurate feed for prediction-market order books, covering all four challenge extensions across Polymarket and Kalshi.
Ordering principle: prove the central bet (the pipeline) and one exchange first, so base + Polymarket is already a complete, submittable deliverable. Everything after is additive; nothing risky sits at the end. The generic abstraction is extracted once a second exchange exists, not designed up front.
Milestones
-
Skeleton + connectivity — done (2026-11-04) Project scaffolding inside the template: module layout (replace
main.rs),scripts/ci.sh(fmt + clippy-D warnings+ tests), project CLAUDE.md, the websocket library (tungstenite+rustls). Runtime settled: thread-only V1 (async deferred to M4; see adr/0001-concurrency-model.md). Output destination is a non-issue (an output path;/dev/stdoutfor stdout). Connect to Polymarket and log raw messages. Ends: CI green; raw Polymarket messages flowing. -
Single-source book pipeline (Polymarket, subset of symbols) — done (2026-11-04) The central bet in its first real form: parse → per-symbol book (fold absolute deltas) → loss-detect → resync → SBE output (deltas + snapshot every minute, on subscribe, after a resync) → receive-time stamping → clean SIGINT, with the
PINGkeepalive every 10 s. Loss detection: Polymarket's book hash, reproduced after every change. One socket, a fixed subset of symbols. Deterministic replay tests + snapshot↔delta reconciliation. Sprint: ../roadmap/02. book pipeline.md. Ends: correct SBE output for a symbol subset; tests green. -
Full coverage: discovery + lanes (base challenge complete) — done (2026-11-11) Every live symbol (~476k tokens, 2026-11-04) spread over lanes, each a connection and a book thread of its own; an orchestrator keeps the tracked set equal to Gamma's live set (a scan at startup and every hour, new and closed markets every minute) through subscribe / unsubscribe on open connections. Sprint: ../roadmap/03. full coverage.md. Ends: every live Polymarket symbol tracked — base challenge satisfied for one exchange.
-
Extension 1 — redundant websockets — done (2026-11-11) Every symbol on two connections chosen independently, its book taking each change from whichever delivers it first (first one wins); connections separated from book threads; a connection the server keeps closing with 1013 is split, so a budget set too high corrects itself. The async checkpoint was held: threads stay (ADR 0001). Sprint: ../roadmap/04. redundant connections.md. Left as ceilings: the writer's in-memory chunk on a hard crash, and the subscription gap when a symbol opens. Ends: redundancy demonstrably reduces data loss: fewer losses and subscription gaps per million changes with two subscriptions than with one. Result: losses a quarter lower in each pair of alternating runs; losses and gaps together lower in one pair, even in the other (sprint 04's Outcome).
-
Extension 2 — subscription monitoring — done (2026-11-11) Per-symbol activity monitoring (data points per second/minute); unsubscribe + resubscribe symbols that go too quiet (with two connections, first one wins hides a subscription gone quiet on one of them: it has to be looked for); periodic full-socket recycle (disconnect → reconnect → resubscribe) to shed cumulative exchange-side bugs. Sprint: ../roadmap/05. subscription monitoring.md. Ends: monitoring + recycling running. Result: a 30-minute run recycled 107 connections with losses no higher than sprint 04's; quiet subscriptions are rare (26 resubscribed, none quiet again) and no connection fell silent (sprint 05's Outcome).
-
Extension 3 — generic core + Kalshi adapter — not started Extract the exchange-agnostic core (the
NormalizedEventseam) from the working Polymarket code, then add a Kalshi adapter so both exchanges run through the same core. The abstraction is extracted from two concretes here, not before. Requires a Kalshi API key. Ends: both Polymarket and Kalshi work through one generic core. -
Extension 4 — latency experiments + writeup — not started Experiment with faster connections (first-after-maintenance, connection racing, source-port tricks) on both exchanges; document findings in a markdown file, with the local clock's offset measured alongside (the dev machine ran ~30 ms slow; see notes.md). Ends: experiments documented.
-
Polish + submission — not started Final review, README for the solution (the template's README still says the helpers are "demonstrated in
main.rs", whose demo M1 replaced), packaging/handoff for the challenge. Ends: ready to submit.
Why this order
- Milestones 1–3 prove the pipeline (the central bet) and finish the base challenge on one exchange — the riskiest and most valuable work, up front.
- Extensions 1 and 2 harden that single exchange before we generalize, so the generic core (6) is extracted from code that already does the hard things.
- The generic abstraction waits for the second exchange (Kalshi), so it's extracted from two real implementations rather than guessed from one.
- Latency (4) is experimentation, least coupled to correctness, so it's last before polish.
Possible features (beyond v1)
- More exchanges beyond Polymarket and Kalshi.
- MVE / multi-outcome market support.
- Non-websocket exchange support.
- Persistence / replay tooling beyond the raw SBE output.
- Observability: metrics, structured logging, dashboards.
- Config-driven symbol filtering and per-exchange tuning.
- Containerized packaging.
- Polymarket's
new_marketwebsocket event as a faster discovery signal (it is broadcast, but lacksorderMinSize/negRisk, so Gamma is still needed;market_resolvedwas never seen). - Closing connections that drain.
- Verifying a symbol again from the trade event's price after a trade, with no resync (sprint 04's prototype did it for 15 of 100 broken chains).
- Telling a real loss from a resync book merely ahead of our stream, so the loss count measures data missed (most of sprint 04's "losses" follow trades, as often with one connection as with two).
- Closing the opening subscription gap with the second connection's opening book, when the first one's chain breaks before any change verified.
- An exact rescued count: a change remembered past the window of 128 until its copies are counted (sprint 04 reports an upper bound).
- Telling a change still in flight from a symbol quiet on both connections,
when a reconnect's book is newer than ours (
maybe: sprint 05 logs both as a loss; the Snapshot is right either way). - Fewer Snapshots for unchanged books (the brief asks for every symbol every minute: ~130 MB a minute at full coverage).
- Re-reading a market's
orderMinSize/negRiskif they can change.