Order-book collector for prediction-market data (Polymarket, then Kalshi). Central bet is the concurrency pipeline; correctness/completeness first, speed later. Roadmap covers base challenge plus all four extensions.
4.1 KiB
Roadmap
Goal: the order-book collector defined in project.md — a correct, complete, time-accurate feed for prediction-market order books, covering all four challenge extensions across Polymarket and Kalshi.
Ordering principle: prove the central bet (the pipeline) and one exchange first, so base + Polymarket is already a complete, submittable deliverable. Everything after is additive; nothing risky sits at the end. The generic abstraction is extracted once a second exchange exists, not designed up front.
Milestones
-
Skeleton + connectivity — not started Project scaffolding inside the template: module layout (replace
main.rs),scripts/ci.sh(fmt + clippy-D warnings+ tests + audit), project CLAUDE.md, chosen websocket/HTTP libraries. Settle the pivotal async-vs-threads call. Resolve the output destination (stdout vs file). Connect to Polymarket and log raw messages. Ends: CI green; raw Polymarket messages flowing. -
Single-source book pipeline (Polymarket, subset of symbols) — not started The central bet in its first real form: normalize events → per-symbol book (fold absolute deltas) → gap-detect → re-snapshot → SBE output (deltas + snapshot every minute, on subscribe, on re-snapshot) → receive-time stamping → clean SIGINT. One/few sockets, a fixed subset of symbols. Deterministic replay tests + snapshot↔delta reconciliation. Ends: correct SBE output for a symbol subset; tests green.
-
Full coverage: discovery + sharding (base challenge complete) — not started REST discovery loop with diffing (subscribe new, unsubscribe gone); load-balance all live regular-market symbols across N sockets. Ends: every live Polymarket symbol tracked — base challenge satisfied for one exchange.
-
Extension 1 — redundant websockets — not started Multiple connections per symbol; choose a trusted source and switch on loss/quiet/delay; emit a fresh snapshot on every source switch. Ends: redundancy demonstrably reduces data loss; source-switch re-snapshots.
-
Extension 2 — subscription monitoring — not started Per-symbol activity monitoring (data points per second/minute); unsubscribe + resubscribe symbols that go too quiet; periodic full-socket recycle (disconnect → reconnect → resubscribe) to shed cumulative exchange-side bugs. Ends: monitoring + recycling running.
-
Extension 3 — generic core + Kalshi adapter — not started Extract the exchange-agnostic core (the
NormalizedEventseam) from the working Polymarket code, then add a Kalshi adapter so both exchanges run through the same core. The abstraction is extracted from two concretes here, not before. Requires a Kalshi API key. Ends: both Polymarket and Kalshi work through one generic core. -
Extension 4 — latency experiments + writeup — not started Experiment with faster connections (first-after-maintenance, connection racing, source-port tricks) on both exchanges; document findings in a markdown file. Ends: experiments documented.
-
Polish + submission — not started Final review, README for the solution, packaging/handoff for the challenge. Ends: ready to submit.
Why this order
- Milestones 1–3 prove the pipeline (the central bet) and finish the base challenge on one exchange — the riskiest and most valuable work, up front.
- Extensions 1 and 2 harden that single exchange before we generalize, so the generic core (6) is extracted from code that already does the hard things.
- The generic abstraction waits for the second exchange (Kalshi), so it's extracted from two real implementations rather than guessed from one.
- Latency (4) is experimentation, least coupled to correctness, so it's last before polish.
Possible features (beyond v1)
- More exchanges beyond Polymarket and Kalshi.
- MVE / multi-outcome market support.
- Non-websocket exchange support.
- Persistence / replay tooling beyond the raw SBE output.
- Observability: metrics, structured logging, dashboards.
- Config-driven symbol filtering and per-exchange tuning.
- Containerized packaging.