This repository has been archived on 2026-11-16. You can view files and clone it. You cannot open issues or pull requests or push a commit.
Files
orderbook-collector/docs/roadmap.md
T
pascal adb7fee8d8 feat(polymarket): hash-verified order books for a subset of symbols
Collects the order books of Polymarket's 20 busiest markets (40 symbols)
into the template's SBE format: a Delta per level change, plus Snapshots on
subscribe, every minute, and whenever a fresh book corrects ours. Every
change is checked against Polymarket's own book hash; a failed check
resyncs the symbol on the open socket, and the log counts losses apart from
the few ms our own resync hides. Runs until SIGINT, which closes the socket
and flushes the output; reconnects when the server drops it. A `reconcile`
binary checks an output file against itself.
2026-11-04 17:10:17 +01:00

6.1 KiB
Raw Blame History

Roadmap

Goal: the order-book collector defined in project.md — a correct, complete, time-accurate feed for prediction-market order books, covering all four challenge extensions across Polymarket and Kalshi.

Ordering principle: prove the central bet (the pipeline) and one exchange first, so base + Polymarket is already a complete, submittable deliverable. Everything after is additive; nothing risky sits at the end. The generic abstraction is extracted once a second exchange exists, not designed up front.

Milestones

  1. Skeleton + connectivity — done (2026-11-04) Project scaffolding inside the template: module layout (replace main.rs), scripts/ci.sh (fmt + clippy -D warnings + tests), project CLAUDE.md, the websocket library (tungstenite + rustls). Runtime settled: thread-only V1 (async deferred to M4; see adr/0001-concurrency-model.md). Output destination is a non-issue (an output path; /dev/stdout for stdout). Connect to Polymarket and log raw messages. Ends: CI green; raw Polymarket messages flowing.

  2. Single-source book pipeline (Polymarket, subset of symbols) — done (2026-11-04) The central bet in its first real form: parse → per-symbol book (fold absolute deltas) → loss-detect → resync → SBE output (deltas + snapshot every minute, on subscribe, after a resync) → receive-time stamping → clean SIGINT, with the PING keepalive every 10 s. Loss detection: Polymarket's book hash, reproduced after every change. One socket, a fixed subset of symbols. Deterministic replay tests + snapshot↔delta reconciliation. Sprint: ../roadmap/02. book pipeline.md. Ends: correct SBE output for a symbol subset; tests green.

  3. Full coverage: discovery + sharding (base challenge complete) — not started Discovery with diffing (subscribe new, unsubscribe gone) — REST polling, or Polymarket's new_market / market_resolved events — using subscribe / unsubscribe on open connections; load-balance all live regular-market symbols across N sockets. Opening questions (from M2): what "live" means, and so how many symbols: Gamma lists at least 100,000 markets active and not closed (2026-11-04, count capped), at least 13,000 of them traded in the last 24 h, against the 40 symbols M2 runs. At that size the minute Snapshots alone (~2 KB each) would be ~0.5 TB a day for 200,000 symbols, mostly unchanged books; and memory (~10 KB a book), sockets per connection, resyncs per minute and the book thread's throughput (M2: ~30,000 frames/s on one thread, 43 frames/s of load) all need measuring. Gamma pages past 10,000 markets only through /markets/keyset with after_cursor. Ends: every live Polymarket symbol tracked — base challenge satisfied for one exchange.

  4. Extension 1 — redundant websockets — not started Multiple connections per symbol; choose a trusted source and switch on loss/quiet/delay; emit a fresh snapshot on every source switch. Async re-evaluation checkpoint: before building this, reassess whether the socket-management complexity (reconnect/timeout/select, redundancy) warrants swapping the thread-only runtime for tokio (see ADR 0001). Also: surviving a hard crash without losing the writer's in-memory chunk, and closing the resync gap (the few ms of a symbol's changes the server doesn't send between our unsubscribe and subscribe), both accepted as ceilings in M2. Ends: redundancy demonstrably reduces data loss; a Snapshot on every source switch.

  5. Extension 2 — subscription monitoring — not started Per-symbol activity monitoring (data points per second/minute); unsubscribe + resubscribe symbols that go too quiet; periodic full-socket recycle (disconnect → reconnect → resubscribe) to shed cumulative exchange-side bugs. Ends: monitoring + recycling running.

  6. Extension 3 — generic core + Kalshi adapter — not started Extract the exchange-agnostic core (the NormalizedEvent seam) from the working Polymarket code, then add a Kalshi adapter so both exchanges run through the same core. The abstraction is extracted from two concretes here, not before. Requires a Kalshi API key. Ends: both Polymarket and Kalshi work through one generic core.

  7. Extension 4 — latency experiments + writeup — not started Experiment with faster connections (first-after-maintenance, connection racing, source-port tricks) on both exchanges; document findings in a markdown file, with the local clock's offset measured alongside (the dev machine ran ~30 ms slow; see notes.md). Ends: experiments documented.

  8. Polish + submission — not started Final review, README for the solution (the template's README still says the helpers are "demonstrated in main.rs", whose demo M1 replaced), packaging/handoff for the challenge. Ends: ready to submit.

Why this order

  • Milestones 1–3 prove the pipeline (the central bet) and finish the base challenge on one exchange — the riskiest and most valuable work, up front.
  • Extensions 1 and 2 harden that single exchange before we generalize, so the generic core (6) is extracted from code that already does the hard things.
  • The generic abstraction waits for the second exchange (Kalshi), so it's extracted from two real implementations rather than guessed from one.
  • Latency (4) is experimentation, least coupled to correctness, so it's last before polish.

Possible features (beyond v1)

  • More exchanges beyond Polymarket and Kalshi.
  • MVE / multi-outcome market support.
  • Non-websocket exchange support.
  • Persistence / replay tooling beyond the raw SBE output.
  • Observability: metrics, structured logging, dashboards.
  • Config-driven symbol filtering and per-exchange tuning.
  • Containerized packaging.
  • Counting a reconnect's differing books as losses rather than resync gaps (a reconnect marker through the frames channel; M2 labels them gaps).
  • A reconcile that reads the collector's log, to tell resync gaps from losses in its own report.