This repository has been archived on 2026-11-16. You can view files and clone it. You cannot open issues or pull requests or push a commit.
Files
orderbook-collector/docs/roadmap.md
T
pascal 0e687e7eba feat(polymarket): skeleton and market-channel connectivity spike
Replaces the template's demo main.rs with a spike that connects to
Polymarket's public market channel, subscribes to one market, and logs every
frame with its receive time. Adds scripts/ci.sh (fmt, clippy, tests), the
project CLAUDE.md, ADR 0001 (OS threads, shared-nothing), and the market
channel's message shapes in docs/notes.md: no sequence number, a book hash on
every change, mirrored Yes/No books.
2026-11-04 14:47:14 +01:00

5.0 KiB
Raw Blame History

Roadmap

Goal: the order-book collector defined in project.md — a correct, complete, time-accurate feed for prediction-market order books, covering all four challenge extensions across Polymarket and Kalshi.

Ordering principle: prove the central bet (the pipeline) and one exchange first, so base + Polymarket is already a complete, submittable deliverable. Everything after is additive; nothing risky sits at the end. The generic abstraction is extracted once a second exchange exists, not designed up front.

Milestones

  1. Skeleton + connectivity — done (2026-11-04) Project scaffolding inside the template: module layout (replace main.rs), scripts/ci.sh (fmt + clippy -D warnings + tests), project CLAUDE.md, the websocket library (tungstenite + rustls). Runtime settled: thread-only V1 (async deferred to M4; see adr/0001-concurrency-model.md). Output destination is a non-issue (an output path; /dev/stdout for stdout). Connect to Polymarket and log raw messages. Ends: CI green; raw Polymarket messages flowing.

  2. Single-source book pipeline (Polymarket, subset of symbols) — not started The central bet in its first real form: normalize events → per-symbol book (fold absolute deltas) → loss-detect → re-snapshot → SBE output (deltas + snapshot every minute, on subscribe, on re-snapshot) → receive-time stamping → clean SIGINT, with the PING keepalive every 10 s. Polymarket has no sequence number, so loss detection (its book hash, or best bid/ask) is the opening question. One/few sockets, a fixed subset of symbols. Deterministic replay tests + snapshot↔delta reconciliation. Ends: correct SBE output for a symbol subset; tests green.

  3. Full coverage: discovery + sharding (base challenge complete) — not started Discovery with diffing (subscribe new, unsubscribe gone) — REST polling, or Polymarket's new_market / market_resolved events — using subscribe / unsubscribe on open connections; load-balance all live regular-market symbols across N sockets. Ends: every live Polymarket symbol tracked — base challenge satisfied for one exchange.

  4. Extension 1 — redundant websockets — not started Multiple connections per symbol; choose a trusted source and switch on loss/quiet/delay; emit a fresh snapshot on every source switch. Async re-evaluation checkpoint: before building this, reassess whether the socket-management complexity (reconnect/timeout/select, redundancy) warrants swapping the thread-only runtime for tokio (see ADR 0001). Ends: redundancy demonstrably reduces data loss; source-switch re-snapshots.

  5. Extension 2 — subscription monitoring — not started Per-symbol activity monitoring (data points per second/minute); unsubscribe + resubscribe symbols that go too quiet; periodic full-socket recycle (disconnect → reconnect → resubscribe) to shed cumulative exchange-side bugs. Ends: monitoring + recycling running.

  6. Extension 3 — generic core + Kalshi adapter — not started Extract the exchange-agnostic core (the NormalizedEvent seam) from the working Polymarket code, then add a Kalshi adapter so both exchanges run through the same core. The abstraction is extracted from two concretes here, not before. Requires a Kalshi API key. Ends: both Polymarket and Kalshi work through one generic core.

  7. Extension 4 — latency experiments + writeup — not started Experiment with faster connections (first-after-maintenance, connection racing, source-port tricks) on both exchanges; document findings in a markdown file, with the local clock's offset measured alongside (the dev machine ran ~30 ms slow; see notes.md). Ends: experiments documented.

  8. Polish + submission — not started Final review, README for the solution (the template's README still says the helpers are "demonstrated in main.rs", whose demo M1 replaced), packaging/handoff for the challenge. Ends: ready to submit.

Why this order

  • Milestones 1–3 prove the pipeline (the central bet) and finish the base challenge on one exchange — the riskiest and most valuable work, up front.
  • Extensions 1 and 2 harden that single exchange before we generalize, so the generic core (6) is extracted from code that already does the hard things.
  • The generic abstraction waits for the second exchange (Kalshi), so it's extracted from two real implementations rather than guessed from one.
  • Latency (4) is experimentation, least coupled to correctness, so it's last before polish.

Possible features (beyond v1)

  • More exchanges beyond Polymarket and Kalshi.
  • MVE / multi-outcome market support.
  • Non-websocket exchange support.
  • Persistence / replay tooling beyond the raw SBE output.
  • Observability: metrics, structured logging, dashboards.
  • Config-driven symbol filtering and per-exchange tuning.
  • Containerized packaging.