Archived
An orchestrator keeps the tracked set equal to Gamma's live markets (a scan at startup and every hour, new and closed markets every minute, reaching back 15 minutes for Gamma's indexing delay) and spreads them over lanes, each a connection and a book thread, all writing through the one writer. Lanes report each market's load in bytes per second; the orchestrator puts new markets on the least-loaded lane and moves the busiest markets off lanes over a 300 KB/s budget, busiest lane first. Differing books our own subscribe explains are subscription gaps, not losses, and every first book after a move is logged, so reconcile's mismatches are all accounted for in the log. A 75-minute live run tracked ~195k markets (~390k symbols) in 89 lanes.
124 lines
6.8 KiB
Markdown
124 lines
6.8 KiB
Markdown
# Roadmap
|
||
|
||
Goal: the order-book collector defined in [project.md](project.md) — a
|
||
correct, complete, time-accurate feed for prediction-market order books,
|
||
covering all four challenge extensions across Polymarket and Kalshi.
|
||
|
||
Ordering principle: **prove the central bet (the pipeline) and one exchange
|
||
first, so base + Polymarket is already a complete, submittable deliverable.**
|
||
Everything after is additive; nothing risky sits at the end. The generic
|
||
abstraction is *extracted* once a second exchange exists, not designed up front.
|
||
|
||
## Milestones
|
||
|
||
1. **Skeleton + connectivity** — *done (2026-11-04)*
|
||
Project scaffolding inside the template: module layout (replace `main.rs`),
|
||
`scripts/ci.sh` (fmt + clippy `-D warnings` + tests), project CLAUDE.md, the
|
||
websocket library (`tungstenite` + `rustls`). Runtime settled: thread-only V1
|
||
(async deferred to M4; see [adr/0001-concurrency-model.md](adr/0001-concurrency-model.md)).
|
||
Output destination is a non-issue (an output path; `/dev/stdout` for stdout).
|
||
Connect to Polymarket and log raw messages.
|
||
*Ends:* CI green; raw Polymarket messages flowing.
|
||
|
||
2. **Single-source book pipeline (Polymarket, subset of symbols)** — *done (2026-11-04)*
|
||
The central bet in its first real form: parse → per-symbol book (fold
|
||
absolute deltas) → loss-detect → resync → SBE output (deltas + snapshot
|
||
every minute, on subscribe, after a resync) → receive-time stamping → clean
|
||
SIGINT, with the `PING` keepalive every 10 s. Loss detection: Polymarket's
|
||
book hash, reproduced after every change. One socket, a fixed subset of
|
||
symbols. Deterministic replay tests + snapshot↔delta reconciliation.
|
||
Sprint: [../roadmap/02. book pipeline.md](../roadmap/02.%20book%20pipeline.md).
|
||
*Ends:* correct SBE output for a symbol subset; tests green.
|
||
|
||
3. **Full coverage: discovery + lanes (base challenge complete)** — *done (2026-11-11)*
|
||
Every live symbol (~476k tokens, 2026-11-04) spread over lanes, each a
|
||
connection and a book thread of its own; an orchestrator keeps the tracked
|
||
set equal to Gamma's live set (a scan at startup and every hour, new and
|
||
closed markets every minute) through subscribe / unsubscribe on open
|
||
connections.
|
||
Sprint: [../roadmap/03. full coverage.md](../roadmap/03.%20full%20coverage.md).
|
||
*Ends:* every live Polymarket symbol tracked — base challenge satisfied for
|
||
one exchange.
|
||
|
||
4. **Extension 1 — redundant websockets** — *not started*
|
||
Multiple connections per symbol (several connections in a lane, its book
|
||
thread choosing among them, the orchestrator coordinating); choose a
|
||
trusted source and switch on loss/quiet/delay; emit a fresh snapshot on
|
||
every source switch. Recovery from errors, with the lane budget scaling
|
||
itself down when connections still get 1013 closes (no rate is safe:
|
||
[notes.md](notes.md), so a budget set too high must correct itself).
|
||
**Async re-evaluation checkpoint:** before building this, reassess whether the
|
||
socket-management complexity (reconnect/timeout/select, redundancy) warrants
|
||
swapping the thread-only runtime for tokio (see ADR 0001).
|
||
The lane that holds markets busy together (the 5- and 15-minute crypto
|
||
markets: closed 16 times in 75 minutes, in bursts a minute's load
|
||
average doesn't show) is the case to beat.
|
||
Also: surviving a hard crash without losing the writer's in-memory chunk,
|
||
a move's Remove reaching the old lane before the new lane's first
|
||
Snapshot, the memory growth seen in sprint 03 (~90 MB every 10 minutes
|
||
over 75 minutes), and closing the subscription gap (the few ms of a
|
||
symbol's changes the server doesn't send between our unsubscribe and
|
||
subscribe); the crash chunk and the gap were accepted as ceilings in M2.
|
||
*Ends:* redundancy demonstrably reduces data loss; a Snapshot on every source
|
||
switch.
|
||
|
||
5. **Extension 2 — subscription monitoring** — *not started*
|
||
Per-symbol activity monitoring (data points per second/minute); unsubscribe +
|
||
resubscribe symbols that go too quiet; periodic full-socket recycle
|
||
(disconnect → reconnect → resubscribe) to shed cumulative exchange-side bugs.
|
||
*Ends:* monitoring + recycling running.
|
||
|
||
6. **Extension 3 — generic core + Kalshi adapter** — *not started*
|
||
Extract the exchange-agnostic core (the `NormalizedEvent` seam) from the
|
||
working Polymarket code, then add a Kalshi adapter so both exchanges run
|
||
through the same core. The abstraction is extracted from two concretes here,
|
||
not before. Requires a Kalshi API key.
|
||
*Ends:* both Polymarket and Kalshi work through one generic core.
|
||
|
||
7. **Extension 4 — latency experiments + writeup** — *not started*
|
||
Experiment with faster connections (first-after-maintenance, connection
|
||
racing, source-port tricks) on both exchanges; document findings in a
|
||
markdown file, with the local clock's offset measured alongside (the dev
|
||
machine ran ~30 ms slow; see notes.md).
|
||
*Ends:* experiments documented.
|
||
|
||
8. **Polish + submission** — *not started*
|
||
Final review, README for the solution (the template's README still says the
|
||
helpers are "demonstrated in `main.rs`", whose demo M1 replaced),
|
||
packaging/handoff for the challenge.
|
||
*Ends:* ready to submit.
|
||
|
||
## Why this order
|
||
|
||
- Milestones 1–3 prove the pipeline (the central bet) and finish the base
|
||
challenge on one exchange — the riskiest and most valuable work, up front.
|
||
- Extensions 1 and 2 harden that single exchange before we generalize, so the
|
||
generic core (6) is extracted from code that already does the hard things.
|
||
- The generic abstraction waits for the second exchange (Kalshi), so it's
|
||
extracted from two real implementations rather than guessed from one.
|
||
- Latency (4) is experimentation, least coupled to correctness, so it's last
|
||
before polish.
|
||
|
||
## Possible features (beyond v1)
|
||
|
||
- More exchanges beyond Polymarket and Kalshi.
|
||
- MVE / multi-outcome market support.
|
||
- Non-websocket exchange support.
|
||
- Persistence / replay tooling beyond the raw SBE output.
|
||
- Observability: metrics, structured logging, dashboards.
|
||
- Config-driven symbol filtering and per-exchange tuning.
|
||
- Containerized packaging.
|
||
- Counting a reconnect's differing books as losses rather than subscription
|
||
gaps when the symbol was resyncing or not yet verified since its opening
|
||
book (a reconnect marker through the frames channel; they are labelled
|
||
gaps).
|
||
- A `reconcile` that reads the collector's log, to tell subscription gaps from
|
||
losses in its own report.
|
||
- Polymarket's `new_market` websocket event as a faster discovery signal (it
|
||
is broadcast, but lacks `orderMinSize` / `negRisk`, so Gamma is still
|
||
needed; `market_resolved` was never seen).
|
||
- Closing lanes that drain.
|
||
- Fewer Snapshots for unchanged books (the brief asks for every symbol every
|
||
minute: ~130 MB a minute at full coverage).
|
||
- Re-reading a market's `orderMinSize` / `negRisk` if they can change.
|