This repository has been archived on 2026-11-16. You can view files and clone it. You cannot open issues or pull requests or push a commit.
Files
orderbook-collector/docs/roadmap.md
T
pascal 1a5bbcd602 feat!: a generic core, and Kalshi through it
The orchestrator, the connection loop and the book thread's shell move to
src/core/, generic over an Exchange and its loss rule (ADR 0004).
Polymarket runs through it unchanged. Kalshi joins: a signed handshake
(KALSHI_KEY_ID_FILE, KALSHI_PRIVATE_KEY_FILE), a reconnect on a skipped
sequence number, first one wins by clock and rank, and its market listing
with markets tracked from three hours before they open.

Breaking: the exchange is now the first argument,
`engineering-challenge <polymarket|kalshi|kalshi-demo> [OUTPUT]`.
2026-11-11 20:54:17 +00:00

141 lines
8.1 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Roadmap
Goal: the order-book collector defined in [project.md](project.md) — a
correct, complete, time-accurate feed for prediction-market order books,
covering all four challenge extensions across Polymarket and Kalshi.
Ordering principle: **prove the central bet (the pipeline) and one exchange
first, so base + Polymarket is already a complete, submittable deliverable.**
Everything after is additive; nothing risky sits at the end. The generic
abstraction is *extracted* once a second exchange exists, not designed up front.
## Milestones
1. **Skeleton + connectivity** — *done (2026-11-04)*
Project scaffolding inside the template: module layout (replace `main.rs`),
`scripts/ci.sh` (fmt + clippy `-D warnings` + tests), project CLAUDE.md, the
websocket library (`tungstenite` + `rustls`). Runtime settled: thread-only V1
(async deferred to M4; see [adr/0001-concurrency-model.md](adr/0001-concurrency-model.md)).
Output destination is a non-issue (an output path; `/dev/stdout` for stdout).
Connect to Polymarket and log raw messages.
*Ends:* CI green; raw Polymarket messages flowing.
2. **Single-source book pipeline (Polymarket, subset of symbols)** — *done (2026-11-04)*
The central bet in its first real form: parse → per-symbol book (fold
absolute deltas) → loss-detect → resync → SBE output (deltas + snapshot
every minute, on subscribe, after a resync) → receive-time stamping → clean
SIGINT, with the `PING` keepalive every 10 s. Loss detection: Polymarket's
book hash, reproduced after every change. One socket, a fixed subset of
symbols. Deterministic replay tests + snapshot↔delta reconciliation.
Sprint: [../roadmap/02. book pipeline.md](../roadmap/02.%20book%20pipeline.md).
*Ends:* correct SBE output for a symbol subset; tests green.
3. **Full coverage: discovery + lanes (base challenge complete)** — *done (2026-11-11)*
Every live symbol (~476k tokens, 2026-11-04) spread over lanes, each a
connection and a book thread of its own; an orchestrator keeps the tracked
set equal to Gamma's live set (a scan at startup and every hour, new and
closed markets every minute) through subscribe / unsubscribe on open
connections.
Sprint: [../roadmap/03. full coverage.md](../roadmap/03.%20full%20coverage.md).
*Ends:* every live Polymarket symbol tracked — base challenge satisfied for
one exchange.
4. **Extension 1 — redundant websockets** — *done (2026-11-11)*
Every symbol on two connections chosen independently, its book taking each
change from whichever delivers it first (first one wins); connections
separated from book threads; a connection the server keeps closing with
1013 is split, so a budget set too high corrects itself. The async
checkpoint was held: threads stay
([ADR 0001](adr/0001-concurrency-model.md)).
Sprint: [../roadmap/04. redundant connections.md](../roadmap/04.%20redundant%20connections.md).
Left as ceilings: the writer's in-memory chunk on a hard crash, and the
subscription gap when a symbol opens.
*Ends:* redundancy demonstrably reduces data loss: fewer losses and
subscription gaps per million changes with two subscriptions than with one.
Result: losses a quarter lower in each pair of alternating runs; losses and
gaps together lower in one pair, even in the other (sprint 04's Outcome).
5. **Extension 2 — subscription monitoring** — *done (2026-11-11)*
Per-symbol activity monitoring (data points per second/minute); unsubscribe +
resubscribe symbols that go too quiet (with two connections, first one wins
hides a subscription gone quiet on one of them: it has to be looked for); periodic full-socket recycle
(disconnect → reconnect → resubscribe) to shed cumulative exchange-side bugs.
Sprint: [../roadmap/05. subscription monitoring.md](../roadmap/05.%20subscription%20monitoring.md).
*Ends:* monitoring + recycling running.
Result: a 30-minute run recycled 107 connections with losses no higher
than sprint 04's; quiet subscriptions are rare (26 resubscribed, none
quiet again) and no connection fell silent (sprint 05's Outcome).
6. **Extension 3 — generic core + Kalshi adapter** — *done (2026-11-11)*
The exchange-agnostic core extracted from the working Polymarket code,
and Kalshi added through it. Each exchange supplies its websocket layer
and its loss rule ([ADR 0004](adr/0004-exchange-seam.md)); on Kalshi,
first one wins by clock and rank.
Sprint: [../roadmap/06. generic core and kalshi.md](../roadmap/06.%20generic%20core%20and%20kalshi.md).
*Ends:* both Polymarket and Kalshi work through one generic core.
Result: 10-minute live runs of each, reconciled; Polymarket's losses no
higher than sprint 05's; Kalshi on its demo (about 245,000 markets on 110
connections), production not run for lack of a key (sprint 06's Outcome).
7. **Extension 4 — latency experiments + writeup** — *not started*
Experiment with faster connections (first-after-maintenance, connection
racing, source-port tricks) on both exchanges; document findings in a
markdown file, with the local clock's offset measured alongside (the dev
machine ran ~30 ms slow; see notes.md).
*Ends:* experiments documented.
8. **Polish + submission** — *not started*
Final review, README for the solution (the template's README still says the
helpers are "demonstrated in `main.rs`", whose demo M1 replaced),
packaging/handoff for the challenge.
*Ends:* ready to submit.
## Why this order
- Milestones 1–3 prove the pipeline (the central bet) and finish the base
challenge on one exchange — the riskiest and most valuable work, up front.
- Extensions 1 and 2 harden that single exchange before we generalize, so the
generic core (6) is extracted from code that already does the hard things.
- The generic abstraction waits for the second exchange (Kalshi), so it's
extracted from two real implementations rather than guessed from one.
- Latency (4) is experimentation, least coupled to correctness, so it's last
before polish.
## Possible features (beyond v1)
- More exchanges beyond Polymarket and Kalshi.
- MVE / multi-outcome market support.
- Non-websocket exchange support.
- Persistence / replay tooling beyond the raw SBE output.
- Observability: metrics, structured logging, dashboards.
- Config-driven symbol filtering and per-exchange tuning.
- Containerized packaging.
- Polymarket's `new_market` websocket event as a faster discovery signal (it
is broadcast, but lacks `orderMinSize` / `negRisk`, so Gamma is still
needed; `market_resolved` was never seen).
- Closing connections that drain.
- Verifying a symbol again from the trade event's price after a trade, with
no resync (sprint 04's prototype did it for 15 of 100 broken chains).
- Telling a real loss from a resync book merely ahead of our stream, so the
loss count measures data missed (most of sprint 04's "losses" follow
trades, as often with one connection as with two).
- Closing the opening subscription gap with the second connection's opening
book, when the first one's chain breaks before any change verified.
- An exact rescued count: a change remembered past the window of 128 until
its copies are counted (sprint 04 reports an upper bound).
- Telling a change still in flight from a symbol quiet on both connections,
when a reconnect's book is newer than ours (`maybe`: sprint 05 logs both
as a loss; the Snapshot is right either way).
- Fewer Snapshots for unchanged books (the brief asks for every symbol every
minute: ~130 MB a minute at full coverage).
- Re-reading a market's `orderMinSize` / `negRisk` if they can change.
- A Kalshi production run, once a production key exists.
- Retrying a failed rescan sooner than the next hourly one (`maybe`: Kalshi's
three-hour opening window covers one failure).
- One place for the two rules' resync bookkeeping (asking for a fresh book,
again on leave and at the minute), in the shell (`maybe`: it means
rewriting Polymarket's).
- Telling Kalshi's book at a market's close, and a connection's book merely
ahead of the other's stream, from a real loss (sprint 06 logs all three as
a loss; the Snapshot is right either way).