Archived
Sprint 08 (roadmap/08. ready to submit.md), the last milestone. - scripts/watch.sh runs the collector with a live view of its log, stops cleanly on Ctrl-C, then checks the output (reconcile) and shows one symbol's book as text (show, a new binary). - The README for the solution, with the final runs: Polymarket through an outage it recovered from on its own, Kalshi's demo with two slow connections recycled. - From a review of the whole codebase: Kalshi takes RSA keys too (RSA-PSS-SHA256); a slow Kalshi connection takes the hourly recycle's next turn, which it nearly never got before; a market tracked again no longer fails reconcile; Book::add checks for overflow; usage and --help name the key variables; the first scan says how far it got. - Tests for every gap the reviews found, an offline run of the whole orchestrator among them; ShellCheck in CI.
173 lines
10 KiB
Markdown
173 lines
10 KiB
Markdown
# Roadmap
|
||
|
||
Goal: the order-book collector defined in [project.md](project.md) — a
|
||
correct, complete, time-accurate feed for prediction-market order books,
|
||
covering all four challenge extensions across Polymarket and Kalshi.
|
||
|
||
Ordering principle: **prove the central bet (the pipeline) and one exchange
|
||
first, so base + Polymarket is already a complete, submittable deliverable.**
|
||
Everything after is additive; nothing risky sits at the end. The generic
|
||
abstraction is *extracted* once a second exchange exists, not designed up front.
|
||
|
||
## Milestones
|
||
|
||
1. **Skeleton + connectivity** — *done (2026-11-04)*
|
||
Project scaffolding inside the template: module layout (replace `main.rs`),
|
||
`scripts/ci.sh` (fmt + clippy `-D warnings` + tests), project CLAUDE.md, the
|
||
websocket library (`tungstenite` + `rustls`). Runtime settled: thread-only V1
|
||
(async deferred to M4; see [adr/0001-concurrency-model.md](adr/0001-concurrency-model.md)).
|
||
Output destination is a non-issue (an output path; `/dev/stdout` for stdout).
|
||
Connect to Polymarket and log raw messages.
|
||
*Ends:* CI green; raw Polymarket messages flowing.
|
||
|
||
2. **Single-source book pipeline (Polymarket, subset of symbols)** — *done (2026-11-04)*
|
||
The central bet in its first real form: parse → per-symbol book (fold
|
||
absolute deltas) → loss-detect → resync → SBE output (deltas + snapshot
|
||
every minute, on subscribe, after a resync) → receive-time stamping → clean
|
||
SIGINT, with the `PING` keepalive every 10 s. Loss detection: Polymarket's
|
||
book hash, reproduced after every change. One socket, a fixed subset of
|
||
symbols. Deterministic replay tests + snapshot↔delta reconciliation.
|
||
Sprint: [../roadmap/02. book pipeline.md](../roadmap/02.%20book%20pipeline.md).
|
||
*Ends:* correct SBE output for a symbol subset; tests green.
|
||
|
||
3. **Full coverage: discovery + lanes (base challenge complete)** — *done (2026-11-11)*
|
||
Every live symbol (~476k tokens, 2026-11-04) spread over lanes, each a
|
||
connection and a book thread of its own; an orchestrator keeps the tracked
|
||
set equal to Gamma's live set (a scan at startup and every hour, new and
|
||
closed markets every minute) through subscribe / unsubscribe on open
|
||
connections.
|
||
Sprint: [../roadmap/03. full coverage.md](../roadmap/03.%20full%20coverage.md).
|
||
*Ends:* every live Polymarket symbol tracked — base challenge satisfied for
|
||
one exchange.
|
||
|
||
4. **Extension 1 — redundant websockets** — *done (2026-11-11)*
|
||
Every symbol on two connections chosen independently, its book taking each
|
||
change from whichever delivers it first (first one wins); connections
|
||
separated from book threads; a connection the server keeps closing with
|
||
1013 is split, so a budget set too high corrects itself. The async
|
||
checkpoint was held: threads stay
|
||
([ADR 0001](adr/0001-concurrency-model.md)).
|
||
Sprint: [../roadmap/04. redundant connections.md](../roadmap/04.%20redundant%20connections.md).
|
||
Left as ceilings: the writer's in-memory chunk on a hard crash, and the
|
||
subscription gap when a symbol opens.
|
||
*Ends:* redundancy demonstrably reduces data loss: fewer losses and
|
||
subscription gaps per million changes with two subscriptions than with one.
|
||
Result: losses a quarter lower in each pair of alternating runs; losses and
|
||
gaps together lower in one pair, even in the other (sprint 04's Outcome).
|
||
|
||
5. **Extension 2 — subscription monitoring** — *done (2026-11-11)*
|
||
Per-symbol activity monitoring (data points per second/minute); unsubscribe +
|
||
resubscribe symbols that go too quiet (with two connections, first one wins
|
||
hides a subscription gone quiet on one of them: it has to be looked for); periodic full-socket recycle
|
||
(disconnect → reconnect → resubscribe) to shed cumulative exchange-side bugs.
|
||
Sprint: [../roadmap/05. subscription monitoring.md](../roadmap/05.%20subscription%20monitoring.md).
|
||
*Ends:* monitoring + recycling running.
|
||
Result: a 30-minute run recycled 107 connections with losses no higher
|
||
than sprint 04's; quiet subscriptions are rare (26 resubscribed, none
|
||
quiet again) and no connection fell silent (sprint 05's Outcome).
|
||
|
||
6. **Extension 3 — generic core + Kalshi adapter** — *done (2026-11-11)*
|
||
The exchange-agnostic core extracted from the working Polymarket code,
|
||
and Kalshi added through it. Each exchange supplies its websocket layer
|
||
and its loss rule ([ADR 0004](adr/0004-exchange-seam.md)); on Kalshi,
|
||
first one wins by clock and rank.
|
||
Sprint: [../roadmap/06. generic core and kalshi.md](../roadmap/06.%20generic%20core%20and%20kalshi.md).
|
||
*Ends:* both Polymarket and Kalshi work through one generic core.
|
||
Result: 10-minute live runs of each, reconciled; Polymarket's losses no
|
||
higher than sprint 05's; Kalshi on its demo (about 245,000 markets on 110
|
||
connections), production not run for lack of a key (sprint 06's Outcome).
|
||
|
||
7. **Extension 4 — latency experiments + writeup** — *done (2026-11-12)*
|
||
Is a connection that is fast now still fast later? 16 connections to the
|
||
same markets, ranked by how long after a change's first copy each
|
||
delivers it (no clock needed), with the clock's offset measured by SNTP;
|
||
written up in [../LATENCY.md](../LATENCY.md).
|
||
Sprint: [../roadmap/07. latency experiments.md](../roadmap/07.%20latency%20experiments.md).
|
||
*Ends:* experiments documented.
|
||
Result: speed persists on both exchanges; keeping the fastest wins under
|
||
a millisecond on Polymarket (not in the tail), tens of milliseconds on
|
||
Kalshi's demo, where the collector now recycles a connection whose mean
|
||
lateness stays above twice everyone's (none was, in 20 live minutes).
|
||
The source port can't be chosen here (a NAT rewrites it); first after
|
||
maintenance wasn't tried.
|
||
|
||
8. **Polish + submission** — *done (2026-11-12)*
|
||
A review of the whole codebase and its fixes, a run anyone can watch
|
||
(`scripts/watch.sh`, `show`), the README, final runs of both
|
||
exchanges, and the handoff as a public repo on git.pband.ch.
|
||
Sprint: [../roadmap/08. ready to submit.md](../roadmap/08.%20ready%20to%20submit.md).
|
||
*Ends:* ready to submit.
|
||
Result: Kalshi takes RSA keys too; the slow recycle, which the hourly
|
||
one nearly always pre-empted, now takes its turn and acted twice in 20
|
||
live minutes; a market tracked again no longer fails `reconcile`; 19
|
||
surviving mutants got their tests. Both final runs reconciled, one
|
||
through an outage of about two minutes (sprint 08's Outcome).
|
||
|
||
## Why this order
|
||
|
||
- Milestones 1–3 prove the pipeline (the central bet) and finish the base
|
||
challenge on one exchange — the riskiest and most valuable work, up front.
|
||
- Extensions 1 and 2 harden that single exchange before we generalize, so the
|
||
generic core (6) is extracted from code that already does the hard things.
|
||
- The generic abstraction waits for the second exchange (Kalshi), so it's
|
||
extracted from two real implementations rather than guessed from one.
|
||
- Latency (4) is experimentation, least coupled to correctness, so it's last
|
||
before polish.
|
||
|
||
## Possible features (beyond v1)
|
||
|
||
- More exchanges beyond Polymarket and Kalshi.
|
||
- MVE / multi-outcome market support.
|
||
- Non-websocket exchange support.
|
||
- Persistence / replay tooling beyond the raw SBE output.
|
||
- Observability: metrics, structured logging, dashboards.
|
||
- Config-driven symbol filtering and per-exchange tuning.
|
||
- Containerized packaging.
|
||
- Polymarket's `new_market` websocket event as a faster discovery signal (it
|
||
is broadcast, but lacks `orderMinSize` / `negRisk`, so Gamma is still
|
||
needed; `market_resolved` was never seen).
|
||
- Closing connections that drain.
|
||
- Verifying a symbol again from the trade event's price after a trade, with
|
||
no resync (sprint 04's prototype did it for 15 of 100 broken chains).
|
||
- Telling a real loss from a resync book merely ahead of our stream, so the
|
||
loss count measures data missed (most of sprint 04's "losses" follow
|
||
trades, as often with one connection as with two).
|
||
- Closing the opening subscription gap with the second connection's opening
|
||
book, when the first one's chain breaks before any change verified.
|
||
- An exact rescued count: a change remembered past the window of 128 until
|
||
its copies are counted (sprint 04 reports an upper bound).
|
||
- Telling a change still in flight from a symbol quiet on both connections,
|
||
when a reconnect's book is newer than ours (`maybe`: sprint 05 logs both
|
||
as a loss; the Snapshot is right either way).
|
||
- Fewer Snapshots for unchanged books (the brief asks for every symbol every
|
||
minute: ~130 MB a minute at full coverage).
|
||
- Re-reading a market's `orderMinSize` / `negRisk` if they can change.
|
||
- A Kalshi production run, once a production key exists.
|
||
- Retrying a failed rescan sooner than the next hourly one (`maybe`: Kalshi's
|
||
three-hour opening window covers one failure).
|
||
- One place for the two rules' resync bookkeeping (asking for a fresh book,
|
||
again on leave and at the minute), in the shell (`maybe`: it means
|
||
rewriting Polymarket's).
|
||
- Telling Kalshi's book at a market's close, and a connection's book merely
|
||
ahead of the other's stream, from a real loss (sprint 06 logs all three as
|
||
a loss; the Snapshot is right either way).
|
||
- The first connection after Kalshi's maintenance (Thursdays 08:00 to
|
||
10:00 UTC in winter): the probe at the window's end against an hour later
|
||
(sprint 07 ran after one).
|
||
- The TCP source port's effect on speed, from a host with a public address
|
||
(here a NAT rewrites it).
|
||
- The slow recycle's gain on Kalshi production, measured on the collector
|
||
(sprint 07 measured racing with the probe, on the demo only).
|
||
- Judging the connections' speed at the book threads' wall-clock minute,
|
||
not on the orchestrator's own timer (a judgment near the crossing sees
|
||
partial samples; sprint 07's review).
|
||
- Kalshi: a resync asked from a connection the symbol has since left stays
|
||
asked there while the other connection is down (sprint 08's review; rare,
|
||
and the resync count grows each minute).
|
||
- Paging Kalshi's listing in parallel: on 2026-11-12 the demo served a page
|
||
of 1,000 markets in 1.7 s, so the first scan took 7.5 minutes (under one
|
||
in sprint 06).
|
||
- Connecting to each of the exchange's addresses in turn, not only the
|
||
first the lookup returns (a host with IPv6 DNS but no IPv6 route would
|
||
never connect; sprint 08's review).
|