This repository has been archived on 2026-11-16. You can view files and clone it. You cannot open issues or pull requests or push a commit.
Files
orderbook-collector/docs/roadmap.md
T
pascal cf6c16c9cf feat: ready to submit, a run to watch, the README, the review's fixes
Sprint 08 (roadmap/08. ready to submit.md), the last milestone.

- scripts/watch.sh runs the collector with a live view of its log, stops
  cleanly on Ctrl-C, then checks the output (reconcile) and shows one
  symbol's book as text (show, a new binary).
- The README for the solution, with the final runs: Polymarket through an
  outage it recovered from on its own, Kalshi's demo with two slow
  connections recycled.
- From a review of the whole codebase: Kalshi takes RSA keys too
  (RSA-PSS-SHA256); a slow Kalshi connection takes the hourly recycle's
  next turn, which it nearly never got before; a market tracked again no
  longer fails reconcile; Book::add checks for overflow; usage and --help
  name the key variables; the first scan says how far it got.
- Tests for every gap the reviews found, an offline run of the whole
  orchestrator among them; ShellCheck in CI.
2026-11-12 22:32:39 +01:00

173 lines
10 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Roadmap
Goal: the order-book collector defined in [project.md](project.md) — a
correct, complete, time-accurate feed for prediction-market order books,
covering all four challenge extensions across Polymarket and Kalshi.
Ordering principle: **prove the central bet (the pipeline) and one exchange
first, so base + Polymarket is already a complete, submittable deliverable.**
Everything after is additive; nothing risky sits at the end. The generic
abstraction is *extracted* once a second exchange exists, not designed up front.
## Milestones
1. **Skeleton + connectivity** — *done (2026-11-04)*
Project scaffolding inside the template: module layout (replace `main.rs`),
`scripts/ci.sh` (fmt + clippy `-D warnings` + tests), project CLAUDE.md, the
websocket library (`tungstenite` + `rustls`). Runtime settled: thread-only V1
(async deferred to M4; see [adr/0001-concurrency-model.md](adr/0001-concurrency-model.md)).
Output destination is a non-issue (an output path; `/dev/stdout` for stdout).
Connect to Polymarket and log raw messages.
*Ends:* CI green; raw Polymarket messages flowing.
2. **Single-source book pipeline (Polymarket, subset of symbols)** — *done (2026-11-04)*
The central bet in its first real form: parse → per-symbol book (fold
absolute deltas) → loss-detect → resync → SBE output (deltas + snapshot
every minute, on subscribe, after a resync) → receive-time stamping → clean
SIGINT, with the `PING` keepalive every 10 s. Loss detection: Polymarket's
book hash, reproduced after every change. One socket, a fixed subset of
symbols. Deterministic replay tests + snapshot↔delta reconciliation.
Sprint: [../roadmap/02. book pipeline.md](../roadmap/02.%20book%20pipeline.md).
*Ends:* correct SBE output for a symbol subset; tests green.
3. **Full coverage: discovery + lanes (base challenge complete)** — *done (2026-11-11)*
Every live symbol (~476k tokens, 2026-11-04) spread over lanes, each a
connection and a book thread of its own; an orchestrator keeps the tracked
set equal to Gamma's live set (a scan at startup and every hour, new and
closed markets every minute) through subscribe / unsubscribe on open
connections.
Sprint: [../roadmap/03. full coverage.md](../roadmap/03.%20full%20coverage.md).
*Ends:* every live Polymarket symbol tracked — base challenge satisfied for
one exchange.
4. **Extension 1 — redundant websockets** — *done (2026-11-11)*
Every symbol on two connections chosen independently, its book taking each
change from whichever delivers it first (first one wins); connections
separated from book threads; a connection the server keeps closing with
1013 is split, so a budget set too high corrects itself. The async
checkpoint was held: threads stay
([ADR 0001](adr/0001-concurrency-model.md)).
Sprint: [../roadmap/04. redundant connections.md](../roadmap/04.%20redundant%20connections.md).
Left as ceilings: the writer's in-memory chunk on a hard crash, and the
subscription gap when a symbol opens.
*Ends:* redundancy demonstrably reduces data loss: fewer losses and
subscription gaps per million changes with two subscriptions than with one.
Result: losses a quarter lower in each pair of alternating runs; losses and
gaps together lower in one pair, even in the other (sprint 04's Outcome).
5. **Extension 2 — subscription monitoring** — *done (2026-11-11)*
Per-symbol activity monitoring (data points per second/minute); unsubscribe +
resubscribe symbols that go too quiet (with two connections, first one wins
hides a subscription gone quiet on one of them: it has to be looked for); periodic full-socket recycle
(disconnect → reconnect → resubscribe) to shed cumulative exchange-side bugs.
Sprint: [../roadmap/05. subscription monitoring.md](../roadmap/05.%20subscription%20monitoring.md).
*Ends:* monitoring + recycling running.
Result: a 30-minute run recycled 107 connections with losses no higher
than sprint 04's; quiet subscriptions are rare (26 resubscribed, none
quiet again) and no connection fell silent (sprint 05's Outcome).
6. **Extension 3 — generic core + Kalshi adapter** — *done (2026-11-11)*
The exchange-agnostic core extracted from the working Polymarket code,
and Kalshi added through it. Each exchange supplies its websocket layer
and its loss rule ([ADR 0004](adr/0004-exchange-seam.md)); on Kalshi,
first one wins by clock and rank.
Sprint: [../roadmap/06. generic core and kalshi.md](../roadmap/06.%20generic%20core%20and%20kalshi.md).
*Ends:* both Polymarket and Kalshi work through one generic core.
Result: 10-minute live runs of each, reconciled; Polymarket's losses no
higher than sprint 05's; Kalshi on its demo (about 245,000 markets on 110
connections), production not run for lack of a key (sprint 06's Outcome).
7. **Extension 4 — latency experiments + writeup** — *done (2026-11-12)*
Is a connection that is fast now still fast later? 16 connections to the
same markets, ranked by how long after a change's first copy each
delivers it (no clock needed), with the clock's offset measured by SNTP;
written up in [../LATENCY.md](../LATENCY.md).
Sprint: [../roadmap/07. latency experiments.md](../roadmap/07.%20latency%20experiments.md).
*Ends:* experiments documented.
Result: speed persists on both exchanges; keeping the fastest wins under
a millisecond on Polymarket (not in the tail), tens of milliseconds on
Kalshi's demo, where the collector now recycles a connection whose mean
lateness stays above twice everyone's (none was, in 20 live minutes).
The source port can't be chosen here (a NAT rewrites it); first after
maintenance wasn't tried.
8. **Polish + submission** — *done (2026-11-12)*
A review of the whole codebase and its fixes, a run anyone can watch
(`scripts/watch.sh`, `show`), the README, final runs of both
exchanges, and the handoff as a public repo on git.pband.ch.
Sprint: [../roadmap/08. ready to submit.md](../roadmap/08.%20ready%20to%20submit.md).
*Ends:* ready to submit.
Result: Kalshi takes RSA keys too; the slow recycle, which the hourly
one nearly always pre-empted, now takes its turn and acted twice in 20
live minutes; a market tracked again no longer fails `reconcile`; 19
surviving mutants got their tests. Both final runs reconciled, one
through an outage of about two minutes (sprint 08's Outcome).
## Why this order
- Milestones 1–3 prove the pipeline (the central bet) and finish the base
challenge on one exchange — the riskiest and most valuable work, up front.
- Extensions 1 and 2 harden that single exchange before we generalize, so the
generic core (6) is extracted from code that already does the hard things.
- The generic abstraction waits for the second exchange (Kalshi), so it's
extracted from two real implementations rather than guessed from one.
- Latency (4) is experimentation, least coupled to correctness, so it's last
before polish.
## Possible features (beyond v1)
- More exchanges beyond Polymarket and Kalshi.
- MVE / multi-outcome market support.
- Non-websocket exchange support.
- Persistence / replay tooling beyond the raw SBE output.
- Observability: metrics, structured logging, dashboards.
- Config-driven symbol filtering and per-exchange tuning.
- Containerized packaging.
- Polymarket's `new_market` websocket event as a faster discovery signal (it
is broadcast, but lacks `orderMinSize` / `negRisk`, so Gamma is still
needed; `market_resolved` was never seen).
- Closing connections that drain.
- Verifying a symbol again from the trade event's price after a trade, with
no resync (sprint 04's prototype did it for 15 of 100 broken chains).
- Telling a real loss from a resync book merely ahead of our stream, so the
loss count measures data missed (most of sprint 04's "losses" follow
trades, as often with one connection as with two).
- Closing the opening subscription gap with the second connection's opening
book, when the first one's chain breaks before any change verified.
- An exact rescued count: a change remembered past the window of 128 until
its copies are counted (sprint 04 reports an upper bound).
- Telling a change still in flight from a symbol quiet on both connections,
when a reconnect's book is newer than ours (`maybe`: sprint 05 logs both
as a loss; the Snapshot is right either way).
- Fewer Snapshots for unchanged books (the brief asks for every symbol every
minute: ~130 MB a minute at full coverage).
- Re-reading a market's `orderMinSize` / `negRisk` if they can change.
- A Kalshi production run, once a production key exists.
- Retrying a failed rescan sooner than the next hourly one (`maybe`: Kalshi's
three-hour opening window covers one failure).
- One place for the two rules' resync bookkeeping (asking for a fresh book,
again on leave and at the minute), in the shell (`maybe`: it means
rewriting Polymarket's).
- Telling Kalshi's book at a market's close, and a connection's book merely
ahead of the other's stream, from a real loss (sprint 06 logs all three as
a loss; the Snapshot is right either way).
- The first connection after Kalshi's maintenance (Thursdays 08:00 to
10:00 UTC in winter): the probe at the window's end against an hour later
(sprint 07 ran after one).
- The TCP source port's effect on speed, from a host with a public address
(here a NAT rewrites it).
- The slow recycle's gain on Kalshi production, measured on the collector
(sprint 07 measured racing with the probe, on the demo only).
- Judging the connections' speed at the book threads' wall-clock minute,
not on the orchestrator's own timer (a judgment near the crossing sees
partial samples; sprint 07's review).
- Kalshi: a resync asked from a connection the symbol has since left stays
asked there while the other connection is down (sprint 08's review; rare,
and the resync count grows each minute).
- Paging Kalshi's listing in parallel: on 2026-11-12 the demo served a page
of 1,000 markets in 1.7 s, so the first scan took 7.5 minutes (under one
in sprint 06).
- Connecting to each of the exchange's addresses in turn, not only the
first the lookup returns (a host with IPv6 DNS but no IPv6 route would
never connect; sprint 08's review).