This repository has been archived on 2026-11-16. You can view files and clone it. You cannot open issues or pull requests or push a commit.
Files
pascal cf6c16c9cf feat: ready to submit, a run to watch, the README, the review's fixes
Sprint 08 (roadmap/08. ready to submit.md), the last milestone.

- scripts/watch.sh runs the collector with a live view of its log, stops
  cleanly on Ctrl-C, then checks the output (reconcile) and shows one
  symbol's book as text (show, a new binary).
- The README for the solution, with the final runs: Polymarket through an
  outage it recovered from on its own, Kalshi's demo with two slow
  connections recycled.
- From a review of the whole codebase: Kalshi takes RSA keys too
  (RSA-PSS-SHA256); a slow Kalshi connection takes the hourly recycle's
  next turn, which it nearly never got before; a market tracked again no
  longer fails reconcile; Book::add checks for overflow; usage and --help
  name the key variables; the first scan says how far it got.
- Tests for every gap the reviews found, an offline run of the whole
  orchestrator among them; ShellCheck in CI.
2026-11-12 22:32:39 +01:00

10 KiB
Raw Permalink Blame History

Roadmap

Goal: the order-book collector defined in project.md — a correct, complete, time-accurate feed for prediction-market order books, covering all four challenge extensions across Polymarket and Kalshi.

Ordering principle: prove the central bet (the pipeline) and one exchange first, so base + Polymarket is already a complete, submittable deliverable. Everything after is additive; nothing risky sits at the end. The generic abstraction is extracted once a second exchange exists, not designed up front.

Milestones

  1. Skeleton + connectivity — done (2026-11-04) Project scaffolding inside the template: module layout (replace main.rs), scripts/ci.sh (fmt + clippy -D warnings + tests), project CLAUDE.md, the websocket library (tungstenite + rustls). Runtime settled: thread-only V1 (async deferred to M4; see adr/0001-concurrency-model.md). Output destination is a non-issue (an output path; /dev/stdout for stdout). Connect to Polymarket and log raw messages. Ends: CI green; raw Polymarket messages flowing.

  2. Single-source book pipeline (Polymarket, subset of symbols) — done (2026-11-04) The central bet in its first real form: parse → per-symbol book (fold absolute deltas) → loss-detect → resync → SBE output (deltas + snapshot every minute, on subscribe, after a resync) → receive-time stamping → clean SIGINT, with the PING keepalive every 10 s. Loss detection: Polymarket's book hash, reproduced after every change. One socket, a fixed subset of symbols. Deterministic replay tests + snapshot↔delta reconciliation. Sprint: ../roadmap/02. book pipeline.md. Ends: correct SBE output for a symbol subset; tests green.

  3. Full coverage: discovery + lanes (base challenge complete) — done (2026-11-11) Every live symbol (~476k tokens, 2026-11-04) spread over lanes, each a connection and a book thread of its own; an orchestrator keeps the tracked set equal to Gamma's live set (a scan at startup and every hour, new and closed markets every minute) through subscribe / unsubscribe on open connections. Sprint: ../roadmap/03. full coverage.md. Ends: every live Polymarket symbol tracked — base challenge satisfied for one exchange.

  4. Extension 1 — redundant websockets — done (2026-11-11) Every symbol on two connections chosen independently, its book taking each change from whichever delivers it first (first one wins); connections separated from book threads; a connection the server keeps closing with 1013 is split, so a budget set too high corrects itself. The async checkpoint was held: threads stay (ADR 0001). Sprint: ../roadmap/04. redundant connections.md. Left as ceilings: the writer's in-memory chunk on a hard crash, and the subscription gap when a symbol opens. Ends: redundancy demonstrably reduces data loss: fewer losses and subscription gaps per million changes with two subscriptions than with one. Result: losses a quarter lower in each pair of alternating runs; losses and gaps together lower in one pair, even in the other (sprint 04's Outcome).

  5. Extension 2 — subscription monitoring — done (2026-11-11) Per-symbol activity monitoring (data points per second/minute); unsubscribe + resubscribe symbols that go too quiet (with two connections, first one wins hides a subscription gone quiet on one of them: it has to be looked for); periodic full-socket recycle (disconnect → reconnect → resubscribe) to shed cumulative exchange-side bugs. Sprint: ../roadmap/05. subscription monitoring.md. Ends: monitoring + recycling running. Result: a 30-minute run recycled 107 connections with losses no higher than sprint 04's; quiet subscriptions are rare (26 resubscribed, none quiet again) and no connection fell silent (sprint 05's Outcome).

  6. Extension 3 — generic core + Kalshi adapter — done (2026-11-11) The exchange-agnostic core extracted from the working Polymarket code, and Kalshi added through it. Each exchange supplies its websocket layer and its loss rule (ADR 0004); on Kalshi, first one wins by clock and rank. Sprint: ../roadmap/06. generic core and kalshi.md. Ends: both Polymarket and Kalshi work through one generic core. Result: 10-minute live runs of each, reconciled; Polymarket's losses no higher than sprint 05's; Kalshi on its demo (about 245,000 markets on 110 connections), production not run for lack of a key (sprint 06's Outcome).

  7. Extension 4 — latency experiments + writeup — done (2026-11-12) Is a connection that is fast now still fast later? 16 connections to the same markets, ranked by how long after a change's first copy each delivers it (no clock needed), with the clock's offset measured by SNTP; written up in ../LATENCY.md. Sprint: ../roadmap/07. latency experiments.md. Ends: experiments documented. Result: speed persists on both exchanges; keeping the fastest wins under a millisecond on Polymarket (not in the tail), tens of milliseconds on Kalshi's demo, where the collector now recycles a connection whose mean lateness stays above twice everyone's (none was, in 20 live minutes). The source port can't be chosen here (a NAT rewrites it); first after maintenance wasn't tried.

  8. Polish + submission — done (2026-11-12) A review of the whole codebase and its fixes, a run anyone can watch (scripts/watch.sh, show), the README, final runs of both exchanges, and the handoff as a public repo on git.pband.ch. Sprint: ../roadmap/08. ready to submit.md. Ends: ready to submit. Result: Kalshi takes RSA keys too; the slow recycle, which the hourly one nearly always pre-empted, now takes its turn and acted twice in 20 live minutes; a market tracked again no longer fails reconcile; 19 surviving mutants got their tests. Both final runs reconciled, one through an outage of about two minutes (sprint 08's Outcome).

Why this order

  • Milestones 1–3 prove the pipeline (the central bet) and finish the base challenge on one exchange — the riskiest and most valuable work, up front.
  • Extensions 1 and 2 harden that single exchange before we generalize, so the generic core (6) is extracted from code that already does the hard things.
  • The generic abstraction waits for the second exchange (Kalshi), so it's extracted from two real implementations rather than guessed from one.
  • Latency (4) is experimentation, least coupled to correctness, so it's last before polish.

Possible features (beyond v1)

  • More exchanges beyond Polymarket and Kalshi.
  • MVE / multi-outcome market support.
  • Non-websocket exchange support.
  • Persistence / replay tooling beyond the raw SBE output.
  • Observability: metrics, structured logging, dashboards.
  • Config-driven symbol filtering and per-exchange tuning.
  • Containerized packaging.
  • Polymarket's new_market websocket event as a faster discovery signal (it is broadcast, but lacks orderMinSize / negRisk, so Gamma is still needed; market_resolved was never seen).
  • Closing connections that drain.
  • Verifying a symbol again from the trade event's price after a trade, with no resync (sprint 04's prototype did it for 15 of 100 broken chains).
  • Telling a real loss from a resync book merely ahead of our stream, so the loss count measures data missed (most of sprint 04's "losses" follow trades, as often with one connection as with two).
  • Closing the opening subscription gap with the second connection's opening book, when the first one's chain breaks before any change verified.
  • An exact rescued count: a change remembered past the window of 128 until its copies are counted (sprint 04 reports an upper bound).
  • Telling a change still in flight from a symbol quiet on both connections, when a reconnect's book is newer than ours (maybe: sprint 05 logs both as a loss; the Snapshot is right either way).
  • Fewer Snapshots for unchanged books (the brief asks for every symbol every minute: ~130 MB a minute at full coverage).
  • Re-reading a market's orderMinSize / negRisk if they can change.
  • A Kalshi production run, once a production key exists.
  • Retrying a failed rescan sooner than the next hourly one (maybe: Kalshi's three-hour opening window covers one failure).
  • One place for the two rules' resync bookkeeping (asking for a fresh book, again on leave and at the minute), in the shell (maybe: it means rewriting Polymarket's).
  • Telling Kalshi's book at a market's close, and a connection's book merely ahead of the other's stream, from a real loss (sprint 06 logs all three as a loss; the Snapshot is right either way).
  • The first connection after Kalshi's maintenance (Thursdays 08:00 to 10:00 UTC in winter): the probe at the window's end against an hour later (sprint 07 ran after one).
  • The TCP source port's effect on speed, from a host with a public address (here a NAT rewrites it).
  • The slow recycle's gain on Kalshi production, measured on the collector (sprint 07 measured racing with the probe, on the demo only).
  • Judging the connections' speed at the book threads' wall-clock minute, not on the orchestrator's own timer (a judgment near the crossing sees partial samples; sprint 07's review).
  • Kalshi: a resync asked from a connection the symbol has since left stays asked there while the other connection is down (sprint 08's review; rare, and the resync count grows each minute).
  • Paging Kalshi's listing in parallel: on 2026-11-12 the demo served a page of 1,000 markets in 1.7 s, so the first scan took 7.5 minutes (under one in sprint 06).
  • Connecting to each of the exchange's addresses in turn, not only the first the lookup returns (a host with IPv6 DNS but no IPv6 route would never connect; sprint 08's review).