# Project definition — order-book collector ## What it is A market-data collector for prediction-market order books. It connects to one exchange (Polymarket first, Kalshi second), tracks every live symbol, maintains an in-memory order book per symbol, and emits the repo's SBE-encoded price-level updates plus a full snapshot per symbol every minute — the backtesting feed described in the challenge brief. For: the Voluntas engineering challenge (and the backtest pipeline it feeds). ## The goal (what "done" means for v1) A running service that, for a chosen exchange: - discovers all live regular-market symbols and keeps that set current (subscribe to new, unsubscribe from gone); - maintains a correct, complete, time-accurate order book per symbol; - emits SBE `PriceLevelUpdate` deltas continuously and a full `Snapshot` per symbol every minute, on subscribe, and after any resync; - shuts down cleanly on SIGINT (close sockets, flush/close output). Full target (all four extensions): redundant websockets, subscription monitoring, a generic exchange-agnostic core with per-exchange adapters for **both** Polymarket and Kalshi, and latency experiments written up. ## The central bet **The concurrency pipeline.** The risky choice this project stands on is the topology that moves the firehose from many sockets to a single output sink: concurrently, without dropping a message, without racing the per-symbol books, and without adding latency that ruins the timestamps. The data model (a sorted map per side) is easy; the pipeline is where completeness, time-accuracy, and throughput collide. If the pipeline is wrong, no amount of correct book logic saves the data. The interacting decisions it is made of: - **Threading / runtime:** settled in sprint 01 — OS threads + channels, shared-nothing; async was re-evaluated at M4 and threads stay ([ADR 0001](adr/0001-concurrency-model.md)). - **Stamping point:** in the socket-read path, immediately, so as little as possible sits between the wire and the receive timestamp. - **Book ownership:** each book is owned by exactly one of 16 book threads, chosen from its market's id; an orchestrator decides which connections carry each market ([ADR 0002](adr/0002-lanes-and-orchestrator.md), its lanes superseded by [ADR 0003](adr/0003-first-one-wins.md) in sprint 04). - **The exchange seam:** settled in sprint 06 — the pipeline is the shared core; an exchange supplies its websocket layer and, inside the book thread, its loss rule ([ADR 0004](adr/0004-exchange-seam.md)). - **Backpressure:** bounded vs unbounded channels — the core correctness/perf tension. Drop-on-full loses data (violates completeness); block adds latency and fills socket buffers. This decision is the bet in miniature. - **Output serialization:** one writer owns the single sink and consumes from all workers. - **Redundancy:** settled in sprint 04 — every symbol on two connections chosen independently, its book taking each change from whichever delivers it first ([ADR 0003](adr/0003-first-one-wins.md)). ## Requirements the pipeline must satisfy The output's value is completeness + time-accuracy, so whatever the topology, it must guarantee: 1. **Receive time stamped at the socket edge**, before any queue or processing. 2. **A per-symbol book folded from the exchange's changes**, written as absolute Deltas (the level's new quantity; 0 deletes it). Polymarket's changes are already absolute; Kalshi's are relative and are added to the level. 3. **Loss detection → resync** (a lost message means the book is silently corrupt; replacing it with a fresh full book is the only safe response). How a loss is detected depends on the exchange: on Polymarket, recomputing its book hash after every change (settled in M2; the formula is in [notes.md](notes.md)); on Kalshi, a skipped sequence number on a connection, which reconnects it (settled in M6). These are proven first, before redundancy or scale. ## Design ownership The **pipeline / concurrency architecture** is driven by the user (the central bet, debated sprint by sprint). The **data model** is mostly delegated to Claude, as a supporting detail that serves the pipeline. ## Design principles Correctness and completeness first; make it fast later. Readiness for the fast version comes from **clean structure, not added complexity** — the seams we want (thread-per-stage joined by channels, the book as a pure `apply(delta) → book` function, symbols shardable later) are also the simplest, clearest design. - **Boring, idiomatic, readable wins.** Plain `BTreeMap` for a price-ordered side, standard channels and threads. No micro-optimization, no bit-packing, no `unsafe`, no 3am-to-decode data structures in V1. - **No speculative abstraction.** No generic `Exchange` trait up front — extract it once two exchanges exist (milestone 6). Readiness ≠ pre-building the flexible version. - **A clean seam is cheap and allowed; a complex implementation behind it is not.** A channel, a function boundary, or a `trait` with two real impls keeps the door open; the clever internals are what we refuse. - **Never simplified away, even in V1:** receive-time stamping at the socket edge, and loss → resync. These are correctness, not readability tradeoffs. - **If a corner is genuinely cut for speed later** (a global lock, an O(n²) scan), mark it with a comment naming the ceiling and the upgrade path, so the debt is visible rather than silently clever. Their operational form, for writing code, is in the project [CLAUDE.md](../CLAUDE.md#writing-code). ## Constraints (already decided) - **Language/stack:** Rust (fixed by the template). Edition 2024. - **Output format:** the repo's SBE helpers (`AlignedSbeWriter` / `SbeEncode`), non-negotiable — output must be in the standard format. - **Build inside `repo/`** (the challenge template): replace `main.rs`, add modules under `src/`. `website/` stays separate as the brief. - **Scope of markets:** regular markets only; MVEs excluded; websocket exchanges only. - **Output destination** (settled in sprints 01 and 02): `AlignedSbeWriter` takes a file path; the program takes the exchange (`polymarket`, `kalshi` or `kalshi-demo`, required since sprint 06: one exchange per process), then that path, default `/dev/stdout`, and logs go to stderr. No rotation: one output per run (rotation can't apply to stdout; a consumer that wants files rotates them). Chunks must be 64 MiB: `AlignedSbeReader::open` reads nothing else. - **Crash ceiling** (accepted in sprint 02): the writer holds the current 64 MiB chunk in memory until it is full, so a hard crash loses it; flushing more often pads each flush to 64 MiB. Kept as a ceiling in extension 1 (M4). ## Workflow differences See the project [CLAUDE.md](../CLAUDE.md#workflow-differences): what we keep and drop from the global workflow for a job-challenge repo. ## Not in scope (v1) - MVEs (multi-outcome event groupings). - Non-websocket exchanges. - Any exchange beyond Polymarket and Kalshi. - Persistence/replay infrastructure beyond what the SBE output already gives. ## Pipeline The primary thing we debate sprint by sprint (this project's equivalent of the schema debate). As settled in sprints 04 and 05: ``` rescan fetcher ──findings──▶ ┌──────────────┐ ◀──findings── recent fetcher │ orchestrator │ ◀── up, down, 1013 ─── connections └──────┬───▲───┘ ──── recycle ────────▶ connections Add / Move / Remove (a market and │ │ load reports, each minute its connections), to its thread │ │ (bytes/s per market); markets │ │ quiet twice on a connection; │ │ lateness per connection (Kalshi) ┌──────▼───┴───┐ connection 0..N ──events───▶ │ book threads │ ──Snapshot | Delta──▶ writer (read, stamp, parse, route) │ 0..15 │ ▲ └──────┬───────┘ └── subscribe / unsubscribe / resync ┘ ``` - The **orchestrator** owns what is tracked (each market's two connections, the timers) and decides: placing, moving, splitting, and recycling one connection at a time, about hourly, and sooner a Kalshi connection whose mean lateness stays above twice everyone's (sprint 07, LATENCY.md). It never blocks on HTTP. - The two **fetchers** only page the exchange's market listing on request and stream what they read. - A **connection** reads, stamps the receive time, parses, and routes each event to the book thread of its market. Every symbol is on two connections, chosen independently. One silent for 30 s reconnects. - A **book thread** owns its markets' books and applies first one wins: a change is a duplicate, the next one, or held until it fits (fold, check, resync, Snapshots as before). It resubscribes a subscription gone quiet on one connection, and judges a reconnect's books of unchanged symbols. - One **writer** owns the output. Settled: OS threads, shared-nothing, no mutexes ([ADR 0001](adr/0001-concurrency-model.md)); the orchestrator ([ADR 0002](adr/0002-lanes-and-orchestrator.md)); two connections per symbol, first one wins ([ADR 0003](adr/0003-first-one-wins.md)). Details: [sprint 04](../roadmap/04.%20redundant%20connections.md). The pipeline is the same for every exchange. What an exchange supplies ([ADR 0004](adr/0004-exchange-seam.md)): the websocket layer of a connection (request, encoding of commands, decoding of frames, keepalive), the fetchers, and the rule a book thread applies to an event (duplicate, next, held, loss). On Polymarket that rule is the hash chain above; on Kalshi a change is the next one when its clock and rank pass the book's, and a held change doesn't exist. ## Data model, first sketch Supporting detail (mostly Claude's to settle), serving the pipeline above. Not a DB — an in-memory domain plus the fixed SBE output schema. Main entities: - **Symbol** (from discovery): one order book, with its trading status and any price/tick metadata the exchange gives. On Polymarket that is a token, not a market: a market has two (Yes / No), each with its own, mirrored book. On Kalshi it is the market: its bids for Yes are the bids, its bids for No at `q` the asks at `1 − q`. - **Event** (settled in sprint 06, in place of the `NormalizedEvent` first sketched): each exchange's own type. The core reads only the market it belongs to and whether it is a full book; the exchange's rule reads the rest (side, price, quantity, and what the exchange offers for loss detection: a book hash, a sequence number, a clock) and folds it into the symbol's book, setting a level's quantity (Polymarket) or moving it by a signed amount (Kalshi). - **Book** (one per symbol): bids and asks as ordered price→quantity maps, plus the last loss-detection marker seen and last exchange timestamp. It is fed by the symbol's two connections, first one wins. - **Connection** (one websocket): the set of symbols it carries. Live markets are placed on connections by market, all its symbols together; Polymarket's lifecycle signal is Gamma (settled in sprint 03: a market is live while active, not closed, with an order book and accepting orders), Kalshi's its market listing (sprint 06: open, or opening within three hours, tracked ahead so the book is there from its first order). Output schema is fixed by the SBE schema: `Snapshot` and `PriceLevelUpdate`. Redundancy was settled in sprint 04: no source is trusted, the first copy of each change wins ([ADR 0003](adr/0003-first-one-wins.md)). On Kalshi the first copy is recognised by clock and rank (sprint 06). ## Open questions None. (Loss detection on Kalshi was the last: settled in M6, by its per-connection sequence number.)