The exchange can fail without telling us, and first one wins hides a subscription gone quiet on one of a symbol's two connections. - A connection that receives nothing for 30 s, not even the keepalive's answer, reconnects. - Each book thread counts a symbol's changes per connection every minute: a subscription with none against three or more on the other is resubscribed, and moved to another connection if it stays quiet. - The orchestrator recycles one connection at a time, each about hourly, and never while another connection is down or just back. - A reconnect's opening book of a symbol unchanged for 10 s is judged like a resync's answer: one newer than ours that differs shows a symbol quiet on both connections, written as a Snapshot and logged as a loss.
10 KiB
Project definition — order-book collector
What it is
A market-data collector for prediction-market order books. It connects to one exchange (Polymarket first, Kalshi second), tracks every live symbol, maintains an in-memory order book per symbol, and emits the repo's SBE-encoded price-level updates plus a full snapshot per symbol every minute — the backtesting feed described in the challenge brief.
For: the Voluntas engineering challenge (and the backtest pipeline it feeds).
The goal (what "done" means for v1)
A running service that, for a chosen exchange:
- discovers all live regular-market symbols and keeps that set current (subscribe to new, unsubscribe from gone);
- maintains a correct, complete, time-accurate order book per symbol;
- emits SBE
PriceLevelUpdatedeltas continuously and a fullSnapshotper symbol every minute, on subscribe, and after any resync; - shuts down cleanly on SIGINT (close sockets, flush/close output).
Full target (all four extensions): redundant websockets, subscription monitoring, a generic exchange-agnostic core with per-exchange adapters for both Polymarket and Kalshi, and latency experiments written up.
The central bet
The concurrency pipeline. The risky choice this project stands on is the topology that moves the firehose from many sockets to a single output sink: concurrently, without dropping a message, without racing the per-symbol books, and without adding latency that ruins the timestamps. The data model (a sorted map per side) is easy; the pipeline is where completeness, time-accuracy, and throughput collide. If the pipeline is wrong, no amount of correct book logic saves the data.
The interacting decisions it is made of:
- Threading / runtime: settled in sprint 01 — OS threads + channels, shared-nothing; async was re-evaluated at M4 and threads stay (ADR 0001).
- Stamping point: in the socket-read path, immediately, so as little as possible sits between the wire and the receive timestamp.
- Book ownership: each book is owned by exactly one of 16 book threads, chosen from its market's id; an orchestrator decides which connections carry each market (ADR 0002, its lanes superseded by ADR 0003 in sprint 04).
- Backpressure: bounded vs unbounded channels — the core correctness/perf tension. Drop-on-full loses data (violates completeness); block adds latency and fills socket buffers. This decision is the bet in miniature.
- Output serialization: one writer owns the single sink and consumes from all workers.
- Redundancy: settled in sprint 04 — every symbol on two connections chosen independently, its book taking each change from whichever delivers it first (ADR 0003).
Requirements the pipeline must satisfy
The output's value is completeness + time-accuracy, so whatever the topology, it must guarantee:
- Receive time stamped at the socket edge, before any queue or processing.
- A per-symbol book folded from absolute deltas (overwrite the level; quantity 0 deletes it).
- Loss detection → resync (a lost message means the book is silently corrupt; replacing it with a fresh full book is the only safe response). How a loss is detected depends on the exchange: a sequence gap where one exists; on Polymarket, which has none, recomputing its book hash after every change (settled in M2; the formula is in notes.md).
These are proven first, before redundancy or scale.
Design ownership
The pipeline / concurrency architecture is driven by the user (the central bet, debated sprint by sprint). The data model is mostly delegated to Claude, as a supporting detail that serves the pipeline.
Design principles
Correctness and completeness first; make it fast later. Readiness for the fast
version comes from clean structure, not added complexity — the seams we want
(thread-per-stage joined by channels, the book as a pure apply(delta) → book
function, symbols shardable later) are also the simplest, clearest design.
- Boring, idiomatic, readable wins. Plain
BTreeMapfor a price-ordered side, standard channels and threads. No micro-optimization, no bit-packing, nounsafe, no 3am-to-decode data structures in V1. - No speculative abstraction. No generic
Exchangetrait up front — extract it once two exchanges exist (milestone 6). Readiness ≠ pre-building the flexible version. - A clean seam is cheap and allowed; a complex implementation behind it is
not. A channel, a function boundary, or a
traitwith two real impls keeps the door open; the clever internals are what we refuse. - Never simplified away, even in V1: receive-time stamping at the socket edge, and loss → resync. These are correctness, not readability tradeoffs.
- If a corner is genuinely cut for speed later (a global lock, an O(n²) scan), mark it with a comment naming the ceiling and the upgrade path, so the debt is visible rather than silently clever.
Their operational form, for writing code, is in the project CLAUDE.md.
Constraints (already decided)
- Language/stack: Rust (fixed by the template). Edition 2024.
- Output format: the repo's SBE helpers (
AlignedSbeWriter/SbeEncode), non-negotiable — output must be in the standard format. - Build inside
repo/(the challenge template): replacemain.rs, add modules undersrc/.website/stays separate as the brief. - Scope of markets: regular markets only; MVEs excluded; websocket exchanges only.
- Output destination (settled in sprints 01 and 02):
AlignedSbeWritertakes a file path; the program's only argument is that path, default/dev/stdout, and logs go to stderr. No rotation: one output per run (rotation can't apply to stdout; a consumer that wants files rotates them). Chunks must be 64 MiB:AlignedSbeReader::openreads nothing else. - Crash ceiling (accepted in sprint 02): the writer holds the current 64 MiB chunk in memory until it is full, so a hard crash loses it; flushing more often pads each flush to 64 MiB. Kept as a ceiling in extension 1 (M4).
Workflow differences
See the project CLAUDE.md: what we keep and drop from the global workflow for a job-challenge repo.
Not in scope (v1)
- MVEs (multi-outcome event groupings).
- Non-websocket exchanges.
- Any exchange beyond Polymarket and Kalshi.
- Persistence/replay infrastructure beyond what the SBE output already gives.
Pipeline
The primary thing we debate sprint by sprint (this project's equivalent of the schema debate). As settled in sprints 04 and 05:
rescan fetcher ──findings──▶ ┌──────────────┐ ◀──findings── recent fetcher
│ orchestrator │ ◀── up, down, 1013 ─── connections
└──────┬───▲───┘ ──── recycle ────────▶ connections
Add / Move / Remove (a market and │ │ load reports, each minute
its connections), to its thread │ │ (bytes/s per market); markets
│ │ quiet twice on a connection
┌──────▼───┴───┐
connection 0..N ──events───▶ │ book threads │ ──Snapshot | Delta──▶ writer
(read, stamp, parse, route) │ 0..15 │
▲ └──────┬───────┘
└── subscribe / unsubscribe / resync ┘
- The orchestrator owns what is tracked (each market's two connections, the timers) and decides: placing, moving, splitting, and recycling one connection at a time, about hourly. It never blocks on HTTP.
- The two fetchers only page Gamma on request and stream what they read.
- A connection reads, stamps the receive time, parses, and routes each event to the book thread of its market. Every symbol is on two connections, chosen independently. One silent for 30 s reconnects.
- A book thread owns its markets' books and applies first one wins: a change is a duplicate, the next one, or held until it fits (fold, check, resync, Snapshots as before). It resubscribes a subscription gone quiet on one connection, and judges a reconnect's books of unchanged symbols.
- One writer owns the output.
Settled: OS threads, shared-nothing, no mutexes (ADR 0001); the orchestrator (ADR 0002); two connections per symbol, first one wins (ADR 0003). Details: sprint 04.
Data model, first sketch
Supporting detail (mostly Claude's to settle), serving the pipeline above.
Not a DB — an in-memory domain plus the fixed SBE output schema. Main entities:
- Symbol (from discovery): one order book, with its trading status and any price/tick metadata the exchange gives. On Polymarket that is a token, not a market: a market has two (Yes / No), each with its own, mirrored book.
- NormalizedEvent (internal, exchange-agnostic): the thing each adapter
produces so the core never sees exchange formats. Either a full snapshot or a
single delta, each carrying
symbol,exchange_timestamp,received_at, whatever the exchange offers for loss detection (a sequence number, a book hash), and payload (side/price/quantity, or the full book). - Book (one per symbol): bids and asks as ordered price→quantity maps, plus the last loss-detection marker seen and last exchange timestamp. It is fed by the symbol's two connections, first one wins.
- Connection (one websocket): the set of symbols it carries. Live markets
are placed on connections by
conditionId, both tokens together; Polymarket's lifecycle signal is Gamma (settled in sprint 03: a market is live while active, not closed, with an order book and accepting orders).
Output schema is fixed by the SBE schema: Snapshot and PriceLevelUpdate.
The debate lives in the NormalizedEvent shape (both exchanges must map cleanly onto it). Redundancy was settled in sprint 04: no source is trusted, the first copy of each change wins (ADR 0003).
Open questions
- Loss detection on Kalshi. Its sequence semantics: checked in M6. (Polymarket: settled in M2, by its book hash.)