Archived
Sprint 08 (roadmap/08. ready to submit.md), the last milestone. - scripts/watch.sh runs the collector with a live view of its log, stops cleanly on Ctrl-C, then checks the output (reconcile) and shows one symbol's book as text (show, a new binary). - The README for the solution, with the final runs: Polymarket through an outage it recovered from on its own, Kalshi's demo with two slow connections recycled. - From a review of the whole codebase: Kalshi takes RSA keys too (RSA-PSS-SHA256); a slow Kalshi connection takes the hourly recycle's next turn, which it nearly never got before; a market tracked again no longer fails reconcile; Book::add checks for overflow; usage and --help name the key variables; the first scan says how far it got. - Tests for every gap the reviews found, an offline run of the whole orchestrator among them; ShellCheck in CI.
230 lines
12 KiB
Markdown
230 lines
12 KiB
Markdown
# Project definition — order-book collector
|
||
|
||
## What it is
|
||
|
||
A market-data collector for prediction-market order books. It connects to one
|
||
exchange (Polymarket first, Kalshi second), tracks every live symbol, maintains
|
||
an in-memory order book per symbol, and emits the repo's SBE-encoded price-level
|
||
updates plus a full snapshot per symbol every minute — the backtesting feed
|
||
described in the challenge brief.
|
||
|
||
For: the Voluntas engineering challenge (and the backtest pipeline it feeds).
|
||
|
||
## The goal (what "done" means for v1)
|
||
|
||
A running service that, for a chosen exchange:
|
||
|
||
- discovers all live regular-market symbols and keeps that set current
|
||
(subscribe to new, unsubscribe from gone);
|
||
- maintains a correct, complete, time-accurate order book per symbol;
|
||
- emits SBE `PriceLevelUpdate` deltas continuously and a full `Snapshot` per
|
||
symbol every minute, on subscribe, and after any resync;
|
||
- shuts down cleanly on SIGINT (close sockets, flush/close output).
|
||
|
||
Full target (all four extensions): redundant websockets, subscription
|
||
monitoring, a generic exchange-agnostic core with per-exchange adapters for
|
||
**both** Polymarket and Kalshi, and latency experiments written up.
|
||
|
||
## The central bet
|
||
|
||
**The concurrency pipeline.** The risky choice this project stands on is the
|
||
topology that moves the firehose from many sockets to a single output sink:
|
||
concurrently, without dropping a message, without racing the per-symbol books,
|
||
and without adding latency that ruins the timestamps. The data model (a sorted
|
||
map per side) is easy; the pipeline is where completeness, time-accuracy, and
|
||
throughput collide. If the pipeline is wrong, no amount of correct book logic
|
||
saves the data.
|
||
|
||
The interacting decisions it is made of:
|
||
|
||
- **Threading / runtime:** settled in sprint 01 — OS threads + channels,
|
||
shared-nothing; async was re-evaluated at M4 and threads stay
|
||
([ADR 0001](adr/0001-concurrency-model.md)).
|
||
- **Stamping point:** in the socket-read path, immediately, so as little as
|
||
possible sits between the wire and the receive timestamp.
|
||
- **Book ownership:** each book is owned by exactly one of 16 book threads,
|
||
chosen from its market's id; an orchestrator decides which connections
|
||
carry each market ([ADR 0002](adr/0002-lanes-and-orchestrator.md), its
|
||
lanes superseded by [ADR 0003](adr/0003-first-one-wins.md) in sprint 04).
|
||
- **The exchange seam:** settled in sprint 06 — the pipeline is the shared
|
||
core; an exchange supplies its websocket layer and, inside the book
|
||
thread, its loss rule ([ADR 0004](adr/0004-exchange-seam.md)).
|
||
- **Backpressure:** bounded vs unbounded channels — the core correctness/perf
|
||
tension. Drop-on-full loses data (violates completeness); block adds latency
|
||
and fills socket buffers. This decision is the bet in miniature.
|
||
- **Output serialization:** one writer owns the single sink and consumes from
|
||
all workers.
|
||
- **Redundancy:** settled in sprint 04 — every symbol on two connections
|
||
chosen independently, its book taking each change from whichever delivers
|
||
it first ([ADR 0003](adr/0003-first-one-wins.md)).
|
||
|
||
## Requirements the pipeline must satisfy
|
||
|
||
The output's value is completeness + time-accuracy, so whatever the topology, it
|
||
must guarantee:
|
||
|
||
1. **Receive time stamped at the socket edge**, before any queue or processing.
|
||
2. **A per-symbol book folded from the exchange's changes**, written as
|
||
absolute Deltas (the level's new quantity; 0 deletes it). Polymarket's
|
||
changes are already absolute; Kalshi's are relative and are added to the
|
||
level.
|
||
3. **Loss detection → resync** (a lost message means the book is silently
|
||
corrupt; replacing it with a fresh full book is the only safe response). How
|
||
a loss is detected depends on the exchange: on Polymarket, recomputing its
|
||
book hash after every change (settled in M2; the formula is in
|
||
[notes.md](notes.md)); on Kalshi, a skipped sequence number on a
|
||
connection, which reconnects it (settled in M6).
|
||
|
||
These are proven first, before redundancy or scale.
|
||
|
||
## Design ownership
|
||
|
||
The **pipeline / concurrency architecture** is driven by the user (the central
|
||
bet, debated sprint by sprint). The **data model** is mostly delegated to Claude,
|
||
as a supporting detail that serves the pipeline.
|
||
|
||
## Design principles
|
||
|
||
Correctness and completeness first; make it fast later. Readiness for the fast
|
||
version comes from **clean structure, not added complexity** — the seams we want
|
||
(thread-per-stage joined by channels, the book as a pure `apply(delta) → book`
|
||
function, symbols shardable later) are also the simplest, clearest design.
|
||
|
||
- **Boring, idiomatic, readable wins.** Plain `BTreeMap` for a price-ordered
|
||
side, standard channels and threads. No micro-optimization, no bit-packing, no
|
||
`unsafe`, no 3am-to-decode data structures in V1.
|
||
- **No speculative abstraction.** No generic `Exchange` trait up front — extract
|
||
it once two exchanges exist (milestone 6). Readiness ≠ pre-building the
|
||
flexible version.
|
||
- **A clean seam is cheap and allowed; a complex implementation behind it is
|
||
not.** A channel, a function boundary, or a `trait` with two real impls keeps
|
||
the door open; the clever internals are what we refuse.
|
||
- **Never simplified away, even in V1:** receive-time stamping at the socket
|
||
edge, and loss → resync. These are correctness, not readability tradeoffs.
|
||
- **If a corner is genuinely cut for speed later** (a global lock, an O(n²)
|
||
scan), mark it with a comment naming the ceiling and the upgrade path, so the
|
||
debt is visible rather than silently clever.
|
||
|
||
Their operational form, for writing code, is in the project
|
||
[CLAUDE.md](../CLAUDE.md#writing-code).
|
||
|
||
## Constraints (already decided)
|
||
|
||
- **Language/stack:** Rust (fixed by the template). Edition 2024.
|
||
- **Output format:** the repo's SBE helpers (`AlignedSbeWriter` / `SbeEncode`),
|
||
non-negotiable — output must be in the standard format.
|
||
- **Build inside `repo/`** (the challenge template): replace `main.rs`, add
|
||
modules under `src/`. `website/` stays separate as the brief.
|
||
- **Scope of markets:** regular markets only; MVEs excluded; websocket exchanges
|
||
only.
|
||
- **Output destination** (settled in sprints 01 and 02): `AlignedSbeWriter`
|
||
takes a file path; the program takes the exchange (`polymarket`, `kalshi`
|
||
or `kalshi-demo`, required since sprint 06: one exchange per process), then
|
||
that path, default `/dev/stdout`, and logs go to stderr. No rotation: one output per run
|
||
(rotation can't apply to stdout; a consumer that wants files rotates them).
|
||
Chunks must be 64 MiB: `AlignedSbeReader::open` reads nothing else.
|
||
- **Crash ceiling** (accepted in sprint 02): the writer holds the current
|
||
64 MiB chunk in memory until it is full, so a hard crash loses it; flushing
|
||
more often pads each flush to 64 MiB. Kept as a ceiling in extension 1 (M4).
|
||
|
||
## Workflow differences
|
||
|
||
See the project [CLAUDE.md](../CLAUDE.md#workflow-differences): what we keep
|
||
and drop from the global workflow for a job-challenge repo.
|
||
|
||
## Not in scope (v1)
|
||
|
||
- MVEs (multi-outcome event groupings).
|
||
- Non-websocket exchanges.
|
||
- Any exchange beyond Polymarket and Kalshi.
|
||
- Persistence/replay infrastructure beyond what the SBE output already gives.
|
||
|
||
## Pipeline
|
||
|
||
The primary thing we debate sprint by sprint (this project's equivalent of the
|
||
schema debate). As settled in sprints 04 and 05:
|
||
|
||
```
|
||
rescan fetcher ──findings──▶ ┌──────────────┐ ◀──findings── recent fetcher
|
||
│ orchestrator │ ◀── up, down, 1013 ─── connections
|
||
└──────┬───▲───┘ ──── recycle ────────▶ connections
|
||
Add / Move / Remove (a market and │ │ load reports, each minute
|
||
its connections), to its thread │ │ (bytes/s per market); markets
|
||
│ │ quiet twice on a connection;
|
||
│ │ lateness per connection (Kalshi)
|
||
┌──────▼───┴───┐
|
||
connection 0..N ──events───▶ │ book threads │ ──Snapshot | Delta──▶ writer
|
||
(read, stamp, parse, route) │ 0..15 │
|
||
▲ └──────┬───────┘
|
||
└── subscribe / unsubscribe / resync ┘
|
||
```
|
||
|
||
- The **orchestrator** owns what is tracked (each market's two connections,
|
||
the timers) and decides: placing, moving, splitting, and recycling one
|
||
connection at a time, about hourly; a Kalshi connection whose mean
|
||
lateness stays above twice everyone's takes the next turn, in place of
|
||
the oldest (sprints 07 and 08, LATENCY.md). It never blocks on HTTP.
|
||
- The two **fetchers** only page the exchange's market listing on request
|
||
and send on what they read.
|
||
- A **connection** reads, stamps the receive time, parses, and routes each
|
||
event to the book thread of its market. Every symbol is on two connections,
|
||
chosen independently. One silent for 30 s reconnects.
|
||
- A **book thread** owns its markets' books and applies first one wins: a
|
||
change is a duplicate, the next one, or held until it fits (fold, check,
|
||
resync, Snapshots as before). It resubscribes a subscription gone quiet on
|
||
one connection, and judges a reconnect's books of unchanged symbols.
|
||
- One **writer** owns the output.
|
||
|
||
Settled: OS threads, shared-nothing, no mutexes
|
||
([ADR 0001](adr/0001-concurrency-model.md)); the orchestrator
|
||
([ADR 0002](adr/0002-lanes-and-orchestrator.md)); two connections per symbol,
|
||
first one wins ([ADR 0003](adr/0003-first-one-wins.md)). Details:
|
||
[sprint 04](../roadmap/04.%20redundant%20connections.md).
|
||
|
||
The pipeline is the same for every exchange. What an exchange supplies
|
||
([ADR 0004](adr/0004-exchange-seam.md)): the websocket layer of a connection
|
||
(request, encoding of commands, decoding of frames, keepalive), the fetchers,
|
||
and the rule a book thread applies to an event (duplicate, next, held, loss).
|
||
On Polymarket that rule is the hash chain above; on Kalshi a change is the
|
||
next one when its clock and rank pass the book's, and a held change doesn't
|
||
exist.
|
||
|
||
## Data model, first sketch
|
||
|
||
Supporting detail (mostly Claude's to settle), serving the pipeline above.
|
||
|
||
Not a DB — an in-memory domain plus the fixed SBE output schema. Main entities:
|
||
|
||
- **Symbol** (from discovery): one order book, with its trading status and any
|
||
price/tick metadata the exchange gives. On Polymarket that is a token, not a
|
||
market: a market has two (Yes / No), each with its own, mirrored book. On
|
||
Kalshi it is the market: its bids for Yes are the bids, its bids for No at
|
||
`q` the asks at `1 − q`.
|
||
- **Event** (settled in sprint 06, in place of the `NormalizedEvent` first
|
||
sketched): each exchange's own type. The core reads only the market it
|
||
belongs to and whether it is a full book; the exchange's rule reads the
|
||
rest (side, price, quantity, and what the exchange offers for loss
|
||
detection: a book hash, a sequence number, a clock) and folds it into the
|
||
symbol's book, setting a level's quantity (Polymarket) or moving it by a
|
||
signed amount (Kalshi).
|
||
- **Book** (one per symbol): bids and asks as ordered price→quantity maps, plus
|
||
the last loss-detection marker seen and last exchange timestamp. It is fed
|
||
by the symbol's two connections, first one wins.
|
||
- **Connection** (one websocket): the set of symbols it carries. Live markets
|
||
are placed on connections by market, all its symbols together;
|
||
Polymarket's lifecycle signal is Gamma (settled in sprint 03: a market is
|
||
live while active, not closed, with an order book and accepting orders),
|
||
Kalshi's its market listing (sprint 06: open, or opening within three
|
||
hours, tracked ahead so the book is there from its first order).
|
||
|
||
Output schema is fixed by the SBE schema: `Snapshot` and `PriceLevelUpdate`.
|
||
|
||
Redundancy was settled in sprint 04: no source is trusted, the first copy of
|
||
each change wins ([ADR 0003](adr/0003-first-one-wins.md)). On Kalshi the
|
||
first copy is recognised by clock and rank (sprint 06).
|
||
|
||
## Open questions
|
||
|
||
None. (Loss detection on Kalshi was the last: settled in M6, by its
|
||
per-connection sequence number.)
|