This repository has been archived on 2026-11-16. You can view files and clone it. You cannot open issues or pull requests or push a commit.
Files
orderbook-collector/docs/project.md
T
pascal cf6c16c9cf feat: ready to submit, a run to watch, the README, the review's fixes
Sprint 08 (roadmap/08. ready to submit.md), the last milestone.

- scripts/watch.sh runs the collector with a live view of its log, stops
  cleanly on Ctrl-C, then checks the output (reconcile) and shows one
  symbol's book as text (show, a new binary).
- The README for the solution, with the final runs: Polymarket through an
  outage it recovered from on its own, Kalshi's demo with two slow
  connections recycled.
- From a review of the whole codebase: Kalshi takes RSA keys too
  (RSA-PSS-SHA256); a slow Kalshi connection takes the hourly recycle's
  next turn, which it nearly never got before; a market tracked again no
  longer fails reconcile; Book::add checks for overflow; usage and --help
  name the key variables; the first scan says how far it got.
- Tests for every gap the reviews found, an offline run of the whole
  orchestrator among them; ShellCheck in CI.
2026-11-12 22:32:39 +01:00

230 lines
12 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Project definition — order-book collector
## What it is
A market-data collector for prediction-market order books. It connects to one
exchange (Polymarket first, Kalshi second), tracks every live symbol, maintains
an in-memory order book per symbol, and emits the repo's SBE-encoded price-level
updates plus a full snapshot per symbol every minute — the backtesting feed
described in the challenge brief.
For: the Voluntas engineering challenge (and the backtest pipeline it feeds).
## The goal (what "done" means for v1)
A running service that, for a chosen exchange:
- discovers all live regular-market symbols and keeps that set current
(subscribe to new, unsubscribe from gone);
- maintains a correct, complete, time-accurate order book per symbol;
- emits SBE `PriceLevelUpdate` deltas continuously and a full `Snapshot` per
symbol every minute, on subscribe, and after any resync;
- shuts down cleanly on SIGINT (close sockets, flush/close output).
Full target (all four extensions): redundant websockets, subscription
monitoring, a generic exchange-agnostic core with per-exchange adapters for
**both** Polymarket and Kalshi, and latency experiments written up.
## The central bet
**The concurrency pipeline.** The risky choice this project stands on is the
topology that moves the firehose from many sockets to a single output sink:
concurrently, without dropping a message, without racing the per-symbol books,
and without adding latency that ruins the timestamps. The data model (a sorted
map per side) is easy; the pipeline is where completeness, time-accuracy, and
throughput collide. If the pipeline is wrong, no amount of correct book logic
saves the data.
The interacting decisions it is made of:
- **Threading / runtime:** settled in sprint 01 — OS threads + channels,
shared-nothing; async was re-evaluated at M4 and threads stay
([ADR 0001](adr/0001-concurrency-model.md)).
- **Stamping point:** in the socket-read path, immediately, so as little as
possible sits between the wire and the receive timestamp.
- **Book ownership:** each book is owned by exactly one of 16 book threads,
chosen from its market's id; an orchestrator decides which connections
carry each market ([ADR 0002](adr/0002-lanes-and-orchestrator.md), its
lanes superseded by [ADR 0003](adr/0003-first-one-wins.md) in sprint 04).
- **The exchange seam:** settled in sprint 06 — the pipeline is the shared
core; an exchange supplies its websocket layer and, inside the book
thread, its loss rule ([ADR 0004](adr/0004-exchange-seam.md)).
- **Backpressure:** bounded vs unbounded channels — the core correctness/perf
tension. Drop-on-full loses data (violates completeness); block adds latency
and fills socket buffers. This decision is the bet in miniature.
- **Output serialization:** one writer owns the single sink and consumes from
all workers.
- **Redundancy:** settled in sprint 04 — every symbol on two connections
chosen independently, its book taking each change from whichever delivers
it first ([ADR 0003](adr/0003-first-one-wins.md)).
## Requirements the pipeline must satisfy
The output's value is completeness + time-accuracy, so whatever the topology, it
must guarantee:
1. **Receive time stamped at the socket edge**, before any queue or processing.
2. **A per-symbol book folded from the exchange's changes**, written as
absolute Deltas (the level's new quantity; 0 deletes it). Polymarket's
changes are already absolute; Kalshi's are relative and are added to the
level.
3. **Loss detection → resync** (a lost message means the book is silently
corrupt; replacing it with a fresh full book is the only safe response). How
a loss is detected depends on the exchange: on Polymarket, recomputing its
book hash after every change (settled in M2; the formula is in
[notes.md](notes.md)); on Kalshi, a skipped sequence number on a
connection, which reconnects it (settled in M6).
These are proven first, before redundancy or scale.
## Design ownership
The **pipeline / concurrency architecture** is driven by the user (the central
bet, debated sprint by sprint). The **data model** is mostly delegated to Claude,
as a supporting detail that serves the pipeline.
## Design principles
Correctness and completeness first; make it fast later. Readiness for the fast
version comes from **clean structure, not added complexity** — the seams we want
(thread-per-stage joined by channels, the book as a pure `apply(delta) → book`
function, symbols shardable later) are also the simplest, clearest design.
- **Boring, idiomatic, readable wins.** Plain `BTreeMap` for a price-ordered
side, standard channels and threads. No micro-optimization, no bit-packing, no
`unsafe`, no 3am-to-decode data structures in V1.
- **No speculative abstraction.** No generic `Exchange` trait up front — extract
it once two exchanges exist (milestone 6). Readiness ≠ pre-building the
flexible version.
- **A clean seam is cheap and allowed; a complex implementation behind it is
not.** A channel, a function boundary, or a `trait` with two real impls keeps
the door open; the clever internals are what we refuse.
- **Never simplified away, even in V1:** receive-time stamping at the socket
edge, and loss → resync. These are correctness, not readability tradeoffs.
- **If a corner is genuinely cut for speed later** (a global lock, an O(n²)
scan), mark it with a comment naming the ceiling and the upgrade path, so the
debt is visible rather than silently clever.
Their operational form, for writing code, is in the project
[CLAUDE.md](../CLAUDE.md#writing-code).
## Constraints (already decided)
- **Language/stack:** Rust (fixed by the template). Edition 2024.
- **Output format:** the repo's SBE helpers (`AlignedSbeWriter` / `SbeEncode`),
non-negotiable — output must be in the standard format.
- **Build inside `repo/`** (the challenge template): replace `main.rs`, add
modules under `src/`. `website/` stays separate as the brief.
- **Scope of markets:** regular markets only; MVEs excluded; websocket exchanges
only.
- **Output destination** (settled in sprints 01 and 02): `AlignedSbeWriter`
takes a file path; the program takes the exchange (`polymarket`, `kalshi`
or `kalshi-demo`, required since sprint 06: one exchange per process), then
that path, default `/dev/stdout`, and logs go to stderr. No rotation: one output per run
(rotation can't apply to stdout; a consumer that wants files rotates them).
Chunks must be 64 MiB: `AlignedSbeReader::open` reads nothing else.
- **Crash ceiling** (accepted in sprint 02): the writer holds the current
64 MiB chunk in memory until it is full, so a hard crash loses it; flushing
more often pads each flush to 64 MiB. Kept as a ceiling in extension 1 (M4).
## Workflow differences
See the project [CLAUDE.md](../CLAUDE.md#workflow-differences): what we keep
and drop from the global workflow for a job-challenge repo.
## Not in scope (v1)
- MVEs (multi-outcome event groupings).
- Non-websocket exchanges.
- Any exchange beyond Polymarket and Kalshi.
- Persistence/replay infrastructure beyond what the SBE output already gives.
## Pipeline
The primary thing we debate sprint by sprint (this project's equivalent of the
schema debate). As settled in sprints 04 and 05:
```
rescan fetcher ──findings──▶ ┌──────────────┐ ◀──findings── recent fetcher
│ orchestrator │ ◀── up, down, 1013 ─── connections
└──────┬───▲───┘ ──── recycle ────────▶ connections
Add / Move / Remove (a market and │ │ load reports, each minute
its connections), to its thread │ │ (bytes/s per market); markets
│ │ quiet twice on a connection;
│ │ lateness per connection (Kalshi)
┌──────▼───┴───┐
connection 0..N ──events───▶ │ book threads │ ──Snapshot | Delta──▶ writer
(read, stamp, parse, route) │ 0..15 │
▲ └──────┬───────┘
└── subscribe / unsubscribe / resync ┘
```
- The **orchestrator** owns what is tracked (each market's two connections,
the timers) and decides: placing, moving, splitting, and recycling one
connection at a time, about hourly; a Kalshi connection whose mean
lateness stays above twice everyone's takes the next turn, in place of
the oldest (sprints 07 and 08, LATENCY.md). It never blocks on HTTP.
- The two **fetchers** only page the exchange's market listing on request
and send on what they read.
- A **connection** reads, stamps the receive time, parses, and routes each
event to the book thread of its market. Every symbol is on two connections,
chosen independently. One silent for 30 s reconnects.
- A **book thread** owns its markets' books and applies first one wins: a
change is a duplicate, the next one, or held until it fits (fold, check,
resync, Snapshots as before). It resubscribes a subscription gone quiet on
one connection, and judges a reconnect's books of unchanged symbols.
- One **writer** owns the output.
Settled: OS threads, shared-nothing, no mutexes
([ADR 0001](adr/0001-concurrency-model.md)); the orchestrator
([ADR 0002](adr/0002-lanes-and-orchestrator.md)); two connections per symbol,
first one wins ([ADR 0003](adr/0003-first-one-wins.md)). Details:
[sprint 04](../roadmap/04.%20redundant%20connections.md).
The pipeline is the same for every exchange. What an exchange supplies
([ADR 0004](adr/0004-exchange-seam.md)): the websocket layer of a connection
(request, encoding of commands, decoding of frames, keepalive), the fetchers,
and the rule a book thread applies to an event (duplicate, next, held, loss).
On Polymarket that rule is the hash chain above; on Kalshi a change is the
next one when its clock and rank pass the book's, and a held change doesn't
exist.
## Data model, first sketch
Supporting detail (mostly Claude's to settle), serving the pipeline above.
Not a DB — an in-memory domain plus the fixed SBE output schema. Main entities:
- **Symbol** (from discovery): one order book, with its trading status and any
price/tick metadata the exchange gives. On Polymarket that is a token, not a
market: a market has two (Yes / No), each with its own, mirrored book. On
Kalshi it is the market: its bids for Yes are the bids, its bids for No at
`q` the asks at `1 − q`.
- **Event** (settled in sprint 06, in place of the `NormalizedEvent` first
sketched): each exchange's own type. The core reads only the market it
belongs to and whether it is a full book; the exchange's rule reads the
rest (side, price, quantity, and what the exchange offers for loss
detection: a book hash, a sequence number, a clock) and folds it into the
symbol's book, setting a level's quantity (Polymarket) or moving it by a
signed amount (Kalshi).
- **Book** (one per symbol): bids and asks as ordered price→quantity maps, plus
the last loss-detection marker seen and last exchange timestamp. It is fed
by the symbol's two connections, first one wins.
- **Connection** (one websocket): the set of symbols it carries. Live markets
are placed on connections by market, all its symbols together;
Polymarket's lifecycle signal is Gamma (settled in sprint 03: a market is
live while active, not closed, with an order book and accepting orders),
Kalshi's its market listing (sprint 06: open, or opening within three
hours, tracked ahead so the book is there from its first order).
Output schema is fixed by the SBE schema: `Snapshot` and `PriceLevelUpdate`.
Redundancy was settled in sprint 04: no source is trusted, the first copy of
each change wins ([ADR 0003](adr/0003-first-one-wins.md)). On Kalshi the
first copy is recognised by clock and rank (sprint 06).
## Open questions
None. (Loss detection on Kalshi was the last: settled in M6, by its
per-connection sequence number.)