Archived
Every live market is subscribed on two connections chosen at random, and its book takes each change from whichever connection delivers it first: the book hash says whether a change is a duplicate, the next one, or one to hold until the change before it arrives. A change one connection misses costs nothing, each Delta carries the earliest receive time, and a connection that drops is covered by the other. When a change fits on no connection in time, the book follows one connection unchecked and asks the other for a fresh book. Connections now decode and route events to 16 book threads, independent of them; a move leaves its old connection only once the new one delivers, so it costs no change; a connection closed with 1013 twice within ten minutes is split. reconcile reads the collector's log and fails only on a mismatch no cause line explains. SIGTERM and SIGHUP stop the collector cleanly. Breaking: lanes are gone (connections and book threads are separate), and reconcile takes the log as a second argument.
203 lines
10 KiB
Markdown
203 lines
10 KiB
Markdown
# Project definition — order-book collector
|
|
|
|
## What it is
|
|
|
|
A market-data collector for prediction-market order books. It connects to one
|
|
exchange (Polymarket first, Kalshi second), tracks every live symbol, maintains
|
|
an in-memory order book per symbol, and emits the repo's SBE-encoded price-level
|
|
updates plus a full snapshot per symbol every minute — the backtesting feed
|
|
described in the challenge brief.
|
|
|
|
For: the Voluntas engineering challenge (and the backtest pipeline it feeds).
|
|
|
|
## The goal (what "done" means for v1)
|
|
|
|
A running service that, for a chosen exchange:
|
|
|
|
- discovers all live regular-market symbols and keeps that set current
|
|
(subscribe to new, unsubscribe from gone);
|
|
- maintains a correct, complete, time-accurate order book per symbol;
|
|
- emits SBE `PriceLevelUpdate` deltas continuously and a full `Snapshot` per
|
|
symbol every minute, on subscribe, and after any resync;
|
|
- shuts down cleanly on SIGINT (close sockets, flush/close output).
|
|
|
|
Full target (all four extensions): redundant websockets, subscription
|
|
monitoring, a generic exchange-agnostic core with per-exchange adapters for
|
|
**both** Polymarket and Kalshi, and latency experiments written up.
|
|
|
|
## The central bet
|
|
|
|
**The concurrency pipeline.** The risky choice this project stands on is the
|
|
topology that moves the firehose from many sockets to a single output sink:
|
|
concurrently, without dropping a message, without racing the per-symbol books,
|
|
and without adding latency that ruins the timestamps. The data model (a sorted
|
|
map per side) is easy; the pipeline is where completeness, time-accuracy, and
|
|
throughput collide. If the pipeline is wrong, no amount of correct book logic
|
|
saves the data.
|
|
|
|
The interacting decisions it is made of:
|
|
|
|
- **Threading / runtime:** settled in sprint 01 — OS threads + channels,
|
|
shared-nothing; async was re-evaluated at M4 and threads stay
|
|
([ADR 0001](adr/0001-concurrency-model.md)).
|
|
- **Stamping point:** in the socket-read path, immediately, so as little as
|
|
possible sits between the wire and the receive timestamp.
|
|
- **Book ownership:** each book is owned by exactly one of 16 book threads,
|
|
chosen from its market's id; an orchestrator decides which connections
|
|
carry each market ([ADR 0002](adr/0002-lanes-and-orchestrator.md), its
|
|
lanes superseded by [ADR 0003](adr/0003-first-one-wins.md) in sprint 04).
|
|
- **Backpressure:** bounded vs unbounded channels — the core correctness/perf
|
|
tension. Drop-on-full loses data (violates completeness); block adds latency
|
|
and fills socket buffers. This decision is the bet in miniature.
|
|
- **Output serialization:** one writer owns the single sink and consumes from
|
|
all workers.
|
|
- **Redundancy:** settled in sprint 04 — every symbol on two connections
|
|
chosen independently, its book taking each change from whichever delivers
|
|
it first ([ADR 0003](adr/0003-first-one-wins.md)).
|
|
|
|
## Requirements the pipeline must satisfy
|
|
|
|
The output's value is completeness + time-accuracy, so whatever the topology, it
|
|
must guarantee:
|
|
|
|
1. **Receive time stamped at the socket edge**, before any queue or processing.
|
|
2. **A per-symbol book folded from absolute deltas** (overwrite the level;
|
|
quantity 0 deletes it).
|
|
3. **Loss detection → resync** (a lost message means the book is silently
|
|
corrupt; replacing it with a fresh full book is the only safe response). How
|
|
a loss is detected depends on the exchange: a sequence gap where one exists;
|
|
on Polymarket, which has none, recomputing its book hash after every change
|
|
(settled in M2; the formula is in [notes.md](notes.md)).
|
|
|
|
These are proven first, before redundancy or scale.
|
|
|
|
## Design ownership
|
|
|
|
The **pipeline / concurrency architecture** is driven by the user (the central
|
|
bet, debated sprint by sprint). The **data model** is mostly delegated to Claude,
|
|
as a supporting detail that serves the pipeline.
|
|
|
|
## Design principles
|
|
|
|
Correctness and completeness first; make it fast later. Readiness for the fast
|
|
version comes from **clean structure, not added complexity** — the seams we want
|
|
(thread-per-stage joined by channels, the book as a pure `apply(delta) → book`
|
|
function, symbols shardable later) are also the simplest, clearest design.
|
|
|
|
- **Boring, idiomatic, readable wins.** Plain `BTreeMap` for a price-ordered
|
|
side, standard channels and threads. No micro-optimization, no bit-packing, no
|
|
`unsafe`, no 3am-to-decode data structures in V1.
|
|
- **No speculative abstraction.** No generic `Exchange` trait up front — extract
|
|
it once two exchanges exist (milestone 6). Readiness ≠ pre-building the
|
|
flexible version.
|
|
- **A clean seam is cheap and allowed; a complex implementation behind it is
|
|
not.** A channel, a function boundary, or a `trait` with two real impls keeps
|
|
the door open; the clever internals are what we refuse.
|
|
- **Never simplified away, even in V1:** receive-time stamping at the socket
|
|
edge, and loss → resync. These are correctness, not readability tradeoffs.
|
|
- **If a corner is genuinely cut for speed later** (a global lock, an O(n²)
|
|
scan), mark it with a comment naming the ceiling and the upgrade path, so the
|
|
debt is visible rather than silently clever.
|
|
|
|
Their operational form, for writing code, is in the project
|
|
[CLAUDE.md](../CLAUDE.md#writing-code).
|
|
|
|
## Constraints (already decided)
|
|
|
|
- **Language/stack:** Rust (fixed by the template). Edition 2024.
|
|
- **Output format:** the repo's SBE helpers (`AlignedSbeWriter` / `SbeEncode`),
|
|
non-negotiable — output must be in the standard format.
|
|
- **Build inside `repo/`** (the challenge template): replace `main.rs`, add
|
|
modules under `src/`. `website/` stays separate as the brief.
|
|
- **Scope of markets:** regular markets only; MVEs excluded; websocket exchanges
|
|
only.
|
|
- **Output destination** (settled in sprints 01 and 02): `AlignedSbeWriter`
|
|
takes a file path; the program's only argument is that path, default
|
|
`/dev/stdout`, and logs go to stderr. No rotation: one output per run
|
|
(rotation can't apply to stdout; a consumer that wants files rotates them).
|
|
Chunks must be 64 MiB: `AlignedSbeReader::open` reads nothing else.
|
|
- **Crash ceiling** (accepted in sprint 02): the writer holds the current
|
|
64 MiB chunk in memory until it is full, so a hard crash loses it; flushing
|
|
more often pads each flush to 64 MiB. Kept as a ceiling in extension 1 (M4).
|
|
|
|
## Workflow differences
|
|
|
|
See the project [CLAUDE.md](../CLAUDE.md#workflow-differences): what we keep
|
|
and drop from the global workflow for a job-challenge repo.
|
|
|
|
## Not in scope (v1)
|
|
|
|
- MVEs (multi-outcome event groupings).
|
|
- Non-websocket exchanges.
|
|
- Any exchange beyond Polymarket and Kalshi.
|
|
- Persistence/replay infrastructure beyond what the SBE output already gives.
|
|
|
|
## Pipeline
|
|
|
|
The primary thing we debate sprint by sprint (this project's equivalent of the
|
|
schema debate). As settled in sprint 04:
|
|
|
|
```
|
|
rescan fetcher ──findings──▶ ┌──────────────┐ ◀──findings── recent fetcher
|
|
│ orchestrator │ ◀── 1013 closes ── connections
|
|
└──────┬───▲───┘
|
|
Add / Move / Remove (a market and │ │ load reports, each minute
|
|
its connections), to its thread │ │ (bytes/s per market)
|
|
┌──────▼───┴───┐
|
|
connection 0..N ──events───▶ │ book threads │ ──Snapshot | Delta──▶ writer
|
|
(read, stamp, parse, route) │ 0..15 │
|
|
▲ └──────┬───────┘
|
|
└── subscribe / unsubscribe / resync ┘
|
|
```
|
|
|
|
- The **orchestrator** owns what is tracked (each market's two connections,
|
|
the timers) and decides: placing, moving, splitting. It never blocks on
|
|
HTTP.
|
|
- The two **fetchers** only page Gamma on request and stream what they read.
|
|
- A **connection** reads, stamps the receive time, parses, and routes each
|
|
event to the book thread of its market. Every symbol is on two connections,
|
|
chosen independently.
|
|
- A **book thread** owns its markets' books and applies first one wins: a
|
|
change is a duplicate, the next one, or held until it fits (fold, check,
|
|
resync, Snapshots as before).
|
|
- One **writer** owns the output.
|
|
|
|
Settled: OS threads, shared-nothing, no mutexes
|
|
([ADR 0001](adr/0001-concurrency-model.md)); the orchestrator
|
|
([ADR 0002](adr/0002-lanes-and-orchestrator.md)); two connections per symbol,
|
|
first one wins ([ADR 0003](adr/0003-first-one-wins.md)). Details:
|
|
[sprint 04](../roadmap/04.%20redundant%20connections.md).
|
|
|
|
## Data model, first sketch
|
|
|
|
Supporting detail (mostly Claude's to settle), serving the pipeline above.
|
|
|
|
Not a DB — an in-memory domain plus the fixed SBE output schema. Main entities:
|
|
|
|
- **Symbol** (from discovery): one order book, with its trading status and any
|
|
price/tick metadata the exchange gives. On Polymarket that is a token, not a
|
|
market: a market has two (Yes / No), each with its own, mirrored book.
|
|
- **NormalizedEvent** (internal, exchange-agnostic): the thing each adapter
|
|
produces so the core never sees exchange formats. Either a full snapshot or a
|
|
single delta, each carrying `symbol`, `exchange_timestamp`, `received_at`,
|
|
whatever the exchange offers for loss detection (a sequence number, a book
|
|
hash), and payload (side/price/quantity, or the full book).
|
|
- **Book** (one per symbol): bids and asks as ordered price→quantity maps, plus
|
|
the last loss-detection marker seen and last exchange timestamp. It is fed
|
|
by the symbol's two connections, first one wins.
|
|
- **Connection** (one websocket): the set of symbols it carries. Live markets
|
|
are placed on connections by `conditionId`, both tokens together;
|
|
Polymarket's lifecycle signal is Gamma (settled in sprint 03: a market is
|
|
live while active, not closed, with an order book and accepting orders).
|
|
|
|
Output schema is fixed by the SBE schema: `Snapshot` and `PriceLevelUpdate`.
|
|
|
|
The debate lives in the **NormalizedEvent** shape (both exchanges must map
|
|
cleanly onto it). Redundancy was settled in sprint 04: no source is trusted,
|
|
the first copy of each change wins ([ADR 0003](adr/0003-first-one-wins.md)).
|
|
|
|
## Open questions
|
|
|
|
- **Loss detection on Kalshi.** Its sequence semantics: checked in M6.
|
|
(Polymarket: settled in M2, by its book hash.)
|