Archived
An orchestrator keeps the tracked set equal to Gamma's live markets (a scan at startup and every hour, new and closed markets every minute, reaching back 15 minutes for Gamma's indexing delay) and spreads them over lanes, each a connection and a book thread, all writing through the one writer. Lanes report each market's load in bytes per second; the orchestrator puts new markets on the least-loaded lane and moves the busiest markets off lanes over a 300 KB/s budget, busiest lane first. Differing books our own subscribe explains are subscription gaps, not losses, and every first book after a move is logged, so reconcile's mismatches are all accounted for in the log. A 75-minute live run tracked ~195k markets (~390k symbols) in 89 lanes.
207 lines
10 KiB
Markdown
207 lines
10 KiB
Markdown
# Project definition — order-book collector
|
|
|
|
## What it is
|
|
|
|
A market-data collector for prediction-market order books. It connects to one
|
|
exchange (Polymarket first, Kalshi second), tracks every live symbol, maintains
|
|
an in-memory order book per symbol, and emits the repo's SBE-encoded price-level
|
|
updates plus a full snapshot per symbol every minute — the backtesting feed
|
|
described in the challenge brief.
|
|
|
|
For: the Voluntas engineering challenge (and the backtest pipeline it feeds).
|
|
|
|
## The goal (what "done" means for v1)
|
|
|
|
A running service that, for a chosen exchange:
|
|
|
|
- discovers all live regular-market symbols and keeps that set current
|
|
(subscribe to new, unsubscribe from gone);
|
|
- maintains a correct, complete, time-accurate order book per symbol;
|
|
- emits SBE `PriceLevelUpdate` deltas continuously and a full `Snapshot` per
|
|
symbol every minute, on subscribe, and after any resync;
|
|
- shuts down cleanly on SIGINT (close sockets, flush/close output).
|
|
|
|
Full target (all four extensions): redundant websockets, subscription
|
|
monitoring, a generic exchange-agnostic core with per-exchange adapters for
|
|
**both** Polymarket and Kalshi, and latency experiments written up.
|
|
|
|
## The central bet
|
|
|
|
**The concurrency pipeline.** The risky choice this project stands on is the
|
|
topology that moves the firehose from many sockets to a single output sink:
|
|
concurrently, without dropping a message, without racing the per-symbol books,
|
|
and without adding latency that ruins the timestamps. The data model (a sorted
|
|
map per side) is easy; the pipeline is where completeness, time-accuracy, and
|
|
throughput collide. If the pipeline is wrong, no amount of correct book logic
|
|
saves the data.
|
|
|
|
The interacting decisions it is made of:
|
|
|
|
- **Threading / runtime:** settled in sprint 01 — OS threads + channels,
|
|
shared-nothing, async re-evaluated at M4
|
|
([ADR 0001](adr/0001-concurrency-model.md)).
|
|
- **Stamping point:** in the socket-read path, immediately, so as little as
|
|
possible sits between the wire and the receive timestamp.
|
|
- **Book ownership:** settled in sprint 03 — symbols are spread over lanes,
|
|
each a connection and a book thread of its own, so each book is owned by
|
|
exactly one thread; an orchestrator assigns them
|
|
([ADR 0002](adr/0002-lanes-and-orchestrator.md)).
|
|
- **Backpressure:** bounded vs unbounded channels — the core correctness/perf
|
|
tension. Drop-on-full loses data (violates completeness); block adds latency
|
|
and fills socket buffers. This decision is the bet in miniature.
|
|
- **Output serialization:** one writer owns the single sink and consumes from
|
|
all workers.
|
|
- **Redundancy fit:** multiple sockets feeding one symbol's worker, which
|
|
dedups / selects by source — the topology must leave room for this from the
|
|
start.
|
|
|
|
## Requirements the pipeline must satisfy
|
|
|
|
The output's value is completeness + time-accuracy, so whatever the topology, it
|
|
must guarantee:
|
|
|
|
1. **Receive time stamped at the socket edge**, before any queue or processing.
|
|
2. **A per-symbol book folded from absolute deltas** (overwrite the level;
|
|
quantity 0 deletes it).
|
|
3. **Loss detection → resync** (a lost message means the book is silently
|
|
corrupt; replacing it with a fresh full book is the only safe response). How
|
|
a loss is detected depends on the exchange: a sequence gap where one exists;
|
|
on Polymarket, which has none, recomputing its book hash after every change
|
|
(settled in M2; the formula is in [notes.md](notes.md)).
|
|
|
|
These are proven first, before redundancy or scale.
|
|
|
|
## Design ownership
|
|
|
|
The **pipeline / concurrency architecture** is driven by the user (the central
|
|
bet, debated sprint by sprint). The **data model** is mostly delegated to Claude,
|
|
as a supporting detail that serves the pipeline.
|
|
|
|
## Design principles
|
|
|
|
Correctness and completeness first; make it fast later. Readiness for the fast
|
|
version comes from **clean structure, not added complexity** — the seams we want
|
|
(thread-per-stage joined by channels, the book as a pure `apply(delta) → book`
|
|
function, symbols shardable later) are also the simplest, clearest design.
|
|
|
|
- **Boring, idiomatic, readable wins.** Plain `BTreeMap` for a price-ordered
|
|
side, standard channels and threads. No micro-optimization, no bit-packing, no
|
|
`unsafe`, no 3am-to-decode data structures in V1.
|
|
- **No speculative abstraction.** No generic `Exchange` trait up front — extract
|
|
it once two exchanges exist (milestone 6). Readiness ≠ pre-building the
|
|
flexible version.
|
|
- **A clean seam is cheap and allowed; a complex implementation behind it is
|
|
not.** A channel, a function boundary, or a `trait` with two real impls keeps
|
|
the door open; the clever internals are what we refuse.
|
|
- **Never simplified away, even in V1:** receive-time stamping at the socket
|
|
edge, and loss → resync. These are correctness, not readability tradeoffs.
|
|
- **If a corner is genuinely cut for speed later** (a global lock, an O(n²)
|
|
scan), mark it with a comment naming the ceiling and the upgrade path, so the
|
|
debt is visible rather than silently clever.
|
|
|
|
Their operational form, for writing code, is in the project
|
|
[CLAUDE.md](../CLAUDE.md#writing-code).
|
|
|
|
## Constraints (already decided)
|
|
|
|
- **Language/stack:** Rust (fixed by the template). Edition 2024.
|
|
- **Output format:** the repo's SBE helpers (`AlignedSbeWriter` / `SbeEncode`),
|
|
non-negotiable — output must be in the standard format.
|
|
- **Build inside `repo/`** (the challenge template): replace `main.rs`, add
|
|
modules under `src/`. `website/` stays separate as the brief.
|
|
- **Scope of markets:** regular markets only; MVEs excluded; websocket exchanges
|
|
only.
|
|
- **Output destination** (settled in sprints 01 and 02): `AlignedSbeWriter`
|
|
takes a file path; the program's only argument is that path, default
|
|
`/dev/stdout`, and logs go to stderr. No rotation: one output per run
|
|
(rotation can't apply to stdout; a consumer that wants files rotates them).
|
|
Chunks must be 64 MiB: `AlignedSbeReader::open` reads nothing else.
|
|
- **Crash ceiling** (accepted in sprint 02): the writer holds the current
|
|
64 MiB chunk in memory until it is full, so a hard crash loses it; flushing
|
|
more often pads each flush to 64 MiB. Taken up with extension 1 (M4).
|
|
|
|
## Workflow differences
|
|
|
|
See the project [CLAUDE.md](../CLAUDE.md#workflow-differences): what we keep
|
|
and drop from the global workflow for a job-challenge repo.
|
|
|
|
## Not in scope (v1)
|
|
|
|
- MVEs (multi-outcome event groupings).
|
|
- Non-websocket exchanges.
|
|
- Any exchange beyond Polymarket and Kalshi.
|
|
- Persistence/replay infrastructure beyond what the SBE output already gives.
|
|
|
|
## Pipeline
|
|
|
|
The primary thing we debate sprint by sprint (this project's equivalent of the
|
|
schema debate). As settled in sprint 03:
|
|
|
|
```
|
|
rescan fetcher ──findings──▶ ┌──────────────┐ ◀──findings── recent fetcher
|
|
(full Gamma scan) ◀─request─│ orchestrator │──request──▶ (new + closed since T)
|
|
└──┬────────▲──┘
|
|
Add / MoveIn / Remove, to │ │ load reports, each minute
|
|
the market's lane │ │ (bytes/s per market)
|
|
┌──────────────────────┼────────┴─────────────────┐
|
|
▼ ▼ ▼
|
|
lane 0: book thread ◀─frames─ connection lane 1: ... lane N: ...
|
|
│ └─subscribe / unsubscribe / resync─▶┘
|
|
└──────────── Snapshot | Delta ─────────────▶ writer thread (one)
|
|
```
|
|
|
|
- The **orchestrator** owns what is tracked (market → lane, the lanes, the
|
|
timers) and decides; it never blocks on HTTP, so later coordination
|
|
(switching sources in M4, recycling in M5) lives there too.
|
|
- The two **fetchers** only page Gamma on request and stream what they read.
|
|
- Each **lane** is a connection (stamps receive time, sends subscriptions and
|
|
resyncs) and a book thread (fold, check, resync, Snapshots); M4 gives a lane
|
|
several connections and has its book thread choose the trusted source.
|
|
- One **writer** owns the output.
|
|
|
|
Settled: OS threads, shared-nothing, no mutexes
|
|
([ADR 0001](adr/0001-concurrency-model.md)); lanes and the orchestrator
|
|
([ADR 0002](adr/0002-lanes-and-orchestrator.md)). Details:
|
|
[sprint 03](../roadmap/03.%20full%20coverage.md).
|
|
|
|
## Data model, first sketch
|
|
|
|
Supporting detail (mostly Claude's to settle), serving the pipeline above.
|
|
|
|
Not a DB — an in-memory domain plus the fixed SBE output schema. Main entities:
|
|
|
|
- **Symbol** (from discovery): one order book, with its trading status and any
|
|
price/tick metadata the exchange gives. On Polymarket that is a token, not a
|
|
market: a market has two (Yes / No), each with its own, mirrored book.
|
|
- **NormalizedEvent** (internal, exchange-agnostic): the thing each adapter
|
|
produces so the core never sees exchange formats. Either a full snapshot or a
|
|
single delta, each carrying `symbol`, `exchange_timestamp`, `received_at`,
|
|
whatever the exchange offers for loss detection (a sequence number, a book
|
|
hash), and payload (side/price/quantity, or the full book).
|
|
- **Book** (one per symbol): bids and asks as ordered price→quantity maps, plus
|
|
the last loss-detection marker seen, last exchange timestamp, and the id of the
|
|
source currently trusted (for redundancy).
|
|
- **Connection** (one websocket): the set of symbols it carries, those of its
|
|
lane.
|
|
- **Lane**: a set of symbols, the book thread that owns their books, and the
|
|
connections that feed it. Live markets are assigned to lanes by
|
|
`conditionId`; Polymarket's lifecycle signal is Gamma (settled in sprint 03:
|
|
a market is live while active, not closed, with an order book and accepting
|
|
orders).
|
|
- **Source / redundancy:** several connections may feed the same symbol; one is
|
|
the trusted source at a time.
|
|
|
|
Output schema is fixed by the SBE schema: `Snapshot` and `PriceLevelUpdate`.
|
|
|
|
The debate lives in two places: the **NormalizedEvent** shape (both exchanges
|
|
must map cleanly onto it) and **redundancy reconciliation** (how a symbol's
|
|
trusted source is chosen and switched, and emitting a fresh snapshot on switch).
|
|
|
|
## Open questions
|
|
|
|
- **Loss detection on Kalshi.** Its sequence semantics: checked in M6.
|
|
(Polymarket: settled in M2, by its book hash.)
|
|
- **Redundancy source-selection.** How to pick/switch the trusted source
|
|
(freshness, message rate, which source is ahead). Designed in the redundancy
|
|
sprint.
|