This repository has been archived on 2026-11-16. You can view files and clone it. You cannot open issues or pull requests or push a commit.
Files
orderbook-collector/docs/project.md
T
pascal adcf205f0b docs: add project definition, roadmap, and evaluation plan
Order-book collector for prediction-market data (Polymarket, then Kalshi). Central bet is the concurrency pipeline; correctness/completeness first, speed later. Roadmap covers base challenge plus all four extensions.
2026-11-04 13:36:36 +01:00

185 lines
8.7 KiB
Markdown

# Project definition — order-book collector
## What it is
A market-data collector for prediction-market order books. It connects to one
exchange (Polymarket first, Kalshi second), tracks every live symbol, maintains
an in-memory order book per symbol, and emits the repo's SBE-encoded price-level
updates plus a full snapshot per symbol every minute — the backtesting feed
described in the challenge brief.
For: the Voluntas engineering challenge (and the backtest pipeline it feeds).
## The goal (what "done" means for v1)
A running service that, for a chosen exchange:
- discovers all live regular-market symbols and keeps that set current
(subscribe to new, unsubscribe from gone);
- maintains a correct, complete, time-accurate order book per symbol;
- emits SBE `PriceLevelUpdate` deltas continuously and a full `Snapshot` per
symbol every minute, on subscribe, and after any re-snapshot;
- shuts down cleanly on SIGINT (close sockets, flush/close output).
Full target (all four extensions): redundant websockets, subscription
monitoring, a generic exchange-agnostic core with per-exchange adapters for
**both** Polymarket and Kalshi, and latency experiments written up.
## The central bet
**The concurrency pipeline.** The risky choice this project stands on is the
topology that moves the firehose from many sockets to a single output sink:
concurrently, without dropping a message, without racing the per-symbol books,
and without adding latency that ruins the timestamps. The data model (a sorted
map per side) is easy; the pipeline is where completeness, time-accuracy, and
throughput collide. If the pipeline is wrong, no amount of correct book logic
saves the data.
The interacting decisions it is made of:
- **Threading / runtime:** threads + channels vs an async runtime. The template
already uses `std` threads + bounded `crossbeam` channels and pulls in no
tokio — a strong hint toward threads + channels. Pivotal sub-decision, settled
in the first sprint.
- **Stamping point:** in the socket-read path, immediately, so as little as
possible sits between the wire and the receive timestamp.
- **Book ownership:** shard symbols so each book is owned by exactly one worker
(lock-free) rather than a shared, locked map.
- **Backpressure:** bounded vs unbounded channels — the core correctness/perf
tension. Drop-on-full loses data (violates completeness); block adds latency
and fills socket buffers. This decision is the bet in miniature.
- **Output serialization:** one writer owns the single sink and consumes from
all workers.
- **Redundancy fit:** multiple sockets feeding one symbol's worker, which
dedups / selects by source + seq — the topology must leave room for this from
the start.
## Requirements the pipeline must satisfy
The output's value is completeness + time-accuracy, so whatever the topology, it
must guarantee:
1. **Receive time stamped at the socket edge**, before any queue or processing.
2. **A per-symbol book folded from absolute deltas** (overwrite the level;
quantity 0 deletes it).
3. **Sequence-gap detection → discard-and-re-snapshot** (a gap means the book is
silently corrupt; rebuilding is the only safe response).
These are proven first, before redundancy or scale.
## Design ownership
The **pipeline / concurrency architecture** is driven by the user (the central
bet, debated sprint by sprint). The **data model** is mostly delegated to Claude,
as a supporting detail that serves the pipeline.
## Design principles
Correctness and completeness first; make it fast later. Readiness for the fast
version comes from **clean structure, not added complexity** — the seams we want
(thread-per-stage joined by channels, the book as a pure `apply(delta) → book`
function, symbols shardable later) are also the simplest, clearest design.
- **Boring, idiomatic, readable wins.** Plain `BTreeMap` for a price-ordered
side, standard channels and threads. No micro-optimization, no bit-packing, no
`unsafe`, no 3am-to-decode data structures in V1.
- **No speculative abstraction.** No generic `Exchange` trait up front — extract
it once two exchanges exist (milestone 6). Readiness ≠ pre-building the
flexible version.
- **A clean seam is cheap and allowed; a complex implementation behind it is
not.** A channel, a function boundary, or a `trait` with two real impls keeps
the door open; the clever internals are what we refuse.
- **Never simplified away, even in V1:** receive-time stamping at the socket
edge, and gap → re-snapshot. These are correctness, not readability tradeoffs.
- **If a corner is genuinely cut for speed later** (a global lock, an O(n²)
scan), mark it with a comment naming the ceiling and the upgrade path, so the
debt is visible rather than silently clever.
Restated in the project `CLAUDE.md` at the skeleton milestone (M1).
## Constraints (already decided)
- **Language/stack:** Rust (fixed by the template). Edition 2024.
- **Output format:** the repo's SBE helpers (`AlignedSbeWriter` / `SbeEncode`),
non-negotiable — output must be in the standard format.
- **Build inside `repo/`** (the challenge template): replace `main.rs`, add
modules under `src/`. `website/` stays separate as the brief.
- **Scope of markets:** regular markets only; MVEs excluded; websocket exchanges
only.
## Workflow differences
This is a job-challenge repo (cloned template, no Gitea/Woodpecker/release
pipeline). From the global workflow we **keep**: `docs/`, `roadmap/`, sprint
files, feature branches, and `scripts/ci.sh` (fmt + clippy `-D warnings` +
tests + dependency audit). We **drop**: Woodpecker pipelines, `release.sh`,
versioning/changelog, rc images, and the `prod`/`dev`/release branch flow.
Work on `feat/*` branches squashed into `main`.
## Not in scope (v1)
- MVEs (multi-outcome event groupings).
- Non-websocket exchanges.
- Any exchange beyond Polymarket and Kalshi.
- Persistence/replay infrastructure beyond what the SBE output already gives.
## Pipeline, first sketch
The primary thing we debate sprint by sprint (this project's equivalent of the
schema debate). Rough shape:
```
discovery ──▶ subscription manager ──▶ N socket readers (stamp receive time here)
│
route by symbol
▼
per-symbol book workers (fold, gap-detect, dedup sources)
│
▼
single output writer (SBE sink)
```
Each stage and the channels between them are open until settled in a sprint; the
async-vs-threads call is the first one.
## Data model, first sketch
Supporting detail (mostly Claude's to settle), serving the pipeline above.
Not a DB — an in-memory domain plus the fixed SBE output schema. Main entities:
- **Symbol / market** (from discovery): id, trading status, any price/tick
metadata the exchange gives.
- **NormalizedEvent** (internal, exchange-agnostic): the thing each adapter
produces so the core never sees exchange formats. Either a full snapshot or a
single delta, each carrying `symbol`, `seq`, `exchange_timestamp`,
`received_at`, and payload (side/price/quantity, or the full book).
- **Book** (one per symbol): bids and asks as ordered price→quantity maps, plus
`last_seq`, last exchange timestamp, and the id of the source currently
trusted (for redundancy).
- **Connection** (one websocket): the set of symbols it carries.
- **Source / redundancy:** several connections may feed the same symbol; one is
the trusted source at a time.
Output schema is fixed by the SBE schema: `Snapshot` and `PriceLevelUpdate`.
The debate lives in two places: the **NormalizedEvent** shape (both exchanges
must map cleanly onto it) and **redundancy reconciliation** (how a symbol's
trusted source is chosen and switched, and emitting a fresh snapshot on switch).
## Open questions
- **Output: stdout vs file.** Brief says stdout; `AlignedSbeWriter` takes a file
path and writes fixed 64 MiB zero-padded chunks. Likely: write to a file
(possibly rotated); resolve in the skeleton milestone. *Recommendation: file
output, treat "stdout" as loose wording; confirm.*
- **Sequence semantics per exchange.** Is `seq` per-symbol or per-connection, and
how are snapshots vs deltas sequenced? Differs by exchange; needs doc check per
adapter.
- **Symbol lifecycle signal.** What marks a market "no longer trading" (drops
from the list vs a status field)? Per-exchange, check docs.
- **File rotation policy.** When to roll to a new output file (size, time), given
64 MiB chunks. Resolve with the output decision.
- **Redundancy source-selection.** How to pick/switch the trusted source
(freshness, message rate, seq lead). Designed in the redundancy sprint.