This repository has been archived on 2026-11-16. You can view files and clone it. You cannot open issues or pull requests or push a commit.
Files
orderbook-collector/docs/project.md
T
pascal adcf205f0b docs: add project definition, roadmap, and evaluation plan
Order-book collector for prediction-market data (Polymarket, then Kalshi). Central bet is the concurrency pipeline; correctness/completeness first, speed later. Roadmap covers base challenge plus all four extensions.
2026-11-04 13:36:36 +01:00

8.7 KiB

Project definition — order-book collector

What it is

A market-data collector for prediction-market order books. It connects to one exchange (Polymarket first, Kalshi second), tracks every live symbol, maintains an in-memory order book per symbol, and emits the repo's SBE-encoded price-level updates plus a full snapshot per symbol every minute — the backtesting feed described in the challenge brief.

For: the Voluntas engineering challenge (and the backtest pipeline it feeds).

The goal (what "done" means for v1)

A running service that, for a chosen exchange:

  • discovers all live regular-market symbols and keeps that set current (subscribe to new, unsubscribe from gone);
  • maintains a correct, complete, time-accurate order book per symbol;
  • emits SBE PriceLevelUpdate deltas continuously and a full Snapshot per symbol every minute, on subscribe, and after any re-snapshot;
  • shuts down cleanly on SIGINT (close sockets, flush/close output).

Full target (all four extensions): redundant websockets, subscription monitoring, a generic exchange-agnostic core with per-exchange adapters for both Polymarket and Kalshi, and latency experiments written up.

The central bet

The concurrency pipeline. The risky choice this project stands on is the topology that moves the firehose from many sockets to a single output sink: concurrently, without dropping a message, without racing the per-symbol books, and without adding latency that ruins the timestamps. The data model (a sorted map per side) is easy; the pipeline is where completeness, time-accuracy, and throughput collide. If the pipeline is wrong, no amount of correct book logic saves the data.

The interacting decisions it is made of:

  • Threading / runtime: threads + channels vs an async runtime. The template already uses std threads + bounded crossbeam channels and pulls in no tokio — a strong hint toward threads + channels. Pivotal sub-decision, settled in the first sprint.
  • Stamping point: in the socket-read path, immediately, so as little as possible sits between the wire and the receive timestamp.
  • Book ownership: shard symbols so each book is owned by exactly one worker (lock-free) rather than a shared, locked map.
  • Backpressure: bounded vs unbounded channels — the core correctness/perf tension. Drop-on-full loses data (violates completeness); block adds latency and fills socket buffers. This decision is the bet in miniature.
  • Output serialization: one writer owns the single sink and consumes from all workers.
  • Redundancy fit: multiple sockets feeding one symbol's worker, which dedups / selects by source + seq — the topology must leave room for this from the start.

Requirements the pipeline must satisfy

The output's value is completeness + time-accuracy, so whatever the topology, it must guarantee:

  1. Receive time stamped at the socket edge, before any queue or processing.
  2. A per-symbol book folded from absolute deltas (overwrite the level; quantity 0 deletes it).
  3. Sequence-gap detection → discard-and-re-snapshot (a gap means the book is silently corrupt; rebuilding is the only safe response).

These are proven first, before redundancy or scale.

Design ownership

The pipeline / concurrency architecture is driven by the user (the central bet, debated sprint by sprint). The data model is mostly delegated to Claude, as a supporting detail that serves the pipeline.

Design principles

Correctness and completeness first; make it fast later. Readiness for the fast version comes from clean structure, not added complexity — the seams we want (thread-per-stage joined by channels, the book as a pure apply(delta) → book function, symbols shardable later) are also the simplest, clearest design.

  • Boring, idiomatic, readable wins. Plain BTreeMap for a price-ordered side, standard channels and threads. No micro-optimization, no bit-packing, no unsafe, no 3am-to-decode data structures in V1.
  • No speculative abstraction. No generic Exchange trait up front — extract it once two exchanges exist (milestone 6). Readiness ≠ pre-building the flexible version.
  • A clean seam is cheap and allowed; a complex implementation behind it is not. A channel, a function boundary, or a trait with two real impls keeps the door open; the clever internals are what we refuse.
  • Never simplified away, even in V1: receive-time stamping at the socket edge, and gap → re-snapshot. These are correctness, not readability tradeoffs.
  • If a corner is genuinely cut for speed later (a global lock, an O(n²) scan), mark it with a comment naming the ceiling and the upgrade path, so the debt is visible rather than silently clever.

Restated in the project CLAUDE.md at the skeleton milestone (M1).

Constraints (already decided)

  • Language/stack: Rust (fixed by the template). Edition 2024.
  • Output format: the repo's SBE helpers (AlignedSbeWriter / SbeEncode), non-negotiable — output must be in the standard format.
  • Build inside repo/ (the challenge template): replace main.rs, add modules under src/. website/ stays separate as the brief.
  • Scope of markets: regular markets only; MVEs excluded; websocket exchanges only.

Workflow differences

This is a job-challenge repo (cloned template, no Gitea/Woodpecker/release pipeline). From the global workflow we keep: docs/, roadmap/, sprint files, feature branches, and scripts/ci.sh (fmt + clippy -D warnings + tests + dependency audit). We drop: Woodpecker pipelines, release.sh, versioning/changelog, rc images, and the prod/dev/release branch flow. Work on feat/* branches squashed into main.

Not in scope (v1)

  • MVEs (multi-outcome event groupings).
  • Non-websocket exchanges.
  • Any exchange beyond Polymarket and Kalshi.
  • Persistence/replay infrastructure beyond what the SBE output already gives.

Pipeline, first sketch

The primary thing we debate sprint by sprint (this project's equivalent of the schema debate). Rough shape:

discovery ──▶ subscription manager ──▶ N socket readers (stamp receive time here)
                                               │
                                   route by symbol
                                               ▼
                         per-symbol book workers (fold, gap-detect, dedup sources)
                                               │
                                               ▼
                                   single output writer (SBE sink)

Each stage and the channels between them are open until settled in a sprint; the async-vs-threads call is the first one.

Data model, first sketch

Supporting detail (mostly Claude's to settle), serving the pipeline above.

Not a DB — an in-memory domain plus the fixed SBE output schema. Main entities:

  • Symbol / market (from discovery): id, trading status, any price/tick metadata the exchange gives.
  • NormalizedEvent (internal, exchange-agnostic): the thing each adapter produces so the core never sees exchange formats. Either a full snapshot or a single delta, each carrying symbol, seq, exchange_timestamp, received_at, and payload (side/price/quantity, or the full book).
  • Book (one per symbol): bids and asks as ordered price→quantity maps, plus last_seq, last exchange timestamp, and the id of the source currently trusted (for redundancy).
  • Connection (one websocket): the set of symbols it carries.
  • Source / redundancy: several connections may feed the same symbol; one is the trusted source at a time.

Output schema is fixed by the SBE schema: Snapshot and PriceLevelUpdate.

The debate lives in two places: the NormalizedEvent shape (both exchanges must map cleanly onto it) and redundancy reconciliation (how a symbol's trusted source is chosen and switched, and emitting a fresh snapshot on switch).

Open questions

  • Output: stdout vs file. Brief says stdout; AlignedSbeWriter takes a file path and writes fixed 64 MiB zero-padded chunks. Likely: write to a file (possibly rotated); resolve in the skeleton milestone. Recommendation: file output, treat "stdout" as loose wording; confirm.
  • Sequence semantics per exchange. Is seq per-symbol or per-connection, and how are snapshots vs deltas sequenced? Differs by exchange; needs doc check per adapter.
  • Symbol lifecycle signal. What marks a market "no longer trading" (drops from the list vs a status field)? Per-exchange, check docs.
  • File rotation policy. When to roll to a new output file (size, time), given 64 MiB chunks. Resolve with the output decision.
  • Redundancy source-selection. How to pick/switch the trusted source (freshness, message rate, seq lead). Designed in the redundancy sprint.