Archived
Replaces the template's demo main.rs with a spike that connects to Polymarket's public market channel, subscribes to one market, and logs every frame with its receive time. Adds scripts/ci.sh (fmt, clippy, tests), the project CLAUDE.md, ADR 0001 (OS threads, shared-nothing), and the market channel's message shapes in docs/notes.md: no sequence number, a book hash on every change, mirrored Yes/No books.
189 lines
8.9 KiB
Markdown
189 lines
8.9 KiB
Markdown
# Project definition — order-book collector
|
|
|
|
## What it is
|
|
|
|
A market-data collector for prediction-market order books. It connects to one
|
|
exchange (Polymarket first, Kalshi second), tracks every live symbol, maintains
|
|
an in-memory order book per symbol, and emits the repo's SBE-encoded price-level
|
|
updates plus a full snapshot per symbol every minute — the backtesting feed
|
|
described in the challenge brief.
|
|
|
|
For: the Voluntas engineering challenge (and the backtest pipeline it feeds).
|
|
|
|
## The goal (what "done" means for v1)
|
|
|
|
A running service that, for a chosen exchange:
|
|
|
|
- discovers all live regular-market symbols and keeps that set current
|
|
(subscribe to new, unsubscribe from gone);
|
|
- maintains a correct, complete, time-accurate order book per symbol;
|
|
- emits SBE `PriceLevelUpdate` deltas continuously and a full `Snapshot` per
|
|
symbol every minute, on subscribe, and after any re-snapshot;
|
|
- shuts down cleanly on SIGINT (close sockets, flush/close output).
|
|
|
|
Full target (all four extensions): redundant websockets, subscription
|
|
monitoring, a generic exchange-agnostic core with per-exchange adapters for
|
|
**both** Polymarket and Kalshi, and latency experiments written up.
|
|
|
|
## The central bet
|
|
|
|
**The concurrency pipeline.** The risky choice this project stands on is the
|
|
topology that moves the firehose from many sockets to a single output sink:
|
|
concurrently, without dropping a message, without racing the per-symbol books,
|
|
and without adding latency that ruins the timestamps. The data model (a sorted
|
|
map per side) is easy; the pipeline is where completeness, time-accuracy, and
|
|
throughput collide. If the pipeline is wrong, no amount of correct book logic
|
|
saves the data.
|
|
|
|
The interacting decisions it is made of:
|
|
|
|
- **Threading / runtime:** settled in sprint 01 — OS threads + channels,
|
|
shared-nothing, async re-evaluated at M4
|
|
([ADR 0001](adr/0001-concurrency-model.md)).
|
|
- **Stamping point:** in the socket-read path, immediately, so as little as
|
|
possible sits between the wire and the receive timestamp.
|
|
- **Book ownership:** shard symbols so each book is owned by exactly one worker
|
|
(lock-free) rather than a shared, locked map.
|
|
- **Backpressure:** bounded vs unbounded channels — the core correctness/perf
|
|
tension. Drop-on-full loses data (violates completeness); block adds latency
|
|
and fills socket buffers. This decision is the bet in miniature.
|
|
- **Output serialization:** one writer owns the single sink and consumes from
|
|
all workers.
|
|
- **Redundancy fit:** multiple sockets feeding one symbol's worker, which
|
|
dedups / selects by source — the topology must leave room for this from the
|
|
start.
|
|
|
|
## Requirements the pipeline must satisfy
|
|
|
|
The output's value is completeness + time-accuracy, so whatever the topology, it
|
|
must guarantee:
|
|
|
|
1. **Receive time stamped at the socket edge**, before any queue or processing.
|
|
2. **A per-symbol book folded from absolute deltas** (overwrite the level;
|
|
quantity 0 deletes it).
|
|
3. **Loss detection → discard-and-re-snapshot** (a lost message means the book
|
|
is silently corrupt; rebuilding is the only safe response). How a loss is
|
|
detected depends on the exchange: a sequence gap where one exists; Polymarket
|
|
has none, so its `hash` or best bid/ask instead
|
|
([notes.md](notes.md)) — settled in M2.
|
|
|
|
These are proven first, before redundancy or scale.
|
|
|
|
## Design ownership
|
|
|
|
The **pipeline / concurrency architecture** is driven by the user (the central
|
|
bet, debated sprint by sprint). The **data model** is mostly delegated to Claude,
|
|
as a supporting detail that serves the pipeline.
|
|
|
|
## Design principles
|
|
|
|
Correctness and completeness first; make it fast later. Readiness for the fast
|
|
version comes from **clean structure, not added complexity** — the seams we want
|
|
(thread-per-stage joined by channels, the book as a pure `apply(delta) → book`
|
|
function, symbols shardable later) are also the simplest, clearest design.
|
|
|
|
- **Boring, idiomatic, readable wins.** Plain `BTreeMap` for a price-ordered
|
|
side, standard channels and threads. No micro-optimization, no bit-packing, no
|
|
`unsafe`, no 3am-to-decode data structures in V1.
|
|
- **No speculative abstraction.** No generic `Exchange` trait up front — extract
|
|
it once two exchanges exist (milestone 6). Readiness ≠ pre-building the
|
|
flexible version.
|
|
- **A clean seam is cheap and allowed; a complex implementation behind it is
|
|
not.** A channel, a function boundary, or a `trait` with two real impls keeps
|
|
the door open; the clever internals are what we refuse.
|
|
- **Never simplified away, even in V1:** receive-time stamping at the socket
|
|
edge, and loss → re-snapshot. These are correctness, not readability tradeoffs.
|
|
- **If a corner is genuinely cut for speed later** (a global lock, an O(n²)
|
|
scan), mark it with a comment naming the ceiling and the upgrade path, so the
|
|
debt is visible rather than silently clever.
|
|
|
|
Their operational form, for writing code, is in the project
|
|
[CLAUDE.md](../CLAUDE.md#writing-code).
|
|
|
|
## Constraints (already decided)
|
|
|
|
- **Language/stack:** Rust (fixed by the template). Edition 2024.
|
|
- **Output format:** the repo's SBE helpers (`AlignedSbeWriter` / `SbeEncode`),
|
|
non-negotiable — output must be in the standard format.
|
|
- **Build inside `repo/`** (the challenge template): replace `main.rs`, add
|
|
modules under `src/`. `website/` stays separate as the brief.
|
|
- **Scope of markets:** regular markets only; MVEs excluded; websocket exchanges
|
|
only.
|
|
- **Output destination** (settled in sprint 01): `AlignedSbeWriter` takes a file
|
|
path, and stdout is just the path `/dev/stdout` (or a tiny writer tweak), so
|
|
one code path covers both. Chunks must be 64 MiB: `AlignedSbeReader::open`
|
|
reads nothing else.
|
|
|
|
## Workflow differences
|
|
|
|
See the project [CLAUDE.md](../CLAUDE.md#workflow-differences): what we keep
|
|
and drop from the global workflow for a job-challenge repo.
|
|
|
|
## Not in scope (v1)
|
|
|
|
- MVEs (multi-outcome event groupings).
|
|
- Non-websocket exchanges.
|
|
- Any exchange beyond Polymarket and Kalshi.
|
|
- Persistence/replay infrastructure beyond what the SBE output already gives.
|
|
|
|
## Pipeline, first sketch
|
|
|
|
The primary thing we debate sprint by sprint (this project's equivalent of the
|
|
schema debate). Rough shape:
|
|
|
|
```
|
|
discovery ──▶ subscription manager ──▶ N socket readers (stamp receive time here)
|
|
│
|
|
route by symbol
|
|
▼
|
|
per-symbol book workers (fold, loss-detect, dedup sources)
|
|
│
|
|
▼
|
|
single output writer (SBE sink)
|
|
```
|
|
|
|
Each stage and the channels between them are open until settled in a sprint.
|
|
Settled so far: OS threads, shared-nothing, no mutexes
|
|
([ADR 0001](adr/0001-concurrency-model.md)).
|
|
|
|
## Data model, first sketch
|
|
|
|
Supporting detail (mostly Claude's to settle), serving the pipeline above.
|
|
|
|
Not a DB — an in-memory domain plus the fixed SBE output schema. Main entities:
|
|
|
|
- **Symbol** (from discovery): one order book, with its trading status and any
|
|
price/tick metadata the exchange gives. On Polymarket that is a token, not a
|
|
market: a market has two (Yes / No), each with its own, mirrored book.
|
|
- **NormalizedEvent** (internal, exchange-agnostic): the thing each adapter
|
|
produces so the core never sees exchange formats. Either a full snapshot or a
|
|
single delta, each carrying `symbol`, `exchange_timestamp`, `received_at`,
|
|
whatever the exchange offers for loss detection (a sequence number, a book
|
|
hash), and payload (side/price/quantity, or the full book).
|
|
- **Book** (one per symbol): bids and asks as ordered price→quantity maps, plus
|
|
the last loss-detection marker seen, last exchange timestamp, and the id of the
|
|
source currently trusted (for redundancy).
|
|
- **Connection** (one websocket): the set of symbols it carries.
|
|
- **Source / redundancy:** several connections may feed the same symbol; one is
|
|
the trusted source at a time.
|
|
|
|
Output schema is fixed by the SBE schema: `Snapshot` and `PriceLevelUpdate`.
|
|
|
|
The debate lives in two places: the **NormalizedEvent** shape (both exchanges
|
|
must map cleanly onto it) and **redundancy reconciliation** (how a symbol's
|
|
trusted source is chosen and switched, and emitting a fresh snapshot on switch).
|
|
|
|
## Open questions
|
|
|
|
- **Loss detection per exchange.** Polymarket has no sequence number: can we
|
|
reproduce its book `hash`, or do we check best bid/ask after each change? M2's
|
|
opening question. Kalshi's sequence semantics: checked in M6.
|
|
- **Symbol lifecycle signal.** What marks a market "no longer trading"?
|
|
Polymarket's `new_market` / `market_resolved` events (with
|
|
`custom_feature_enabled`) are candidates; settle in M3.
|
|
- **File rotation policy.** When to roll to a new output file (size, time), given
|
|
64 MiB chunks. Resolve in M2 with the output writer.
|
|
- **Redundancy source-selection.** How to pick/switch the trusted source
|
|
(freshness, message rate, which source is ahead). Designed in the redundancy
|
|
sprint.
|