From adcf205f0be2901d16946df71cf89a2a945eeafb Mon Sep 17 00:00:00 2001 From: Pascal Perrenoud Date: Wed, 4 Nov 2026 13:36:36 +0100 Subject: [PATCH] docs: add project definition, roadmap, and evaluation plan Order-book collector for prediction-market data (Polymarket, then Kalshi). Central bet is the concurrency pipeline; correctness/completeness first, speed later. Roadmap covers base challenge plus all four extensions. --- docs/evaluation.md | 57 ++++++++++++++ docs/notes.md | 13 ++++ docs/project.md | 184 +++++++++++++++++++++++++++++++++++++++++++++ docs/roadmap.md | 83 ++++++++++++++++++++ 4 files changed, 337 insertions(+) create mode 100644 docs/evaluation.md create mode 100644 docs/notes.md create mode 100644 docs/project.md create mode 100644 docs/roadmap.md diff --git a/docs/evaluation.md b/docs/evaluation.md new file mode 100644 index 0000000..9bcba04 --- /dev/null +++ b/docs/evaluation.md @@ -0,0 +1,57 @@ +# Evaluation + +How we judge the finished software. This is a **post-completion evaluation**, not +a metric we develop against. Correctness (soundness/completeness tests), +benchmarks, and profiling drive development; this is the "does the data have +financial value" story told afterward. + +## The idea: a trading simulator on top of the collector + +Write a *separate* program that consumes our collector's output as its source of +truth, makes trading decisions, runs for a while, and reports P&L — a tangible +demonstration of the "faster/more complete makes more money" claim. + +## The trap, and how to design around it + +A trading sim measures the **strategy**, not the collector. Absolute P&L is +mostly "was the bot any good," which is noise for our purposes. + +**Fix: make it differential — same strategy, two feeds.** Run the identical bot +on (a) our full feed and (b) a deliberately degraded copy, and compare the P&L +*difference*. That delta is attributable to data quality, which is what the +challenge claims matters. + +The two claims split apart: + +- **Completeness is cleanly measurable on recorded data.** Treat our output as + ground truth, make a lossy copy (drop a % of messages, skip some + re-snapshots), run the same strategy on both, compare P&L. The gap = cost of + dropped data. No live trading needed. +- **Latency is NOT measurable from recorded data.** Collection latency is baked + into a recording; a backtest can't show "we'd have been a few ms faster." A + dollar figure on latency needs a live/forward A/B (two collectors, one slower, + same strategy in real time, or injected delay). Much bigger apparatus. + +**The fill model is the hard part.** A naive "see best ask → buy → book profit" +bot overstates P&L massively: it ignores queue position, the level vanishing +before the order arrives, latency to the exchange, and market impact. A credible +fill model for Kalshi/Polymarket microstructure is arguably bigger than the +challenge itself. Scope it carefully or not at all. + +## Cheaper metrics that target the claims directly + +Prefer these; they can't be gamed by strategy luck: + +- **Time-accuracy:** distribution of `received_at − exchange_timestamp`, and + separately our *internal* added latency (edge-stamp → output byte). +- **Completeness:** reconciliation error rate (snapshot↔delta divergence); count + of missed deletes / stale levels under induced loss. +- **Redundancy value (once ext1 exists):** messages the backup socket caught that + the primary missed — literal "data loss avoided." + +## Recommendation + +Keep the trading-sim as the narrative. For an actual P&L demo, build only the +**differential completeness** version (full vs lossy, recorded data, same +strategy) — the one variant that's both doable and meaningful. Treat the full +live-latency-P&L rig as out of scope. Revisit at the polish milestone. diff --git a/docs/notes.md b/docs/notes.md new file mode 100644 index 0000000..bff365d --- /dev/null +++ b/docs/notes.md @@ -0,0 +1,13 @@ +# Challenge notes + +## Deltas are absolute, not incremental +A delta carries the **new total quantity** at a price level, not a change to it. +`quantity: 400` means "this level is now 400," not "+400". Overwrite the level; +never add. `quantity: 0` means delete the level. (Schema confirms: "absolute +available quantity".) + +## Sequence numbers are the thread you must not drop +Each symbol's messages carry a monotonically increasing `seq`. If you receive a +gap (e.g. 101 then 103), you missed a message, so your in-memory book is now +wrong. You cannot patch it — throw the book away and re-snapshot that symbol. +This is the "completeness" requirement made mechanical. diff --git a/docs/project.md b/docs/project.md new file mode 100644 index 0000000..486f210 --- /dev/null +++ b/docs/project.md @@ -0,0 +1,184 @@ +# Project definition — order-book collector + +## What it is + +A market-data collector for prediction-market order books. It connects to one +exchange (Polymarket first, Kalshi second), tracks every live symbol, maintains +an in-memory order book per symbol, and emits the repo's SBE-encoded price-level +updates plus a full snapshot per symbol every minute — the backtesting feed +described in the challenge brief. + +For: the Voluntas engineering challenge (and the backtest pipeline it feeds). + +## The goal (what "done" means for v1) + +A running service that, for a chosen exchange: + +- discovers all live regular-market symbols and keeps that set current + (subscribe to new, unsubscribe from gone); +- maintains a correct, complete, time-accurate order book per symbol; +- emits SBE `PriceLevelUpdate` deltas continuously and a full `Snapshot` per + symbol every minute, on subscribe, and after any re-snapshot; +- shuts down cleanly on SIGINT (close sockets, flush/close output). + +Full target (all four extensions): redundant websockets, subscription +monitoring, a generic exchange-agnostic core with per-exchange adapters for +**both** Polymarket and Kalshi, and latency experiments written up. + +## The central bet + +**The concurrency pipeline.** The risky choice this project stands on is the +topology that moves the firehose from many sockets to a single output sink: +concurrently, without dropping a message, without racing the per-symbol books, +and without adding latency that ruins the timestamps. The data model (a sorted +map per side) is easy; the pipeline is where completeness, time-accuracy, and +throughput collide. If the pipeline is wrong, no amount of correct book logic +saves the data. + +The interacting decisions it is made of: + +- **Threading / runtime:** threads + channels vs an async runtime. The template + already uses `std` threads + bounded `crossbeam` channels and pulls in no + tokio — a strong hint toward threads + channels. Pivotal sub-decision, settled + in the first sprint. +- **Stamping point:** in the socket-read path, immediately, so as little as + possible sits between the wire and the receive timestamp. +- **Book ownership:** shard symbols so each book is owned by exactly one worker + (lock-free) rather than a shared, locked map. +- **Backpressure:** bounded vs unbounded channels — the core correctness/perf + tension. Drop-on-full loses data (violates completeness); block adds latency + and fills socket buffers. This decision is the bet in miniature. +- **Output serialization:** one writer owns the single sink and consumes from + all workers. +- **Redundancy fit:** multiple sockets feeding one symbol's worker, which + dedups / selects by source + seq — the topology must leave room for this from + the start. + +## Requirements the pipeline must satisfy + +The output's value is completeness + time-accuracy, so whatever the topology, it +must guarantee: + +1. **Receive time stamped at the socket edge**, before any queue or processing. +2. **A per-symbol book folded from absolute deltas** (overwrite the level; + quantity 0 deletes it). +3. **Sequence-gap detection → discard-and-re-snapshot** (a gap means the book is + silently corrupt; rebuilding is the only safe response). + +These are proven first, before redundancy or scale. + +## Design ownership + +The **pipeline / concurrency architecture** is driven by the user (the central +bet, debated sprint by sprint). The **data model** is mostly delegated to Claude, +as a supporting detail that serves the pipeline. + +## Design principles + +Correctness and completeness first; make it fast later. Readiness for the fast +version comes from **clean structure, not added complexity** — the seams we want +(thread-per-stage joined by channels, the book as a pure `apply(delta) → book` +function, symbols shardable later) are also the simplest, clearest design. + +- **Boring, idiomatic, readable wins.** Plain `BTreeMap` for a price-ordered + side, standard channels and threads. No micro-optimization, no bit-packing, no + `unsafe`, no 3am-to-decode data structures in V1. +- **No speculative abstraction.** No generic `Exchange` trait up front — extract + it once two exchanges exist (milestone 6). Readiness ≠ pre-building the + flexible version. +- **A clean seam is cheap and allowed; a complex implementation behind it is + not.** A channel, a function boundary, or a `trait` with two real impls keeps + the door open; the clever internals are what we refuse. +- **Never simplified away, even in V1:** receive-time stamping at the socket + edge, and gap → re-snapshot. These are correctness, not readability tradeoffs. +- **If a corner is genuinely cut for speed later** (a global lock, an O(n²) + scan), mark it with a comment naming the ceiling and the upgrade path, so the + debt is visible rather than silently clever. + +Restated in the project `CLAUDE.md` at the skeleton milestone (M1). + +## Constraints (already decided) + +- **Language/stack:** Rust (fixed by the template). Edition 2024. +- **Output format:** the repo's SBE helpers (`AlignedSbeWriter` / `SbeEncode`), + non-negotiable — output must be in the standard format. +- **Build inside `repo/`** (the challenge template): replace `main.rs`, add + modules under `src/`. `website/` stays separate as the brief. +- **Scope of markets:** regular markets only; MVEs excluded; websocket exchanges + only. + +## Workflow differences + +This is a job-challenge repo (cloned template, no Gitea/Woodpecker/release +pipeline). From the global workflow we **keep**: `docs/`, `roadmap/`, sprint +files, feature branches, and `scripts/ci.sh` (fmt + clippy `-D warnings` + +tests + dependency audit). We **drop**: Woodpecker pipelines, `release.sh`, +versioning/changelog, rc images, and the `prod`/`dev`/release branch flow. +Work on `feat/*` branches squashed into `main`. + +## Not in scope (v1) + +- MVEs (multi-outcome event groupings). +- Non-websocket exchanges. +- Any exchange beyond Polymarket and Kalshi. +- Persistence/replay infrastructure beyond what the SBE output already gives. + +## Pipeline, first sketch + +The primary thing we debate sprint by sprint (this project's equivalent of the +schema debate). Rough shape: + +``` +discovery ──▶ subscription manager ──▶ N socket readers (stamp receive time here) + │ + route by symbol + ▼ + per-symbol book workers (fold, gap-detect, dedup sources) + │ + ▼ + single output writer (SBE sink) +``` + +Each stage and the channels between them are open until settled in a sprint; the +async-vs-threads call is the first one. + +## Data model, first sketch + +Supporting detail (mostly Claude's to settle), serving the pipeline above. + +Not a DB — an in-memory domain plus the fixed SBE output schema. Main entities: + +- **Symbol / market** (from discovery): id, trading status, any price/tick + metadata the exchange gives. +- **NormalizedEvent** (internal, exchange-agnostic): the thing each adapter + produces so the core never sees exchange formats. Either a full snapshot or a + single delta, each carrying `symbol`, `seq`, `exchange_timestamp`, + `received_at`, and payload (side/price/quantity, or the full book). +- **Book** (one per symbol): bids and asks as ordered price→quantity maps, plus + `last_seq`, last exchange timestamp, and the id of the source currently + trusted (for redundancy). +- **Connection** (one websocket): the set of symbols it carries. +- **Source / redundancy:** several connections may feed the same symbol; one is + the trusted source at a time. + +Output schema is fixed by the SBE schema: `Snapshot` and `PriceLevelUpdate`. + +The debate lives in two places: the **NormalizedEvent** shape (both exchanges +must map cleanly onto it) and **redundancy reconciliation** (how a symbol's +trusted source is chosen and switched, and emitting a fresh snapshot on switch). + +## Open questions + +- **Output: stdout vs file.** Brief says stdout; `AlignedSbeWriter` takes a file + path and writes fixed 64 MiB zero-padded chunks. Likely: write to a file + (possibly rotated); resolve in the skeleton milestone. *Recommendation: file + output, treat "stdout" as loose wording; confirm.* +- **Sequence semantics per exchange.** Is `seq` per-symbol or per-connection, and + how are snapshots vs deltas sequenced? Differs by exchange; needs doc check per + adapter. +- **Symbol lifecycle signal.** What marks a market "no longer trading" (drops + from the list vs a status field)? Per-exchange, check docs. +- **File rotation policy.** When to roll to a new output file (size, time), given + 64 MiB chunks. Resolve with the output decision. +- **Redundancy source-selection.** How to pick/switch the trusted source + (freshness, message rate, seq lead). Designed in the redundancy sprint. diff --git a/docs/roadmap.md b/docs/roadmap.md new file mode 100644 index 0000000..7d5172c --- /dev/null +++ b/docs/roadmap.md @@ -0,0 +1,83 @@ +# Roadmap + +Goal: the order-book collector defined in [project.md](project.md) — a +correct, complete, time-accurate feed for prediction-market order books, +covering all four challenge extensions across Polymarket and Kalshi. + +Ordering principle: **prove the central bet (the pipeline) and one exchange +first, so base + Polymarket is already a complete, submittable deliverable.** +Everything after is additive; nothing risky sits at the end. The generic +abstraction is *extracted* once a second exchange exists, not designed up front. + +## Milestones + +1. **Skeleton + connectivity** — *not started* + Project scaffolding inside the template: module layout (replace `main.rs`), + `scripts/ci.sh` (fmt + clippy `-D warnings` + tests + audit), project + CLAUDE.md, chosen websocket/HTTP libraries. Settle the pivotal + async-vs-threads call. Resolve the output destination (stdout vs file). + Connect to Polymarket and log raw messages. + *Ends:* CI green; raw Polymarket messages flowing. + +2. **Single-source book pipeline (Polymarket, subset of symbols)** — *not started* + The central bet in its first real form: normalize events → per-symbol book + (fold absolute deltas) → gap-detect → re-snapshot → SBE output (deltas + + snapshot every minute, on subscribe, on re-snapshot) → receive-time stamping + → clean SIGINT. One/few sockets, a fixed subset of symbols. Deterministic + replay tests + snapshot↔delta reconciliation. + *Ends:* correct SBE output for a symbol subset; tests green. + +3. **Full coverage: discovery + sharding (base challenge complete)** — *not started* + REST discovery loop with diffing (subscribe new, unsubscribe gone); + load-balance all live regular-market symbols across N sockets. + *Ends:* every live Polymarket symbol tracked — base challenge satisfied for + one exchange. + +4. **Extension 1 — redundant websockets** — *not started* + Multiple connections per symbol; choose a trusted source and switch on + loss/quiet/delay; emit a fresh snapshot on every source switch. + *Ends:* redundancy demonstrably reduces data loss; source-switch re-snapshots. + +5. **Extension 2 — subscription monitoring** — *not started* + Per-symbol activity monitoring (data points per second/minute); unsubscribe + + resubscribe symbols that go too quiet; periodic full-socket recycle + (disconnect → reconnect → resubscribe) to shed cumulative exchange-side bugs. + *Ends:* monitoring + recycling running. + +6. **Extension 3 — generic core + Kalshi adapter** — *not started* + Extract the exchange-agnostic core (the `NormalizedEvent` seam) from the + working Polymarket code, then add a Kalshi adapter so both exchanges run + through the same core. The abstraction is extracted from two concretes here, + not before. Requires a Kalshi API key. + *Ends:* both Polymarket and Kalshi work through one generic core. + +7. **Extension 4 — latency experiments + writeup** — *not started* + Experiment with faster connections (first-after-maintenance, connection + racing, source-port tricks) on both exchanges; document findings in a + markdown file. + *Ends:* experiments documented. + +8. **Polish + submission** — *not started* + Final review, README for the solution, packaging/handoff for the challenge. + *Ends:* ready to submit. + +## Why this order + +- Milestones 1–3 prove the pipeline (the central bet) and finish the base + challenge on one exchange — the riskiest and most valuable work, up front. +- Extensions 1 and 2 harden that single exchange before we generalize, so the + generic core (6) is extracted from code that already does the hard things. +- The generic abstraction waits for the second exchange (Kalshi), so it's + extracted from two real implementations rather than guessed from one. +- Latency (4) is experimentation, least coupled to correctness, so it's last + before polish. + +## Possible features (beyond v1) + +- More exchanges beyond Polymarket and Kalshi. +- MVE / multi-outcome market support. +- Non-websocket exchange support. +- Persistence / replay tooling beyond the raw SBE output. +- Observability: metrics, structured logging, dashboards. +- Config-driven symbol filtering and per-exchange tuning. +- Containerized packaging.