Archived
docs: add project definition, roadmap, and evaluation plan
Order-book collector for prediction-market data (Polymarket, then Kalshi). Central bet is the concurrency pipeline; correctness/completeness first, speed later. Roadmap covers base challenge plus all four extensions.
This commit is contained in:
4 files changed
+337
No files matched your search
@@ -0,0 +1,57 @@
|
||||
# Evaluation
|
||||
|
||||
How we judge the finished software. This is a **post-completion evaluation**, not
|
||||
a metric we develop against. Correctness (soundness/completeness tests),
|
||||
benchmarks, and profiling drive development; this is the "does the data have
|
||||
financial value" story told afterward.
|
||||
|
||||
## The idea: a trading simulator on top of the collector
|
||||
|
||||
Write a *separate* program that consumes our collector's output as its source of
|
||||
truth, makes trading decisions, runs for a while, and reports P&L — a tangible
|
||||
demonstration of the "faster/more complete makes more money" claim.
|
||||
|
||||
## The trap, and how to design around it
|
||||
|
||||
A trading sim measures the **strategy**, not the collector. Absolute P&L is
|
||||
mostly "was the bot any good," which is noise for our purposes.
|
||||
|
||||
**Fix: make it differential — same strategy, two feeds.** Run the identical bot
|
||||
on (a) our full feed and (b) a deliberately degraded copy, and compare the P&L
|
||||
*difference*. That delta is attributable to data quality, which is what the
|
||||
challenge claims matters.
|
||||
|
||||
The two claims split apart:
|
||||
|
||||
- **Completeness is cleanly measurable on recorded data.** Treat our output as
|
||||
ground truth, make a lossy copy (drop a % of messages, skip some
|
||||
re-snapshots), run the same strategy on both, compare P&L. The gap = cost of
|
||||
dropped data. No live trading needed.
|
||||
- **Latency is NOT measurable from recorded data.** Collection latency is baked
|
||||
into a recording; a backtest can't show "we'd have been a few ms faster." A
|
||||
dollar figure on latency needs a live/forward A/B (two collectors, one slower,
|
||||
same strategy in real time, or injected delay). Much bigger apparatus.
|
||||
|
||||
**The fill model is the hard part.** A naive "see best ask → buy → book profit"
|
||||
bot overstates P&L massively: it ignores queue position, the level vanishing
|
||||
before the order arrives, latency to the exchange, and market impact. A credible
|
||||
fill model for Kalshi/Polymarket microstructure is arguably bigger than the
|
||||
challenge itself. Scope it carefully or not at all.
|
||||
|
||||
## Cheaper metrics that target the claims directly
|
||||
|
||||
Prefer these; they can't be gamed by strategy luck:
|
||||
|
||||
- **Time-accuracy:** distribution of `received_at − exchange_timestamp`, and
|
||||
separately our *internal* added latency (edge-stamp → output byte).
|
||||
- **Completeness:** reconciliation error rate (snapshot↔delta divergence); count
|
||||
of missed deletes / stale levels under induced loss.
|
||||
- **Redundancy value (once ext1 exists):** messages the backup socket caught that
|
||||
the primary missed — literal "data loss avoided."
|
||||
|
||||
## Recommendation
|
||||
|
||||
Keep the trading-sim as the narrative. For an actual P&L demo, build only the
|
||||
**differential completeness** version (full vs lossy, recorded data, same
|
||||
strategy) — the one variant that's both doable and meaningful. Treat the full
|
||||
live-latency-P&L rig as out of scope. Revisit at the polish milestone.
|
||||
@@ -0,0 +1,13 @@
|
||||
# Challenge notes
|
||||
|
||||
## Deltas are absolute, not incremental
|
||||
A delta carries the **new total quantity** at a price level, not a change to it.
|
||||
`quantity: 400` means "this level is now 400," not "+400". Overwrite the level;
|
||||
never add. `quantity: 0` means delete the level. (Schema confirms: "absolute
|
||||
available quantity".)
|
||||
|
||||
## Sequence numbers are the thread you must not drop
|
||||
Each symbol's messages carry a monotonically increasing `seq`. If you receive a
|
||||
gap (e.g. 101 then 103), you missed a message, so your in-memory book is now
|
||||
wrong. You cannot patch it — throw the book away and re-snapshot that symbol.
|
||||
This is the "completeness" requirement made mechanical.
|
||||
+184
@@ -0,0 +1,184 @@
|
||||
# Project definition — order-book collector
|
||||
|
||||
## What it is
|
||||
|
||||
A market-data collector for prediction-market order books. It connects to one
|
||||
exchange (Polymarket first, Kalshi second), tracks every live symbol, maintains
|
||||
an in-memory order book per symbol, and emits the repo's SBE-encoded price-level
|
||||
updates plus a full snapshot per symbol every minute — the backtesting feed
|
||||
described in the challenge brief.
|
||||
|
||||
For: the Voluntas engineering challenge (and the backtest pipeline it feeds).
|
||||
|
||||
## The goal (what "done" means for v1)
|
||||
|
||||
A running service that, for a chosen exchange:
|
||||
|
||||
- discovers all live regular-market symbols and keeps that set current
|
||||
(subscribe to new, unsubscribe from gone);
|
||||
- maintains a correct, complete, time-accurate order book per symbol;
|
||||
- emits SBE `PriceLevelUpdate` deltas continuously and a full `Snapshot` per
|
||||
symbol every minute, on subscribe, and after any re-snapshot;
|
||||
- shuts down cleanly on SIGINT (close sockets, flush/close output).
|
||||
|
||||
Full target (all four extensions): redundant websockets, subscription
|
||||
monitoring, a generic exchange-agnostic core with per-exchange adapters for
|
||||
**both** Polymarket and Kalshi, and latency experiments written up.
|
||||
|
||||
## The central bet
|
||||
|
||||
**The concurrency pipeline.** The risky choice this project stands on is the
|
||||
topology that moves the firehose from many sockets to a single output sink:
|
||||
concurrently, without dropping a message, without racing the per-symbol books,
|
||||
and without adding latency that ruins the timestamps. The data model (a sorted
|
||||
map per side) is easy; the pipeline is where completeness, time-accuracy, and
|
||||
throughput collide. If the pipeline is wrong, no amount of correct book logic
|
||||
saves the data.
|
||||
|
||||
The interacting decisions it is made of:
|
||||
|
||||
- **Threading / runtime:** threads + channels vs an async runtime. The template
|
||||
already uses `std` threads + bounded `crossbeam` channels and pulls in no
|
||||
tokio — a strong hint toward threads + channels. Pivotal sub-decision, settled
|
||||
in the first sprint.
|
||||
- **Stamping point:** in the socket-read path, immediately, so as little as
|
||||
possible sits between the wire and the receive timestamp.
|
||||
- **Book ownership:** shard symbols so each book is owned by exactly one worker
|
||||
(lock-free) rather than a shared, locked map.
|
||||
- **Backpressure:** bounded vs unbounded channels — the core correctness/perf
|
||||
tension. Drop-on-full loses data (violates completeness); block adds latency
|
||||
and fills socket buffers. This decision is the bet in miniature.
|
||||
- **Output serialization:** one writer owns the single sink and consumes from
|
||||
all workers.
|
||||
- **Redundancy fit:** multiple sockets feeding one symbol's worker, which
|
||||
dedups / selects by source + seq — the topology must leave room for this from
|
||||
the start.
|
||||
|
||||
## Requirements the pipeline must satisfy
|
||||
|
||||
The output's value is completeness + time-accuracy, so whatever the topology, it
|
||||
must guarantee:
|
||||
|
||||
1. **Receive time stamped at the socket edge**, before any queue or processing.
|
||||
2. **A per-symbol book folded from absolute deltas** (overwrite the level;
|
||||
quantity 0 deletes it).
|
||||
3. **Sequence-gap detection → discard-and-re-snapshot** (a gap means the book is
|
||||
silently corrupt; rebuilding is the only safe response).
|
||||
|
||||
These are proven first, before redundancy or scale.
|
||||
|
||||
## Design ownership
|
||||
|
||||
The **pipeline / concurrency architecture** is driven by the user (the central
|
||||
bet, debated sprint by sprint). The **data model** is mostly delegated to Claude,
|
||||
as a supporting detail that serves the pipeline.
|
||||
|
||||
## Design principles
|
||||
|
||||
Correctness and completeness first; make it fast later. Readiness for the fast
|
||||
version comes from **clean structure, not added complexity** — the seams we want
|
||||
(thread-per-stage joined by channels, the book as a pure `apply(delta) → book`
|
||||
function, symbols shardable later) are also the simplest, clearest design.
|
||||
|
||||
- **Boring, idiomatic, readable wins.** Plain `BTreeMap` for a price-ordered
|
||||
side, standard channels and threads. No micro-optimization, no bit-packing, no
|
||||
`unsafe`, no 3am-to-decode data structures in V1.
|
||||
- **No speculative abstraction.** No generic `Exchange` trait up front — extract
|
||||
it once two exchanges exist (milestone 6). Readiness ≠ pre-building the
|
||||
flexible version.
|
||||
- **A clean seam is cheap and allowed; a complex implementation behind it is
|
||||
not.** A channel, a function boundary, or a `trait` with two real impls keeps
|
||||
the door open; the clever internals are what we refuse.
|
||||
- **Never simplified away, even in V1:** receive-time stamping at the socket
|
||||
edge, and gap → re-snapshot. These are correctness, not readability tradeoffs.
|
||||
- **If a corner is genuinely cut for speed later** (a global lock, an O(n²)
|
||||
scan), mark it with a comment naming the ceiling and the upgrade path, so the
|
||||
debt is visible rather than silently clever.
|
||||
|
||||
Restated in the project `CLAUDE.md` at the skeleton milestone (M1).
|
||||
|
||||
## Constraints (already decided)
|
||||
|
||||
- **Language/stack:** Rust (fixed by the template). Edition 2024.
|
||||
- **Output format:** the repo's SBE helpers (`AlignedSbeWriter` / `SbeEncode`),
|
||||
non-negotiable — output must be in the standard format.
|
||||
- **Build inside `repo/`** (the challenge template): replace `main.rs`, add
|
||||
modules under `src/`. `website/` stays separate as the brief.
|
||||
- **Scope of markets:** regular markets only; MVEs excluded; websocket exchanges
|
||||
only.
|
||||
|
||||
## Workflow differences
|
||||
|
||||
This is a job-challenge repo (cloned template, no Gitea/Woodpecker/release
|
||||
pipeline). From the global workflow we **keep**: `docs/`, `roadmap/`, sprint
|
||||
files, feature branches, and `scripts/ci.sh` (fmt + clippy `-D warnings` +
|
||||
tests + dependency audit). We **drop**: Woodpecker pipelines, `release.sh`,
|
||||
versioning/changelog, rc images, and the `prod`/`dev`/release branch flow.
|
||||
Work on `feat/*` branches squashed into `main`.
|
||||
|
||||
## Not in scope (v1)
|
||||
|
||||
- MVEs (multi-outcome event groupings).
|
||||
- Non-websocket exchanges.
|
||||
- Any exchange beyond Polymarket and Kalshi.
|
||||
- Persistence/replay infrastructure beyond what the SBE output already gives.
|
||||
|
||||
## Pipeline, first sketch
|
||||
|
||||
The primary thing we debate sprint by sprint (this project's equivalent of the
|
||||
schema debate). Rough shape:
|
||||
|
||||
```
|
||||
discovery ──▶ subscription manager ──▶ N socket readers (stamp receive time here)
|
||||
│
|
||||
route by symbol
|
||||
▼
|
||||
per-symbol book workers (fold, gap-detect, dedup sources)
|
||||
│
|
||||
▼
|
||||
single output writer (SBE sink)
|
||||
```
|
||||
|
||||
Each stage and the channels between them are open until settled in a sprint; the
|
||||
async-vs-threads call is the first one.
|
||||
|
||||
## Data model, first sketch
|
||||
|
||||
Supporting detail (mostly Claude's to settle), serving the pipeline above.
|
||||
|
||||
Not a DB — an in-memory domain plus the fixed SBE output schema. Main entities:
|
||||
|
||||
- **Symbol / market** (from discovery): id, trading status, any price/tick
|
||||
metadata the exchange gives.
|
||||
- **NormalizedEvent** (internal, exchange-agnostic): the thing each adapter
|
||||
produces so the core never sees exchange formats. Either a full snapshot or a
|
||||
single delta, each carrying `symbol`, `seq`, `exchange_timestamp`,
|
||||
`received_at`, and payload (side/price/quantity, or the full book).
|
||||
- **Book** (one per symbol): bids and asks as ordered price→quantity maps, plus
|
||||
`last_seq`, last exchange timestamp, and the id of the source currently
|
||||
trusted (for redundancy).
|
||||
- **Connection** (one websocket): the set of symbols it carries.
|
||||
- **Source / redundancy:** several connections may feed the same symbol; one is
|
||||
the trusted source at a time.
|
||||
|
||||
Output schema is fixed by the SBE schema: `Snapshot` and `PriceLevelUpdate`.
|
||||
|
||||
The debate lives in two places: the **NormalizedEvent** shape (both exchanges
|
||||
must map cleanly onto it) and **redundancy reconciliation** (how a symbol's
|
||||
trusted source is chosen and switched, and emitting a fresh snapshot on switch).
|
||||
|
||||
## Open questions
|
||||
|
||||
- **Output: stdout vs file.** Brief says stdout; `AlignedSbeWriter` takes a file
|
||||
path and writes fixed 64 MiB zero-padded chunks. Likely: write to a file
|
||||
(possibly rotated); resolve in the skeleton milestone. *Recommendation: file
|
||||
output, treat "stdout" as loose wording; confirm.*
|
||||
- **Sequence semantics per exchange.** Is `seq` per-symbol or per-connection, and
|
||||
how are snapshots vs deltas sequenced? Differs by exchange; needs doc check per
|
||||
adapter.
|
||||
- **Symbol lifecycle signal.** What marks a market "no longer trading" (drops
|
||||
from the list vs a status field)? Per-exchange, check docs.
|
||||
- **File rotation policy.** When to roll to a new output file (size, time), given
|
||||
64 MiB chunks. Resolve with the output decision.
|
||||
- **Redundancy source-selection.** How to pick/switch the trusted source
|
||||
(freshness, message rate, seq lead). Designed in the redundancy sprint.
|
||||
@@ -0,0 +1,83 @@
|
||||
# Roadmap
|
||||
|
||||
Goal: the order-book collector defined in [project.md](project.md) — a
|
||||
correct, complete, time-accurate feed for prediction-market order books,
|
||||
covering all four challenge extensions across Polymarket and Kalshi.
|
||||
|
||||
Ordering principle: **prove the central bet (the pipeline) and one exchange
|
||||
first, so base + Polymarket is already a complete, submittable deliverable.**
|
||||
Everything after is additive; nothing risky sits at the end. The generic
|
||||
abstraction is *extracted* once a second exchange exists, not designed up front.
|
||||
|
||||
## Milestones
|
||||
|
||||
1. **Skeleton + connectivity** — *not started*
|
||||
Project scaffolding inside the template: module layout (replace `main.rs`),
|
||||
`scripts/ci.sh` (fmt + clippy `-D warnings` + tests + audit), project
|
||||
CLAUDE.md, chosen websocket/HTTP libraries. Settle the pivotal
|
||||
async-vs-threads call. Resolve the output destination (stdout vs file).
|
||||
Connect to Polymarket and log raw messages.
|
||||
*Ends:* CI green; raw Polymarket messages flowing.
|
||||
|
||||
2. **Single-source book pipeline (Polymarket, subset of symbols)** — *not started*
|
||||
The central bet in its first real form: normalize events → per-symbol book
|
||||
(fold absolute deltas) → gap-detect → re-snapshot → SBE output (deltas +
|
||||
snapshot every minute, on subscribe, on re-snapshot) → receive-time stamping
|
||||
→ clean SIGINT. One/few sockets, a fixed subset of symbols. Deterministic
|
||||
replay tests + snapshot↔delta reconciliation.
|
||||
*Ends:* correct SBE output for a symbol subset; tests green.
|
||||
|
||||
3. **Full coverage: discovery + sharding (base challenge complete)** — *not started*
|
||||
REST discovery loop with diffing (subscribe new, unsubscribe gone);
|
||||
load-balance all live regular-market symbols across N sockets.
|
||||
*Ends:* every live Polymarket symbol tracked — base challenge satisfied for
|
||||
one exchange.
|
||||
|
||||
4. **Extension 1 — redundant websockets** — *not started*
|
||||
Multiple connections per symbol; choose a trusted source and switch on
|
||||
loss/quiet/delay; emit a fresh snapshot on every source switch.
|
||||
*Ends:* redundancy demonstrably reduces data loss; source-switch re-snapshots.
|
||||
|
||||
5. **Extension 2 — subscription monitoring** — *not started*
|
||||
Per-symbol activity monitoring (data points per second/minute); unsubscribe +
|
||||
resubscribe symbols that go too quiet; periodic full-socket recycle
|
||||
(disconnect → reconnect → resubscribe) to shed cumulative exchange-side bugs.
|
||||
*Ends:* monitoring + recycling running.
|
||||
|
||||
6. **Extension 3 — generic core + Kalshi adapter** — *not started*
|
||||
Extract the exchange-agnostic core (the `NormalizedEvent` seam) from the
|
||||
working Polymarket code, then add a Kalshi adapter so both exchanges run
|
||||
through the same core. The abstraction is extracted from two concretes here,
|
||||
not before. Requires a Kalshi API key.
|
||||
*Ends:* both Polymarket and Kalshi work through one generic core.
|
||||
|
||||
7. **Extension 4 — latency experiments + writeup** — *not started*
|
||||
Experiment with faster connections (first-after-maintenance, connection
|
||||
racing, source-port tricks) on both exchanges; document findings in a
|
||||
markdown file.
|
||||
*Ends:* experiments documented.
|
||||
|
||||
8. **Polish + submission** — *not started*
|
||||
Final review, README for the solution, packaging/handoff for the challenge.
|
||||
*Ends:* ready to submit.
|
||||
|
||||
## Why this order
|
||||
|
||||
- Milestones 1–3 prove the pipeline (the central bet) and finish the base
|
||||
challenge on one exchange — the riskiest and most valuable work, up front.
|
||||
- Extensions 1 and 2 harden that single exchange before we generalize, so the
|
||||
generic core (6) is extracted from code that already does the hard things.
|
||||
- The generic abstraction waits for the second exchange (Kalshi), so it's
|
||||
extracted from two real implementations rather than guessed from one.
|
||||
- Latency (4) is experimentation, least coupled to correctness, so it's last
|
||||
before polish.
|
||||
|
||||
## Possible features (beyond v1)
|
||||
|
||||
- More exchanges beyond Polymarket and Kalshi.
|
||||
- MVE / multi-outcome market support.
|
||||
- Non-websocket exchange support.
|
||||
- Persistence / replay tooling beyond the raw SBE output.
|
||||
- Observability: metrics, structured logging, dashboards.
|
||||
- Config-driven symbol filtering and per-exchange tuning.
|
||||
- Containerized packaging.
|
||||
Reference in new issue
Block a user