# Order-book collector (Voluntas engineering challenge) A collector for prediction-market order books (Polymarket, then Kalshi) that emits SBE-encoded updates and per-minute snapshots, built inside the challenge template. The brief is https://voluntas.io/engineering-challenge. ## Where things are - `docs/project.md`: definition, central bet (the concurrency pipeline), and the reasons behind the design principles. Read before any design discussion. - `docs/roadmap.md` and `roadmap/NN. name.md`: milestones and each sprint's agreed plan. Read the current sprint file before writing code. - `docs/adr/`: hard-to-reverse decisions (0001: threads, shared-nothing; 0002: the orchestrator, its lanes superseded; 0003: two connections per symbol, first one wins; 0004: the exchange seam, a websocket layer and a loss rule per exchange). - `docs/notes.md`: domain rules and exchange protocol facts (delta semantics, loss detection, message shapes). Read before touching exchange parsing. - `GLOSSARY.md`: the terms (symbol, connection, load, first one wins, duplicate, held, rescued, follow, resync, loss, subscription gap, split, quiet, resubscribe, silent, recycle, sequence number, clock, Snapshot, Delta, lateness, lag, speed). - `src/core/`: what is the same for every exchange (ADR 0004). `orchestrator.rs`: what is tracked, which connections carry each market, their balancing and their recycling (hourly, a Kalshi connection that stays among the slowest taking the next turn); `socket.rs`: a connection (read, stamp, decode, route; reconnects when dropped, silent, lost a message or recycled); `feed.rs`: a book thread's shell (Snapshots, moves, the quiet watch) and the `Rule` trait; `mod.rs`: the `Exchange`, `Session`, `Market` and `Event` traits. - `src/polymarket/`: `gamma.rs` the fetchers (discovery), `wire.rs` the exchange's messages, `feed.rs` the hash rule (the subtle part), `mod.rs` the exchange and its session. - `src/kalshi/`: `listing.rs` the fetchers, `wire.rs` the messages, `sign.rs` the key and the handshake's signature, `feed.rs` the clock and rank rule, `mod.rs` the exchange and its session (the `seq` check). - `src/book.rs`: the book and fixed point; `src/reconcile.rs` + `src/bin/reconcile.rs`: the output check, reading the log's cause lines (`src/cause.rs`); `src/show.rs` + `src/bin/show.rs`: one symbol of the output as text. `tests/fixtures/`: recordings of the live feeds that the tests replay. - `scripts/watch.sh [EXCHANGE] [MINUTES]`: a run to watch, also the final runs' and the README's quick start (roadmap/08): the collector under `timeout -s INT`, a live view of its log (`scripts/watch.awk`, which parses the minute lines' wording: change one, check the other), then `reconcile` and `show`. Output and log in `watch/`. - `src/latency.rs` + `scripts/latency/analyse.py`: the latency probe (`engineering-challenge latency EXCHANGE OUT.csv`, until SIGINT: one CSV line per copy of a change on each of 16 connections) and its analysis (speed, persistence, racing, by address, lag); `LATENCY.md` is the writeup (roadmap/07). - `src/util/` and `sbe_schemas/`: the template's SBE machinery, used as given so output stays readable by the template's `AlignedSbeReader`. Our code is the rest of `src/`. ## Working together The user drives the pipeline / concurrency architecture: propose and debate it, and let the user settle it. Claude owns the data model. ## Writing code - Shared-nothing threads: each stage owns its state on one thread or is sharded, and stages talk over channels. Mutexes stay out of the pipeline. - Boring and readable: std collections (`BTreeMap` for a price-ordered side), plain threads and channels. Speed work waits for the profiler, after V1. - An abstraction arrives with its second implementation (the exchange trait in M6). - Stamp receive time immediately after the socket read, and resync a book (a fresh full book from the exchange) whenever a loss is detected (how depends on the exchange: `docs/notes.md`). - Use `GLOSSARY.md`'s terms: a **Snapshot** is only the output message; getting a fresh book is a **resync**. - A corner cut for speed carries a comment naming its ceiling and upgrade path. - A figure quoted in a doc (README, `LATENCY.md`, a sprint's Outcome) comes from the tool that checks it, `reconcile`'s output or the log itself, never from `watch.sh`'s view or a count grouped by hand: sprint 08's first README figures had four errors from those. ## Environment traps - **DNS: a network can block the exchanges** (its resolver answers a block page's address): check what the hosts resolve to before blaming the code. A 403 is not DNS (see `docs/notes.md`, REST from scripts). - **Write 64 MiB chunks**: `AlignedSbeReader::open` hardcodes 64 MiB (`open_with_chunk_size` is private), so any other chunk size is unreadable. - **Only a clean stop flushes the output**: SIGINT, SIGTERM or SIGHUP. SIGKILL or SIGALRM leaves the last 64 MiB chunk unwritten. Bound a run with GNU `timeout -s INT`, then check the output against the log: `timeout -s INT 600 ./target/release/engineering-challenge polymarket out.sbe 2> run.log; ./target/release/reconcile out.sbe run.log`. With the log, `reconcile` pairs each mismatch with its cause line and fails only on one left unexplained. - **Wait on a PID, never on a `pgrep -f` pattern**: `until ! pgrep -f "capacity scan"` matches the waiting shell's own command line and never ends, and `pkill -f PATTERN` can kill the shell running it. Take the PID (`pgrep -f '^./target/release/...'`, anchored) and loop on `kill -0 $pid`. - **No `/usr/bin/time` on the server**: for a run's peak memory and thread count, read `VmHWM` and `Threads` from `/proc//status` while it runs (`scripts/watch.sh` samples them every 10 s). - **mawk buffers a pipe's input** unless `-W interactive`: a live view through `awk` shows nothing until the end (`scripts/watch.sh` passes it when the awk is mawk, as on the server). - **A job started with `&` in `sh` ignores Ctrl-C**: a script that runs the collector in the background must pass SIGINT on itself (`watch.sh`'s trap). To try Ctrl-C without a terminal: `(sleep N; printf '\003') | script -qfec "CMD" /dev/null`. - **Polymarket's Gamma answers 429 to runs started close together**: two collectors' first scans at once (or a few runs within minutes) got every page refused; the scan retries, but leave a few minutes between runs. - **`*.pem` is gitignored globally** (`~/.gitignore`): a test key goes in the source, as `src/kalshi/sign.rs` does with RFC 8032's and RFC 9500's published test keys. - **The server is shared and gets overloaded** (load up to 92 on 12 cores, 2026-11-11): a starved collector falls behind and gets closed with 1013. Check `cat /proc/loadavg` before a live run whose numbers are compared, and alternate the builds in short rounds rather than running them back to back. - **Kalshi needs a key; nothing else does**: `KALSHI_KEY_ID_FILE` and `KALSHI_PRIVATE_KEY_FILE` name its two files (never read or print them). The key is Ed25519 or RSA (RSA-PSS-SHA256), told apart by the parsed key. Opening ~100 connections at once gets some handshakes refused with 429; each retries after 1 s and gets in. - **cargo keeps a build whose source got an older mtime**: a script that restores a file from a copy (`shutil.move`, `cp -p`) must `touch` it, or the next `cargo test` runs the old binary (a mutation check did, and a "flaky" test was only the last mutant). - **Live data beats the docs**: Polymarket's market channel has rules its docs don't state (`docs/notes.md`); check a protocol assumption on a recording of the real feed (`tests/fixtures/record.py`) before relying on it. - **The workspace can't choose what the exchange sees**: a NAT rewrites every connection's source port (local 41000 left as 36996, then as 37086 a minute later; `curl --local-port P https://ifconfig.me/all.json`), and there is no IPv6 route (connections to an IPv6 address fail with "Network is unreachable"). The probe uses IPv4 addresses only. - **Python checks run through `uvx`**: no ruff or pyflakes is installed; `scripts/ci.sh` runs `uvx ruff@0.15.0` (config in `scripts/latency/ruff.toml`). ## Workflow differences A job-challenge repo cloned from the template: no Gitea, Woodpecker or release pipeline. - **Kept**: `docs/`, `roadmap/` sprint files, `feat/*` branches, and `scripts/ci.sh` (fmt, clippy `-D warnings`, tests). - **Dropped**: the dependency audit (supply-chain checks are for deployed products), Woodpecker, `release.sh`, versioning and changelog, rc images, and the `prod` / `dev` branches. CI targets application quality, not prod-grade. - **Branches**: `main` plays the role of `dev`; `feat/*` branches are squashed into `main`.