# Order-book collector (Voluntas engineering challenge) A collector for prediction-market order books (Polymarket, then Kalshi) that emits SBE-encoded updates and per-minute snapshots, built inside the challenge template. The brief is `../website/Engineering Challenge _ Voluntas.html`, outside this repo. ## Where things are - `docs/project.md`: definition, central bet (the concurrency pipeline), and the reasons behind the design principles. Read before any design discussion. - `docs/roadmap.md` and `roadmap/NN. name.md`: milestones and each sprint's agreed plan. Read the current sprint file before writing code. - `docs/adr/`: hard-to-reverse decisions (0001: threads, shared-nothing; 0002: lanes and the orchestrator). - `docs/notes.md`: domain rules and exchange protocol facts (delta semantics, loss detection, message shapes). Read before touching exchange parsing. - `GLOSSARY.md`: the terms (symbol, resync, loss, subscription gap, lane, load, Snapshot, Delta). - `src/polymarket/orchestrator.rs`: what is tracked, the lanes and their balancing; `src/polymarket/gamma.rs`: the fetchers (discovery); `src/polymarket/feed.rs`: a lane's book thread and its loss rules (the subtle part); `src/polymarket/socket.rs`: a lane's connection; `src/book.rs`: the book and fixed point; `src/reconcile.rs` + `src/bin/reconcile.rs`: the output check. `tests/fixtures/`: a 90 s recording of the live feed the tests replay. - `src/util/` and `sbe_schemas/`: the template's SBE machinery, used as given so output stays readable by the template's `AlignedSbeReader`. Our code is the rest of `src/`. ## Working together The user drives the pipeline / concurrency architecture: propose and debate it, and let the user settle it. Claude owns the data model. ## Writing code - Shared-nothing threads: each stage owns its state on one thread or is sharded, and stages talk over channels. Mutexes stay out of the pipeline. - Boring and readable: std collections (`BTreeMap` for a price-ordered side), plain threads and channels. Speed work waits for the profiler, after V1. - An abstraction arrives with its second implementation (the exchange trait in M6). - Stamp receive time immediately after the socket read, and resync a book (a fresh full book from the exchange) whenever a loss is detected (how depends on the exchange: `docs/notes.md`). - Use `GLOSSARY.md`'s terms: a **Snapshot** is only the output message; getting a fresh book is a **resync**. - A corner cut for speed carries a comment naming its ceiling and upgrade path. ## Environment traps - **DNS: the exchanges resolve normally on the server** (a remote-claude workspace, since 2026-11-09). The Mac's dev network blocks them (its resolver answers `195.186.4.210` / `2001:4d98:4:b10c::1`, a Swisscom block page); there the user pins the hosts with `sudo ../host.sh on` (workspace root, outside this repo; `off` removes them), rerun when the CDN addresses rotate. A 403 is not DNS (see `docs/notes.md`, REST from scripts). - **Write 64 MiB chunks**: `AlignedSbeReader::open` hardcodes 64 MiB (`open_with_chunk_size` is private), so any other chunk size is unreadable. - **The collector runs until SIGINT, and only SIGINT flushes its output.** Any other signal (SIGTERM, SIGALRM) leaves the last 64 MiB chunk unwritten. Bound a run with GNU `timeout -s INT`, then check the output: `timeout -s INT 600 ./target/release/engineering-challenge out.sbe 2> run.log; ./target/release/reconcile out.sbe`. `reconcile` counts subscription gaps as mismatches too: compare its list with the log's "loss", "subscription gap" and "first book after a move" lines. - **Wait on a PID, never on a `pgrep -f` pattern**: `until ! pgrep -f "capacity scan"` matches the waiting shell's own command line and never ends, and `pkill -f PATTERN` can kill the shell running it. Take the PID (`pgrep -f '^./target/release/...'`, anchored) and loop on `kill -0 $pid`. - **Live data beats the docs**: Polymarket's market channel has rules its docs don't state (`docs/notes.md`); check a protocol assumption on a recording of the real feed (`tests/fixtures/record.py`) before relying on it. ## Workflow differences A job-challenge repo cloned from the template: no Gitea, Woodpecker or release pipeline. - **Kept**: `docs/`, `roadmap/` sprint files, `feat/*` branches, and `scripts/ci.sh` (fmt, clippy `-D warnings`, tests). - **Dropped**: the dependency audit (supply-chain checks are for deployed products), Woodpecker, `release.sh`, versioning and changelog, rc images, and the `prod` / `dev` branches. CI targets application quality, not prod-grade. - **Branches**: `main` plays the role of `dev`; `feat/*` branches are squashed into `main`.