Archived
An orchestrator keeps the tracked set equal to Gamma's live markets (a scan at startup and every hour, new and closed markets every minute, reaching back 15 minutes for Gamma's indexing delay) and spreads them over lanes, each a connection and a book thread, all writing through the one writer. Lanes report each market's load in bytes per second; the orchestrator puts new markets on the least-loaded lane and moves the busiest markets off lanes over a 300 KB/s budget, busiest lane first. Differing books our own subscribe explains are subscription gaps, not losses, and every first book after a move is logged, so reconcile's mismatches are all accounted for in the log. A 75-minute live run tracked ~195k markets (~390k symbols) in 89 lanes.
4.6 KiB
4.6 KiB
Order-book collector (Voluntas engineering challenge)
A collector for prediction-market order books (Polymarket, then Kalshi) that
emits SBE-encoded updates and per-minute snapshots, built inside the challenge
template. The brief is ../website/Engineering Challenge _ Voluntas.html,
outside this repo.
Where things are
docs/project.md: definition, central bet (the concurrency pipeline), and the reasons behind the design principles. Read before any design discussion.docs/roadmap.mdandroadmap/NN. name.md: milestones and each sprint's agreed plan. Read the current sprint file before writing code.docs/adr/: hard-to-reverse decisions (0001: threads, shared-nothing; 0002: lanes and the orchestrator).docs/notes.md: domain rules and exchange protocol facts (delta semantics, loss detection, message shapes). Read before touching exchange parsing.GLOSSARY.md: the terms (symbol, resync, loss, subscription gap, lane, load, Snapshot, Delta).src/polymarket/orchestrator.rs: what is tracked, the lanes and their balancing;src/polymarket/gamma.rs: the fetchers (discovery);src/polymarket/feed.rs: a lane's book thread and its loss rules (the subtle part);src/polymarket/socket.rs: a lane's connection;src/book.rs: the book and fixed point;src/reconcile.rs+src/bin/reconcile.rs: the output check.tests/fixtures/: a 90 s recording of the live feed the tests replay.src/util/andsbe_schemas/: the template's SBE machinery, used as given so output stays readable by the template'sAlignedSbeReader. Our code is the rest ofsrc/.
Working together
The user drives the pipeline / concurrency architecture: propose and debate it, and let the user settle it. Claude owns the data model.
Writing code
- Shared-nothing threads: each stage owns its state on one thread or is sharded, and stages talk over channels. Mutexes stay out of the pipeline.
- Boring and readable: std collections (
BTreeMapfor a price-ordered side), plain threads and channels. Speed work waits for the profiler, after V1. - An abstraction arrives with its second implementation (the exchange trait in M6).
- Stamp receive time immediately after the socket read, and resync a book (a
fresh full book from the exchange) whenever a loss is detected (how depends
on the exchange:
docs/notes.md). - Use
GLOSSARY.md's terms: a Snapshot is only the output message; getting a fresh book is a resync. - A corner cut for speed carries a comment naming its ceiling and upgrade path.
Environment traps
- DNS: the exchanges resolve normally on the server (a remote-claude
workspace, since 2026-11-09). The Mac's dev network blocks them (its resolver
answers
195.186.4.210/2001:4d98:4:b10c::1, a Swisscom block page); there the user pins the hosts withsudo ../host.sh on(workspace root, outside this repo;offremoves them), rerun when the CDN addresses rotate. A 403 is not DNS (seedocs/notes.md, REST from scripts). - Write 64 MiB chunks:
AlignedSbeReader::openhardcodes 64 MiB (open_with_chunk_sizeis private), so any other chunk size is unreadable. - The collector runs until SIGINT, and only SIGINT flushes its output.
Any other signal (SIGTERM, SIGALRM) leaves the last 64 MiB chunk unwritten.
Bound a run with GNU
timeout -s INT, then check the output:timeout -s INT 600 ./target/release/engineering-challenge out.sbe 2> run.log; ./target/release/reconcile out.sbe.reconcilecounts subscription gaps as mismatches too: compare its list with the log's "loss", "subscription gap" and "first book after a move" lines. - Wait on a PID, never on a
pgrep -fpattern:until ! pgrep -f "capacity scan"matches the waiting shell's own command line and never ends, andpkill -f PATTERNcan kill the shell running it. Take the PID (pgrep -f '^./target/release/...', anchored) and loop onkill -0 $pid. - Live data beats the docs: Polymarket's market channel has rules its
docs don't state (
docs/notes.md); check a protocol assumption on a recording of the real feed (tests/fixtures/record.py) before relying on it.
Workflow differences
A job-challenge repo cloned from the template: no Gitea, Woodpecker or release pipeline.
- Kept:
docs/,roadmap/sprint files,feat/*branches, andscripts/ci.sh(fmt, clippy-D warnings, tests). - Dropped: the dependency audit (supply-chain checks are for deployed
products), Woodpecker,
release.sh, versioning and changelog, rc images, and theprod/devbranches. CI targets application quality, not prod-grade. - Branches:
mainplays the role ofdev;feat/*branches are squashed intomain.