Archived
The exchange can fail without telling us, and first one wins hides a subscription gone quiet on one of a symbol's two connections. - A connection that receives nothing for 30 s, not even the keepalive's answer, reconnects. - Each book thread counts a symbol's changes per connection every minute: a subscription with none against three or more on the other is resubscribed, and moved to another connection if it stays quiet. - The orchestrator recycles one connection at a time, each about hourly, and never while another connection is down or just back. - A reconnect's opening book of a symbol unchanged for 10 s is judged like a resync's answer: one newer than ours that differs shows a symbol quiet on both connections, written as a Snapshot and logged as a loss.
5.5 KiB
5.5 KiB
Order-book collector (Voluntas engineering challenge)
A collector for prediction-market order books (Polymarket, then Kalshi) that
emits SBE-encoded updates and per-minute snapshots, built inside the challenge
template. The brief is ../website/Engineering Challenge _ Voluntas.html,
outside this repo.
Where things are
docs/project.md: definition, central bet (the concurrency pipeline), and the reasons behind the design principles. Read before any design discussion.docs/roadmap.mdandroadmap/NN. name.md: milestones and each sprint's agreed plan. Read the current sprint file before writing code.docs/adr/: hard-to-reverse decisions (0001: threads, shared-nothing; 0002: the orchestrator, its lanes superseded; 0003: two connections per symbol, first one wins).docs/notes.md: domain rules and exchange protocol facts (delta semantics, loss detection, message shapes). Read before touching exchange parsing.GLOSSARY.md: the terms (symbol, connection, load, first one wins, duplicate, held, rescued, follow, resync, loss, subscription gap, split, quiet, resubscribe, silent, recycle, Snapshot, Delta).src/polymarket/orchestrator.rs: what is tracked, which connections carry each market, their balancing and their recycling;src/polymarket/gamma.rs: the fetchers (discovery);src/polymarket/socket.rs: a connection (read, stamp, decode, route; reconnects when dropped, silent or recycled);src/polymarket/wire.rs: the exchange's messages;src/polymarket/feed.rs: a book thread, its loss rules (the subtle part) and its watch for quiet subscriptions;src/book.rs: the book and fixed point;src/reconcile.rs+src/bin/reconcile.rs: the output check, reading the log's cause lines (src/cause.rs).tests/fixtures/: recordings of the live feed, on one and two connections, that the tests replay.src/util/andsbe_schemas/: the template's SBE machinery, used as given so output stays readable by the template'sAlignedSbeReader. Our code is the rest ofsrc/.
Working together
The user drives the pipeline / concurrency architecture: propose and debate it, and let the user settle it. Claude owns the data model.
Writing code
- Shared-nothing threads: each stage owns its state on one thread or is sharded, and stages talk over channels. Mutexes stay out of the pipeline.
- Boring and readable: std collections (
BTreeMapfor a price-ordered side), plain threads and channels. Speed work waits for the profiler, after V1. - An abstraction arrives with its second implementation (the exchange trait in M6).
- Stamp receive time immediately after the socket read, and resync a book (a
fresh full book from the exchange) whenever a loss is detected (how depends
on the exchange:
docs/notes.md). - Use
GLOSSARY.md's terms: a Snapshot is only the output message; getting a fresh book is a resync. - A corner cut for speed carries a comment naming its ceiling and upgrade path.
Environment traps
- DNS: the exchanges resolve normally on the server (a remote-claude
workspace, since 2026-11-09). The Mac's dev network blocks them (its resolver
answers
195.186.4.210/2001:4d98:4:b10c::1, a Swisscom block page); there the user pins the hosts withsudo ../host.sh on(workspace root, outside this repo;offremoves them), rerun when the CDN addresses rotate. A 403 is not DNS (seedocs/notes.md, REST from scripts). - Write 64 MiB chunks:
AlignedSbeReader::openhardcodes 64 MiB (open_with_chunk_sizeis private), so any other chunk size is unreadable. - Only a clean stop flushes the output: SIGINT, SIGTERM or SIGHUP. SIGKILL
or SIGALRM leaves the last 64 MiB chunk unwritten. Bound a run with GNU
timeout -s INT, then check the output against the log:timeout -s INT 600 ./target/release/engineering-challenge out.sbe 2> run.log; ./target/release/reconcile out.sbe run.log. With the log,reconcilepairs each mismatch with its cause line and fails only on one left unexplained. - Wait on a PID, never on a
pgrep -fpattern:until ! pgrep -f "capacity scan"matches the waiting shell's own command line and never ends, andpkill -f PATTERNcan kill the shell running it. Take the PID (pgrep -f '^./target/release/...', anchored) and loop onkill -0 $pid. - No
/usr/bin/timeon the server: for a run's peak memory and thread count, readVmHWMandThreadsfrom/proc/<pid>/statuswhile it runs (sprint 05's run script sampled them every 30 s). - The server is shared and gets overloaded (load up to 92 on 12 cores,
2026-11-11): a starved collector falls behind and gets closed with 1013.
Check
cat /proc/loadavgbefore a live run whose numbers are compared, and alternate the builds in short rounds rather than running them back to back. - Live data beats the docs: Polymarket's market channel has rules its
docs don't state (
docs/notes.md); check a protocol assumption on a recording of the real feed (tests/fixtures/record.py) before relying on it.
Workflow differences
A job-challenge repo cloned from the template: no Gitea, Woodpecker or release pipeline.
- Kept:
docs/,roadmap/sprint files,feat/*branches, andscripts/ci.sh(fmt, clippy-D warnings, tests). - Dropped: the dependency audit (supply-chain checks are for deployed
products), Woodpecker,
release.sh, versioning and changelog, rc images, and theprod/devbranches. CI targets application quality, not prod-grade. - Branches:
mainplays the role ofdev;feat/*branches are squashed intomain.