Archived
The README said the quiet rule holds for the quietest markets (it can't judge one under 3 changes a minute) and that all four extensions were done (two of the brief's latency ideas were not tried); LATENCY.md said the slow recycle was not seen to fire live, two lines after the run where it did. Limits gains the process going down, the quiet rule's blind spots and Kalshi's skipped-change case. CLAUDE.md loses the home network's details and the server's key paths, of no use to a reader of the public repo. scripts/test_watch.sh no longer puts a here-document inside `$( )`: bash 3.2, which is sh on macOS, stopped at an apostrophe there and scripts/ci.sh ended on a syntax error.
8.7 KiB
8.7 KiB
Order-book collector (Voluntas engineering challenge)
A collector for prediction-market order books (Polymarket, then Kalshi) that emits SBE-encoded updates and per-minute snapshots, built inside the challenge template. The brief is https://voluntas.io/engineering-challenge.
Where things are
docs/project.md: definition, central bet (the concurrency pipeline), and the reasons behind the design principles. Read before any design discussion.docs/roadmap.mdandroadmap/NN. name.md: milestones and each sprint's agreed plan. Read the current sprint file before writing code.docs/adr/: hard-to-reverse decisions (0001: threads, shared-nothing; 0002: the orchestrator, its lanes superseded; 0003: two connections per symbol, first one wins; 0004: the exchange seam, a websocket layer and a loss rule per exchange).docs/notes.md: domain rules and exchange protocol facts (delta semantics, loss detection, message shapes). Read before touching exchange parsing.GLOSSARY.md: the terms (symbol, connection, load, first one wins, duplicate, held, rescued, follow, resync, loss, subscription gap, split, quiet, resubscribe, silent, recycle, sequence number, clock, Snapshot, Delta, lateness, lag, speed).src/core/: what is the same for every exchange (ADR 0004).orchestrator.rs: what is tracked, which connections carry each market, their balancing and their recycling (hourly, a Kalshi connection that stays among the slowest taking the next turn);socket.rs: a connection (read, stamp, decode, route; reconnects when dropped, silent, lost a message or recycled);feed.rs: a book thread's shell (Snapshots, moves, the quiet watch) and theRuletrait;mod.rs: theExchange,Session,MarketandEventtraits.src/polymarket/:gamma.rsthe fetchers (discovery),wire.rsthe exchange's messages,feed.rsthe hash rule (the subtle part),mod.rsthe exchange and its session.src/kalshi/:listing.rsthe fetchers,wire.rsthe messages,sign.rsthe key and the handshake's signature,feed.rsthe clock and rank rule,mod.rsthe exchange and its session (theseqcheck).src/book.rs: the book and fixed point;src/reconcile.rs+src/bin/reconcile.rs: the output check, reading the log's cause lines (src/cause.rs);src/show.rs+src/bin/show.rs: one symbol of the output as text.tests/fixtures/: recordings of the live feeds that the tests replay.scripts/watch.sh [EXCHANGE] [MINUTES]: a run to watch, also the final runs' and the README's quick start (roadmap/08): the collector undertimeout -s INT, a live view of its log (scripts/watch.awk, which parses the minute lines' wording: change one, check the other), thenreconcileandshow. Output and log inwatch/.src/latency.rs+scripts/latency/analyse.py: the latency probe (engineering-challenge latency EXCHANGE OUT.csv, until SIGINT: one CSV line per copy of a change on each of 16 connections) and its analysis (speed, persistence, racing, by address, lag);LATENCY.mdis the writeup (roadmap/07).src/util/andsbe_schemas/: the template's SBE machinery, used as given so output stays readable by the template'sAlignedSbeReader. Our code is the rest ofsrc/.
Working together
The user drives the pipeline / concurrency architecture: propose and debate it, and let the user settle it. Claude owns the data model.
Writing code
- Shared-nothing threads: each stage owns its state on one thread or is sharded, and stages talk over channels. Mutexes stay out of the pipeline.
- Boring and readable: std collections (
BTreeMapfor a price-ordered side), plain threads and channels. Speed work waits for the profiler, after V1. - An abstraction arrives with its second implementation (the exchange trait in M6).
- Stamp receive time immediately after the socket read, and resync a book (a
fresh full book from the exchange) whenever a loss is detected (how depends
on the exchange:
docs/notes.md). - Use
GLOSSARY.md's terms: a Snapshot is only the output message; getting a fresh book is a resync. - A corner cut for speed carries a comment naming its ceiling and upgrade path.
- A figure quoted in a doc (README,
LATENCY.md, a sprint's Outcome) comes from the tool that checks it,reconcile's output or the log itself, never fromwatch.sh's view or a count grouped by hand: sprint 08's first README figures had four errors from those.
Environment traps
- DNS: a network can block the exchanges (its resolver answers a block
page's address): check what the hosts resolve to before blaming the code.
A 403 is not DNS (see
docs/notes.md, REST from scripts). - Write 64 MiB chunks:
AlignedSbeReader::openhardcodes 64 MiB (open_with_chunk_sizeis private), so any other chunk size is unreadable. - Only a clean stop flushes the output: SIGINT, SIGTERM or SIGHUP. SIGKILL
or SIGALRM leaves the last 64 MiB chunk unwritten. Bound a run with GNU
timeout -s INT, then check the output against the log:timeout -s INT 600 ./target/release/engineering-challenge polymarket out.sbe 2> run.log; ./target/release/reconcile out.sbe run.log. With the log,reconcilepairs each mismatch with its cause line and fails only on one left unexplained. - Wait on a PID, never on a
pgrep -fpattern:until ! pgrep -f "capacity scan"matches the waiting shell's own command line and never ends, andpkill -f PATTERNcan kill the shell running it. Take the PID (pgrep -f '^./target/release/...', anchored) and loop onkill -0 $pid. - No
/usr/bin/timeon the server: for a run's peak memory and thread count, readVmHWMandThreadsfrom/proc/<pid>/statuswhile it runs (scripts/watch.shsamples them every 10 s). - mawk buffers a pipe's input unless
-W interactive: a live view throughawkshows nothing until the end (scripts/watch.shpasses it when the awk is mawk, as on the server). - A job started with
&inshignores Ctrl-C: a script that runs the collector in the background must pass SIGINT on itself (watch.sh's trap). To try Ctrl-C without a terminal:(sleep N; printf '\003') | script -qfec "CMD" /dev/null. - Polymarket's Gamma answers 429 to runs started close together: two collectors' first scans at once (or a few runs within minutes) got every page refused; the scan retries, but leave a few minutes between runs.
*.pemis gitignored globally (~/.gitignore): a test key goes in the source, assrc/kalshi/sign.rsdoes with RFC 8032's and RFC 9500's published test keys.- The server is shared and gets overloaded (load up to 92 on 12 cores,
2026-11-11): a starved collector falls behind and gets closed with 1013.
Check
cat /proc/loadavgbefore a live run whose numbers are compared, and alternate the builds in short rounds rather than running them back to back. - Kalshi needs a key; nothing else does:
KALSHI_KEY_ID_FILEandKALSHI_PRIVATE_KEY_FILEname its two files (never read or print them). The key is Ed25519 or RSA (RSA-PSS-SHA256), told apart by the parsed key. Opening ~100 connections at once gets some handshakes refused with 429; each retries after 1 s and gets in. - cargo keeps a build whose source got an older mtime: a script that
restores a file from a copy (
shutil.move,cp -p) musttouchit, or the nextcargo testruns the old binary (a mutation check did, and a "flaky" test was only the last mutant). - Live data beats the docs: Polymarket's market channel has rules its
docs don't state (
docs/notes.md); check a protocol assumption on a recording of the real feed (tests/fixtures/record.py) before relying on it. - The workspace can't choose what the exchange sees: a NAT rewrites
every connection's source port (local 41000 left as 36996, then as 37086
a minute later;
curl --local-port P https://ifconfig.me/all.json), and there is no IPv6 route (connections to an IPv6 address fail with "Network is unreachable"). The probe uses IPv4 addresses only. - Python checks run through
uvx: no ruff or pyflakes is installed;scripts/ci.shrunsuvx ruff@0.15.0(config inscripts/latency/ruff.toml).
Workflow differences
A job-challenge repo cloned from the template: no Gitea, Woodpecker or release pipeline.
- Kept:
docs/,roadmap/sprint files,feat/*branches, andscripts/ci.sh(fmt, clippy-D warnings, tests). - Dropped: the dependency audit (supply-chain checks are for deployed
products), Woodpecker,
release.sh, versioning and changelog, rc images, and theprod/devbranches. CI targets application quality, not prod-grade. - Branches:
mainplays the role ofdev;feat/*branches are squashed intomain.