This repository has been archived on 2026-11-16. You can view files and clone it. You cannot open issues or pull requests or push a commit.
Files
orderbook-collector/CLAUDE.md
T
pascal 00202bbc51 docs: correct the README's claims, state the limits, and CI on macOS
The README said the quiet rule holds for the quietest markets (it can't
judge one under 3 changes a minute) and that all four extensions were
done (two of the brief's latency ideas were not tried); LATENCY.md said
the slow recycle was not seen to fire live, two lines after the run where
it did. Limits gains the process going down, the quiet rule's blind
spots and Kalshi's skipped-change case.

CLAUDE.md loses the home network's details and the server's key paths,
of no use to a reader of the public repo.

scripts/test_watch.sh no longer puts a here-document inside `$( )`:
bash 3.2, which is sh on macOS, stopped at an apostrophe there and
scripts/ci.sh ended on a syntax error.
2026-11-12 23:17:10 +01:00

8.7 KiB

Order-book collector (Voluntas engineering challenge)

A collector for prediction-market order books (Polymarket, then Kalshi) that emits SBE-encoded updates and per-minute snapshots, built inside the challenge template. The brief is https://voluntas.io/engineering-challenge.

Where things are

  • docs/project.md: definition, central bet (the concurrency pipeline), and the reasons behind the design principles. Read before any design discussion.
  • docs/roadmap.md and roadmap/NN. name.md: milestones and each sprint's agreed plan. Read the current sprint file before writing code.
  • docs/adr/: hard-to-reverse decisions (0001: threads, shared-nothing; 0002: the orchestrator, its lanes superseded; 0003: two connections per symbol, first one wins; 0004: the exchange seam, a websocket layer and a loss rule per exchange).
  • docs/notes.md: domain rules and exchange protocol facts (delta semantics, loss detection, message shapes). Read before touching exchange parsing.
  • GLOSSARY.md: the terms (symbol, connection, load, first one wins, duplicate, held, rescued, follow, resync, loss, subscription gap, split, quiet, resubscribe, silent, recycle, sequence number, clock, Snapshot, Delta, lateness, lag, speed).
  • src/core/: what is the same for every exchange (ADR 0004). orchestrator.rs: what is tracked, which connections carry each market, their balancing and their recycling (hourly, a Kalshi connection that stays among the slowest taking the next turn); socket.rs: a connection (read, stamp, decode, route; reconnects when dropped, silent, lost a message or recycled); feed.rs: a book thread's shell (Snapshots, moves, the quiet watch) and the Rule trait; mod.rs: the Exchange, Session, Market and Event traits.
  • src/polymarket/: gamma.rs the fetchers (discovery), wire.rs the exchange's messages, feed.rs the hash rule (the subtle part), mod.rs the exchange and its session.
  • src/kalshi/: listing.rs the fetchers, wire.rs the messages, sign.rs the key and the handshake's signature, feed.rs the clock and rank rule, mod.rs the exchange and its session (the seq check).
  • src/book.rs: the book and fixed point; src/reconcile.rs + src/bin/reconcile.rs: the output check, reading the log's cause lines (src/cause.rs); src/show.rs + src/bin/show.rs: one symbol of the output as text. tests/fixtures/: recordings of the live feeds that the tests replay.
  • scripts/watch.sh [EXCHANGE] [MINUTES]: a run to watch, also the final runs' and the README's quick start (roadmap/08): the collector under timeout -s INT, a live view of its log (scripts/watch.awk, which parses the minute lines' wording: change one, check the other), then reconcile and show. Output and log in watch/.
  • src/latency.rs + scripts/latency/analyse.py: the latency probe (engineering-challenge latency EXCHANGE OUT.csv, until SIGINT: one CSV line per copy of a change on each of 16 connections) and its analysis (speed, persistence, racing, by address, lag); LATENCY.md is the writeup (roadmap/07).
  • src/util/ and sbe_schemas/: the template's SBE machinery, used as given so output stays readable by the template's AlignedSbeReader. Our code is the rest of src/.

Working together

The user drives the pipeline / concurrency architecture: propose and debate it, and let the user settle it. Claude owns the data model.

Writing code

  • Shared-nothing threads: each stage owns its state on one thread or is sharded, and stages talk over channels. Mutexes stay out of the pipeline.
  • Boring and readable: std collections (BTreeMap for a price-ordered side), plain threads and channels. Speed work waits for the profiler, after V1.
  • An abstraction arrives with its second implementation (the exchange trait in M6).
  • Stamp receive time immediately after the socket read, and resync a book (a fresh full book from the exchange) whenever a loss is detected (how depends on the exchange: docs/notes.md).
  • Use GLOSSARY.md's terms: a Snapshot is only the output message; getting a fresh book is a resync.
  • A corner cut for speed carries a comment naming its ceiling and upgrade path.
  • A figure quoted in a doc (README, LATENCY.md, a sprint's Outcome) comes from the tool that checks it, reconcile's output or the log itself, never from watch.sh's view or a count grouped by hand: sprint 08's first README figures had four errors from those.

Environment traps

  • DNS: a network can block the exchanges (its resolver answers a block page's address): check what the hosts resolve to before blaming the code. A 403 is not DNS (see docs/notes.md, REST from scripts).
  • Write 64 MiB chunks: AlignedSbeReader::open hardcodes 64 MiB (open_with_chunk_size is private), so any other chunk size is unreadable.
  • Only a clean stop flushes the output: SIGINT, SIGTERM or SIGHUP. SIGKILL or SIGALRM leaves the last 64 MiB chunk unwritten. Bound a run with GNU timeout -s INT, then check the output against the log: timeout -s INT 600 ./target/release/engineering-challenge polymarket out.sbe 2> run.log; ./target/release/reconcile out.sbe run.log. With the log, reconcile pairs each mismatch with its cause line and fails only on one left unexplained.
  • Wait on a PID, never on a pgrep -f pattern: until ! pgrep -f "capacity scan" matches the waiting shell's own command line and never ends, and pkill -f PATTERN can kill the shell running it. Take the PID (pgrep -f '^./target/release/...', anchored) and loop on kill -0 $pid.
  • No /usr/bin/time on the server: for a run's peak memory and thread count, read VmHWM and Threads from /proc/<pid>/status while it runs (scripts/watch.sh samples them every 10 s).
  • mawk buffers a pipe's input unless -W interactive: a live view through awk shows nothing until the end (scripts/watch.sh passes it when the awk is mawk, as on the server).
  • A job started with & in sh ignores Ctrl-C: a script that runs the collector in the background must pass SIGINT on itself (watch.sh's trap). To try Ctrl-C without a terminal: (sleep N; printf '\003') | script -qfec "CMD" /dev/null.
  • Polymarket's Gamma answers 429 to runs started close together: two collectors' first scans at once (or a few runs within minutes) got every page refused; the scan retries, but leave a few minutes between runs.
  • *.pem is gitignored globally (~/.gitignore): a test key goes in the source, as src/kalshi/sign.rs does with RFC 8032's and RFC 9500's published test keys.
  • The server is shared and gets overloaded (load up to 92 on 12 cores, 2026-11-11): a starved collector falls behind and gets closed with 1013. Check cat /proc/loadavg before a live run whose numbers are compared, and alternate the builds in short rounds rather than running them back to back.
  • Kalshi needs a key; nothing else does: KALSHI_KEY_ID_FILE and KALSHI_PRIVATE_KEY_FILE name its two files (never read or print them). The key is Ed25519 or RSA (RSA-PSS-SHA256), told apart by the parsed key. Opening ~100 connections at once gets some handshakes refused with 429; each retries after 1 s and gets in.
  • cargo keeps a build whose source got an older mtime: a script that restores a file from a copy (shutil.move, cp -p) must touch it, or the next cargo test runs the old binary (a mutation check did, and a "flaky" test was only the last mutant).
  • Live data beats the docs: Polymarket's market channel has rules its docs don't state (docs/notes.md); check a protocol assumption on a recording of the real feed (tests/fixtures/record.py) before relying on it.
  • The workspace can't choose what the exchange sees: a NAT rewrites every connection's source port (local 41000 left as 36996, then as 37086 a minute later; curl --local-port P https://ifconfig.me/all.json), and there is no IPv6 route (connections to an IPv6 address fail with "Network is unreachable"). The probe uses IPv4 addresses only.
  • Python checks run through uvx: no ruff or pyflakes is installed; scripts/ci.sh runs uvx ruff@0.15.0 (config in scripts/latency/ruff.toml).

Workflow differences

A job-challenge repo cloned from the template: no Gitea, Woodpecker or release pipeline.

  • Kept: docs/, roadmap/ sprint files, feat/* branches, and scripts/ci.sh (fmt, clippy -D warnings, tests).
  • Dropped: the dependency audit (supply-chain checks are for deployed products), Woodpecker, release.sh, versioning and changelog, rc images, and the prod / dev branches. CI targets application quality, not prod-grade.
  • Branches: main plays the role of dev; feat/* branches are squashed into main.