This repository has been archived on 2026-11-16. You can view files and clone it. You cannot open issues or pull requests or push a commit.
Files
orderbook-collector/CLAUDE.md
T
pascal cf6c16c9cf feat: ready to submit, a run to watch, the README, the review's fixes
Sprint 08 (roadmap/08. ready to submit.md), the last milestone.

- scripts/watch.sh runs the collector with a live view of its log, stops
  cleanly on Ctrl-C, then checks the output (reconcile) and shows one
  symbol's book as text (show, a new binary).
- The README for the solution, with the final runs: Polymarket through an
  outage it recovered from on its own, Kalshi's demo with two slow
  connections recycled.
- From a review of the whole codebase: Kalshi takes RSA keys too
  (RSA-PSS-SHA256); a slow Kalshi connection takes the hourly recycle's
  next turn, which it nearly never got before; a market tracked again no
  longer fails reconcile; Book::add checks for overflow; usage and --help
  name the key variables; the first scan says how far it got.
- Tests for every gap the reviews found, an offline run of the whole
  orchestrator among them; ShellCheck in CI.
2026-11-12 22:32:39 +01:00

9.0 KiB

Order-book collector (Voluntas engineering challenge)

A collector for prediction-market order books (Polymarket, then Kalshi) that emits SBE-encoded updates and per-minute snapshots, built inside the challenge template. The brief is ../website/Engineering Challenge _ Voluntas.html, outside this repo.

Where things are

  • docs/project.md: definition, central bet (the concurrency pipeline), and the reasons behind the design principles. Read before any design discussion.
  • docs/roadmap.md and roadmap/NN. name.md: milestones and each sprint's agreed plan. Read the current sprint file before writing code.
  • docs/adr/: hard-to-reverse decisions (0001: threads, shared-nothing; 0002: the orchestrator, its lanes superseded; 0003: two connections per symbol, first one wins; 0004: the exchange seam, a websocket layer and a loss rule per exchange).
  • docs/notes.md: domain rules and exchange protocol facts (delta semantics, loss detection, message shapes). Read before touching exchange parsing.
  • GLOSSARY.md: the terms (symbol, connection, load, first one wins, duplicate, held, rescued, follow, resync, loss, subscription gap, split, quiet, resubscribe, silent, recycle, sequence number, clock, Snapshot, Delta, lateness, lag, speed).
  • src/core/: what is the same for every exchange (ADR 0004). orchestrator.rs: what is tracked, which connections carry each market, their balancing and their recycling (hourly, a Kalshi connection that stays among the slowest taking the next turn); socket.rs: a connection (read, stamp, decode, route; reconnects when dropped, silent, lost a message or recycled); feed.rs: a book thread's shell (Snapshots, moves, the quiet watch) and the Rule trait; mod.rs: the Exchange, Session, Market and Event traits.
  • src/polymarket/: gamma.rs the fetchers (discovery), wire.rs the exchange's messages, feed.rs the hash rule (the subtle part), mod.rs the exchange and its session.
  • src/kalshi/: listing.rs the fetchers, wire.rs the messages, sign.rs the key and the handshake's signature, feed.rs the clock and rank rule, mod.rs the exchange and its session (the seq check).
  • src/book.rs: the book and fixed point; src/reconcile.rs + src/bin/reconcile.rs: the output check, reading the log's cause lines (src/cause.rs); src/show.rs + src/bin/show.rs: one symbol of the output as text. tests/fixtures/: recordings of the live feeds that the tests replay.
  • scripts/watch.sh [EXCHANGE] [MINUTES]: a run to watch, also the final runs' and the README's quick start (roadmap/08): the collector under timeout -s INT, a live view of its log (scripts/watch.awk, which parses the minute lines' wording: change one, check the other), then reconcile and show. Output and log in watch/.
  • src/latency.rs + scripts/latency/analyse.py: the latency probe (engineering-challenge latency EXCHANGE OUT.csv, until SIGINT: one CSV line per copy of a change on each of 16 connections) and its analysis (speed, persistence, racing, by address, lag); LATENCY.md is the writeup (roadmap/07).
  • src/util/ and sbe_schemas/: the template's SBE machinery, used as given so output stays readable by the template's AlignedSbeReader. Our code is the rest of src/.

Working together

The user drives the pipeline / concurrency architecture: propose and debate it, and let the user settle it. Claude owns the data model.

Writing code

  • Shared-nothing threads: each stage owns its state on one thread or is sharded, and stages talk over channels. Mutexes stay out of the pipeline.
  • Boring and readable: std collections (BTreeMap for a price-ordered side), plain threads and channels. Speed work waits for the profiler, after V1.
  • An abstraction arrives with its second implementation (the exchange trait in M6).
  • Stamp receive time immediately after the socket read, and resync a book (a fresh full book from the exchange) whenever a loss is detected (how depends on the exchange: docs/notes.md).
  • Use GLOSSARY.md's terms: a Snapshot is only the output message; getting a fresh book is a resync.
  • A corner cut for speed carries a comment naming its ceiling and upgrade path.
  • A figure quoted in a doc (README, LATENCY.md, a sprint's Outcome) comes from the tool that checks it, reconcile's output or the log itself, never from watch.sh's view or a count grouped by hand: sprint 08's first README figures had four errors from those.

Environment traps

  • DNS: the exchanges resolve normally on the server (a remote-claude workspace, since 2026-11-09). The Mac's dev network blocks them (its resolver answers 195.186.4.210 / 2001:4d98:4:b10c::1, a Swisscom block page); there the user pins the hosts with sudo ../host.sh on (workspace root, outside this repo; off removes them), rerun when the CDN addresses rotate. A 403 is not DNS (see docs/notes.md, REST from scripts).
  • Write 64 MiB chunks: AlignedSbeReader::open hardcodes 64 MiB (open_with_chunk_size is private), so any other chunk size is unreadable.
  • Only a clean stop flushes the output: SIGINT, SIGTERM or SIGHUP. SIGKILL or SIGALRM leaves the last 64 MiB chunk unwritten. Bound a run with GNU timeout -s INT, then check the output against the log: timeout -s INT 600 ./target/release/engineering-challenge polymarket out.sbe 2> run.log; ./target/release/reconcile out.sbe run.log. With the log, reconcile pairs each mismatch with its cause line and fails only on one left unexplained.
  • Wait on a PID, never on a pgrep -f pattern: until ! pgrep -f "capacity scan" matches the waiting shell's own command line and never ends, and pkill -f PATTERN can kill the shell running it. Take the PID (pgrep -f '^./target/release/...', anchored) and loop on kill -0 $pid.
  • No /usr/bin/time on the server: for a run's peak memory and thread count, read VmHWM and Threads from /proc/<pid>/status while it runs (scripts/watch.sh samples them every 10 s).
  • mawk buffers a pipe's input unless -W interactive: a live view through awk shows nothing until the end (scripts/watch.sh passes it when the awk is mawk, as on the server).
  • A job started with & in sh ignores Ctrl-C: a script that runs the collector in the background must pass SIGINT on itself (watch.sh's trap). To try Ctrl-C without a terminal: (sleep N; printf '\003') | script -qfec "CMD" /dev/null.
  • Polymarket's Gamma answers 429 to runs started close together: two collectors' first scans at once (or a few runs within minutes) got every page refused; the scan retries, but leave a few minutes between runs.
  • *.pem is gitignored globally (~/.gitignore): a test key goes in the source, as src/kalshi/sign.rs does with RFC 8032's and RFC 9500's published test keys.
  • The server is shared and gets overloaded (load up to 92 on 12 cores, 2026-11-11): a starved collector falls behind and gets closed with 1013. Check cat /proc/loadavg before a live run whose numbers are compared, and alternate the builds in short rounds rather than running them back to back.
  • Kalshi needs a key; nothing else does: KALSHI_KEY_ID_FILE and KALSHI_PRIVATE_KEY_FILE name its two files (on the server /run/secrets/kalshi-demo-key-id and kalshi-demo-private-key; never read or print them). The key is Ed25519 or RSA (RSA-PSS-SHA256), told apart by the parsed key. Opening ~100 connections at once gets some handshakes refused with 429; each retries after 1 s and gets in.
  • cargo keeps a build whose source got an older mtime: a script that restores a file from a copy (shutil.move, cp -p) must touch it, or the next cargo test runs the old binary (a mutation check did, and a "flaky" test was only the last mutant).
  • Live data beats the docs: Polymarket's market channel has rules its docs don't state (docs/notes.md); check a protocol assumption on a recording of the real feed (tests/fixtures/record.py) before relying on it.
  • The workspace can't choose what the exchange sees: a NAT rewrites every connection's source port (local 41000 left as 36996, then as 37086 a minute later; curl --local-port P https://ifconfig.me/all.json), and there is no IPv6 route (connections to an IPv6 address fail with "Network is unreachable"). The probe uses IPv4 addresses only.
  • Python checks run through uvx: no ruff or pyflakes is installed; scripts/ci.sh runs uvx ruff@0.15.0 (config in scripts/latency/ruff.toml).

Workflow differences

A job-challenge repo cloned from the template: no Gitea, Woodpecker or release pipeline.

  • Kept: docs/, roadmap/ sprint files, feat/* branches, and scripts/ci.sh (fmt, clippy -D warnings, tests).
  • Dropped: the dependency audit (supply-chain checks are for deployed products), Woodpecker, release.sh, versioning and changelog, rc images, and the prod / dev branches. CI targets application quality, not prod-grade.
  • Branches: main plays the role of dev; feat/* branches are squashed into main.