This repository has been archived on 2026-11-16. You can view files and clone it. You cannot open issues or pull requests or push a commit.
Files
orderbook-collector/CLAUDE.md
T
pascal adb7fee8d8 feat(polymarket): hash-verified order books for a subset of symbols
Collects the order books of Polymarket's 20 busiest markets (40 symbols)
into the template's SBE format: a Delta per level change, plus Snapshots on
subscribe, every minute, and whenever a fresh book corrects ours. Every
change is checked against Polymarket's own book hash; a failed check
resyncs the symbol on the open socket, and the log counts losses apart from
the few ms our own resync hides. Runs until SIGINT, which closes the socket
and flushes the output; reconnects when the server drops it. A `reconcile`
binary checks an output file against itself.
2026-11-04 17:10:17 +01:00

4.3 KiB

Order-book collector (Voluntas engineering challenge)

A collector for prediction-market order books (Polymarket, then Kalshi) that emits SBE-encoded updates and per-minute snapshots, built inside the challenge template. The brief is ../website/Engineering Challenge _ Voluntas.html, outside this repo.

Where things are

  • docs/project.md: definition, central bet (the concurrency pipeline), and the reasons behind the design principles. Read before any design discussion.
  • docs/roadmap.md and roadmap/NN. name.md: milestones and each sprint's agreed plan. Read the current sprint file before writing code.
  • docs/adr/: hard-to-reverse decisions (0001: threads, shared-nothing).
  • docs/notes.md: domain rules and exchange protocol facts (delta semantics, loss detection, message shapes). Read before touching exchange parsing.
  • GLOSSARY.md: the terms (symbol, resync, loss, resync gap, Snapshot, Delta).
  • src/polymarket/feed.rs: the book thread and its loss rules (the subtle part); src/polymarket/socket.rs: the socket thread; src/book.rs: the book and fixed point; src/reconcile.rs + src/bin/reconcile.rs: the output check. tests/fixtures/: a 90 s recording of the live feed the tests replay.
  • src/util/ and sbe_schemas/: the template's SBE machinery, used as given so output stays readable by the template's AlignedSbeReader. Our code is the rest of src/.

Working together

The user drives the pipeline / concurrency architecture: propose and debate it, and let the user settle it. Claude owns the data model.

Writing code

  • Shared-nothing threads: each stage owns its state on one thread or is sharded, and stages talk over channels. Mutexes stay out of the pipeline.
  • Boring and readable: std collections (BTreeMap for a price-ordered side), plain threads and channels. Speed work waits for the profiler, after V1.
  • An abstraction arrives with its second implementation (the exchange trait in M6).
  • Stamp receive time immediately after the socket read, and resync a book (a fresh full book from the exchange) whenever a loss is detected (how depends on the exchange: docs/notes.md).
  • Use GLOSSARY.md's terms: a Snapshot is only the output message; getting a fresh book is a resync.
  • A corner cut for speed carries a comment naming its ceiling and upgrade path.

Environment traps

  • Polymarket and Kalshi are DNS-blocked on the dev network: the local resolver answers 195.186.4.210 / 2001:4d98:4:b10c::1 (a Swisscom block page). The user pins the hosts with sudo ../host.sh on (workspace root, outside this repo; off removes them). A timeout or the block address again means the CDN addresses rotated: ask the user to rerun it. A host not yet in host.sh's HOSTS list fails the same way: add it there and ask the user to rerun it. A 403 is not DNS (see docs/notes.md, REST from scripts).
  • Write 64 MiB chunks: AlignedSbeReader::open hardcodes 64 MiB (open_with_chunk_size is private), so any other chunk size is unreadable.
  • The collector runs until SIGINT, and only SIGINT flushes its output. macOS has no timeout, and a perl alarm kill (SIGALRM) leaves the last 64 MiB chunk unwritten. Bound a run with SIGINT, then check the output: ./target/release/engineering-challenge out.sbe 2> run.log & pid=$!; sleep 600; kill -INT $pid; wait $pid; ./target/release/reconcile out.sbe. reconcile counts resync gaps as mismatches too: compare its list with the log's "loss" and "resync gap" lines.
  • Live data beats the docs: Polymarket's market channel has rules its docs don't state (docs/notes.md); check a protocol assumption on a recording of the real feed (tests/fixtures/record.py) before relying on it.

Workflow differences

A job-challenge repo cloned from the template: no Gitea, Woodpecker or release pipeline.

  • Kept: docs/, roadmap/ sprint files, feat/* branches, and scripts/ci.sh (fmt, clippy -D warnings, tests).
  • Dropped: the dependency audit (supply-chain checks are for deployed products), Woodpecker, release.sh, versioning and changelog, rc images, and the prod / dev branches. CI targets application quality, not prod-grade.
  • Branches: main plays the role of dev; feat/* branches are squashed into main.