This repository has been archived on 2026-11-16. You can view files and clone it. You cannot open issues or pull requests or push a commit.
Files
orderbook-collector/CLAUDE.md
T
pascal 0e687e7eba feat(polymarket): skeleton and market-channel connectivity spike
Replaces the template's demo main.rs with a spike that connects to
Polymarket's public market channel, subscribes to one market, and logs every
frame with its receive time. Adds scripts/ci.sh (fmt, clippy, tests), the
project CLAUDE.md, ADR 0001 (OS threads, shared-nothing), and the market
channel's message shapes in docs/notes.md: no sequence number, a book hash on
every change, mirrored Yes/No books.
2026-11-04 14:47:14 +01:00

3.1 KiB

Order-book collector (Voluntas engineering challenge)

A collector for prediction-market order books (Polymarket, then Kalshi) that emits SBE-encoded updates and per-minute snapshots, built inside the challenge template. The brief is ../website/Engineering Challenge _ Voluntas.html, outside this repo.

Where things are

  • docs/project.md: definition, central bet (the concurrency pipeline), and the reasons behind the design principles. Read before any design discussion.
  • docs/roadmap.md and roadmap/NN. name.md: milestones and each sprint's agreed plan. Read the current sprint file before writing code.
  • docs/adr/: hard-to-reverse decisions (0001: threads, shared-nothing).
  • docs/notes.md: domain rules and exchange protocol facts (delta semantics, loss detection, message shapes). Read before touching exchange parsing.
  • src/util/ and sbe_schemas/: the template's SBE machinery, used as given so output stays readable by the template's AlignedSbeReader. Our code is the rest of src/.

Working together

The user drives the pipeline / concurrency architecture: propose and debate it, and let the user settle it. Claude owns the data model.

Writing code

  • Shared-nothing threads: each stage owns its state on one thread or is sharded, and stages talk over channels. Mutexes stay out of the pipeline.
  • Boring and readable: std collections (BTreeMap for a price-ordered side), plain threads and channels. Speed work waits for the profiler, after V1.
  • An abstraction arrives with its second implementation (the exchange trait in M6).
  • Stamp receive time immediately after the socket read, and rebuild a book from a fresh snapshot whenever a lost message is detected (how depends on the exchange: docs/notes.md).
  • A corner cut for speed carries a comment naming its ceiling and upgrade path.

Environment traps

  • Polymarket and Kalshi are DNS-blocked on the dev network: the local resolver answers 195.186.4.210 / 2001:4d98:4:b10c::1 (a Swisscom block page). The user pins the hosts with sudo ../host.sh on (workspace root, outside this repo; off removes them). A timeout or the block address again means the CDN addresses rotated: ask the user to rerun it.
  • Write 64 MiB chunks: AlignedSbeReader::open hardcodes 64 MiB (open_with_chunk_size is private), so any other chunk size is unreadable.
  • The binary runs until the server closes the socket. macOS has no timeout; bound a run with perl -e 'alarm 10; exec @ARGV' ./target/debug/engineering-challenge.

Workflow differences

A job-challenge repo cloned from the template: no Gitea, Woodpecker or release pipeline.

  • Kept: docs/, roadmap/ sprint files, feat/* branches, and scripts/ci.sh (fmt, clippy -D warnings, tests).
  • Dropped: the dependency audit (supply-chain checks are for deployed products), Woodpecker, release.sh, versioning and changelog, rc images, and the prod / dev branches. CI targets application quality, not prod-grade.
  • Branches: main plays the role of dev; feat/* branches are squashed into main.