Archived
Replaces the template's demo main.rs with a spike that connects to Polymarket's public market channel, subscribes to one market, and logs every frame with its receive time. Adds scripts/ci.sh (fmt, clippy, tests), the project CLAUDE.md, ADR 0001 (OS threads, shared-nothing), and the market channel's message shapes in docs/notes.md: no sequence number, a book hash on every change, mirrored Yes/No books.
3.1 KiB
3.1 KiB
Order-book collector (Voluntas engineering challenge)
A collector for prediction-market order books (Polymarket, then Kalshi) that
emits SBE-encoded updates and per-minute snapshots, built inside the challenge
template. The brief is ../website/Engineering Challenge _ Voluntas.html,
outside this repo.
Where things are
docs/project.md: definition, central bet (the concurrency pipeline), and the reasons behind the design principles. Read before any design discussion.docs/roadmap.mdandroadmap/NN. name.md: milestones and each sprint's agreed plan. Read the current sprint file before writing code.docs/adr/: hard-to-reverse decisions (0001: threads, shared-nothing).docs/notes.md: domain rules and exchange protocol facts (delta semantics, loss detection, message shapes). Read before touching exchange parsing.src/util/andsbe_schemas/: the template's SBE machinery, used as given so output stays readable by the template'sAlignedSbeReader. Our code is the rest ofsrc/.
Working together
The user drives the pipeline / concurrency architecture: propose and debate it, and let the user settle it. Claude owns the data model.
Writing code
- Shared-nothing threads: each stage owns its state on one thread or is sharded, and stages talk over channels. Mutexes stay out of the pipeline.
- Boring and readable: std collections (
BTreeMapfor a price-ordered side), plain threads and channels. Speed work waits for the profiler, after V1. - An abstraction arrives with its second implementation (the exchange trait in M6).
- Stamp receive time immediately after the socket read, and rebuild a book from
a fresh snapshot whenever a lost message is detected (how depends on the
exchange:
docs/notes.md). - A corner cut for speed carries a comment naming its ceiling and upgrade path.
Environment traps
- Polymarket and Kalshi are DNS-blocked on the dev network: the local
resolver answers
195.186.4.210/2001:4d98:4:b10c::1(a Swisscom block page). The user pins the hosts withsudo ../host.sh on(workspace root, outside this repo;offremoves them). A timeout or the block address again means the CDN addresses rotated: ask the user to rerun it. - Write 64 MiB chunks:
AlignedSbeReader::openhardcodes 64 MiB (open_with_chunk_sizeis private), so any other chunk size is unreadable. - The binary runs until the server closes the socket. macOS has no
timeout; bound a run withperl -e 'alarm 10; exec @ARGV' ./target/debug/engineering-challenge.
Workflow differences
A job-challenge repo cloned from the template: no Gitea, Woodpecker or release pipeline.
- Kept:
docs/,roadmap/sprint files,feat/*branches, andscripts/ci.sh(fmt, clippy-D warnings, tests). - Dropped: the dependency audit (supply-chain checks are for deployed
products), Woodpecker,
release.sh, versioning and changelog, rc images, and theprod/devbranches. CI targets application quality, not prod-grade. - Branches:
mainplays the role ofdev;feat/*branches are squashed intomain.