This repository has been archived on 2026-11-16. You can view files and clone it. You cannot open issues or pull requests or push a commit.
Files
pascal 00202bbc51 docs: correct the README's claims, state the limits, and CI on macOS
The README said the quiet rule holds for the quietest markets (it can't
judge one under 3 changes a minute) and that all four extensions were
done (two of the brief's latency ideas were not tried); LATENCY.md said
the slow recycle was not seen to fire live, two lines after the run where
it did. Limits gains the process going down, the quiet rule's blind
spots and Kalshi's skipped-change case.

CLAUDE.md loses the home network's details and the server's key paths,
of no use to a reader of the public repo.

scripts/test_watch.sh no longer puts a here-document inside `$( )`:
bash 3.2, which is sh on macOS, stopped at an apostrophe there and
scripts/ci.sh ended on a syntax error.
2026-11-12 23:17:10 +01:00

151 lines
8.7 KiB
Markdown

# Order-book collector (Voluntas engineering challenge)
A collector for prediction-market order books (Polymarket, then Kalshi) that
emits SBE-encoded updates and per-minute snapshots, built inside the challenge
template. The brief is https://voluntas.io/engineering-challenge.
## Where things are
- `docs/project.md`: definition, central bet (the concurrency pipeline), and the
reasons behind the design principles. Read before any design discussion.
- `docs/roadmap.md` and `roadmap/NN. name.md`: milestones and each sprint's
agreed plan. Read the current sprint file before writing code.
- `docs/adr/`: hard-to-reverse decisions (0001: threads, shared-nothing;
0002: the orchestrator, its lanes superseded; 0003: two connections per
symbol, first one wins; 0004: the exchange seam, a websocket layer and a
loss rule per exchange).
- `docs/notes.md`: domain rules and exchange protocol facts (delta semantics,
loss detection, message shapes). Read before touching exchange parsing.
- `GLOSSARY.md`: the terms (symbol, connection, load, first one wins,
duplicate, held, rescued, follow, resync, loss, subscription gap, split,
quiet, resubscribe, silent, recycle, sequence number, clock, Snapshot,
Delta, lateness, lag, speed).
- `src/core/`: what is the same for every exchange (ADR 0004).
`orchestrator.rs`: what is tracked, which connections carry each market,
their balancing and their recycling (hourly, a Kalshi connection that
stays among the slowest taking the next turn); `socket.rs`: a connection (read,
stamp, decode, route; reconnects when dropped, silent, lost a message or
recycled); `feed.rs`: a book thread's shell (Snapshots, moves, the quiet
watch) and the `Rule` trait; `mod.rs`: the `Exchange`, `Session`, `Market`
and `Event` traits.
- `src/polymarket/`: `gamma.rs` the fetchers (discovery), `wire.rs` the
exchange's messages, `feed.rs` the hash rule (the subtle part),
`mod.rs` the exchange and its session.
- `src/kalshi/`: `listing.rs` the fetchers, `wire.rs` the messages,
`sign.rs` the key and the handshake's signature, `feed.rs` the clock and
rank rule, `mod.rs` the exchange and its session (the `seq` check).
- `src/book.rs`: the book and fixed point; `src/reconcile.rs` +
`src/bin/reconcile.rs`: the output check, reading the log's cause lines
(`src/cause.rs`); `src/show.rs` + `src/bin/show.rs`: one symbol of the
output as text. `tests/fixtures/`: recordings of the live feeds that the
tests replay.
- `scripts/watch.sh [EXCHANGE] [MINUTES]`: a run to watch, also the final
runs' and the README's quick start (roadmap/08): the collector under
`timeout -s INT`, a live view of its log (`scripts/watch.awk`, which
parses the minute lines' wording: change one, check the other), then
`reconcile` and `show`. Output and log in `watch/`.
- `src/latency.rs` + `scripts/latency/analyse.py`: the latency probe
(`engineering-challenge latency EXCHANGE OUT.csv`, until SIGINT: one CSV
line per copy of a change on each of 16 connections) and its analysis
(speed, persistence, racing, by address, lag); `LATENCY.md` is the
writeup (roadmap/07).
- `src/util/` and `sbe_schemas/`: the template's SBE machinery, used as given so
output stays readable by the template's `AlignedSbeReader`. Our code is the
rest of `src/`.
## Working together
The user drives the pipeline / concurrency architecture: propose and debate it,
and let the user settle it. Claude owns the data model.
## Writing code
- Shared-nothing threads: each stage owns its state on one thread or is sharded,
and stages talk over channels. Mutexes stay out of the pipeline.
- Boring and readable: std collections (`BTreeMap` for a price-ordered side),
plain threads and channels. Speed work waits for the profiler, after V1.
- An abstraction arrives with its second implementation (the exchange trait in
M6).
- Stamp receive time immediately after the socket read, and resync a book (a
fresh full book from the exchange) whenever a loss is detected (how depends
on the exchange: `docs/notes.md`).
- Use `GLOSSARY.md`'s terms: a **Snapshot** is only the output message; getting
a fresh book is a **resync**.
- A corner cut for speed carries a comment naming its ceiling and upgrade path.
- A figure quoted in a doc (README, `LATENCY.md`, a sprint's Outcome) comes
from the tool that checks it, `reconcile`'s output or the log itself,
never from `watch.sh`'s view or a count grouped by hand: sprint 08's
first README figures had four errors from those.
## Environment traps
- **DNS: a network can block the exchanges** (its resolver answers a block
page's address): check what the hosts resolve to before blaming the code.
A 403 is not DNS (see `docs/notes.md`, REST from scripts).
- **Write 64 MiB chunks**: `AlignedSbeReader::open` hardcodes 64 MiB
(`open_with_chunk_size` is private), so any other chunk size is unreadable.
- **Only a clean stop flushes the output**: SIGINT, SIGTERM or SIGHUP. SIGKILL
or SIGALRM leaves the last 64 MiB chunk unwritten. Bound a run with GNU
`timeout -s INT`, then check the output against the log:
`timeout -s INT 600 ./target/release/engineering-challenge polymarket out.sbe 2> run.log;
./target/release/reconcile out.sbe run.log`. With the log, `reconcile`
pairs each mismatch with its cause line and fails only on one left
unexplained.
- **Wait on a PID, never on a `pgrep -f` pattern**: `until ! pgrep -f
"capacity scan"` matches the waiting shell's own command line and never
ends, and `pkill -f PATTERN` can kill the shell running it. Take the PID
(`pgrep -f '^./target/release/...'`, anchored) and loop on `kill -0 $pid`.
- **No `/usr/bin/time` on the server**: for a run's peak memory and thread
count, read `VmHWM` and `Threads` from `/proc/<pid>/status` while it runs
(`scripts/watch.sh` samples them every 10 s).
- **mawk buffers a pipe's input** unless `-W interactive`: a live view
through `awk` shows nothing until the end (`scripts/watch.sh` passes it
when the awk is mawk, as on the server).
- **A job started with `&` in `sh` ignores Ctrl-C**: a script that runs the
collector in the background must pass SIGINT on itself (`watch.sh`'s
trap). To try Ctrl-C without a terminal: `(sleep N; printf '\003') |
script -qfec "CMD" /dev/null`.
- **Polymarket's Gamma answers 429 to runs started close together**: two
collectors' first scans at once (or a few runs within minutes) got every
page refused; the scan retries, but leave a few minutes between runs.
- **`*.pem` is gitignored globally** (`~/.gitignore`): a test key goes in
the source, as `src/kalshi/sign.rs` does with RFC 8032's and RFC 9500's
published test keys.
- **The server is shared and gets overloaded** (load up to 92 on 12 cores,
2026-11-11): a starved collector falls behind and gets closed with 1013.
Check `cat /proc/loadavg` before a live run whose numbers are compared, and
alternate the builds in short rounds rather than running them back to back.
- **Kalshi needs a key; nothing else does**: `KALSHI_KEY_ID_FILE` and
`KALSHI_PRIVATE_KEY_FILE` name its two files (never read or print
them). The key is Ed25519 or RSA (RSA-PSS-SHA256), told apart
by the parsed key. Opening ~100 connections at once gets some handshakes
refused with 429; each retries after 1 s and gets in.
- **cargo keeps a build whose source got an older mtime**: a script that
restores a file from a copy (`shutil.move`, `cp -p`) must `touch` it, or
the next `cargo test` runs the old binary (a mutation check did, and a
"flaky" test was only the last mutant).
- **Live data beats the docs**: Polymarket's market channel has rules its
docs don't state (`docs/notes.md`); check a protocol assumption on a
recording of the real feed (`tests/fixtures/record.py`) before relying on it.
- **The workspace can't choose what the exchange sees**: a NAT rewrites
every connection's source port (local 41000 left as 36996, then as 37086
a minute later; `curl --local-port P https://ifconfig.me/all.json`), and
there is no IPv6 route (connections to an IPv6 address fail with "Network
is unreachable"). The probe uses IPv4 addresses only.
- **Python checks run through `uvx`**: no ruff or pyflakes is installed;
`scripts/ci.sh` runs `uvx ruff@0.15.0` (config in
`scripts/latency/ruff.toml`).
## Workflow differences
A job-challenge repo cloned from the template: no Gitea, Woodpecker or release
pipeline.
- **Kept**: `docs/`, `roadmap/` sprint files, `feat/*` branches, and
`scripts/ci.sh` (fmt, clippy `-D warnings`, tests).
- **Dropped**: the dependency audit (supply-chain checks are for deployed
products), Woodpecker, `release.sh`, versioning and changelog, rc images, and
the `prod` / `dev` branches. CI targets application quality, not prod-grade.
- **Branches**: `main` plays the role of `dev`; `feat/*` branches are squashed
into `main`.