Archived
The README said the quiet rule holds for the quietest markets (it can't judge one under 3 changes a minute) and that all four extensions were done (two of the brief's latency ideas were not tried); LATENCY.md said the slow recycle was not seen to fire live, two lines after the run where it did. Limits gains the process going down, the quiet rule's blind spots and Kalshi's skipped-change case. CLAUDE.md loses the home network's details and the server's key paths, of no use to a reader of the public repo. scripts/test_watch.sh no longer puts a here-document inside `$( )`: bash 3.2, which is sh on macOS, stopped at an apostrophe there and scripts/ci.sh ended on a syntax error.
151 lines
8.7 KiB
Markdown
151 lines
8.7 KiB
Markdown
# Order-book collector (Voluntas engineering challenge)
|
|
|
|
A collector for prediction-market order books (Polymarket, then Kalshi) that
|
|
emits SBE-encoded updates and per-minute snapshots, built inside the challenge
|
|
template. The brief is https://voluntas.io/engineering-challenge.
|
|
|
|
## Where things are
|
|
|
|
- `docs/project.md`: definition, central bet (the concurrency pipeline), and the
|
|
reasons behind the design principles. Read before any design discussion.
|
|
- `docs/roadmap.md` and `roadmap/NN. name.md`: milestones and each sprint's
|
|
agreed plan. Read the current sprint file before writing code.
|
|
- `docs/adr/`: hard-to-reverse decisions (0001: threads, shared-nothing;
|
|
0002: the orchestrator, its lanes superseded; 0003: two connections per
|
|
symbol, first one wins; 0004: the exchange seam, a websocket layer and a
|
|
loss rule per exchange).
|
|
- `docs/notes.md`: domain rules and exchange protocol facts (delta semantics,
|
|
loss detection, message shapes). Read before touching exchange parsing.
|
|
- `GLOSSARY.md`: the terms (symbol, connection, load, first one wins,
|
|
duplicate, held, rescued, follow, resync, loss, subscription gap, split,
|
|
quiet, resubscribe, silent, recycle, sequence number, clock, Snapshot,
|
|
Delta, lateness, lag, speed).
|
|
- `src/core/`: what is the same for every exchange (ADR 0004).
|
|
`orchestrator.rs`: what is tracked, which connections carry each market,
|
|
their balancing and their recycling (hourly, a Kalshi connection that
|
|
stays among the slowest taking the next turn); `socket.rs`: a connection (read,
|
|
stamp, decode, route; reconnects when dropped, silent, lost a message or
|
|
recycled); `feed.rs`: a book thread's shell (Snapshots, moves, the quiet
|
|
watch) and the `Rule` trait; `mod.rs`: the `Exchange`, `Session`, `Market`
|
|
and `Event` traits.
|
|
- `src/polymarket/`: `gamma.rs` the fetchers (discovery), `wire.rs` the
|
|
exchange's messages, `feed.rs` the hash rule (the subtle part),
|
|
`mod.rs` the exchange and its session.
|
|
- `src/kalshi/`: `listing.rs` the fetchers, `wire.rs` the messages,
|
|
`sign.rs` the key and the handshake's signature, `feed.rs` the clock and
|
|
rank rule, `mod.rs` the exchange and its session (the `seq` check).
|
|
- `src/book.rs`: the book and fixed point; `src/reconcile.rs` +
|
|
`src/bin/reconcile.rs`: the output check, reading the log's cause lines
|
|
(`src/cause.rs`); `src/show.rs` + `src/bin/show.rs`: one symbol of the
|
|
output as text. `tests/fixtures/`: recordings of the live feeds that the
|
|
tests replay.
|
|
- `scripts/watch.sh [EXCHANGE] [MINUTES]`: a run to watch, also the final
|
|
runs' and the README's quick start (roadmap/08): the collector under
|
|
`timeout -s INT`, a live view of its log (`scripts/watch.awk`, which
|
|
parses the minute lines' wording: change one, check the other), then
|
|
`reconcile` and `show`. Output and log in `watch/`.
|
|
- `src/latency.rs` + `scripts/latency/analyse.py`: the latency probe
|
|
(`engineering-challenge latency EXCHANGE OUT.csv`, until SIGINT: one CSV
|
|
line per copy of a change on each of 16 connections) and its analysis
|
|
(speed, persistence, racing, by address, lag); `LATENCY.md` is the
|
|
writeup (roadmap/07).
|
|
- `src/util/` and `sbe_schemas/`: the template's SBE machinery, used as given so
|
|
output stays readable by the template's `AlignedSbeReader`. Our code is the
|
|
rest of `src/`.
|
|
|
|
## Working together
|
|
|
|
The user drives the pipeline / concurrency architecture: propose and debate it,
|
|
and let the user settle it. Claude owns the data model.
|
|
|
|
## Writing code
|
|
|
|
- Shared-nothing threads: each stage owns its state on one thread or is sharded,
|
|
and stages talk over channels. Mutexes stay out of the pipeline.
|
|
- Boring and readable: std collections (`BTreeMap` for a price-ordered side),
|
|
plain threads and channels. Speed work waits for the profiler, after V1.
|
|
- An abstraction arrives with its second implementation (the exchange trait in
|
|
M6).
|
|
- Stamp receive time immediately after the socket read, and resync a book (a
|
|
fresh full book from the exchange) whenever a loss is detected (how depends
|
|
on the exchange: `docs/notes.md`).
|
|
- Use `GLOSSARY.md`'s terms: a **Snapshot** is only the output message; getting
|
|
a fresh book is a **resync**.
|
|
- A corner cut for speed carries a comment naming its ceiling and upgrade path.
|
|
- A figure quoted in a doc (README, `LATENCY.md`, a sprint's Outcome) comes
|
|
from the tool that checks it, `reconcile`'s output or the log itself,
|
|
never from `watch.sh`'s view or a count grouped by hand: sprint 08's
|
|
first README figures had four errors from those.
|
|
|
|
## Environment traps
|
|
|
|
- **DNS: a network can block the exchanges** (its resolver answers a block
|
|
page's address): check what the hosts resolve to before blaming the code.
|
|
A 403 is not DNS (see `docs/notes.md`, REST from scripts).
|
|
- **Write 64 MiB chunks**: `AlignedSbeReader::open` hardcodes 64 MiB
|
|
(`open_with_chunk_size` is private), so any other chunk size is unreadable.
|
|
- **Only a clean stop flushes the output**: SIGINT, SIGTERM or SIGHUP. SIGKILL
|
|
or SIGALRM leaves the last 64 MiB chunk unwritten. Bound a run with GNU
|
|
`timeout -s INT`, then check the output against the log:
|
|
`timeout -s INT 600 ./target/release/engineering-challenge polymarket out.sbe 2> run.log;
|
|
./target/release/reconcile out.sbe run.log`. With the log, `reconcile`
|
|
pairs each mismatch with its cause line and fails only on one left
|
|
unexplained.
|
|
- **Wait on a PID, never on a `pgrep -f` pattern**: `until ! pgrep -f
|
|
"capacity scan"` matches the waiting shell's own command line and never
|
|
ends, and `pkill -f PATTERN` can kill the shell running it. Take the PID
|
|
(`pgrep -f '^./target/release/...'`, anchored) and loop on `kill -0 $pid`.
|
|
- **No `/usr/bin/time` on the server**: for a run's peak memory and thread
|
|
count, read `VmHWM` and `Threads` from `/proc/<pid>/status` while it runs
|
|
(`scripts/watch.sh` samples them every 10 s).
|
|
- **mawk buffers a pipe's input** unless `-W interactive`: a live view
|
|
through `awk` shows nothing until the end (`scripts/watch.sh` passes it
|
|
when the awk is mawk, as on the server).
|
|
- **A job started with `&` in `sh` ignores Ctrl-C**: a script that runs the
|
|
collector in the background must pass SIGINT on itself (`watch.sh`'s
|
|
trap). To try Ctrl-C without a terminal: `(sleep N; printf '\003') |
|
|
script -qfec "CMD" /dev/null`.
|
|
- **Polymarket's Gamma answers 429 to runs started close together**: two
|
|
collectors' first scans at once (or a few runs within minutes) got every
|
|
page refused; the scan retries, but leave a few minutes between runs.
|
|
- **`*.pem` is gitignored globally** (`~/.gitignore`): a test key goes in
|
|
the source, as `src/kalshi/sign.rs` does with RFC 8032's and RFC 9500's
|
|
published test keys.
|
|
- **The server is shared and gets overloaded** (load up to 92 on 12 cores,
|
|
2026-11-11): a starved collector falls behind and gets closed with 1013.
|
|
Check `cat /proc/loadavg` before a live run whose numbers are compared, and
|
|
alternate the builds in short rounds rather than running them back to back.
|
|
- **Kalshi needs a key; nothing else does**: `KALSHI_KEY_ID_FILE` and
|
|
`KALSHI_PRIVATE_KEY_FILE` name its two files (never read or print
|
|
them). The key is Ed25519 or RSA (RSA-PSS-SHA256), told apart
|
|
by the parsed key. Opening ~100 connections at once gets some handshakes
|
|
refused with 429; each retries after 1 s and gets in.
|
|
- **cargo keeps a build whose source got an older mtime**: a script that
|
|
restores a file from a copy (`shutil.move`, `cp -p`) must `touch` it, or
|
|
the next `cargo test` runs the old binary (a mutation check did, and a
|
|
"flaky" test was only the last mutant).
|
|
- **Live data beats the docs**: Polymarket's market channel has rules its
|
|
docs don't state (`docs/notes.md`); check a protocol assumption on a
|
|
recording of the real feed (`tests/fixtures/record.py`) before relying on it.
|
|
- **The workspace can't choose what the exchange sees**: a NAT rewrites
|
|
every connection's source port (local 41000 left as 36996, then as 37086
|
|
a minute later; `curl --local-port P https://ifconfig.me/all.json`), and
|
|
there is no IPv6 route (connections to an IPv6 address fail with "Network
|
|
is unreachable"). The probe uses IPv4 addresses only.
|
|
- **Python checks run through `uvx`**: no ruff or pyflakes is installed;
|
|
`scripts/ci.sh` runs `uvx ruff@0.15.0` (config in
|
|
`scripts/latency/ruff.toml`).
|
|
|
|
## Workflow differences
|
|
|
|
A job-challenge repo cloned from the template: no Gitea, Woodpecker or release
|
|
pipeline.
|
|
|
|
- **Kept**: `docs/`, `roadmap/` sprint files, `feat/*` branches, and
|
|
`scripts/ci.sh` (fmt, clippy `-D warnings`, tests).
|
|
- **Dropped**: the dependency audit (supply-chain checks are for deployed
|
|
products), Woodpecker, `release.sh`, versioning and changelog, rc images, and
|
|
the `prod` / `dev` branches. CI targets application quality, not prod-grade.
|
|
- **Branches**: `main` plays the role of `dev`; `feat/*` branches are squashed
|
|
into `main`.
|