Archived
The exchange can fail without telling us, and first one wins hides a subscription gone quiet on one of a symbol's two connections. - A connection that receives nothing for 30 s, not even the keepalive's answer, reconnects. - Each book thread counts a symbol's changes per connection every minute: a subscription with none against three or more on the other is resubscribed, and moved to another connection if it stays quiet. - The orchestrator recycles one connection at a time, each about hourly, and never while another connection is down or just back. - A reconnect's opening book of a symbol unchanged for 10 s is judged like a resync's answer: one newer than ours that differs shows a symbol quiet on both connections, written as a Snapshot and logged as a loss.
100 lines
5.5 KiB
Markdown
100 lines
5.5 KiB
Markdown
# Order-book collector (Voluntas engineering challenge)
|
|
|
|
A collector for prediction-market order books (Polymarket, then Kalshi) that
|
|
emits SBE-encoded updates and per-minute snapshots, built inside the challenge
|
|
template. The brief is `../website/Engineering Challenge _ Voluntas.html`,
|
|
outside this repo.
|
|
|
|
## Where things are
|
|
|
|
- `docs/project.md`: definition, central bet (the concurrency pipeline), and the
|
|
reasons behind the design principles. Read before any design discussion.
|
|
- `docs/roadmap.md` and `roadmap/NN. name.md`: milestones and each sprint's
|
|
agreed plan. Read the current sprint file before writing code.
|
|
- `docs/adr/`: hard-to-reverse decisions (0001: threads, shared-nothing;
|
|
0002: the orchestrator, its lanes superseded; 0003: two connections per
|
|
symbol, first one wins).
|
|
- `docs/notes.md`: domain rules and exchange protocol facts (delta semantics,
|
|
loss detection, message shapes). Read before touching exchange parsing.
|
|
- `GLOSSARY.md`: the terms (symbol, connection, load, first one wins,
|
|
duplicate, held, rescued, follow, resync, loss, subscription gap, split,
|
|
quiet, resubscribe, silent, recycle, Snapshot, Delta).
|
|
- `src/polymarket/orchestrator.rs`: what is tracked, which connections carry
|
|
each market, their balancing and their recycling; `src/polymarket/gamma.rs`: the fetchers
|
|
(discovery); `src/polymarket/socket.rs`: a connection (read, stamp, decode,
|
|
route; reconnects when dropped, silent or recycled);
|
|
`src/polymarket/wire.rs`: the exchange's messages;
|
|
`src/polymarket/feed.rs`: a book thread, its loss rules (the subtle
|
|
part) and its watch for quiet subscriptions; `src/book.rs`: the book and fixed point; `src/reconcile.rs` +
|
|
`src/bin/reconcile.rs`: the output check, reading the log's cause lines
|
|
(`src/cause.rs`). `tests/fixtures/`: recordings of the live feed, on one and two
|
|
connections, that the tests replay.
|
|
- `src/util/` and `sbe_schemas/`: the template's SBE machinery, used as given so
|
|
output stays readable by the template's `AlignedSbeReader`. Our code is the
|
|
rest of `src/`.
|
|
|
|
## Working together
|
|
|
|
The user drives the pipeline / concurrency architecture: propose and debate it,
|
|
and let the user settle it. Claude owns the data model.
|
|
|
|
## Writing code
|
|
|
|
- Shared-nothing threads: each stage owns its state on one thread or is sharded,
|
|
and stages talk over channels. Mutexes stay out of the pipeline.
|
|
- Boring and readable: std collections (`BTreeMap` for a price-ordered side),
|
|
plain threads and channels. Speed work waits for the profiler, after V1.
|
|
- An abstraction arrives with its second implementation (the exchange trait in
|
|
M6).
|
|
- Stamp receive time immediately after the socket read, and resync a book (a
|
|
fresh full book from the exchange) whenever a loss is detected (how depends
|
|
on the exchange: `docs/notes.md`).
|
|
- Use `GLOSSARY.md`'s terms: a **Snapshot** is only the output message; getting
|
|
a fresh book is a **resync**.
|
|
- A corner cut for speed carries a comment naming its ceiling and upgrade path.
|
|
|
|
## Environment traps
|
|
|
|
- **DNS: the exchanges resolve normally on the server** (a remote-claude
|
|
workspace, since 2026-11-09). The Mac's dev network blocks them (its resolver
|
|
answers `195.186.4.210` / `2001:4d98:4:b10c::1`, a Swisscom block page);
|
|
there the user pins the hosts with `sudo ../host.sh on` (workspace root,
|
|
outside this repo; `off` removes them), rerun when the CDN addresses rotate.
|
|
A 403 is not DNS (see `docs/notes.md`, REST from scripts).
|
|
- **Write 64 MiB chunks**: `AlignedSbeReader::open` hardcodes 64 MiB
|
|
(`open_with_chunk_size` is private), so any other chunk size is unreadable.
|
|
- **Only a clean stop flushes the output**: SIGINT, SIGTERM or SIGHUP. SIGKILL
|
|
or SIGALRM leaves the last 64 MiB chunk unwritten. Bound a run with GNU
|
|
`timeout -s INT`, then check the output against the log:
|
|
`timeout -s INT 600 ./target/release/engineering-challenge out.sbe 2> run.log;
|
|
./target/release/reconcile out.sbe run.log`. With the log, `reconcile`
|
|
pairs each mismatch with its cause line and fails only on one left
|
|
unexplained.
|
|
- **Wait on a PID, never on a `pgrep -f` pattern**: `until ! pgrep -f
|
|
"capacity scan"` matches the waiting shell's own command line and never
|
|
ends, and `pkill -f PATTERN` can kill the shell running it. Take the PID
|
|
(`pgrep -f '^./target/release/...'`, anchored) and loop on `kill -0 $pid`.
|
|
- **No `/usr/bin/time` on the server**: for a run's peak memory and thread
|
|
count, read `VmHWM` and `Threads` from `/proc/<pid>/status` while it runs
|
|
(sprint 05's run script sampled them every 30 s).
|
|
- **The server is shared and gets overloaded** (load up to 92 on 12 cores,
|
|
2026-11-11): a starved collector falls behind and gets closed with 1013.
|
|
Check `cat /proc/loadavg` before a live run whose numbers are compared, and
|
|
alternate the builds in short rounds rather than running them back to back.
|
|
- **Live data beats the docs**: Polymarket's market channel has rules its
|
|
docs don't state (`docs/notes.md`); check a protocol assumption on a
|
|
recording of the real feed (`tests/fixtures/record.py`) before relying on it.
|
|
|
|
## Workflow differences
|
|
|
|
A job-challenge repo cloned from the template: no Gitea, Woodpecker or release
|
|
pipeline.
|
|
|
|
- **Kept**: `docs/`, `roadmap/` sprint files, `feat/*` branches, and
|
|
`scripts/ci.sh` (fmt, clippy `-D warnings`, tests).
|
|
- **Dropped**: the dependency audit (supply-chain checks are for deployed
|
|
products), Woodpecker, `release.sh`, versioning and changelog, rc images, and
|
|
the `prod` / `dev` branches. CI targets application quality, not prod-grade.
|
|
- **Branches**: `main` plays the role of `dev`; `feat/*` branches are squashed
|
|
into `main`.
|