feat: ready to submit, a run to watch, the README, the review's fixes

Sprint 08 (roadmap/08. ready to submit.md), the last milestone.

- scripts/watch.sh runs the collector with a live view of its log, stops
  cleanly on Ctrl-C, then checks the output (reconcile) and shows one
  symbol's book as text (show, a new binary).
- The README for the solution, with the final runs: Polymarket through an
  outage it recovered from on its own, Kalshi's demo with two slow
  connections recycled.
- From a review of the whole codebase: Kalshi takes RSA keys too
  (RSA-PSS-SHA256); a slow Kalshi connection takes the hourly recycle's
  next turn, which it nearly never got before; a market tracked again no
  longer fails reconcile; Book::add checks for overflow; usage and --help
  name the key variables; the first scan says how far it got.
- Tests for every gap the reviews found, an offline run of the whole
  orchestrator among them; ShellCheck in CI.
This commit is contained in:
pascal committed 2026-11-12 22:32:39 +01:00
1 parent 515e85941d
commit cf6c16c9cf
35 files changed
+2791 -207

No files matched your search

+208 -15
View File
@@ -1,21 +1,214 @@
# Read before getting started!!
# Order-book collector for Polymarket and Kalshi
First, read [src/main.rs](https://github.com/voluntas-io/engineering-challenge/blob/main/src/main.rs). Then, some background on this repo:
My answer to the Voluntas engineering challenge. The collector subscribes
to every live market of one exchange (Polymarket, or Kalshi), keeps an order
book for each symbol, and writes the template's SBE messages: a Delta for
every change, and a Snapshot of every book each minute and whenever a book
is replaced. All four extensions are done: redundant connections,
subscription monitoring, one core for both exchanges, and latency
experiments ([LATENCY.md](LATENCY.md)).
This repo uses the [FIX SBE standard](https://fixtrading.org/packages/simple-binary-encoding-technical-specification-v2-rc2/)
for writing binary messages. The rust code abstracts this away for you, and you can simply use the helpers, as demonstrated in `main.rs`.
Built with Claude Code. I set the architecture (the threading model,
redundancy, the exchange seam) and settled every design decision; Claude
proposed the options, wrote the code and ran the experiments. Each
decision and its reasons are in [`roadmap/`](roadmap) and
[`docs/adr/`](docs/adr).
### More background, in case you are curious!
## Quick start
FIX SBE is defined in XML, which I find extremely cumbersome to edit, especially
when updating schemas frequently. To solve this, I have also introduced some
helper code that takes our own custom YAML format, and turns it into FIX SBE
XML, which in turn is used to generate rust code to encode and decode the
messages. **You don't need to edit any of these files**. This explanation is
simply to help you understand the origins of the various files in this repo.
On Linux (the script reads `/proc`), with Rust; built and tested with
1.98:
If you're curious to see the YAML schema, take a look at [challenge_orderbook.yaml](https://github.com/voluntas-io/engineering-challenge/blob/main/sbe_schemas/engineering_challenge/challenge_orderbook.yaml).
We also use a custom timestamp field, so that the Rust struct can use
`chrono::DateTime` in the generated struct, and in SBE schema, it is a uint64
representing the timestamp as the number of microseconds since unix epoch.
```sh
scripts/watch.sh # Polymarket for 3 minutes, no key needed
scripts/watch.sh polymarket 10 # or longer
```
It builds the collector, runs it, and shows what it does each minute:
markets and connections; changes written; duplicates (a change's second
copy, from the other connection carrying its symbol); rescued changes (one
connection missed them, the other delivered them); resyncs (a fresh book
asked for, when ours can't be checked); losses (a fresh book that differed
from ours: changes neither connection delivered). The output and the full
log go to `watch/`. Ctrl-C stops it
early and cleanly. Then it checks the output against the log (`reconcile`)
and prints one busy symbol's book as text (`show`). The first scan lists
every market before subscribing: a minute or two.
## Running it
```sh
cargo build --release
./target/release/engineering-challenge polymarket > out.sbe 2> run.log
./target/release/engineering-challenge polymarket out.sbe 2> run.log # same
```
The output goes to stdout unless a path is given; the log goes to stderr.
SIGINT, SIGTERM or SIGHUP closes the websockets and flushes the output.
Only a clean stop flushes it: the writer fills 64 MiB chunks, the only size
the template's reader opens.
Kalshi needs an API key. Two variables name its files (paths keep the key
out of the shell's history):
```sh
export KALSHI_KEY_ID_FILE=~/kalshi/key-id
export KALSHI_PRIVATE_KEY_FILE=~/kalshi/private-key.pem # Ed25519 or RSA
./target/release/engineering-challenge kalshi out.sbe 2> run.log
./target/release/engineering-challenge kalshi-demo out.sbe 2> run.log
```
I had a key for Kalshi's demo only, so production has never been run; the
code is the same, only the URLs differ.
## Checking the output
```sh
./target/release/reconcile out.sbe run.log
./target/release/show out.sbe [SYMBOL]
```
`reconcile` replays the output: per symbol, the Deltas since a Snapshot,
applied to it, must give the next Snapshot. A Snapshot that differs is
paired with the log line that says why it was written: a loss; a
subscription gap (the few changes a subscribe keeps from us, which the
exchange's opening book already holds); a switch (Polymarket's book now
followed on the other connection); a market tracked again after it left.
It fails on any left unexplained, or on a Delta before its symbol's first
Snapshot.
## Final runs
`scripts/watch.sh`, from the submitted code, on a shared 12-core server.
**Polymarket, 10 minutes** (2026-11-12, system load 7.2 at the start): the
first scan listed 199,531 live markets in 77 s, subscribed on 178
connections. `reconcile`: 14.4 million Deltas and 3.7 million Snapshots;
the 142,990 Snapshots that differ from the book before them each explained
by their log line. 95% of the changes came on both connections, and
417,852 were rescued. Peak memory 2.3 GB. The thread count wasn't sampled
at its peak in this run; it is one per connection plus about 20 (the Kalshi
run's 137 for 116), so about 200.
The run met an outage of about two minutes. Between 16:50:17 and 16:50:25
UTC, 177 of the 178 connections were found silent (a connection is silent
after 30 s with nothing received, so since about 16:49:47). Connects then
timed out or found no route to the host, and between 16:51:37 and 16:51:59
every connection came back on its own. Each book that changed meanwhile was found
wrong on the reconnect and replaced: 138,950 of the run's 140,129 losses
are those two minutes, 1,179 the other eight. Whether the network or
Polymarket went down, the log can't say.
**Kalshi's demo, 20 minutes** (system load 0.8 at the start): the first
scan took 5 min 17 s, the demo's listing being slow (about 1.2 s a page of
1,000 markets in that run); then 260,865 markets on 116 connections.
`reconcile`: 4.1 million Deltas and 4.0 million Snapshots; the 12 that
differ each a logged loss. 98% of the changes came on both connections;
1,228 resyncs. Peak memory 830 MB, 137 threads. Every connection is
recycled (closed and opened again) about once an hour, one at a time; on
Kalshi one that stays slow takes the next turn, and that acted: a
connection's lateness is how long after the other connection's copy it
delivers a change, and connections 99 (246 ms on average) and 57 (80 ms)
were each among the slowest 3 minutes running, so each was recycled at
the next turn, 13 and 21 s later.
## Design
```
rescan fetcher ──findings──▶ ┌──────────────┐ ◀──findings── recent fetcher
│ orchestrator │ ◀── up, down, 1013 ─── connections
└──────┬───▲───┘ ──── recycle ────────▶ connections
Add / Move / Remove (a market and │ │ loads, quiet markets,
its connections), to its thread │ │ lateness, each minute
┌──────▼───┴───┐
connection 0..N ──events───▶ │ book threads │ ──Snapshot | Delta──▶ writer
(read, stamp, parse, route) │ 0..15 │
▲ └──────┬───────┘
└── subscribe / unsubscribe / resync ┘
```
Plain threads and channels, nothing shared ([ADR 0001](docs/adr/0001-concurrency-model.md)).
A connection stamps the receive time right after the socket read, then
parses and routes each event to the book thread that owns its market. Each
book is owned by one of 16 book threads; one writer owns the output. The
orchestrator decides which connections carry each market, balances the
bytes per second each carries, and recycles them ([ADR 0002](docs/adr/0002-lanes-and-orchestrator.md)).
[docs/project.md](docs/project.md) has the whole design.
A lost change leaves a book silently wrong, so every sign of one replaces
the book with a fresh one from the exchange (a resync). On Polymarket the
exchange's book hash is recomputed after every change; on Kalshi it is a
skipped sequence number on a connection. A fresh book that differs from
ours counts as a loss; some of those are only books ahead of our changes,
so the count is an upper bound. The protocol facts the
code relies on, several found only in live data, are in
[docs/notes.md](docs/notes.md).
## The extensions
- **Redundant websockets.** Every symbol is on two connections chosen
independently; its book takes each change from whichever delivers it
first ([ADR 0003](docs/adr/0003-first-one-wins.md)). There is no "source"
to switch, so the brief's Snapshot on a switch becomes a Snapshot
whenever a fresh book from either connection replaces ours. A connection
the exchange closes twice within 10 minutes as too slow (code 1013, "try
again later") has half its markets moved to others.
- **Subscription monitoring.** A subscription is quiet when it delivered
nothing for a minute while the other connection delivered at least 3
changes of the same symbol: it is resubscribed, and moved if quiet
again. Judged against the other connection, not a rate, it holds for
the quietest markets too. Every
connection is recycled about once an hour, one at a time, never two of a
market's; one silent for 30 s reconnects.
- **Generic core.** `src/core/` is the same for both exchanges. An exchange
supplies its websocket codec and its loss rule
([ADR 0004](docs/adr/0004-exchange-seam.md)): the rule is more than
decoding, since telling a duplicate from a loss depends on what the
exchange sends (a book hash, or a sequence number and a clock).
- **Latency.** [LATENCY.md](LATENCY.md): a connection that delivers
changes sooner than the others keeps doing so, on both exchanges.
Keeping the fastest gains under a millisecond on Polymarket, and tens of
milliseconds on Kalshi's demo, where the collector now recycles a
connection that stays slow.
## Limits
- A hard crash (SIGKILL, power) loses the chunk being filled, up to 64 MiB.
- When a symbol is subscribed, the exchange's opening book holds a few
changes it never sends as changes: logged as a subscription gap.
- Polymarket changes held for ordering when the run stops (at most 1 s) are
not written.
- Channels are unbounded: a burst never blocks a socket or drops an event,
but a book thread or writer that can't keep up grows memory.
- Kalshi production was never run.
## Next
From the roadmap's possible features ([docs/roadmap.md](docs/roadmap.md)):
telling a real loss from a fresh book merely ahead of the changes we have,
so the loss count measures data missed; Polymarket's `new_market` event as
a faster discovery signal; fewer Snapshots for unchanged books (the brief's
every minute is about 130 MB a minute at full coverage); a Kalshi
production run.
## Reading the code
- [docs/project.md](docs/project.md): what it is, the central bet (the
pipeline), the design.
- [docs/adr/](docs/adr): the four decisions that are hard to reverse.
- [docs/notes.md](docs/notes.md): each exchange's rules, as found.
- [LATENCY.md](LATENCY.md): extension 4.
- [GLOSSARY.md](GLOSSARY.md): the terms (a Snapshot is only the output
message; getting a fresh book is a resync).
- [roadmap/](roadmap): each sprint's plan, decisions and outcome, in order.
- [CLAUDE.md](CLAUDE.md): the instructions the coding agent worked under.
## Output format
The output uses the template's helpers unchanged: `AlignedSbeWriter`
writes the SBE messages of
[sbe_schemas/engineering_challenge/challenge_orderbook.yaml](sbe_schemas/engineering_challenge/challenge_orderbook.yaml)
(`Snapshot` and `PriceLevelUpdate`, prices and quantities in fixed point,
timestamps in microseconds since the epoch), readable by the template's
`AlignedSbeReader`. The template's code is `src/util/` and `sbe_schemas/`;
everything else in `src/` is the collector.