Archived
Sprint 08 (roadmap/08. ready to submit.md), the last milestone. - scripts/watch.sh runs the collector with a live view of its log, stops cleanly on Ctrl-C, then checks the output (reconcile) and shows one symbol's book as text (show, a new binary). - The README for the solution, with the final runs: Polymarket through an outage it recovered from on its own, Kalshi's demo with two slow connections recycled. - From a review of the whole codebase: Kalshi takes RSA keys too (RSA-PSS-SHA256); a slow Kalshi connection takes the hourly recycle's next turn, which it nearly never got before; a market tracked again no longer fails reconcile; Book::add checks for overflow; usage and --help name the key variables; the first scan says how far it got. - Tests for every gap the reviews found, an offline run of the whole orchestrator among them; ShellCheck in CI.
215 lines
11 KiB
Markdown
215 lines
11 KiB
Markdown
# Order-book collector for Polymarket and Kalshi
|
|
|
|
My answer to the Voluntas engineering challenge. The collector subscribes
|
|
to every live market of one exchange (Polymarket, or Kalshi), keeps an order
|
|
book for each symbol, and writes the template's SBE messages: a Delta for
|
|
every change, and a Snapshot of every book each minute and whenever a book
|
|
is replaced. All four extensions are done: redundant connections,
|
|
subscription monitoring, one core for both exchanges, and latency
|
|
experiments ([LATENCY.md](LATENCY.md)).
|
|
|
|
Built with Claude Code. I set the architecture (the threading model,
|
|
redundancy, the exchange seam) and settled every design decision; Claude
|
|
proposed the options, wrote the code and ran the experiments. Each
|
|
decision and its reasons are in [`roadmap/`](roadmap) and
|
|
[`docs/adr/`](docs/adr).
|
|
|
|
## Quick start
|
|
|
|
On Linux (the script reads `/proc`), with Rust; built and tested with
|
|
1.98:
|
|
|
|
```sh
|
|
scripts/watch.sh # Polymarket for 3 minutes, no key needed
|
|
scripts/watch.sh polymarket 10 # or longer
|
|
```
|
|
|
|
It builds the collector, runs it, and shows what it does each minute:
|
|
markets and connections; changes written; duplicates (a change's second
|
|
copy, from the other connection carrying its symbol); rescued changes (one
|
|
connection missed them, the other delivered them); resyncs (a fresh book
|
|
asked for, when ours can't be checked); losses (a fresh book that differed
|
|
from ours: changes neither connection delivered). The output and the full
|
|
log go to `watch/`. Ctrl-C stops it
|
|
early and cleanly. Then it checks the output against the log (`reconcile`)
|
|
and prints one busy symbol's book as text (`show`). The first scan lists
|
|
every market before subscribing: a minute or two.
|
|
|
|
## Running it
|
|
|
|
```sh
|
|
cargo build --release
|
|
./target/release/engineering-challenge polymarket > out.sbe 2> run.log
|
|
./target/release/engineering-challenge polymarket out.sbe 2> run.log # same
|
|
```
|
|
|
|
The output goes to stdout unless a path is given; the log goes to stderr.
|
|
SIGINT, SIGTERM or SIGHUP closes the websockets and flushes the output.
|
|
Only a clean stop flushes it: the writer fills 64 MiB chunks, the only size
|
|
the template's reader opens.
|
|
|
|
Kalshi needs an API key. Two variables name its files (paths keep the key
|
|
out of the shell's history):
|
|
|
|
```sh
|
|
export KALSHI_KEY_ID_FILE=~/kalshi/key-id
|
|
export KALSHI_PRIVATE_KEY_FILE=~/kalshi/private-key.pem # Ed25519 or RSA
|
|
./target/release/engineering-challenge kalshi out.sbe 2> run.log
|
|
./target/release/engineering-challenge kalshi-demo out.sbe 2> run.log
|
|
```
|
|
|
|
I had a key for Kalshi's demo only, so production has never been run; the
|
|
code is the same, only the URLs differ.
|
|
|
|
## Checking the output
|
|
|
|
```sh
|
|
./target/release/reconcile out.sbe run.log
|
|
./target/release/show out.sbe [SYMBOL]
|
|
```
|
|
|
|
`reconcile` replays the output: per symbol, the Deltas since a Snapshot,
|
|
applied to it, must give the next Snapshot. A Snapshot that differs is
|
|
paired with the log line that says why it was written: a loss; a
|
|
subscription gap (the few changes a subscribe keeps from us, which the
|
|
exchange's opening book already holds); a switch (Polymarket's book now
|
|
followed on the other connection); a market tracked again after it left.
|
|
It fails on any left unexplained, or on a Delta before its symbol's first
|
|
Snapshot.
|
|
|
|
## Final runs
|
|
|
|
`scripts/watch.sh`, from the submitted code, on a shared 12-core server.
|
|
|
|
**Polymarket, 10 minutes** (2026-11-12, system load 7.2 at the start): the
|
|
first scan listed 199,531 live markets in 77 s, subscribed on 178
|
|
connections. `reconcile`: 14.4 million Deltas and 3.7 million Snapshots;
|
|
the 142,990 Snapshots that differ from the book before them each explained
|
|
by their log line. 95% of the changes came on both connections, and
|
|
417,852 were rescued. Peak memory 2.3 GB. The thread count wasn't sampled
|
|
at its peak in this run; it is one per connection plus about 20 (the Kalshi
|
|
run's 137 for 116), so about 200.
|
|
|
|
The run met an outage of about two minutes. Between 16:50:17 and 16:50:25
|
|
UTC, 177 of the 178 connections were found silent (a connection is silent
|
|
after 30 s with nothing received, so since about 16:49:47). Connects then
|
|
timed out or found no route to the host, and between 16:51:37 and 16:51:59
|
|
every connection came back on its own. Each book that changed meanwhile was found
|
|
wrong on the reconnect and replaced: 138,950 of the run's 140,129 losses
|
|
are those two minutes, 1,179 the other eight. Whether the network or
|
|
Polymarket went down, the log can't say.
|
|
|
|
**Kalshi's demo, 20 minutes** (system load 0.8 at the start): the first
|
|
scan took 5 min 17 s, the demo's listing being slow (about 1.2 s a page of
|
|
1,000 markets in that run); then 260,865 markets on 116 connections.
|
|
`reconcile`: 4.1 million Deltas and 4.0 million Snapshots; the 12 that
|
|
differ each a logged loss. 98% of the changes came on both connections;
|
|
1,228 resyncs. Peak memory 830 MB, 137 threads. Every connection is
|
|
recycled (closed and opened again) about once an hour, one at a time; on
|
|
Kalshi one that stays slow takes the next turn, and that acted: a
|
|
connection's lateness is how long after the other connection's copy it
|
|
delivers a change, and connections 99 (246 ms on average) and 57 (80 ms)
|
|
were each among the slowest 3 minutes running, so each was recycled at
|
|
the next turn, 13 and 21 s later.
|
|
|
|
## Design
|
|
|
|
```
|
|
rescan fetcher ──findings──▶ ┌──────────────┐ ◀──findings── recent fetcher
|
|
│ orchestrator │ ◀── up, down, 1013 ─── connections
|
|
└──────┬───▲───┘ ──── recycle ────────▶ connections
|
|
Add / Move / Remove (a market and │ │ loads, quiet markets,
|
|
its connections), to its thread │ │ lateness, each minute
|
|
┌──────▼───┴───┐
|
|
connection 0..N ──events───▶ │ book threads │ ──Snapshot | Delta──▶ writer
|
|
(read, stamp, parse, route) │ 0..15 │
|
|
▲ └──────┬───────┘
|
|
└── subscribe / unsubscribe / resync ┘
|
|
```
|
|
|
|
Plain threads and channels, nothing shared ([ADR 0001](docs/adr/0001-concurrency-model.md)).
|
|
A connection stamps the receive time right after the socket read, then
|
|
parses and routes each event to the book thread that owns its market. Each
|
|
book is owned by one of 16 book threads; one writer owns the output. The
|
|
orchestrator decides which connections carry each market, balances the
|
|
bytes per second each carries, and recycles them ([ADR 0002](docs/adr/0002-lanes-and-orchestrator.md)).
|
|
[docs/project.md](docs/project.md) has the whole design.
|
|
|
|
A lost change leaves a book silently wrong, so every sign of one replaces
|
|
the book with a fresh one from the exchange (a resync). On Polymarket the
|
|
exchange's book hash is recomputed after every change; on Kalshi it is a
|
|
skipped sequence number on a connection. A fresh book that differs from
|
|
ours counts as a loss; some of those are only books ahead of our changes,
|
|
so the count is an upper bound. The protocol facts the
|
|
code relies on, several found only in live data, are in
|
|
[docs/notes.md](docs/notes.md).
|
|
|
|
## The extensions
|
|
|
|
- **Redundant websockets.** Every symbol is on two connections chosen
|
|
independently; its book takes each change from whichever delivers it
|
|
first ([ADR 0003](docs/adr/0003-first-one-wins.md)). There is no "source"
|
|
to switch, so the brief's Snapshot on a switch becomes a Snapshot
|
|
whenever a fresh book from either connection replaces ours. A connection
|
|
the exchange closes twice within 10 minutes as too slow (code 1013, "try
|
|
again later") has half its markets moved to others.
|
|
- **Subscription monitoring.** A subscription is quiet when it delivered
|
|
nothing for a minute while the other connection delivered at least 3
|
|
changes of the same symbol: it is resubscribed, and moved if quiet
|
|
again. Judged against the other connection, not a rate, it holds for
|
|
the quietest markets too. Every
|
|
connection is recycled about once an hour, one at a time, never two of a
|
|
market's; one silent for 30 s reconnects.
|
|
- **Generic core.** `src/core/` is the same for both exchanges. An exchange
|
|
supplies its websocket codec and its loss rule
|
|
([ADR 0004](docs/adr/0004-exchange-seam.md)): the rule is more than
|
|
decoding, since telling a duplicate from a loss depends on what the
|
|
exchange sends (a book hash, or a sequence number and a clock).
|
|
- **Latency.** [LATENCY.md](LATENCY.md): a connection that delivers
|
|
changes sooner than the others keeps doing so, on both exchanges.
|
|
Keeping the fastest gains under a millisecond on Polymarket, and tens of
|
|
milliseconds on Kalshi's demo, where the collector now recycles a
|
|
connection that stays slow.
|
|
|
|
## Limits
|
|
|
|
- A hard crash (SIGKILL, power) loses the chunk being filled, up to 64 MiB.
|
|
- When a symbol is subscribed, the exchange's opening book holds a few
|
|
changes it never sends as changes: logged as a subscription gap.
|
|
- Polymarket changes held for ordering when the run stops (at most 1 s) are
|
|
not written.
|
|
- Channels are unbounded: a burst never blocks a socket or drops an event,
|
|
but a book thread or writer that can't keep up grows memory.
|
|
- Kalshi production was never run.
|
|
|
|
## Next
|
|
|
|
From the roadmap's possible features ([docs/roadmap.md](docs/roadmap.md)):
|
|
telling a real loss from a fresh book merely ahead of the changes we have,
|
|
so the loss count measures data missed; Polymarket's `new_market` event as
|
|
a faster discovery signal; fewer Snapshots for unchanged books (the brief's
|
|
every minute is about 130 MB a minute at full coverage); a Kalshi
|
|
production run.
|
|
|
|
## Reading the code
|
|
|
|
- [docs/project.md](docs/project.md): what it is, the central bet (the
|
|
pipeline), the design.
|
|
- [docs/adr/](docs/adr): the four decisions that are hard to reverse.
|
|
- [docs/notes.md](docs/notes.md): each exchange's rules, as found.
|
|
- [LATENCY.md](LATENCY.md): extension 4.
|
|
- [GLOSSARY.md](GLOSSARY.md): the terms (a Snapshot is only the output
|
|
message; getting a fresh book is a resync).
|
|
- [roadmap/](roadmap): each sprint's plan, decisions and outcome, in order.
|
|
- [CLAUDE.md](CLAUDE.md): the instructions the coding agent worked under.
|
|
|
|
## Output format
|
|
|
|
The output uses the template's helpers unchanged: `AlignedSbeWriter`
|
|
writes the SBE messages of
|
|
[sbe_schemas/engineering_challenge/challenge_orderbook.yaml](sbe_schemas/engineering_challenge/challenge_orderbook.yaml)
|
|
(`Snapshot` and `PriceLevelUpdate`, prices and quantities in fixed point,
|
|
timestamps in microseconds since the epoch), readable by the template's
|
|
`AlignedSbeReader`. The template's code is `src/util/` and `sbe_schemas/`;
|
|
everything else in `src/` is the collector.
|