Archived
feat: ready to submit, a run to watch, the README, the review's fixes
Sprint 08 (roadmap/08. ready to submit.md), the last milestone. - scripts/watch.sh runs the collector with a live view of its log, stops cleanly on Ctrl-C, then checks the output (reconcile) and shows one symbol's book as text (show, a new binary). - The README for the solution, with the final runs: Polymarket through an outage it recovered from on its own, Kalshi's demo with two slow connections recycled. - From a review of the whole codebase: Kalshi takes RSA keys too (RSA-PSS-SHA256); a slow Kalshi connection takes the hourly recycle's next turn, which it nearly never got before; a market tracked again no longer fails reconcile; Book::add checks for overflow; usage and --help name the key variables; the first scan says how far it got. - Tests for every gap the reviews found, an offline run of the whole orchestrator among them; ShellCheck in CI.
This commit is contained in:
35 files changed
+2791
-207
No files matched your search
@@ -1,21 +1,214 @@
|
||||
# Read before getting started!!
|
||||
# Order-book collector for Polymarket and Kalshi
|
||||
|
||||
First, read [src/main.rs](https://github.com/voluntas-io/engineering-challenge/blob/main/src/main.rs). Then, some background on this repo:
|
||||
My answer to the Voluntas engineering challenge. The collector subscribes
|
||||
to every live market of one exchange (Polymarket, or Kalshi), keeps an order
|
||||
book for each symbol, and writes the template's SBE messages: a Delta for
|
||||
every change, and a Snapshot of every book each minute and whenever a book
|
||||
is replaced. All four extensions are done: redundant connections,
|
||||
subscription monitoring, one core for both exchanges, and latency
|
||||
experiments ([LATENCY.md](LATENCY.md)).
|
||||
|
||||
This repo uses the [FIX SBE standard](https://fixtrading.org/packages/simple-binary-encoding-technical-specification-v2-rc2/)
|
||||
for writing binary messages. The rust code abstracts this away for you, and you can simply use the helpers, as demonstrated in `main.rs`.
|
||||
Built with Claude Code. I set the architecture (the threading model,
|
||||
redundancy, the exchange seam) and settled every design decision; Claude
|
||||
proposed the options, wrote the code and ran the experiments. Each
|
||||
decision and its reasons are in [`roadmap/`](roadmap) and
|
||||
[`docs/adr/`](docs/adr).
|
||||
|
||||
### More background, in case you are curious!
|
||||
## Quick start
|
||||
|
||||
FIX SBE is defined in XML, which I find extremely cumbersome to edit, especially
|
||||
when updating schemas frequently. To solve this, I have also introduced some
|
||||
helper code that takes our own custom YAML format, and turns it into FIX SBE
|
||||
XML, which in turn is used to generate rust code to encode and decode the
|
||||
messages. **You don't need to edit any of these files**. This explanation is
|
||||
simply to help you understand the origins of the various files in this repo.
|
||||
On Linux (the script reads `/proc`), with Rust; built and tested with
|
||||
1.98:
|
||||
|
||||
If you're curious to see the YAML schema, take a look at [challenge_orderbook.yaml](https://github.com/voluntas-io/engineering-challenge/blob/main/sbe_schemas/engineering_challenge/challenge_orderbook.yaml).
|
||||
We also use a custom timestamp field, so that the Rust struct can use
|
||||
`chrono::DateTime` in the generated struct, and in SBE schema, it is a uint64
|
||||
representing the timestamp as the number of microseconds since unix epoch.
|
||||
```sh
|
||||
scripts/watch.sh # Polymarket for 3 minutes, no key needed
|
||||
scripts/watch.sh polymarket 10 # or longer
|
||||
```
|
||||
|
||||
It builds the collector, runs it, and shows what it does each minute:
|
||||
markets and connections; changes written; duplicates (a change's second
|
||||
copy, from the other connection carrying its symbol); rescued changes (one
|
||||
connection missed them, the other delivered them); resyncs (a fresh book
|
||||
asked for, when ours can't be checked); losses (a fresh book that differed
|
||||
from ours: changes neither connection delivered). The output and the full
|
||||
log go to `watch/`. Ctrl-C stops it
|
||||
early and cleanly. Then it checks the output against the log (`reconcile`)
|
||||
and prints one busy symbol's book as text (`show`). The first scan lists
|
||||
every market before subscribing: a minute or two.
|
||||
|
||||
## Running it
|
||||
|
||||
```sh
|
||||
cargo build --release
|
||||
./target/release/engineering-challenge polymarket > out.sbe 2> run.log
|
||||
./target/release/engineering-challenge polymarket out.sbe 2> run.log # same
|
||||
```
|
||||
|
||||
The output goes to stdout unless a path is given; the log goes to stderr.
|
||||
SIGINT, SIGTERM or SIGHUP closes the websockets and flushes the output.
|
||||
Only a clean stop flushes it: the writer fills 64 MiB chunks, the only size
|
||||
the template's reader opens.
|
||||
|
||||
Kalshi needs an API key. Two variables name its files (paths keep the key
|
||||
out of the shell's history):
|
||||
|
||||
```sh
|
||||
export KALSHI_KEY_ID_FILE=~/kalshi/key-id
|
||||
export KALSHI_PRIVATE_KEY_FILE=~/kalshi/private-key.pem # Ed25519 or RSA
|
||||
./target/release/engineering-challenge kalshi out.sbe 2> run.log
|
||||
./target/release/engineering-challenge kalshi-demo out.sbe 2> run.log
|
||||
```
|
||||
|
||||
I had a key for Kalshi's demo only, so production has never been run; the
|
||||
code is the same, only the URLs differ.
|
||||
|
||||
## Checking the output
|
||||
|
||||
```sh
|
||||
./target/release/reconcile out.sbe run.log
|
||||
./target/release/show out.sbe [SYMBOL]
|
||||
```
|
||||
|
||||
`reconcile` replays the output: per symbol, the Deltas since a Snapshot,
|
||||
applied to it, must give the next Snapshot. A Snapshot that differs is
|
||||
paired with the log line that says why it was written: a loss; a
|
||||
subscription gap (the few changes a subscribe keeps from us, which the
|
||||
exchange's opening book already holds); a switch (Polymarket's book now
|
||||
followed on the other connection); a market tracked again after it left.
|
||||
It fails on any left unexplained, or on a Delta before its symbol's first
|
||||
Snapshot.
|
||||
|
||||
## Final runs
|
||||
|
||||
`scripts/watch.sh`, from the submitted code, on a shared 12-core server.
|
||||
|
||||
**Polymarket, 10 minutes** (2026-11-12, system load 7.2 at the start): the
|
||||
first scan listed 199,531 live markets in 77 s, subscribed on 178
|
||||
connections. `reconcile`: 14.4 million Deltas and 3.7 million Snapshots;
|
||||
the 142,990 Snapshots that differ from the book before them each explained
|
||||
by their log line. 95% of the changes came on both connections, and
|
||||
417,852 were rescued. Peak memory 2.3 GB. The thread count wasn't sampled
|
||||
at its peak in this run; it is one per connection plus about 20 (the Kalshi
|
||||
run's 137 for 116), so about 200.
|
||||
|
||||
The run met an outage of about two minutes. Between 16:50:17 and 16:50:25
|
||||
UTC, 177 of the 178 connections were found silent (a connection is silent
|
||||
after 30 s with nothing received, so since about 16:49:47). Connects then
|
||||
timed out or found no route to the host, and between 16:51:37 and 16:51:59
|
||||
every connection came back on its own. Each book that changed meanwhile was found
|
||||
wrong on the reconnect and replaced: 138,950 of the run's 140,129 losses
|
||||
are those two minutes, 1,179 the other eight. Whether the network or
|
||||
Polymarket went down, the log can't say.
|
||||
|
||||
**Kalshi's demo, 20 minutes** (system load 0.8 at the start): the first
|
||||
scan took 5 min 17 s, the demo's listing being slow (about 1.2 s a page of
|
||||
1,000 markets in that run); then 260,865 markets on 116 connections.
|
||||
`reconcile`: 4.1 million Deltas and 4.0 million Snapshots; the 12 that
|
||||
differ each a logged loss. 98% of the changes came on both connections;
|
||||
1,228 resyncs. Peak memory 830 MB, 137 threads. Every connection is
|
||||
recycled (closed and opened again) about once an hour, one at a time; on
|
||||
Kalshi one that stays slow takes the next turn, and that acted: a
|
||||
connection's lateness is how long after the other connection's copy it
|
||||
delivers a change, and connections 99 (246 ms on average) and 57 (80 ms)
|
||||
were each among the slowest 3 minutes running, so each was recycled at
|
||||
the next turn, 13 and 21 s later.
|
||||
|
||||
## Design
|
||||
|
||||
```
|
||||
rescan fetcher ──findings──▶ ┌──────────────┐ ◀──findings── recent fetcher
|
||||
│ orchestrator │ ◀── up, down, 1013 ─── connections
|
||||
└──────┬───▲───┘ ──── recycle ────────▶ connections
|
||||
Add / Move / Remove (a market and │ │ loads, quiet markets,
|
||||
its connections), to its thread │ │ lateness, each minute
|
||||
┌──────▼───┴───┐
|
||||
connection 0..N ──events───▶ │ book threads │ ──Snapshot | Delta──▶ writer
|
||||
(read, stamp, parse, route) │ 0..15 │
|
||||
▲ └──────┬───────┘
|
||||
└── subscribe / unsubscribe / resync ┘
|
||||
```
|
||||
|
||||
Plain threads and channels, nothing shared ([ADR 0001](docs/adr/0001-concurrency-model.md)).
|
||||
A connection stamps the receive time right after the socket read, then
|
||||
parses and routes each event to the book thread that owns its market. Each
|
||||
book is owned by one of 16 book threads; one writer owns the output. The
|
||||
orchestrator decides which connections carry each market, balances the
|
||||
bytes per second each carries, and recycles them ([ADR 0002](docs/adr/0002-lanes-and-orchestrator.md)).
|
||||
[docs/project.md](docs/project.md) has the whole design.
|
||||
|
||||
A lost change leaves a book silently wrong, so every sign of one replaces
|
||||
the book with a fresh one from the exchange (a resync). On Polymarket the
|
||||
exchange's book hash is recomputed after every change; on Kalshi it is a
|
||||
skipped sequence number on a connection. A fresh book that differs from
|
||||
ours counts as a loss; some of those are only books ahead of our changes,
|
||||
so the count is an upper bound. The protocol facts the
|
||||
code relies on, several found only in live data, are in
|
||||
[docs/notes.md](docs/notes.md).
|
||||
|
||||
## The extensions
|
||||
|
||||
- **Redundant websockets.** Every symbol is on two connections chosen
|
||||
independently; its book takes each change from whichever delivers it
|
||||
first ([ADR 0003](docs/adr/0003-first-one-wins.md)). There is no "source"
|
||||
to switch, so the brief's Snapshot on a switch becomes a Snapshot
|
||||
whenever a fresh book from either connection replaces ours. A connection
|
||||
the exchange closes twice within 10 minutes as too slow (code 1013, "try
|
||||
again later") has half its markets moved to others.
|
||||
- **Subscription monitoring.** A subscription is quiet when it delivered
|
||||
nothing for a minute while the other connection delivered at least 3
|
||||
changes of the same symbol: it is resubscribed, and moved if quiet
|
||||
again. Judged against the other connection, not a rate, it holds for
|
||||
the quietest markets too. Every
|
||||
connection is recycled about once an hour, one at a time, never two of a
|
||||
market's; one silent for 30 s reconnects.
|
||||
- **Generic core.** `src/core/` is the same for both exchanges. An exchange
|
||||
supplies its websocket codec and its loss rule
|
||||
([ADR 0004](docs/adr/0004-exchange-seam.md)): the rule is more than
|
||||
decoding, since telling a duplicate from a loss depends on what the
|
||||
exchange sends (a book hash, or a sequence number and a clock).
|
||||
- **Latency.** [LATENCY.md](LATENCY.md): a connection that delivers
|
||||
changes sooner than the others keeps doing so, on both exchanges.
|
||||
Keeping the fastest gains under a millisecond on Polymarket, and tens of
|
||||
milliseconds on Kalshi's demo, where the collector now recycles a
|
||||
connection that stays slow.
|
||||
|
||||
## Limits
|
||||
|
||||
- A hard crash (SIGKILL, power) loses the chunk being filled, up to 64 MiB.
|
||||
- When a symbol is subscribed, the exchange's opening book holds a few
|
||||
changes it never sends as changes: logged as a subscription gap.
|
||||
- Polymarket changes held for ordering when the run stops (at most 1 s) are
|
||||
not written.
|
||||
- Channels are unbounded: a burst never blocks a socket or drops an event,
|
||||
but a book thread or writer that can't keep up grows memory.
|
||||
- Kalshi production was never run.
|
||||
|
||||
## Next
|
||||
|
||||
From the roadmap's possible features ([docs/roadmap.md](docs/roadmap.md)):
|
||||
telling a real loss from a fresh book merely ahead of the changes we have,
|
||||
so the loss count measures data missed; Polymarket's `new_market` event as
|
||||
a faster discovery signal; fewer Snapshots for unchanged books (the brief's
|
||||
every minute is about 130 MB a minute at full coverage); a Kalshi
|
||||
production run.
|
||||
|
||||
## Reading the code
|
||||
|
||||
- [docs/project.md](docs/project.md): what it is, the central bet (the
|
||||
pipeline), the design.
|
||||
- [docs/adr/](docs/adr): the four decisions that are hard to reverse.
|
||||
- [docs/notes.md](docs/notes.md): each exchange's rules, as found.
|
||||
- [LATENCY.md](LATENCY.md): extension 4.
|
||||
- [GLOSSARY.md](GLOSSARY.md): the terms (a Snapshot is only the output
|
||||
message; getting a fresh book is a resync).
|
||||
- [roadmap/](roadmap): each sprint's plan, decisions and outcome, in order.
|
||||
- [CLAUDE.md](CLAUDE.md): the instructions the coding agent worked under.
|
||||
|
||||
## Output format
|
||||
|
||||
The output uses the template's helpers unchanged: `AlignedSbeWriter`
|
||||
writes the SBE messages of
|
||||
[sbe_schemas/engineering_challenge/challenge_orderbook.yaml](sbe_schemas/engineering_challenge/challenge_orderbook.yaml)
|
||||
(`Snapshot` and `PriceLevelUpdate`, prices and quantities in fixed point,
|
||||
timestamps in microseconds since the epoch), readable by the template's
|
||||
`AlignedSbeReader`. The template's code is `src/util/` and `sbe_schemas/`;
|
||||
everything else in `src/` is the collector.
|
||||
Reference in new issue
Block a user