Archived
The README said the quiet rule holds for the quietest markets (it can't judge one under 3 changes a minute) and that all four extensions were done (two of the brief's latency ideas were not tried); LATENCY.md said the slow recycle was not seen to fire live, two lines after the run where it did. Limits gains the process going down, the quiet rule's blind spots and Kalshi's skipped-change case. CLAUDE.md loses the home network's details and the server's key paths, of no use to a reader of the public repo. scripts/test_watch.sh no longer puts a here-document inside `$( )`: bash 3.2, which is sh on macOS, stopped at an apostrophe there and scripts/ci.sh ended on a syntax error.
232 lines
12 KiB
Markdown
232 lines
12 KiB
Markdown
# Order-book collector for Polymarket and Kalshi
|
|
|
|
My answer to the Voluntas engineering challenge. The collector subscribes
|
|
to every live market of one exchange (Polymarket, or Kalshi), keeps an order
|
|
book for each symbol, and writes the template's SBE messages: a Delta for
|
|
every change, and a Snapshot of every book each minute and whenever a book
|
|
is replaced. The four extensions are built: redundant connections,
|
|
subscription monitoring, one core for both exchanges, and latency
|
|
experiments ([LATENCY.md](LATENCY.md); of the brief's three ideas, two
|
|
could not be tried from here). What each leaves out is under
|
|
[Limits](#limits).
|
|
|
|
The design is mine: the threading model, the redundancy (two connections
|
|
per symbol, first one wins), the exchange seam, and every design decision,
|
|
each with its reasons in [`roadmap/`](roadmap) and [`docs/adr/`](docs/adr).
|
|
I built it with Claude Code, which proposed the options, wrote the code and
|
|
ran the experiments.
|
|
|
|
## Quick start
|
|
|
|
On Linux (the script reads `/proc`), with Rust; built and tested with
|
|
1.98:
|
|
|
|
```sh
|
|
scripts/watch.sh # Polymarket for 3 minutes, no key needed
|
|
scripts/watch.sh polymarket 10 # or longer
|
|
```
|
|
|
|
It builds the collector, runs it, and shows what it does each minute:
|
|
markets and connections; changes written; duplicates (a change's second
|
|
copy, from the other connection carrying its symbol); rescued changes (one
|
|
connection missed them, the other delivered them); resyncs (a fresh book
|
|
asked for, when ours can't be checked); losses (a fresh book that differed
|
|
from ours: changes neither connection delivered). The output and the full
|
|
log go to `watch/`. Ctrl-C stops it
|
|
early and cleanly. Then it checks the output against the log (`reconcile`)
|
|
and prints one busy symbol's book as text (`show`). The first scan lists
|
|
every market before subscribing: a minute or two.
|
|
|
|
## Running it
|
|
|
|
```sh
|
|
cargo build --release
|
|
./target/release/engineering-challenge polymarket > out.sbe 2> run.log
|
|
./target/release/engineering-challenge polymarket out.sbe 2> run.log # same
|
|
```
|
|
|
|
The output goes to stdout unless a path is given; the log goes to stderr.
|
|
SIGINT, SIGTERM or SIGHUP closes the websockets and flushes the output.
|
|
Only a clean stop flushes it: the writer fills 64 MiB chunks, the only size
|
|
the template's reader opens.
|
|
|
|
Kalshi needs an API key. Two variables name its files (paths keep the key
|
|
out of the shell's history):
|
|
|
|
```sh
|
|
export KALSHI_KEY_ID_FILE=~/kalshi/key-id
|
|
export KALSHI_PRIVATE_KEY_FILE=~/kalshi/private-key.pem # Ed25519 or RSA
|
|
./target/release/engineering-challenge kalshi out.sbe 2> run.log
|
|
./target/release/engineering-challenge kalshi-demo out.sbe 2> run.log
|
|
```
|
|
|
|
I had a key for Kalshi's demo only, so production has never been run; the
|
|
code is the same, only the URLs differ.
|
|
|
|
## Checking the output
|
|
|
|
```sh
|
|
./target/release/reconcile out.sbe run.log
|
|
./target/release/show out.sbe [SYMBOL]
|
|
```
|
|
|
|
`reconcile` replays the output: per symbol, the Deltas since a Snapshot,
|
|
applied to it, must give the next Snapshot. A Snapshot that differs is
|
|
paired with the log line that says why it was written: a loss; a
|
|
subscription gap (the few changes a subscribe keeps from us, which the
|
|
exchange's opening book already holds); a switch (Polymarket's book now
|
|
followed on the other connection); a market tracked again after it left.
|
|
It fails on any left unexplained, or on a Delta before its symbol's first
|
|
Snapshot.
|
|
|
|
## Final runs
|
|
|
|
`scripts/watch.sh`, from the submitted code, on a shared 12-core server.
|
|
|
|
**Polymarket, 10 minutes** (2026-11-12, system load 7.2 at the start): the
|
|
first scan listed 199,531 live markets in 77 s, subscribed on 178
|
|
connections. `reconcile`: 14.4 million Deltas and 3.7 million Snapshots;
|
|
the 142,990 Snapshots that differ from the book before them each explained
|
|
by their log line. 95% of the changes came on both connections, and
|
|
417,852 were rescued. Peak memory 2.3 GB. The thread count wasn't sampled
|
|
at its peak in this run; it is one per connection plus about 20 (the Kalshi
|
|
run's 137 for 116), so about 200.
|
|
|
|
The run met an outage of about two minutes. Between 16:50:17 and 16:50:25
|
|
UTC, 177 of the 178 connections were found silent (a connection is silent
|
|
after 30 s with nothing received, so since about 16:49:47). Connects then
|
|
timed out or found no route to the host, and between 16:51:37 and 16:51:59
|
|
every connection came back on its own. Each book that changed meanwhile was found
|
|
wrong on the reconnect and replaced: 138,950 of the run's 140,129 losses
|
|
are those two minutes, 1,179 the other eight. Whether the network or
|
|
Polymarket went down, the log can't say.
|
|
|
|
**Kalshi's demo, 20 minutes** (system load 0.8 at the start): the first
|
|
scan took 5 min 17 s, the demo's listing being slow (about 1.2 s a page of
|
|
1,000 markets in that run); then 260,865 markets on 116 connections.
|
|
`reconcile`: 4.1 million Deltas and 4.0 million Snapshots; the 12 that
|
|
differ each a logged loss. 98% of the changes came on both connections;
|
|
1,228 resyncs. Peak memory 830 MB, 137 threads. Every connection is
|
|
recycled (closed and opened again) about once an hour, one at a time; on
|
|
Kalshi one that stays slow takes the next turn, and that acted: a
|
|
connection's lateness is how long after the other connection's copy it
|
|
delivers a change, and connections 99 (246 ms on average) and 57 (80 ms)
|
|
were each among the slowest 3 minutes running, so each was recycled at
|
|
the next turn, 13 and 21 s later.
|
|
|
|
## Design
|
|
|
|
```
|
|
rescan fetcher ──findings──▶ ┌──────────────┐ ◀──findings── recent fetcher
|
|
│ orchestrator │ ◀── up, down, 1013 ─── connections
|
|
└──────┬───▲───┘ ──── recycle ────────▶ connections
|
|
Add / Move / Remove (a market and │ │ loads, quiet markets,
|
|
its connections), to its thread │ │ lateness, each minute
|
|
┌──────▼───┴───┐
|
|
connection 0..N ──events───▶ │ book threads │ ──Snapshot | Delta──▶ writer
|
|
(read, stamp, parse, route) │ 0..15 │
|
|
▲ └──────┬───────┘
|
|
└── subscribe / unsubscribe / resync ┘
|
|
```
|
|
|
|
Plain threads and channels, nothing shared ([ADR 0001](docs/adr/0001-concurrency-model.md)).
|
|
A connection stamps the receive time right after the socket read, then
|
|
parses and routes each event to the book thread that owns its market. Each
|
|
book is owned by one of 16 book threads; one writer owns the output. The
|
|
orchestrator decides which connections carry each market, balances the
|
|
bytes per second each carries, and recycles them ([ADR 0002](docs/adr/0002-lanes-and-orchestrator.md)).
|
|
[docs/project.md](docs/project.md) has the whole design.
|
|
|
|
A lost change leaves a book silently wrong, so every sign of one replaces
|
|
the book with a fresh one from the exchange (a resync). On Polymarket the
|
|
exchange's book hash is recomputed after every change; on Kalshi it is a
|
|
skipped sequence number on a connection. A fresh book that differs from
|
|
ours counts as a loss; some of those are only books ahead of our changes,
|
|
so the count is an upper bound. The protocol facts the
|
|
code relies on, several found only in live data, are in
|
|
[docs/notes.md](docs/notes.md).
|
|
|
|
## The extensions
|
|
|
|
- **Redundant websockets.** Every symbol is on two connections chosen
|
|
independently; its book takes each change from whichever delivers it
|
|
first ([ADR 0003](docs/adr/0003-first-one-wins.md)). There is no "source"
|
|
to switch, so the brief's Snapshot on a switch becomes a Snapshot
|
|
whenever a fresh book from either connection replaces ours. A connection
|
|
the exchange closes twice within 10 minutes as too slow (code 1013, "try
|
|
again later") has half its markets moved to others.
|
|
- **Subscription monitoring.** A subscription is quiet when it delivered
|
|
nothing for a minute while the other connection delivered at least 3
|
|
changes of the same symbol: it is resubscribed, and moved if quiet
|
|
again. Judged against the other connection, not a rate, it needs no
|
|
threshold per market; it can't judge a market under 3 changes a minute,
|
|
nor one quiet on both connections at once (Limits). Every
|
|
connection is recycled about once an hour, one at a time, never two of a
|
|
market's; one silent for 30 s reconnects.
|
|
- **Generic core.** `src/core/` is the same for both exchanges. An exchange
|
|
supplies its websocket codec and its loss rule
|
|
([ADR 0004](docs/adr/0004-exchange-seam.md)): the rule is more than
|
|
decoding, since telling a duplicate from a loss depends on what the
|
|
exchange sends (a book hash, or a sequence number and a clock).
|
|
- **Latency.** [LATENCY.md](LATENCY.md): a connection that delivers
|
|
changes sooner than the others keeps doing so, on Polymarket and on
|
|
Kalshi's demo. Keeping the fastest gains under a millisecond on
|
|
Polymarket, and tens of milliseconds on Kalshi's demo, where the
|
|
collector now recycles a connection that stays slow. Connecting first
|
|
after maintenance and choosing the TCP source port were not tried; the
|
|
file says why, and how each would be.
|
|
|
|
## Limits
|
|
|
|
- A hard crash (SIGKILL, power) loses the chunk being filled, up to 64 MiB.
|
|
- Nothing covers the process itself going down, the fourth failure the
|
|
brief's extension 1 names: it is one process, and starting it again on
|
|
the same output path truncates the file. A second collector on another
|
|
host, the two outputs merged downstream, would cover it; not built.
|
|
- Subscription monitoring compares a symbol's two connections, so it
|
|
doesn't see a market under 3 changes a minute, nor a symbol quiet on both
|
|
at once: only the next fresh book shows what was missed. A rate per
|
|
symbol, as the brief describes, would see the second.
|
|
- On Kalshi, a change that one connection skipped without a gap in its
|
|
sequence numbers would be dropped as a duplicate when the other delivers
|
|
it, and the book would stay wrong until the next fresh book. Never
|
|
observed; the rule rests on the demo's behaviour.
|
|
- When a symbol is subscribed, the exchange's opening book holds a few
|
|
changes it never sends as changes: logged as a subscription gap.
|
|
- Polymarket changes held for ordering when the run stops (at most 1 s) are
|
|
not written.
|
|
- Channels are unbounded: a burst never blocks a socket or drops an event,
|
|
but a book thread or writer that can't keep up grows memory.
|
|
- Kalshi production was never run.
|
|
|
|
## Next
|
|
|
|
From the roadmap's possible features ([docs/roadmap.md](docs/roadmap.md)):
|
|
telling a real loss from a fresh book merely ahead of the changes we have,
|
|
so the loss count measures data missed; Polymarket's `new_market` event as
|
|
a faster discovery signal; fewer Snapshots for unchanged books (the brief's
|
|
every minute is about 130 MB a minute at full coverage); a Kalshi
|
|
production run.
|
|
|
|
## Reading the code
|
|
|
|
- [docs/project.md](docs/project.md): what it is, the central bet (the
|
|
pipeline), the design.
|
|
- [docs/adr/](docs/adr): the four decisions that are hard to reverse.
|
|
- [docs/notes.md](docs/notes.md): each exchange's rules, as found.
|
|
- [LATENCY.md](LATENCY.md): extension 4.
|
|
- [GLOSSARY.md](GLOSSARY.md): the terms (a Snapshot is only the output
|
|
message; getting a fresh book is a resync).
|
|
- [roadmap/](roadmap): each sprint's plan, decisions and outcome, in order.
|
|
- [CLAUDE.md](CLAUDE.md): the instructions the coding agent worked under.
|
|
|
|
## Output format
|
|
|
|
The output uses the template's helpers unchanged: `AlignedSbeWriter`
|
|
writes the SBE messages of
|
|
[sbe_schemas/engineering_challenge/challenge_orderbook.yaml](sbe_schemas/engineering_challenge/challenge_orderbook.yaml)
|
|
(`Snapshot` and `PriceLevelUpdate`, prices and quantities in fixed point,
|
|
timestamps in microseconds since the epoch), readable by the template's
|
|
`AlignedSbeReader`. The template's code is `src/util/` and `sbe_schemas/`;
|
|
everything else in `src/` is the collector.
|