feat: ready to submit, a run to watch, the README, the review's fixes

Sprint 08 (roadmap/08. ready to submit.md), the last milestone.

- scripts/watch.sh runs the collector with a live view of its log, stops
  cleanly on Ctrl-C, then checks the output (reconcile) and shows one
  symbol's book as text (show, a new binary).
- The README for the solution, with the final runs: Polymarket through an
  outage it recovered from on its own, Kalshi's demo with two slow
  connections recycled.
- From a review of the whole codebase: Kalshi takes RSA keys too
  (RSA-PSS-SHA256); a slow Kalshi connection takes the hourly recycle's
  next turn, which it nearly never got before; a market tracked again no
  longer fails reconcile; Book::add checks for overflow; usage and --help
  name the key variables; the first scan says how far it got.
- Tests for every gap the reviews found, an offline run of the whole
  orchestrator among them; ShellCheck in CI.
This commit is contained in:
pascal committed 2026-11-12 22:32:39 +01:00
1 parent 515e85941d
commit cf6c16c9cf
35 files changed
+2791 -207

No files matched your search

+1
View File
@@ -1,2 +1,3 @@
/target/ /target/
__pycache__/ __pycache__/
watch/
+29 -5
View File
@@ -23,8 +23,8 @@ outside this repo.
Delta, lateness, lag, speed). Delta, lateness, lag, speed).
- `src/core/`: what is the same for every exchange (ADR 0004). - `src/core/`: what is the same for every exchange (ADR 0004).
`orchestrator.rs`: what is tracked, which connections carry each market, `orchestrator.rs`: what is tracked, which connections carry each market,
their balancing and their recycling (hourly, and sooner for a Kalshi their balancing and their recycling (hourly, a Kalshi connection that
connection that stays among the slowest); `socket.rs`: a connection (read, stays among the slowest taking the next turn); `socket.rs`: a connection (read,
stamp, decode, route; reconnects when dropped, silent, lost a message or stamp, decode, route; reconnects when dropped, silent, lost a message or
recycled); `feed.rs`: a book thread's shell (Snapshots, moves, the quiet recycled); `feed.rs`: a book thread's shell (Snapshots, moves, the quiet
watch) and the `Rule` trait; `mod.rs`: the `Exchange`, `Session`, `Market` watch) and the `Rule` trait; `mod.rs`: the `Exchange`, `Session`, `Market`
@@ -37,8 +37,14 @@ outside this repo.
rank rule, `mod.rs` the exchange and its session (the `seq` check). rank rule, `mod.rs` the exchange and its session (the `seq` check).
- `src/book.rs`: the book and fixed point; `src/reconcile.rs` + - `src/book.rs`: the book and fixed point; `src/reconcile.rs` +
`src/bin/reconcile.rs`: the output check, reading the log's cause lines `src/bin/reconcile.rs`: the output check, reading the log's cause lines
(`src/cause.rs`). `tests/fixtures/`: recordings of the live feeds that the (`src/cause.rs`); `src/show.rs` + `src/bin/show.rs`: one symbol of the
output as text. `tests/fixtures/`: recordings of the live feeds that the
tests replay. tests replay.
- `scripts/watch.sh [EXCHANGE] [MINUTES]`: a run to watch, also the final
runs' and the README's quick start (roadmap/08): the collector under
`timeout -s INT`, a live view of its log (`scripts/watch.awk`, which
parses the minute lines' wording: change one, check the other), then
`reconcile` and `show`. Output and log in `watch/`.
- `src/latency.rs` + `scripts/latency/analyse.py`: the latency probe - `src/latency.rs` + `scripts/latency/analyse.py`: the latency probe
(`engineering-challenge latency EXCHANGE OUT.csv`, until SIGINT: one CSV (`engineering-challenge latency EXCHANGE OUT.csv`, until SIGINT: one CSV
line per copy of a change on each of 16 connections) and its analysis line per copy of a change on each of 16 connections) and its analysis
@@ -67,6 +73,10 @@ and let the user settle it. Claude owns the data model.
- Use `GLOSSARY.md`'s terms: a **Snapshot** is only the output message; getting - Use `GLOSSARY.md`'s terms: a **Snapshot** is only the output message; getting
a fresh book is a **resync**. a fresh book is a **resync**.
- A corner cut for speed carries a comment naming its ceiling and upgrade path. - A corner cut for speed carries a comment naming its ceiling and upgrade path.
- A figure quoted in a doc (README, `LATENCY.md`, a sprint's Outcome) comes
from the tool that checks it, `reconcile`'s output or the log itself,
never from `watch.sh`'s view or a count grouped by hand: sprint 08's
first README figures had four errors from those.
## Environment traps ## Environment traps
@@ -91,7 +101,20 @@ and let the user settle it. Claude owns the data model.
(`pgrep -f '^./target/release/...'`, anchored) and loop on `kill -0 $pid`. (`pgrep -f '^./target/release/...'`, anchored) and loop on `kill -0 $pid`.
- **No `/usr/bin/time` on the server**: for a run's peak memory and thread - **No `/usr/bin/time` on the server**: for a run's peak memory and thread
count, read `VmHWM` and `Threads` from `/proc/<pid>/status` while it runs count, read `VmHWM` and `Threads` from `/proc/<pid>/status` while it runs
(sprint 05's run script sampled them every 30 s). (`scripts/watch.sh` samples them every 10 s).
- **mawk buffers a pipe's input** unless `-W interactive`: a live view
through `awk` shows nothing until the end (`scripts/watch.sh` passes it
when the awk is mawk, as on the server).
- **A job started with `&` in `sh` ignores Ctrl-C**: a script that runs the
collector in the background must pass SIGINT on itself (`watch.sh`'s
trap). To try Ctrl-C without a terminal: `(sleep N; printf '\003') |
script -qfec "CMD" /dev/null`.
- **Polymarket's Gamma answers 429 to runs started close together**: two
collectors' first scans at once (or a few runs within minutes) got every
page refused; the scan retries, but leave a few minutes between runs.
- **`*.pem` is gitignored globally** (`~/.gitignore`): a test key goes in
the source, as `src/kalshi/sign.rs` does with RFC 8032's and RFC 9500's
published test keys.
- **The server is shared and gets overloaded** (load up to 92 on 12 cores, - **The server is shared and gets overloaded** (load up to 92 on 12 cores,
2026-11-11): a starved collector falls behind and gets closed with 1013. 2026-11-11): a starved collector falls behind and gets closed with 1013.
Check `cat /proc/loadavg` before a live run whose numbers are compared, and Check `cat /proc/loadavg` before a live run whose numbers are compared, and
@@ -99,7 +122,8 @@ and let the user settle it. Claude owns the data model.
- **Kalshi needs a key; nothing else does**: `KALSHI_KEY_ID_FILE` and - **Kalshi needs a key; nothing else does**: `KALSHI_KEY_ID_FILE` and
`KALSHI_PRIVATE_KEY_FILE` name its two files (on the server `KALSHI_PRIVATE_KEY_FILE` name its two files (on the server
`/run/secrets/kalshi-demo-key-id` and `kalshi-demo-private-key`; never read `/run/secrets/kalshi-demo-key-id` and `kalshi-demo-private-key`; never read
or print them). Opening ~100 connections at once gets some handshakes or print them). The key is Ed25519 or RSA (RSA-PSS-SHA256), told apart
by the parsed key. Opening ~100 connections at once gets some handshakes
refused with 429; each retries after 1 s and gets in. refused with 429; each retries after 1 s and gets in.
- **cargo keeps a build whose source got an older mtime**: a script that - **cargo keeps a build whose source got an older mtime**: a script that
restores a file from a copy (`shutil.move`, `cp -p`) must `touch` it, or restores a file from a copy (`shutil.move`, `cp -p`) must `touch` it, or
+4 -3
View File
@@ -3,7 +3,7 @@ name = "engineering-challenge"
version = "0.1.0" version = "0.1.0"
edition = "2024" edition = "2024"
build = "src/util/build.rs" build = "src/util/build.rs"
# `cargo run` runs the collector, not `src/bin/reconcile.rs`. # `cargo run` runs the collector, not `src/bin/reconcile.rs` or `show.rs`.
default-run = "engineering-challenge" default-run = "engineering-challenge"
[dependencies] [dependencies]
@@ -14,10 +14,11 @@ chrono = "0.4"
crossbeam = "0.8.4" crossbeam = "0.8.4"
# `termination`: SIGTERM and SIGHUP stop the collector like SIGINT. # `termination`: SIGTERM and SIGHUP stop the collector like SIGINT.
ctrlc = { version = "3", features = ["termination"] } ctrlc = { version = "3", features = ["termination"] }
# Signs Kalshi's handshake (Ed25519 or RSA-PSS); already rustls's crypto
# provider.
ring = "0.17"
# Not used directly: tungstenite leaves rustls's crypto provider to the app, and # Not used directly: tungstenite leaves rustls's crypto provider to the app, and
# rustls panics at runtime unless exactly one (here ring) is enabled. # rustls panics at runtime unless exactly one (here ring) is enabled.
# Signs Kalshi's handshake (Ed25519); already rustls's crypto provider.
ring = "0.17"
rustls = { version = "0.23", default-features = false, features = ["ring"] } rustls = { version = "0.23", default-features = false, features = ["ring"] }
# Reads the PEM of Kalshi's key. # Reads the PEM of Kalshi's key.
rustls-pki-types = { version = "1", features = ["std"] } rustls-pki-types = { version = "1", features = ["std"] }
+8 -2
View File
@@ -48,8 +48,14 @@ _Avoid_: Order book state, depth
**Connection**: **Connection**:
One websocket to the exchange. Every live symbol is subscribed on two One websocket to the exchange. Every live symbol is subscribed on two
connections, chosen independently of each other. connections, chosen independently of each other.
_Avoid_: Socket (for the concept; fine for the code), stream, lane (the _Avoid_: Socket (for the concept; fine for the code), stream (that is one
retired pairing of one connection with one book thread) connection's changes of a symbol), lane (the retired pairing of one
connection with one book thread)
**Stream**:
One connection's changes of one symbol, in the order it delivers them. The
two streams of a symbol carry the same changes, each at its own pace.
_Avoid_: Feed (the book thread's code), using it for the connection
**Load**: **Load**:
How much a market weighs on a connection: the bytes per second of its frames How much a market weighs on a connection: the bytes per second of its frames
+13 -5
View File
@@ -186,8 +186,9 @@ as the median lag), but at the median it wins 9 ms, a tenth of the lag.
- **On Kalshi's demo it is worth tens of milliseconds**, in the tail too - **On Kalshi's demo it is worth tens of milliseconds**, in the tail too
(p99 28% and 42% lower with the fastest pair kept). This met the (p99 28% and 42% lower with the fastest pair kept). This met the
threshold agreed before the runs, so the collector now recycles a Kalshi threshold agreed before the runs, so the collector now recycles a Kalshi
connection that stays slow for 3 minutes running, ahead of its hourly connection that stays slow for 3 minutes running at the hourly
turn: its replacement's speed is drawn afresh. Each duplicate tells how recycle's next turn (one every 30 s or so), in place of the oldest: its
replacement's speed is drawn afresh. Each duplicate tells how
late its connection was against the other's copy (`src/kalshi/feed.rs`); late its connection was against the other's copy (`src/kalshi/feed.rs`);
once a minute the orchestrator ranks the connections by their mean once a minute the orchestrator ranks the connections by their mean
lateness, and one is slow if it is in the slowest quarter and more than lateness, and one is slow if it is in the slowest quarter and more than
@@ -197,9 +198,16 @@ as the median lag), but at the median it wins 9 ms, a tenth of the lag.
quarter always exists: a first version, by median alone, recycled 6 quarter always exists: a first version, by median alone, recycled 6
connections in 20 minutes, one a minute, connections no slower than the connections in 20 minutes, one a minute, connections no slower than the
rest. **In 20 minutes of the collector on the demo (270,000 markets on rest. **In 20 minutes of the collector on the demo (270,000 markets on
120 connections, load 5.7 at the start), no connection was slow by that 120 connections, load 5.7 at the start), none was recycled for it**; the
rule**, so none was recycled for it; the hourly recycle went on (33), and hourly recycle went on (33), and the output reconciled (25 mismatches,
the output reconciled (25 mismatches, each a logged loss). The probe's 16 each a logged loss). That run doesn't say whether one was judged slow: a
later review found the slow recycle waited for the same moment as the
hourly one, which nearly always took it first, and only the recycle was
logged. Now the slow one takes the hourly recycle's turn, and the
judgment is logged on its own. **In 20 minutes on the demo after that
fix** (260,865 markets on 116 connections, 2026-11-12), two connections
were judged slow and recycled: one at a mean lateness of 246 ms, one at
80 ms. The probe's 16
connections, on the 100 busiest markets, differed far more than the connections, on the 100 busiest markets, differed far more than the
collector's; assumed to be because each of the collector's shares its collector's; assumed to be because each of the collector's shares its
markets with many others and is judged against all their copies, which markets with many others and is judged against all their copies, which
+208 -15
View File
@@ -1,21 +1,214 @@
# Read before getting started!! # Order-book collector for Polymarket and Kalshi
First, read [src/main.rs](https://github.com/voluntas-io/engineering-challenge/blob/main/src/main.rs). Then, some background on this repo: My answer to the Voluntas engineering challenge. The collector subscribes
to every live market of one exchange (Polymarket, or Kalshi), keeps an order
book for each symbol, and writes the template's SBE messages: a Delta for
every change, and a Snapshot of every book each minute and whenever a book
is replaced. All four extensions are done: redundant connections,
subscription monitoring, one core for both exchanges, and latency
experiments ([LATENCY.md](LATENCY.md)).
This repo uses the [FIX SBE standard](https://fixtrading.org/packages/simple-binary-encoding-technical-specification-v2-rc2/) Built with Claude Code. I set the architecture (the threading model,
for writing binary messages. The rust code abstracts this away for you, and you can simply use the helpers, as demonstrated in `main.rs`. redundancy, the exchange seam) and settled every design decision; Claude
proposed the options, wrote the code and ran the experiments. Each
decision and its reasons are in [`roadmap/`](roadmap) and
[`docs/adr/`](docs/adr).
### More background, in case you are curious! ## Quick start
FIX SBE is defined in XML, which I find extremely cumbersome to edit, especially On Linux (the script reads `/proc`), with Rust; built and tested with
when updating schemas frequently. To solve this, I have also introduced some 1.98:
helper code that takes our own custom YAML format, and turns it into FIX SBE
XML, which in turn is used to generate rust code to encode and decode the
messages. **You don't need to edit any of these files**. This explanation is
simply to help you understand the origins of the various files in this repo.
If you're curious to see the YAML schema, take a look at [challenge_orderbook.yaml](https://github.com/voluntas-io/engineering-challenge/blob/main/sbe_schemas/engineering_challenge/challenge_orderbook.yaml). ```sh
We also use a custom timestamp field, so that the Rust struct can use scripts/watch.sh # Polymarket for 3 minutes, no key needed
`chrono::DateTime` in the generated struct, and in SBE schema, it is a uint64 scripts/watch.sh polymarket 10 # or longer
representing the timestamp as the number of microseconds since unix epoch. ```
It builds the collector, runs it, and shows what it does each minute:
markets and connections; changes written; duplicates (a change's second
copy, from the other connection carrying its symbol); rescued changes (one
connection missed them, the other delivered them); resyncs (a fresh book
asked for, when ours can't be checked); losses (a fresh book that differed
from ours: changes neither connection delivered). The output and the full
log go to `watch/`. Ctrl-C stops it
early and cleanly. Then it checks the output against the log (`reconcile`)
and prints one busy symbol's book as text (`show`). The first scan lists
every market before subscribing: a minute or two.
## Running it
```sh
cargo build --release
./target/release/engineering-challenge polymarket > out.sbe 2> run.log
./target/release/engineering-challenge polymarket out.sbe 2> run.log # same
```
The output goes to stdout unless a path is given; the log goes to stderr.
SIGINT, SIGTERM or SIGHUP closes the websockets and flushes the output.
Only a clean stop flushes it: the writer fills 64 MiB chunks, the only size
the template's reader opens.
Kalshi needs an API key. Two variables name its files (paths keep the key
out of the shell's history):
```sh
export KALSHI_KEY_ID_FILE=~/kalshi/key-id
export KALSHI_PRIVATE_KEY_FILE=~/kalshi/private-key.pem # Ed25519 or RSA
./target/release/engineering-challenge kalshi out.sbe 2> run.log
./target/release/engineering-challenge kalshi-demo out.sbe 2> run.log
```
I had a key for Kalshi's demo only, so production has never been run; the
code is the same, only the URLs differ.
## Checking the output
```sh
./target/release/reconcile out.sbe run.log
./target/release/show out.sbe [SYMBOL]
```
`reconcile` replays the output: per symbol, the Deltas since a Snapshot,
applied to it, must give the next Snapshot. A Snapshot that differs is
paired with the log line that says why it was written: a loss; a
subscription gap (the few changes a subscribe keeps from us, which the
exchange's opening book already holds); a switch (Polymarket's book now
followed on the other connection); a market tracked again after it left.
It fails on any left unexplained, or on a Delta before its symbol's first
Snapshot.
## Final runs
`scripts/watch.sh`, from the submitted code, on a shared 12-core server.
**Polymarket, 10 minutes** (2026-11-12, system load 7.2 at the start): the
first scan listed 199,531 live markets in 77 s, subscribed on 178
connections. `reconcile`: 14.4 million Deltas and 3.7 million Snapshots;
the 142,990 Snapshots that differ from the book before them each explained
by their log line. 95% of the changes came on both connections, and
417,852 were rescued. Peak memory 2.3 GB. The thread count wasn't sampled
at its peak in this run; it is one per connection plus about 20 (the Kalshi
run's 137 for 116), so about 200.
The run met an outage of about two minutes. Between 16:50:17 and 16:50:25
UTC, 177 of the 178 connections were found silent (a connection is silent
after 30 s with nothing received, so since about 16:49:47). Connects then
timed out or found no route to the host, and between 16:51:37 and 16:51:59
every connection came back on its own. Each book that changed meanwhile was found
wrong on the reconnect and replaced: 138,950 of the run's 140,129 losses
are those two minutes, 1,179 the other eight. Whether the network or
Polymarket went down, the log can't say.
**Kalshi's demo, 20 minutes** (system load 0.8 at the start): the first
scan took 5 min 17 s, the demo's listing being slow (about 1.2 s a page of
1,000 markets in that run); then 260,865 markets on 116 connections.
`reconcile`: 4.1 million Deltas and 4.0 million Snapshots; the 12 that
differ each a logged loss. 98% of the changes came on both connections;
1,228 resyncs. Peak memory 830 MB, 137 threads. Every connection is
recycled (closed and opened again) about once an hour, one at a time; on
Kalshi one that stays slow takes the next turn, and that acted: a
connection's lateness is how long after the other connection's copy it
delivers a change, and connections 99 (246 ms on average) and 57 (80 ms)
were each among the slowest 3 minutes running, so each was recycled at
the next turn, 13 and 21 s later.
## Design
```
rescan fetcher ──findings──▶ ┌──────────────┐ ◀──findings── recent fetcher
│ orchestrator │ ◀── up, down, 1013 ─── connections
└──────┬───▲───┘ ──── recycle ────────▶ connections
Add / Move / Remove (a market and │ │ loads, quiet markets,
its connections), to its thread │ │ lateness, each minute
┌──────▼───┴───┐
connection 0..N ──events───▶ │ book threads │ ──Snapshot | Delta──▶ writer
(read, stamp, parse, route) │ 0..15 │
▲ └──────┬───────┘
└── subscribe / unsubscribe / resync ┘
```
Plain threads and channels, nothing shared ([ADR 0001](docs/adr/0001-concurrency-model.md)).
A connection stamps the receive time right after the socket read, then
parses and routes each event to the book thread that owns its market. Each
book is owned by one of 16 book threads; one writer owns the output. The
orchestrator decides which connections carry each market, balances the
bytes per second each carries, and recycles them ([ADR 0002](docs/adr/0002-lanes-and-orchestrator.md)).
[docs/project.md](docs/project.md) has the whole design.
A lost change leaves a book silently wrong, so every sign of one replaces
the book with a fresh one from the exchange (a resync). On Polymarket the
exchange's book hash is recomputed after every change; on Kalshi it is a
skipped sequence number on a connection. A fresh book that differs from
ours counts as a loss; some of those are only books ahead of our changes,
so the count is an upper bound. The protocol facts the
code relies on, several found only in live data, are in
[docs/notes.md](docs/notes.md).
## The extensions
- **Redundant websockets.** Every symbol is on two connections chosen
independently; its book takes each change from whichever delivers it
first ([ADR 0003](docs/adr/0003-first-one-wins.md)). There is no "source"
to switch, so the brief's Snapshot on a switch becomes a Snapshot
whenever a fresh book from either connection replaces ours. A connection
the exchange closes twice within 10 minutes as too slow (code 1013, "try
again later") has half its markets moved to others.
- **Subscription monitoring.** A subscription is quiet when it delivered
nothing for a minute while the other connection delivered at least 3
changes of the same symbol: it is resubscribed, and moved if quiet
again. Judged against the other connection, not a rate, it holds for
the quietest markets too. Every
connection is recycled about once an hour, one at a time, never two of a
market's; one silent for 30 s reconnects.
- **Generic core.** `src/core/` is the same for both exchanges. An exchange
supplies its websocket codec and its loss rule
([ADR 0004](docs/adr/0004-exchange-seam.md)): the rule is more than
decoding, since telling a duplicate from a loss depends on what the
exchange sends (a book hash, or a sequence number and a clock).
- **Latency.** [LATENCY.md](LATENCY.md): a connection that delivers
changes sooner than the others keeps doing so, on both exchanges.
Keeping the fastest gains under a millisecond on Polymarket, and tens of
milliseconds on Kalshi's demo, where the collector now recycles a
connection that stays slow.
## Limits
- A hard crash (SIGKILL, power) loses the chunk being filled, up to 64 MiB.
- When a symbol is subscribed, the exchange's opening book holds a few
changes it never sends as changes: logged as a subscription gap.
- Polymarket changes held for ordering when the run stops (at most 1 s) are
not written.
- Channels are unbounded: a burst never blocks a socket or drops an event,
but a book thread or writer that can't keep up grows memory.
- Kalshi production was never run.
## Next
From the roadmap's possible features ([docs/roadmap.md](docs/roadmap.md)):
telling a real loss from a fresh book merely ahead of the changes we have,
so the loss count measures data missed; Polymarket's `new_market` event as
a faster discovery signal; fewer Snapshots for unchanged books (the brief's
every minute is about 130 MB a minute at full coverage); a Kalshi
production run.
## Reading the code
- [docs/project.md](docs/project.md): what it is, the central bet (the
pipeline), the design.
- [docs/adr/](docs/adr): the four decisions that are hard to reverse.
- [docs/notes.md](docs/notes.md): each exchange's rules, as found.
- [LATENCY.md](LATENCY.md): extension 4.
- [GLOSSARY.md](GLOSSARY.md): the terms (a Snapshot is only the output
message; getting a fresh book is a resync).
- [roadmap/](roadmap): each sprint's plan, decisions and outcome, in order.
- [CLAUDE.md](CLAUDE.md): the instructions the coding agent worked under.
## Output format
The output uses the template's helpers unchanged: `AlignedSbeWriter`
writes the SBE messages of
[sbe_schemas/engineering_challenge/challenge_orderbook.yaml](sbe_schemas/engineering_challenge/challenge_orderbook.yaml)
(`Snapshot` and `PriceLevelUpdate`, prices and quantities in fixed point,
timestamps in microseconds since the epoch), readable by the template's
`AlignedSbeReader`. The template's code is `src/util/` and `sbe_schemas/`;
everything else in `src/` is the collector.
+8 -3
View File
@@ -237,9 +237,14 @@ is the demo's.
`api.elections.kalshi.com` and `demo-api.kalshi.co` still answer. `api.elections.kalshi.com` and `demo-api.kalshi.co` still answer.
- **Auth:** the websocket handshake carries `KALSHI-ACCESS-KEY` (the key id), - **Auth:** the websocket handshake carries `KALSHI-ACCESS-KEY` (the key id),
`KALSHI-ACCESS-TIMESTAMP` (ms) and `KALSHI-ACCESS-SIGNATURE`: base64 of the `KALSHI-ACCESS-TIMESTAMP` (ms) and `KALSHI-ACCESS-SIGNATURE`: base64 of the
Ed25519 signature of `<timestamp>GET/trade-api/ws/v2`. Without them: 401. signature of `<timestamp>GET/trade-api/ws/v2`. Without them: 401. The
The market listing needs no key. Keys are created per environment key is Ed25519 (RFC 8032 over that text) or 2048-bit RSA (RSA-PSS,
(Account → Profile → API Keys); the private key is a PKCS#8 PEM shown once. SHA-256, MGF1 SHA-256, a salt of the digest's length); Kalshi picks the
check from the public key it holds, and a PEM's header doesn't say which
(an RSA key may be PKCS#1, as Kalshi gives it, or PKCS#8; Kalshi's API
keys page, read 2026-11-12). The market listing needs no key. Keys are
created per environment (Account → Profile → API Keys); the private key
is shown once.
- **Subscribe:** `{"id":1,"cmd":"subscribe","params":{"channels": - **Subscribe:** `{"id":1,"cmd":"subscribe","params":{"channels":
["orderbook_delta"],"market_tickers":[...]}}` is answered ["orderbook_delta"],"market_tickers":[...]}}` is answered
`{"type":"subscribed","id":1,"msg":{"channel":"orderbook_delta","sid":1}}`, `{"type":"subscribed","id":1,"msg":{"channel":"orderbook_delta","sid":1}}`,
+4 -4
View File
@@ -161,11 +161,11 @@ schema debate). As settled in sprints 04 and 05:
- The **orchestrator** owns what is tracked (each market's two connections, - The **orchestrator** owns what is tracked (each market's two connections,
the timers) and decides: placing, moving, splitting, and recycling one the timers) and decides: placing, moving, splitting, and recycling one
connection at a time, about hourly, and sooner a Kalshi connection whose connection at a time, about hourly; a Kalshi connection whose mean
mean lateness stays above twice everyone's (sprint 07, LATENCY.md). It lateness stays above twice everyone's takes the next turn, in place of
never blocks on HTTP. the oldest (sprints 07 and 08, LATENCY.md). It never blocks on HTTP.
- The two **fetchers** only page the exchange's market listing on request - The two **fetchers** only page the exchange's market listing on request
and stream what they read. and send on what they read.
- A **connection** reads, stamps the receive time, parses, and routes each - A **connection** reads, stamps the receive time, parses, and routes each
event to the book thread of its market. Every symbol is on two connections, event to the book thread of its market. Every symbol is on two connections,
chosen independently. One silent for 30 s reconnects. chosen independently. One silent for 30 s reconnects.
+19 -4
View File
@@ -91,11 +91,17 @@ abstraction is *extracted* once a second exchange exists, not designed up front.
The source port can't be chosen here (a NAT rewrites it); first after The source port can't be chosen here (a NAT rewrites it); first after
maintenance wasn't tried. maintenance wasn't tried.
8. **Polish + submission** — *not started* 8. **Polish + submission** — *done (2026-11-12)*
Final review, README for the solution (the template's README still says the A review of the whole codebase and its fixes, a run anyone can watch
helpers are "demonstrated in `main.rs`", whose demo M1 replaced), (`scripts/watch.sh`, `show`), the README, final runs of both
packaging/handoff for the challenge. exchanges, and the handoff as a public repo on git.pband.ch.
Sprint: [../roadmap/08. ready to submit.md](../roadmap/08.%20ready%20to%20submit.md).
*Ends:* ready to submit. *Ends:* ready to submit.
Result: Kalshi takes RSA keys too; the slow recycle, which the hourly
one nearly always pre-empted, now takes its turn and acted twice in 20
live minutes; a market tracked again no longer fails `reconcile`; 19
surviving mutants got their tests. Both final runs reconciled, one
through an outage of about two minutes (sprint 08's Outcome).
## Why this order ## Why this order
@@ -155,3 +161,12 @@ abstraction is *extracted* once a second exchange exists, not designed up front.
- Judging the connections' speed at the book threads' wall-clock minute, - Judging the connections' speed at the book threads' wall-clock minute,
not on the orchestrator's own timer (a judgment near the crossing sees not on the orchestrator's own timer (a judgment near the crossing sees
partial samples; sprint 07's review). partial samples; sprint 07's review).
- Kalshi: a resync asked from a connection the symbol has since left stays
asked there while the other connection is down (sprint 08's review; rare,
and the resync count grows each minute).
- Paging Kalshi's listing in parallel: on 2026-11-12 the demo served a page
of 1,000 markets in 1.7 s, so the first scan took 7.5 minutes (under one
in sprint 06).
- Connecting to each of the exchange's addresses in turn, not only the
first the lookup returns (a host with IPv6 DNS but no IPv6 route would
never connect; sprint 08's review).
+425
View File
@@ -0,0 +1,425 @@
# 08. Ready to submit
**Status: done (2026-11-12)**
M8, the last milestone: the collector handed to Voluntas in a state a
reviewer can read, run and check on their own. A review of the whole
codebase (not one sprint's diff) and its fixes; one command that runs the
collector and shows what it does; a README for the solution in place of
the template's; a final live run of each exchange, reconciled, whose
figures the README quotes; and the handoff as a public repo.
## The watched run
`scripts/watch.sh [EXCHANGE] [MINUTES]`, defaults `polymarket` (no key) and
3 minutes (two minute summaries; Polymarket tracks about 200,000 markets
within its first minute, `../runs/poly10.log`). It builds release, runs the
collector under `timeout -s INT`, and writes the output and the full log
to a gitignored `watch/`. On the terminal, a curated view of the log:
- each minute, the orchestrator's line as is, and the 16 book threads'
lines summed into one (changes written, duplicates, rescued, resyncs,
losses, subscription gaps; their counts are cumulative, "so far");
- between them, the notable events only: a connection lost, recycled
(hourly or slow), split, silent; the startup's connections as one count,
not a line each. The Snapshot cause lines (3,300 of 3,952 lines in a
10-minute Polymarket run) stay in the file.
Ctrl-C stops the run early and cleanly: the script goes on to the checks
(how Ctrl-C reaches `timeout` and the collector from a script is
**assumed**; a quick try without a terminal didn't deliver the signal, so
the first task tries it in one). Then it runs `reconcile` on the output
and the log, and `show` on the output.
On the workspace: `rc shell voluntas`, then `cd voluntas/repo &&
scripts/watch.sh` (`rc shell`'s interactive use is assumed from
`rc shell exec`). Kalshi needs `KALSHI_KEY_ID_FILE` and
`KALSHI_PRIVATE_KEY_FILE` exported; without them the collector stops
before connecting, naming the variable (`src/kalshi/sign.rs:26`). The
script adds nothing for keys.
## show
`src/bin/show.rs`: `show OUTPUT [SYMBOL]`, reading through the template's
`AlignedSbeReader` as `src/reconcile.rs` does. With no symbol, the one
with the most Deltas in the file. It prints that symbol's first Snapshot
(top 5 levels a side, its receive time), its next 20 Deltas (receive time,
side, price, new quantity; 0 shown as deleted), then the top 5 levels a
side of the book folded from them, in exchange units through
`book::format_fixed` (`src/book.rs:46`). A Polymarket symbol is its token
id: the output holds no market name. An unknown symbol is an error.
## The review
The whole codebase, not a diff: `src/`, `scripts/`, the tests, by five
`reviewer` subagents:
- **brief**, in place of a spec reviewer: the brief checked line by line
against the code (every live symbol, MVEs excluded on Kalshi, new and
gone symbols, Deltas to stdout and a Snapshot each minute, SIGINT closes
the websockets and the file, each extension);
- **smells**, read as a stranger: leftovers of superseded designs (lanes,
the template's demo), comments that need sprint history (19 citations
of `roadmap/0N` or "sprint 0N" in `src/`), the two exchanges
inconsistent with each other;
- **bugs**, **tests**, **security**, as usual; sprint 07's 11 post-review
fixes, never reviewed a second time, included.
Findings are sorted with the user: fix now (a task below, with its test),
or a limit or a "next" in the README.
## The README
Replaces the template's, for the Voluntas reader, in this order: what the
collector does, with the line on how it was built (below); quick start
(`scripts/watch.sh`); build and run per exchange, Kalshi's two `export`
lines, and that Kalshi ran only against the demo; checking the output
(`reconcile` with the log); the final runs' figures; the design in one
diagram (`docs/project.md`'s pipeline); each extension and where it is
met (ADRs, `LATENCY.md`); the limits (the crash ceiling, the subscription
gap when a symbol opens, Kalshi production never run); next (a short
pick of the roadmap's possible features); what to read first
(`docs/project.md`, `docs/adr/`, `LATENCY.md`, `docs/notes.md`, and
`GLOSSARY.md`, `CLAUDE.md`) and that `roadmap/` holds each sprint's plan
and outcome; an "Output format" section (the template's
`AlignedSbeWriter` unchanged, 64 MiB chunks, the schema's path). The
order as built supersedes the first one (the runs after the extensions,
the line on how it was built last): the runs read as evidence for the
design that follows, and the line belongs with what the project is. The
line:
> Built with Claude Code. I set the architecture (the threading model,
> redundancy, the exchange seam) and settled every design decision;
> Claude proposed the options, wrote the code and ran the experiments.
> Each decision and its reasons are in `roadmap/` and `docs/adr/`.
## Decisions
- **The watched run is a script over the existing log**, not pipeline
code. Rejected: a status line in the collector (counts across threads,
new pipeline code); a TUI.
- **The data is shown, not only counted** (`show`), after the run: the
writer emits 64 MiB chunks, so a live stream of the output would come
in bursts.
- **One script for everything**: the final runs and the README's quick
start are `scripts/watch.sh`, so the grader runs what made the
README's figures. The workspace's `../runs/run.sh` is retired.
- **Everything ships unchanged** (`docs/`, `roadmap/`, `CLAUDE.md`,
`GLOSSARY.md`, `LATENCY.md`); the README says what to read first.
Rejected: stripping the process files; the work being AI-assisted is
said openly instead.
- **Sprint citations in comments**: a file path stays (`roadmap/` ships);
a reference that leans on history is rewritten to say what the code
does, or to cite the ADR or `docs/notes.md`.
- **The final runs** pass `reconcile` (every mismatch explained, no
orphan Delta), load and peak memory recorded (`VmHWM` from `/proc`);
no comparison with earlier sprints' runs (done in their sprints; the
load differs from day to day).
- **The template's README text**: its stale lines go ("read `main.rs`",
"as demonstrated in `main.rs`"); the rest becomes "Output format".
- **Delivery: a public repo on the user's Gitea (git.pband.ch), not on
GitHub.** The brief says nothing on how (it links only the template,
public, 25 forks). Public, so the link opens with no account; on
GitHub it would publish a solution to the challenge where other
candidates search, which the user doesn't want. Rejected: a GitHub
fork (listed under the template), a GitHub repo of our own (found by
search), a private repo (an invite to a handle not known), a PR to the
template (public). Full history, the template's 3 commits first, so a
diff against it shows what was added. Pushed over SSH (which reaches
Gitea from here), with the user's go. Supersedes "a public repo of the
user's own on GitHub", at the user's request during the build.
- **Kalshi's RSA keys are taken** (from the review): RSA-PSS-SHA256 with
`ring`, PKCS#1 or PKCS#8, told from Ed25519 by the parsed key; tested
on RFC 9500's published test key (no private key of our own in a public
repo; `*.pem` is gitignored globally); checked
against `openssl`. Supersedes sprint 06's "Ed25519 only", because a
grader's key from Kalshi may well be RSA.
- **A slow connection takes the hourly recycle's next turn** (from the
review), in place of being recycled at once: the slow recycle waited for
the same moment as the hourly one, which nearly always took it first.
One path; the mark is logged when made. Supersedes sprint 07's "recycled
at once, ahead of the hourly order". `LATENCY.md`'s "no connection was
slow" in 20 live minutes rested on the starved count: reworded.
- **The final runs, as they went.** Polymarket's 10 minutes met an outage
of about two minutes (177 of 178 connections found silent at 16:50:17,
so since about 16:49:47; back from 16:51:38 to 16:51:59): kept in the
README, as it shows the recovery; 138,950 of its 140,129 losses (the
log's and `reconcile`'s) are those two minutes. Kalshi's 10-minute run spent 7.5
of them on the first scan (the demo's listing at 1.7 s a page) and saw
74 normal closes by the demo, never seen before; an A/B of `main`'s
build against the branch's, 10 minutes each, saw none on either, so
the closes were the demo's. The README's Kalshi run is 20 minutes,
long enough for the slow judgment (from a connection's 10th minute):
it recycled two. Supersedes "kalshi-demo 10". Logs in `../runs/`.
- **ShellCheck in CI**: `uvx --from shellcheck-py==0.10.0.1 shellcheck`
over `scripts/*.sh` (tried: 0.10.0, `scripts/ci.sh` clean).
## Tasks
- [x] Ctrl-C through the script, tried in a terminal: `timeout -s INT`
and the collector stop cleanly and the script goes on to the
checks; the script's handling settled from what happens.
- [x] `scripts/watch.sh`: build, bounded run, `watch/` (gitignored), the
curated view, then `reconcile` and `show`.
- [x] The view's summed book-thread line checked once by hand against
the 16 raw lines of the same minute.
- [x] ShellCheck in `scripts/ci.sh`.
- [x] `src/bin/show.rs`.
- [x] Test (`show`): on an output written by `AlignedSbeWriter` in the
test, the symbol with the most Deltas is picked; a symbol named is
used; its Deltas come in order and the folded book is right; an
unknown symbol is an error.
- [x] The review: five reviewers over the whole codebase; findings
sorted with the user, the fix-now ones added here as tasks.
- [x] The citations in comments the review lists, rewritten.
- [x] A short run, stopped with Ctrl-C partway: the output flushes and
reconciles.
- [x] The final runs, `scripts/watch.sh polymarket 10` and
`scripts/watch.sh kalshi-demo 20`, from the branch with every fix
in, load checked before each; both reconcile; load and peak
memory recorded.
- [x] `README.md`, its figures from those runs.
- [x] `docs/roadmap.md` (M8 done) and `CLAUDE.md` (watch, show) brought
up to date.
- [x] The stale branches `feat/full-coverage` and
`feat/redundant-connections` deleted.
- [ ] After the squash into `main`, with the user's go: the public repo
created on git.pband.ch and pushed, and its link opened logged
out.
From the review (2026-11-12, at `73a1551`):
- [x] Kalshi's RSA keys (PKCS#1 PEM, as Kalshi generates them), signed
RSA-PSS-SHA256 with `ring` (the scheme assumed until checked in
Kalshi's docs); Ed25519 kept. Test: an RSA signature verifies (with
`ring`; checked once by hand with `openssl`); a key of neither kind
is refused.
- [x] Usage names the two `KALSHI_*` variables; `--help` and `-h` print
it; the missing-key error names both variables.
- [x] The first scan is logged: when it starts, and every 10 s the
markets listed so far; `watch.awk` shows them.
- [x] A slow connection takes the hourly recycle's next turn, in place of
the oldest: one recycle path. Test: with the hourly recycle running,
a connection slow 3 minutes is the next recycled.
- [x] A symbol tracked again after its market was removed: its first
Snapshot logs a cause, tracked again, which `reconcile` reads. Test:
remove, add, the Snapshot reconciles with the log.
- [x] `watch.sh` exits non-zero when the collector or `reconcile` fails.
- [x] `book::add` checks for overflow; an overflow counts as a loss and
resyncs. Test.
- [x] Test: `orchestrator::run` offline against the local listing and
websocket servers; on stop, each socket gets a Close and the output
reconciles.
- [x] Tests: a command sent on an open connection reaches the server; a
1013 close becomes a split.
- [x] Tests for the review's surviving mutants: Kalshi's rule
(`src/kalshi/feed.rs` 94, 114, 215, 285), the shell
(`src/core/feed.rs` 483, 539, 589, 683, 687), `show` (several asks,
the 20-Delta cap, the first ladder), `reconcile` (orphan Deltas, one
cause line explains one Snapshot), the socket (a dead book thread
stops it, the PING).
- [x] Test: a sample log through `watch.awk`, from `scripts/ci.sh`.
- [x] `a_silent_connection_reconnects_with_its_tokens` fails instead of
hanging (a timeout on the server's accept).
- [x] "ponytail:" becomes "Ceiling:" (4 places); "lag" for lateness and
"stream" reworded (or "stream" in `GLOSSARY.md`); the 4 history
citations rewritten; `Cargo.toml`'s comments, the `CHUNK`
constants' names, `impl RecentChange`'s place, "the server" in
`watch.sh`.
From the review of `26b4d67`:
- [x] `watch.sh` records the peak thread count, not the last sample.
- [x] README: Polymarket's threads (one per connection plus about 20,
derived from the Kalshi run: 137 = 116 + 21, labelled so); losses
140,129 (the log's and `reconcile`'s), the same in this file; the
first reconnect at 16:51:38; 177 of the 178 connections; the outage
"about two minutes" (silence is noticed after 30 s), the roadmap
too; Kalshi's scan at about 1.2 s a page; "every sign of a loss
resyncs, and the count includes some that weren't"; "system load";
each term defined at its first use; `reconcile`'s Delta count only;
the unwrapped line.
- [x] This file: the RSA task says the `openssl` check was by hand; the
README section's order follows the README; "Done when" names the
20-minute Kalshi run.
- [x] Test: a "tracked again" line explains a mismatch in `reconcile`.
- [x] Test: an overflowing Kalshi change asks a resync and removes the
level.
- [x] The view prints a minute's sum on the first line 2 s or more after
its first book line, too (a book thread with nothing logs none).
Test: a minute missing one thread is printed before the
orchestrator's line.
- [x] "Next to recycle" is logged when the mark is new only; the slow log
text in one place. Test.
- [x] Test: the `reconcile` binary fails on an unexplained mismatch.
- [x] Test: a mark is cleared when a later judgment finds the connection
no longer slow.
- [x] Test: `--help` prints the usage and exits 0.
- [x] The two timing-dependent tests poll until their condition, under a
timeout, in place of a fixed sleep.
- [x] `gamma.rs`'s "stream what they read"; `orchestrator.rs:60`'s
unwrapped doc comment; the Kalshi test's hand-built `Market` uses
`market()`.
From the review of `d94f683`:
- [x] The view's window is 10 s, to the millisecond and across midnight:
a minute's book-thread lines spread 1.6 s in the final Polymarket
run, and the 2 s rule, in whole seconds, could split its totals.
Tests: a thread 1.6 s late, live and final; across midnight.
- [x] "Next to recycle" said once per connection while it stays slow,
even when two marked connections swap ranks. Test.
- [x] README: silent between 16:50:17 and 16:50:25; back from 16:51:37;
recycling explained before the slow recycle; a line wrapped.
- [x] The `reconcile` binary's test matches `unexplained: 1 at `, not the
symbol alone.
Each test is seen to fail once (the code it protects removed, then put
back).
## Review of `d94f683` (2026-11-12)
The 7 commits after the second review's tasks, one reviewer on the five
questions. Fix now: the four tasks above. No: the test of an unknown
exchange (unasked, kept: cheap and useful); the 1013 test's copy of
`client` (changing `client` touches every socket test); `output(orphan,
mismatched)`'s two flags (a test helper). Assumed, written so in
`watch.awk` before: "they come within milliseconds", which the final
run's log contradicted.
## Review of `26b4d67` (2026-11-12)
The diff since `73a1551`, five reviewers; none of the findings dropped.
Fix now: the tasks above. Otherwise:
- **Spec:** no: the Polymarket run predating the view's fix (`src/`
unchanged since; the sums recomputed on the log); "sprint 08's Outcome"
(written at the close); the README's longer reading list.
- **Tests:** no: `watch.sh`'s exit status (needs the live exchange or a
fake collector; tried by hand); "neither kind" with an EC key (garbage
covers the refusal); the first-scan lines typed in `test_watch.sh`
(`CLAUDE.md` says change both).
- **Smells:** no: `Symbol::again` carrying the exchange's name
(documented); `64 << 20` in the library's tests (the binary's constant
isn't visible there); `run_tests` serving Kalshi locally (real sockets).
- **Security:** no vulnerability.
## Review of `73a1551` (2026-11-12)
The whole codebase, five reviewers (brief, bugs, tests, smells,
security); the tests reviewer ran 40 mutants, 19 survived. 2 findings
dropped (files since committed; a 429 that recovered). Fix now: the
tasks above. Otherwise:
- **Brief:** quiet judged against the twin connection, a `Rule` beyond
encode/decode, first one wins in place of a source switch: the README
explains each with its ADR. No: `/dev/stdout` truncating a `>>`.
- **Bugs:** later (`docs/roadmap.md`): a Kalshi resync left asked on a
connection the symbol left; only the first resolved address tried. No:
Polymarket changes held at stop, at most 1 s (a README limit); the
probe's exit 0 when every connection drops.
- **Smells:** no: the two fetchers, `Rule::Symbol`, the socket's data
clumps, `on_book`'s length, the exchanges' decoding and construction.
- **Tests, as built:** every new test seen red on its mutant, but one:
removing `drop(outputs)` at the top of `orchestrator::run` changes
nothing a test can see (`outputs` drops when `run` returns, which is
after `threads.join()`); kept, as it says when the writer may end.
- **Security:** no secret in the tree or history. No: unbounded channels
(never dropping is the design; a README limit), raw exchange text in
the log, the template's `unsafe-libyaml`. Kept, the user's call: the
commit email and `CLAUDE.md`'s DNS lines in the public history.
## Not in this sprint
The trading simulator of `docs/evaluation.md` (its fill model is a project
of its own; reconcile, lateness and rescued counts already measure the
claims); the roadmap's possible features (the README's "next");
containerized packaging; a Kalshi production run (no key).
## Done when
- The review's findings are sorted; the fix-now ones are fixed and
tested.
- `scripts/watch.sh` and `show` work; Ctrl-C was tried.
- Both final runs (Polymarket 10 minutes, Kalshi's demo 20) reconcile,
load and memory recorded.
- `README.md` is written, its figures from those runs.
- The stale branches are deleted.
- `scripts/ci.sh` is green, ShellCheck included.
After the squash:
- The public repo is created on git.pband.ch and pushed, with the user's
go, and its link works logged out.
## Outcome
The collector is ready to hand over. `scripts/watch.sh` runs it with a live
view, then `reconcile` and `show` (one symbol's book as text); the README
replaces the template's; three reviews (the whole codebase, then each round
of fixes) and their fixes are in; both final runs reconciled.
- **Fixed, from the whole-codebase review:** Kalshi takes RSA keys too
(RSA-PSS-SHA256, PKCS#1 or PKCS#8, told by the parsed key); the slow
recycle, which the hourly one nearly always took first, now marks the
connection and takes the next turn, and acted twice in the 20-minute
Kalshi run; a market tracked again says so, so `reconcile` doesn't fail
on it; `Book::add` checks for overflow; usage, `--help` and the
missing-key error name both key variables; the first scan says how far
it got. 19 surviving mutants got their tests, and an offline run of the
whole orchestrator proves SIGINT closes every socket and the output
reconciles.
- **Final runs:** Polymarket, 10 minutes, 199,531 markets on 178
connections, 14.4 million Deltas, through an outage of about two minutes
that it recovered from on its own (138,950 of its 140,129 losses).
Kalshi's demo, 20 minutes, 260,865 markets on 116 connections, 4.1
million Deltas, 12 losses, two slow connections recycled.
**Diverged from the plan**, each written into this file when it
happened: delivery on the user's Gitea, not GitHub (the user's call, so as
not to publish a solution where candidates search); RSA keys taken
(sprint 06 had Ed25519 only), tested on RFC 9500's published key since
`*.pem` is ignored globally; the slow recycle takes the hourly turn, not
recycled at once; the Kalshi final run is 20 minutes, not 10 (the demo's
listing took 5 to 7.5 minutes to page); the README's sections in another
order; `LATENCY.md`'s "no connection was slow" withdrawn, since it rested
on the starved count.
**Left out**, and where it went: a Kalshi resync left asked on a
connection the symbol left; connecting to every resolved address;
paging Kalshi's listing in parallel (`docs/roadmap.md`, possible
features). Unexplained, recorded: the Polymarket outage's cause, and the
74 normal closes by Kalshi's demo in one run (an A/B ruled out our code).
**Assumed**, not shown: that the outage was on the network's side ("no
route to host"); that RSA keys work against Kalshi itself (tried offline
only, against `openssl`); `rc shell`'s interactive use; Polymarket's
thread count, about 200, derived from the Kalshi run's.
**Reviewed** at `73a1551` (the whole codebase), `26b4d67` and `d94f683`;
the 3 commits after `d94f683` are its fixes, each with a test seen to
fail, and had no further review.
### Retrospective
- **Went well:** reviewing the whole codebase, not the sprint's diff, found
a starved mechanism that had made a sprint 07 claim false; a sprint diff
never would have. Rerunning one key mutant of each subagent's tests, and
an A/B against `main`'s build, kept their claims and the demo's closes
honest. The user's watchable run turned out to be the best test of the
view: each live run found something.
- **Went wrong:** my own instruments misled me four times: `head` cut the
mutation output so a red test looked green, `$?` after a pipe reported
the pipe's last command, the view summed part of a minute, and a count
grouped by 10 s gave a reconnect time 8 s early. The README's first
figures came from the view and from memory, not from `reconcile` and the
log, and two reviews corrected them. The view's 2 s window rested on
"they come within milliseconds", never measured. Three review rounds
where one careful round of fixes would have done.
- **Tested enough?** The fixes are, by removal. Not: `watch.sh`'s exit
status and its sampling (live only), RSA against Kalshi, the "tracked
again" log line itself (stderr), the first-scan lines' wording against
the view (typed in the test).
- **Next time:** take every quoted figure from the tool that checks it
(`reconcile`, the log), never from a summary of it; check an exit status
without a pipe; measure a timing before building a rule on it.
+5
View File
@@ -13,3 +13,8 @@ cargo test
uvx ruff@0.15.0 format --check scripts/latency uvx ruff@0.15.0 format --check scripts/latency
uvx ruff@0.15.0 check scripts/latency uvx ruff@0.15.0 check scripts/latency
python3 -m unittest discover -s scripts/latency python3 -m unittest discover -s scripts/latency
# The shell scripts: ShellCheck pinned, run through uvx as ruff is.
uvx --from shellcheck-py==0.10.0.1 shellcheck scripts/*.sh
# The view of watch.sh, on a sample log.
sh scripts/test_watch.sh
+157
View File
@@ -0,0 +1,157 @@
#!/bin/sh
# Pipes a sample log of each exchange through scripts/watch.awk, live and
# with -v final=1, and compares what it prints with what the view must show;
# exits non-zero on a difference. The lines are worded as the collector
# writes them: the book threads' minute lines (log_minute in
# src/polymarket/feed.rs and src/kalshi/feed.rs), the orchestrator's
# (src/core/orchestrator.rs), the connections' (src/core/socket.rs) and a
# Snapshot's cause (src/cause.rs).
set -eu
cd "$(dirname "$0")/.."
failed=0
# check NAME LOG EXPECTED [AWK OPTION...]
check() {
name=$1 log=$2 expected=$3
shift 3
actual=$(printf '%s\n' "$log" | awk -v book_threads=2 "$@" -f scripts/watch.awk)
if [ "$actual" != "$expected" ]; then
printf 'watch.awk, %s: printed\n%s\nexpected\n%s\n' "$name" "$actual" "$expected"
failed=1
fi
}
polymarket=$(
cat <<'EOF'
12:00:00.100 polymarket: orchestrator: first scan, listing the exchange's markets
12:00:30.200 polymarket: orchestrator: first scan done, 40 live markets, subscribing
12:00:30.300 polymarket: connection 0: connected, 40 symbols
12:00:30.310 polymarket: connection 1: connected, 40 symbols
12:01:00.001 polymarket: book thread 0: 10 symbols; so far 100 changes written (90 verified), 50 duplicates, 2 rescued, 0 held dropped, 1 chains broken, 3 resyncs, 0 full books behind ours, 1 subscription gaps, 2 losses, 0 switches, 0 quiet subscriptions resubscribed, 0 markets reported quiet again, 4 reconnect books judged (1 losses); duplicates' lateness 1.5 ms at the median, 9.0 ms at p99; this minute 150 changes received, at most 4 events waited
12:01:00.002 polymarket: book thread 1: 30 symbols; so far 300 changes written (290 verified), 150 duplicates, 4 rescued, 0 held dropped, 0 chains broken, 1 resyncs, 0 full books behind ours, 0 subscription gaps, 1 losses, 0 switches, 0 quiet subscriptions resubscribed, 0 markets reported quiet again, 0 reconnect books judged (0 losses); duplicates' lateness 2.0 ms at the median, 8.0 ms at p99; this minute 450 changes received, at most 9 events waited
12:01:00.050 polymarket: Snapshot of 123 at 1794390060050000: loss
12:01:30.400 polymarket: orchestrator: 40 markets on 2 connections; 40 added, 0 removed, 0 moved (0 quiet), 0 splits, 0 recycled (0 slow), 0 silent this minute; busiest connection at 1200 bytes/s
12:02:00.001 polymarket: book thread 0: 10 symbols; so far 160 changes written (150 verified), 80 duplicates, 3 rescued, 0 held dropped, 1 chains broken, 3 resyncs, 0 full books behind ours, 1 subscription gaps, 2 losses, 0 switches, 1 quiet subscriptions resubscribed, 0 markets reported quiet again, 4 reconnect books judged (1 losses); duplicates' lateness 1.5 ms at the median, 9.0 ms at p99; this minute 90 changes received, at most 2 events waited
12:02:00.002 polymarket: book thread 1: 30 symbols; so far 500 changes written (480 verified), 250 duplicates, 6 rescued, 0 held dropped, 0 chains broken, 2 resyncs, 0 full books behind ours, 0 subscription gaps, 1 losses, 0 switches, 0 quiet subscriptions resubscribed, 0 markets reported quiet again, 0 reconnect books judged (0 losses); duplicates' lateness 2.0 ms at the median, 8.0 ms at p99; this minute 300 changes received, at most 5 events waited
12:02:10.000 polymarket: connection 1: recycled
12:02:11.000 polymarket: connection 1: connected, 40 symbols
12:02:30.400 polymarket: orchestrator: 40 markets on 2 connections; 0 added, 0 removed, 0 moved (0 quiet), 0 splits, 1 recycled (0 slow), 0 silent this minute; busiest connection at 1300 bytes/s
EOF
)
check "Polymarket, live" "$polymarket" "$(
cat <<'EOF'
12:00:00 polymarket: orchestrator: first scan, listing the exchange's markets
12:00:30 polymarket: orchestrator: first scan done, 40 live markets, subscribing
12:00:30 connections opening (counted in the minute line)
12:01 books (2 threads): 40 symbols, 400 changes written, 200 duplicates, 6 rescued, 4 resyncs, 1 subscription gaps, 3 losses
12:01:30 40 markets on 2 connections; 40 added, 0 removed, 0 moved (0 quiet), 0 splits, 0 recycled (0 slow), 0 silent this minute; busiest connection at 1200 bytes/s; 2 connected (0 refused, retried), 1 Snapshots with a cause
12:02 books (2 threads): 40 symbols, 660 changes written, 330 duplicates, 9 rescued, 5 resyncs, 1 subscription gaps, 3 losses
12:02:10 polymarket: connection 1: recycled
12:02:30 40 markets on 2 connections; 0 added, 0 removed, 0 moved (0 quiet), 0 splits, 1 recycled (0 slow), 0 silent this minute; busiest connection at 1300 bytes/s; 1 connected (0 refused, retried), 0 Snapshots with a cause
EOF
)"
check "Polymarket, final" "$polymarket" \
"12:02 books (2 threads): 40 symbols, 660 changes written, 330 duplicates, 9 rescued, 5 resyncs, 1 subscription gaps, 3 losses" \
-v final=1
# A minute with one of its two book threads' lines: printed with the first
# line 10 s after it, before the orchestrator's.
missing=$(
cat <<'EOF'
12:01:00.001 polymarket: book thread 0: 10 symbols; so far 100 changes written (90 verified), 50 duplicates, 2 rescued, 0 held dropped, 1 chains broken, 3 resyncs, 0 full books behind ours, 1 subscription gaps, 2 losses, 0 switches, 0 quiet subscriptions resubscribed, 0 markets reported quiet again, 4 reconnect books judged (1 losses); duplicates' lateness 1.5 ms at the median, 9.0 ms at p99; this minute 150 changes received, at most 4 events waited
12:01:00.500 polymarket: connection 1: recycled
12:01:30.400 polymarket: orchestrator: 40 markets on 2 connections; 0 added, 0 removed, 0 moved (0 quiet), 0 splits, 1 recycled (0 slow), 0 silent this minute; busiest connection at 1200 bytes/s
EOF
)
check "a book thread missing" "$missing" "$(
cat <<'EOF'
12:01:00 polymarket: connection 1: recycled
12:01 books (1 threads): 10 symbols, 100 changes written, 50 duplicates, 2 rescued, 3 resyncs, 1 subscription gaps, 2 losses
12:01:30 40 markets on 2 connections; 0 added, 0 removed, 0 moved (0 quiet), 0 splits, 1 recycled (0 slow), 0 silent this minute; busiest connection at 1200 bytes/s; 0 connected (0 refused, retried), 0 Snapshots with a cause
EOF
)"
# A thread's line 1.6 s after the first, past a line between them: still
# the same minute's, summed whole.
late=$(
cat <<'EOF'
12:01:00.075 polymarket: book thread 0: 10 symbols; so far 100 changes written (90 verified), 50 duplicates, 2 rescued, 0 held dropped, 1 chains broken, 3 resyncs, 0 full books behind ours, 1 subscription gaps, 2 losses, 0 switches, 0 quiet subscriptions resubscribed, 0 markets reported quiet again, 4 reconnect books judged (1 losses); duplicates' lateness 1.5 ms at the median, 9.0 ms at p99; this minute 150 changes received, at most 4 events waited
12:01:01.600 polymarket: connection 1: recycled
12:01:01.672 polymarket: book thread 1: 30 symbols; so far 300 changes written (290 verified), 150 duplicates, 4 rescued, 0 held dropped, 0 chains broken, 1 resyncs, 0 full books behind ours, 0 subscription gaps, 1 losses, 0 switches, 0 quiet subscriptions resubscribed, 0 markets reported quiet again, 0 reconnect books judged (0 losses); duplicates' lateness 2.0 ms at the median, 8.0 ms at p99; this minute 450 changes received, at most 9 events waited
EOF
)
check "a thread 1.6 s late" "$late" "$(
cat <<'EOF'
12:01:01 polymarket: connection 1: recycled
12:01 books (2 threads): 40 symbols, 400 changes written, 200 duplicates, 6 rescued, 4 resyncs, 1 subscription gaps, 3 losses
EOF
)"
check "a thread 1.6 s late, final" "$late" \
"12:01 books (2 threads): 40 symbols, 400 changes written, 200 duplicates, 6 rescued, 4 resyncs, 1 subscription gaps, 3 losses" \
-v final=1
# Across midnight: a minute begun at 23:59 still ends 10 s on.
midnight=$(
cat <<'EOF'
23:59:00.001 polymarket: book thread 0: 10 symbols; so far 100 changes written (90 verified), 50 duplicates, 2 rescued, 0 held dropped, 1 chains broken, 3 resyncs, 0 full books behind ours, 1 subscription gaps, 2 losses, 0 switches, 0 quiet subscriptions resubscribed, 0 markets reported quiet again, 4 reconnect books judged (1 losses); duplicates' lateness 1.5 ms at the median, 9.0 ms at p99; this minute 150 changes received, at most 4 events waited
00:00:30.000 polymarket: connection 1: recycled
EOF
)
check "across midnight" "$midnight" "$(
cat <<'EOF'
23:59 books (1 threads): 10 symbols, 100 changes written, 50 duplicates, 2 rescued, 3 resyncs, 1 subscription gaps, 2 losses
00:00:30 polymarket: connection 1: recycled
EOF
)"
# A line between a minute's book threads: the minute is still summed whole.
interleaved=$(
cat <<'EOF'
12:01:00.001 polymarket: book thread 0: 10 symbols; so far 100 changes written (90 verified), 50 duplicates, 2 rescued, 0 held dropped, 1 chains broken, 3 resyncs, 0 full books behind ours, 1 subscription gaps, 2 losses, 0 switches, 0 quiet subscriptions resubscribed, 0 markets reported quiet again, 4 reconnect books judged (1 losses); duplicates' lateness 1.5 ms at the median, 9.0 ms at p99; this minute 150 changes received, at most 4 events waited
12:01:00.002 polymarket: connection 1: recycled
12:01:00.003 polymarket: book thread 1: 30 symbols; so far 300 changes written (290 verified), 150 duplicates, 4 rescued, 0 held dropped, 0 chains broken, 1 resyncs, 0 full books behind ours, 0 subscription gaps, 1 losses, 0 switches, 0 quiet subscriptions resubscribed, 0 markets reported quiet again, 0 reconnect books judged (0 losses); duplicates' lateness 2.0 ms at the median, 8.0 ms at p99; this minute 450 changes received, at most 9 events waited
EOF
)
check "a line between book threads" "$interleaved" "$(
cat <<'EOF'
12:01:00 polymarket: connection 1: recycled
12:01 books (2 threads): 40 symbols, 400 changes written, 200 duplicates, 6 rescued, 4 resyncs, 1 subscription gaps, 3 losses
EOF
)"
kalshi=$(
cat <<'EOF'
12:00:00.100 kalshi: orchestrator: first scan, listing the exchange's markets
12:00:10.100 kalshi: orchestrator: first scan, 1000 live markets listed so far
12:00:20.200 kalshi: orchestrator: first scan done, 1500 live markets, subscribing
12:00:20.300 kalshi: connection 0: connected, 1500 symbols
12:00:20.310 kalshi: connection 1: connect failed: handshake refused: 429 Too Many Requests
12:00:21.320 kalshi: connection 1: connected, 1500 symbols
12:01:00.001 kalshi: book thread 0: 700 symbols; so far 100 changes written, 90 duplicates, 1 before their connection's book, 0 ahead of the book, 3 fresh books (1 losses), 0 below zero, 2 resyncs, 0 quiet subscriptions resubscribed, 0 markets reported quiet again; this minute 190 changes received, at most 3 events waited
12:01:00.002 kalshi: book thread 1: 800 symbols; so far 200 changes written, 190 duplicates, 0 before their connection's book, 1 ahead of the book, 2 fresh books (0 losses), 1 below zero, 1 resyncs, 0 quiet subscriptions resubscribed, 0 markets reported quiet again; this minute 390 changes received, at most 6 events waited
12:01:00.030 kalshi: Snapshot of KXA at 1794390060030000: loss (its clock 12, ours 11)
12:01:20.400 kalshi: orchestrator: 1500 markets on 2 connections; 1500 added, 0 removed, 0 moved (0 quiet), 0 splits, 0 recycled (0 slow), 0 silent this minute; busiest connection at 9000 bytes/s
12:02:00.001 kalshi: book thread 0: 700 symbols; so far 150 changes written, 140 duplicates, 1 before their connection's book, 0 ahead of the book, 3 fresh books (1 losses), 0 below zero, 2 resyncs, 0 quiet subscriptions resubscribed, 0 markets reported quiet again; this minute 100 changes received, at most 2 events waited
12:02:00.002 kalshi: book thread 1: 800 symbols; so far 260 changes written, 250 duplicates, 0 before their connection's book, 1 ahead of the book, 2 fresh books (0 losses), 1 below zero, 1 resyncs, 0 quiet subscriptions resubscribed, 0 markets reported quiet again; this minute 120 changes received, at most 4 events waited
12:02:20.400 kalshi: orchestrator: 1500 markets on 2 connections; 0 added, 0 removed, 0 moved (0 quiet), 0 splits, 0 recycled (0 slow), 0 silent this minute; busiest connection at 8000 bytes/s
EOF
)
check "Kalshi, live" "$kalshi" "$(
cat <<'EOF'
12:00:00 kalshi: orchestrator: first scan, listing the exchange's markets
12:00:10 kalshi: orchestrator: first scan, 1000 live markets listed so far
12:00:20 kalshi: orchestrator: first scan done, 1500 live markets, subscribing
12:00:20 connections opening (counted in the minute line)
12:01 books (2 threads): 1500 symbols, 300 changes written, 280 duplicates, 3 resyncs, 1 losses
12:01:20 1500 markets on 2 connections; 1500 added, 0 removed, 0 moved (0 quiet), 0 splits, 0 recycled (0 slow), 0 silent this minute; busiest connection at 9000 bytes/s; 2 connected (1 refused, retried), 1 Snapshots with a cause
12:02 books (2 threads): 1500 symbols, 410 changes written, 390 duplicates, 3 resyncs, 1 losses
12:02:20 1500 markets on 2 connections; 0 added, 0 removed, 0 moved (0 quiet), 0 splits, 0 recycled (0 slow), 0 silent this minute; busiest connection at 8000 bytes/s; 0 connected (0 refused, retried), 0 Snapshots with a cause
EOF
)"
check "Kalshi, final" "$kalshi" \
"12:02 books (2 threads): 1500 symbols, 410 changes written, 390 duplicates, 3 resyncs, 1 losses" \
-v final=1
exit "$failed"
+107
View File
@@ -0,0 +1,107 @@
# The view of scripts/watch.sh: reads the collector's log and prints, each
# minute, the orchestrator's line and the book threads' lines summed into one,
# and between them only the notable events. The Snapshot cause lines and the
# connections coming up are counted, not shown.
#
# With -v final=1 it prints nothing live, only the last minute's sums: the
# run's totals, since the book threads count "so far". A minute's sum is
# printed once all its book threads' lines are in (-v book_threads=N, 16 by
# default, src/core/mod.rs's BOOK_THREADS), or with the first other line 10 s
# or more after the minute's first, or when the next minute begins. A book
# thread that received nothing logs no line; the others' come within a few
# seconds of the minute (1.6 s apart at most in a 10-minute Polymarket run,
# 2026-11-12), and other lines often fall between them.
# The book threads' counters shown, in this order; a name missing from an
# exchange's line (Kalshi has no "rescued") is left out.
BEGIN {
if (book_threads == "") book_threads = 16
shown = "symbols|changes written|duplicates|rescued|resyncs|subscription gaps|losses"
count = split(shown, names, "|")
}
function flush( line, i) {
if (threads == 0) return
line = minute " books (" threads " threads):"
for (i = 1; i <= count; i++)
if (names[i] in sum) line = line " " sum[names[i]] " " names[i] ","
sub(/,$/, "", line)
last = line
event(line)
threads = 0
split("", sum)
}
# Printed at once: the view is read while the run goes on, and mawk
# buffers its output otherwise.
function event(text) {
if (final) return
print text
fflush()
}
{
time = substr($1, 1, 8)
text = substr($0, index($0, " ") + 1)
# To the millisecond: $1 is HH:MM:SS.mmm.
seconds = substr($1, 1, 2) * 3600 + substr($1, 4, 2) * 60 + substr($1, 7)
}
/: book thread [0-9]+: / {
this = substr(time, 1, 5)
if (threads > 0 && this != minute) flush()
if (threads == 0) first_at = seconds
minute = this
threads++
body = text
sub(/; this minute .*/, "", body)
split("", seen)
# Each "N name": a count and its name, up to the next comma, semicolon
# or parenthesis. A name is summed once a line (its first occurrence).
while (match(body, /[0-9]+ [a-z][a-z' ]*[a-z]/)) {
pair = substr(body, RSTART, RLENGTH)
body = substr(body, RSTART + RLENGTH)
n = substr(pair, 1, index(pair, " ") - 1)
name = substr(pair, index(pair, " ") + 1)
sub(/^so far /, "", name)
if (name in seen) continue
seen[name] = 1
sum[name] += n
}
if (threads == book_threads) flush()
next
}
# A minute with a book thread's line missing: printed once it is clearly over
# (a day's seconds added back across midnight).
threads > 0 && (seconds - first_at + 86400) % 86400 >= 10 { flush() }
/: orchestrator: [0-9]+ markets on/ {
line = text
sub(/^[a-z-]+: orchestrator: /, "", line)
event(time " " line "; " up + 0 " connected (" refused + 0 " refused, retried), " snapshots + 0 " Snapshots with a cause")
up = 0
refused = 0
snapshots = 0
next
}
/: connection [0-9]+: connected, / {
if (!opened) event(time " connections opening (counted in the minute line)")
opened = 1
up++
next
}
# Kalshi refuses some of ~100 handshakes opened at once (429); each retries.
/: connect failed: handshake refused: 429/ { refused++; next }
/: Snapshot of / { snapshots++; next }
/ changes go unchecked$| tick size changed, resync$| unreadable / { next }
# Everything else is an event worth seeing: a connection lost, recycled,
# split, moved off when quiet, a refused handshake, a failed fetch, the end.
{ event(time " " text) }
END {
flush()
if (final && last != "") print last
}
+101
View File
@@ -0,0 +1,101 @@
#!/bin/sh
# watch.sh [EXCHANGE] [MINUTES]: runs the collector and shows what it does.
# EXCHANGE is polymarket (the default, no key needed), kalshi or kalshi-demo
# (KALSHI_KEY_ID_FILE and KALSHI_PRIVATE_KEY_FILE exported); MINUTES
# defaults to 3. Ctrl-C stops the run early, cleanly.
#
# The output and the full log go to watch/; the terminal shows, each minute,
# the orchestrator's line and the book threads' summed (scripts/watch.awk),
# and the notable events between. Then the output is checked against the
# log (reconcile) and one symbol is shown as text (show). It exits non-zero
# when the collector failed or the output doesn't reconcile.
set -eu
cd "$(dirname "$0")/.."
exchange=${1:-polymarket}
minutes=${2:-3}
case $exchange in
polymarket | kalshi | kalshi-demo) ;;
*)
echo "usage: scripts/watch.sh [polymarket|kalshi|kalshi-demo] [MINUTES]" >&2
exit 2
;;
esac
echo "building (cargo build --release)"
cargo build --release --quiet
mkdir -p watch
out=watch/$exchange.sbe log=watch/$exchange.log memory=watch/$exchange.memory
rm -f "$out" "$log" "$memory"
: >"$log"
echo "load before the run: $(cut -d' ' -f1-3 /proc/loadavg) (1, 5, 15 min)"
echo "collecting $exchange for $minutes minutes into $out (log: $log); Ctrl-C stops it"
echo "the markets are listed first (a minute or two), then subscribed"
timeout -s INT "${minutes}m" ./target/release/engineering-challenge "$exchange" "$out" 2>>"$log" &
run=$!
# The view ends by itself once the run is over and the log read to its end.
# mawk reads a pipe a buffer at a time unless interactive: the view would
# show nothing until the end.
live=
if awk -W version 2>/dev/null | grep -q mawk; then live='-W interactive'; fi
# shellcheck disable=SC2086 # $live is empty or two words, both wanted.
tail -n +1 -F --pid="$run" "$log" 2>/dev/null | awk $live -f scripts/watch.awk &
view=$!
# A job started with & ignores Ctrl-C (sh without job control), so the
# script passes it on: timeout hands SIGINT to the collector, which stops
# cleanly, and the script carries on to the checks. The view, also a job,
# keeps showing the end.
trap 'echo; echo "stopping (Ctrl-C)"; kill -INT "$run" 2>/dev/null || true' INT
# The collector's peak memory (VmHWM, the kernel's own peak) and most
# threads, sampled every 10 s while it runs, from /proc: /usr/bin/time isn't
# installed everywhere. The last sample may fall in the shutdown, when the
# connections' threads are gone: the most seen is kept.
(
sleep 1
pid=$(pgrep -P "$run" || true)
most=0
while [ -n "$pid" ] && kill -0 "$pid" 2>/dev/null; do
proc=$(cat "/proc/$pid/status" 2>/dev/null || true)
peak=$(printf '%s\n' "$proc" | awk '/^VmHWM:/ {print $2}')
threads=$(printf '%s\n' "$proc" | awk '/^Threads:/ {print $2}')
if [ -n "$threads" ] && [ "$threads" -gt "$most" ]; then most=$threads; fi
if [ -n "$peak" ]; then echo "$peak kB, at most $most threads" >"$memory"; fi
sleep 10
done
) &
# wait returns early when Ctrl-C interrupts it: wait again until the
# collector has flushed and exited.
status=0
while kill -0 "$run" 2>/dev/null; do
wait "$run" && status=0 || status=$?
done
trap - INT
wait "$view" || true
# 124: stopped by timeout, at the end or on Ctrl-C (passed on by it).
failed=0
case $status in
0 | 124 | 130) ;;
*)
echo "the collector failed (exit $status); its log: $log"
failed=1
;;
esac
if [ ! -e "$out" ]; then
echo "no output was written"
exit 1
fi
echo
echo "== totals at the last minute"
awk -v final=1 -f scripts/watch.awk "$log"
if [ -s "$memory" ]; then echo "peak memory $(cat "$memory")"; fi
echo
echo "== the output against the log (reconcile)"
./target/release/reconcile "$out" "$log" || failed=1
echo
echo "== one symbol, as text (show)"
# An output stopped before any book is empty: said, not a failure.
./target/release/show "$out" || true
exit "$failed"
+15
View File
@@ -0,0 +1,15 @@
//! `show OUTPUT [SYMBOL]`: one symbol's first Snapshot, the Deltas after it
//! and the book they fold into, as text (`src/show.rs`).
fn main() -> anyhow::Result<()> {
let mut args = std::env::args().skip(1);
let path = args
.next()
.ok_or_else(|| anyhow::anyhow!("usage: show OUTPUT [SYMBOL]"))?;
let symbol = args.next();
print!(
"{}",
engineering_challenge::show::show(path, symbol.as_deref())?
);
Ok(())
}
+24 -9
View File
@@ -78,15 +78,21 @@ impl Book {
} }
/// Moves a level's quantity by `delta`, which can be negative, and returns /// Moves a level's quantity by `delta`, which can be negative, and returns
/// the sum. A level can't hold less than nothing: a negative sum removes /// the sum. A level can't hold less than nothing, nor more than an `i64`:
/// it, and tells the caller the book was wrong. /// either sum removes it and returns `None`, which tells the caller the
pub fn add(&mut self, side: Side, price: i64, delta: i64) -> i64 { /// book was wrong.
pub fn add(&mut self, side: Side, price: i64, delta: i64) -> Option<i64> {
let levels = match side { let levels = match side {
Side::Buy => &self.bids, Side::Buy => &self.bids,
Side::Sell => &self.asks, Side::Sell => &self.asks,
}; };
let quantity = levels.get(&price).copied().unwrap_or(0) + delta; let quantity = levels
self.set(side, price, quantity.max(0)); .get(&price)
.copied()
.unwrap_or(0)
.checked_add(delta)
.filter(|&quantity| quantity >= 0);
self.set(side, price, quantity.unwrap_or(0));
quantity quantity
} }
@@ -173,12 +179,21 @@ mod tests {
#[test] #[test]
fn adding_moves_a_level_and_never_leaves_it_below_zero() { fn adding_moves_a_level_and_never_leaves_it_below_zero() {
let mut book = Book::default(); let mut book = Book::default();
assert_eq!(book.add(Side::Sell, 600_000, 5), 5); assert_eq!(book.add(Side::Sell, 600_000, 5), Some(5));
assert_eq!(book.add(Side::Sell, 600_000, -2), 3); assert_eq!(book.add(Side::Sell, 600_000, -2), Some(3));
assert_eq!(book.asks, BTreeMap::from([(600_000, 3)])); assert_eq!(book.asks, BTreeMap::from([(600_000, 3)]));
assert_eq!(book.add(Side::Sell, 600_000, -3), 0); assert_eq!(book.add(Side::Sell, 600_000, -3), Some(0));
assert!(book.asks.is_empty()); assert!(book.asks.is_empty());
assert_eq!(book.add(Side::Buy, 400_000, -1), -1); assert_eq!(book.add(Side::Buy, 400_000, -1), None);
assert!(book.bids.is_empty());
}
#[test]
fn a_sum_past_the_largest_quantity_removes_the_level() {
let mut book = Book::default();
assert_eq!(book.add(Side::Buy, 400_000, i64::MAX), Some(i64::MAX));
assert_eq!(book.bids, BTreeMap::from([(400_000, i64::MAX)]));
assert_eq!(book.add(Side::Buy, 400_000, 1), None);
assert!(book.bids.is_empty()); assert!(book.bids.is_empty());
} }
+10 -1
View File
@@ -17,16 +17,25 @@ pub enum Cause {
/// The connection followed changed: the new one's book, which may hold /// The connection followed changed: the new one's book, which may hold
/// changes the old one never delivered. /// changes the old one never delivered.
Switch, Switch,
/// Its market was removed and is tracked again: its first book since,
/// against the last one written before.
TrackedAgain,
} }
impl Cause { impl Cause {
const ALL: [Cause; 3] = [Cause::Loss, Cause::SubscriptionGap, Cause::Switch]; const ALL: [Cause; 4] = [
Cause::Loss,
Cause::SubscriptionGap,
Cause::Switch,
Cause::TrackedAgain,
];
fn name(self) -> &'static str { fn name(self) -> &'static str {
match self { match self {
Cause::Loss => "loss", Cause::Loss => "loss",
Cause::SubscriptionGap => "subscription gap", Cause::SubscriptionGap => "subscription gap",
Cause::Switch => "switch", Cause::Switch => "switch",
Cause::TrackedAgain => "tracked again",
} }
} }
} }
+29 -9
View File
@@ -6,13 +6,14 @@
//! subscription gone quiet on one connection (roadmap/05). What an event does //! subscription gone quiet on one connection (roadmap/05). What an event does
//! to a book, how a loss is found and when to resync is the rule's. //! to a book, how a loss is found and when to resync is the rule's.
use std::collections::{BTreeMap, BTreeSet, HashMap}; use std::collections::{BTreeMap, BTreeSet, HashMap, HashSet};
use std::time::Duration; use std::time::Duration;
use chrono::{DateTime, DurationRound, TimeDelta, Utc}; use chrono::{DateTime, DurationRound, TimeDelta, Utc};
use crossbeam::channel::{Receiver, Sender, never, select}; use crossbeam::channel::{Receiver, Sender, never, select};
use engineering_challenge::Side; use engineering_challenge::Side;
use engineering_challenge::book::{Book, Output}; use engineering_challenge::book::{Book, Output};
use engineering_challenge::cause::{self, Cause};
use engineering_challenge::util::dtos::challenge_orderbook::PriceLevelUpdateDomain; use engineering_challenge::util::dtos::challenge_orderbook::PriceLevelUpdateDomain;
use super::socket::Command; use super::socket::Command;
@@ -24,9 +25,9 @@ pub(crate) const MIN_MEASURED: TimeDelta = TimeDelta::seconds(30);
/// How often a book thread whose rule asks for it wakes for the rule's timer. /// How often a book thread whose rule asks for it wakes for the rule's timer.
const WAKE_EVERY: Duration = Duration::from_millis(10); const WAKE_EVERY: Duration = Duration::from_millis(10);
/// A connection that delivered none of a symbol's changes in a minute while /// A connection that delivered none of a symbol's changes in a minute while
/// another delivered at least this many is quiet for it (sprint 05: crossed /// another delivered at least this many is quiet for it (crossed once in 21
/// once in 21 live minutes; at 1, copies landing in the next minute would /// live minutes, docs/notes.md; at 1, copies landing in the next minute
/// count). /// would count).
const QUIET_AGAINST: u32 = 3; const QUIET_AGAINST: u32 = 3;
/// What the book thread does with an event or a command: write output, ask a /// What the book thread does with an event or a command: write output, ask a
@@ -222,6 +223,10 @@ pub struct Symbol<S> {
/// The receive time of its last Delta or Snapshot: a later Delta is /// The receive time of its last Delta or Snapshot: a later Delta is
/// stamped no earlier, so replaying the output by time keeps its order. /// stamped no earlier, so replaying the output by time keeps its order.
pub last_written: DateTime<Utc>, pub last_written: DateTime<Utc>,
/// The exchange's name while its market, removed before, is tracked
/// again and no Snapshot was written since: the first says so (its
/// book can't be told from the one written before it left).
pub again: Option<&'static str>,
pub rule: S, pub rule: S,
} }
@@ -238,6 +243,7 @@ impl<S> Symbol<S> {
.collect(), .collect(),
quiet_on: None, quiet_on: None,
last_written: DateTime::UNIX_EPOCH, last_written: DateTime::UNIX_EPOCH,
again: None,
rule, rule,
} }
} }
@@ -281,6 +287,12 @@ impl<S> Symbol<S> {
/// The Snapshot of its book, received at `received_at`. /// The Snapshot of its book, received at `received_at`.
pub fn snapshot(&mut self, name: &str, received_at: DateTime<Utc>) -> Output { pub fn snapshot(&mut self, name: &str, received_at: DateTime<Utc>) -> Output {
self.last_written = self.last_written.max(received_at); self.last_written = self.last_written.max(received_at);
if let Some(exchange) = self.again.take() {
log!(
"{exchange}: {}",
cause::line(name, received_at, Cause::TrackedAgain, "")
);
}
self.book self.book
.snapshot(name, received_at, self.exchange_timestamp) .snapshot(name, received_at, self.exchange_timestamp)
} }
@@ -378,6 +390,11 @@ pub struct Shell<S> {
pub peak_waiting: usize, pub peak_waiting: usize,
pub quiet: u64, pub quiet: u64,
pub quiet_reported: u64, pub quiet_reported: u64,
/// The symbols of the markets removed: one tracked again says so in its
/// first Snapshot. Ceiling: it grows with every market closed in the
/// run (tens of bytes a symbol, a few thousand a day); forgetting them
/// after a day would bound it.
pub removed: HashSet<String>,
} }
impl<S> Shell<S> { impl<S> Shell<S> {
@@ -429,6 +446,7 @@ impl<R: Rule> Feed<R> {
peak_waiting: 0, peak_waiting: 0,
quiet: 0, quiet: 0,
quiet_reported: 0, quiet_reported: 0,
removed: HashSet::new(),
}, },
rule, rule,
} }
@@ -446,10 +464,11 @@ impl<R: Rule> Feed<R> {
let names = market.symbols().to_vec(); let names = market.symbols().to_vec();
for name in &names { for name in &names {
let rule = self.rule.symbol(market); let rule = self.rule.symbol(market);
self.shell.symbols.insert( let mut symbol = Symbol::new(market.id(), connections, now, rule);
name.clone(), if self.shell.removed.remove(name) {
Symbol::new(market.id(), connections, now, rule), symbol.again = Some(self.shell.exchange);
); }
self.shell.symbols.insert(name.clone(), symbol);
} }
let tracked = Tracked { let tracked = Tracked {
names: names.clone(), names: names.clone(),
@@ -535,6 +554,7 @@ impl<R: Rule> Feed<R> {
for name in market.symbols() { for name in market.symbols() {
if let Some(symbol) = self.shell.symbols.remove(name) { if let Some(symbol) = self.shell.symbols.remove(name) {
connections = symbol.connections; connections = symbol.connections;
self.shell.removed.insert(name.clone());
} }
self.rule.on_removed(name); self.rule.on_removed(name);
} }
@@ -700,7 +720,7 @@ impl<R: Rule> Feed<R> {
/// A book thread: runs until every connection and the orchestrator have /// A book thread: runs until every connection and the orchestrator have
/// dropped its channels, then drops its senders so the writer thread finishes /// dropped its channels, then drops its senders so the writer thread finishes
/// once every book thread has. /// once every book thread has.
// ponytail: unbounded channels never block a socket read nor drop an event; // Ceiling: unbounded channels never block a socket read nor drop an event;
// the ceiling is memory if a consumer stays behind. The minute log's peak of // the ceiling is memory if a consumer stays behind. The minute log's peak of
// events waiting makes that visible; bounded channels are a constructor change // events waiting makes that visible; bounded channels are a constructor change
// if it ever matters. // if it ever matters.
+1 -1
View File
@@ -18,7 +18,7 @@ use crossbeam::channel::{Receiver, Sender};
use feed::Rule; use feed::Rule;
/// How many book threads own the books. One handled ~30,000 frames a second /// How many book threads own the books. One handled ~30,000 frames a second
/// in sprint 02. /// when measured (roadmap/04).
pub const BOOK_THREADS: usize = 16; pub const BOOK_THREADS: usize = 16;
/// The book thread that owns a market's books, from a hash of its id. /// The book thread that owns a market's books, from a hash of its id.
+395 -63
View File
@@ -44,6 +44,8 @@ const MOVED_STAYS: Duration = Duration::from_secs(10 * 60);
const SPLIT_WITHIN: Duration = Duration::from_secs(10 * 60); const SPLIT_WITHIN: Duration = Duration::from_secs(10 * 60);
const RECENT_EVERY: Duration = Duration::from_secs(60); const RECENT_EVERY: Duration = Duration::from_secs(60);
const RESCAN_EVERY: Duration = Duration::from_secs(60 * 60); const RESCAN_EVERY: Duration = Duration::from_secs(60 * 60);
/// How often the first scan says how far it got.
const SCAN_NOTE_EVERY: Duration = Duration::from_secs(10);
/// The listing's indexing delay: each recent check reaches this far before /// The listing's indexing delay: each recent check reaches this far before
/// the last successful one. New markets were listed up to 5 minutes after /// the last successful one. New markets were listed up to 5 minutes after
/// their creation (docs/notes.md); known markets are skipped, so the margin /// their creation (docs/notes.md); known markets are skipped, so the margin
@@ -54,9 +56,10 @@ const RECENT_MARGIN: TimeDelta = TimeDelta::minutes(15);
const RECYCLE_EVERY: Duration = Duration::from_secs(60 * 60); const RECYCLE_EVERY: Duration = Duration::from_secs(60 * 60);
/// A connection slow (among the slowest quarter by mean lateness, and more /// A connection slow (among the slowest quarter by mean lateness, and more
/// than [`SLOW_FACTOR`] times the mean of every copy judged) this many /// than [`SLOW_FACTOR`] times the mean of every copy judged) this many
/// minutes in a row is recycled ahead of its turn: the next one is drawn /// minutes in a row takes the next recycle's turn, in place of the oldest:
/// afresh. Kalshi's demo kept the 2 fastest of 16 connections 28% and 42% /// its replacement's speed is drawn afresh. Kalshi's demo kept the 2 fastest
/// lower at p99 (LATENCY.md); only Kalshi's rule reports lateness. The mean, /// of 16 connections 28% and 42% lower at p99 (LATENCY.md); only Kalshi's
/// rule reports lateness. The mean,
/// not the median: with two copies of each change, a connection first more /// not the median: with two copies of each change, a connection first more
/// than half the time has a median of 0 (roadmap/07). /// than half the time has a median of 0 (roadmap/07).
const SLOW_MINUTES: u32 = 3; const SLOW_MINUTES: u32 = 3;
@@ -123,6 +126,9 @@ pub fn run<E: Exchange>(
let mut fetcher_died = false; let mut fetcher_died = false;
let mut next_rescan = Instant::now(); let mut next_rescan = Instant::now();
let mut next_recent = Instant::now() + RECENT_EVERY; let mut next_recent = Instant::now() + RECENT_EVERY;
// The first scan pages the whole listing (a minute or two): said as it
// goes, so a run that has written nothing yet doesn't look hung.
let mut next_scan_note = Instant::now() + SCAN_NOTE_EVERY;
while !stop.load(Ordering::Relaxed) { while !stop.load(Ordering::Relaxed) {
// Only a panic ends a fetcher while this runs, even one in the middle // Only a panic ends a fetcher while this runs, even one in the middle
@@ -136,6 +142,16 @@ pub fn run<E: Exchange>(
} }
let now = Instant::now(); let now = Instant::now();
let mut commands = Vec::new(); let mut commands = Vec::new();
if let Some(waiting) = &first_scan
&& scanning
&& now >= next_scan_note
{
log!(
"{name}: orchestrator: first scan, {} live markets listed so far",
waiting.len()
);
next_scan_note = now + SCAN_NOTE_EVERY;
}
if now >= next_rescan { if now >= next_rescan {
if !scanning { if !scanning {
_ = rescan.send(Request::Rescan); _ = rescan.send(Request::Rescan);
@@ -155,15 +171,11 @@ pub fn run<E: Exchange>(
let moves = tracker.rebalance(now); let moves = tracker.rebalance(now);
moved += moves.len(); moved += moves.len();
commands.extend(moves); commands.extend(moves);
if let Some((connection, mean)) = tracker.judge_speeds(now) { if let Some(marked) = tracker.judge_speeds(now) {
log!( log!(
"{name}: orchestrator: connection {connection} among the slowest for \ "{name}: orchestrator: {}, next to recycle",
{SLOW_MINUTES} minutes (mean lateness {:.1} ms), recycled", slow_line(marked)
mean as f64 / 1_000.0
); );
threads.recycle(connection);
recycled += 1;
slow_recycled += 1;
} }
let busiest = tracker.connections.iter().map(|one| one.load).max(); let busiest = tracker.connections.iter().map(|one| one.load).max();
log!( log!(
@@ -181,7 +193,14 @@ pub fn run<E: Exchange>(
if first_scan.is_none() && queue.is_empty() { if first_scan.is_none() && queue.is_empty() {
tracker.start_recycling(now); tracker.start_recycling(now);
} }
if let Some(connection) = tracker.recycle(now) { if let Some((connection, slow)) = tracker.recycle(now) {
if let Some(mean) = slow {
log!(
"{name}: orchestrator: {}, recycled",
slow_line((connection, mean))
);
slow_recycled += 1;
}
threads.recycle(connection); threads.recycle(connection);
recycled += 1; recycled += 1;
} }
@@ -203,6 +222,10 @@ pub fn run<E: Exchange>(
recv(findings) -> finding => match finding { recv(findings) -> finding => match finding {
Ok(Finding::ScanStarted) => { Ok(Finding::ScanStarted) => {
tracker.scan_started(); tracker.scan_started();
if first_scan.is_some() {
log!("{name}: orchestrator: first scan, listing the exchange's markets");
next_scan_note = Instant::now() + SCAN_NOTE_EVERY;
}
// A first scan that failed starts over. // A first scan that failed starts over.
if let Some(waiting) = &mut first_scan { if let Some(waiting) = &mut first_scan {
waiting.clear(); waiting.clear();
@@ -224,6 +247,10 @@ pub fn run<E: Exchange>(
scanned = true; scanned = true;
match first_scan.take() { match first_scan.take() {
Some(markets) => { Some(markets) => {
log!(
"{name}: orchestrator: first scan done, {} live markets, subscribing",
markets.len()
);
let adds = tracker.deal(markets); let adds = tracker.deal(markets);
added += adds.len(); added += adds.len();
commands.extend(adds); commands.extend(adds);
@@ -466,6 +493,21 @@ struct Tracker<M> {
recycling: Option<usize>, recycling: Option<usize>,
/// The book threads' lateness samples since the last judgment. /// The book threads' lateness samples since the last judgment.
lateness: Lateness, lateness: Lateness,
/// The connection judged slow, with its mean lateness (µs): it takes the
/// next recycle's turn in place of the oldest.
slow: Option<(usize, i64)>,
/// The connections whose mark was logged, while they stay slow: two that
/// swap ranks are each said once.
said: HashSet<usize>,
}
/// A slow connection, as the log says it: its number and mean lateness (µs).
fn slow_line((connection, mean): (usize, i64)) -> String {
format!(
"connection {connection} among the slowest for {SLOW_MINUTES} minutes (mean lateness \
{:.1} ms)",
mean as f64 / 1_000.0
)
} }
impl<M: Market> Tracker<M> { impl<M: Market> Tracker<M> {
@@ -479,6 +521,8 @@ impl<M: Market> Tracker<M> {
next_recycle: None, next_recycle: None,
recycling: None, recycling: None,
lateness: Lateness::new(), lateness: Lateness::new(),
slow: None,
said: HashSet::new(),
} }
} }
@@ -602,7 +646,7 @@ impl<M: Market> Tracker<M> {
.collect(); .collect();
over.sort_by_key(|&number| Reverse(self.connections[number].load)); over.sort_by_key(|&number| Reverse(self.connections[number].load));
for from in over { for from in over {
// ponytail: a scan of every market per connection over budget, // Ceiling: a scan of every market per connection over budget,
// fine at a few hundred thousand once a minute; per-connection // fine at a few hundred thousand once a minute; per-connection
// market sets if not. // market sets if not.
let mut busiest: Vec<(String, u64)> = self let mut busiest: Vec<(String, u64)> = self
@@ -715,6 +759,11 @@ impl<M: Market> Tracker<M> {
if self.recycling == Some(connection) { if self.recycling == Some(connection) {
self.recycling = None; self.recycling = None;
} }
// A fresh connection: what was judged slow is gone with the old one.
if self.slow.is_some_and(|(slow, _)| slow == connection) {
self.slow = None;
}
self.said.remove(&connection);
let settled = now + self.recycle_interval(); let settled = now + self.recycle_interval();
if let Some(next) = &mut self.next_recycle { if let Some(next) = &mut self.next_recycle {
*next = (*next).max(settled); *next = (*next).max(settled);
@@ -738,11 +787,12 @@ impl<M: Market> Tracker<M> {
RECYCLE_EVERY / self.connections.len().max(1) as u32 RECYCLE_EVERY / self.connections.len().max(1) as u32
} }
/// The connection to recycle now, if one is due: the one up the longest, /// The connection to recycle now, if one is due: the one judged slow if
/// once the last recycled is up again, and only while every connection is /// there is one (with its mean lateness), else the one up the longest;
/// up and has been for an interval, so no market loses both of its /// once the last recycled is up again, and only while every connection
/// is up and has been for an interval, so no market loses both of its
/// connections at once. /// connections at once.
fn recycle(&mut self, now: Instant) -> Option<usize> { fn recycle(&mut self, now: Instant) -> Option<(usize, Option<i64>)> {
let due = self.next_recycle.is_some_and(|next| now >= next); let due = self.next_recycle.is_some_and(|next| now >= next);
if !due if !due
|| self.recycling.is_some() || self.recycling.is_some()
@@ -750,11 +800,17 @@ impl<M: Market> Tracker<M> {
{ {
return None; return None;
} }
let oldest = (0..self.connections.len()) let (connection, slow) = match self.slow.take() {
.min_by_key(|&number| self.connections[number].connected_at)?; Some((slow, mean)) => (slow, Some(mean)),
self.recycling = Some(oldest); None => (
(0..self.connections.len())
.min_by_key(|&number| self.connections[number].connected_at)?,
None,
),
};
self.recycling = Some(connection);
self.next_recycle = Some(now + self.recycle_interval()); self.next_recycle = Some(now + self.recycle_interval());
Some(oldest) Some((connection, slow))
} }
/// A book thread's lateness samples, by connection. /// A book thread's lateness samples, by connection.
@@ -768,10 +824,10 @@ impl<M: Market> Tracker<M> {
/// since the last time; those of the slowest 1 in [`SLOW_SHARE`] more /// since the last time; those of the slowest 1 in [`SLOW_SHARE`] more
/// than [`SLOW_FACTOR`] times the mean of every judged copy count a slow /// than [`SLOW_FACTOR`] times the mean of every judged copy count a slow
/// minute, the others start over. The slowest of those slow for /// minute, the others start over. The slowest of those slow for
/// [`SLOW_MINUTES`] is recycled now, with its mean (µs), under /// [`SLOW_MINUTES`] is marked, with its mean (µs), to take the next
/// [`Tracker::recycle`]'s /// recycle's turn ([`Tracker::recycle`]): one path, one at a time.
/// rule: one at a time, every connection up, none up for less than an /// Returns the mark when it is new: a connection marked again while it
/// interval. /// stays slow, waiting for its turn, is said once.
fn judge_speeds(&mut self, now: Instant) -> Option<(usize, i64)> { fn judge_speeds(&mut self, now: Instant) -> Option<(usize, i64)> {
let samples = std::mem::take(&mut self.lateness); let samples = std::mem::take(&mut self.lateness);
let speeds: Vec<(usize, i64, i64)> = samples let speeds: Vec<(usize, i64, i64)> = samples
@@ -814,22 +870,15 @@ impl<M: Market> Tracker<M> {
0 0
}; };
} }
// As for the hourly recycle: one at a time, every connection up, and let connections = &self.connections;
// the last to come up past its opening books. self.said
let settled = self .retain(|&connection| connections[connection].slow_minutes > 0);
.connections self.slow = slow
.iter() .iter()
.filter_map(|state| state.connected_at) .copied()
.max() .find(|&(connection, _)| self.connections[connection].slow_minutes >= SLOW_MINUTES);
.is_none_or(|last| now.duration_since(last) >= self.recycle_interval()); self.slow
if self.recycling.is_some() || self.connections.iter().any(|state| !state.up) || !settled { .filter(|&(connection, _)| self.said.insert(connection))
return None;
}
let &(connection, speed) = slow
.iter()
.find(|&&(connection, _)| self.connections[connection].slow_minutes >= SLOW_MINUTES)?;
self.recycling = Some(connection);
Some((connection, speed))
} }
fn scan_started(&mut self) { fn scan_started(&mut self) {
@@ -1189,7 +1238,7 @@ mod tests {
tracker.start_recycling(at(10)); tracker.start_recycling(at(10));
// One every hour / 3 connections. // One every hour / 3 connections.
assert_eq!(tracker.recycle(at(1_209)), None); assert_eq!(tracker.recycle(at(1_209)), None);
assert_eq!(tracker.recycle(at(1_210)), Some(2)); assert_eq!(tracker.recycle(at(1_210)), Some((2, None)));
// The next waits for the recycled one, whose Down may not even have // The next waits for the recycled one, whose Down may not even have
// come yet, to be up again... // come yet, to be up again...
assert_eq!(tracker.recycle(at(2_410)), None); assert_eq!(tracker.recycle(at(2_410)), None);
@@ -1198,7 +1247,7 @@ mod tests {
tracker.up(2, at(2_411)); tracker.up(2, at(2_411));
// ...and for its opening books: an interval more. // ...and for its opening books: an interval more.
assert_eq!(tracker.recycle(at(3_610)), None); assert_eq!(tracker.recycle(at(3_610)), None);
assert_eq!(tracker.recycle(at(3_611)), Some(0)); assert_eq!(tracker.recycle(at(3_611)), Some((0, None)));
tracker.down(0); tracker.down(0);
tracker.up(0, at(3_612)); tracker.up(0, at(3_612));
// A connection down for its own reasons holds every recycle, and its // A connection down for its own reasons holds every recycle, and its
@@ -1207,7 +1256,7 @@ mod tests {
assert_eq!(tracker.recycle(at(6_000)), None); assert_eq!(tracker.recycle(at(6_000)), None);
tracker.up(2, at(6_001)); tracker.up(2, at(6_001));
assert_eq!(tracker.recycle(at(7_200)), None); assert_eq!(tracker.recycle(at(7_200)), None);
assert_eq!(tracker.recycle(at(7_201)), Some(1)); assert_eq!(tracker.recycle(at(7_201)), Some((1, None)));
} }
/// Twelve connections, up from `start`. /// Twelve connections, up from `start`.
@@ -1236,36 +1285,119 @@ mod tests {
} }
#[test] #[test]
fn a_connection_among_the_slowest_three_minutes_running_is_recycled() { fn a_connection_among_the_slowest_three_minutes_running_takes_the_next_recycle() {
let start = Instant::now(); let start = Instant::now();
let at = |seconds: u64| start + Duration::from_secs(seconds); let at = |seconds: u64| start + Duration::from_secs(seconds);
let mut tracker = twelve_up(start); let mut tracker = twelve_up(start);
// The slowest quarter is three of twelve. 3 and 7 are slow every // The hourly recycle runs: one every 300 s (an hour / 12), the
// minute, 5 for two minutes and then not. // first due at 600.
tracker.start_recycling(at(300));
// The slowest quarter is three of twelve, then two of the eleven
// judged once 0 is fresh. 3 and 7 are slow every minute, 5 only in
// the first.
minute(&mut tracker, &[(3, 90), (5, 50), (7, 40)], 100); minute(&mut tracker, &[(3, 90), (5, 50), (7, 40)], 100);
assert_eq!(tracker.judge_speeds(at(601)), None); assert_eq!(tracker.judge_speeds(at(601)), None);
minute(&mut tracker, &[(3, 90), (5, 50), (7, 40)], 100); // The hourly turn takes the oldest (all equal: the first).
assert_eq!(tracker.recycle(at(601)), Some((0, None)));
tracker.down(0);
tracker.up(0, at(603));
minute(&mut tracker, &[(3, 90), (7, 40), (5, 35)], 100);
assert_eq!(tracker.judge_speeds(at(661)), None); assert_eq!(tracker.judge_speeds(at(661)), None);
// 3 and 7 both reach three minutes: one recycle a minute, the slowest. // 3 and 7 both reach three minutes: the slowest is marked.
minute(&mut tracker, &[(3, 90), (7, 40), (9, 30)], 100); minute(&mut tracker, &[(3, 90), (7, 40), (9, 30)], 100);
assert_eq!(tracker.judge_speeds(at(721)), Some((3, 90_000))); assert_eq!(tracker.judge_speeds(at(721)), Some((3, 90_000)));
assert_eq!(tracker.connections[5].slow_minutes, 0); assert_eq!(tracker.connections[5].slow_minutes, 0);
// While 3 recycles, 7 stays slow but isn't recycled. // Not before the turn (0 came up at 603: due at 903)...
minute(&mut tracker, &[(3, 90), (7, 40), (9, 30)], 100); assert_eq!(tracker.recycle(at(902)), None);
assert_eq!(tracker.judge_speeds(at(781)), None); // ...then the turn is 3's, not the oldest's.
assert_eq!(tracker.connections[7].slow_minutes, 4); assert_eq!(tracker.recycle(at(903)), Some((3, Some(90_000))));
// While 3 recycles, nothing else is.
assert_eq!(tracker.recycle(at(1_300)), None);
tracker.down(3); tracker.down(3);
tracker.up(3, at(790)); tracker.up(3, at(1_301));
// 3 is fresh: slow, but not judged. 7 waits, as the hourly recycle // 3 is fresh: not judged, its slow minutes start over; 7, still
// would, for 3's opening books: an interval (an hour / 12) after it // slow, takes the next turn, 300 s after 3 came up.
// came up.
minute(&mut tracker, &[(3, 90), (7, 40), (9, 30)], 100); minute(&mut tracker, &[(3, 90), (7, 40), (9, 30)], 100);
assert_eq!(tracker.judge_speeds(at(841)), None); assert_eq!(tracker.judge_speeds(at(1_302)), Some((7, 40_000)));
assert_eq!(tracker.connections[3].slow_minutes, 0); assert_eq!(tracker.connections[3].slow_minutes, 0);
minute(&mut tracker, &[(3, 90), (7, 40), (9, 30)], 100); assert_eq!(tracker.recycle(at(1_600)), None);
assert_eq!(tracker.judge_speeds(at(1_089)), None); assert_eq!(tracker.recycle(at(1_601)), Some((7, Some(40_000))));
minute(&mut tracker, &[(3, 90), (7, 40), (9, 30)], 100); // Then the hourly order again.
assert_eq!(tracker.judge_speeds(at(1_090)), Some((7, 40_000))); tracker.down(7);
tracker.up(7, at(1_602));
assert_eq!(tracker.recycle(at(1_902)), Some((1, None)));
}
#[test]
fn a_mark_is_said_once_and_cleared_once_the_connection_is_no_longer_slow() {
let start = Instant::now();
let at = |seconds: u64| start + Duration::from_secs(seconds);
let mut tracker = twelve_up(start);
for seconds in [601, 661] {
minute(&mut tracker, &[(3, 90)], 100);
assert_eq!(tracker.judge_speeds(at(seconds)), None);
}
minute(&mut tracker, &[(3, 90)], 100);
assert_eq!(tracker.judge_speeds(at(721)), Some((3, 90_000)));
// Still slow, its turn not come: marked still, but not said again.
minute(&mut tracker, &[(3, 90)], 100);
assert_eq!(tracker.judge_speeds(at(781)), None);
assert_eq!(tracker.slow, Some((3, 90_000)));
// As fast as the others before its turn: the mark goes, and the
// turn is the oldest's.
minute(&mut tracker, &[], 100);
assert_eq!(tracker.judge_speeds(at(841)), None);
assert_eq!(tracker.slow, None);
tracker.start_recycling(at(841));
assert_eq!(tracker.recycle(at(1_141)), Some((0, None)));
}
#[test]
fn two_marked_connections_swapping_ranks_are_each_said_once() {
let start = Instant::now();
let at = |seconds: u64| start + Duration::from_secs(seconds);
let mut tracker = twelve_up(start);
for seconds in [601, 661] {
minute(&mut tracker, &[(3, 90), (7, 40)], 100);
assert_eq!(tracker.judge_speeds(at(seconds)), None);
}
minute(&mut tracker, &[(3, 90), (7, 40)], 100);
assert_eq!(tracker.judge_speeds(at(721)), Some((3, 90_000)));
// 7 overtakes 3: the mark moves to 7, said once.
minute(&mut tracker, &[(3, 40), (7, 90)], 100);
assert_eq!(tracker.judge_speeds(at(781)), Some((7, 90_000)));
// Back to 3, still slow since it was said: not again.
minute(&mut tracker, &[(3, 90), (7, 40)], 100);
assert_eq!(tracker.judge_speeds(at(841)), None);
assert_eq!(tracker.slow, Some((3, 90_000)));
// 3 fast for a minute, then slow 3 minutes again: said anew.
minute(&mut tracker, &[(7, 40)], 100);
tracker.judge_speeds(at(901));
for seconds in [961, 1_021] {
minute(&mut tracker, &[(3, 90), (7, 40)], 100);
tracker.judge_speeds(at(seconds));
}
minute(&mut tracker, &[(3, 90), (7, 40)], 100);
assert_eq!(tracker.judge_speeds(at(1_081)), Some((3, 90_000)));
}
#[test]
fn a_slow_mark_is_dropped_when_its_connection_comes_up_fresh() {
let start = Instant::now();
let at = |seconds: u64| start + Duration::from_secs(seconds);
let mut tracker = twelve_up(start);
for seconds in [601, 661, 721] {
minute(&mut tracker, &[(3, 90), (7, 40)], 100);
tracker.judge_speeds(at(seconds));
}
assert_eq!(tracker.slow, Some((3, 90_000)));
// 3 reconnects for its own reasons before its turn: renewed, so
// the turn goes back to the oldest.
tracker.down(3);
tracker.up(3, at(730));
assert_eq!(tracker.slow, None);
tracker.start_recycling(at(730));
assert_eq!(tracker.recycle(at(1_030)), Some((0, None)));
} }
#[test] #[test]
@@ -1326,13 +1458,16 @@ mod tests {
minute(&mut tracker, &[(3, 90), (5, 50)], 99); minute(&mut tracker, &[(3, 90), (5, 50)], 99);
assert_eq!(tracker.judge_speeds(at(seconds)), None); assert_eq!(tracker.judge_speeds(at(seconds)), None);
} }
// A connection down holds every recycle, as the hourly one's rule. // A connection down holds the slow one's turn, as every recycle.
let mut tracker = twelve_up(start); let mut tracker = twelve_up(start);
tracker.start_recycling(start);
tracker.down(0); tracker.down(0);
for seconds in [601, 661, 721, 781] { for seconds in [601, 661, 721] {
minute(&mut tracker, &[(3, 90), (5, 50)], 100); minute(&mut tracker, &[(3, 90), (5, 50)], 100);
assert_eq!(tracker.judge_speeds(at(seconds)), None); tracker.judge_speeds(at(seconds));
} }
assert_eq!(tracker.slow, Some((3, 90_000)));
assert_eq!(tracker.recycle(at(1_000)), None);
} }
#[test] #[test]
@@ -1447,3 +1582,200 @@ mod tests {
assert_eq!(tracker.markets.len(), 1); assert_eq!(tracker.markets.len(), 1);
} }
} }
/// [`run`] end to end, offline: Kalshi's listing and websocket served
/// locally, the writer as `main` runs it.
#[cfg(test)]
mod run_tests {
use std::io::{BufRead, BufReader, Write};
use std::net::{TcpListener, TcpStream};
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
use std::thread;
use std::time::Duration;
use crossbeam::channel::{Sender, unbounded};
use tungstenite::Message;
use super::SUBSCRIPTIONS;
use crate::kalshi::{Kalshi, sign};
const TICKERS: [&str; 2] = ["KXA", "KXB"];
/// A local listing: the tickers open, nothing unopened.
fn listing() -> String {
let listener = TcpListener::bind("127.0.0.1:0").unwrap();
let url = format!("http://{}", listener.local_addr().unwrap());
thread::spawn(move || {
for stream in listener.incoming() {
let stream = stream.unwrap();
let mut reader = BufReader::new(&stream);
let mut request = String::new();
reader.read_line(&mut request).unwrap();
// The headers, read whole so the close doesn't reset them.
let mut line = String::new();
while reader.read_line(&mut line).unwrap() > 2 {
line.clear();
}
let markets: Vec<String> = TICKERS
.iter()
.map(|ticker| format!(r#"{{"ticker":"{ticker}","status":"active"}}"#))
.collect();
let open = request.contains("status=open") && !request.contains("cursor=");
let markets = if open {
markets.join(",")
} else {
String::new()
};
let body = format!(r#"{{"markets":[{markets}],"cursor":""}}"#);
write!(
&stream,
"HTTP/1.1 200 OK\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}",
body.len()
)
.unwrap();
}
});
url
}
/// A local websocket: each connection answers every ticker it subscribes
/// with a book and a change, the same on every connection. `books` hears
/// of each ticker served, `closed` whether each connection was closed
/// with a Close frame.
fn websocket(books: Sender<()>, closed: Sender<bool>) -> String {
let listener = TcpListener::bind("127.0.0.1:0").unwrap();
let url = format!("ws://{}/trade-api/ws/v2", listener.local_addr().unwrap());
thread::spawn(move || {
for stream in listener.incoming() {
let (books, closed) = (books.clone(), closed.clone());
thread::spawn(move || serve(stream.unwrap(), &books, &closed));
}
});
url
}
fn serve(stream: TcpStream, books: &Sender<()>, closed: &Sender<bool>) {
let mut socket = tungstenite::accept(stream).unwrap();
let mut seq = 0;
loop {
match socket.read() {
Ok(Message::Text(text)) => {
let command: serde_json::Value = serde_json::from_str(&text).unwrap();
if command["cmd"] != "subscribe" {
continue;
}
if seq == 0 {
let subscribed = r#"{"type":"subscribed","id":1,"msg":{"channel":"orderbook_delta","sid":1}}"#;
socket.send(Message::text(subscribed)).unwrap();
}
for ticker in command["params"]["market_tickers"].as_array().unwrap() {
let ticker = ticker.as_str().unwrap();
let book = format!(
r#"{{"type":"orderbook_snapshot","sid":1,"seq":{},"msg":{{"market_ticker":"{ticker}","market_id":"x","clock":10,"yes_dollars_fp":[["0.4000","5.00"]]}},"sending_ts_ms":1794422918369}}"#,
seq + 1
);
let change = format!(
r#"{{"type":"orderbook_delta","sid":1,"seq":{},"msg":{{"market_ticker":"{ticker}","market_id":"x","clock":11,"price_dollars":"0.4000","delta_fp":"1.00","side":"yes","ts_ms":1794422918370}}}}"#,
seq + 2
);
seq += 2;
socket.send(Message::text(book)).unwrap();
socket.send(Message::text(change)).unwrap();
_ = books.send(());
}
}
Ok(Message::Close(_)) => {
// The reply goes out with the next read.
while socket.read().is_ok() {}
_ = closed.send(true);
return;
}
Ok(_) => {}
Err(_) => {
_ = closed.send(false);
return;
}
}
}
}
#[test]
fn a_run_subscribes_writes_and_stops_on_the_flag_closing_every_socket() {
let (books_tx, books) = unbounded();
let (closed_tx, closed) = unbounded();
let kalshi = Kalshi {
websocket: websocket(books_tx, closed_tx),
listing: listing(),
key: sign::test_key(),
};
let path =
std::env::temp_dir().join(format!("orchestrator-run-{}.sbe", std::process::id()));
let (outputs, outputs_rx) = unbounded();
// Between the run and the writer: says when every change has gone
// through, so the test stops then, not after a guessed sleep.
let (relayed, writer_rx) = unbounded();
let (all_deltas_tx, all_deltas) = unbounded();
thread::spawn(move || {
let mut deltas = 0;
for output in outputs_rx {
if matches!(output, engineering_challenge::book::Output::Delta(_)) {
deltas += 1;
if deltas == TICKERS.len() {
_ = all_deltas_tx.send(());
}
}
_ = relayed.send(output);
}
});
let (written_tx, written) = unbounded();
let writer_path = path.to_str().unwrap().to_string();
thread::spawn(move || {
let result = crate::write(&writer_path, writer_rx).map_err(|error| error.to_string());
written_tx.send(result).unwrap();
});
let stop = Arc::new(AtomicBool::new(false));
let run_stop = Arc::clone(&stop);
let (failed_tx, failed) = unbounded();
thread::spawn(move || failed_tx.send(super::run(Arc::new(kalshi), outputs, &run_stop)));
// Each ticker served on each of its connections.
for _ in 0..TICKERS.len() * SUBSCRIPTIONS {
books
.recv_timeout(Duration::from_secs(10))
.expect("every subscription served");
}
// Their events reach the book threads, and the writer.
all_deltas
.recv_timeout(Duration::from_secs(10))
.expect("every change written");
stop.store(true, Ordering::Relaxed);
// Timed, so a run that never ends fails rather than hangs.
let failed = failed
.recv_timeout(Duration::from_secs(10))
.expect("the run returns once stopped");
assert_eq!(failed, 0);
let written = written
.recv_timeout(Duration::from_secs(10))
.expect("the writer ends once the run has");
let closes: Vec<bool> = (0..SUBSCRIPTIONS)
.map(|_| closed.recv_timeout(Duration::from_secs(5)).unwrap())
.collect();
assert_eq!(
closes, [true; SUBSCRIPTIONS],
"a Close frame on every socket"
);
// No other connection.
assert!(closed.recv_timeout(Duration::from_millis(200)).is_err());
written.unwrap();
let report = engineering_challenge::reconcile::reconcile(&path, None);
std::fs::remove_file(&path).unwrap();
let report = report.unwrap();
// A first book per ticker, and one per minute crossed; each change
// once, the other connection's copy a duplicate.
assert!(report.snapshots >= TICKERS.len(), "{report:?}");
assert_eq!(report.deltas, TICKERS.len(), "{report:?}");
assert_eq!(report.orphan_deltas, 0);
assert_eq!(report.mismatches, []);
}
}
+1
View File
@@ -31,6 +31,7 @@ const CLOSE_TIMEOUT: Duration = Duration::from_secs(2);
/// Nothing received this long, not even a keepalive's answer or a ping: the /// Nothing received this long, not even a keepalive's answer or a ping: the
/// connection is silent, and reconnects. /// connection is silent, and reconnects.
pub const SILENT_AFTER: Duration = Duration::from_secs(30); pub const SILENT_AFTER: Duration = Duration::from_secs(30);
/// What a book thread asks a connection for. /// What a book thread asks a connection for.
#[derive(Debug, PartialEq, Eq)] #[derive(Debug, PartialEq, Eq)]
pub enum Command { pub enum Command {
+174 -5
View File
@@ -29,7 +29,7 @@ const BOOK: u32 = u32::MAX;
/// Changes remembered per symbol for their second copy's lateness: two /// Changes remembered per symbol for their second copy's lateness: two
/// connections deliver a market's changes 2 ms apart at the median, 344 ms /// connections deliver a market's changes 2 ms apart at the median, 344 ms
/// at worst (docs/notes.md), far fewer changes than this. /// at worst (docs/notes.md), far fewer changes than this.
// ponytail: a second copy more than 128 changes behind goes unmeasured, and // Ceiling: a second copy more than 128 changes behind goes unmeasured, and
// the lookup is linear; a map by position, emptied as the book passes, // the lookup is linear; a map by position, emptied as the book passes,
// would lift both. // would lift both.
const RECENT: usize = 128; const RECENT: usize = 128;
@@ -205,8 +205,11 @@ impl Clock {
.recent .recent
.push_back((position, connection, received_at)); .push_back((position, connection, received_at));
let quantity = symbol.book.add(change.side, change.price, change.delta); let quantity = symbol.book.add(change.side, change.price, change.delta);
let quantity = if quantity < 0 { let quantity = if let Some(quantity) = quantity {
// A level can't hold less than nothing: ours was wrong. quantity
} else {
// A level can't hold less than nothing (nor more than an i64):
// ours was wrong.
self.counts.below_zero += 1; self.counts.below_zero += 1;
log!( log!(
"kalshi: {ticker}: a change takes a level below zero, resync on connection \ "kalshi: {ticker}: a change takes a level below zero, resync on connection \
@@ -217,8 +220,6 @@ impl Clock {
self.counts.resyncs += 1; self.counts.resyncs += 1;
} }
0 0
} else {
quantity
}; };
symbol.exchange_timestamp = change.at; symbol.exchange_timestamp = change.at;
self.counts.written += 1; self.counts.written += 1;
@@ -677,6 +678,20 @@ mod tests {
assert_eq!(deltas(&after)[0].quantity, 10_000_000); assert_eq!(deltas(&after)[0].quantity, 10_000_000);
} }
/// A change that would take a level past an i64: the book is wrong, as
/// below zero.
#[test]
fn a_level_past_the_largest_quantity_resyncs() {
let mut feed = both_on("M");
let level = feed.shell.symbols["M"].book.bids[&400_000];
assert!(level > 0);
let effects = feed.from(0, 3, change("M", 11, Side::Buy, 400_000, i64::MAX));
assert_eq!(deltas(&effects)[0].quantity, 0);
assert_eq!(resyncs(&effects), [(0, "M")]);
assert_eq!(feed.shell.symbols["M"].book.bids.get(&400_000), None);
assert_eq!(feed.rule.counts.below_zero, 1);
}
/// The answer to a resync can be behind what the other connection has /// The answer to a resync can be behind what the other connection has
/// delivered: the book goes back to it, and only a stream that continues /// delivered: the book goes back to it, and only a stream that continues
/// from there is taken until the other fits again. /// from there is taken until the other fits again.
@@ -778,6 +793,140 @@ mod tests {
assert_eq!(snapshots(&feed.tick(minute + TimeDelta::minutes(2))), 1); assert_eq!(snapshots(&feed.tick(minute + TimeDelta::minutes(2))), 1);
} }
fn unsubscribes(effects: &[Effect]) -> Vec<usize> {
effects
.iter()
.filter_map(|effect| match effect {
Effect::Send(connection, Command::Unsubscribe(_)) => Some(*connection),
_ => None,
})
.collect()
}
fn market(ticker: &str) -> Market {
Market {
ticker: ticker.into(),
}
}
/// `both_on`'s market moved from connection 0 to 2, the move ended by
/// connection 2's book: connection 0 left it.
fn moved_off_zero() -> Feed {
let mut feed = both_on("M");
feed.move_market(&market("M"), 0, 2, at(3));
let ended = feed.from(2, 4, book("M", 10, &[(400_000, 5_000_000)], &[]));
assert_eq!(unsubscribes(&ended), [0]);
feed
}
#[test]
fn a_book_from_a_connection_the_symbol_left_counts_for_nothing() {
let mut feed = moved_off_zero();
// Still on its way when it left, past ours and different.
let late = feed.from(0, 5, book("M", 20, &[(400_000, 1_000_000)], &[]));
assert!(late.is_empty(), "{late:?}");
assert_eq!(feed.rule.counts.losses, 0);
assert_eq!(feed.shell.symbols["M"].book.bids[&400_000], 5_000_000);
}
#[test]
fn a_change_from_a_connection_the_symbol_left_has_no_place() {
let mut feed = moved_off_zero();
let late = feed.from(0, 5, change("M", 11, Side::Buy, 400_000, 1_000_000));
assert!(deltas(&late).is_empty(), "{late:?}");
assert_eq!(feed.rule.counts.unplaced, 1);
}
/// The answer to a resync can be our very book: the resync is over, not
/// asked again every minute.
#[test]
fn an_answer_equal_to_ours_ends_the_resync() {
let mut feed = both_on("M");
feed.on_received(Received::Down(0));
feed.on_received(Received::Down(1));
feed.on_received(Received::Up(0));
feed.on_received(Received::Up(1));
let minute = next_minute(at(10));
assert_eq!(resyncs(&feed.tick(minute)), [(0, "M")]);
let answer = feed.from(0, 70_000, book("M", 10, &[(400_000, 5_000_000)], &[]));
assert!(answer.is_empty(), "{answer:?}");
assert_eq!(feed.shell.symbols["M"].rule.asked, None);
assert!(resyncs(&feed.tick(minute + TimeDelta::minutes(1))).is_empty());
}
#[test]
fn a_second_level_below_zero_while_a_resync_is_asked_asks_no_other() {
let mut feed = both_on("M");
let first = feed.from(0, 3, change("M", 11, Side::Buy, 400_000, -6_000_000));
assert_eq!(resyncs(&first), [(0, "M")]);
let second = feed.from(0, 4, change("M", 12, Side::Sell, 600_000, -1_000_000));
assert_eq!(deltas(&second)[0].quantity, 0);
assert!(resyncs(&second).is_empty(), "{second:?}");
assert_eq!(feed.rule.counts.below_zero, 2);
assert_eq!(feed.rule.counts.resyncs, 1);
}
// The shell's moves and quiet watch (src/core/feed.rs), through Kalshi's
// rule, the simpler one.
#[test]
fn a_move_ends_on_the_new_connections_book_not_on_its_change() {
let mut feed = both_on("M");
feed.move_market(&market("M"), 0, 2, at(3));
let change = feed.from(2, 4, change("M", 11, Side::Buy, 400_000, 1_000_000));
assert!(unsubscribes(&change).is_empty(), "{change:?}");
let ended = feed.from(2, 5, book("M", 10, &[(400_000, 5_000_000)], &[]));
assert_eq!(unsubscribes(&ended), [0]);
}
#[test]
fn a_second_move_while_one_is_under_way_ends_the_first() {
let mut feed = both_on("M");
feed.move_market(&market("M"), 0, 2, at(3));
let second = feed.move_market(&market("M"), 1, 3, at(4));
assert_eq!(unsubscribes(&second), [0]);
assert!(second.iter().any(|effect| matches!(
effect,
Effect::Send(3, Command::Subscribe(tickers)) if tickers == &["M".to_string()]
)));
let mut on = feed.shell.symbols["M"].connections.clone();
on.sort();
assert_eq!(on, [1, 2, 3]);
}
/// Quiet on connection 1, resubscribed; a minute delivering on both;
/// quiet again: resubscribed again, not reported for a move.
#[test]
fn a_good_minute_between_two_quiet_ones_forgets_the_first() {
let mut feed = both_on("M");
let minute = |n: i64| next_minute(at(10)) + TimeDelta::minutes(n);
feed.tick(minute(0));
let mut clock = 10;
let mut changes = |feed: &mut Feed, n: i64, on: &[usize]| {
for step in 0..3 {
clock += 1;
for &connection in on {
let millis = (minute(n) - at(0)).num_milliseconds() + 10_000 + step;
feed.from(connection, millis, change("M", clock, Side::Buy, 1, 1));
}
}
};
changes(&mut feed, 0, &[0]);
assert_eq!(resyncs(&feed.tick(minute(1))), [(1, "M")]);
changes(&mut feed, 1, &[0, 1]);
assert!(resyncs(&feed.tick(minute(2))).is_empty());
changes(&mut feed, 2, &[0]);
let again = feed.tick(minute(3));
assert_eq!(resyncs(&again), [(1, "M")]);
assert!(
!again
.iter()
.any(|effect| matches!(effect, Effect::Report(feed::Report::Quiet { .. }))),
"{again:?}"
);
assert_eq!(feed.shell.quiet, 2);
}
/// The demo's recording on two connections (tests/fixtures/README.md). /// The demo's recording on two connections (tests/fixtures/README.md).
fn recording() -> Vec<(DateTime<Utc>, usize, Event)> { fn recording() -> Vec<(DateTime<Utc>, usize, Event)> {
#[derive(Deserialize)] #[derive(Deserialize)]
@@ -871,6 +1020,26 @@ mod tests {
assert!(two_sided >= 10, "{two_sided}"); assert!(two_sided >= 10, "{two_sided}");
} }
/// A market removed and tracked again: its first book's Snapshot says
/// so, once.
#[test]
fn a_market_tracked_again_says_so_in_its_first_snapshot() {
let mut feed = feed_of(&["A"], &[0]);
let first = feed.from(0, 1, book("A", 1, &[(400_000, 1_000_000)], &[]));
assert_eq!(snapshots(&first), 1);
assert_eq!(feed.shell.symbols["A"].again, None);
let a = market("A");
feed.remove_market(&a);
feed.add_market(&a, &[0], at(10));
assert_eq!(feed.shell.symbols["A"].again, Some("kalshi"));
let back = feed.from(0, 20, book("A", 9, &[(500_000, 2_000_000)], &[]));
assert_eq!(snapshots(&back), 1);
assert_eq!(feed.shell.symbols["A"].again, None);
// A market new to the run is not "again".
feed.add_market(&market("B"), &[0], at(20));
assert_eq!(feed.shell.symbols["B"].again, None);
}
/// The book and writer threads as main runs them, fed the recording: /// The book and writer threads as main runs them, fed the recording:
/// every Snapshot is the one before it with the Deltas between. /// every Snapshot is the one before it with the Deltas between.
#[test] #[test]
+4 -3
View File
@@ -1,6 +1,7 @@
//! Kalshi's market listing: which markets are live, or about to be. Two fetcher threads page //! Kalshi's market listing: which markets are live, or about to be. Two
//! it on the orchestrator's request and stream what they read; they hold no //! fetcher threads page it on the orchestrator's request and send on what
//! state and no timer, as Gamma's (roadmap/06). Read without a key. //! they read; they hold no state and no timer, as Gamma's (roadmap/06). Read
//! without a key.
use std::time::Duration; use std::time::Duration;
+6 -5
View File
@@ -15,8 +15,9 @@ use tungstenite::client::IntoClientRequest;
use crate::core::{self, Finding, Request, Undecoded}; use crate::core::{self, Finding, Request, Undecoded};
use sign::Key; use sign::Key;
/// Tickers per subscribe or unsubscribe message. /// Tickers per subscribe or unsubscribe message, as Polymarket's (whose
const CHUNK: usize = 500; /// limit is known, docs/notes.md); Kalshi documents none.
const PER_MESSAGE: usize = 500;
/// Errors after which the subscription is gone (Kalshi's docs: a channel /// Errors after which the subscription is gone (Kalshi's docs: a channel
/// error, a subscription buffer overflow): the connection starts over. /// error, a subscription buffer overflow): the connection starts over.
const TERMINAL_ERRORS: [u32; 2] = [10, 25]; const TERMINAL_ERRORS: [u32; 2] = [10, 25];
@@ -74,7 +75,7 @@ impl core::Exchange for Kalshi {
fn request(&self) -> anyhow::Result<tungstenite::handshake::client::Request> { fn request(&self) -> anyhow::Result<tungstenite::handshake::client::Request> {
let mut request = self.websocket.as_str().into_client_request()?; let mut request = self.websocket.as_str().into_client_request()?;
for (name, value) in self.key.headers(Utc::now().timestamp_millis()) { for (name, value) in self.key.headers(Utc::now().timestamp_millis())? {
request.headers_mut().insert(name, value.parse()?); request.headers_mut().insert(name, value.parse()?);
} }
Ok(request) Ok(request)
@@ -125,7 +126,7 @@ impl core::Session for Session {
/// default is announced to flip. /// default is announced to flip.
fn subscribe(&mut self, tickers: &[&String]) -> Vec<String> { fn subscribe(&mut self, tickers: &[&String]) -> Vec<String> {
tickers tickers
.chunks(CHUNK) .chunks(PER_MESSAGE)
.map(|chunk| { .map(|chunk| {
let params = json!({ let params = json!({
"channels": ["orderbook_delta"], "channels": ["orderbook_delta"],
@@ -143,7 +144,7 @@ impl core::Session for Session {
// unsubscribe sent before its answer names it so. // unsubscribe sent before its answer names it so.
let sid = self.sid.unwrap_or(1); let sid = self.sid.unwrap_or(1);
tickers tickers
.chunks(CHUNK) .chunks(PER_MESSAGE)
.map(|chunk| { .map(|chunk| {
let params = json!({ let params = json!({
"sids": [sid], "sids": [sid],
+191 -25
View File
@@ -1,11 +1,13 @@
//! The key that signs Kalshi's websocket handshake (docs/notes.md, Kalshi's //! The key that signs Kalshi's websocket handshake (docs/notes.md, Kalshi's
//! websocket): an Ed25519 key, read once from the files that the environment //! websocket): Ed25519, or RSA signed with RSA-PSS and SHA-256, the two
//! names. Neither file's content is ever logged. //! kinds Kalshi takes; read once from the files that the environment names.
//! Neither file's content is ever logged.
use std::path::Path; use std::path::Path;
use base64::Engine; use base64::Engine;
use ring::signature::Ed25519KeyPair; use ring::rand::SystemRandom;
use ring::signature::{Ed25519KeyPair, RSA_PSS_SHA256, RsaKeyPair};
use rustls_pki_types::PrivateKeyDer; use rustls_pki_types::PrivateKeyDer;
use rustls_pki_types::pem::PemObject; use rustls_pki_types::pem::PemObject;
@@ -18,15 +20,33 @@ const SIGNED_PATH: &str = "GET/trade-api/ws/v2";
pub struct Key { pub struct Key {
id: String, id: String,
pair: Ed25519KeyPair, pair: Pair,
}
/// Kalshi tells the two apart by the public key it holds; the PEM's header
/// doesn't say (an RSA key can be PKCS#8 too), so the parsed key does.
enum Pair {
Ed25519(Ed25519KeyPair),
Rsa(RsaKeyPair),
} }
impl Key { impl Key {
/// The key from the files [`KEY_ID_FILE`] and [`PRIVATE_KEY_FILE`] name. /// The key from the files [`KEY_ID_FILE`] and [`PRIVATE_KEY_FILE`] name.
pub fn from_env() -> anyhow::Result<Self> { pub fn from_env() -> anyhow::Result<Self> {
let path = |variable: &str| { Self::from_variables(|name| std::env::var_os(name))
std::env::var_os(variable) }
.ok_or_else(|| anyhow::anyhow!("{variable} is not set: it names a file of the key"))
/// [`Self::from_env`], the variables read through `variable`.
fn from_variables(
variable: impl Fn(&str) -> Option<std::ffi::OsString>,
) -> anyhow::Result<Self> {
let path = |name: &str| {
variable(name).ok_or_else(|| {
anyhow::anyhow!(
"{name} is not set: Kalshi needs {KEY_ID_FILE} and {PRIVATE_KEY_FILE}, \
the files of its key id and its private key"
)
})
}; };
Self::from_files( Self::from_files(
path(KEY_ID_FILE)?.as_ref(), path(KEY_ID_FILE)?.as_ref(),
@@ -37,7 +57,8 @@ impl Key {
}) })
} }
/// The key id in `id_file`, the private key (a PKCS#8 PEM) in `key_file`. /// The key id in `id_file`, the private key in `key_file`: a PEM,
/// Ed25519 in PKCS#8, or RSA in PKCS#1 (as Kalshi gives it) or PKCS#8.
pub fn from_files(id_file: &Path, key_file: &Path) -> anyhow::Result<Self> { pub fn from_files(id_file: &Path, key_file: &Path) -> anyhow::Result<Self> {
let id = std::fs::read_to_string(id_file) let id = std::fs::read_to_string(id_file)
.map_err(|error| anyhow::anyhow!("{}: {error}", id_file.display()))? .map_err(|error| anyhow::anyhow!("{}: {error}", id_file.display()))?
@@ -50,28 +71,54 @@ impl Key {
// error quotes what it read, so it is dropped. // error quotes what it read, so it is dropped.
let der = PrivateKeyDer::from_pem_slice(&pem) let der = PrivateKeyDer::from_pem_slice(&pem)
.map_err(|_| anyhow::anyhow!("{}: not a PEM private key", key_file.display()))?; .map_err(|_| anyhow::anyhow!("{}: not a PEM private key", key_file.display()))?;
let PrivateKeyDer::Pkcs8(der) = der else { let pair = match &der {
anyhow::bail!( PrivateKeyDer::Pkcs8(der) => {
"{}: not a PKCS#8 key; Kalshi's Ed25519 keys are", Ed25519KeyPair::from_pkcs8_maybe_unchecked(der.secret_pkcs8_der())
key_file.display() .map(Pair::Ed25519)
); .ok()
.or_else(|| {
RsaKeyPair::from_pkcs8(der.secret_pkcs8_der())
.ok()
.map(Pair::Rsa)
})
}
PrivateKeyDer::Pkcs1(der) => RsaKeyPair::from_der(der.secret_pkcs1_der())
.ok()
.map(Pair::Rsa),
_ => None,
}; };
let pair = Ed25519KeyPair::from_pkcs8_maybe_unchecked(der.secret_pkcs8_der()) let pair = pair.ok_or_else(|| {
.map_err(|_| anyhow::anyhow!("{}: not an Ed25519 key", key_file.display()))?; anyhow::anyhow!("{}: neither an Ed25519 nor an RSA key", key_file.display())
})?;
Ok(Self { id, pair }) Ok(Self { id, pair })
} }
/// The handshake's three headers at `millis` since the epoch. /// The handshake's three headers at `millis` since the epoch.
pub fn headers(&self, millis: i64) -> [(&'static str, String); 3] { pub fn headers(&self, millis: i64) -> anyhow::Result<[(&'static str, String); 3]> {
let signature = self.pair.sign(format!("{millis}{SIGNED_PATH}").as_bytes()); let signed = format!("{millis}{SIGNED_PATH}");
[ let signature = match &self.pair {
Pair::Ed25519(pair) => pair.sign(signed.as_bytes()).as_ref().to_vec(),
Pair::Rsa(pair) => {
// Salted: a new signature each time, from the system's random.
let mut signature = vec![0; pair.public().modulus_len()];
pair.sign(
&RSA_PSS_SHA256,
&SystemRandom::new(),
signed.as_bytes(),
&mut signature,
)
.map_err(|_| anyhow::anyhow!("the RSA signature failed"))?;
signature
}
};
Ok([
("KALSHI-ACCESS-KEY", self.id.clone()), ("KALSHI-ACCESS-KEY", self.id.clone()),
("KALSHI-ACCESS-TIMESTAMP", millis.to_string()), ("KALSHI-ACCESS-TIMESTAMP", millis.to_string()),
( (
"KALSHI-ACCESS-SIGNATURE", "KALSHI-ACCESS-SIGNATURE",
base64::engine::general_purpose::STANDARD.encode(signature.as_ref()), base64::engine::general_purpose::STANDARD.encode(signature),
), ),
] ])
} }
} }
@@ -92,7 +139,9 @@ pub fn test_key() -> Key {
}; };
Key { Key {
id: "a-key-id".into(), id: "a-key-id".into(),
pair: Ed25519KeyPair::from_pkcs8_maybe_unchecked(der.secret_pkcs8_der()).unwrap(), pair: Pair::Ed25519(
Ed25519KeyPair::from_pkcs8_maybe_unchecked(der.secret_pkcs8_der()).unwrap(),
),
} }
} }
@@ -120,7 +169,7 @@ mod tests {
&write(&dir, "key.pem", TEST_KEY), &write(&dir, "key.pem", TEST_KEY),
) )
.unwrap(); .unwrap();
let [id, time, signature] = key.headers(1_669_149_841_234); let [id, time, signature] = key.headers(1_669_149_841_234).unwrap();
assert_eq!(id, ("KALSHI-ACCESS-KEY", "a-key-id".to_string())); assert_eq!(id, ("KALSHI-ACCESS-KEY", "a-key-id".to_string()));
assert_eq!( assert_eq!(
time, time,
@@ -143,10 +192,122 @@ mod tests {
.verify(b"1669149841234GET/trade-api/ws/v2", &bytes) .verify(b"1669149841234GET/trade-api/ws/v2", &bytes)
.unwrap(); .unwrap();
// Ed25519 is deterministic: the same time, the same signature. // Ed25519 is deterministic: the same time, the same signature.
assert_eq!(key.headers(1_669_149_841_234)[2], signature); assert_eq!(key.headers(1_669_149_841_234).unwrap()[2], signature);
std::fs::remove_dir_all(dir).unwrap(); std::fs::remove_dir_all(dir).unwrap();
} }
/// RFC 9500's "testRSA2048" (section 2.1), a key published for tests,
/// in PKCS#1 as Kalshi gives it.
const RSA_PKCS1: &str = "-----BEGIN RSA PRIVATE KEY-----
MIIEowIBAAKCAQEAsPnoGUOnrpiSqt4XynxA+HRP7S+BSObI6qJ7fQAVSPtRkqso
tWxQYLEYzNEx5ZSHTGypibVsJylvCfuToDTfMul8b/CZjP2Ob0LdpYrNH6l5hvFE
89FU1nZQF15oVLOpUgA7wGiHuEVawrGfey92UE68mOyUVXGweJIVDdxqdMoPvNNU
l86BU02vlBiESxOuox+dWmuVV7vfYZ79Toh/LUK43YvJh+rhv4nKuF7iHjVjBd9s
B6iDjj70HFldzOQ9r8SRI+9NirupPTkF5AKNe6kUhKJ1luB7S27ZkvB3tSTT3P59
3VVJvnzOjaA1z6Cz+4+eRvcysqhrRgFlwI9TEwIDAQABAoIBAEEYiyDP29vCzx/+
dS3LqnI5BjUuJhXUnc6AWX/PCgVAO+8A+gZRgvct7PtZb0sM6P9ZcLrweomlGezI
FrL0/6xQaa8bBr/ve/a8155OgcjFo6fZEw3Dz7ra5fbSiPmu4/b/kvrg+Br1l77J
aun6uUAs1f5B9wW+vbR7tzbT/mxaUeDiBzKpe15GwcvbJtdIVMa2YErtRjc1/5B2
BGVXyvlJv0SIlcIEMsHgnAFOp1ZgQ08aDzvilLq8XVMOahAhP1O2A3X8hKdXPyrx
IVWE9bS9ptTo+eF6eNl+d7htpKGEZHUxinoQpWEBTv+iOoHsVunkEJ3vjLP3lyI/
fY0NQ1ECgYEA3RBXAjgvIys2gfU3keImF8e/TprLge1I2vbWmV2j6rZCg5r/AS0u
pii5CvJ5/T5vfJPNgPBy8B/yRDs+6PJO1GmnlhOkG9JAIPkv0RBZvR0PMBtbp6nT
Y3yo1lwamBVBfY6rc0sLTzosZh2aGoLzrHNMQFMGaauORzBFpY5lU50CgYEAzPHl
u5DI6Xgep1vr8QvCUuEesCOgJg8Yh1UqVoY/SmQh6MYAv1I9bLGwrb3WW/7kqIoD
fj0aQV5buVZI2loMomtU9KY5SFIsPV+JuUpy7/+VE01ZQM5FdY8wiYCQiVZYju9X
Wz5LxMNoz+gT7pwlLCsC4N+R8aoBk404aF1gum8CgYAJ7VTq7Zj4TFV7Soa/T1eE
k9y8a+kdoYk3BASpCHJ29M5R2KEA7YV9wrBklHTz8VzSTFTbKHEQ5W5csAhoL5Fo
qoHzFFi3Qx7MHESQb9qHyolHEMNx6QdsHUn7rlEnaTTyrXh3ifQtD6C0yTmFXUIS
CW9wKApOrnyKJ9nI0HcuZQKBgQCMtoV6e9VGX4AEfpuHvAAnMYQFgeBiYTkBKltQ
XwozhH63uMMomUmtSG87Sz1TmrXadjAhy8gsG6I0pWaN7QgBuFnzQ/HOkwTm+qKw
AsrZt4zeXNwsH7QXHEJCFnCmqw9QzEoZTrNtHJHpNboBuVnYcoueZEJrP8OnUG3r
UjmopwKBgAqB2KYYMUqAOvYcBnEfLDmyZv9BTVNHbR2lKkMYqv5LlvDaBxVfilE0
2riO4p6BaAdvzXjKeRrGNEKoHNBpOSfYCOM16NjL8hIZB1CaV3WbT5oY+jp7Mzd5
7d56RZOE+ERK2uz/7JX9VSsM/LbH9pJibd4e8mikDS9ntciqOH/3
-----END RSA PRIVATE KEY-----
";
/// The same key in PKCS#8 (`openssl pkcs8 -topk8 -nocrypt`).
const RSA_PKCS8: &str = "-----BEGIN PRIVATE KEY-----
MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCw+egZQ6eumJKq
3hfKfED4dE/tL4FI5sjqont9ABVI+1GSqyi1bFBgsRjM0THllIdMbKmJtWwnKW8J
+5OgNN8y6Xxv8JmM/Y5vQt2lis0fqXmG8UTz0VTWdlAXXmhUs6lSADvAaIe4RVrC
sZ97L3ZQTryY7JRVcbB4khUN3Gp0yg+801SXzoFTTa+UGIRLE66jH51aa5VXu99h
nv1OiH8tQrjdi8mH6uG/icq4XuIeNWMF32wHqIOOPvQcWV3M5D2vxJEj702Ku6k9
OQXkAo17qRSEonWW4HtLbtmS8He1JNPc/n3dVUm+fM6NoDXPoLP7j55G9zKyqGtG
AWXAj1MTAgMBAAECggEAQRiLIM/b28LPH/51LcuqcjkGNS4mFdSdzoBZf88KBUA7
7wD6BlGC9y3s+1lvSwzo/1lwuvB6iaUZ7MgWsvT/rFBprxsGv+979rzXnk6ByMWj
p9kTDcPPutrl9tKI+a7j9v+S+uD4GvWXvslq6fq5QCzV/kH3Bb69tHu3NtP+bFpR
4OIHMql7XkbBy9sm10hUxrZgSu1GNzX/kHYEZVfK+Um/RIiVwgQyweCcAU6nVmBD
TxoPO+KUurxdUw5qECE/U7YDdfyEp1c/KvEhVYT1tL2m1Oj54Xp42X53uG2koYRk
dTGKehClYQFO/6I6gexW6eQQne+Ms/eXIj99jQ1DUQKBgQDdEFcCOC8jKzaB9TeR
4iYXx79OmsuB7Uja9taZXaPqtkKDmv8BLS6mKLkK8nn9Pm98k82A8HLwH/JEOz7o
8k7UaaeWE6Qb0kAg+S/REFm9HQ8wG1unqdNjfKjWXBqYFUF9jqtzSwtPOixmHZoa
gvOsc0xAUwZpq45HMEWljmVTnQKBgQDM8eW7kMjpeB6nW+vxC8JS4R6wI6AmDxiH
VSpWhj9KZCHoxgC/Uj1ssbCtvdZb/uSoigN+PRpBXlu5VkjaWgyia1T0pjlIUiw9
X4m5SnLv/5UTTVlAzkV1jzCJgJCJVliO71dbPkvEw2jP6BPunCUsKwLg35HxqgGT
jThoXWC6bwKBgAntVOrtmPhMVXtKhr9PV4ST3Lxr6R2hiTcEBKkIcnb0zlHYoQDt
hX3CsGSUdPPxXNJMVNsocRDlblywCGgvkWiqgfMUWLdDHswcRJBv2ofKiUcQw3Hp
B2wdSfuuUSdpNPKteHeJ9C0PoLTJOYVdQhIJb3AoCk6ufIon2cjQdy5lAoGBAIy2
hXp71UZfgAR+m4e8ACcxhAWB4GJhOQEqW1BfCjOEfre4wyiZSa1IbztLPVOatdp2
MCHLyCwbojSlZo3tCAG4WfND8c6TBOb6orACytm3jN5c3CwftBccQkIWcKarD1DM
ShlOs20ckek1ugG5Wdhyi55kQms/w6dQbetSOainAoGACoHYphgxSoA69hwGcR8s
ObJm/0FNU0dtHaUqQxiq/kuW8NoHFV+KUTTauI7inoFoB2/NeMp5GsY0Qqgc0Gk5
J9gI4zXo2MvyEhkHUJpXdZtPmhj6OnszN3nt3npFk4T4REra7P/slf1VKwz8tsf2
kmJt3h7yaKQNL2e1yKo4f/c=
-----END PRIVATE KEY-----
";
#[test]
fn signs_with_an_rsa_key_in_pss_and_sha256() {
let dir = scratch("rsa");
for (name, pem) in [("pkcs1.pem", RSA_PKCS1), ("pkcs8.pem", RSA_PKCS8)] {
let key = Key::from_files(&write(&dir, "id", "an-rsa-key\n"), &write(&dir, name, pem))
.unwrap();
let Pair::Rsa(pair) = &key.pair else {
panic!("{name}: read as Ed25519");
};
let [id, _, signature] = key.headers(1_669_149_841_234).unwrap();
assert_eq!(id.1, "an-rsa-key");
let bytes = base64::engine::general_purpose::STANDARD
.decode(&signature.1)
.unwrap();
let public = ring::signature::UnparsedPublicKey::new(
&ring::signature::RSA_PSS_2048_8192_SHA256,
pair.public().as_ref(),
);
public
.verify(b"1669149841234GET/trade-api/ws/v2", &bytes)
.unwrap();
// Not PKCS#1 v1.5, which Kalshi doesn't take.
let pkcs1v15 = ring::signature::UnparsedPublicKey::new(
&ring::signature::RSA_PKCS1_2048_8192_SHA256,
pair.public().as_ref(),
);
assert!(
pkcs1v15
.verify(b"1669149841234GET/trade-api/ws/v2", &bytes)
.is_err()
);
}
std::fs::remove_dir_all(dir).unwrap();
}
#[test]
fn a_missing_variable_names_both() {
let error = Key::from_variables(|_| None).err().unwrap().to_string();
assert!(
error.starts_with("KALSHI_KEY_ID_FILE is not set"),
"{error}"
);
assert!(error.contains("KALSHI_PRIVATE_KEY_FILE"), "{error}");
let only_id = |name: &str| (name == KEY_ID_FILE).then(|| "id".into());
let error = Key::from_variables(only_id).err().unwrap().to_string();
assert!(
error.starts_with("KALSHI_PRIVATE_KEY_FILE is not set"),
"{error}"
);
}
#[test] #[test]
fn a_missing_or_wrong_file_says_which_and_never_what_it_holds() { fn a_missing_or_wrong_file_says_which_and_never_what_it_holds() {
let dir = scratch("errors"); let dir = scratch("errors");
@@ -170,13 +331,18 @@ mod tests {
"{unreadable}" "{unreadable}"
); );
// An RSA key in PKCS#1, as Kalshi generates its RSA keys: refused. // An RSA header over what isn't an RSA key: refused, its content
// unquoted.
let rsa = write( let rsa = write(
&dir, &dir,
"rsa.pem", "rsa.pem",
"-----BEGIN RSA PRIVATE KEY-----\nMAA=\n-----END RSA PRIVATE KEY-----\n", "-----BEGIN RSA PRIVATE KEY-----\nMAA=\n-----END RSA PRIVATE KEY-----\n",
); );
assert!(error(&id, &rsa).contains("not a PKCS#8 key")); let refused = error(&id, &rsa);
assert!(
refused.ends_with("neither an Ed25519 nor an RSA key"),
"{refused}"
);
let empty = write(&dir, "empty-id", "\n"); let empty = write(&dir, "empty-id", "\n");
let key = write(&dir, "key.pem", TEST_KEY); let key = write(&dir, "key.pem", TEST_KEY);
+1
View File
@@ -1,6 +1,7 @@
pub mod book; pub mod book;
pub mod cause; pub mod cause;
pub mod reconcile; pub mod reconcile;
pub mod show;
pub mod util; pub mod util;
mod side; mod side;
+19 -7
View File
@@ -29,13 +29,25 @@ mod polymarket;
use engineering_challenge::book::Output; use engineering_challenge::book::Output;
/// `AlignedSbeReader::open` reads nothing else. /// `AlignedSbeReader::open` reads nothing else.
const CHUNK: usize = 64 << 20; const OUTPUT_CHUNK: usize = 64 << 20;
const USAGE: &str = "usage: engineering-challenge <polymarket|kalshi|kalshi-demo> [OUTPUT], or latency <polymarket|kalshi|kalshi-demo> OUTPUT"; const USAGE: &str = "\
usage: engineering-challenge EXCHANGE [OUTPUT]
EXCHANGE polymarket (needs no key), kalshi, or kalshi-demo (Kalshi's demo)
OUTPUT the SBE output, stdout by default; the log goes to stderr
Kalshi's key: KALSHI_KEY_ID_FILE and KALSHI_PRIVATE_KEY_FILE name the files
of its id and its private key (a PEM, Ed25519 or RSA).
SIGINT, SIGTERM or SIGHUP stops it and flushes the output.
or: engineering-challenge latency EXCHANGE OUT.csv, the latency probe
(LATENCY.md); SIGINT stops it.";
fn main() -> anyhow::Result<()> { fn main() -> anyhow::Result<()> {
let mut args = std::env::args().skip(1); let mut args = std::env::args().skip(1);
let exchange = args.next().ok_or_else(|| anyhow::anyhow!(USAGE))?; let exchange = args.next().ok_or_else(|| anyhow::anyhow!(USAGE))?;
if exchange == "--help" || exchange == "-h" {
println!("{USAGE}");
return Ok(());
}
if exchange == "latency" { if exchange == "latency" {
return latency::run(args); return latency::run(args);
} }
@@ -53,7 +65,7 @@ fn main() -> anyhow::Result<()> {
output, output,
), ),
"kalshi-demo" => collect(kalshi::Kalshi::demo(kalshi::sign::Key::from_env()?), output), "kalshi-demo" => collect(kalshi::Kalshi::demo(kalshi::sign::Key::from_env()?), output),
_ => anyhow::bail!("unknown exchange {exchange:?}; {USAGE}"), _ => anyhow::bail!("unknown exchange {exchange:?}\n{USAGE}"),
} }
} }
@@ -87,12 +99,12 @@ fn collect<E: core::Exchange>(exchange: E, output: String) -> anyhow::Result<()>
/// The writer thread: writes outputs in arrival order until every book /// The writer thread: writes outputs in arrival order until every book
/// thread is gone, then flushes the last chunk. /// thread is gone, then flushes the last chunk.
// ponytail: the chunk being filled lives only in memory until it is full, so // Ceiling: the chunk being filled lives only in memory until it is full, so
// a hard crash loses it; flushing more often pads every flush to 64 MiB. // a hard crash loses it; flushing more often pads every flush to 64 MiB.
// Kept as a ceiling in M4 (roadmap/04): writing to a file of our own and // Accepted for redundancy's sake too (docs/project.md, Constraints): writing
// copying it into chunks would lift it. // to a file of our own and copying it into chunks would lift it.
fn write(path: &str, outputs: Receiver<Output>) -> anyhow::Result<()> { fn write(path: &str, outputs: Receiver<Output>) -> anyhow::Result<()> {
let mut writer = AlignedSbeWriter::create(path, CHUNK)?; let mut writer = AlignedSbeWriter::create(path, OUTPUT_CHUNK)?;
for output in outputs { for output in outputs {
match output { match output {
Output::Snapshot(snapshot) => writer.write_msg(&snapshot)?, Output::Snapshot(snapshot) => writer.write_msg(&snapshot)?,
+76 -19
View File
@@ -28,7 +28,7 @@ use engineering_challenge::book::{Book, Output, format_fixed, parse_fixed};
/// so is a fresh book behind our stream. /// so is a fresh book behind our stream.
const RECENT_CHANGES: usize = 128; const RECENT_CHANGES: usize = 128;
/// How long a change may be held while another connection might still deliver /// How long a change may be held while another connection might still deliver
/// the one before it: the second delivery lagged up to 0.7 s. /// the one before it: the second delivery was up to 0.7 s late.
const HOLD: TimeDelta = TimeDelta::seconds(1); const HOLD: TimeDelta = TimeDelta::seconds(1);
/// How long once no other connection can: only the rest of a batch is still to /// How long once no other connection can: only the rest of a batch is still to
/// come, from the same connection (40 ms at worst). /// come, from the same connection (40 ms at worst).
@@ -91,12 +91,6 @@ impl Held {
} }
} }
impl RecentChange {
fn is_copy(&self, change: &Held) -> bool {
(self.hash, self.side, self.price, self.quantity) == change.key()
}
}
/// A change as applied, with the exchange's hash of the book after it. /// A change as applied, with the exchange's hash of the book after it.
struct RecentChange { struct RecentChange {
side: Side, side: Side,
@@ -115,6 +109,12 @@ struct RecentChange {
counted: bool, counted: bool,
} }
impl RecentChange {
fn is_copy(&self, change: &Held) -> bool {
(self.hash, self.side, self.price, self.quantity) == change.key()
}
}
/// A symbol's state for the rule. /// A symbol's state for the rule.
pub struct Chained { pub struct Chained {
state: State, state: State,
@@ -524,7 +524,7 @@ impl Chain {
}; };
if reconnect { if reconnect {
self.counts.reconnect_books += 1; self.counts.reconnect_books += 1;
// The exchange's opening books lag by minutes (docs/notes.md): one // The exchange's opening books trail by minutes (docs/notes.md): one
// that is ours, or a state we passed, leaves ours, and the hash // that is ours, or a state we passed, leaves ours, and the hash
// inputs, as they are. // inputs, as they are.
let brought = symbol let brought = symbol
@@ -537,7 +537,7 @@ impl Chain {
return; return;
} }
// Our chain holds up to our last change, so a book no newer than // Our chain holds up to our last change, so a book no newer than
// it holds nothing we missed, however far back it lags: further // it holds nothing we missed, however far behind it is: further
// than the changes we remember, too. // than the changes we remember, too.
if exchange_timestamp <= symbol.exchange_timestamp || symbol.was_ours(&book) { if exchange_timestamp <= symbol.exchange_timestamp || symbol.was_ours(&book) {
self.counts.books_behind += 1; self.counts.books_behind += 1;
@@ -583,7 +583,7 @@ impl Chain {
let mut book = book; let mut book = book;
let book_time = exchange_timestamp; let book_time = exchange_timestamp;
let mut exchange_timestamp = exchange_timestamp; let mut exchange_timestamp = exchange_timestamp;
// The book a resync brings can lag the changes already applied, with a // The book a resync brings can trail the changes already applied, with a
// newer timestamp or last trade price than its levels (docs/notes.md). // newer timestamp or last trade price than its levels (docs/notes.md).
// It is behind if it is ours as it was a few changes ago: that past // It is behind if it is ours as it was a few changes ago: that past
// state was the exchange's own, and the changes since are what it // state was the exchange's own, and the changes since are what it
@@ -621,7 +621,8 @@ impl Chain {
Some(Cause::Loss) => self.counts.losses += 1, Some(Cause::Loss) => self.counts.losses += 1,
Some(Cause::SubscriptionGap) => self.counts.subscription_gaps += 1, Some(Cause::SubscriptionGap) => self.counts.subscription_gaps += 1,
Some(Cause::Switch) => self.counts.switches += 1, Some(Cause::Switch) => self.counts.switches += 1,
None => {} // Not found here: the shell says it, at the Snapshot.
Some(Cause::TrackedAgain) | None => {}
} }
symbol.book = book; symbol.book = book;
symbol.exchange_timestamp = exchange_timestamp; symbol.exchange_timestamp = exchange_timestamp;
@@ -633,11 +634,7 @@ impl Chain {
// Our changes before this book no longer lead to ours: rewinding // Our changes before this book no longer lead to ours: rewinding
// through them would make up states that never existed. // through them would make up states that never existed.
symbol.rule.recent_changes.clear(); symbol.rule.recent_changes.clear();
let snapshot = symbol effects.push(Effect::Write(symbol.snapshot(&token, received_at)));
.book
.snapshot(&token, received_at, exchange_timestamp);
effects.push(Effect::Write(snapshot));
symbol.last_written = symbol.last_written.max(received_at);
if let Some(cause) = cause { if let Some(cause) = cause {
let found = if reconnect { let found = if reconnect {
", found by a reconnect" ", found by a reconnect"
@@ -1370,7 +1367,7 @@ mod tests {
(token.clone(), free) (token.clone(), free)
} }
// --- one connection: sprint 02's rules still hold --- // --- one connection: the hash check alone ---
/// Split batches and trades make checks wait or fail with nothing lost: /// Split batches and trades make checks wait or fail with nothing lost:
/// the recording must replay without a single loss. /// the recording must replay without a single loss.
@@ -1980,6 +1977,38 @@ mod tests {
assert_eq!(feed.rule.counts.rescued, rescued + 1); assert_eq!(feed.rule.counts.rescued, rescued + 1);
} }
/// A market removed while a change of it waits: the book thread stops
/// waking for it.
#[test]
fn a_market_removed_while_a_change_waits_no_longer_wakes_the_book_thread() {
let (mut feed, token, book, free, millis, at) = two_connection_symbol();
let symbol = &feed.shell.symbols[&token];
let (_, after_x) = changes_frame(
&token,
symbol,
&book,
&[(Side::Buy, free, 1_000_000)],
millis,
);
let (y, _) = changes_frame(
&token,
symbol,
&after_x,
&[(Side::Buy, free + 1_000, 2_000_000)],
millis + 1,
);
assert!(feed.on_frame_from(0, at, &y).is_empty());
assert!(feed.rule.wakes_soon());
let markets = fixture_markets();
let market = markets
.iter()
.find(|market| market.tokens.contains(&token))
.unwrap();
feed.remove_market(market);
assert!(!feed.rule.waiting.contains(&token));
assert!(!feed.rule.wakes_soon());
}
/// A burst on a connection that missed the change before it: the held /// A burst on a connection that missed the change before it: the held
/// changes stop at the cap, the chain breaks, and it is all quick. /// changes stop at the cap, the chain breaks, and it is all quick.
#[test] #[test]
@@ -2276,10 +2305,13 @@ mod tests {
assert_eq!(feed.shell.quiet, 1); assert_eq!(feed.shell.quiet, 1);
assert_eq!(feed.shell.symbols[&token].rule.state, State::Verified); assert_eq!(feed.shell.symbols[&token].rule.state, State::Verified);
// The book the resubscribe brings back is not asked for: ignored. // The book the resubscribe brings back is not asked for: ignored.
// Nor judged as a reconnect's: the subscription there is renewed.
let judged = feed.rule.counts.reconnect_books;
let book = feed.shell.symbols[&token].book.clone(); let book = feed.shell.symbols[&token].book.clone();
let back = fresh_book(&token, &feed.shell.symbols[&token], &book); let back = fresh_book(&token, &feed.shell.symbols[&token], &book);
let effects = feed.on_frame_from(1, minute(2) + TimeDelta::seconds(1), &back); let effects = feed.on_frame_from(1, minute(2) + TimeDelta::seconds(1), &back);
assert!(snapshot_symbols(&effects).is_empty()); assert!(snapshot_symbols(&effects).is_empty());
assert_eq!(feed.rule.counts.reconnect_books, judged);
// Still quiet the minute after: reported for a move, not resubscribed. // Still quiet the minute after: reported for a move, not resubscribed.
changes(&mut feed, &token, &[0], minute(2), free, 3); changes(&mut feed, &token, &[0], minute(2), free, 3);
@@ -2385,14 +2417,14 @@ mod tests {
let effects = reconnect_with(&mut feed, &token, &ours, later); let effects = reconnect_with(&mut feed, &token, &ours, later);
assert!(snapshot_symbols(&effects).is_empty()); assert!(snapshot_symbols(&effects).is_empty());
assert_eq!(feed.rule.counts.reconnect_books, 1); assert_eq!(feed.rule.counts.reconnect_books, 1);
// The server's opening books lag: a state we passed is no loss. // The server's opening books trail: a state we passed is no loss.
let behind = feed.rule.counts.books_behind; let behind = feed.rule.counts.books_behind;
let effects = reconnect_with(&mut feed, &token, &before, later + TimeDelta::seconds(1)); let effects = reconnect_with(&mut feed, &token, &before, later + TimeDelta::seconds(1));
assert!(snapshot_symbols(&effects).is_empty()); assert!(snapshot_symbols(&effects).is_empty());
assert_eq!(feed.rule.counts.reconnect_books, 2); assert_eq!(feed.rule.counts.reconnect_books, 2);
assert_eq!(feed.rule.counts.books_behind, behind + 1); assert_eq!(feed.rule.counts.books_behind, behind + 1);
assert_eq!(feed.shell.symbols[&token].book, ours); assert_eq!(feed.shell.symbols[&token].book, ours);
// It lags further back than the changes we remember: no newer than // It trails further back than the changes we remember: no newer than
// our last change, it still holds nothing we missed. // our last change, it still holds nothing we missed.
let symbol = feed.shell.symbols.get_mut(&token).unwrap(); let symbol = feed.shell.symbols.get_mut(&token).unwrap();
symbol.rule.recent_changes.clear(); symbol.rule.recent_changes.clear();
@@ -2719,6 +2751,31 @@ mod tests {
assert_eq!(snapshot_symbols(&effects).len(), 38 * 2); assert_eq!(snapshot_symbols(&effects).len(), 38 * 2);
} }
/// Removed, then tracked again: its first books' Snapshots say so
/// (the shell's flag, cleared once written).
#[test]
fn a_market_tracked_again_says_so_in_its_first_snapshot() {
let frames = fixture_frames();
let markets = fixture_markets();
let mut feed = feed_of(&markets);
replay(&mut feed, &frames[..1]);
feed.remove_market(&markets[0]);
feed.add_market(&markets[0], &[0], frames[100].0);
let tokens = &markets[0].tokens;
assert!(
tokens
.iter()
.all(|token| feed.shell.symbols[token].again.is_some())
);
let effects = feed.on_frame(frames[100].0, &opening_books(&frames, &markets[0]));
assert_eq!(snapshot_symbols(&effects).len(), tokens.len());
assert!(
tokens
.iter()
.all(|token| feed.shell.symbols[token].again.is_none())
);
}
#[test] #[test]
fn each_minute_reports_the_bytes_per_second_of_every_market() { fn each_minute_reports_the_bytes_per_second_of_every_market() {
let minute = "2026-11-10T16:00:00Z".parse::<DateTime<Utc>>().unwrap(); let minute = "2026-11-10T16:00:00Z".parse::<DateTime<Utc>>().unwrap();
+2 -2
View File
@@ -1,6 +1,6 @@
//! Gamma, Polymarket's market listing: which markets are live. Two fetcher //! Gamma, Polymarket's market listing: which markets are live. Two fetcher
//! threads page it on the orchestrator's request and stream what they read; //! threads page it on the orchestrator's request and send on what they
//! they hold no state and no timer (roadmap/03). //! read; they hold no state and no timer (roadmap/03).
use std::time::Duration; use std::time::Duration;
+2 -2
View File
@@ -15,7 +15,7 @@ use crate::core::{self, Finding, Request, Undecoded};
pub const MARKET_CHANNEL_URL: &str = "wss://ws-subscriptions-clob.polymarket.com/ws/market"; pub const MARKET_CHANNEL_URL: &str = "wss://ws-subscriptions-clob.polymarket.com/ws/market";
/// Tokens per subscribe or unsubscribe message: a first message of more than /// Tokens per subscribe or unsubscribe message: a first message of more than
/// 500 gets no opening books (docs/notes.md). /// 500 gets no opening books (docs/notes.md).
const CHUNK: usize = 500; const PER_MESSAGE: usize = 500;
/// A live market: its symbols and the two inputs of their book hash that the /// A live market: its symbols and the two inputs of their book hash that the
/// websocket doesn't carry (docs/notes.md). /// websocket doesn't carry (docs/notes.md).
@@ -78,7 +78,7 @@ impl Session {
/// connection. /// connection.
fn operations(operation: &str, tokens: &[&String]) -> Vec<String> { fn operations(operation: &str, tokens: &[&String]) -> Vec<String> {
tokens tokens
.chunks(CHUNK) .chunks(PER_MESSAGE)
.map(|chunk| json!({"assets_ids": chunk, "operation": operation}).to_string()) .map(|chunk| json!({"assets_ids": chunk, "operation": operation}).to_string())
.collect() .collect()
} }
+174 -9
View File
@@ -2,22 +2,48 @@
//! Polymarket's codec. //! Polymarket's codec.
use std::collections::BTreeSet; use std::collections::BTreeSet;
use std::io::ErrorKind;
use std::net::{TcpListener, TcpStream}; use std::net::{TcpListener, TcpStream};
use std::sync::Arc; use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering}; use std::sync::atomic::{AtomicBool, Ordering};
use std::thread; use std::thread;
use std::time::Duration; use std::time::{Duration, Instant};
use chrono::Utc; use chrono::Utc;
use crossbeam::channel::{Sender, unbounded}; use crossbeam::channel::{Sender, unbounded};
use serde_json::json; use serde_json::json;
use tungstenite::protocol::CloseFrame;
use tungstenite::protocol::frame::coding::CloseCode;
use tungstenite::{Message, WebSocket}; use tungstenite::{Message, WebSocket};
use super::{CHUNK, Polymarket, Session}; use super::{PER_MESSAGE, Polymarket, Session};
use crate::core::Event; use crate::core::Event;
use crate::core::socket::{Command, Notice, Routed, SILENT_AFTER, apply, route, run}; use crate::core::socket::{Command, Notice, PING_EVERY, Routed, SILENT_AFTER, apply, route, run};
use crate::core::{BOOK_THREADS, Received, book_thread}; use crate::core::{BOOK_THREADS, Received, book_thread};
/// The next connection to `listener`, within 5 s: a client that never
/// connects (again) fails the test rather than hangs it.
fn accept(listener: &TcpListener) -> TcpStream {
listener.set_nonblocking(true).unwrap();
let deadline = Instant::now() + Duration::from_secs(5);
loop {
match listener.accept() {
Ok((stream, _)) => {
stream.set_nonblocking(false).unwrap();
stream
.set_read_timeout(Some(Duration::from_secs(5)))
.unwrap();
return stream;
}
Err(error) if error.kind() == ErrorKind::WouldBlock => {
assert!(Instant::now() < deadline, "no connection within 5 s");
thread::sleep(Duration::from_millis(10));
}
Err(error) => panic!("accept: {error}"),
}
}
}
/// The tokens each message of one connection subscribed, after the first. /// The tokens each message of one connection subscribed, after the first.
fn subscribed(socket: &mut WebSocket<TcpStream>, tokens: usize) -> Vec<Vec<String>> { fn subscribed(socket: &mut WebSocket<TcpStream>, tokens: usize) -> Vec<Vec<String>> {
let first = socket.read().unwrap().into_text().unwrap(); let first = socket.read().unwrap().into_text().unwrap();
@@ -83,7 +109,7 @@ fn a_reconnect_subscribes_every_token_again_in_chunks() {
let mut tokens = tokens; let mut tokens = tokens;
tokens.sort(); tokens.sort();
for chunks in connections { for chunks in connections {
assert!(chunks.iter().all(|chunk| chunk.len() <= CHUNK)); assert!(chunks.iter().all(|chunk| chunk.len() <= PER_MESSAGE));
let mut subscribed = chunks.concat(); let mut subscribed = chunks.concat();
subscribed.sort(); subscribed.sort();
assert_eq!(subscribed, tokens); assert_eq!(subscribed, tokens);
@@ -102,11 +128,7 @@ fn serving_twice(
let server = thread::spawn(move || { let server = thread::spawn(move || {
(0..2) (0..2)
.map(|_| { .map(|_| {
let (stream, _) = listener.accept().unwrap(); let mut socket = tungstenite::accept(accept(&listener)).unwrap();
stream
.set_read_timeout(Some(Duration::from_secs(5)))
.unwrap();
let mut socket = tungstenite::accept(stream).unwrap();
let chunks = subscribed(&mut socket, total); let chunks = subscribed(&mut socket, total);
serve(&mut socket); serve(&mut socket);
chunks chunks
@@ -270,3 +292,146 @@ fn commands_update_the_token_set_and_say_what_to_send() {
assert!(apply(&mut session, &mut tokens, &Command::Resubscribe("a".into())).is_empty()); assert!(apply(&mut session, &mut tokens, &Command::Resubscribe("a".into())).is_empty());
assert_eq!(tokens.into_iter().collect::<Vec<_>>(), ["b"]); assert_eq!(tokens.into_iter().collect::<Vec<_>>(), ["b"]);
} }
/// A local server for one connection of one token: once subscribed, `serve`
/// handles it and sends what it found on the channel returned.
fn serving_once<T: Send + 'static>(
serve: impl FnOnce(&mut WebSocket<TcpStream>, &Sender<T>) + Send + 'static,
) -> (String, crossbeam::channel::Receiver<T>) {
let listener = TcpListener::bind("127.0.0.1:0").unwrap();
let url = format!("ws://{}/ws/market", listener.local_addr().unwrap());
let (found, found_rx) = unbounded();
thread::spawn(move || {
let mut socket = tungstenite::accept(accept(&listener)).unwrap();
subscribed(&mut socket, 1);
serve(&mut socket, &found);
while socket.read().is_ok() {}
});
(url, found_rx)
}
/// Every other socket test queues its Subscribe before the connection is up.
#[test]
fn a_command_sent_on_an_open_connection_reaches_the_server() {
let (url, sent) = serving_once(|socket, sent| {
sent.send(String::new()).unwrap();
// The keepalive comes only after 10 s: the next text is the command.
let text = socket.read().unwrap().into_text().unwrap();
sent.send(text.to_string()).unwrap();
});
client(url, vec!["a".into()], SILENT_AFTER, |commands| {
// Subscribed: the connection is open.
sent.recv_timeout(Duration::from_secs(5)).unwrap();
commands.send(Command::Subscribe(vec!["b".into()])).unwrap();
let text = sent.recv_timeout(Duration::from_secs(5)).unwrap();
let message: serde_json::Value = serde_json::from_str(&text).unwrap();
assert_eq!(
message,
json!({"assets_ids": ["b"], "operation": "subscribe"})
);
});
}
#[test]
fn a_close_with_1013_tells_the_orchestrator_the_connection_is_too_slow() {
let (url, closed) = serving_once(|socket, closed| {
let frame = CloseFrame {
code: CloseCode::Again,
reason: "".into(),
};
socket.close(Some(frame)).unwrap();
closed.send(()).unwrap();
});
// As `client` does, but waiting for the notices themselves, each under
// a timeout, rather than a sleep before reading them.
let (events_tx, _events_rx) = unbounded();
let (commands_tx, commands_rx) = unbounded();
commands_tx
.send(Command::Subscribe(vec!["a".into()]))
.unwrap();
let stop = Arc::new(AtomicBool::new(false));
let client_stop = Arc::clone(&stop);
let (notices, notices_rx) = unbounded();
let client = thread::spawn(move || {
run(
&Polymarket { url },
0,
&[events_tx],
&commands_rx,
&notices,
SILENT_AFTER,
&client_stop,
)
});
closed.recv_timeout(Duration::from_secs(5)).unwrap();
let first: Vec<Notice> = (0..3)
.map(|_| notices_rx.recv_timeout(Duration::from_secs(5)).unwrap())
.collect();
stop.store(true, Ordering::Relaxed);
client.join().unwrap();
assert_eq!(first, [Notice::Up(0), Notice::TooSlow(0), Notice::Down(0)]);
}
#[test]
fn a_connection_sends_its_keepalive_every_ten_seconds() {
let (url, received) = serving_once(|socket, received| {
let stream = socket.get_ref();
stream
.set_read_timeout(Some(PING_EVERY + Duration::from_secs(5)))
.unwrap();
let text = socket.read().unwrap().into_text().unwrap();
received.send(text.to_string()).unwrap();
});
client(url, vec!["a".into()], SILENT_AFTER, |_| {
let text = received
.recv_timeout(PING_EVERY + Duration::from_secs(5))
.unwrap();
assert_eq!(text, "PING");
});
}
/// Only a panic ends a book thread before its connections: the run stops.
#[test]
fn a_book_thread_gone_stops_the_connection_and_the_run() {
#[derive(serde::Deserialize)]
struct Line {
text: String,
}
let path = concat!(
env!("CARGO_MANIFEST_DIR"),
"/tests/fixtures/polymarket-frames.jsonl"
);
let recording = std::fs::read_to_string(path).unwrap();
let line: Line = serde_json::from_str(recording.lines().next().unwrap()).unwrap();
let (url, _sent) = serving_once(move |socket, sent| {
socket.send(Message::text(line.text)).unwrap();
sent.send(()).unwrap();
});
let (book_threads, receivers): (Vec<_>, Vec<_>) =
(0..BOOK_THREADS).map(|_| unbounded()).unzip();
drop(receivers);
let (commands_tx, commands) = unbounded();
commands_tx
.send(Command::Subscribe(vec!["a".into()]))
.unwrap();
let (notices, _notices_rx) = unbounded();
let stop = Arc::new(AtomicBool::new(false));
let client_stop = Arc::clone(&stop);
let (ended, ended_rx) = unbounded();
thread::spawn(move || {
run(
&Polymarket { url },
0,
&book_threads,
&commands,
&notices,
SILENT_AFTER,
&client_stop,
);
ended.send(()).unwrap();
});
ended_rx
.recv_timeout(Duration::from_secs(5))
.expect("the connection stops by itself");
assert!(stop.load(Ordering::Relaxed), "and stops the run");
}
+99 -6
View File
@@ -99,10 +99,11 @@ mod tests {
use crate::util::dtos::challenge_orderbook::{PriceLevelUpdateDomain, SnapshotBidsEntryDomain}; use crate::util::dtos::challenge_orderbook::{PriceLevelUpdateDomain, SnapshotBidsEntryDomain};
/// An output with a symbol's Snapshot, a Delta, then a Snapshot that /// An output with a symbol's Snapshot, a Delta, then a Snapshot that
/// misses it; and a log line for that second Snapshot when `explained`. /// misses it; and a log line with `cause` for that second Snapshot when
fn reconcile_with(explained: bool) -> Report { /// `explained`.
fn reconcile_with(explained: bool, cause: Cause) -> Report {
let dir = std::env::temp_dir(); let dir = std::env::temp_dir();
let id = format!("{}-{explained}", std::process::id()); let id = format!("{}-{explained}-{cause}", std::process::id()).replace(' ', "-");
let (output, log) = ( let (output, log) = (
dir.join(format!("reconcile-{id}.sbe")), dir.join(format!("reconcile-{id}.sbe")),
dir.join(format!("reconcile-{id}.log")), dir.join(format!("reconcile-{id}.log")),
@@ -132,7 +133,7 @@ mod tests {
.unwrap(); .unwrap();
writer.write_msg(&snapshot(at(2), 3_000_000)).unwrap(); writer.write_msg(&snapshot(at(2), 3_000_000)).unwrap();
writer.flush().unwrap(); writer.flush().unwrap();
let line = cause::line("1", at(if explained { 2 } else { 3 }), Cause::Loss, ""); let line = cause::line("1", at(if explained { 2 } else { 3 }), cause, "");
std::fs::write(&log, format!("12:00:00.000 polymarket: {line}\n")).unwrap(); std::fs::write(&log, format!("12:00:00.000 polymarket: {line}\n")).unwrap();
let report = reconcile(&output, Some(&log)).unwrap(); let report = reconcile(&output, Some(&log)).unwrap();
std::fs::remove_file(output).unwrap(); std::fs::remove_file(output).unwrap();
@@ -140,14 +141,106 @@ mod tests {
report report
} }
fn at(seconds: i64) -> DateTime<Utc> {
DateTime::from_timestamp(1_794_390_000 + seconds, 0).unwrap()
}
fn snapshot(received_at: DateTime<Utc>, quantity: i64) -> SnapshotDomain {
SnapshotDomain {
received_at,
exchange_timestamp: received_at,
bids: vec![SnapshotBidsEntryDomain {
price_level: 500_000,
quantity,
}],
asks: Vec::new(),
symbol: "1".to_string(),
}
}
fn delta(received_at: DateTime<Utc>, quantity: i64) -> PriceLevelUpdateDomain {
PriceLevelUpdateDomain {
backtest_timestamp: received_at,
exchange_timestamp: received_at,
price_level: 500_000,
quantity,
side: Side::Buy,
symbol: "1".to_string(),
}
}
/// The report on the output `write` writes, with `log` as the log;
/// `name` keeps parallel tests apart.
fn reconciled(name: &str, write: impl FnOnce(&mut AlignedSbeWriter), log: &str) -> Report {
let dir = std::env::temp_dir();
let id = format!("{}-{name}", std::process::id());
let (output, log_path) = (
dir.join(format!("reconcile-{id}.sbe")),
dir.join(format!("reconcile-{id}.log")),
);
let mut writer = AlignedSbeWriter::create(output.to_str().unwrap(), 64 << 20).unwrap();
write(&mut writer);
writer.flush().unwrap();
std::fs::write(&log_path, log).unwrap();
let report = reconcile(&output, Some(&log_path)).unwrap();
std::fs::remove_file(output).unwrap();
std::fs::remove_file(log_path).unwrap();
report
}
#[test]
fn a_delta_before_its_symbols_first_snapshot_is_an_orphan() {
let report = reconciled(
"orphan",
|writer| {
writer.write_msg(&delta(at(0), 2_000_000)).unwrap();
writer.write_msg(&snapshot(at(1), 1_000_000)).unwrap();
writer.write_msg(&delta(at(2), 3_000_000)).unwrap();
},
"",
);
assert_eq!((report.deltas, report.orphan_deltas), (2, 1));
}
/// Two Snapshots of one symbol stamped alike, both mismatched: the cause
/// line explains one of them only.
#[test]
fn a_cause_line_explains_one_snapshot_only() {
let line = cause::line("1", at(2), Cause::Loss, "");
let report = reconciled(
"once",
|writer| {
writer.write_msg(&snapshot(at(0), 1_000_000)).unwrap();
writer.write_msg(&delta(at(1), 2_000_000)).unwrap();
writer.write_msg(&snapshot(at(2), 3_000_000)).unwrap();
writer.write_msg(&delta(at(2), 4_000_000)).unwrap();
writer.write_msg(&snapshot(at(2), 5_000_000)).unwrap();
},
&format!("12:00:00.000 kalshi: {line}\n"),
);
assert_eq!(report.mismatches.len(), 2);
assert_eq!(report.explained, HashMap::from([(Cause::Loss, 1)]));
assert_eq!(report.unexplained, [("1".to_string(), at(2))]);
}
#[test] #[test]
fn a_mismatch_with_its_log_line_is_explained_and_one_without_is_not() { fn a_mismatch_with_its_log_line_is_explained_and_one_without_is_not() {
let report = reconcile_with(true); let report = reconcile_with(true, Cause::Loss);
assert_eq!(report.mismatches.len(), 1); assert_eq!(report.mismatches.len(), 1);
assert_eq!(report.explained, HashMap::from([(Cause::Loss, 1)])); assert_eq!(report.explained, HashMap::from([(Cause::Loss, 1)]));
assert!(report.unexplained.is_empty()); assert!(report.unexplained.is_empty());
let report = reconcile_with(false); let report = reconcile_with(false, Cause::Loss);
assert_eq!(report.unexplained.len(), 1); assert_eq!(report.unexplained.len(), 1);
assert!(report.explained.is_empty()); assert!(report.explained.is_empty());
} }
/// A market tracked again: its first Snapshot since differs from the one
/// written before it left, and its log line says why.
#[test]
fn a_market_tracked_again_explains_its_first_snapshot() {
let report = reconcile_with(true, Cause::TrackedAgain);
assert_eq!(report.mismatches.len(), 1);
assert_eq!(report.explained, HashMap::from([(Cause::TrackedAgain, 1)]));
assert!(report.unexplained.is_empty());
}
} }
+350
View File
@@ -0,0 +1,350 @@
//! A look at the output as text: one symbol's first Snapshot, the Deltas
//! after it, and the book they fold into. With no symbol given, the one with
//! the most Deltas in the file.
use std::collections::HashMap;
use std::fmt::Write;
use std::path::Path;
use chrono::{DateTime, Utc};
use crate::Side;
use crate::book::{Book, format_fixed};
use crate::util::dtos::challenge_orderbook::{AnyMessage, PriceLevelUpdateDomain, SnapshotDomain};
use crate::util::{AlignedSbeReader, OrderbookSchema};
/// The Deltas shown after the Snapshot.
const DELTAS: usize = 20;
/// The levels shown on each side of a book.
const LEVELS: usize = 5;
/// The text for `symbol` in the output at `path`, or for its busiest symbol.
pub fn show(path: impl AsRef<Path>, symbol: Option<&str>) -> anyhow::Result<String> {
let path = path.as_ref();
let mut counts: HashMap<String, (usize, usize)> = HashMap::new();
each(path, |message| {
let (symbol, is_snapshot) = match message {
Message::Snapshot(snapshot) => (snapshot.symbol, true),
Message::Delta(delta) => (delta.symbol, false),
};
let (snapshots, deltas) = counts.entry(symbol).or_default();
if is_snapshot {
*snapshots += 1;
} else {
*deltas += 1;
}
})?;
let symbol = match symbol {
Some(symbol) if counts.contains_key(symbol) => symbol.to_string(),
Some(symbol) => anyhow::bail!("{symbol} is not in {}", path.display()),
// Ties go to the smallest symbol, so the pick doesn't depend on hashing.
None => counts
.iter()
.max_by(|a, b| a.1.1.cmp(&b.1.1).then(b.0.cmp(a.0)))
.map(|(symbol, _)| symbol.clone())
.ok_or_else(|| anyhow::anyhow!("{} holds no message", path.display()))?,
};
let (snapshots, deltas) = counts[&symbol];
let mut first: Option<SnapshotDomain> = None;
let mut after: Vec<PriceLevelUpdateDomain> = Vec::new();
each(path, |message| match message {
Message::Snapshot(snapshot) if snapshot.symbol == symbol && first.is_none() => {
first = Some(snapshot)
}
Message::Delta(delta)
if delta.symbol == symbol && first.is_some() && after.len() < DELTAS =>
{
after.push(delta)
}
_ => {}
})?;
let first = first.ok_or_else(|| anyhow::anyhow!("{symbol} has no Snapshot"))?;
let mut text = String::new();
writeln!(
text,
"{symbol}: Snapshots {snapshots}, Deltas {deltas}, in {}",
path.display()
)?;
let mut book = Book {
bids: first
.bids
.iter()
.map(|level| (level.price_level, level.quantity))
.collect(),
asks: first
.asks
.iter()
.map(|level| (level.price_level, level.quantity))
.collect(),
};
writeln!(
text,
"\nIts first Snapshot, received {} (exchange time {}):",
time(first.received_at),
time(first.exchange_timestamp)
)?;
ladder(&mut text, &book)?;
writeln!(text, "\nThe {} Deltas after it:", after.len())?;
for delta in &after {
let side = match delta.side {
Side::Buy => "bid",
Side::Sell => "ask",
};
let quantity = match delta.quantity {
0 => "deleted".to_string(),
quantity => format!("now {}", format_fixed(quantity)),
};
writeln!(
text,
" {} {side} {:>8} {quantity}",
time(delta.backtest_timestamp),
format_fixed(delta.price_level)
)?;
book.set(delta.side, delta.price_level, delta.quantity);
}
writeln!(text, "\nThe book they fold into:")?;
ladder(&mut text, &book)?;
Ok(text)
}
enum Message {
Snapshot(SnapshotDomain),
Delta(PriceLevelUpdateDomain),
}
/// Calls `f` with every message of the output at `path`, in order.
fn each(path: &Path, mut f: impl FnMut(Message)) -> anyhow::Result<()> {
let mut reader = AlignedSbeReader::<OrderbookSchema>::open(path)?;
while let Some(message) = reader.read_msg()? {
f(match message {
AnyMessage::Snapshot(decoder) => {
Message::Snapshot(SnapshotDomain::try_from_decoder(decoder)?)
}
AnyMessage::PriceLevelUpdate(decoder) => {
Message::Delta(PriceLevelUpdateDomain::try_from_decoder(decoder)?)
}
_ => anyhow::bail!("unknown message in the output"),
});
}
Ok(())
}
/// The best `LEVELS` asks above the best `LEVELS` bids, prices descending.
fn ladder(text: &mut String, book: &Book) -> std::fmt::Result {
let asks: Vec<_> = book.asks.iter().take(LEVELS).collect();
for (price, quantity) in asks.into_iter().rev() {
writeln!(
text,
" ask {:>8} {}",
format_fixed(*price),
format_fixed(*quantity)
)?;
}
writeln!(text, " ----")?;
for (price, quantity) in book.bids.iter().rev().take(LEVELS) {
writeln!(
text,
" bid {:>8} {}",
format_fixed(*price),
format_fixed(*quantity)
)?;
}
Ok(())
}
fn time(at: DateTime<Utc>) -> String {
at.format("%H:%M:%S%.6f").to_string()
}
#[cfg(test)]
mod tests {
use super::*;
use crate::util::AlignedSbeWriter;
use crate::util::dtos::challenge_orderbook::{
SnapshotAsksEntryDomain, SnapshotBidsEntryDomain,
};
const AT: i64 = 1_794_390_000;
fn at(seconds: i64) -> DateTime<Utc> {
DateTime::from_timestamp(AT + seconds, 0).unwrap()
}
fn snapshot(symbol: &str, seconds: i64) -> SnapshotDomain {
SnapshotDomain {
received_at: at(seconds),
exchange_timestamp: at(seconds),
bids: vec![
SnapshotBidsEntryDomain {
price_level: 400_000,
quantity: 10_000_000,
},
SnapshotBidsEntryDomain {
price_level: 500_000,
quantity: 20_000_000,
},
],
asks: vec![SnapshotAsksEntryDomain {
price_level: 600_000,
quantity: 30_000_000,
}],
symbol: symbol.to_string(),
}
}
fn delta(
symbol: &str,
seconds: i64,
side: Side,
price: i64,
quantity: i64,
) -> PriceLevelUpdateDomain {
PriceLevelUpdateDomain {
backtest_timestamp: at(seconds),
exchange_timestamp: at(seconds),
price_level: price,
quantity,
side,
symbol: symbol.to_string(),
}
}
/// "quiet" has one Delta, "busy" three: a new bid, a deleted bid, an ask
/// changed. `name` keeps parallel tests apart.
fn output(name: &str) -> std::path::PathBuf {
let path = std::env::temp_dir().join(format!("show-{}-{name}.sbe", std::process::id()));
let mut writer = AlignedSbeWriter::create(path.to_str().unwrap(), 64 << 20).unwrap();
writer.write_msg(&snapshot("quiet", 0)).unwrap();
writer.write_msg(&snapshot("busy", 0)).unwrap();
writer
.write_msg(&delta("busy", 1, Side::Buy, 550_000, 5_000_000))
.unwrap();
writer
.write_msg(&delta("quiet", 2, Side::Buy, 400_000, 1_000_000))
.unwrap();
writer
.write_msg(&delta("busy", 3, Side::Buy, 400_000, 0))
.unwrap();
writer
.write_msg(&delta("busy", 4, Side::Sell, 600_000, 7_000_000))
.unwrap();
writer.flush().unwrap();
path
}
#[test]
fn shows_the_busiest_symbol_its_deltas_in_order_and_the_folded_book() {
let path = output("busiest");
let text = show(&path, None).unwrap();
std::fs::remove_file(&path).unwrap();
assert!(
text.starts_with("busy: Snapshots 1, Deltas 3, in"),
"{text}"
);
let deltas = text.split("The 3 Deltas after it:\n").nth(1).unwrap();
let deltas: Vec<_> = deltas.lines().take(3).map(str::trim).collect();
assert_eq!(
deltas,
[
"09:40:01.000000 bid 0.55 now 5",
"09:40:03.000000 bid 0.4 deleted",
"09:40:04.000000 ask 0.6 now 7",
]
);
let folded = text.split("The book they fold into:\n").nth(1).unwrap();
let folded: Vec<_> = folded.lines().map(str::trim).collect();
assert_eq!(
folded,
[
"ask 0.6 7",
"----",
"bid 0.55 5",
"bid 0.5 20"
]
);
}
/// "deep": a Delta before its first Snapshot, a Snapshot of 7 asks and 7
/// bids, a second Snapshot, then 25 Deltas.
#[test]
fn shows_the_first_snapshots_best_levels_and_the_first_20_deltas_after_it() {
let path = std::env::temp_dir().join(format!("show-{}-deep.sbe", std::process::id()));
let mut writer = AlignedSbeWriter::create(path.to_str().unwrap(), 64 << 20).unwrap();
writer
.write_msg(&delta("deep", 0, Side::Buy, 100_000, 1_000_000))
.unwrap();
let mut deep = snapshot("deep", 1);
deep.asks = (1..=7)
.map(|level| SnapshotAsksEntryDomain {
price_level: 500_000 + level * 10_000,
quantity: level * 1_000_000,
})
.collect();
deep.bids = (1..=7)
.map(|level| SnapshotBidsEntryDomain {
price_level: 500_000 - level * 10_000,
quantity: level * 1_000_000,
})
.collect();
writer.write_msg(&deep).unwrap();
writer.write_msg(&snapshot("deep", 2)).unwrap();
for second in 3..28 {
writer
.write_msg(&delta("deep", second, Side::Buy, 100_000, 2_000_000))
.unwrap();
}
writer.flush().unwrap();
let text = show(&path, Some("deep")).unwrap();
std::fs::remove_file(&path).unwrap();
assert!(
text.starts_with("deep: Snapshots 2, Deltas 26, in"),
"{text}"
);
let first = text.split("09:40:01.000000):\n").nth(1).unwrap();
let first: Vec<_> = first.lines().take(11).map(str::trim).collect();
assert_eq!(
first,
[
"ask 0.55 5",
"ask 0.54 4",
"ask 0.53 3",
"ask 0.52 2",
"ask 0.51 1",
"----",
"bid 0.49 1",
"bid 0.48 2",
"bid 0.47 3",
"bid 0.46 4",
"bid 0.45 5",
]
);
let deltas = text.split("The 20 Deltas after it:\n").nth(1).unwrap();
let deltas: Vec<_> = deltas.lines().take_while(|line| !line.is_empty()).collect();
assert_eq!(deltas.len(), 20, "{text}");
assert!(deltas[0].contains("09:40:03.000000"), "{text}");
assert!(deltas[19].contains("09:40:22.000000"), "{text}");
}
#[test]
fn shows_the_symbol_named() {
let path = output("named");
let text = show(&path, Some("quiet")).unwrap();
std::fs::remove_file(&path).unwrap();
assert!(
text.starts_with("quiet: Snapshots 1, Deltas 1, in"),
"{text}"
);
assert!(text.contains("The 1 Deltas after it:"), "{text}");
}
#[test]
fn an_unknown_symbol_is_an_error() {
let path = output("unknown");
let shown = show(&path, Some("absent"));
std::fs::remove_file(&path).unwrap();
assert!(shown.unwrap_err().to_string().contains("absent is not in"));
}
}
+35
View File
@@ -0,0 +1,35 @@
//! The collector's command line, run as a user would.
use std::process::Command;
#[test]
fn help_prints_the_usage_and_exits_0() {
for flag in ["--help", "-h"] {
let output = Command::new(env!("CARGO_BIN_EXE_engineering-challenge"))
.arg(flag)
.output()
.unwrap();
let text = String::from_utf8(output.stdout).unwrap();
assert!(output.status.success(), "{flag}: {text}");
assert!(
text.starts_with("usage: engineering-challenge EXCHANGE"),
"{flag}: {text}"
);
assert!(text.contains("KALSHI_PRIVATE_KEY_FILE"), "{flag}: {text}");
}
}
#[test]
fn an_unknown_exchange_fails_with_the_usage() {
let output = Command::new(env!("CARGO_BIN_EXE_engineering-challenge"))
.arg("kraken")
.output()
.unwrap();
let text = String::from_utf8(output.stderr).unwrap();
assert!(!output.status.success());
assert!(text.contains("unknown exchange \"kraken\""), "{text}");
assert!(
text.contains("usage: engineering-challenge EXCHANGE"),
"{text}"
);
}
+94
View File
@@ -0,0 +1,94 @@
//! The `reconcile` binary's exit status: what `scripts/watch.sh` relies on.
use std::process::Command;
use chrono::DateTime;
use engineering_challenge::Side;
use engineering_challenge::util::AlignedSbeWriter;
use engineering_challenge::util::dtos::challenge_orderbook::{
PriceLevelUpdateDomain, SnapshotBidsEntryDomain, SnapshotDomain,
};
/// An output of a symbol's Snapshot and a Delta, after an orphan Delta of
/// another symbol if `orphan`; then, if `mismatched`, a Snapshot of the
/// symbol that the Delta doesn't lead to.
fn output(orphan: bool, mismatched: bool) -> std::path::PathBuf {
let at = DateTime::from_timestamp(1_794_390_000, 0).unwrap();
let delta = |symbol: &str| PriceLevelUpdateDomain {
backtest_timestamp: at,
exchange_timestamp: at,
price_level: 500_000,
quantity: 2_000_000,
side: Side::Buy,
symbol: symbol.to_string(),
};
let path = std::env::temp_dir().join(format!(
"reconcile-exit-{}-{orphan}-{mismatched}.sbe",
std::process::id()
));
let mut writer = AlignedSbeWriter::create(path.to_str().unwrap(), 64 << 20).unwrap();
if orphan {
writer.write_msg(&delta("2")).unwrap();
}
writer
.write_msg(&SnapshotDomain {
received_at: at,
exchange_timestamp: at,
bids: vec![SnapshotBidsEntryDomain {
price_level: 500_000,
quantity: 1_000_000,
}],
asks: Vec::new(),
symbol: "1".to_string(),
})
.unwrap();
writer.write_msg(&delta("1")).unwrap();
if mismatched {
writer
.write_msg(&SnapshotDomain {
received_at: at,
exchange_timestamp: at,
bids: vec![SnapshotBidsEntryDomain {
price_level: 500_000,
quantity: 3_000_000,
}],
asks: Vec::new(),
symbol: "1".to_string(),
})
.unwrap();
}
writer.flush().unwrap();
path
}
#[test]
fn reconcile_fails_on_an_orphan_delta_or_an_unexplained_mismatch_and_passes_otherwise() {
let status = |orphan: bool, mismatched: bool| {
let path = output(orphan, mismatched);
let output = Command::new(env!("CARGO_BIN_EXE_reconcile"))
.arg(&path)
.output()
.unwrap();
std::fs::remove_file(&path).unwrap();
(
output.status.success(),
String::from_utf8(output.stdout).unwrap(),
)
};
let (passed, text) = status(false, false);
assert!(passed, "{text}");
assert_eq!(
text,
"1 snapshots, 1 deltas, 0 orphan deltas, 0 mismatches\n"
);
let (passed, text) = status(true, false);
assert!(!passed, "{text}");
assert!(text.contains("1 orphan deltas"), "{text}");
// A Snapshot no log line explains (no log given): it fails too.
let (passed, text) = status(false, true);
assert!(!passed, "{text}");
assert!(
text.contains("1 mismatches") && text.contains("unexplained: 1 at "),
"{text}"
);
}