diff --git a/.gitignore b/.gitignore index 4d26375..0c0e780 100644 --- a/.gitignore +++ b/.gitignore @@ -1,2 +1,3 @@ /target/ __pycache__/ +watch/ diff --git a/CLAUDE.md b/CLAUDE.md index 38d4fe2..5d415eb 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -23,8 +23,8 @@ outside this repo. Delta, lateness, lag, speed). - `src/core/`: what is the same for every exchange (ADR 0004). `orchestrator.rs`: what is tracked, which connections carry each market, - their balancing and their recycling (hourly, and sooner for a Kalshi - connection that stays among the slowest); `socket.rs`: a connection (read, + their balancing and their recycling (hourly, a Kalshi connection that + stays among the slowest taking the next turn); `socket.rs`: a connection (read, stamp, decode, route; reconnects when dropped, silent, lost a message or recycled); `feed.rs`: a book thread's shell (Snapshots, moves, the quiet watch) and the `Rule` trait; `mod.rs`: the `Exchange`, `Session`, `Market` @@ -37,8 +37,14 @@ outside this repo. rank rule, `mod.rs` the exchange and its session (the `seq` check). - `src/book.rs`: the book and fixed point; `src/reconcile.rs` + `src/bin/reconcile.rs`: the output check, reading the log's cause lines - (`src/cause.rs`). `tests/fixtures/`: recordings of the live feeds that the + (`src/cause.rs`); `src/show.rs` + `src/bin/show.rs`: one symbol of the + output as text. `tests/fixtures/`: recordings of the live feeds that the tests replay. +- `scripts/watch.sh [EXCHANGE] [MINUTES]`: a run to watch, also the final + runs' and the README's quick start (roadmap/08): the collector under + `timeout -s INT`, a live view of its log (`scripts/watch.awk`, which + parses the minute lines' wording: change one, check the other), then + `reconcile` and `show`. Output and log in `watch/`. - `src/latency.rs` + `scripts/latency/analyse.py`: the latency probe (`engineering-challenge latency EXCHANGE OUT.csv`, until SIGINT: one CSV line per copy of a change on each of 16 connections) and its analysis @@ -67,6 +73,10 @@ and let the user settle it. Claude owns the data model. - Use `GLOSSARY.md`'s terms: a **Snapshot** is only the output message; getting a fresh book is a **resync**. - A corner cut for speed carries a comment naming its ceiling and upgrade path. +- A figure quoted in a doc (README, `LATENCY.md`, a sprint's Outcome) comes + from the tool that checks it, `reconcile`'s output or the log itself, + never from `watch.sh`'s view or a count grouped by hand: sprint 08's + first README figures had four errors from those. ## Environment traps @@ -91,7 +101,20 @@ and let the user settle it. Claude owns the data model. (`pgrep -f '^./target/release/...'`, anchored) and loop on `kill -0 $pid`. - **No `/usr/bin/time` on the server**: for a run's peak memory and thread count, read `VmHWM` and `Threads` from `/proc//status` while it runs - (sprint 05's run script sampled them every 30 s). + (`scripts/watch.sh` samples them every 10 s). +- **mawk buffers a pipe's input** unless `-W interactive`: a live view + through `awk` shows nothing until the end (`scripts/watch.sh` passes it + when the awk is mawk, as on the server). +- **A job started with `&` in `sh` ignores Ctrl-C**: a script that runs the + collector in the background must pass SIGINT on itself (`watch.sh`'s + trap). To try Ctrl-C without a terminal: `(sleep N; printf '\003') | + script -qfec "CMD" /dev/null`. +- **Polymarket's Gamma answers 429 to runs started close together**: two + collectors' first scans at once (or a few runs within minutes) got every + page refused; the scan retries, but leave a few minutes between runs. +- **`*.pem` is gitignored globally** (`~/.gitignore`): a test key goes in + the source, as `src/kalshi/sign.rs` does with RFC 8032's and RFC 9500's + published test keys. - **The server is shared and gets overloaded** (load up to 92 on 12 cores, 2026-11-11): a starved collector falls behind and gets closed with 1013. Check `cat /proc/loadavg` before a live run whose numbers are compared, and @@ -99,7 +122,8 @@ and let the user settle it. Claude owns the data model. - **Kalshi needs a key; nothing else does**: `KALSHI_KEY_ID_FILE` and `KALSHI_PRIVATE_KEY_FILE` name its two files (on the server `/run/secrets/kalshi-demo-key-id` and `kalshi-demo-private-key`; never read - or print them). Opening ~100 connections at once gets some handshakes + or print them). The key is Ed25519 or RSA (RSA-PSS-SHA256), told apart + by the parsed key. Opening ~100 connections at once gets some handshakes refused with 429; each retries after 1 s and gets in. - **cargo keeps a build whose source got an older mtime**: a script that restores a file from a copy (`shutil.move`, `cp -p`) must `touch` it, or diff --git a/Cargo.toml b/Cargo.toml index cdcd9b0..1c4b4a1 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -3,7 +3,7 @@ name = "engineering-challenge" version = "0.1.0" edition = "2024" build = "src/util/build.rs" -# `cargo run` runs the collector, not `src/bin/reconcile.rs`. +# `cargo run` runs the collector, not `src/bin/reconcile.rs` or `show.rs`. default-run = "engineering-challenge" [dependencies] @@ -14,10 +14,11 @@ chrono = "0.4" crossbeam = "0.8.4" # `termination`: SIGTERM and SIGHUP stop the collector like SIGINT. ctrlc = { version = "3", features = ["termination"] } +# Signs Kalshi's handshake (Ed25519 or RSA-PSS); already rustls's crypto +# provider. +ring = "0.17" # Not used directly: tungstenite leaves rustls's crypto provider to the app, and # rustls panics at runtime unless exactly one (here ring) is enabled. -# Signs Kalshi's handshake (Ed25519); already rustls's crypto provider. -ring = "0.17" rustls = { version = "0.23", default-features = false, features = ["ring"] } # Reads the PEM of Kalshi's key. rustls-pki-types = { version = "1", features = ["std"] } diff --git a/GLOSSARY.md b/GLOSSARY.md index 146f366..3f2fdee 100644 --- a/GLOSSARY.md +++ b/GLOSSARY.md @@ -48,8 +48,14 @@ _Avoid_: Order book state, depth **Connection**: One websocket to the exchange. Every live symbol is subscribed on two connections, chosen independently of each other. -_Avoid_: Socket (for the concept; fine for the code), stream, lane (the -retired pairing of one connection with one book thread) +_Avoid_: Socket (for the concept; fine for the code), stream (that is one +connection's changes of a symbol), lane (the retired pairing of one +connection with one book thread) + +**Stream**: +One connection's changes of one symbol, in the order it delivers them. The +two streams of a symbol carry the same changes, each at its own pace. +_Avoid_: Feed (the book thread's code), using it for the connection **Load**: How much a market weighs on a connection: the bytes per second of its frames diff --git a/LATENCY.md b/LATENCY.md index 0116688..366c091 100644 --- a/LATENCY.md +++ b/LATENCY.md @@ -186,8 +186,9 @@ as the median lag), but at the median it wins 9 ms, a tenth of the lag. - **On Kalshi's demo it is worth tens of milliseconds**, in the tail too (p99 28% and 42% lower with the fastest pair kept). This met the threshold agreed before the runs, so the collector now recycles a Kalshi - connection that stays slow for 3 minutes running, ahead of its hourly - turn: its replacement's speed is drawn afresh. Each duplicate tells how + connection that stays slow for 3 minutes running at the hourly + recycle's next turn (one every 30 s or so), in place of the oldest: its + replacement's speed is drawn afresh. Each duplicate tells how late its connection was against the other's copy (`src/kalshi/feed.rs`); once a minute the orchestrator ranks the connections by their mean lateness, and one is slow if it is in the slowest quarter and more than @@ -197,9 +198,16 @@ as the median lag), but at the median it wins 9 ms, a tenth of the lag. quarter always exists: a first version, by median alone, recycled 6 connections in 20 minutes, one a minute, connections no slower than the rest. **In 20 minutes of the collector on the demo (270,000 markets on - 120 connections, load 5.7 at the start), no connection was slow by that - rule**, so none was recycled for it; the hourly recycle went on (33), and - the output reconciled (25 mismatches, each a logged loss). The probe's 16 + 120 connections, load 5.7 at the start), none was recycled for it**; the + hourly recycle went on (33), and the output reconciled (25 mismatches, + each a logged loss). That run doesn't say whether one was judged slow: a + later review found the slow recycle waited for the same moment as the + hourly one, which nearly always took it first, and only the recycle was + logged. Now the slow one takes the hourly recycle's turn, and the + judgment is logged on its own. **In 20 minutes on the demo after that + fix** (260,865 markets on 116 connections, 2026-11-12), two connections + were judged slow and recycled: one at a mean lateness of 246 ms, one at + 80 ms. The probe's 16 connections, on the 100 busiest markets, differed far more than the collector's; assumed to be because each of the collector's shares its markets with many others and is judged against all their copies, which diff --git a/README.md b/README.md index 4dab97f..c3f5217 100644 --- a/README.md +++ b/README.md @@ -1,21 +1,214 @@ -# Read before getting started!! +# Order-book collector for Polymarket and Kalshi -First, read [src/main.rs](https://github.com/voluntas-io/engineering-challenge/blob/main/src/main.rs). Then, some background on this repo: +My answer to the Voluntas engineering challenge. The collector subscribes +to every live market of one exchange (Polymarket, or Kalshi), keeps an order +book for each symbol, and writes the template's SBE messages: a Delta for +every change, and a Snapshot of every book each minute and whenever a book +is replaced. All four extensions are done: redundant connections, +subscription monitoring, one core for both exchanges, and latency +experiments ([LATENCY.md](LATENCY.md)). -This repo uses the [FIX SBE standard](https://fixtrading.org/packages/simple-binary-encoding-technical-specification-v2-rc2/) -for writing binary messages. The rust code abstracts this away for you, and you can simply use the helpers, as demonstrated in `main.rs`. +Built with Claude Code. I set the architecture (the threading model, +redundancy, the exchange seam) and settled every design decision; Claude +proposed the options, wrote the code and ran the experiments. Each +decision and its reasons are in [`roadmap/`](roadmap) and +[`docs/adr/`](docs/adr). -### More background, in case you are curious! +## Quick start -FIX SBE is defined in XML, which I find extremely cumbersome to edit, especially -when updating schemas frequently. To solve this, I have also introduced some -helper code that takes our own custom YAML format, and turns it into FIX SBE -XML, which in turn is used to generate rust code to encode and decode the -messages. **You don't need to edit any of these files**. This explanation is -simply to help you understand the origins of the various files in this repo. +On Linux (the script reads `/proc`), with Rust; built and tested with +1.98: -If you're curious to see the YAML schema, take a look at [challenge_orderbook.yaml](https://github.com/voluntas-io/engineering-challenge/blob/main/sbe_schemas/engineering_challenge/challenge_orderbook.yaml). -We also use a custom timestamp field, so that the Rust struct can use -`chrono::DateTime` in the generated struct, and in SBE schema, it is a uint64 -representing the timestamp as the number of microseconds since unix epoch. +```sh +scripts/watch.sh # Polymarket for 3 minutes, no key needed +scripts/watch.sh polymarket 10 # or longer +``` +It builds the collector, runs it, and shows what it does each minute: +markets and connections; changes written; duplicates (a change's second +copy, from the other connection carrying its symbol); rescued changes (one +connection missed them, the other delivered them); resyncs (a fresh book +asked for, when ours can't be checked); losses (a fresh book that differed +from ours: changes neither connection delivered). The output and the full +log go to `watch/`. Ctrl-C stops it +early and cleanly. Then it checks the output against the log (`reconcile`) +and prints one busy symbol's book as text (`show`). The first scan lists +every market before subscribing: a minute or two. + +## Running it + +```sh +cargo build --release +./target/release/engineering-challenge polymarket > out.sbe 2> run.log +./target/release/engineering-challenge polymarket out.sbe 2> run.log # same +``` + +The output goes to stdout unless a path is given; the log goes to stderr. +SIGINT, SIGTERM or SIGHUP closes the websockets and flushes the output. +Only a clean stop flushes it: the writer fills 64 MiB chunks, the only size +the template's reader opens. + +Kalshi needs an API key. Two variables name its files (paths keep the key +out of the shell's history): + +```sh +export KALSHI_KEY_ID_FILE=~/kalshi/key-id +export KALSHI_PRIVATE_KEY_FILE=~/kalshi/private-key.pem # Ed25519 or RSA +./target/release/engineering-challenge kalshi out.sbe 2> run.log +./target/release/engineering-challenge kalshi-demo out.sbe 2> run.log +``` + +I had a key for Kalshi's demo only, so production has never been run; the +code is the same, only the URLs differ. + +## Checking the output + +```sh +./target/release/reconcile out.sbe run.log +./target/release/show out.sbe [SYMBOL] +``` + +`reconcile` replays the output: per symbol, the Deltas since a Snapshot, +applied to it, must give the next Snapshot. A Snapshot that differs is +paired with the log line that says why it was written: a loss; a +subscription gap (the few changes a subscribe keeps from us, which the +exchange's opening book already holds); a switch (Polymarket's book now +followed on the other connection); a market tracked again after it left. +It fails on any left unexplained, or on a Delta before its symbol's first +Snapshot. + +## Final runs + +`scripts/watch.sh`, from the submitted code, on a shared 12-core server. + +**Polymarket, 10 minutes** (2026-11-12, system load 7.2 at the start): the +first scan listed 199,531 live markets in 77 s, subscribed on 178 +connections. `reconcile`: 14.4 million Deltas and 3.7 million Snapshots; +the 142,990 Snapshots that differ from the book before them each explained +by their log line. 95% of the changes came on both connections, and +417,852 were rescued. Peak memory 2.3 GB. The thread count wasn't sampled +at its peak in this run; it is one per connection plus about 20 (the Kalshi +run's 137 for 116), so about 200. + +The run met an outage of about two minutes. Between 16:50:17 and 16:50:25 +UTC, 177 of the 178 connections were found silent (a connection is silent +after 30 s with nothing received, so since about 16:49:47). Connects then +timed out or found no route to the host, and between 16:51:37 and 16:51:59 +every connection came back on its own. Each book that changed meanwhile was found +wrong on the reconnect and replaced: 138,950 of the run's 140,129 losses +are those two minutes, 1,179 the other eight. Whether the network or +Polymarket went down, the log can't say. + +**Kalshi's demo, 20 minutes** (system load 0.8 at the start): the first +scan took 5 min 17 s, the demo's listing being slow (about 1.2 s a page of +1,000 markets in that run); then 260,865 markets on 116 connections. +`reconcile`: 4.1 million Deltas and 4.0 million Snapshots; the 12 that +differ each a logged loss. 98% of the changes came on both connections; +1,228 resyncs. Peak memory 830 MB, 137 threads. Every connection is +recycled (closed and opened again) about once an hour, one at a time; on +Kalshi one that stays slow takes the next turn, and that acted: a +connection's lateness is how long after the other connection's copy it +delivers a change, and connections 99 (246 ms on average) and 57 (80 ms) +were each among the slowest 3 minutes running, so each was recycled at +the next turn, 13 and 21 s later. + +## Design + +``` + rescan fetcher ──findings──▶ ┌──────────────┐ ◀──findings── recent fetcher + │ orchestrator │ ◀── up, down, 1013 ─── connections + └──────┬───▲───┘ ──── recycle ────────▶ connections + Add / Move / Remove (a market and │ │ loads, quiet markets, + its connections), to its thread │ │ lateness, each minute + ┌──────▼───┴───┐ + connection 0..N ──events───▶ │ book threads │ ──Snapshot | Delta──▶ writer + (read, stamp, parse, route) │ 0..15 │ + ▲ └──────┬───────┘ + └── subscribe / unsubscribe / resync ┘ +``` + +Plain threads and channels, nothing shared ([ADR 0001](docs/adr/0001-concurrency-model.md)). +A connection stamps the receive time right after the socket read, then +parses and routes each event to the book thread that owns its market. Each +book is owned by one of 16 book threads; one writer owns the output. The +orchestrator decides which connections carry each market, balances the +bytes per second each carries, and recycles them ([ADR 0002](docs/adr/0002-lanes-and-orchestrator.md)). +[docs/project.md](docs/project.md) has the whole design. + +A lost change leaves a book silently wrong, so every sign of one replaces +the book with a fresh one from the exchange (a resync). On Polymarket the +exchange's book hash is recomputed after every change; on Kalshi it is a +skipped sequence number on a connection. A fresh book that differs from +ours counts as a loss; some of those are only books ahead of our changes, +so the count is an upper bound. The protocol facts the +code relies on, several found only in live data, are in +[docs/notes.md](docs/notes.md). + +## The extensions + +- **Redundant websockets.** Every symbol is on two connections chosen + independently; its book takes each change from whichever delivers it + first ([ADR 0003](docs/adr/0003-first-one-wins.md)). There is no "source" + to switch, so the brief's Snapshot on a switch becomes a Snapshot + whenever a fresh book from either connection replaces ours. A connection + the exchange closes twice within 10 minutes as too slow (code 1013, "try + again later") has half its markets moved to others. +- **Subscription monitoring.** A subscription is quiet when it delivered + nothing for a minute while the other connection delivered at least 3 + changes of the same symbol: it is resubscribed, and moved if quiet + again. Judged against the other connection, not a rate, it holds for + the quietest markets too. Every + connection is recycled about once an hour, one at a time, never two of a + market's; one silent for 30 s reconnects. +- **Generic core.** `src/core/` is the same for both exchanges. An exchange + supplies its websocket codec and its loss rule + ([ADR 0004](docs/adr/0004-exchange-seam.md)): the rule is more than + decoding, since telling a duplicate from a loss depends on what the + exchange sends (a book hash, or a sequence number and a clock). +- **Latency.** [LATENCY.md](LATENCY.md): a connection that delivers + changes sooner than the others keeps doing so, on both exchanges. + Keeping the fastest gains under a millisecond on Polymarket, and tens of + milliseconds on Kalshi's demo, where the collector now recycles a + connection that stays slow. + +## Limits + +- A hard crash (SIGKILL, power) loses the chunk being filled, up to 64 MiB. +- When a symbol is subscribed, the exchange's opening book holds a few + changes it never sends as changes: logged as a subscription gap. +- Polymarket changes held for ordering when the run stops (at most 1 s) are + not written. +- Channels are unbounded: a burst never blocks a socket or drops an event, + but a book thread or writer that can't keep up grows memory. +- Kalshi production was never run. + +## Next + +From the roadmap's possible features ([docs/roadmap.md](docs/roadmap.md)): +telling a real loss from a fresh book merely ahead of the changes we have, +so the loss count measures data missed; Polymarket's `new_market` event as +a faster discovery signal; fewer Snapshots for unchanged books (the brief's +every minute is about 130 MB a minute at full coverage); a Kalshi +production run. + +## Reading the code + +- [docs/project.md](docs/project.md): what it is, the central bet (the + pipeline), the design. +- [docs/adr/](docs/adr): the four decisions that are hard to reverse. +- [docs/notes.md](docs/notes.md): each exchange's rules, as found. +- [LATENCY.md](LATENCY.md): extension 4. +- [GLOSSARY.md](GLOSSARY.md): the terms (a Snapshot is only the output + message; getting a fresh book is a resync). +- [roadmap/](roadmap): each sprint's plan, decisions and outcome, in order. +- [CLAUDE.md](CLAUDE.md): the instructions the coding agent worked under. + +## Output format + +The output uses the template's helpers unchanged: `AlignedSbeWriter` +writes the SBE messages of +[sbe_schemas/engineering_challenge/challenge_orderbook.yaml](sbe_schemas/engineering_challenge/challenge_orderbook.yaml) +(`Snapshot` and `PriceLevelUpdate`, prices and quantities in fixed point, +timestamps in microseconds since the epoch), readable by the template's +`AlignedSbeReader`. The template's code is `src/util/` and `sbe_schemas/`; +everything else in `src/` is the collector. diff --git a/docs/notes.md b/docs/notes.md index 710d341..e6f4c83 100644 --- a/docs/notes.md +++ b/docs/notes.md @@ -237,9 +237,14 @@ is the demo's. `api.elections.kalshi.com` and `demo-api.kalshi.co` still answer. - **Auth:** the websocket handshake carries `KALSHI-ACCESS-KEY` (the key id), `KALSHI-ACCESS-TIMESTAMP` (ms) and `KALSHI-ACCESS-SIGNATURE`: base64 of the - Ed25519 signature of `GET/trade-api/ws/v2`. Without them: 401. - The market listing needs no key. Keys are created per environment - (Account → Profile → API Keys); the private key is a PKCS#8 PEM shown once. + signature of `GET/trade-api/ws/v2`. Without them: 401. The + key is Ed25519 (RFC 8032 over that text) or 2048-bit RSA (RSA-PSS, + SHA-256, MGF1 SHA-256, a salt of the digest's length); Kalshi picks the + check from the public key it holds, and a PEM's header doesn't say which + (an RSA key may be PKCS#1, as Kalshi gives it, or PKCS#8; Kalshi's API + keys page, read 2026-11-12). The market listing needs no key. Keys are + created per environment (Account → Profile → API Keys); the private key + is shown once. - **Subscribe:** `{"id":1,"cmd":"subscribe","params":{"channels": ["orderbook_delta"],"market_tickers":[...]}}` is answered `{"type":"subscribed","id":1,"msg":{"channel":"orderbook_delta","sid":1}}`, diff --git a/docs/project.md b/docs/project.md index ef6cb99..9babb0c 100644 --- a/docs/project.md +++ b/docs/project.md @@ -161,11 +161,11 @@ schema debate). As settled in sprints 04 and 05: - The **orchestrator** owns what is tracked (each market's two connections, the timers) and decides: placing, moving, splitting, and recycling one - connection at a time, about hourly, and sooner a Kalshi connection whose - mean lateness stays above twice everyone's (sprint 07, LATENCY.md). It - never blocks on HTTP. + connection at a time, about hourly; a Kalshi connection whose mean + lateness stays above twice everyone's takes the next turn, in place of + the oldest (sprints 07 and 08, LATENCY.md). It never blocks on HTTP. - The two **fetchers** only page the exchange's market listing on request - and stream what they read. + and send on what they read. - A **connection** reads, stamps the receive time, parses, and routes each event to the book thread of its market. Every symbol is on two connections, chosen independently. One silent for 30 s reconnects. diff --git a/docs/roadmap.md b/docs/roadmap.md index 1656330..facf38b 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -91,11 +91,17 @@ abstraction is *extracted* once a second exchange exists, not designed up front. The source port can't be chosen here (a NAT rewrites it); first after maintenance wasn't tried. -8. **Polish + submission** — *not started* - Final review, README for the solution (the template's README still says the - helpers are "demonstrated in `main.rs`", whose demo M1 replaced), - packaging/handoff for the challenge. +8. **Polish + submission** — *done (2026-11-12)* + A review of the whole codebase and its fixes, a run anyone can watch + (`scripts/watch.sh`, `show`), the README, final runs of both + exchanges, and the handoff as a public repo on git.pband.ch. + Sprint: [../roadmap/08. ready to submit.md](../roadmap/08.%20ready%20to%20submit.md). *Ends:* ready to submit. + Result: Kalshi takes RSA keys too; the slow recycle, which the hourly + one nearly always pre-empted, now takes its turn and acted twice in 20 + live minutes; a market tracked again no longer fails `reconcile`; 19 + surviving mutants got their tests. Both final runs reconciled, one + through an outage of about two minutes (sprint 08's Outcome). ## Why this order @@ -155,3 +161,12 @@ abstraction is *extracted* once a second exchange exists, not designed up front. - Judging the connections' speed at the book threads' wall-clock minute, not on the orchestrator's own timer (a judgment near the crossing sees partial samples; sprint 07's review). +- Kalshi: a resync asked from a connection the symbol has since left stays + asked there while the other connection is down (sprint 08's review; rare, + and the resync count grows each minute). +- Paging Kalshi's listing in parallel: on 2026-11-12 the demo served a page + of 1,000 markets in 1.7 s, so the first scan took 7.5 minutes (under one + in sprint 06). +- Connecting to each of the exchange's addresses in turn, not only the + first the lookup returns (a host with IPv6 DNS but no IPv6 route would + never connect; sprint 08's review). diff --git a/roadmap/08. ready to submit.md b/roadmap/08. ready to submit.md new file mode 100644 index 0000000..a3cd230 --- /dev/null +++ b/roadmap/08. ready to submit.md @@ -0,0 +1,425 @@ +# 08. Ready to submit + +**Status: done (2026-11-12)** + +M8, the last milestone: the collector handed to Voluntas in a state a +reviewer can read, run and check on their own. A review of the whole +codebase (not one sprint's diff) and its fixes; one command that runs the +collector and shows what it does; a README for the solution in place of +the template's; a final live run of each exchange, reconciled, whose +figures the README quotes; and the handoff as a public repo. + +## The watched run + +`scripts/watch.sh [EXCHANGE] [MINUTES]`, defaults `polymarket` (no key) and +3 minutes (two minute summaries; Polymarket tracks about 200,000 markets +within its first minute, `../runs/poly10.log`). It builds release, runs the +collector under `timeout -s INT`, and writes the output and the full log +to a gitignored `watch/`. On the terminal, a curated view of the log: + +- each minute, the orchestrator's line as is, and the 16 book threads' + lines summed into one (changes written, duplicates, rescued, resyncs, + losses, subscription gaps; their counts are cumulative, "so far"); +- between them, the notable events only: a connection lost, recycled + (hourly or slow), split, silent; the startup's connections as one count, + not a line each. The Snapshot cause lines (3,300 of 3,952 lines in a + 10-minute Polymarket run) stay in the file. + +Ctrl-C stops the run early and cleanly: the script goes on to the checks +(how Ctrl-C reaches `timeout` and the collector from a script is +**assumed**; a quick try without a terminal didn't deliver the signal, so +the first task tries it in one). Then it runs `reconcile` on the output +and the log, and `show` on the output. + +On the workspace: `rc shell voluntas`, then `cd voluntas/repo && +scripts/watch.sh` (`rc shell`'s interactive use is assumed from +`rc shell exec`). Kalshi needs `KALSHI_KEY_ID_FILE` and +`KALSHI_PRIVATE_KEY_FILE` exported; without them the collector stops +before connecting, naming the variable (`src/kalshi/sign.rs:26`). The +script adds nothing for keys. + +## show + +`src/bin/show.rs`: `show OUTPUT [SYMBOL]`, reading through the template's +`AlignedSbeReader` as `src/reconcile.rs` does. With no symbol, the one +with the most Deltas in the file. It prints that symbol's first Snapshot +(top 5 levels a side, its receive time), its next 20 Deltas (receive time, +side, price, new quantity; 0 shown as deleted), then the top 5 levels a +side of the book folded from them, in exchange units through +`book::format_fixed` (`src/book.rs:46`). A Polymarket symbol is its token +id: the output holds no market name. An unknown symbol is an error. + +## The review + +The whole codebase, not a diff: `src/`, `scripts/`, the tests, by five +`reviewer` subagents: + +- **brief**, in place of a spec reviewer: the brief checked line by line + against the code (every live symbol, MVEs excluded on Kalshi, new and + gone symbols, Deltas to stdout and a Snapshot each minute, SIGINT closes + the websockets and the file, each extension); +- **smells**, read as a stranger: leftovers of superseded designs (lanes, + the template's demo), comments that need sprint history (19 citations + of `roadmap/0N` or "sprint 0N" in `src/`), the two exchanges + inconsistent with each other; +- **bugs**, **tests**, **security**, as usual; sprint 07's 11 post-review + fixes, never reviewed a second time, included. + +Findings are sorted with the user: fix now (a task below, with its test), +or a limit or a "next" in the README. + +## The README + +Replaces the template's, for the Voluntas reader, in this order: what the +collector does, with the line on how it was built (below); quick start +(`scripts/watch.sh`); build and run per exchange, Kalshi's two `export` +lines, and that Kalshi ran only against the demo; checking the output +(`reconcile` with the log); the final runs' figures; the design in one +diagram (`docs/project.md`'s pipeline); each extension and where it is +met (ADRs, `LATENCY.md`); the limits (the crash ceiling, the subscription +gap when a symbol opens, Kalshi production never run); next (a short +pick of the roadmap's possible features); what to read first +(`docs/project.md`, `docs/adr/`, `LATENCY.md`, `docs/notes.md`, and +`GLOSSARY.md`, `CLAUDE.md`) and that `roadmap/` holds each sprint's plan +and outcome; an "Output format" section (the template's +`AlignedSbeWriter` unchanged, 64 MiB chunks, the schema's path). The +order as built supersedes the first one (the runs after the extensions, +the line on how it was built last): the runs read as evidence for the +design that follows, and the line belongs with what the project is. The +line: + +> Built with Claude Code. I set the architecture (the threading model, +> redundancy, the exchange seam) and settled every design decision; +> Claude proposed the options, wrote the code and ran the experiments. +> Each decision and its reasons are in `roadmap/` and `docs/adr/`. + +## Decisions + +- **The watched run is a script over the existing log**, not pipeline + code. Rejected: a status line in the collector (counts across threads, + new pipeline code); a TUI. +- **The data is shown, not only counted** (`show`), after the run: the + writer emits 64 MiB chunks, so a live stream of the output would come + in bursts. +- **One script for everything**: the final runs and the README's quick + start are `scripts/watch.sh`, so the grader runs what made the + README's figures. The workspace's `../runs/run.sh` is retired. +- **Everything ships unchanged** (`docs/`, `roadmap/`, `CLAUDE.md`, + `GLOSSARY.md`, `LATENCY.md`); the README says what to read first. + Rejected: stripping the process files; the work being AI-assisted is + said openly instead. +- **Sprint citations in comments**: a file path stays (`roadmap/` ships); + a reference that leans on history is rewritten to say what the code + does, or to cite the ADR or `docs/notes.md`. +- **The final runs** pass `reconcile` (every mismatch explained, no + orphan Delta), load and peak memory recorded (`VmHWM` from `/proc`); + no comparison with earlier sprints' runs (done in their sprints; the + load differs from day to day). +- **The template's README text**: its stale lines go ("read `main.rs`", + "as demonstrated in `main.rs`"); the rest becomes "Output format". +- **Delivery: a public repo on the user's Gitea (git.pband.ch), not on + GitHub.** The brief says nothing on how (it links only the template, + public, 25 forks). Public, so the link opens with no account; on + GitHub it would publish a solution to the challenge where other + candidates search, which the user doesn't want. Rejected: a GitHub + fork (listed under the template), a GitHub repo of our own (found by + search), a private repo (an invite to a handle not known), a PR to the + template (public). Full history, the template's 3 commits first, so a + diff against it shows what was added. Pushed over SSH (which reaches + Gitea from here), with the user's go. Supersedes "a public repo of the + user's own on GitHub", at the user's request during the build. +- **Kalshi's RSA keys are taken** (from the review): RSA-PSS-SHA256 with + `ring`, PKCS#1 or PKCS#8, told from Ed25519 by the parsed key; tested + on RFC 9500's published test key (no private key of our own in a public + repo; `*.pem` is gitignored globally); checked + against `openssl`. Supersedes sprint 06's "Ed25519 only", because a + grader's key from Kalshi may well be RSA. +- **A slow connection takes the hourly recycle's next turn** (from the + review), in place of being recycled at once: the slow recycle waited for + the same moment as the hourly one, which nearly always took it first. + One path; the mark is logged when made. Supersedes sprint 07's "recycled + at once, ahead of the hourly order". `LATENCY.md`'s "no connection was + slow" in 20 live minutes rested on the starved count: reworded. +- **The final runs, as they went.** Polymarket's 10 minutes met an outage + of about two minutes (177 of 178 connections found silent at 16:50:17, + so since about 16:49:47; back from 16:51:38 to 16:51:59): kept in the + README, as it shows the recovery; 138,950 of its 140,129 losses (the + log's and `reconcile`'s) are those two minutes. Kalshi's 10-minute run spent 7.5 + of them on the first scan (the demo's listing at 1.7 s a page) and saw + 74 normal closes by the demo, never seen before; an A/B of `main`'s + build against the branch's, 10 minutes each, saw none on either, so + the closes were the demo's. The README's Kalshi run is 20 minutes, + long enough for the slow judgment (from a connection's 10th minute): + it recycled two. Supersedes "kalshi-demo 10". Logs in `../runs/`. +- **ShellCheck in CI**: `uvx --from shellcheck-py==0.10.0.1 shellcheck` + over `scripts/*.sh` (tried: 0.10.0, `scripts/ci.sh` clean). + +## Tasks + +- [x] Ctrl-C through the script, tried in a terminal: `timeout -s INT` + and the collector stop cleanly and the script goes on to the + checks; the script's handling settled from what happens. +- [x] `scripts/watch.sh`: build, bounded run, `watch/` (gitignored), the + curated view, then `reconcile` and `show`. +- [x] The view's summed book-thread line checked once by hand against + the 16 raw lines of the same minute. +- [x] ShellCheck in `scripts/ci.sh`. +- [x] `src/bin/show.rs`. +- [x] Test (`show`): on an output written by `AlignedSbeWriter` in the + test, the symbol with the most Deltas is picked; a symbol named is + used; its Deltas come in order and the folded book is right; an + unknown symbol is an error. +- [x] The review: five reviewers over the whole codebase; findings + sorted with the user, the fix-now ones added here as tasks. +- [x] The citations in comments the review lists, rewritten. +- [x] A short run, stopped with Ctrl-C partway: the output flushes and + reconciles. +- [x] The final runs, `scripts/watch.sh polymarket 10` and + `scripts/watch.sh kalshi-demo 20`, from the branch with every fix + in, load checked before each; both reconcile; load and peak + memory recorded. +- [x] `README.md`, its figures from those runs. +- [x] `docs/roadmap.md` (M8 done) and `CLAUDE.md` (watch, show) brought + up to date. +- [x] The stale branches `feat/full-coverage` and + `feat/redundant-connections` deleted. +- [ ] After the squash into `main`, with the user's go: the public repo + created on git.pband.ch and pushed, and its link opened logged + out. + +From the review (2026-11-12, at `73a1551`): + +- [x] Kalshi's RSA keys (PKCS#1 PEM, as Kalshi generates them), signed + RSA-PSS-SHA256 with `ring` (the scheme assumed until checked in + Kalshi's docs); Ed25519 kept. Test: an RSA signature verifies (with + `ring`; checked once by hand with `openssl`); a key of neither kind + is refused. +- [x] Usage names the two `KALSHI_*` variables; `--help` and `-h` print + it; the missing-key error names both variables. +- [x] The first scan is logged: when it starts, and every 10 s the + markets listed so far; `watch.awk` shows them. +- [x] A slow connection takes the hourly recycle's next turn, in place of + the oldest: one recycle path. Test: with the hourly recycle running, + a connection slow 3 minutes is the next recycled. +- [x] A symbol tracked again after its market was removed: its first + Snapshot logs a cause, tracked again, which `reconcile` reads. Test: + remove, add, the Snapshot reconciles with the log. +- [x] `watch.sh` exits non-zero when the collector or `reconcile` fails. +- [x] `book::add` checks for overflow; an overflow counts as a loss and + resyncs. Test. +- [x] Test: `orchestrator::run` offline against the local listing and + websocket servers; on stop, each socket gets a Close and the output + reconciles. +- [x] Tests: a command sent on an open connection reaches the server; a + 1013 close becomes a split. +- [x] Tests for the review's surviving mutants: Kalshi's rule + (`src/kalshi/feed.rs` 94, 114, 215, 285), the shell + (`src/core/feed.rs` 483, 539, 589, 683, 687), `show` (several asks, + the 20-Delta cap, the first ladder), `reconcile` (orphan Deltas, one + cause line explains one Snapshot), the socket (a dead book thread + stops it, the PING). +- [x] Test: a sample log through `watch.awk`, from `scripts/ci.sh`. +- [x] `a_silent_connection_reconnects_with_its_tokens` fails instead of + hanging (a timeout on the server's accept). +- [x] "ponytail:" becomes "Ceiling:" (4 places); "lag" for lateness and + "stream" reworded (or "stream" in `GLOSSARY.md`); the 4 history + citations rewritten; `Cargo.toml`'s comments, the `CHUNK` + constants' names, `impl RecentChange`'s place, "the server" in + `watch.sh`. + +From the review of `26b4d67`: + +- [x] `watch.sh` records the peak thread count, not the last sample. +- [x] README: Polymarket's threads (one per connection plus about 20, + derived from the Kalshi run: 137 = 116 + 21, labelled so); losses + 140,129 (the log's and `reconcile`'s), the same in this file; the + first reconnect at 16:51:38; 177 of the 178 connections; the outage + "about two minutes" (silence is noticed after 30 s), the roadmap + too; Kalshi's scan at about 1.2 s a page; "every sign of a loss + resyncs, and the count includes some that weren't"; "system load"; + each term defined at its first use; `reconcile`'s Delta count only; + the unwrapped line. +- [x] This file: the RSA task says the `openssl` check was by hand; the + README section's order follows the README; "Done when" names the + 20-minute Kalshi run. +- [x] Test: a "tracked again" line explains a mismatch in `reconcile`. +- [x] Test: an overflowing Kalshi change asks a resync and removes the + level. +- [x] The view prints a minute's sum on the first line 2 s or more after + its first book line, too (a book thread with nothing logs none). + Test: a minute missing one thread is printed before the + orchestrator's line. +- [x] "Next to recycle" is logged when the mark is new only; the slow log + text in one place. Test. +- [x] Test: the `reconcile` binary fails on an unexplained mismatch. +- [x] Test: a mark is cleared when a later judgment finds the connection + no longer slow. +- [x] Test: `--help` prints the usage and exits 0. +- [x] The two timing-dependent tests poll until their condition, under a + timeout, in place of a fixed sleep. +- [x] `gamma.rs`'s "stream what they read"; `orchestrator.rs:60`'s + unwrapped doc comment; the Kalshi test's hand-built `Market` uses + `market()`. + +From the review of `d94f683`: + +- [x] The view's window is 10 s, to the millisecond and across midnight: + a minute's book-thread lines spread 1.6 s in the final Polymarket + run, and the 2 s rule, in whole seconds, could split its totals. + Tests: a thread 1.6 s late, live and final; across midnight. +- [x] "Next to recycle" said once per connection while it stays slow, + even when two marked connections swap ranks. Test. +- [x] README: silent between 16:50:17 and 16:50:25; back from 16:51:37; + recycling explained before the slow recycle; a line wrapped. +- [x] The `reconcile` binary's test matches `unexplained: 1 at `, not the + symbol alone. + +Each test is seen to fail once (the code it protects removed, then put +back). + +## Review of `d94f683` (2026-11-12) + +The 7 commits after the second review's tasks, one reviewer on the five +questions. Fix now: the four tasks above. No: the test of an unknown +exchange (unasked, kept: cheap and useful); the 1013 test's copy of +`client` (changing `client` touches every socket test); `output(orphan, +mismatched)`'s two flags (a test helper). Assumed, written so in +`watch.awk` before: "they come within milliseconds", which the final +run's log contradicted. + +## Review of `26b4d67` (2026-11-12) + +The diff since `73a1551`, five reviewers; none of the findings dropped. +Fix now: the tasks above. Otherwise: + +- **Spec:** no: the Polymarket run predating the view's fix (`src/` + unchanged since; the sums recomputed on the log); "sprint 08's Outcome" + (written at the close); the README's longer reading list. +- **Tests:** no: `watch.sh`'s exit status (needs the live exchange or a + fake collector; tried by hand); "neither kind" with an EC key (garbage + covers the refusal); the first-scan lines typed in `test_watch.sh` + (`CLAUDE.md` says change both). +- **Smells:** no: `Symbol::again` carrying the exchange's name + (documented); `64 << 20` in the library's tests (the binary's constant + isn't visible there); `run_tests` serving Kalshi locally (real sockets). +- **Security:** no vulnerability. + +## Review of `73a1551` (2026-11-12) + +The whole codebase, five reviewers (brief, bugs, tests, smells, +security); the tests reviewer ran 40 mutants, 19 survived. 2 findings +dropped (files since committed; a 429 that recovered). Fix now: the +tasks above. Otherwise: + +- **Brief:** quiet judged against the twin connection, a `Rule` beyond + encode/decode, first one wins in place of a source switch: the README + explains each with its ADR. No: `/dev/stdout` truncating a `>>`. +- **Bugs:** later (`docs/roadmap.md`): a Kalshi resync left asked on a + connection the symbol left; only the first resolved address tried. No: + Polymarket changes held at stop, at most 1 s (a README limit); the + probe's exit 0 when every connection drops. +- **Smells:** no: the two fetchers, `Rule::Symbol`, the socket's data + clumps, `on_book`'s length, the exchanges' decoding and construction. +- **Tests, as built:** every new test seen red on its mutant, but one: + removing `drop(outputs)` at the top of `orchestrator::run` changes + nothing a test can see (`outputs` drops when `run` returns, which is + after `threads.join()`); kept, as it says when the writer may end. +- **Security:** no secret in the tree or history. No: unbounded channels + (never dropping is the design; a README limit), raw exchange text in + the log, the template's `unsafe-libyaml`. Kept, the user's call: the + commit email and `CLAUDE.md`'s DNS lines in the public history. + +## Not in this sprint + +The trading simulator of `docs/evaluation.md` (its fill model is a project +of its own; reconcile, lateness and rescued counts already measure the +claims); the roadmap's possible features (the README's "next"); +containerized packaging; a Kalshi production run (no key). + +## Done when + +- The review's findings are sorted; the fix-now ones are fixed and + tested. +- `scripts/watch.sh` and `show` work; Ctrl-C was tried. +- Both final runs (Polymarket 10 minutes, Kalshi's demo 20) reconcile, + load and memory recorded. +- `README.md` is written, its figures from those runs. +- The stale branches are deleted. +- `scripts/ci.sh` is green, ShellCheck included. + +After the squash: + +- The public repo is created on git.pband.ch and pushed, with the user's + go, and its link works logged out. + +## Outcome + +The collector is ready to hand over. `scripts/watch.sh` runs it with a live +view, then `reconcile` and `show` (one symbol's book as text); the README +replaces the template's; three reviews (the whole codebase, then each round +of fixes) and their fixes are in; both final runs reconciled. + +- **Fixed, from the whole-codebase review:** Kalshi takes RSA keys too + (RSA-PSS-SHA256, PKCS#1 or PKCS#8, told by the parsed key); the slow + recycle, which the hourly one nearly always took first, now marks the + connection and takes the next turn, and acted twice in the 20-minute + Kalshi run; a market tracked again says so, so `reconcile` doesn't fail + on it; `Book::add` checks for overflow; usage, `--help` and the + missing-key error name both key variables; the first scan says how far + it got. 19 surviving mutants got their tests, and an offline run of the + whole orchestrator proves SIGINT closes every socket and the output + reconciles. +- **Final runs:** Polymarket, 10 minutes, 199,531 markets on 178 + connections, 14.4 million Deltas, through an outage of about two minutes + that it recovered from on its own (138,950 of its 140,129 losses). + Kalshi's demo, 20 minutes, 260,865 markets on 116 connections, 4.1 + million Deltas, 12 losses, two slow connections recycled. + +**Diverged from the plan**, each written into this file when it +happened: delivery on the user's Gitea, not GitHub (the user's call, so as +not to publish a solution where candidates search); RSA keys taken +(sprint 06 had Ed25519 only), tested on RFC 9500's published key since +`*.pem` is ignored globally; the slow recycle takes the hourly turn, not +recycled at once; the Kalshi final run is 20 minutes, not 10 (the demo's +listing took 5 to 7.5 minutes to page); the README's sections in another +order; `LATENCY.md`'s "no connection was slow" withdrawn, since it rested +on the starved count. + +**Left out**, and where it went: a Kalshi resync left asked on a +connection the symbol left; connecting to every resolved address; +paging Kalshi's listing in parallel (`docs/roadmap.md`, possible +features). Unexplained, recorded: the Polymarket outage's cause, and the +74 normal closes by Kalshi's demo in one run (an A/B ruled out our code). + +**Assumed**, not shown: that the outage was on the network's side ("no +route to host"); that RSA keys work against Kalshi itself (tried offline +only, against `openssl`); `rc shell`'s interactive use; Polymarket's +thread count, about 200, derived from the Kalshi run's. + +**Reviewed** at `73a1551` (the whole codebase), `26b4d67` and `d94f683`; +the 3 commits after `d94f683` are its fixes, each with a test seen to +fail, and had no further review. + +### Retrospective + +- **Went well:** reviewing the whole codebase, not the sprint's diff, found + a starved mechanism that had made a sprint 07 claim false; a sprint diff + never would have. Rerunning one key mutant of each subagent's tests, and + an A/B against `main`'s build, kept their claims and the demo's closes + honest. The user's watchable run turned out to be the best test of the + view: each live run found something. +- **Went wrong:** my own instruments misled me four times: `head` cut the + mutation output so a red test looked green, `$?` after a pipe reported + the pipe's last command, the view summed part of a minute, and a count + grouped by 10 s gave a reconnect time 8 s early. The README's first + figures came from the view and from memory, not from `reconcile` and the + log, and two reviews corrected them. The view's 2 s window rested on + "they come within milliseconds", never measured. Three review rounds + where one careful round of fixes would have done. +- **Tested enough?** The fixes are, by removal. Not: `watch.sh`'s exit + status and its sampling (live only), RSA against Kalshi, the "tracked + again" log line itself (stderr), the first-scan lines' wording against + the view (typed in the test). +- **Next time:** take every quoted figure from the tool that checks it + (`reconcile`, the log), never from a summary of it; check an exit status + without a pipe; measure a timing before building a rule on it. diff --git a/scripts/ci.sh b/scripts/ci.sh index 8740764..4d6fff8 100755 --- a/scripts/ci.sh +++ b/scripts/ci.sh @@ -13,3 +13,8 @@ cargo test uvx ruff@0.15.0 format --check scripts/latency uvx ruff@0.15.0 check scripts/latency python3 -m unittest discover -s scripts/latency + +# The shell scripts: ShellCheck pinned, run through uvx as ruff is. +uvx --from shellcheck-py==0.10.0.1 shellcheck scripts/*.sh +# The view of watch.sh, on a sample log. +sh scripts/test_watch.sh diff --git a/scripts/test_watch.sh b/scripts/test_watch.sh new file mode 100755 index 0000000..63d1f9a --- /dev/null +++ b/scripts/test_watch.sh @@ -0,0 +1,157 @@ +#!/bin/sh +# Pipes a sample log of each exchange through scripts/watch.awk, live and +# with -v final=1, and compares what it prints with what the view must show; +# exits non-zero on a difference. The lines are worded as the collector +# writes them: the book threads' minute lines (log_minute in +# src/polymarket/feed.rs and src/kalshi/feed.rs), the orchestrator's +# (src/core/orchestrator.rs), the connections' (src/core/socket.rs) and a +# Snapshot's cause (src/cause.rs). +set -eu +cd "$(dirname "$0")/.." + +failed=0 +# check NAME LOG EXPECTED [AWK OPTION...] +check() { + name=$1 log=$2 expected=$3 + shift 3 + actual=$(printf '%s\n' "$log" | awk -v book_threads=2 "$@" -f scripts/watch.awk) + if [ "$actual" != "$expected" ]; then + printf 'watch.awk, %s: printed\n%s\nexpected\n%s\n' "$name" "$actual" "$expected" + failed=1 + fi +} + +polymarket=$( + cat <<'EOF' +12:00:00.100 polymarket: orchestrator: first scan, listing the exchange's markets +12:00:30.200 polymarket: orchestrator: first scan done, 40 live markets, subscribing +12:00:30.300 polymarket: connection 0: connected, 40 symbols +12:00:30.310 polymarket: connection 1: connected, 40 symbols +12:01:00.001 polymarket: book thread 0: 10 symbols; so far 100 changes written (90 verified), 50 duplicates, 2 rescued, 0 held dropped, 1 chains broken, 3 resyncs, 0 full books behind ours, 1 subscription gaps, 2 losses, 0 switches, 0 quiet subscriptions resubscribed, 0 markets reported quiet again, 4 reconnect books judged (1 losses); duplicates' lateness 1.5 ms at the median, 9.0 ms at p99; this minute 150 changes received, at most 4 events waited +12:01:00.002 polymarket: book thread 1: 30 symbols; so far 300 changes written (290 verified), 150 duplicates, 4 rescued, 0 held dropped, 0 chains broken, 1 resyncs, 0 full books behind ours, 0 subscription gaps, 1 losses, 0 switches, 0 quiet subscriptions resubscribed, 0 markets reported quiet again, 0 reconnect books judged (0 losses); duplicates' lateness 2.0 ms at the median, 8.0 ms at p99; this minute 450 changes received, at most 9 events waited +12:01:00.050 polymarket: Snapshot of 123 at 1794390060050000: loss +12:01:30.400 polymarket: orchestrator: 40 markets on 2 connections; 40 added, 0 removed, 0 moved (0 quiet), 0 splits, 0 recycled (0 slow), 0 silent this minute; busiest connection at 1200 bytes/s +12:02:00.001 polymarket: book thread 0: 10 symbols; so far 160 changes written (150 verified), 80 duplicates, 3 rescued, 0 held dropped, 1 chains broken, 3 resyncs, 0 full books behind ours, 1 subscription gaps, 2 losses, 0 switches, 1 quiet subscriptions resubscribed, 0 markets reported quiet again, 4 reconnect books judged (1 losses); duplicates' lateness 1.5 ms at the median, 9.0 ms at p99; this minute 90 changes received, at most 2 events waited +12:02:00.002 polymarket: book thread 1: 30 symbols; so far 500 changes written (480 verified), 250 duplicates, 6 rescued, 0 held dropped, 0 chains broken, 2 resyncs, 0 full books behind ours, 0 subscription gaps, 1 losses, 0 switches, 0 quiet subscriptions resubscribed, 0 markets reported quiet again, 0 reconnect books judged (0 losses); duplicates' lateness 2.0 ms at the median, 8.0 ms at p99; this minute 300 changes received, at most 5 events waited +12:02:10.000 polymarket: connection 1: recycled +12:02:11.000 polymarket: connection 1: connected, 40 symbols +12:02:30.400 polymarket: orchestrator: 40 markets on 2 connections; 0 added, 0 removed, 0 moved (0 quiet), 0 splits, 1 recycled (0 slow), 0 silent this minute; busiest connection at 1300 bytes/s +EOF +) + +check "Polymarket, live" "$polymarket" "$( + cat <<'EOF' +12:00:00 polymarket: orchestrator: first scan, listing the exchange's markets +12:00:30 polymarket: orchestrator: first scan done, 40 live markets, subscribing +12:00:30 connections opening (counted in the minute line) +12:01 books (2 threads): 40 symbols, 400 changes written, 200 duplicates, 6 rescued, 4 resyncs, 1 subscription gaps, 3 losses +12:01:30 40 markets on 2 connections; 40 added, 0 removed, 0 moved (0 quiet), 0 splits, 0 recycled (0 slow), 0 silent this minute; busiest connection at 1200 bytes/s; 2 connected (0 refused, retried), 1 Snapshots with a cause +12:02 books (2 threads): 40 symbols, 660 changes written, 330 duplicates, 9 rescued, 5 resyncs, 1 subscription gaps, 3 losses +12:02:10 polymarket: connection 1: recycled +12:02:30 40 markets on 2 connections; 0 added, 0 removed, 0 moved (0 quiet), 0 splits, 1 recycled (0 slow), 0 silent this minute; busiest connection at 1300 bytes/s; 1 connected (0 refused, retried), 0 Snapshots with a cause +EOF +)" +check "Polymarket, final" "$polymarket" \ + "12:02 books (2 threads): 40 symbols, 660 changes written, 330 duplicates, 9 rescued, 5 resyncs, 1 subscription gaps, 3 losses" \ + -v final=1 + +# A minute with one of its two book threads' lines: printed with the first +# line 10 s after it, before the orchestrator's. +missing=$( + cat <<'EOF' +12:01:00.001 polymarket: book thread 0: 10 symbols; so far 100 changes written (90 verified), 50 duplicates, 2 rescued, 0 held dropped, 1 chains broken, 3 resyncs, 0 full books behind ours, 1 subscription gaps, 2 losses, 0 switches, 0 quiet subscriptions resubscribed, 0 markets reported quiet again, 4 reconnect books judged (1 losses); duplicates' lateness 1.5 ms at the median, 9.0 ms at p99; this minute 150 changes received, at most 4 events waited +12:01:00.500 polymarket: connection 1: recycled +12:01:30.400 polymarket: orchestrator: 40 markets on 2 connections; 0 added, 0 removed, 0 moved (0 quiet), 0 splits, 1 recycled (0 slow), 0 silent this minute; busiest connection at 1200 bytes/s +EOF +) +check "a book thread missing" "$missing" "$( + cat <<'EOF' +12:01:00 polymarket: connection 1: recycled +12:01 books (1 threads): 10 symbols, 100 changes written, 50 duplicates, 2 rescued, 3 resyncs, 1 subscription gaps, 2 losses +12:01:30 40 markets on 2 connections; 0 added, 0 removed, 0 moved (0 quiet), 0 splits, 1 recycled (0 slow), 0 silent this minute; busiest connection at 1200 bytes/s; 0 connected (0 refused, retried), 0 Snapshots with a cause +EOF +)" + +# A thread's line 1.6 s after the first, past a line between them: still +# the same minute's, summed whole. +late=$( + cat <<'EOF' +12:01:00.075 polymarket: book thread 0: 10 symbols; so far 100 changes written (90 verified), 50 duplicates, 2 rescued, 0 held dropped, 1 chains broken, 3 resyncs, 0 full books behind ours, 1 subscription gaps, 2 losses, 0 switches, 0 quiet subscriptions resubscribed, 0 markets reported quiet again, 4 reconnect books judged (1 losses); duplicates' lateness 1.5 ms at the median, 9.0 ms at p99; this minute 150 changes received, at most 4 events waited +12:01:01.600 polymarket: connection 1: recycled +12:01:01.672 polymarket: book thread 1: 30 symbols; so far 300 changes written (290 verified), 150 duplicates, 4 rescued, 0 held dropped, 0 chains broken, 1 resyncs, 0 full books behind ours, 0 subscription gaps, 1 losses, 0 switches, 0 quiet subscriptions resubscribed, 0 markets reported quiet again, 0 reconnect books judged (0 losses); duplicates' lateness 2.0 ms at the median, 8.0 ms at p99; this minute 450 changes received, at most 9 events waited +EOF +) +check "a thread 1.6 s late" "$late" "$( + cat <<'EOF' +12:01:01 polymarket: connection 1: recycled +12:01 books (2 threads): 40 symbols, 400 changes written, 200 duplicates, 6 rescued, 4 resyncs, 1 subscription gaps, 3 losses +EOF +)" +check "a thread 1.6 s late, final" "$late" \ + "12:01 books (2 threads): 40 symbols, 400 changes written, 200 duplicates, 6 rescued, 4 resyncs, 1 subscription gaps, 3 losses" \ + -v final=1 + +# Across midnight: a minute begun at 23:59 still ends 10 s on. +midnight=$( + cat <<'EOF' +23:59:00.001 polymarket: book thread 0: 10 symbols; so far 100 changes written (90 verified), 50 duplicates, 2 rescued, 0 held dropped, 1 chains broken, 3 resyncs, 0 full books behind ours, 1 subscription gaps, 2 losses, 0 switches, 0 quiet subscriptions resubscribed, 0 markets reported quiet again, 4 reconnect books judged (1 losses); duplicates' lateness 1.5 ms at the median, 9.0 ms at p99; this minute 150 changes received, at most 4 events waited +00:00:30.000 polymarket: connection 1: recycled +EOF +) +check "across midnight" "$midnight" "$( + cat <<'EOF' +23:59 books (1 threads): 10 symbols, 100 changes written, 50 duplicates, 2 rescued, 3 resyncs, 1 subscription gaps, 2 losses +00:00:30 polymarket: connection 1: recycled +EOF +)" + +# A line between a minute's book threads: the minute is still summed whole. +interleaved=$( + cat <<'EOF' +12:01:00.001 polymarket: book thread 0: 10 symbols; so far 100 changes written (90 verified), 50 duplicates, 2 rescued, 0 held dropped, 1 chains broken, 3 resyncs, 0 full books behind ours, 1 subscription gaps, 2 losses, 0 switches, 0 quiet subscriptions resubscribed, 0 markets reported quiet again, 4 reconnect books judged (1 losses); duplicates' lateness 1.5 ms at the median, 9.0 ms at p99; this minute 150 changes received, at most 4 events waited +12:01:00.002 polymarket: connection 1: recycled +12:01:00.003 polymarket: book thread 1: 30 symbols; so far 300 changes written (290 verified), 150 duplicates, 4 rescued, 0 held dropped, 0 chains broken, 1 resyncs, 0 full books behind ours, 0 subscription gaps, 1 losses, 0 switches, 0 quiet subscriptions resubscribed, 0 markets reported quiet again, 0 reconnect books judged (0 losses); duplicates' lateness 2.0 ms at the median, 8.0 ms at p99; this minute 450 changes received, at most 9 events waited +EOF +) +check "a line between book threads" "$interleaved" "$( + cat <<'EOF' +12:01:00 polymarket: connection 1: recycled +12:01 books (2 threads): 40 symbols, 400 changes written, 200 duplicates, 6 rescued, 4 resyncs, 1 subscription gaps, 3 losses +EOF +)" + +kalshi=$( + cat <<'EOF' +12:00:00.100 kalshi: orchestrator: first scan, listing the exchange's markets +12:00:10.100 kalshi: orchestrator: first scan, 1000 live markets listed so far +12:00:20.200 kalshi: orchestrator: first scan done, 1500 live markets, subscribing +12:00:20.300 kalshi: connection 0: connected, 1500 symbols +12:00:20.310 kalshi: connection 1: connect failed: handshake refused: 429 Too Many Requests +12:00:21.320 kalshi: connection 1: connected, 1500 symbols +12:01:00.001 kalshi: book thread 0: 700 symbols; so far 100 changes written, 90 duplicates, 1 before their connection's book, 0 ahead of the book, 3 fresh books (1 losses), 0 below zero, 2 resyncs, 0 quiet subscriptions resubscribed, 0 markets reported quiet again; this minute 190 changes received, at most 3 events waited +12:01:00.002 kalshi: book thread 1: 800 symbols; so far 200 changes written, 190 duplicates, 0 before their connection's book, 1 ahead of the book, 2 fresh books (0 losses), 1 below zero, 1 resyncs, 0 quiet subscriptions resubscribed, 0 markets reported quiet again; this minute 390 changes received, at most 6 events waited +12:01:00.030 kalshi: Snapshot of KXA at 1794390060030000: loss (its clock 12, ours 11) +12:01:20.400 kalshi: orchestrator: 1500 markets on 2 connections; 1500 added, 0 removed, 0 moved (0 quiet), 0 splits, 0 recycled (0 slow), 0 silent this minute; busiest connection at 9000 bytes/s +12:02:00.001 kalshi: book thread 0: 700 symbols; so far 150 changes written, 140 duplicates, 1 before their connection's book, 0 ahead of the book, 3 fresh books (1 losses), 0 below zero, 2 resyncs, 0 quiet subscriptions resubscribed, 0 markets reported quiet again; this minute 100 changes received, at most 2 events waited +12:02:00.002 kalshi: book thread 1: 800 symbols; so far 260 changes written, 250 duplicates, 0 before their connection's book, 1 ahead of the book, 2 fresh books (0 losses), 1 below zero, 1 resyncs, 0 quiet subscriptions resubscribed, 0 markets reported quiet again; this minute 120 changes received, at most 4 events waited +12:02:20.400 kalshi: orchestrator: 1500 markets on 2 connections; 0 added, 0 removed, 0 moved (0 quiet), 0 splits, 0 recycled (0 slow), 0 silent this minute; busiest connection at 8000 bytes/s +EOF +) + +check "Kalshi, live" "$kalshi" "$( + cat <<'EOF' +12:00:00 kalshi: orchestrator: first scan, listing the exchange's markets +12:00:10 kalshi: orchestrator: first scan, 1000 live markets listed so far +12:00:20 kalshi: orchestrator: first scan done, 1500 live markets, subscribing +12:00:20 connections opening (counted in the minute line) +12:01 books (2 threads): 1500 symbols, 300 changes written, 280 duplicates, 3 resyncs, 1 losses +12:01:20 1500 markets on 2 connections; 1500 added, 0 removed, 0 moved (0 quiet), 0 splits, 0 recycled (0 slow), 0 silent this minute; busiest connection at 9000 bytes/s; 2 connected (1 refused, retried), 1 Snapshots with a cause +12:02 books (2 threads): 1500 symbols, 410 changes written, 390 duplicates, 3 resyncs, 1 losses +12:02:20 1500 markets on 2 connections; 0 added, 0 removed, 0 moved (0 quiet), 0 splits, 0 recycled (0 slow), 0 silent this minute; busiest connection at 8000 bytes/s; 0 connected (0 refused, retried), 0 Snapshots with a cause +EOF +)" +check "Kalshi, final" "$kalshi" \ + "12:02 books (2 threads): 1500 symbols, 410 changes written, 390 duplicates, 3 resyncs, 1 losses" \ + -v final=1 + +exit "$failed" diff --git a/scripts/watch.awk b/scripts/watch.awk new file mode 100644 index 0000000..8d0a621 --- /dev/null +++ b/scripts/watch.awk @@ -0,0 +1,107 @@ +# The view of scripts/watch.sh: reads the collector's log and prints, each +# minute, the orchestrator's line and the book threads' lines summed into one, +# and between them only the notable events. The Snapshot cause lines and the +# connections coming up are counted, not shown. +# +# With -v final=1 it prints nothing live, only the last minute's sums: the +# run's totals, since the book threads count "so far". A minute's sum is +# printed once all its book threads' lines are in (-v book_threads=N, 16 by +# default, src/core/mod.rs's BOOK_THREADS), or with the first other line 10 s +# or more after the minute's first, or when the next minute begins. A book +# thread that received nothing logs no line; the others' come within a few +# seconds of the minute (1.6 s apart at most in a 10-minute Polymarket run, +# 2026-11-12), and other lines often fall between them. + +# The book threads' counters shown, in this order; a name missing from an +# exchange's line (Kalshi has no "rescued") is left out. +BEGIN { + if (book_threads == "") book_threads = 16 + shown = "symbols|changes written|duplicates|rescued|resyncs|subscription gaps|losses" + count = split(shown, names, "|") +} + +function flush( line, i) { + if (threads == 0) return + line = minute " books (" threads " threads):" + for (i = 1; i <= count; i++) + if (names[i] in sum) line = line " " sum[names[i]] " " names[i] "," + sub(/,$/, "", line) + last = line + event(line) + threads = 0 + split("", sum) +} + +# Printed at once: the view is read while the run goes on, and mawk +# buffers its output otherwise. +function event(text) { + if (final) return + print text + fflush() +} + +{ + time = substr($1, 1, 8) + text = substr($0, index($0, " ") + 1) + # To the millisecond: $1 is HH:MM:SS.mmm. + seconds = substr($1, 1, 2) * 3600 + substr($1, 4, 2) * 60 + substr($1, 7) +} + +/: book thread [0-9]+: / { + this = substr(time, 1, 5) + if (threads > 0 && this != minute) flush() + if (threads == 0) first_at = seconds + minute = this + threads++ + body = text + sub(/; this minute .*/, "", body) + split("", seen) + # Each "N name": a count and its name, up to the next comma, semicolon + # or parenthesis. A name is summed once a line (its first occurrence). + while (match(body, /[0-9]+ [a-z][a-z' ]*[a-z]/)) { + pair = substr(body, RSTART, RLENGTH) + body = substr(body, RSTART + RLENGTH) + n = substr(pair, 1, index(pair, " ") - 1) + name = substr(pair, index(pair, " ") + 1) + sub(/^so far /, "", name) + if (name in seen) continue + seen[name] = 1 + sum[name] += n + } + if (threads == book_threads) flush() + next +} + +# A minute with a book thread's line missing: printed once it is clearly over +# (a day's seconds added back across midnight). +threads > 0 && (seconds - first_at + 86400) % 86400 >= 10 { flush() } + +/: orchestrator: [0-9]+ markets on/ { + line = text + sub(/^[a-z-]+: orchestrator: /, "", line) + event(time " " line "; " up + 0 " connected (" refused + 0 " refused, retried), " snapshots + 0 " Snapshots with a cause") + up = 0 + refused = 0 + snapshots = 0 + next +} + +/: connection [0-9]+: connected, / { + if (!opened) event(time " connections opening (counted in the minute line)") + opened = 1 + up++ + next +} +# Kalshi refuses some of ~100 handshakes opened at once (429); each retries. +/: connect failed: handshake refused: 429/ { refused++; next } +/: Snapshot of / { snapshots++; next } +/ changes go unchecked$| tick size changed, resync$| unreadable / { next } + +# Everything else is an event worth seeing: a connection lost, recycled, +# split, moved off when quiet, a refused handshake, a failed fetch, the end. +{ event(time " " text) } + +END { + flush() + if (final && last != "") print last +} diff --git a/scripts/watch.sh b/scripts/watch.sh new file mode 100755 index 0000000..beb22a4 --- /dev/null +++ b/scripts/watch.sh @@ -0,0 +1,101 @@ +#!/bin/sh +# watch.sh [EXCHANGE] [MINUTES]: runs the collector and shows what it does. +# EXCHANGE is polymarket (the default, no key needed), kalshi or kalshi-demo +# (KALSHI_KEY_ID_FILE and KALSHI_PRIVATE_KEY_FILE exported); MINUTES +# defaults to 3. Ctrl-C stops the run early, cleanly. +# +# The output and the full log go to watch/; the terminal shows, each minute, +# the orchestrator's line and the book threads' summed (scripts/watch.awk), +# and the notable events between. Then the output is checked against the +# log (reconcile) and one symbol is shown as text (show). It exits non-zero +# when the collector failed or the output doesn't reconcile. +set -eu +cd "$(dirname "$0")/.." +exchange=${1:-polymarket} +minutes=${2:-3} +case $exchange in +polymarket | kalshi | kalshi-demo) ;; +*) + echo "usage: scripts/watch.sh [polymarket|kalshi|kalshi-demo] [MINUTES]" >&2 + exit 2 + ;; +esac + +echo "building (cargo build --release)" +cargo build --release --quiet +mkdir -p watch +out=watch/$exchange.sbe log=watch/$exchange.log memory=watch/$exchange.memory +rm -f "$out" "$log" "$memory" +: >"$log" + +echo "load before the run: $(cut -d' ' -f1-3 /proc/loadavg) (1, 5, 15 min)" +echo "collecting $exchange for $minutes minutes into $out (log: $log); Ctrl-C stops it" +echo "the markets are listed first (a minute or two), then subscribed" +timeout -s INT "${minutes}m" ./target/release/engineering-challenge "$exchange" "$out" 2>>"$log" & +run=$! +# The view ends by itself once the run is over and the log read to its end. +# mawk reads a pipe a buffer at a time unless interactive: the view would +# show nothing until the end. +live= +if awk -W version 2>/dev/null | grep -q mawk; then live='-W interactive'; fi +# shellcheck disable=SC2086 # $live is empty or two words, both wanted. +tail -n +1 -F --pid="$run" "$log" 2>/dev/null | awk $live -f scripts/watch.awk & +view=$! +# A job started with & ignores Ctrl-C (sh without job control), so the +# script passes it on: timeout hands SIGINT to the collector, which stops +# cleanly, and the script carries on to the checks. The view, also a job, +# keeps showing the end. +trap 'echo; echo "stopping (Ctrl-C)"; kill -INT "$run" 2>/dev/null || true' INT + +# The collector's peak memory (VmHWM, the kernel's own peak) and most +# threads, sampled every 10 s while it runs, from /proc: /usr/bin/time isn't +# installed everywhere. The last sample may fall in the shutdown, when the +# connections' threads are gone: the most seen is kept. +( + sleep 1 + pid=$(pgrep -P "$run" || true) + most=0 + while [ -n "$pid" ] && kill -0 "$pid" 2>/dev/null; do + proc=$(cat "/proc/$pid/status" 2>/dev/null || true) + peak=$(printf '%s\n' "$proc" | awk '/^VmHWM:/ {print $2}') + threads=$(printf '%s\n' "$proc" | awk '/^Threads:/ {print $2}') + if [ -n "$threads" ] && [ "$threads" -gt "$most" ]; then most=$threads; fi + if [ -n "$peak" ]; then echo "$peak kB, at most $most threads" >"$memory"; fi + sleep 10 + done +) & + +# wait returns early when Ctrl-C interrupts it: wait again until the +# collector has flushed and exited. +status=0 +while kill -0 "$run" 2>/dev/null; do + wait "$run" && status=0 || status=$? +done +trap - INT +wait "$view" || true +# 124: stopped by timeout, at the end or on Ctrl-C (passed on by it). +failed=0 +case $status in +0 | 124 | 130) ;; +*) + echo "the collector failed (exit $status); its log: $log" + failed=1 + ;; +esac + +if [ ! -e "$out" ]; then + echo "no output was written" + exit 1 +fi +echo +echo "== totals at the last minute" +awk -v final=1 -f scripts/watch.awk "$log" +if [ -s "$memory" ]; then echo "peak memory $(cat "$memory")"; fi +echo +echo "== the output against the log (reconcile)" +./target/release/reconcile "$out" "$log" || failed=1 +echo +echo "== one symbol, as text (show)" +# An output stopped before any book is empty: said, not a failure. +./target/release/show "$out" || true +exit "$failed" diff --git a/src/bin/show.rs b/src/bin/show.rs new file mode 100644 index 0000000..83f3c79 --- /dev/null +++ b/src/bin/show.rs @@ -0,0 +1,15 @@ +//! `show OUTPUT [SYMBOL]`: one symbol's first Snapshot, the Deltas after it +//! and the book they fold into, as text (`src/show.rs`). + +fn main() -> anyhow::Result<()> { + let mut args = std::env::args().skip(1); + let path = args + .next() + .ok_or_else(|| anyhow::anyhow!("usage: show OUTPUT [SYMBOL]"))?; + let symbol = args.next(); + print!( + "{}", + engineering_challenge::show::show(path, symbol.as_deref())? + ); + Ok(()) +} diff --git a/src/book.rs b/src/book.rs index 100348b..6213f6b 100644 --- a/src/book.rs +++ b/src/book.rs @@ -78,15 +78,21 @@ impl Book { } /// Moves a level's quantity by `delta`, which can be negative, and returns - /// the sum. A level can't hold less than nothing: a negative sum removes - /// it, and tells the caller the book was wrong. - pub fn add(&mut self, side: Side, price: i64, delta: i64) -> i64 { + /// the sum. A level can't hold less than nothing, nor more than an `i64`: + /// either sum removes it and returns `None`, which tells the caller the + /// book was wrong. + pub fn add(&mut self, side: Side, price: i64, delta: i64) -> Option { let levels = match side { Side::Buy => &self.bids, Side::Sell => &self.asks, }; - let quantity = levels.get(&price).copied().unwrap_or(0) + delta; - self.set(side, price, quantity.max(0)); + let quantity = levels + .get(&price) + .copied() + .unwrap_or(0) + .checked_add(delta) + .filter(|&quantity| quantity >= 0); + self.set(side, price, quantity.unwrap_or(0)); quantity } @@ -173,12 +179,21 @@ mod tests { #[test] fn adding_moves_a_level_and_never_leaves_it_below_zero() { let mut book = Book::default(); - assert_eq!(book.add(Side::Sell, 600_000, 5), 5); - assert_eq!(book.add(Side::Sell, 600_000, -2), 3); + assert_eq!(book.add(Side::Sell, 600_000, 5), Some(5)); + assert_eq!(book.add(Side::Sell, 600_000, -2), Some(3)); assert_eq!(book.asks, BTreeMap::from([(600_000, 3)])); - assert_eq!(book.add(Side::Sell, 600_000, -3), 0); + assert_eq!(book.add(Side::Sell, 600_000, -3), Some(0)); assert!(book.asks.is_empty()); - assert_eq!(book.add(Side::Buy, 400_000, -1), -1); + assert_eq!(book.add(Side::Buy, 400_000, -1), None); + assert!(book.bids.is_empty()); + } + + #[test] + fn a_sum_past_the_largest_quantity_removes_the_level() { + let mut book = Book::default(); + assert_eq!(book.add(Side::Buy, 400_000, i64::MAX), Some(i64::MAX)); + assert_eq!(book.bids, BTreeMap::from([(400_000, i64::MAX)])); + assert_eq!(book.add(Side::Buy, 400_000, 1), None); assert!(book.bids.is_empty()); } diff --git a/src/cause.rs b/src/cause.rs index 880f17c..a8f8577 100644 --- a/src/cause.rs +++ b/src/cause.rs @@ -17,16 +17,25 @@ pub enum Cause { /// The connection followed changed: the new one's book, which may hold /// changes the old one never delivered. Switch, + /// Its market was removed and is tracked again: its first book since, + /// against the last one written before. + TrackedAgain, } impl Cause { - const ALL: [Cause; 3] = [Cause::Loss, Cause::SubscriptionGap, Cause::Switch]; + const ALL: [Cause; 4] = [ + Cause::Loss, + Cause::SubscriptionGap, + Cause::Switch, + Cause::TrackedAgain, + ]; fn name(self) -> &'static str { match self { Cause::Loss => "loss", Cause::SubscriptionGap => "subscription gap", Cause::Switch => "switch", + Cause::TrackedAgain => "tracked again", } } } diff --git a/src/core/feed.rs b/src/core/feed.rs index 38d46ec..86c882f 100644 --- a/src/core/feed.rs +++ b/src/core/feed.rs @@ -6,13 +6,14 @@ //! subscription gone quiet on one connection (roadmap/05). What an event does //! to a book, how a loss is found and when to resync is the rule's. -use std::collections::{BTreeMap, BTreeSet, HashMap}; +use std::collections::{BTreeMap, BTreeSet, HashMap, HashSet}; use std::time::Duration; use chrono::{DateTime, DurationRound, TimeDelta, Utc}; use crossbeam::channel::{Receiver, Sender, never, select}; use engineering_challenge::Side; use engineering_challenge::book::{Book, Output}; +use engineering_challenge::cause::{self, Cause}; use engineering_challenge::util::dtos::challenge_orderbook::PriceLevelUpdateDomain; use super::socket::Command; @@ -24,9 +25,9 @@ pub(crate) const MIN_MEASURED: TimeDelta = TimeDelta::seconds(30); /// How often a book thread whose rule asks for it wakes for the rule's timer. const WAKE_EVERY: Duration = Duration::from_millis(10); /// A connection that delivered none of a symbol's changes in a minute while -/// another delivered at least this many is quiet for it (sprint 05: crossed -/// once in 21 live minutes; at 1, copies landing in the next minute would -/// count). +/// another delivered at least this many is quiet for it (crossed once in 21 +/// live minutes, docs/notes.md; at 1, copies landing in the next minute +/// would count). const QUIET_AGAINST: u32 = 3; /// What the book thread does with an event or a command: write output, ask a @@ -222,6 +223,10 @@ pub struct Symbol { /// The receive time of its last Delta or Snapshot: a later Delta is /// stamped no earlier, so replaying the output by time keeps its order. pub last_written: DateTime, + /// The exchange's name while its market, removed before, is tracked + /// again and no Snapshot was written since: the first says so (its + /// book can't be told from the one written before it left). + pub again: Option<&'static str>, pub rule: S, } @@ -238,6 +243,7 @@ impl Symbol { .collect(), quiet_on: None, last_written: DateTime::UNIX_EPOCH, + again: None, rule, } } @@ -281,6 +287,12 @@ impl Symbol { /// The Snapshot of its book, received at `received_at`. pub fn snapshot(&mut self, name: &str, received_at: DateTime) -> Output { self.last_written = self.last_written.max(received_at); + if let Some(exchange) = self.again.take() { + log!( + "{exchange}: {}", + cause::line(name, received_at, Cause::TrackedAgain, "") + ); + } self.book .snapshot(name, received_at, self.exchange_timestamp) } @@ -378,6 +390,11 @@ pub struct Shell { pub peak_waiting: usize, pub quiet: u64, pub quiet_reported: u64, + /// The symbols of the markets removed: one tracked again says so in its + /// first Snapshot. Ceiling: it grows with every market closed in the + /// run (tens of bytes a symbol, a few thousand a day); forgetting them + /// after a day would bound it. + pub removed: HashSet, } impl Shell { @@ -429,6 +446,7 @@ impl Feed { peak_waiting: 0, quiet: 0, quiet_reported: 0, + removed: HashSet::new(), }, rule, } @@ -446,10 +464,11 @@ impl Feed { let names = market.symbols().to_vec(); for name in &names { let rule = self.rule.symbol(market); - self.shell.symbols.insert( - name.clone(), - Symbol::new(market.id(), connections, now, rule), - ); + let mut symbol = Symbol::new(market.id(), connections, now, rule); + if self.shell.removed.remove(name) { + symbol.again = Some(self.shell.exchange); + } + self.shell.symbols.insert(name.clone(), symbol); } let tracked = Tracked { names: names.clone(), @@ -535,6 +554,7 @@ impl Feed { for name in market.symbols() { if let Some(symbol) = self.shell.symbols.remove(name) { connections = symbol.connections; + self.shell.removed.insert(name.clone()); } self.rule.on_removed(name); } @@ -700,7 +720,7 @@ impl Feed { /// A book thread: runs until every connection and the orchestrator have /// dropped its channels, then drops its senders so the writer thread finishes /// once every book thread has. -// ponytail: unbounded channels never block a socket read nor drop an event; +// Ceiling: unbounded channels never block a socket read nor drop an event; // the ceiling is memory if a consumer stays behind. The minute log's peak of // events waiting makes that visible; bounded channels are a constructor change // if it ever matters. diff --git a/src/core/mod.rs b/src/core/mod.rs index f323ba9..e8367d4 100644 --- a/src/core/mod.rs +++ b/src/core/mod.rs @@ -18,7 +18,7 @@ use crossbeam::channel::{Receiver, Sender}; use feed::Rule; /// How many book threads own the books. One handled ~30,000 frames a second -/// in sprint 02. +/// when measured (roadmap/04). pub const BOOK_THREADS: usize = 16; /// The book thread that owns a market's books, from a hash of its id. diff --git a/src/core/orchestrator.rs b/src/core/orchestrator.rs index dda0390..53f33a0 100644 --- a/src/core/orchestrator.rs +++ b/src/core/orchestrator.rs @@ -44,6 +44,8 @@ const MOVED_STAYS: Duration = Duration::from_secs(10 * 60); const SPLIT_WITHIN: Duration = Duration::from_secs(10 * 60); const RECENT_EVERY: Duration = Duration::from_secs(60); const RESCAN_EVERY: Duration = Duration::from_secs(60 * 60); +/// How often the first scan says how far it got. +const SCAN_NOTE_EVERY: Duration = Duration::from_secs(10); /// The listing's indexing delay: each recent check reaches this far before /// the last successful one. New markets were listed up to 5 minutes after /// their creation (docs/notes.md); known markets are skipped, so the margin @@ -54,9 +56,10 @@ const RECENT_MARGIN: TimeDelta = TimeDelta::minutes(15); const RECYCLE_EVERY: Duration = Duration::from_secs(60 * 60); /// A connection slow (among the slowest quarter by mean lateness, and more /// than [`SLOW_FACTOR`] times the mean of every copy judged) this many -/// minutes in a row is recycled ahead of its turn: the next one is drawn -/// afresh. Kalshi's demo kept the 2 fastest of 16 connections 28% and 42% -/// lower at p99 (LATENCY.md); only Kalshi's rule reports lateness. The mean, +/// minutes in a row takes the next recycle's turn, in place of the oldest: +/// its replacement's speed is drawn afresh. Kalshi's demo kept the 2 fastest +/// of 16 connections 28% and 42% lower at p99 (LATENCY.md); only Kalshi's +/// rule reports lateness. The mean, /// not the median: with two copies of each change, a connection first more /// than half the time has a median of 0 (roadmap/07). const SLOW_MINUTES: u32 = 3; @@ -123,6 +126,9 @@ pub fn run( let mut fetcher_died = false; let mut next_rescan = Instant::now(); let mut next_recent = Instant::now() + RECENT_EVERY; + // The first scan pages the whole listing (a minute or two): said as it + // goes, so a run that has written nothing yet doesn't look hung. + let mut next_scan_note = Instant::now() + SCAN_NOTE_EVERY; while !stop.load(Ordering::Relaxed) { // Only a panic ends a fetcher while this runs, even one in the middle @@ -136,6 +142,16 @@ pub fn run( } let now = Instant::now(); let mut commands = Vec::new(); + if let Some(waiting) = &first_scan + && scanning + && now >= next_scan_note + { + log!( + "{name}: orchestrator: first scan, {} live markets listed so far", + waiting.len() + ); + next_scan_note = now + SCAN_NOTE_EVERY; + } if now >= next_rescan { if !scanning { _ = rescan.send(Request::Rescan); @@ -155,15 +171,11 @@ pub fn run( let moves = tracker.rebalance(now); moved += moves.len(); commands.extend(moves); - if let Some((connection, mean)) = tracker.judge_speeds(now) { + if let Some(marked) = tracker.judge_speeds(now) { log!( - "{name}: orchestrator: connection {connection} among the slowest for \ - {SLOW_MINUTES} minutes (mean lateness {:.1} ms), recycled", - mean as f64 / 1_000.0 + "{name}: orchestrator: {}, next to recycle", + slow_line(marked) ); - threads.recycle(connection); - recycled += 1; - slow_recycled += 1; } let busiest = tracker.connections.iter().map(|one| one.load).max(); log!( @@ -181,7 +193,14 @@ pub fn run( if first_scan.is_none() && queue.is_empty() { tracker.start_recycling(now); } - if let Some(connection) = tracker.recycle(now) { + if let Some((connection, slow)) = tracker.recycle(now) { + if let Some(mean) = slow { + log!( + "{name}: orchestrator: {}, recycled", + slow_line((connection, mean)) + ); + slow_recycled += 1; + } threads.recycle(connection); recycled += 1; } @@ -203,6 +222,10 @@ pub fn run( recv(findings) -> finding => match finding { Ok(Finding::ScanStarted) => { tracker.scan_started(); + if first_scan.is_some() { + log!("{name}: orchestrator: first scan, listing the exchange's markets"); + next_scan_note = Instant::now() + SCAN_NOTE_EVERY; + } // A first scan that failed starts over. if let Some(waiting) = &mut first_scan { waiting.clear(); @@ -224,6 +247,10 @@ pub fn run( scanned = true; match first_scan.take() { Some(markets) => { + log!( + "{name}: orchestrator: first scan done, {} live markets, subscribing", + markets.len() + ); let adds = tracker.deal(markets); added += adds.len(); commands.extend(adds); @@ -466,6 +493,21 @@ struct Tracker { recycling: Option, /// The book threads' lateness samples since the last judgment. lateness: Lateness, + /// The connection judged slow, with its mean lateness (µs): it takes the + /// next recycle's turn in place of the oldest. + slow: Option<(usize, i64)>, + /// The connections whose mark was logged, while they stay slow: two that + /// swap ranks are each said once. + said: HashSet, +} + +/// A slow connection, as the log says it: its number and mean lateness (µs). +fn slow_line((connection, mean): (usize, i64)) -> String { + format!( + "connection {connection} among the slowest for {SLOW_MINUTES} minutes (mean lateness \ + {:.1} ms)", + mean as f64 / 1_000.0 + ) } impl Tracker { @@ -479,6 +521,8 @@ impl Tracker { next_recycle: None, recycling: None, lateness: Lateness::new(), + slow: None, + said: HashSet::new(), } } @@ -602,7 +646,7 @@ impl Tracker { .collect(); over.sort_by_key(|&number| Reverse(self.connections[number].load)); for from in over { - // ponytail: a scan of every market per connection over budget, + // Ceiling: a scan of every market per connection over budget, // fine at a few hundred thousand once a minute; per-connection // market sets if not. let mut busiest: Vec<(String, u64)> = self @@ -715,6 +759,11 @@ impl Tracker { if self.recycling == Some(connection) { self.recycling = None; } + // A fresh connection: what was judged slow is gone with the old one. + if self.slow.is_some_and(|(slow, _)| slow == connection) { + self.slow = None; + } + self.said.remove(&connection); let settled = now + self.recycle_interval(); if let Some(next) = &mut self.next_recycle { *next = (*next).max(settled); @@ -738,11 +787,12 @@ impl Tracker { RECYCLE_EVERY / self.connections.len().max(1) as u32 } - /// The connection to recycle now, if one is due: the one up the longest, - /// once the last recycled is up again, and only while every connection is - /// up and has been for an interval, so no market loses both of its + /// The connection to recycle now, if one is due: the one judged slow if + /// there is one (with its mean lateness), else the one up the longest; + /// once the last recycled is up again, and only while every connection + /// is up and has been for an interval, so no market loses both of its /// connections at once. - fn recycle(&mut self, now: Instant) -> Option { + fn recycle(&mut self, now: Instant) -> Option<(usize, Option)> { let due = self.next_recycle.is_some_and(|next| now >= next); if !due || self.recycling.is_some() @@ -750,11 +800,17 @@ impl Tracker { { return None; } - let oldest = (0..self.connections.len()) - .min_by_key(|&number| self.connections[number].connected_at)?; - self.recycling = Some(oldest); + let (connection, slow) = match self.slow.take() { + Some((slow, mean)) => (slow, Some(mean)), + None => ( + (0..self.connections.len()) + .min_by_key(|&number| self.connections[number].connected_at)?, + None, + ), + }; + self.recycling = Some(connection); self.next_recycle = Some(now + self.recycle_interval()); - Some(oldest) + Some((connection, slow)) } /// A book thread's lateness samples, by connection. @@ -768,10 +824,10 @@ impl Tracker { /// since the last time; those of the slowest 1 in [`SLOW_SHARE`] more /// than [`SLOW_FACTOR`] times the mean of every judged copy count a slow /// minute, the others start over. The slowest of those slow for - /// [`SLOW_MINUTES`] is recycled now, with its mean (µs), under - /// [`Tracker::recycle`]'s - /// rule: one at a time, every connection up, none up for less than an - /// interval. + /// [`SLOW_MINUTES`] is marked, with its mean (µs), to take the next + /// recycle's turn ([`Tracker::recycle`]): one path, one at a time. + /// Returns the mark when it is new: a connection marked again while it + /// stays slow, waiting for its turn, is said once. fn judge_speeds(&mut self, now: Instant) -> Option<(usize, i64)> { let samples = std::mem::take(&mut self.lateness); let speeds: Vec<(usize, i64, i64)> = samples @@ -814,22 +870,15 @@ impl Tracker { 0 }; } - // As for the hourly recycle: one at a time, every connection up, and - // the last to come up past its opening books. - let settled = self - .connections + let connections = &self.connections; + self.said + .retain(|&connection| connections[connection].slow_minutes > 0); + self.slow = slow .iter() - .filter_map(|state| state.connected_at) - .max() - .is_none_or(|last| now.duration_since(last) >= self.recycle_interval()); - if self.recycling.is_some() || self.connections.iter().any(|state| !state.up) || !settled { - return None; - } - let &(connection, speed) = slow - .iter() - .find(|&&(connection, _)| self.connections[connection].slow_minutes >= SLOW_MINUTES)?; - self.recycling = Some(connection); - Some((connection, speed)) + .copied() + .find(|&(connection, _)| self.connections[connection].slow_minutes >= SLOW_MINUTES); + self.slow + .filter(|&(connection, _)| self.said.insert(connection)) } fn scan_started(&mut self) { @@ -1189,7 +1238,7 @@ mod tests { tracker.start_recycling(at(10)); // One every hour / 3 connections. assert_eq!(tracker.recycle(at(1_209)), None); - assert_eq!(tracker.recycle(at(1_210)), Some(2)); + assert_eq!(tracker.recycle(at(1_210)), Some((2, None))); // The next waits for the recycled one, whose Down may not even have // come yet, to be up again... assert_eq!(tracker.recycle(at(2_410)), None); @@ -1198,7 +1247,7 @@ mod tests { tracker.up(2, at(2_411)); // ...and for its opening books: an interval more. assert_eq!(tracker.recycle(at(3_610)), None); - assert_eq!(tracker.recycle(at(3_611)), Some(0)); + assert_eq!(tracker.recycle(at(3_611)), Some((0, None))); tracker.down(0); tracker.up(0, at(3_612)); // A connection down for its own reasons holds every recycle, and its @@ -1207,7 +1256,7 @@ mod tests { assert_eq!(tracker.recycle(at(6_000)), None); tracker.up(2, at(6_001)); assert_eq!(tracker.recycle(at(7_200)), None); - assert_eq!(tracker.recycle(at(7_201)), Some(1)); + assert_eq!(tracker.recycle(at(7_201)), Some((1, None))); } /// Twelve connections, up from `start`. @@ -1236,36 +1285,119 @@ mod tests { } #[test] - fn a_connection_among_the_slowest_three_minutes_running_is_recycled() { + fn a_connection_among_the_slowest_three_minutes_running_takes_the_next_recycle() { let start = Instant::now(); let at = |seconds: u64| start + Duration::from_secs(seconds); let mut tracker = twelve_up(start); - // The slowest quarter is three of twelve. 3 and 7 are slow every - // minute, 5 for two minutes and then not. + // The hourly recycle runs: one every 300 s (an hour / 12), the + // first due at 600. + tracker.start_recycling(at(300)); + // The slowest quarter is three of twelve, then two of the eleven + // judged once 0 is fresh. 3 and 7 are slow every minute, 5 only in + // the first. minute(&mut tracker, &[(3, 90), (5, 50), (7, 40)], 100); assert_eq!(tracker.judge_speeds(at(601)), None); - minute(&mut tracker, &[(3, 90), (5, 50), (7, 40)], 100); + // The hourly turn takes the oldest (all equal: the first). + assert_eq!(tracker.recycle(at(601)), Some((0, None))); + tracker.down(0); + tracker.up(0, at(603)); + minute(&mut tracker, &[(3, 90), (7, 40), (5, 35)], 100); assert_eq!(tracker.judge_speeds(at(661)), None); - // 3 and 7 both reach three minutes: one recycle a minute, the slowest. + // 3 and 7 both reach three minutes: the slowest is marked. minute(&mut tracker, &[(3, 90), (7, 40), (9, 30)], 100); assert_eq!(tracker.judge_speeds(at(721)), Some((3, 90_000))); assert_eq!(tracker.connections[5].slow_minutes, 0); - // While 3 recycles, 7 stays slow but isn't recycled. - minute(&mut tracker, &[(3, 90), (7, 40), (9, 30)], 100); - assert_eq!(tracker.judge_speeds(at(781)), None); - assert_eq!(tracker.connections[7].slow_minutes, 4); + // Not before the turn (0 came up at 603: due at 903)... + assert_eq!(tracker.recycle(at(902)), None); + // ...then the turn is 3's, not the oldest's. + assert_eq!(tracker.recycle(at(903)), Some((3, Some(90_000)))); + // While 3 recycles, nothing else is. + assert_eq!(tracker.recycle(at(1_300)), None); tracker.down(3); - tracker.up(3, at(790)); - // 3 is fresh: slow, but not judged. 7 waits, as the hourly recycle - // would, for 3's opening books: an interval (an hour / 12) after it - // came up. + tracker.up(3, at(1_301)); + // 3 is fresh: not judged, its slow minutes start over; 7, still + // slow, takes the next turn, 300 s after 3 came up. minute(&mut tracker, &[(3, 90), (7, 40), (9, 30)], 100); - assert_eq!(tracker.judge_speeds(at(841)), None); + assert_eq!(tracker.judge_speeds(at(1_302)), Some((7, 40_000))); assert_eq!(tracker.connections[3].slow_minutes, 0); - minute(&mut tracker, &[(3, 90), (7, 40), (9, 30)], 100); - assert_eq!(tracker.judge_speeds(at(1_089)), None); - minute(&mut tracker, &[(3, 90), (7, 40), (9, 30)], 100); - assert_eq!(tracker.judge_speeds(at(1_090)), Some((7, 40_000))); + assert_eq!(tracker.recycle(at(1_600)), None); + assert_eq!(tracker.recycle(at(1_601)), Some((7, Some(40_000)))); + // Then the hourly order again. + tracker.down(7); + tracker.up(7, at(1_602)); + assert_eq!(tracker.recycle(at(1_902)), Some((1, None))); + } + + #[test] + fn a_mark_is_said_once_and_cleared_once_the_connection_is_no_longer_slow() { + let start = Instant::now(); + let at = |seconds: u64| start + Duration::from_secs(seconds); + let mut tracker = twelve_up(start); + for seconds in [601, 661] { + minute(&mut tracker, &[(3, 90)], 100); + assert_eq!(tracker.judge_speeds(at(seconds)), None); + } + minute(&mut tracker, &[(3, 90)], 100); + assert_eq!(tracker.judge_speeds(at(721)), Some((3, 90_000))); + // Still slow, its turn not come: marked still, but not said again. + minute(&mut tracker, &[(3, 90)], 100); + assert_eq!(tracker.judge_speeds(at(781)), None); + assert_eq!(tracker.slow, Some((3, 90_000))); + // As fast as the others before its turn: the mark goes, and the + // turn is the oldest's. + minute(&mut tracker, &[], 100); + assert_eq!(tracker.judge_speeds(at(841)), None); + assert_eq!(tracker.slow, None); + tracker.start_recycling(at(841)); + assert_eq!(tracker.recycle(at(1_141)), Some((0, None))); + } + + #[test] + fn two_marked_connections_swapping_ranks_are_each_said_once() { + let start = Instant::now(); + let at = |seconds: u64| start + Duration::from_secs(seconds); + let mut tracker = twelve_up(start); + for seconds in [601, 661] { + minute(&mut tracker, &[(3, 90), (7, 40)], 100); + assert_eq!(tracker.judge_speeds(at(seconds)), None); + } + minute(&mut tracker, &[(3, 90), (7, 40)], 100); + assert_eq!(tracker.judge_speeds(at(721)), Some((3, 90_000))); + // 7 overtakes 3: the mark moves to 7, said once. + minute(&mut tracker, &[(3, 40), (7, 90)], 100); + assert_eq!(tracker.judge_speeds(at(781)), Some((7, 90_000))); + // Back to 3, still slow since it was said: not again. + minute(&mut tracker, &[(3, 90), (7, 40)], 100); + assert_eq!(tracker.judge_speeds(at(841)), None); + assert_eq!(tracker.slow, Some((3, 90_000))); + // 3 fast for a minute, then slow 3 minutes again: said anew. + minute(&mut tracker, &[(7, 40)], 100); + tracker.judge_speeds(at(901)); + for seconds in [961, 1_021] { + minute(&mut tracker, &[(3, 90), (7, 40)], 100); + tracker.judge_speeds(at(seconds)); + } + minute(&mut tracker, &[(3, 90), (7, 40)], 100); + assert_eq!(tracker.judge_speeds(at(1_081)), Some((3, 90_000))); + } + + #[test] + fn a_slow_mark_is_dropped_when_its_connection_comes_up_fresh() { + let start = Instant::now(); + let at = |seconds: u64| start + Duration::from_secs(seconds); + let mut tracker = twelve_up(start); + for seconds in [601, 661, 721] { + minute(&mut tracker, &[(3, 90), (7, 40)], 100); + tracker.judge_speeds(at(seconds)); + } + assert_eq!(tracker.slow, Some((3, 90_000))); + // 3 reconnects for its own reasons before its turn: renewed, so + // the turn goes back to the oldest. + tracker.down(3); + tracker.up(3, at(730)); + assert_eq!(tracker.slow, None); + tracker.start_recycling(at(730)); + assert_eq!(tracker.recycle(at(1_030)), Some((0, None))); } #[test] @@ -1326,13 +1458,16 @@ mod tests { minute(&mut tracker, &[(3, 90), (5, 50)], 99); assert_eq!(tracker.judge_speeds(at(seconds)), None); } - // A connection down holds every recycle, as the hourly one's rule. + // A connection down holds the slow one's turn, as every recycle. let mut tracker = twelve_up(start); + tracker.start_recycling(start); tracker.down(0); - for seconds in [601, 661, 721, 781] { + for seconds in [601, 661, 721] { minute(&mut tracker, &[(3, 90), (5, 50)], 100); - assert_eq!(tracker.judge_speeds(at(seconds)), None); + tracker.judge_speeds(at(seconds)); } + assert_eq!(tracker.slow, Some((3, 90_000))); + assert_eq!(tracker.recycle(at(1_000)), None); } #[test] @@ -1447,3 +1582,200 @@ mod tests { assert_eq!(tracker.markets.len(), 1); } } + +/// [`run`] end to end, offline: Kalshi's listing and websocket served +/// locally, the writer as `main` runs it. +#[cfg(test)] +mod run_tests { + use std::io::{BufRead, BufReader, Write}; + use std::net::{TcpListener, TcpStream}; + use std::sync::Arc; + use std::sync::atomic::{AtomicBool, Ordering}; + use std::thread; + use std::time::Duration; + + use crossbeam::channel::{Sender, unbounded}; + use tungstenite::Message; + + use super::SUBSCRIPTIONS; + use crate::kalshi::{Kalshi, sign}; + + const TICKERS: [&str; 2] = ["KXA", "KXB"]; + + /// A local listing: the tickers open, nothing unopened. + fn listing() -> String { + let listener = TcpListener::bind("127.0.0.1:0").unwrap(); + let url = format!("http://{}", listener.local_addr().unwrap()); + thread::spawn(move || { + for stream in listener.incoming() { + let stream = stream.unwrap(); + let mut reader = BufReader::new(&stream); + let mut request = String::new(); + reader.read_line(&mut request).unwrap(); + // The headers, read whole so the close doesn't reset them. + let mut line = String::new(); + while reader.read_line(&mut line).unwrap() > 2 { + line.clear(); + } + let markets: Vec = TICKERS + .iter() + .map(|ticker| format!(r#"{{"ticker":"{ticker}","status":"active"}}"#)) + .collect(); + let open = request.contains("status=open") && !request.contains("cursor="); + let markets = if open { + markets.join(",") + } else { + String::new() + }; + let body = format!(r#"{{"markets":[{markets}],"cursor":""}}"#); + write!( + &stream, + "HTTP/1.1 200 OK\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}", + body.len() + ) + .unwrap(); + } + }); + url + } + + /// A local websocket: each connection answers every ticker it subscribes + /// with a book and a change, the same on every connection. `books` hears + /// of each ticker served, `closed` whether each connection was closed + /// with a Close frame. + fn websocket(books: Sender<()>, closed: Sender) -> String { + let listener = TcpListener::bind("127.0.0.1:0").unwrap(); + let url = format!("ws://{}/trade-api/ws/v2", listener.local_addr().unwrap()); + thread::spawn(move || { + for stream in listener.incoming() { + let (books, closed) = (books.clone(), closed.clone()); + thread::spawn(move || serve(stream.unwrap(), &books, &closed)); + } + }); + url + } + + fn serve(stream: TcpStream, books: &Sender<()>, closed: &Sender) { + let mut socket = tungstenite::accept(stream).unwrap(); + let mut seq = 0; + loop { + match socket.read() { + Ok(Message::Text(text)) => { + let command: serde_json::Value = serde_json::from_str(&text).unwrap(); + if command["cmd"] != "subscribe" { + continue; + } + if seq == 0 { + let subscribed = r#"{"type":"subscribed","id":1,"msg":{"channel":"orderbook_delta","sid":1}}"#; + socket.send(Message::text(subscribed)).unwrap(); + } + for ticker in command["params"]["market_tickers"].as_array().unwrap() { + let ticker = ticker.as_str().unwrap(); + let book = format!( + r#"{{"type":"orderbook_snapshot","sid":1,"seq":{},"msg":{{"market_ticker":"{ticker}","market_id":"x","clock":10,"yes_dollars_fp":[["0.4000","5.00"]]}},"sending_ts_ms":1794422918369}}"#, + seq + 1 + ); + let change = format!( + r#"{{"type":"orderbook_delta","sid":1,"seq":{},"msg":{{"market_ticker":"{ticker}","market_id":"x","clock":11,"price_dollars":"0.4000","delta_fp":"1.00","side":"yes","ts_ms":1794422918370}}}}"#, + seq + 2 + ); + seq += 2; + socket.send(Message::text(book)).unwrap(); + socket.send(Message::text(change)).unwrap(); + _ = books.send(()); + } + } + Ok(Message::Close(_)) => { + // The reply goes out with the next read. + while socket.read().is_ok() {} + _ = closed.send(true); + return; + } + Ok(_) => {} + Err(_) => { + _ = closed.send(false); + return; + } + } + } + } + + #[test] + fn a_run_subscribes_writes_and_stops_on_the_flag_closing_every_socket() { + let (books_tx, books) = unbounded(); + let (closed_tx, closed) = unbounded(); + let kalshi = Kalshi { + websocket: websocket(books_tx, closed_tx), + listing: listing(), + key: sign::test_key(), + }; + let path = + std::env::temp_dir().join(format!("orchestrator-run-{}.sbe", std::process::id())); + let (outputs, outputs_rx) = unbounded(); + // Between the run and the writer: says when every change has gone + // through, so the test stops then, not after a guessed sleep. + let (relayed, writer_rx) = unbounded(); + let (all_deltas_tx, all_deltas) = unbounded(); + thread::spawn(move || { + let mut deltas = 0; + for output in outputs_rx { + if matches!(output, engineering_challenge::book::Output::Delta(_)) { + deltas += 1; + if deltas == TICKERS.len() { + _ = all_deltas_tx.send(()); + } + } + _ = relayed.send(output); + } + }); + let (written_tx, written) = unbounded(); + let writer_path = path.to_str().unwrap().to_string(); + thread::spawn(move || { + let result = crate::write(&writer_path, writer_rx).map_err(|error| error.to_string()); + written_tx.send(result).unwrap(); + }); + let stop = Arc::new(AtomicBool::new(false)); + let run_stop = Arc::clone(&stop); + let (failed_tx, failed) = unbounded(); + thread::spawn(move || failed_tx.send(super::run(Arc::new(kalshi), outputs, &run_stop))); + + // Each ticker served on each of its connections. + for _ in 0..TICKERS.len() * SUBSCRIPTIONS { + books + .recv_timeout(Duration::from_secs(10)) + .expect("every subscription served"); + } + // Their events reach the book threads, and the writer. + all_deltas + .recv_timeout(Duration::from_secs(10)) + .expect("every change written"); + stop.store(true, Ordering::Relaxed); + // Timed, so a run that never ends fails rather than hangs. + let failed = failed + .recv_timeout(Duration::from_secs(10)) + .expect("the run returns once stopped"); + assert_eq!(failed, 0); + let written = written + .recv_timeout(Duration::from_secs(10)) + .expect("the writer ends once the run has"); + let closes: Vec = (0..SUBSCRIPTIONS) + .map(|_| closed.recv_timeout(Duration::from_secs(5)).unwrap()) + .collect(); + assert_eq!( + closes, [true; SUBSCRIPTIONS], + "a Close frame on every socket" + ); + // No other connection. + assert!(closed.recv_timeout(Duration::from_millis(200)).is_err()); + written.unwrap(); + let report = engineering_challenge::reconcile::reconcile(&path, None); + std::fs::remove_file(&path).unwrap(); + let report = report.unwrap(); + // A first book per ticker, and one per minute crossed; each change + // once, the other connection's copy a duplicate. + assert!(report.snapshots >= TICKERS.len(), "{report:?}"); + assert_eq!(report.deltas, TICKERS.len(), "{report:?}"); + assert_eq!(report.orphan_deltas, 0); + assert_eq!(report.mismatches, []); + } +} diff --git a/src/core/socket.rs b/src/core/socket.rs index 125f0cd..b38e387 100644 --- a/src/core/socket.rs +++ b/src/core/socket.rs @@ -31,6 +31,7 @@ const CLOSE_TIMEOUT: Duration = Duration::from_secs(2); /// Nothing received this long, not even a keepalive's answer or a ping: the /// connection is silent, and reconnects. pub const SILENT_AFTER: Duration = Duration::from_secs(30); + /// What a book thread asks a connection for. #[derive(Debug, PartialEq, Eq)] pub enum Command { diff --git a/src/kalshi/feed.rs b/src/kalshi/feed.rs index d7ddf28..51ba539 100644 --- a/src/kalshi/feed.rs +++ b/src/kalshi/feed.rs @@ -29,7 +29,7 @@ const BOOK: u32 = u32::MAX; /// Changes remembered per symbol for their second copy's lateness: two /// connections deliver a market's changes 2 ms apart at the median, 344 ms /// at worst (docs/notes.md), far fewer changes than this. -// ponytail: a second copy more than 128 changes behind goes unmeasured, and +// Ceiling: a second copy more than 128 changes behind goes unmeasured, and // the lookup is linear; a map by position, emptied as the book passes, // would lift both. const RECENT: usize = 128; @@ -205,8 +205,11 @@ impl Clock { .recent .push_back((position, connection, received_at)); let quantity = symbol.book.add(change.side, change.price, change.delta); - let quantity = if quantity < 0 { - // A level can't hold less than nothing: ours was wrong. + let quantity = if let Some(quantity) = quantity { + quantity + } else { + // A level can't hold less than nothing (nor more than an i64): + // ours was wrong. self.counts.below_zero += 1; log!( "kalshi: {ticker}: a change takes a level below zero, resync on connection \ @@ -217,8 +220,6 @@ impl Clock { self.counts.resyncs += 1; } 0 - } else { - quantity }; symbol.exchange_timestamp = change.at; self.counts.written += 1; @@ -677,6 +678,20 @@ mod tests { assert_eq!(deltas(&after)[0].quantity, 10_000_000); } + /// A change that would take a level past an i64: the book is wrong, as + /// below zero. + #[test] + fn a_level_past_the_largest_quantity_resyncs() { + let mut feed = both_on("M"); + let level = feed.shell.symbols["M"].book.bids[&400_000]; + assert!(level > 0); + let effects = feed.from(0, 3, change("M", 11, Side::Buy, 400_000, i64::MAX)); + assert_eq!(deltas(&effects)[0].quantity, 0); + assert_eq!(resyncs(&effects), [(0, "M")]); + assert_eq!(feed.shell.symbols["M"].book.bids.get(&400_000), None); + assert_eq!(feed.rule.counts.below_zero, 1); + } + /// The answer to a resync can be behind what the other connection has /// delivered: the book goes back to it, and only a stream that continues /// from there is taken until the other fits again. @@ -778,6 +793,140 @@ mod tests { assert_eq!(snapshots(&feed.tick(minute + TimeDelta::minutes(2))), 1); } + fn unsubscribes(effects: &[Effect]) -> Vec { + effects + .iter() + .filter_map(|effect| match effect { + Effect::Send(connection, Command::Unsubscribe(_)) => Some(*connection), + _ => None, + }) + .collect() + } + + fn market(ticker: &str) -> Market { + Market { + ticker: ticker.into(), + } + } + + /// `both_on`'s market moved from connection 0 to 2, the move ended by + /// connection 2's book: connection 0 left it. + fn moved_off_zero() -> Feed { + let mut feed = both_on("M"); + feed.move_market(&market("M"), 0, 2, at(3)); + let ended = feed.from(2, 4, book("M", 10, &[(400_000, 5_000_000)], &[])); + assert_eq!(unsubscribes(&ended), [0]); + feed + } + + #[test] + fn a_book_from_a_connection_the_symbol_left_counts_for_nothing() { + let mut feed = moved_off_zero(); + // Still on its way when it left, past ours and different. + let late = feed.from(0, 5, book("M", 20, &[(400_000, 1_000_000)], &[])); + assert!(late.is_empty(), "{late:?}"); + assert_eq!(feed.rule.counts.losses, 0); + assert_eq!(feed.shell.symbols["M"].book.bids[&400_000], 5_000_000); + } + + #[test] + fn a_change_from_a_connection_the_symbol_left_has_no_place() { + let mut feed = moved_off_zero(); + let late = feed.from(0, 5, change("M", 11, Side::Buy, 400_000, 1_000_000)); + assert!(deltas(&late).is_empty(), "{late:?}"); + assert_eq!(feed.rule.counts.unplaced, 1); + } + + /// The answer to a resync can be our very book: the resync is over, not + /// asked again every minute. + #[test] + fn an_answer_equal_to_ours_ends_the_resync() { + let mut feed = both_on("M"); + feed.on_received(Received::Down(0)); + feed.on_received(Received::Down(1)); + feed.on_received(Received::Up(0)); + feed.on_received(Received::Up(1)); + let minute = next_minute(at(10)); + assert_eq!(resyncs(&feed.tick(minute)), [(0, "M")]); + let answer = feed.from(0, 70_000, book("M", 10, &[(400_000, 5_000_000)], &[])); + assert!(answer.is_empty(), "{answer:?}"); + assert_eq!(feed.shell.symbols["M"].rule.asked, None); + assert!(resyncs(&feed.tick(minute + TimeDelta::minutes(1))).is_empty()); + } + + #[test] + fn a_second_level_below_zero_while_a_resync_is_asked_asks_no_other() { + let mut feed = both_on("M"); + let first = feed.from(0, 3, change("M", 11, Side::Buy, 400_000, -6_000_000)); + assert_eq!(resyncs(&first), [(0, "M")]); + let second = feed.from(0, 4, change("M", 12, Side::Sell, 600_000, -1_000_000)); + assert_eq!(deltas(&second)[0].quantity, 0); + assert!(resyncs(&second).is_empty(), "{second:?}"); + assert_eq!(feed.rule.counts.below_zero, 2); + assert_eq!(feed.rule.counts.resyncs, 1); + } + + // The shell's moves and quiet watch (src/core/feed.rs), through Kalshi's + // rule, the simpler one. + + #[test] + fn a_move_ends_on_the_new_connections_book_not_on_its_change() { + let mut feed = both_on("M"); + feed.move_market(&market("M"), 0, 2, at(3)); + let change = feed.from(2, 4, change("M", 11, Side::Buy, 400_000, 1_000_000)); + assert!(unsubscribes(&change).is_empty(), "{change:?}"); + let ended = feed.from(2, 5, book("M", 10, &[(400_000, 5_000_000)], &[])); + assert_eq!(unsubscribes(&ended), [0]); + } + + #[test] + fn a_second_move_while_one_is_under_way_ends_the_first() { + let mut feed = both_on("M"); + feed.move_market(&market("M"), 0, 2, at(3)); + let second = feed.move_market(&market("M"), 1, 3, at(4)); + assert_eq!(unsubscribes(&second), [0]); + assert!(second.iter().any(|effect| matches!( + effect, + Effect::Send(3, Command::Subscribe(tickers)) if tickers == &["M".to_string()] + ))); + let mut on = feed.shell.symbols["M"].connections.clone(); + on.sort(); + assert_eq!(on, [1, 2, 3]); + } + + /// Quiet on connection 1, resubscribed; a minute delivering on both; + /// quiet again: resubscribed again, not reported for a move. + #[test] + fn a_good_minute_between_two_quiet_ones_forgets_the_first() { + let mut feed = both_on("M"); + let minute = |n: i64| next_minute(at(10)) + TimeDelta::minutes(n); + feed.tick(minute(0)); + let mut clock = 10; + let mut changes = |feed: &mut Feed, n: i64, on: &[usize]| { + for step in 0..3 { + clock += 1; + for &connection in on { + let millis = (minute(n) - at(0)).num_milliseconds() + 10_000 + step; + feed.from(connection, millis, change("M", clock, Side::Buy, 1, 1)); + } + } + }; + changes(&mut feed, 0, &[0]); + assert_eq!(resyncs(&feed.tick(minute(1))), [(1, "M")]); + changes(&mut feed, 1, &[0, 1]); + assert!(resyncs(&feed.tick(minute(2))).is_empty()); + changes(&mut feed, 2, &[0]); + let again = feed.tick(minute(3)); + assert_eq!(resyncs(&again), [(1, "M")]); + assert!( + !again + .iter() + .any(|effect| matches!(effect, Effect::Report(feed::Report::Quiet { .. }))), + "{again:?}" + ); + assert_eq!(feed.shell.quiet, 2); + } + /// The demo's recording on two connections (tests/fixtures/README.md). fn recording() -> Vec<(DateTime, usize, Event)> { #[derive(Deserialize)] @@ -871,6 +1020,26 @@ mod tests { assert!(two_sided >= 10, "{two_sided}"); } + /// A market removed and tracked again: its first book's Snapshot says + /// so, once. + #[test] + fn a_market_tracked_again_says_so_in_its_first_snapshot() { + let mut feed = feed_of(&["A"], &[0]); + let first = feed.from(0, 1, book("A", 1, &[(400_000, 1_000_000)], &[])); + assert_eq!(snapshots(&first), 1); + assert_eq!(feed.shell.symbols["A"].again, None); + let a = market("A"); + feed.remove_market(&a); + feed.add_market(&a, &[0], at(10)); + assert_eq!(feed.shell.symbols["A"].again, Some("kalshi")); + let back = feed.from(0, 20, book("A", 9, &[(500_000, 2_000_000)], &[])); + assert_eq!(snapshots(&back), 1); + assert_eq!(feed.shell.symbols["A"].again, None); + // A market new to the run is not "again". + feed.add_market(&market("B"), &[0], at(20)); + assert_eq!(feed.shell.symbols["B"].again, None); + } + /// The book and writer threads as main runs them, fed the recording: /// every Snapshot is the one before it with the Deltas between. #[test] diff --git a/src/kalshi/listing.rs b/src/kalshi/listing.rs index 07cf99f..7a06148 100644 --- a/src/kalshi/listing.rs +++ b/src/kalshi/listing.rs @@ -1,6 +1,7 @@ -//! Kalshi's market listing: which markets are live, or about to be. Two fetcher threads page -//! it on the orchestrator's request and stream what they read; they hold no -//! state and no timer, as Gamma's (roadmap/06). Read without a key. +//! Kalshi's market listing: which markets are live, or about to be. Two +//! fetcher threads page it on the orchestrator's request and send on what +//! they read; they hold no state and no timer, as Gamma's (roadmap/06). Read +//! without a key. use std::time::Duration; diff --git a/src/kalshi/mod.rs b/src/kalshi/mod.rs index d2088f4..aa8204e 100644 --- a/src/kalshi/mod.rs +++ b/src/kalshi/mod.rs @@ -15,8 +15,9 @@ use tungstenite::client::IntoClientRequest; use crate::core::{self, Finding, Request, Undecoded}; use sign::Key; -/// Tickers per subscribe or unsubscribe message. -const CHUNK: usize = 500; +/// Tickers per subscribe or unsubscribe message, as Polymarket's (whose +/// limit is known, docs/notes.md); Kalshi documents none. +const PER_MESSAGE: usize = 500; /// Errors after which the subscription is gone (Kalshi's docs: a channel /// error, a subscription buffer overflow): the connection starts over. const TERMINAL_ERRORS: [u32; 2] = [10, 25]; @@ -74,7 +75,7 @@ impl core::Exchange for Kalshi { fn request(&self) -> anyhow::Result { let mut request = self.websocket.as_str().into_client_request()?; - for (name, value) in self.key.headers(Utc::now().timestamp_millis()) { + for (name, value) in self.key.headers(Utc::now().timestamp_millis())? { request.headers_mut().insert(name, value.parse()?); } Ok(request) @@ -125,7 +126,7 @@ impl core::Session for Session { /// default is announced to flip. fn subscribe(&mut self, tickers: &[&String]) -> Vec { tickers - .chunks(CHUNK) + .chunks(PER_MESSAGE) .map(|chunk| { let params = json!({ "channels": ["orderbook_delta"], @@ -143,7 +144,7 @@ impl core::Session for Session { // unsubscribe sent before its answer names it so. let sid = self.sid.unwrap_or(1); tickers - .chunks(CHUNK) + .chunks(PER_MESSAGE) .map(|chunk| { let params = json!({ "sids": [sid], diff --git a/src/kalshi/sign.rs b/src/kalshi/sign.rs index bd2fa0d..68c650f 100644 --- a/src/kalshi/sign.rs +++ b/src/kalshi/sign.rs @@ -1,11 +1,13 @@ //! The key that signs Kalshi's websocket handshake (docs/notes.md, Kalshi's -//! websocket): an Ed25519 key, read once from the files that the environment -//! names. Neither file's content is ever logged. +//! websocket): Ed25519, or RSA signed with RSA-PSS and SHA-256, the two +//! kinds Kalshi takes; read once from the files that the environment names. +//! Neither file's content is ever logged. use std::path::Path; use base64::Engine; -use ring::signature::Ed25519KeyPair; +use ring::rand::SystemRandom; +use ring::signature::{Ed25519KeyPair, RSA_PSS_SHA256, RsaKeyPair}; use rustls_pki_types::PrivateKeyDer; use rustls_pki_types::pem::PemObject; @@ -18,15 +20,33 @@ const SIGNED_PATH: &str = "GET/trade-api/ws/v2"; pub struct Key { id: String, - pair: Ed25519KeyPair, + pair: Pair, +} + +/// Kalshi tells the two apart by the public key it holds; the PEM's header +/// doesn't say (an RSA key can be PKCS#8 too), so the parsed key does. +enum Pair { + Ed25519(Ed25519KeyPair), + Rsa(RsaKeyPair), } impl Key { /// The key from the files [`KEY_ID_FILE`] and [`PRIVATE_KEY_FILE`] name. pub fn from_env() -> anyhow::Result { - let path = |variable: &str| { - std::env::var_os(variable) - .ok_or_else(|| anyhow::anyhow!("{variable} is not set: it names a file of the key")) + Self::from_variables(|name| std::env::var_os(name)) + } + + /// [`Self::from_env`], the variables read through `variable`. + fn from_variables( + variable: impl Fn(&str) -> Option, + ) -> anyhow::Result { + let path = |name: &str| { + variable(name).ok_or_else(|| { + anyhow::anyhow!( + "{name} is not set: Kalshi needs {KEY_ID_FILE} and {PRIVATE_KEY_FILE}, \ + the files of its key id and its private key" + ) + }) }; Self::from_files( path(KEY_ID_FILE)?.as_ref(), @@ -37,7 +57,8 @@ impl Key { }) } - /// The key id in `id_file`, the private key (a PKCS#8 PEM) in `key_file`. + /// The key id in `id_file`, the private key in `key_file`: a PEM, + /// Ed25519 in PKCS#8, or RSA in PKCS#1 (as Kalshi gives it) or PKCS#8. pub fn from_files(id_file: &Path, key_file: &Path) -> anyhow::Result { let id = std::fs::read_to_string(id_file) .map_err(|error| anyhow::anyhow!("{}: {error}", id_file.display()))? @@ -50,28 +71,54 @@ impl Key { // error quotes what it read, so it is dropped. let der = PrivateKeyDer::from_pem_slice(&pem) .map_err(|_| anyhow::anyhow!("{}: not a PEM private key", key_file.display()))?; - let PrivateKeyDer::Pkcs8(der) = der else { - anyhow::bail!( - "{}: not a PKCS#8 key; Kalshi's Ed25519 keys are", - key_file.display() - ); + let pair = match &der { + PrivateKeyDer::Pkcs8(der) => { + Ed25519KeyPair::from_pkcs8_maybe_unchecked(der.secret_pkcs8_der()) + .map(Pair::Ed25519) + .ok() + .or_else(|| { + RsaKeyPair::from_pkcs8(der.secret_pkcs8_der()) + .ok() + .map(Pair::Rsa) + }) + } + PrivateKeyDer::Pkcs1(der) => RsaKeyPair::from_der(der.secret_pkcs1_der()) + .ok() + .map(Pair::Rsa), + _ => None, }; - let pair = Ed25519KeyPair::from_pkcs8_maybe_unchecked(der.secret_pkcs8_der()) - .map_err(|_| anyhow::anyhow!("{}: not an Ed25519 key", key_file.display()))?; + let pair = pair.ok_or_else(|| { + anyhow::anyhow!("{}: neither an Ed25519 nor an RSA key", key_file.display()) + })?; Ok(Self { id, pair }) } /// The handshake's three headers at `millis` since the epoch. - pub fn headers(&self, millis: i64) -> [(&'static str, String); 3] { - let signature = self.pair.sign(format!("{millis}{SIGNED_PATH}").as_bytes()); - [ + pub fn headers(&self, millis: i64) -> anyhow::Result<[(&'static str, String); 3]> { + let signed = format!("{millis}{SIGNED_PATH}"); + let signature = match &self.pair { + Pair::Ed25519(pair) => pair.sign(signed.as_bytes()).as_ref().to_vec(), + Pair::Rsa(pair) => { + // Salted: a new signature each time, from the system's random. + let mut signature = vec![0; pair.public().modulus_len()]; + pair.sign( + &RSA_PSS_SHA256, + &SystemRandom::new(), + signed.as_bytes(), + &mut signature, + ) + .map_err(|_| anyhow::anyhow!("the RSA signature failed"))?; + signature + } + }; + Ok([ ("KALSHI-ACCESS-KEY", self.id.clone()), ("KALSHI-ACCESS-TIMESTAMP", millis.to_string()), ( "KALSHI-ACCESS-SIGNATURE", - base64::engine::general_purpose::STANDARD.encode(signature.as_ref()), + base64::engine::general_purpose::STANDARD.encode(signature), ), - ] + ]) } } @@ -92,7 +139,9 @@ pub fn test_key() -> Key { }; Key { id: "a-key-id".into(), - pair: Ed25519KeyPair::from_pkcs8_maybe_unchecked(der.secret_pkcs8_der()).unwrap(), + pair: Pair::Ed25519( + Ed25519KeyPair::from_pkcs8_maybe_unchecked(der.secret_pkcs8_der()).unwrap(), + ), } } @@ -120,7 +169,7 @@ mod tests { &write(&dir, "key.pem", TEST_KEY), ) .unwrap(); - let [id, time, signature] = key.headers(1_669_149_841_234); + let [id, time, signature] = key.headers(1_669_149_841_234).unwrap(); assert_eq!(id, ("KALSHI-ACCESS-KEY", "a-key-id".to_string())); assert_eq!( time, @@ -143,10 +192,122 @@ mod tests { .verify(b"1669149841234GET/trade-api/ws/v2", &bytes) .unwrap(); // Ed25519 is deterministic: the same time, the same signature. - assert_eq!(key.headers(1_669_149_841_234)[2], signature); + assert_eq!(key.headers(1_669_149_841_234).unwrap()[2], signature); std::fs::remove_dir_all(dir).unwrap(); } + /// RFC 9500's "testRSA2048" (section 2.1), a key published for tests, + /// in PKCS#1 as Kalshi gives it. + const RSA_PKCS1: &str = "-----BEGIN RSA PRIVATE KEY----- +MIIEowIBAAKCAQEAsPnoGUOnrpiSqt4XynxA+HRP7S+BSObI6qJ7fQAVSPtRkqso +tWxQYLEYzNEx5ZSHTGypibVsJylvCfuToDTfMul8b/CZjP2Ob0LdpYrNH6l5hvFE +89FU1nZQF15oVLOpUgA7wGiHuEVawrGfey92UE68mOyUVXGweJIVDdxqdMoPvNNU +l86BU02vlBiESxOuox+dWmuVV7vfYZ79Toh/LUK43YvJh+rhv4nKuF7iHjVjBd9s +B6iDjj70HFldzOQ9r8SRI+9NirupPTkF5AKNe6kUhKJ1luB7S27ZkvB3tSTT3P59 +3VVJvnzOjaA1z6Cz+4+eRvcysqhrRgFlwI9TEwIDAQABAoIBAEEYiyDP29vCzx/+ +dS3LqnI5BjUuJhXUnc6AWX/PCgVAO+8A+gZRgvct7PtZb0sM6P9ZcLrweomlGezI +FrL0/6xQaa8bBr/ve/a8155OgcjFo6fZEw3Dz7ra5fbSiPmu4/b/kvrg+Br1l77J +aun6uUAs1f5B9wW+vbR7tzbT/mxaUeDiBzKpe15GwcvbJtdIVMa2YErtRjc1/5B2 +BGVXyvlJv0SIlcIEMsHgnAFOp1ZgQ08aDzvilLq8XVMOahAhP1O2A3X8hKdXPyrx +IVWE9bS9ptTo+eF6eNl+d7htpKGEZHUxinoQpWEBTv+iOoHsVunkEJ3vjLP3lyI/ +fY0NQ1ECgYEA3RBXAjgvIys2gfU3keImF8e/TprLge1I2vbWmV2j6rZCg5r/AS0u +pii5CvJ5/T5vfJPNgPBy8B/yRDs+6PJO1GmnlhOkG9JAIPkv0RBZvR0PMBtbp6nT +Y3yo1lwamBVBfY6rc0sLTzosZh2aGoLzrHNMQFMGaauORzBFpY5lU50CgYEAzPHl +u5DI6Xgep1vr8QvCUuEesCOgJg8Yh1UqVoY/SmQh6MYAv1I9bLGwrb3WW/7kqIoD +fj0aQV5buVZI2loMomtU9KY5SFIsPV+JuUpy7/+VE01ZQM5FdY8wiYCQiVZYju9X +Wz5LxMNoz+gT7pwlLCsC4N+R8aoBk404aF1gum8CgYAJ7VTq7Zj4TFV7Soa/T1eE +k9y8a+kdoYk3BASpCHJ29M5R2KEA7YV9wrBklHTz8VzSTFTbKHEQ5W5csAhoL5Fo +qoHzFFi3Qx7MHESQb9qHyolHEMNx6QdsHUn7rlEnaTTyrXh3ifQtD6C0yTmFXUIS +CW9wKApOrnyKJ9nI0HcuZQKBgQCMtoV6e9VGX4AEfpuHvAAnMYQFgeBiYTkBKltQ +XwozhH63uMMomUmtSG87Sz1TmrXadjAhy8gsG6I0pWaN7QgBuFnzQ/HOkwTm+qKw +AsrZt4zeXNwsH7QXHEJCFnCmqw9QzEoZTrNtHJHpNboBuVnYcoueZEJrP8OnUG3r +UjmopwKBgAqB2KYYMUqAOvYcBnEfLDmyZv9BTVNHbR2lKkMYqv5LlvDaBxVfilE0 +2riO4p6BaAdvzXjKeRrGNEKoHNBpOSfYCOM16NjL8hIZB1CaV3WbT5oY+jp7Mzd5 +7d56RZOE+ERK2uz/7JX9VSsM/LbH9pJibd4e8mikDS9ntciqOH/3 +-----END RSA PRIVATE KEY----- +"; + /// The same key in PKCS#8 (`openssl pkcs8 -topk8 -nocrypt`). + const RSA_PKCS8: &str = "-----BEGIN PRIVATE KEY----- +MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCw+egZQ6eumJKq +3hfKfED4dE/tL4FI5sjqont9ABVI+1GSqyi1bFBgsRjM0THllIdMbKmJtWwnKW8J ++5OgNN8y6Xxv8JmM/Y5vQt2lis0fqXmG8UTz0VTWdlAXXmhUs6lSADvAaIe4RVrC +sZ97L3ZQTryY7JRVcbB4khUN3Gp0yg+801SXzoFTTa+UGIRLE66jH51aa5VXu99h +nv1OiH8tQrjdi8mH6uG/icq4XuIeNWMF32wHqIOOPvQcWV3M5D2vxJEj702Ku6k9 +OQXkAo17qRSEonWW4HtLbtmS8He1JNPc/n3dVUm+fM6NoDXPoLP7j55G9zKyqGtG +AWXAj1MTAgMBAAECggEAQRiLIM/b28LPH/51LcuqcjkGNS4mFdSdzoBZf88KBUA7 +7wD6BlGC9y3s+1lvSwzo/1lwuvB6iaUZ7MgWsvT/rFBprxsGv+979rzXnk6ByMWj +p9kTDcPPutrl9tKI+a7j9v+S+uD4GvWXvslq6fq5QCzV/kH3Bb69tHu3NtP+bFpR +4OIHMql7XkbBy9sm10hUxrZgSu1GNzX/kHYEZVfK+Um/RIiVwgQyweCcAU6nVmBD +TxoPO+KUurxdUw5qECE/U7YDdfyEp1c/KvEhVYT1tL2m1Oj54Xp42X53uG2koYRk +dTGKehClYQFO/6I6gexW6eQQne+Ms/eXIj99jQ1DUQKBgQDdEFcCOC8jKzaB9TeR +4iYXx79OmsuB7Uja9taZXaPqtkKDmv8BLS6mKLkK8nn9Pm98k82A8HLwH/JEOz7o +8k7UaaeWE6Qb0kAg+S/REFm9HQ8wG1unqdNjfKjWXBqYFUF9jqtzSwtPOixmHZoa +gvOsc0xAUwZpq45HMEWljmVTnQKBgQDM8eW7kMjpeB6nW+vxC8JS4R6wI6AmDxiH +VSpWhj9KZCHoxgC/Uj1ssbCtvdZb/uSoigN+PRpBXlu5VkjaWgyia1T0pjlIUiw9 +X4m5SnLv/5UTTVlAzkV1jzCJgJCJVliO71dbPkvEw2jP6BPunCUsKwLg35HxqgGT +jThoXWC6bwKBgAntVOrtmPhMVXtKhr9PV4ST3Lxr6R2hiTcEBKkIcnb0zlHYoQDt +hX3CsGSUdPPxXNJMVNsocRDlblywCGgvkWiqgfMUWLdDHswcRJBv2ofKiUcQw3Hp +B2wdSfuuUSdpNPKteHeJ9C0PoLTJOYVdQhIJb3AoCk6ufIon2cjQdy5lAoGBAIy2 +hXp71UZfgAR+m4e8ACcxhAWB4GJhOQEqW1BfCjOEfre4wyiZSa1IbztLPVOatdp2 +MCHLyCwbojSlZo3tCAG4WfND8c6TBOb6orACytm3jN5c3CwftBccQkIWcKarD1DM +ShlOs20ckek1ugG5Wdhyi55kQms/w6dQbetSOainAoGACoHYphgxSoA69hwGcR8s +ObJm/0FNU0dtHaUqQxiq/kuW8NoHFV+KUTTauI7inoFoB2/NeMp5GsY0Qqgc0Gk5 +J9gI4zXo2MvyEhkHUJpXdZtPmhj6OnszN3nt3npFk4T4REra7P/slf1VKwz8tsf2 +kmJt3h7yaKQNL2e1yKo4f/c= +-----END PRIVATE KEY----- +"; + + #[test] + fn signs_with_an_rsa_key_in_pss_and_sha256() { + let dir = scratch("rsa"); + for (name, pem) in [("pkcs1.pem", RSA_PKCS1), ("pkcs8.pem", RSA_PKCS8)] { + let key = Key::from_files(&write(&dir, "id", "an-rsa-key\n"), &write(&dir, name, pem)) + .unwrap(); + let Pair::Rsa(pair) = &key.pair else { + panic!("{name}: read as Ed25519"); + }; + let [id, _, signature] = key.headers(1_669_149_841_234).unwrap(); + assert_eq!(id.1, "an-rsa-key"); + let bytes = base64::engine::general_purpose::STANDARD + .decode(&signature.1) + .unwrap(); + let public = ring::signature::UnparsedPublicKey::new( + &ring::signature::RSA_PSS_2048_8192_SHA256, + pair.public().as_ref(), + ); + public + .verify(b"1669149841234GET/trade-api/ws/v2", &bytes) + .unwrap(); + // Not PKCS#1 v1.5, which Kalshi doesn't take. + let pkcs1v15 = ring::signature::UnparsedPublicKey::new( + &ring::signature::RSA_PKCS1_2048_8192_SHA256, + pair.public().as_ref(), + ); + assert!( + pkcs1v15 + .verify(b"1669149841234GET/trade-api/ws/v2", &bytes) + .is_err() + ); + } + std::fs::remove_dir_all(dir).unwrap(); + } + + #[test] + fn a_missing_variable_names_both() { + let error = Key::from_variables(|_| None).err().unwrap().to_string(); + assert!( + error.starts_with("KALSHI_KEY_ID_FILE is not set"), + "{error}" + ); + assert!(error.contains("KALSHI_PRIVATE_KEY_FILE"), "{error}"); + let only_id = |name: &str| (name == KEY_ID_FILE).then(|| "id".into()); + let error = Key::from_variables(only_id).err().unwrap().to_string(); + assert!( + error.starts_with("KALSHI_PRIVATE_KEY_FILE is not set"), + "{error}" + ); + } + #[test] fn a_missing_or_wrong_file_says_which_and_never_what_it_holds() { let dir = scratch("errors"); @@ -170,13 +331,18 @@ mod tests { "{unreadable}" ); - // An RSA key in PKCS#1, as Kalshi generates its RSA keys: refused. + // An RSA header over what isn't an RSA key: refused, its content + // unquoted. let rsa = write( &dir, "rsa.pem", "-----BEGIN RSA PRIVATE KEY-----\nMAA=\n-----END RSA PRIVATE KEY-----\n", ); - assert!(error(&id, &rsa).contains("not a PKCS#8 key")); + let refused = error(&id, &rsa); + assert!( + refused.ends_with("neither an Ed25519 nor an RSA key"), + "{refused}" + ); let empty = write(&dir, "empty-id", "\n"); let key = write(&dir, "key.pem", TEST_KEY); diff --git a/src/lib.rs b/src/lib.rs index 039e6f5..38400ef 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -1,6 +1,7 @@ pub mod book; pub mod cause; pub mod reconcile; +pub mod show; pub mod util; mod side; diff --git a/src/main.rs b/src/main.rs index 6b029c2..8bf75ff 100644 --- a/src/main.rs +++ b/src/main.rs @@ -29,13 +29,25 @@ mod polymarket; use engineering_challenge::book::Output; /// `AlignedSbeReader::open` reads nothing else. -const CHUNK: usize = 64 << 20; +const OUTPUT_CHUNK: usize = 64 << 20; -const USAGE: &str = "usage: engineering-challenge [OUTPUT], or latency OUTPUT"; +const USAGE: &str = "\ +usage: engineering-challenge EXCHANGE [OUTPUT] + EXCHANGE polymarket (needs no key), kalshi, or kalshi-demo (Kalshi's demo) + OUTPUT the SBE output, stdout by default; the log goes to stderr + Kalshi's key: KALSHI_KEY_ID_FILE and KALSHI_PRIVATE_KEY_FILE name the files + of its id and its private key (a PEM, Ed25519 or RSA). + SIGINT, SIGTERM or SIGHUP stops it and flushes the output. +or: engineering-challenge latency EXCHANGE OUT.csv, the latency probe + (LATENCY.md); SIGINT stops it."; fn main() -> anyhow::Result<()> { let mut args = std::env::args().skip(1); let exchange = args.next().ok_or_else(|| anyhow::anyhow!(USAGE))?; + if exchange == "--help" || exchange == "-h" { + println!("{USAGE}"); + return Ok(()); + } if exchange == "latency" { return latency::run(args); } @@ -53,7 +65,7 @@ fn main() -> anyhow::Result<()> { output, ), "kalshi-demo" => collect(kalshi::Kalshi::demo(kalshi::sign::Key::from_env()?), output), - _ => anyhow::bail!("unknown exchange {exchange:?}; {USAGE}"), + _ => anyhow::bail!("unknown exchange {exchange:?}\n{USAGE}"), } } @@ -87,12 +99,12 @@ fn collect(exchange: E, output: String) -> anyhow::Result<()> /// The writer thread: writes outputs in arrival order until every book /// thread is gone, then flushes the last chunk. -// ponytail: the chunk being filled lives only in memory until it is full, so +// Ceiling: the chunk being filled lives only in memory until it is full, so // a hard crash loses it; flushing more often pads every flush to 64 MiB. -// Kept as a ceiling in M4 (roadmap/04): writing to a file of our own and -// copying it into chunks would lift it. +// Accepted for redundancy's sake too (docs/project.md, Constraints): writing +// to a file of our own and copying it into chunks would lift it. fn write(path: &str, outputs: Receiver) -> anyhow::Result<()> { - let mut writer = AlignedSbeWriter::create(path, CHUNK)?; + let mut writer = AlignedSbeWriter::create(path, OUTPUT_CHUNK)?; for output in outputs { match output { Output::Snapshot(snapshot) => writer.write_msg(&snapshot)?, diff --git a/src/polymarket/feed.rs b/src/polymarket/feed.rs index 1b9dd06..91e2281 100644 --- a/src/polymarket/feed.rs +++ b/src/polymarket/feed.rs @@ -28,7 +28,7 @@ use engineering_challenge::book::{Book, Output, format_fixed, parse_fixed}; /// so is a fresh book behind our stream. const RECENT_CHANGES: usize = 128; /// How long a change may be held while another connection might still deliver -/// the one before it: the second delivery lagged up to 0.7 s. +/// the one before it: the second delivery was up to 0.7 s late. const HOLD: TimeDelta = TimeDelta::seconds(1); /// How long once no other connection can: only the rest of a batch is still to /// come, from the same connection (40 ms at worst). @@ -91,12 +91,6 @@ impl Held { } } -impl RecentChange { - fn is_copy(&self, change: &Held) -> bool { - (self.hash, self.side, self.price, self.quantity) == change.key() - } -} - /// A change as applied, with the exchange's hash of the book after it. struct RecentChange { side: Side, @@ -115,6 +109,12 @@ struct RecentChange { counted: bool, } +impl RecentChange { + fn is_copy(&self, change: &Held) -> bool { + (self.hash, self.side, self.price, self.quantity) == change.key() + } +} + /// A symbol's state for the rule. pub struct Chained { state: State, @@ -524,7 +524,7 @@ impl Chain { }; if reconnect { self.counts.reconnect_books += 1; - // The exchange's opening books lag by minutes (docs/notes.md): one + // The exchange's opening books trail by minutes (docs/notes.md): one // that is ours, or a state we passed, leaves ours, and the hash // inputs, as they are. let brought = symbol @@ -537,7 +537,7 @@ impl Chain { return; } // Our chain holds up to our last change, so a book no newer than - // it holds nothing we missed, however far back it lags: further + // it holds nothing we missed, however far behind it is: further // than the changes we remember, too. if exchange_timestamp <= symbol.exchange_timestamp || symbol.was_ours(&book) { self.counts.books_behind += 1; @@ -583,7 +583,7 @@ impl Chain { let mut book = book; let book_time = exchange_timestamp; let mut exchange_timestamp = exchange_timestamp; - // The book a resync brings can lag the changes already applied, with a + // The book a resync brings can trail the changes already applied, with a // newer timestamp or last trade price than its levels (docs/notes.md). // It is behind if it is ours as it was a few changes ago: that past // state was the exchange's own, and the changes since are what it @@ -621,7 +621,8 @@ impl Chain { Some(Cause::Loss) => self.counts.losses += 1, Some(Cause::SubscriptionGap) => self.counts.subscription_gaps += 1, Some(Cause::Switch) => self.counts.switches += 1, - None => {} + // Not found here: the shell says it, at the Snapshot. + Some(Cause::TrackedAgain) | None => {} } symbol.book = book; symbol.exchange_timestamp = exchange_timestamp; @@ -633,11 +634,7 @@ impl Chain { // Our changes before this book no longer lead to ours: rewinding // through them would make up states that never existed. symbol.rule.recent_changes.clear(); - let snapshot = symbol - .book - .snapshot(&token, received_at, exchange_timestamp); - effects.push(Effect::Write(snapshot)); - symbol.last_written = symbol.last_written.max(received_at); + effects.push(Effect::Write(symbol.snapshot(&token, received_at))); if let Some(cause) = cause { let found = if reconnect { ", found by a reconnect" @@ -1370,7 +1367,7 @@ mod tests { (token.clone(), free) } - // --- one connection: sprint 02's rules still hold --- + // --- one connection: the hash check alone --- /// Split batches and trades make checks wait or fail with nothing lost: /// the recording must replay without a single loss. @@ -1980,6 +1977,38 @@ mod tests { assert_eq!(feed.rule.counts.rescued, rescued + 1); } + /// A market removed while a change of it waits: the book thread stops + /// waking for it. + #[test] + fn a_market_removed_while_a_change_waits_no_longer_wakes_the_book_thread() { + let (mut feed, token, book, free, millis, at) = two_connection_symbol(); + let symbol = &feed.shell.symbols[&token]; + let (_, after_x) = changes_frame( + &token, + symbol, + &book, + &[(Side::Buy, free, 1_000_000)], + millis, + ); + let (y, _) = changes_frame( + &token, + symbol, + &after_x, + &[(Side::Buy, free + 1_000, 2_000_000)], + millis + 1, + ); + assert!(feed.on_frame_from(0, at, &y).is_empty()); + assert!(feed.rule.wakes_soon()); + let markets = fixture_markets(); + let market = markets + .iter() + .find(|market| market.tokens.contains(&token)) + .unwrap(); + feed.remove_market(market); + assert!(!feed.rule.waiting.contains(&token)); + assert!(!feed.rule.wakes_soon()); + } + /// A burst on a connection that missed the change before it: the held /// changes stop at the cap, the chain breaks, and it is all quick. #[test] @@ -2276,10 +2305,13 @@ mod tests { assert_eq!(feed.shell.quiet, 1); assert_eq!(feed.shell.symbols[&token].rule.state, State::Verified); // The book the resubscribe brings back is not asked for: ignored. + // Nor judged as a reconnect's: the subscription there is renewed. + let judged = feed.rule.counts.reconnect_books; let book = feed.shell.symbols[&token].book.clone(); let back = fresh_book(&token, &feed.shell.symbols[&token], &book); let effects = feed.on_frame_from(1, minute(2) + TimeDelta::seconds(1), &back); assert!(snapshot_symbols(&effects).is_empty()); + assert_eq!(feed.rule.counts.reconnect_books, judged); // Still quiet the minute after: reported for a move, not resubscribed. changes(&mut feed, &token, &[0], minute(2), free, 3); @@ -2385,14 +2417,14 @@ mod tests { let effects = reconnect_with(&mut feed, &token, &ours, later); assert!(snapshot_symbols(&effects).is_empty()); assert_eq!(feed.rule.counts.reconnect_books, 1); - // The server's opening books lag: a state we passed is no loss. + // The server's opening books trail: a state we passed is no loss. let behind = feed.rule.counts.books_behind; let effects = reconnect_with(&mut feed, &token, &before, later + TimeDelta::seconds(1)); assert!(snapshot_symbols(&effects).is_empty()); assert_eq!(feed.rule.counts.reconnect_books, 2); assert_eq!(feed.rule.counts.books_behind, behind + 1); assert_eq!(feed.shell.symbols[&token].book, ours); - // It lags further back than the changes we remember: no newer than + // It trails further back than the changes we remember: no newer than // our last change, it still holds nothing we missed. let symbol = feed.shell.symbols.get_mut(&token).unwrap(); symbol.rule.recent_changes.clear(); @@ -2719,6 +2751,31 @@ mod tests { assert_eq!(snapshot_symbols(&effects).len(), 38 * 2); } + /// Removed, then tracked again: its first books' Snapshots say so + /// (the shell's flag, cleared once written). + #[test] + fn a_market_tracked_again_says_so_in_its_first_snapshot() { + let frames = fixture_frames(); + let markets = fixture_markets(); + let mut feed = feed_of(&markets); + replay(&mut feed, &frames[..1]); + feed.remove_market(&markets[0]); + feed.add_market(&markets[0], &[0], frames[100].0); + let tokens = &markets[0].tokens; + assert!( + tokens + .iter() + .all(|token| feed.shell.symbols[token].again.is_some()) + ); + let effects = feed.on_frame(frames[100].0, &opening_books(&frames, &markets[0])); + assert_eq!(snapshot_symbols(&effects).len(), tokens.len()); + assert!( + tokens + .iter() + .all(|token| feed.shell.symbols[token].again.is_none()) + ); + } + #[test] fn each_minute_reports_the_bytes_per_second_of_every_market() { let minute = "2026-11-10T16:00:00Z".parse::>().unwrap(); diff --git a/src/polymarket/gamma.rs b/src/polymarket/gamma.rs index 167cbc0..dc43aa8 100644 --- a/src/polymarket/gamma.rs +++ b/src/polymarket/gamma.rs @@ -1,6 +1,6 @@ //! Gamma, Polymarket's market listing: which markets are live. Two fetcher -//! threads page it on the orchestrator's request and stream what they read; -//! they hold no state and no timer (roadmap/03). +//! threads page it on the orchestrator's request and send on what they +//! read; they hold no state and no timer (roadmap/03). use std::time::Duration; diff --git a/src/polymarket/mod.rs b/src/polymarket/mod.rs index f10273e..0077e3d 100644 --- a/src/polymarket/mod.rs +++ b/src/polymarket/mod.rs @@ -15,7 +15,7 @@ use crate::core::{self, Finding, Request, Undecoded}; pub const MARKET_CHANNEL_URL: &str = "wss://ws-subscriptions-clob.polymarket.com/ws/market"; /// Tokens per subscribe or unsubscribe message: a first message of more than /// 500 gets no opening books (docs/notes.md). -const CHUNK: usize = 500; +const PER_MESSAGE: usize = 500; /// A live market: its symbols and the two inputs of their book hash that the /// websocket doesn't carry (docs/notes.md). @@ -78,7 +78,7 @@ impl Session { /// connection. fn operations(operation: &str, tokens: &[&String]) -> Vec { tokens - .chunks(CHUNK) + .chunks(PER_MESSAGE) .map(|chunk| json!({"assets_ids": chunk, "operation": operation}).to_string()) .collect() } diff --git a/src/polymarket/socket_tests.rs b/src/polymarket/socket_tests.rs index ed75c68..447c716 100644 --- a/src/polymarket/socket_tests.rs +++ b/src/polymarket/socket_tests.rs @@ -2,22 +2,48 @@ //! Polymarket's codec. use std::collections::BTreeSet; +use std::io::ErrorKind; use std::net::{TcpListener, TcpStream}; use std::sync::Arc; use std::sync::atomic::{AtomicBool, Ordering}; use std::thread; -use std::time::Duration; +use std::time::{Duration, Instant}; use chrono::Utc; use crossbeam::channel::{Sender, unbounded}; use serde_json::json; +use tungstenite::protocol::CloseFrame; +use tungstenite::protocol::frame::coding::CloseCode; use tungstenite::{Message, WebSocket}; -use super::{CHUNK, Polymarket, Session}; +use super::{PER_MESSAGE, Polymarket, Session}; use crate::core::Event; -use crate::core::socket::{Command, Notice, Routed, SILENT_AFTER, apply, route, run}; +use crate::core::socket::{Command, Notice, PING_EVERY, Routed, SILENT_AFTER, apply, route, run}; use crate::core::{BOOK_THREADS, Received, book_thread}; +/// The next connection to `listener`, within 5 s: a client that never +/// connects (again) fails the test rather than hangs it. +fn accept(listener: &TcpListener) -> TcpStream { + listener.set_nonblocking(true).unwrap(); + let deadline = Instant::now() + Duration::from_secs(5); + loop { + match listener.accept() { + Ok((stream, _)) => { + stream.set_nonblocking(false).unwrap(); + stream + .set_read_timeout(Some(Duration::from_secs(5))) + .unwrap(); + return stream; + } + Err(error) if error.kind() == ErrorKind::WouldBlock => { + assert!(Instant::now() < deadline, "no connection within 5 s"); + thread::sleep(Duration::from_millis(10)); + } + Err(error) => panic!("accept: {error}"), + } + } +} + /// The tokens each message of one connection subscribed, after the first. fn subscribed(socket: &mut WebSocket, tokens: usize) -> Vec> { let first = socket.read().unwrap().into_text().unwrap(); @@ -83,7 +109,7 @@ fn a_reconnect_subscribes_every_token_again_in_chunks() { let mut tokens = tokens; tokens.sort(); for chunks in connections { - assert!(chunks.iter().all(|chunk| chunk.len() <= CHUNK)); + assert!(chunks.iter().all(|chunk| chunk.len() <= PER_MESSAGE)); let mut subscribed = chunks.concat(); subscribed.sort(); assert_eq!(subscribed, tokens); @@ -102,11 +128,7 @@ fn serving_twice( let server = thread::spawn(move || { (0..2) .map(|_| { - let (stream, _) = listener.accept().unwrap(); - stream - .set_read_timeout(Some(Duration::from_secs(5))) - .unwrap(); - let mut socket = tungstenite::accept(stream).unwrap(); + let mut socket = tungstenite::accept(accept(&listener)).unwrap(); let chunks = subscribed(&mut socket, total); serve(&mut socket); chunks @@ -270,3 +292,146 @@ fn commands_update_the_token_set_and_say_what_to_send() { assert!(apply(&mut session, &mut tokens, &Command::Resubscribe("a".into())).is_empty()); assert_eq!(tokens.into_iter().collect::>(), ["b"]); } + +/// A local server for one connection of one token: once subscribed, `serve` +/// handles it and sends what it found on the channel returned. +fn serving_once( + serve: impl FnOnce(&mut WebSocket, &Sender) + Send + 'static, +) -> (String, crossbeam::channel::Receiver) { + let listener = TcpListener::bind("127.0.0.1:0").unwrap(); + let url = format!("ws://{}/ws/market", listener.local_addr().unwrap()); + let (found, found_rx) = unbounded(); + thread::spawn(move || { + let mut socket = tungstenite::accept(accept(&listener)).unwrap(); + subscribed(&mut socket, 1); + serve(&mut socket, &found); + while socket.read().is_ok() {} + }); + (url, found_rx) +} + +/// Every other socket test queues its Subscribe before the connection is up. +#[test] +fn a_command_sent_on_an_open_connection_reaches_the_server() { + let (url, sent) = serving_once(|socket, sent| { + sent.send(String::new()).unwrap(); + // The keepalive comes only after 10 s: the next text is the command. + let text = socket.read().unwrap().into_text().unwrap(); + sent.send(text.to_string()).unwrap(); + }); + client(url, vec!["a".into()], SILENT_AFTER, |commands| { + // Subscribed: the connection is open. + sent.recv_timeout(Duration::from_secs(5)).unwrap(); + commands.send(Command::Subscribe(vec!["b".into()])).unwrap(); + let text = sent.recv_timeout(Duration::from_secs(5)).unwrap(); + let message: serde_json::Value = serde_json::from_str(&text).unwrap(); + assert_eq!( + message, + json!({"assets_ids": ["b"], "operation": "subscribe"}) + ); + }); +} + +#[test] +fn a_close_with_1013_tells_the_orchestrator_the_connection_is_too_slow() { + let (url, closed) = serving_once(|socket, closed| { + let frame = CloseFrame { + code: CloseCode::Again, + reason: "".into(), + }; + socket.close(Some(frame)).unwrap(); + closed.send(()).unwrap(); + }); + // As `client` does, but waiting for the notices themselves, each under + // a timeout, rather than a sleep before reading them. + let (events_tx, _events_rx) = unbounded(); + let (commands_tx, commands_rx) = unbounded(); + commands_tx + .send(Command::Subscribe(vec!["a".into()])) + .unwrap(); + let stop = Arc::new(AtomicBool::new(false)); + let client_stop = Arc::clone(&stop); + let (notices, notices_rx) = unbounded(); + let client = thread::spawn(move || { + run( + &Polymarket { url }, + 0, + &[events_tx], + &commands_rx, + ¬ices, + SILENT_AFTER, + &client_stop, + ) + }); + closed.recv_timeout(Duration::from_secs(5)).unwrap(); + let first: Vec = (0..3) + .map(|_| notices_rx.recv_timeout(Duration::from_secs(5)).unwrap()) + .collect(); + stop.store(true, Ordering::Relaxed); + client.join().unwrap(); + assert_eq!(first, [Notice::Up(0), Notice::TooSlow(0), Notice::Down(0)]); +} + +#[test] +fn a_connection_sends_its_keepalive_every_ten_seconds() { + let (url, received) = serving_once(|socket, received| { + let stream = socket.get_ref(); + stream + .set_read_timeout(Some(PING_EVERY + Duration::from_secs(5))) + .unwrap(); + let text = socket.read().unwrap().into_text().unwrap(); + received.send(text.to_string()).unwrap(); + }); + client(url, vec!["a".into()], SILENT_AFTER, |_| { + let text = received + .recv_timeout(PING_EVERY + Duration::from_secs(5)) + .unwrap(); + assert_eq!(text, "PING"); + }); +} + +/// Only a panic ends a book thread before its connections: the run stops. +#[test] +fn a_book_thread_gone_stops_the_connection_and_the_run() { + #[derive(serde::Deserialize)] + struct Line { + text: String, + } + let path = concat!( + env!("CARGO_MANIFEST_DIR"), + "/tests/fixtures/polymarket-frames.jsonl" + ); + let recording = std::fs::read_to_string(path).unwrap(); + let line: Line = serde_json::from_str(recording.lines().next().unwrap()).unwrap(); + let (url, _sent) = serving_once(move |socket, sent| { + socket.send(Message::text(line.text)).unwrap(); + sent.send(()).unwrap(); + }); + let (book_threads, receivers): (Vec<_>, Vec<_>) = + (0..BOOK_THREADS).map(|_| unbounded()).unzip(); + drop(receivers); + let (commands_tx, commands) = unbounded(); + commands_tx + .send(Command::Subscribe(vec!["a".into()])) + .unwrap(); + let (notices, _notices_rx) = unbounded(); + let stop = Arc::new(AtomicBool::new(false)); + let client_stop = Arc::clone(&stop); + let (ended, ended_rx) = unbounded(); + thread::spawn(move || { + run( + &Polymarket { url }, + 0, + &book_threads, + &commands, + ¬ices, + SILENT_AFTER, + &client_stop, + ); + ended.send(()).unwrap(); + }); + ended_rx + .recv_timeout(Duration::from_secs(5)) + .expect("the connection stops by itself"); + assert!(stop.load(Ordering::Relaxed), "and stops the run"); +} diff --git a/src/reconcile.rs b/src/reconcile.rs index f7ed059..af09407 100644 --- a/src/reconcile.rs +++ b/src/reconcile.rs @@ -99,10 +99,11 @@ mod tests { use crate::util::dtos::challenge_orderbook::{PriceLevelUpdateDomain, SnapshotBidsEntryDomain}; /// An output with a symbol's Snapshot, a Delta, then a Snapshot that - /// misses it; and a log line for that second Snapshot when `explained`. - fn reconcile_with(explained: bool) -> Report { + /// misses it; and a log line with `cause` for that second Snapshot when + /// `explained`. + fn reconcile_with(explained: bool, cause: Cause) -> Report { let dir = std::env::temp_dir(); - let id = format!("{}-{explained}", std::process::id()); + let id = format!("{}-{explained}-{cause}", std::process::id()).replace(' ', "-"); let (output, log) = ( dir.join(format!("reconcile-{id}.sbe")), dir.join(format!("reconcile-{id}.log")), @@ -132,7 +133,7 @@ mod tests { .unwrap(); writer.write_msg(&snapshot(at(2), 3_000_000)).unwrap(); writer.flush().unwrap(); - let line = cause::line("1", at(if explained { 2 } else { 3 }), Cause::Loss, ""); + let line = cause::line("1", at(if explained { 2 } else { 3 }), cause, ""); std::fs::write(&log, format!("12:00:00.000 polymarket: {line}\n")).unwrap(); let report = reconcile(&output, Some(&log)).unwrap(); std::fs::remove_file(output).unwrap(); @@ -140,14 +141,106 @@ mod tests { report } + fn at(seconds: i64) -> DateTime { + DateTime::from_timestamp(1_794_390_000 + seconds, 0).unwrap() + } + + fn snapshot(received_at: DateTime, quantity: i64) -> SnapshotDomain { + SnapshotDomain { + received_at, + exchange_timestamp: received_at, + bids: vec![SnapshotBidsEntryDomain { + price_level: 500_000, + quantity, + }], + asks: Vec::new(), + symbol: "1".to_string(), + } + } + + fn delta(received_at: DateTime, quantity: i64) -> PriceLevelUpdateDomain { + PriceLevelUpdateDomain { + backtest_timestamp: received_at, + exchange_timestamp: received_at, + price_level: 500_000, + quantity, + side: Side::Buy, + symbol: "1".to_string(), + } + } + + /// The report on the output `write` writes, with `log` as the log; + /// `name` keeps parallel tests apart. + fn reconciled(name: &str, write: impl FnOnce(&mut AlignedSbeWriter), log: &str) -> Report { + let dir = std::env::temp_dir(); + let id = format!("{}-{name}", std::process::id()); + let (output, log_path) = ( + dir.join(format!("reconcile-{id}.sbe")), + dir.join(format!("reconcile-{id}.log")), + ); + let mut writer = AlignedSbeWriter::create(output.to_str().unwrap(), 64 << 20).unwrap(); + write(&mut writer); + writer.flush().unwrap(); + std::fs::write(&log_path, log).unwrap(); + let report = reconcile(&output, Some(&log_path)).unwrap(); + std::fs::remove_file(output).unwrap(); + std::fs::remove_file(log_path).unwrap(); + report + } + + #[test] + fn a_delta_before_its_symbols_first_snapshot_is_an_orphan() { + let report = reconciled( + "orphan", + |writer| { + writer.write_msg(&delta(at(0), 2_000_000)).unwrap(); + writer.write_msg(&snapshot(at(1), 1_000_000)).unwrap(); + writer.write_msg(&delta(at(2), 3_000_000)).unwrap(); + }, + "", + ); + assert_eq!((report.deltas, report.orphan_deltas), (2, 1)); + } + + /// Two Snapshots of one symbol stamped alike, both mismatched: the cause + /// line explains one of them only. + #[test] + fn a_cause_line_explains_one_snapshot_only() { + let line = cause::line("1", at(2), Cause::Loss, ""); + let report = reconciled( + "once", + |writer| { + writer.write_msg(&snapshot(at(0), 1_000_000)).unwrap(); + writer.write_msg(&delta(at(1), 2_000_000)).unwrap(); + writer.write_msg(&snapshot(at(2), 3_000_000)).unwrap(); + writer.write_msg(&delta(at(2), 4_000_000)).unwrap(); + writer.write_msg(&snapshot(at(2), 5_000_000)).unwrap(); + }, + &format!("12:00:00.000 kalshi: {line}\n"), + ); + assert_eq!(report.mismatches.len(), 2); + assert_eq!(report.explained, HashMap::from([(Cause::Loss, 1)])); + assert_eq!(report.unexplained, [("1".to_string(), at(2))]); + } + #[test] fn a_mismatch_with_its_log_line_is_explained_and_one_without_is_not() { - let report = reconcile_with(true); + let report = reconcile_with(true, Cause::Loss); assert_eq!(report.mismatches.len(), 1); assert_eq!(report.explained, HashMap::from([(Cause::Loss, 1)])); assert!(report.unexplained.is_empty()); - let report = reconcile_with(false); + let report = reconcile_with(false, Cause::Loss); assert_eq!(report.unexplained.len(), 1); assert!(report.explained.is_empty()); } + + /// A market tracked again: its first Snapshot since differs from the one + /// written before it left, and its log line says why. + #[test] + fn a_market_tracked_again_explains_its_first_snapshot() { + let report = reconcile_with(true, Cause::TrackedAgain); + assert_eq!(report.mismatches.len(), 1); + assert_eq!(report.explained, HashMap::from([(Cause::TrackedAgain, 1)])); + assert!(report.unexplained.is_empty()); + } } diff --git a/src/show.rs b/src/show.rs new file mode 100644 index 0000000..716bbbf --- /dev/null +++ b/src/show.rs @@ -0,0 +1,350 @@ +//! A look at the output as text: one symbol's first Snapshot, the Deltas +//! after it, and the book they fold into. With no symbol given, the one with +//! the most Deltas in the file. + +use std::collections::HashMap; +use std::fmt::Write; +use std::path::Path; + +use chrono::{DateTime, Utc}; + +use crate::Side; +use crate::book::{Book, format_fixed}; +use crate::util::dtos::challenge_orderbook::{AnyMessage, PriceLevelUpdateDomain, SnapshotDomain}; +use crate::util::{AlignedSbeReader, OrderbookSchema}; + +/// The Deltas shown after the Snapshot. +const DELTAS: usize = 20; +/// The levels shown on each side of a book. +const LEVELS: usize = 5; + +/// The text for `symbol` in the output at `path`, or for its busiest symbol. +pub fn show(path: impl AsRef, symbol: Option<&str>) -> anyhow::Result { + let path = path.as_ref(); + let mut counts: HashMap = HashMap::new(); + each(path, |message| { + let (symbol, is_snapshot) = match message { + Message::Snapshot(snapshot) => (snapshot.symbol, true), + Message::Delta(delta) => (delta.symbol, false), + }; + let (snapshots, deltas) = counts.entry(symbol).or_default(); + if is_snapshot { + *snapshots += 1; + } else { + *deltas += 1; + } + })?; + let symbol = match symbol { + Some(symbol) if counts.contains_key(symbol) => symbol.to_string(), + Some(symbol) => anyhow::bail!("{symbol} is not in {}", path.display()), + // Ties go to the smallest symbol, so the pick doesn't depend on hashing. + None => counts + .iter() + .max_by(|a, b| a.1.1.cmp(&b.1.1).then(b.0.cmp(a.0))) + .map(|(symbol, _)| symbol.clone()) + .ok_or_else(|| anyhow::anyhow!("{} holds no message", path.display()))?, + }; + let (snapshots, deltas) = counts[&symbol]; + + let mut first: Option = None; + let mut after: Vec = Vec::new(); + each(path, |message| match message { + Message::Snapshot(snapshot) if snapshot.symbol == symbol && first.is_none() => { + first = Some(snapshot) + } + Message::Delta(delta) + if delta.symbol == symbol && first.is_some() && after.len() < DELTAS => + { + after.push(delta) + } + _ => {} + })?; + let first = first.ok_or_else(|| anyhow::anyhow!("{symbol} has no Snapshot"))?; + + let mut text = String::new(); + writeln!( + text, + "{symbol}: Snapshots {snapshots}, Deltas {deltas}, in {}", + path.display() + )?; + let mut book = Book { + bids: first + .bids + .iter() + .map(|level| (level.price_level, level.quantity)) + .collect(), + asks: first + .asks + .iter() + .map(|level| (level.price_level, level.quantity)) + .collect(), + }; + writeln!( + text, + "\nIts first Snapshot, received {} (exchange time {}):", + time(first.received_at), + time(first.exchange_timestamp) + )?; + ladder(&mut text, &book)?; + writeln!(text, "\nThe {} Deltas after it:", after.len())?; + for delta in &after { + let side = match delta.side { + Side::Buy => "bid", + Side::Sell => "ask", + }; + let quantity = match delta.quantity { + 0 => "deleted".to_string(), + quantity => format!("now {}", format_fixed(quantity)), + }; + writeln!( + text, + " {} {side} {:>8} {quantity}", + time(delta.backtest_timestamp), + format_fixed(delta.price_level) + )?; + book.set(delta.side, delta.price_level, delta.quantity); + } + writeln!(text, "\nThe book they fold into:")?; + ladder(&mut text, &book)?; + Ok(text) +} + +enum Message { + Snapshot(SnapshotDomain), + Delta(PriceLevelUpdateDomain), +} + +/// Calls `f` with every message of the output at `path`, in order. +fn each(path: &Path, mut f: impl FnMut(Message)) -> anyhow::Result<()> { + let mut reader = AlignedSbeReader::::open(path)?; + while let Some(message) = reader.read_msg()? { + f(match message { + AnyMessage::Snapshot(decoder) => { + Message::Snapshot(SnapshotDomain::try_from_decoder(decoder)?) + } + AnyMessage::PriceLevelUpdate(decoder) => { + Message::Delta(PriceLevelUpdateDomain::try_from_decoder(decoder)?) + } + _ => anyhow::bail!("unknown message in the output"), + }); + } + Ok(()) +} + +/// The best `LEVELS` asks above the best `LEVELS` bids, prices descending. +fn ladder(text: &mut String, book: &Book) -> std::fmt::Result { + let asks: Vec<_> = book.asks.iter().take(LEVELS).collect(); + for (price, quantity) in asks.into_iter().rev() { + writeln!( + text, + " ask {:>8} {}", + format_fixed(*price), + format_fixed(*quantity) + )?; + } + writeln!(text, " ----")?; + for (price, quantity) in book.bids.iter().rev().take(LEVELS) { + writeln!( + text, + " bid {:>8} {}", + format_fixed(*price), + format_fixed(*quantity) + )?; + } + Ok(()) +} + +fn time(at: DateTime) -> String { + at.format("%H:%M:%S%.6f").to_string() +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::util::AlignedSbeWriter; + use crate::util::dtos::challenge_orderbook::{ + SnapshotAsksEntryDomain, SnapshotBidsEntryDomain, + }; + + const AT: i64 = 1_794_390_000; + + fn at(seconds: i64) -> DateTime { + DateTime::from_timestamp(AT + seconds, 0).unwrap() + } + + fn snapshot(symbol: &str, seconds: i64) -> SnapshotDomain { + SnapshotDomain { + received_at: at(seconds), + exchange_timestamp: at(seconds), + bids: vec![ + SnapshotBidsEntryDomain { + price_level: 400_000, + quantity: 10_000_000, + }, + SnapshotBidsEntryDomain { + price_level: 500_000, + quantity: 20_000_000, + }, + ], + asks: vec![SnapshotAsksEntryDomain { + price_level: 600_000, + quantity: 30_000_000, + }], + symbol: symbol.to_string(), + } + } + + fn delta( + symbol: &str, + seconds: i64, + side: Side, + price: i64, + quantity: i64, + ) -> PriceLevelUpdateDomain { + PriceLevelUpdateDomain { + backtest_timestamp: at(seconds), + exchange_timestamp: at(seconds), + price_level: price, + quantity, + side, + symbol: symbol.to_string(), + } + } + + /// "quiet" has one Delta, "busy" three: a new bid, a deleted bid, an ask + /// changed. `name` keeps parallel tests apart. + fn output(name: &str) -> std::path::PathBuf { + let path = std::env::temp_dir().join(format!("show-{}-{name}.sbe", std::process::id())); + let mut writer = AlignedSbeWriter::create(path.to_str().unwrap(), 64 << 20).unwrap(); + writer.write_msg(&snapshot("quiet", 0)).unwrap(); + writer.write_msg(&snapshot("busy", 0)).unwrap(); + writer + .write_msg(&delta("busy", 1, Side::Buy, 550_000, 5_000_000)) + .unwrap(); + writer + .write_msg(&delta("quiet", 2, Side::Buy, 400_000, 1_000_000)) + .unwrap(); + writer + .write_msg(&delta("busy", 3, Side::Buy, 400_000, 0)) + .unwrap(); + writer + .write_msg(&delta("busy", 4, Side::Sell, 600_000, 7_000_000)) + .unwrap(); + writer.flush().unwrap(); + path + } + + #[test] + fn shows_the_busiest_symbol_its_deltas_in_order_and_the_folded_book() { + let path = output("busiest"); + let text = show(&path, None).unwrap(); + std::fs::remove_file(&path).unwrap(); + assert!( + text.starts_with("busy: Snapshots 1, Deltas 3, in"), + "{text}" + ); + let deltas = text.split("The 3 Deltas after it:\n").nth(1).unwrap(); + let deltas: Vec<_> = deltas.lines().take(3).map(str::trim).collect(); + assert_eq!( + deltas, + [ + "09:40:01.000000 bid 0.55 now 5", + "09:40:03.000000 bid 0.4 deleted", + "09:40:04.000000 ask 0.6 now 7", + ] + ); + let folded = text.split("The book they fold into:\n").nth(1).unwrap(); + let folded: Vec<_> = folded.lines().map(str::trim).collect(); + assert_eq!( + folded, + [ + "ask 0.6 7", + "----", + "bid 0.55 5", + "bid 0.5 20" + ] + ); + } + + /// "deep": a Delta before its first Snapshot, a Snapshot of 7 asks and 7 + /// bids, a second Snapshot, then 25 Deltas. + #[test] + fn shows_the_first_snapshots_best_levels_and_the_first_20_deltas_after_it() { + let path = std::env::temp_dir().join(format!("show-{}-deep.sbe", std::process::id())); + let mut writer = AlignedSbeWriter::create(path.to_str().unwrap(), 64 << 20).unwrap(); + writer + .write_msg(&delta("deep", 0, Side::Buy, 100_000, 1_000_000)) + .unwrap(); + let mut deep = snapshot("deep", 1); + deep.asks = (1..=7) + .map(|level| SnapshotAsksEntryDomain { + price_level: 500_000 + level * 10_000, + quantity: level * 1_000_000, + }) + .collect(); + deep.bids = (1..=7) + .map(|level| SnapshotBidsEntryDomain { + price_level: 500_000 - level * 10_000, + quantity: level * 1_000_000, + }) + .collect(); + writer.write_msg(&deep).unwrap(); + writer.write_msg(&snapshot("deep", 2)).unwrap(); + for second in 3..28 { + writer + .write_msg(&delta("deep", second, Side::Buy, 100_000, 2_000_000)) + .unwrap(); + } + writer.flush().unwrap(); + let text = show(&path, Some("deep")).unwrap(); + std::fs::remove_file(&path).unwrap(); + + assert!( + text.starts_with("deep: Snapshots 2, Deltas 26, in"), + "{text}" + ); + let first = text.split("09:40:01.000000):\n").nth(1).unwrap(); + let first: Vec<_> = first.lines().take(11).map(str::trim).collect(); + assert_eq!( + first, + [ + "ask 0.55 5", + "ask 0.54 4", + "ask 0.53 3", + "ask 0.52 2", + "ask 0.51 1", + "----", + "bid 0.49 1", + "bid 0.48 2", + "bid 0.47 3", + "bid 0.46 4", + "bid 0.45 5", + ] + ); + let deltas = text.split("The 20 Deltas after it:\n").nth(1).unwrap(); + let deltas: Vec<_> = deltas.lines().take_while(|line| !line.is_empty()).collect(); + assert_eq!(deltas.len(), 20, "{text}"); + assert!(deltas[0].contains("09:40:03.000000"), "{text}"); + assert!(deltas[19].contains("09:40:22.000000"), "{text}"); + } + + #[test] + fn shows_the_symbol_named() { + let path = output("named"); + let text = show(&path, Some("quiet")).unwrap(); + std::fs::remove_file(&path).unwrap(); + assert!( + text.starts_with("quiet: Snapshots 1, Deltas 1, in"), + "{text}" + ); + assert!(text.contains("The 1 Deltas after it:"), "{text}"); + } + + #[test] + fn an_unknown_symbol_is_an_error() { + let path = output("unknown"); + let shown = show(&path, Some("absent")); + std::fs::remove_file(&path).unwrap(); + assert!(shown.unwrap_err().to_string().contains("absent is not in")); + } +} diff --git a/tests/cli.rs b/tests/cli.rs new file mode 100644 index 0000000..5c1b62f --- /dev/null +++ b/tests/cli.rs @@ -0,0 +1,35 @@ +//! The collector's command line, run as a user would. + +use std::process::Command; + +#[test] +fn help_prints_the_usage_and_exits_0() { + for flag in ["--help", "-h"] { + let output = Command::new(env!("CARGO_BIN_EXE_engineering-challenge")) + .arg(flag) + .output() + .unwrap(); + let text = String::from_utf8(output.stdout).unwrap(); + assert!(output.status.success(), "{flag}: {text}"); + assert!( + text.starts_with("usage: engineering-challenge EXCHANGE"), + "{flag}: {text}" + ); + assert!(text.contains("KALSHI_PRIVATE_KEY_FILE"), "{flag}: {text}"); + } +} + +#[test] +fn an_unknown_exchange_fails_with_the_usage() { + let output = Command::new(env!("CARGO_BIN_EXE_engineering-challenge")) + .arg("kraken") + .output() + .unwrap(); + let text = String::from_utf8(output.stderr).unwrap(); + assert!(!output.status.success()); + assert!(text.contains("unknown exchange \"kraken\""), "{text}"); + assert!( + text.contains("usage: engineering-challenge EXCHANGE"), + "{text}" + ); +} diff --git a/tests/reconcile.rs b/tests/reconcile.rs new file mode 100644 index 0000000..03d364a --- /dev/null +++ b/tests/reconcile.rs @@ -0,0 +1,94 @@ +//! The `reconcile` binary's exit status: what `scripts/watch.sh` relies on. + +use std::process::Command; + +use chrono::DateTime; +use engineering_challenge::Side; +use engineering_challenge::util::AlignedSbeWriter; +use engineering_challenge::util::dtos::challenge_orderbook::{ + PriceLevelUpdateDomain, SnapshotBidsEntryDomain, SnapshotDomain, +}; + +/// An output of a symbol's Snapshot and a Delta, after an orphan Delta of +/// another symbol if `orphan`; then, if `mismatched`, a Snapshot of the +/// symbol that the Delta doesn't lead to. +fn output(orphan: bool, mismatched: bool) -> std::path::PathBuf { + let at = DateTime::from_timestamp(1_794_390_000, 0).unwrap(); + let delta = |symbol: &str| PriceLevelUpdateDomain { + backtest_timestamp: at, + exchange_timestamp: at, + price_level: 500_000, + quantity: 2_000_000, + side: Side::Buy, + symbol: symbol.to_string(), + }; + let path = std::env::temp_dir().join(format!( + "reconcile-exit-{}-{orphan}-{mismatched}.sbe", + std::process::id() + )); + let mut writer = AlignedSbeWriter::create(path.to_str().unwrap(), 64 << 20).unwrap(); + if orphan { + writer.write_msg(&delta("2")).unwrap(); + } + writer + .write_msg(&SnapshotDomain { + received_at: at, + exchange_timestamp: at, + bids: vec![SnapshotBidsEntryDomain { + price_level: 500_000, + quantity: 1_000_000, + }], + asks: Vec::new(), + symbol: "1".to_string(), + }) + .unwrap(); + writer.write_msg(&delta("1")).unwrap(); + if mismatched { + writer + .write_msg(&SnapshotDomain { + received_at: at, + exchange_timestamp: at, + bids: vec![SnapshotBidsEntryDomain { + price_level: 500_000, + quantity: 3_000_000, + }], + asks: Vec::new(), + symbol: "1".to_string(), + }) + .unwrap(); + } + writer.flush().unwrap(); + path +} + +#[test] +fn reconcile_fails_on_an_orphan_delta_or_an_unexplained_mismatch_and_passes_otherwise() { + let status = |orphan: bool, mismatched: bool| { + let path = output(orphan, mismatched); + let output = Command::new(env!("CARGO_BIN_EXE_reconcile")) + .arg(&path) + .output() + .unwrap(); + std::fs::remove_file(&path).unwrap(); + ( + output.status.success(), + String::from_utf8(output.stdout).unwrap(), + ) + }; + let (passed, text) = status(false, false); + assert!(passed, "{text}"); + assert_eq!( + text, + "1 snapshots, 1 deltas, 0 orphan deltas, 0 mismatches\n" + ); + let (passed, text) = status(true, false); + assert!(!passed, "{text}"); + assert!(text.contains("1 orphan deltas"), "{text}"); + // A Snapshot no log line explains (no log given): it fails too. + let (passed, text) = status(false, true); + assert!(!passed, "{text}"); + assert!( + text.contains("1 mismatches") && text.contains("unexplained: 1 at "), + "{text}" + ); +}