Archived
feat: ready to submit, a run to watch, the README, the review's fixes
Sprint 08 (roadmap/08. ready to submit.md), the last milestone. - scripts/watch.sh runs the collector with a live view of its log, stops cleanly on Ctrl-C, then checks the output (reconcile) and shows one symbol's book as text (show, a new binary). - The README for the solution, with the final runs: Polymarket through an outage it recovered from on its own, Kalshi's demo with two slow connections recycled. - From a review of the whole codebase: Kalshi takes RSA keys too (RSA-PSS-SHA256); a slow Kalshi connection takes the hourly recycle's next turn, which it nearly never got before; a market tracked again no longer fails reconcile; Book::add checks for overflow; usage and --help name the key variables; the first scan says how far it got. - Tests for every gap the reviews found, an offline run of the whole orchestrator among them; ShellCheck in CI.
This commit is contained in:
35 files changed
+2791
-207
No files matched your search
@@ -1,2 +1,3 @@
|
|||||||
/target/
|
/target/
|
||||||
__pycache__/
|
__pycache__/
|
||||||
|
watch/
|
||||||
@@ -23,8 +23,8 @@ outside this repo.
|
|||||||
Delta, lateness, lag, speed).
|
Delta, lateness, lag, speed).
|
||||||
- `src/core/`: what is the same for every exchange (ADR 0004).
|
- `src/core/`: what is the same for every exchange (ADR 0004).
|
||||||
`orchestrator.rs`: what is tracked, which connections carry each market,
|
`orchestrator.rs`: what is tracked, which connections carry each market,
|
||||||
their balancing and their recycling (hourly, and sooner for a Kalshi
|
their balancing and their recycling (hourly, a Kalshi connection that
|
||||||
connection that stays among the slowest); `socket.rs`: a connection (read,
|
stays among the slowest taking the next turn); `socket.rs`: a connection (read,
|
||||||
stamp, decode, route; reconnects when dropped, silent, lost a message or
|
stamp, decode, route; reconnects when dropped, silent, lost a message or
|
||||||
recycled); `feed.rs`: a book thread's shell (Snapshots, moves, the quiet
|
recycled); `feed.rs`: a book thread's shell (Snapshots, moves, the quiet
|
||||||
watch) and the `Rule` trait; `mod.rs`: the `Exchange`, `Session`, `Market`
|
watch) and the `Rule` trait; `mod.rs`: the `Exchange`, `Session`, `Market`
|
||||||
@@ -37,8 +37,14 @@ outside this repo.
|
|||||||
rank rule, `mod.rs` the exchange and its session (the `seq` check).
|
rank rule, `mod.rs` the exchange and its session (the `seq` check).
|
||||||
- `src/book.rs`: the book and fixed point; `src/reconcile.rs` +
|
- `src/book.rs`: the book and fixed point; `src/reconcile.rs` +
|
||||||
`src/bin/reconcile.rs`: the output check, reading the log's cause lines
|
`src/bin/reconcile.rs`: the output check, reading the log's cause lines
|
||||||
(`src/cause.rs`). `tests/fixtures/`: recordings of the live feeds that the
|
(`src/cause.rs`); `src/show.rs` + `src/bin/show.rs`: one symbol of the
|
||||||
|
output as text. `tests/fixtures/`: recordings of the live feeds that the
|
||||||
tests replay.
|
tests replay.
|
||||||
|
- `scripts/watch.sh [EXCHANGE] [MINUTES]`: a run to watch, also the final
|
||||||
|
runs' and the README's quick start (roadmap/08): the collector under
|
||||||
|
`timeout -s INT`, a live view of its log (`scripts/watch.awk`, which
|
||||||
|
parses the minute lines' wording: change one, check the other), then
|
||||||
|
`reconcile` and `show`. Output and log in `watch/`.
|
||||||
- `src/latency.rs` + `scripts/latency/analyse.py`: the latency probe
|
- `src/latency.rs` + `scripts/latency/analyse.py`: the latency probe
|
||||||
(`engineering-challenge latency EXCHANGE OUT.csv`, until SIGINT: one CSV
|
(`engineering-challenge latency EXCHANGE OUT.csv`, until SIGINT: one CSV
|
||||||
line per copy of a change on each of 16 connections) and its analysis
|
line per copy of a change on each of 16 connections) and its analysis
|
||||||
@@ -67,6 +73,10 @@ and let the user settle it. Claude owns the data model.
|
|||||||
- Use `GLOSSARY.md`'s terms: a **Snapshot** is only the output message; getting
|
- Use `GLOSSARY.md`'s terms: a **Snapshot** is only the output message; getting
|
||||||
a fresh book is a **resync**.
|
a fresh book is a **resync**.
|
||||||
- A corner cut for speed carries a comment naming its ceiling and upgrade path.
|
- A corner cut for speed carries a comment naming its ceiling and upgrade path.
|
||||||
|
- A figure quoted in a doc (README, `LATENCY.md`, a sprint's Outcome) comes
|
||||||
|
from the tool that checks it, `reconcile`'s output or the log itself,
|
||||||
|
never from `watch.sh`'s view or a count grouped by hand: sprint 08's
|
||||||
|
first README figures had four errors from those.
|
||||||
|
|
||||||
## Environment traps
|
## Environment traps
|
||||||
|
|
||||||
@@ -91,7 +101,20 @@ and let the user settle it. Claude owns the data model.
|
|||||||
(`pgrep -f '^./target/release/...'`, anchored) and loop on `kill -0 $pid`.
|
(`pgrep -f '^./target/release/...'`, anchored) and loop on `kill -0 $pid`.
|
||||||
- **No `/usr/bin/time` on the server**: for a run's peak memory and thread
|
- **No `/usr/bin/time` on the server**: for a run's peak memory and thread
|
||||||
count, read `VmHWM` and `Threads` from `/proc/<pid>/status` while it runs
|
count, read `VmHWM` and `Threads` from `/proc/<pid>/status` while it runs
|
||||||
(sprint 05's run script sampled them every 30 s).
|
(`scripts/watch.sh` samples them every 10 s).
|
||||||
|
- **mawk buffers a pipe's input** unless `-W interactive`: a live view
|
||||||
|
through `awk` shows nothing until the end (`scripts/watch.sh` passes it
|
||||||
|
when the awk is mawk, as on the server).
|
||||||
|
- **A job started with `&` in `sh` ignores Ctrl-C**: a script that runs the
|
||||||
|
collector in the background must pass SIGINT on itself (`watch.sh`'s
|
||||||
|
trap). To try Ctrl-C without a terminal: `(sleep N; printf '\003') |
|
||||||
|
script -qfec "CMD" /dev/null`.
|
||||||
|
- **Polymarket's Gamma answers 429 to runs started close together**: two
|
||||||
|
collectors' first scans at once (or a few runs within minutes) got every
|
||||||
|
page refused; the scan retries, but leave a few minutes between runs.
|
||||||
|
- **`*.pem` is gitignored globally** (`~/.gitignore`): a test key goes in
|
||||||
|
the source, as `src/kalshi/sign.rs` does with RFC 8032's and RFC 9500's
|
||||||
|
published test keys.
|
||||||
- **The server is shared and gets overloaded** (load up to 92 on 12 cores,
|
- **The server is shared and gets overloaded** (load up to 92 on 12 cores,
|
||||||
2026-11-11): a starved collector falls behind and gets closed with 1013.
|
2026-11-11): a starved collector falls behind and gets closed with 1013.
|
||||||
Check `cat /proc/loadavg` before a live run whose numbers are compared, and
|
Check `cat /proc/loadavg` before a live run whose numbers are compared, and
|
||||||
@@ -99,7 +122,8 @@ and let the user settle it. Claude owns the data model.
|
|||||||
- **Kalshi needs a key; nothing else does**: `KALSHI_KEY_ID_FILE` and
|
- **Kalshi needs a key; nothing else does**: `KALSHI_KEY_ID_FILE` and
|
||||||
`KALSHI_PRIVATE_KEY_FILE` name its two files (on the server
|
`KALSHI_PRIVATE_KEY_FILE` name its two files (on the server
|
||||||
`/run/secrets/kalshi-demo-key-id` and `kalshi-demo-private-key`; never read
|
`/run/secrets/kalshi-demo-key-id` and `kalshi-demo-private-key`; never read
|
||||||
or print them). Opening ~100 connections at once gets some handshakes
|
or print them). The key is Ed25519 or RSA (RSA-PSS-SHA256), told apart
|
||||||
|
by the parsed key. Opening ~100 connections at once gets some handshakes
|
||||||
refused with 429; each retries after 1 s and gets in.
|
refused with 429; each retries after 1 s and gets in.
|
||||||
- **cargo keeps a build whose source got an older mtime**: a script that
|
- **cargo keeps a build whose source got an older mtime**: a script that
|
||||||
restores a file from a copy (`shutil.move`, `cp -p`) must `touch` it, or
|
restores a file from a copy (`shutil.move`, `cp -p`) must `touch` it, or
|
||||||
|
|||||||
+4
-3
@@ -3,7 +3,7 @@ name = "engineering-challenge"
|
|||||||
version = "0.1.0"
|
version = "0.1.0"
|
||||||
edition = "2024"
|
edition = "2024"
|
||||||
build = "src/util/build.rs"
|
build = "src/util/build.rs"
|
||||||
# `cargo run` runs the collector, not `src/bin/reconcile.rs`.
|
# `cargo run` runs the collector, not `src/bin/reconcile.rs` or `show.rs`.
|
||||||
default-run = "engineering-challenge"
|
default-run = "engineering-challenge"
|
||||||
|
|
||||||
[dependencies]
|
[dependencies]
|
||||||
@@ -14,10 +14,11 @@ chrono = "0.4"
|
|||||||
crossbeam = "0.8.4"
|
crossbeam = "0.8.4"
|
||||||
# `termination`: SIGTERM and SIGHUP stop the collector like SIGINT.
|
# `termination`: SIGTERM and SIGHUP stop the collector like SIGINT.
|
||||||
ctrlc = { version = "3", features = ["termination"] }
|
ctrlc = { version = "3", features = ["termination"] }
|
||||||
|
# Signs Kalshi's handshake (Ed25519 or RSA-PSS); already rustls's crypto
|
||||||
|
# provider.
|
||||||
|
ring = "0.17"
|
||||||
# Not used directly: tungstenite leaves rustls's crypto provider to the app, and
|
# Not used directly: tungstenite leaves rustls's crypto provider to the app, and
|
||||||
# rustls panics at runtime unless exactly one (here ring) is enabled.
|
# rustls panics at runtime unless exactly one (here ring) is enabled.
|
||||||
# Signs Kalshi's handshake (Ed25519); already rustls's crypto provider.
|
|
||||||
ring = "0.17"
|
|
||||||
rustls = { version = "0.23", default-features = false, features = ["ring"] }
|
rustls = { version = "0.23", default-features = false, features = ["ring"] }
|
||||||
# Reads the PEM of Kalshi's key.
|
# Reads the PEM of Kalshi's key.
|
||||||
rustls-pki-types = { version = "1", features = ["std"] }
|
rustls-pki-types = { version = "1", features = ["std"] }
|
||||||
|
|||||||
+8
-2
@@ -48,8 +48,14 @@ _Avoid_: Order book state, depth
|
|||||||
**Connection**:
|
**Connection**:
|
||||||
One websocket to the exchange. Every live symbol is subscribed on two
|
One websocket to the exchange. Every live symbol is subscribed on two
|
||||||
connections, chosen independently of each other.
|
connections, chosen independently of each other.
|
||||||
_Avoid_: Socket (for the concept; fine for the code), stream, lane (the
|
_Avoid_: Socket (for the concept; fine for the code), stream (that is one
|
||||||
retired pairing of one connection with one book thread)
|
connection's changes of a symbol), lane (the retired pairing of one
|
||||||
|
connection with one book thread)
|
||||||
|
|
||||||
|
**Stream**:
|
||||||
|
One connection's changes of one symbol, in the order it delivers them. The
|
||||||
|
two streams of a symbol carry the same changes, each at its own pace.
|
||||||
|
_Avoid_: Feed (the book thread's code), using it for the connection
|
||||||
|
|
||||||
**Load**:
|
**Load**:
|
||||||
How much a market weighs on a connection: the bytes per second of its frames
|
How much a market weighs on a connection: the bytes per second of its frames
|
||||||
|
|||||||
+13
-5
@@ -186,8 +186,9 @@ as the median lag), but at the median it wins 9 ms, a tenth of the lag.
|
|||||||
- **On Kalshi's demo it is worth tens of milliseconds**, in the tail too
|
- **On Kalshi's demo it is worth tens of milliseconds**, in the tail too
|
||||||
(p99 28% and 42% lower with the fastest pair kept). This met the
|
(p99 28% and 42% lower with the fastest pair kept). This met the
|
||||||
threshold agreed before the runs, so the collector now recycles a Kalshi
|
threshold agreed before the runs, so the collector now recycles a Kalshi
|
||||||
connection that stays slow for 3 minutes running, ahead of its hourly
|
connection that stays slow for 3 minutes running at the hourly
|
||||||
turn: its replacement's speed is drawn afresh. Each duplicate tells how
|
recycle's next turn (one every 30 s or so), in place of the oldest: its
|
||||||
|
replacement's speed is drawn afresh. Each duplicate tells how
|
||||||
late its connection was against the other's copy (`src/kalshi/feed.rs`);
|
late its connection was against the other's copy (`src/kalshi/feed.rs`);
|
||||||
once a minute the orchestrator ranks the connections by their mean
|
once a minute the orchestrator ranks the connections by their mean
|
||||||
lateness, and one is slow if it is in the slowest quarter and more than
|
lateness, and one is slow if it is in the slowest quarter and more than
|
||||||
@@ -197,9 +198,16 @@ as the median lag), but at the median it wins 9 ms, a tenth of the lag.
|
|||||||
quarter always exists: a first version, by median alone, recycled 6
|
quarter always exists: a first version, by median alone, recycled 6
|
||||||
connections in 20 minutes, one a minute, connections no slower than the
|
connections in 20 minutes, one a minute, connections no slower than the
|
||||||
rest. **In 20 minutes of the collector on the demo (270,000 markets on
|
rest. **In 20 minutes of the collector on the demo (270,000 markets on
|
||||||
120 connections, load 5.7 at the start), no connection was slow by that
|
120 connections, load 5.7 at the start), none was recycled for it**; the
|
||||||
rule**, so none was recycled for it; the hourly recycle went on (33), and
|
hourly recycle went on (33), and the output reconciled (25 mismatches,
|
||||||
the output reconciled (25 mismatches, each a logged loss). The probe's 16
|
each a logged loss). That run doesn't say whether one was judged slow: a
|
||||||
|
later review found the slow recycle waited for the same moment as the
|
||||||
|
hourly one, which nearly always took it first, and only the recycle was
|
||||||
|
logged. Now the slow one takes the hourly recycle's turn, and the
|
||||||
|
judgment is logged on its own. **In 20 minutes on the demo after that
|
||||||
|
fix** (260,865 markets on 116 connections, 2026-11-12), two connections
|
||||||
|
were judged slow and recycled: one at a mean lateness of 246 ms, one at
|
||||||
|
80 ms. The probe's 16
|
||||||
connections, on the 100 busiest markets, differed far more than the
|
connections, on the 100 busiest markets, differed far more than the
|
||||||
collector's; assumed to be because each of the collector's shares its
|
collector's; assumed to be because each of the collector's shares its
|
||||||
markets with many others and is judged against all their copies, which
|
markets with many others and is judged against all their copies, which
|
||||||
|
|||||||
@@ -1,21 +1,214 @@
|
|||||||
# Read before getting started!!
|
# Order-book collector for Polymarket and Kalshi
|
||||||
|
|
||||||
First, read [src/main.rs](https://github.com/voluntas-io/engineering-challenge/blob/main/src/main.rs). Then, some background on this repo:
|
My answer to the Voluntas engineering challenge. The collector subscribes
|
||||||
|
to every live market of one exchange (Polymarket, or Kalshi), keeps an order
|
||||||
|
book for each symbol, and writes the template's SBE messages: a Delta for
|
||||||
|
every change, and a Snapshot of every book each minute and whenever a book
|
||||||
|
is replaced. All four extensions are done: redundant connections,
|
||||||
|
subscription monitoring, one core for both exchanges, and latency
|
||||||
|
experiments ([LATENCY.md](LATENCY.md)).
|
||||||
|
|
||||||
This repo uses the [FIX SBE standard](https://fixtrading.org/packages/simple-binary-encoding-technical-specification-v2-rc2/)
|
Built with Claude Code. I set the architecture (the threading model,
|
||||||
for writing binary messages. The rust code abstracts this away for you, and you can simply use the helpers, as demonstrated in `main.rs`.
|
redundancy, the exchange seam) and settled every design decision; Claude
|
||||||
|
proposed the options, wrote the code and ran the experiments. Each
|
||||||
|
decision and its reasons are in [`roadmap/`](roadmap) and
|
||||||
|
[`docs/adr/`](docs/adr).
|
||||||
|
|
||||||
### More background, in case you are curious!
|
## Quick start
|
||||||
|
|
||||||
FIX SBE is defined in XML, which I find extremely cumbersome to edit, especially
|
On Linux (the script reads `/proc`), with Rust; built and tested with
|
||||||
when updating schemas frequently. To solve this, I have also introduced some
|
1.98:
|
||||||
helper code that takes our own custom YAML format, and turns it into FIX SBE
|
|
||||||
XML, which in turn is used to generate rust code to encode and decode the
|
|
||||||
messages. **You don't need to edit any of these files**. This explanation is
|
|
||||||
simply to help you understand the origins of the various files in this repo.
|
|
||||||
|
|
||||||
If you're curious to see the YAML schema, take a look at [challenge_orderbook.yaml](https://github.com/voluntas-io/engineering-challenge/blob/main/sbe_schemas/engineering_challenge/challenge_orderbook.yaml).
|
```sh
|
||||||
We also use a custom timestamp field, so that the Rust struct can use
|
scripts/watch.sh # Polymarket for 3 minutes, no key needed
|
||||||
`chrono::DateTime` in the generated struct, and in SBE schema, it is a uint64
|
scripts/watch.sh polymarket 10 # or longer
|
||||||
representing the timestamp as the number of microseconds since unix epoch.
|
```
|
||||||
|
|
||||||
|
It builds the collector, runs it, and shows what it does each minute:
|
||||||
|
markets and connections; changes written; duplicates (a change's second
|
||||||
|
copy, from the other connection carrying its symbol); rescued changes (one
|
||||||
|
connection missed them, the other delivered them); resyncs (a fresh book
|
||||||
|
asked for, when ours can't be checked); losses (a fresh book that differed
|
||||||
|
from ours: changes neither connection delivered). The output and the full
|
||||||
|
log go to `watch/`. Ctrl-C stops it
|
||||||
|
early and cleanly. Then it checks the output against the log (`reconcile`)
|
||||||
|
and prints one busy symbol's book as text (`show`). The first scan lists
|
||||||
|
every market before subscribing: a minute or two.
|
||||||
|
|
||||||
|
## Running it
|
||||||
|
|
||||||
|
```sh
|
||||||
|
cargo build --release
|
||||||
|
./target/release/engineering-challenge polymarket > out.sbe 2> run.log
|
||||||
|
./target/release/engineering-challenge polymarket out.sbe 2> run.log # same
|
||||||
|
```
|
||||||
|
|
||||||
|
The output goes to stdout unless a path is given; the log goes to stderr.
|
||||||
|
SIGINT, SIGTERM or SIGHUP closes the websockets and flushes the output.
|
||||||
|
Only a clean stop flushes it: the writer fills 64 MiB chunks, the only size
|
||||||
|
the template's reader opens.
|
||||||
|
|
||||||
|
Kalshi needs an API key. Two variables name its files (paths keep the key
|
||||||
|
out of the shell's history):
|
||||||
|
|
||||||
|
```sh
|
||||||
|
export KALSHI_KEY_ID_FILE=~/kalshi/key-id
|
||||||
|
export KALSHI_PRIVATE_KEY_FILE=~/kalshi/private-key.pem # Ed25519 or RSA
|
||||||
|
./target/release/engineering-challenge kalshi out.sbe 2> run.log
|
||||||
|
./target/release/engineering-challenge kalshi-demo out.sbe 2> run.log
|
||||||
|
```
|
||||||
|
|
||||||
|
I had a key for Kalshi's demo only, so production has never been run; the
|
||||||
|
code is the same, only the URLs differ.
|
||||||
|
|
||||||
|
## Checking the output
|
||||||
|
|
||||||
|
```sh
|
||||||
|
./target/release/reconcile out.sbe run.log
|
||||||
|
./target/release/show out.sbe [SYMBOL]
|
||||||
|
```
|
||||||
|
|
||||||
|
`reconcile` replays the output: per symbol, the Deltas since a Snapshot,
|
||||||
|
applied to it, must give the next Snapshot. A Snapshot that differs is
|
||||||
|
paired with the log line that says why it was written: a loss; a
|
||||||
|
subscription gap (the few changes a subscribe keeps from us, which the
|
||||||
|
exchange's opening book already holds); a switch (Polymarket's book now
|
||||||
|
followed on the other connection); a market tracked again after it left.
|
||||||
|
It fails on any left unexplained, or on a Delta before its symbol's first
|
||||||
|
Snapshot.
|
||||||
|
|
||||||
|
## Final runs
|
||||||
|
|
||||||
|
`scripts/watch.sh`, from the submitted code, on a shared 12-core server.
|
||||||
|
|
||||||
|
**Polymarket, 10 minutes** (2026-11-12, system load 7.2 at the start): the
|
||||||
|
first scan listed 199,531 live markets in 77 s, subscribed on 178
|
||||||
|
connections. `reconcile`: 14.4 million Deltas and 3.7 million Snapshots;
|
||||||
|
the 142,990 Snapshots that differ from the book before them each explained
|
||||||
|
by their log line. 95% of the changes came on both connections, and
|
||||||
|
417,852 were rescued. Peak memory 2.3 GB. The thread count wasn't sampled
|
||||||
|
at its peak in this run; it is one per connection plus about 20 (the Kalshi
|
||||||
|
run's 137 for 116), so about 200.
|
||||||
|
|
||||||
|
The run met an outage of about two minutes. Between 16:50:17 and 16:50:25
|
||||||
|
UTC, 177 of the 178 connections were found silent (a connection is silent
|
||||||
|
after 30 s with nothing received, so since about 16:49:47). Connects then
|
||||||
|
timed out or found no route to the host, and between 16:51:37 and 16:51:59
|
||||||
|
every connection came back on its own. Each book that changed meanwhile was found
|
||||||
|
wrong on the reconnect and replaced: 138,950 of the run's 140,129 losses
|
||||||
|
are those two minutes, 1,179 the other eight. Whether the network or
|
||||||
|
Polymarket went down, the log can't say.
|
||||||
|
|
||||||
|
**Kalshi's demo, 20 minutes** (system load 0.8 at the start): the first
|
||||||
|
scan took 5 min 17 s, the demo's listing being slow (about 1.2 s a page of
|
||||||
|
1,000 markets in that run); then 260,865 markets on 116 connections.
|
||||||
|
`reconcile`: 4.1 million Deltas and 4.0 million Snapshots; the 12 that
|
||||||
|
differ each a logged loss. 98% of the changes came on both connections;
|
||||||
|
1,228 resyncs. Peak memory 830 MB, 137 threads. Every connection is
|
||||||
|
recycled (closed and opened again) about once an hour, one at a time; on
|
||||||
|
Kalshi one that stays slow takes the next turn, and that acted: a
|
||||||
|
connection's lateness is how long after the other connection's copy it
|
||||||
|
delivers a change, and connections 99 (246 ms on average) and 57 (80 ms)
|
||||||
|
were each among the slowest 3 minutes running, so each was recycled at
|
||||||
|
the next turn, 13 and 21 s later.
|
||||||
|
|
||||||
|
## Design
|
||||||
|
|
||||||
|
```
|
||||||
|
rescan fetcher ──findings──▶ ┌──────────────┐ ◀──findings── recent fetcher
|
||||||
|
│ orchestrator │ ◀── up, down, 1013 ─── connections
|
||||||
|
└──────┬───▲───┘ ──── recycle ────────▶ connections
|
||||||
|
Add / Move / Remove (a market and │ │ loads, quiet markets,
|
||||||
|
its connections), to its thread │ │ lateness, each minute
|
||||||
|
┌──────▼───┴───┐
|
||||||
|
connection 0..N ──events───▶ │ book threads │ ──Snapshot | Delta──▶ writer
|
||||||
|
(read, stamp, parse, route) │ 0..15 │
|
||||||
|
▲ └──────┬───────┘
|
||||||
|
└── subscribe / unsubscribe / resync ┘
|
||||||
|
```
|
||||||
|
|
||||||
|
Plain threads and channels, nothing shared ([ADR 0001](docs/adr/0001-concurrency-model.md)).
|
||||||
|
A connection stamps the receive time right after the socket read, then
|
||||||
|
parses and routes each event to the book thread that owns its market. Each
|
||||||
|
book is owned by one of 16 book threads; one writer owns the output. The
|
||||||
|
orchestrator decides which connections carry each market, balances the
|
||||||
|
bytes per second each carries, and recycles them ([ADR 0002](docs/adr/0002-lanes-and-orchestrator.md)).
|
||||||
|
[docs/project.md](docs/project.md) has the whole design.
|
||||||
|
|
||||||
|
A lost change leaves a book silently wrong, so every sign of one replaces
|
||||||
|
the book with a fresh one from the exchange (a resync). On Polymarket the
|
||||||
|
exchange's book hash is recomputed after every change; on Kalshi it is a
|
||||||
|
skipped sequence number on a connection. A fresh book that differs from
|
||||||
|
ours counts as a loss; some of those are only books ahead of our changes,
|
||||||
|
so the count is an upper bound. The protocol facts the
|
||||||
|
code relies on, several found only in live data, are in
|
||||||
|
[docs/notes.md](docs/notes.md).
|
||||||
|
|
||||||
|
## The extensions
|
||||||
|
|
||||||
|
- **Redundant websockets.** Every symbol is on two connections chosen
|
||||||
|
independently; its book takes each change from whichever delivers it
|
||||||
|
first ([ADR 0003](docs/adr/0003-first-one-wins.md)). There is no "source"
|
||||||
|
to switch, so the brief's Snapshot on a switch becomes a Snapshot
|
||||||
|
whenever a fresh book from either connection replaces ours. A connection
|
||||||
|
the exchange closes twice within 10 minutes as too slow (code 1013, "try
|
||||||
|
again later") has half its markets moved to others.
|
||||||
|
- **Subscription monitoring.** A subscription is quiet when it delivered
|
||||||
|
nothing for a minute while the other connection delivered at least 3
|
||||||
|
changes of the same symbol: it is resubscribed, and moved if quiet
|
||||||
|
again. Judged against the other connection, not a rate, it holds for
|
||||||
|
the quietest markets too. Every
|
||||||
|
connection is recycled about once an hour, one at a time, never two of a
|
||||||
|
market's; one silent for 30 s reconnects.
|
||||||
|
- **Generic core.** `src/core/` is the same for both exchanges. An exchange
|
||||||
|
supplies its websocket codec and its loss rule
|
||||||
|
([ADR 0004](docs/adr/0004-exchange-seam.md)): the rule is more than
|
||||||
|
decoding, since telling a duplicate from a loss depends on what the
|
||||||
|
exchange sends (a book hash, or a sequence number and a clock).
|
||||||
|
- **Latency.** [LATENCY.md](LATENCY.md): a connection that delivers
|
||||||
|
changes sooner than the others keeps doing so, on both exchanges.
|
||||||
|
Keeping the fastest gains under a millisecond on Polymarket, and tens of
|
||||||
|
milliseconds on Kalshi's demo, where the collector now recycles a
|
||||||
|
connection that stays slow.
|
||||||
|
|
||||||
|
## Limits
|
||||||
|
|
||||||
|
- A hard crash (SIGKILL, power) loses the chunk being filled, up to 64 MiB.
|
||||||
|
- When a symbol is subscribed, the exchange's opening book holds a few
|
||||||
|
changes it never sends as changes: logged as a subscription gap.
|
||||||
|
- Polymarket changes held for ordering when the run stops (at most 1 s) are
|
||||||
|
not written.
|
||||||
|
- Channels are unbounded: a burst never blocks a socket or drops an event,
|
||||||
|
but a book thread or writer that can't keep up grows memory.
|
||||||
|
- Kalshi production was never run.
|
||||||
|
|
||||||
|
## Next
|
||||||
|
|
||||||
|
From the roadmap's possible features ([docs/roadmap.md](docs/roadmap.md)):
|
||||||
|
telling a real loss from a fresh book merely ahead of the changes we have,
|
||||||
|
so the loss count measures data missed; Polymarket's `new_market` event as
|
||||||
|
a faster discovery signal; fewer Snapshots for unchanged books (the brief's
|
||||||
|
every minute is about 130 MB a minute at full coverage); a Kalshi
|
||||||
|
production run.
|
||||||
|
|
||||||
|
## Reading the code
|
||||||
|
|
||||||
|
- [docs/project.md](docs/project.md): what it is, the central bet (the
|
||||||
|
pipeline), the design.
|
||||||
|
- [docs/adr/](docs/adr): the four decisions that are hard to reverse.
|
||||||
|
- [docs/notes.md](docs/notes.md): each exchange's rules, as found.
|
||||||
|
- [LATENCY.md](LATENCY.md): extension 4.
|
||||||
|
- [GLOSSARY.md](GLOSSARY.md): the terms (a Snapshot is only the output
|
||||||
|
message; getting a fresh book is a resync).
|
||||||
|
- [roadmap/](roadmap): each sprint's plan, decisions and outcome, in order.
|
||||||
|
- [CLAUDE.md](CLAUDE.md): the instructions the coding agent worked under.
|
||||||
|
|
||||||
|
## Output format
|
||||||
|
|
||||||
|
The output uses the template's helpers unchanged: `AlignedSbeWriter`
|
||||||
|
writes the SBE messages of
|
||||||
|
[sbe_schemas/engineering_challenge/challenge_orderbook.yaml](sbe_schemas/engineering_challenge/challenge_orderbook.yaml)
|
||||||
|
(`Snapshot` and `PriceLevelUpdate`, prices and quantities in fixed point,
|
||||||
|
timestamps in microseconds since the epoch), readable by the template's
|
||||||
|
`AlignedSbeReader`. The template's code is `src/util/` and `sbe_schemas/`;
|
||||||
|
everything else in `src/` is the collector.
|
||||||
+8
-3
@@ -237,9 +237,14 @@ is the demo's.
|
|||||||
`api.elections.kalshi.com` and `demo-api.kalshi.co` still answer.
|
`api.elections.kalshi.com` and `demo-api.kalshi.co` still answer.
|
||||||
- **Auth:** the websocket handshake carries `KALSHI-ACCESS-KEY` (the key id),
|
- **Auth:** the websocket handshake carries `KALSHI-ACCESS-KEY` (the key id),
|
||||||
`KALSHI-ACCESS-TIMESTAMP` (ms) and `KALSHI-ACCESS-SIGNATURE`: base64 of the
|
`KALSHI-ACCESS-TIMESTAMP` (ms) and `KALSHI-ACCESS-SIGNATURE`: base64 of the
|
||||||
Ed25519 signature of `<timestamp>GET/trade-api/ws/v2`. Without them: 401.
|
signature of `<timestamp>GET/trade-api/ws/v2`. Without them: 401. The
|
||||||
The market listing needs no key. Keys are created per environment
|
key is Ed25519 (RFC 8032 over that text) or 2048-bit RSA (RSA-PSS,
|
||||||
(Account → Profile → API Keys); the private key is a PKCS#8 PEM shown once.
|
SHA-256, MGF1 SHA-256, a salt of the digest's length); Kalshi picks the
|
||||||
|
check from the public key it holds, and a PEM's header doesn't say which
|
||||||
|
(an RSA key may be PKCS#1, as Kalshi gives it, or PKCS#8; Kalshi's API
|
||||||
|
keys page, read 2026-11-12). The market listing needs no key. Keys are
|
||||||
|
created per environment (Account → Profile → API Keys); the private key
|
||||||
|
is shown once.
|
||||||
- **Subscribe:** `{"id":1,"cmd":"subscribe","params":{"channels":
|
- **Subscribe:** `{"id":1,"cmd":"subscribe","params":{"channels":
|
||||||
["orderbook_delta"],"market_tickers":[...]}}` is answered
|
["orderbook_delta"],"market_tickers":[...]}}` is answered
|
||||||
`{"type":"subscribed","id":1,"msg":{"channel":"orderbook_delta","sid":1}}`,
|
`{"type":"subscribed","id":1,"msg":{"channel":"orderbook_delta","sid":1}}`,
|
||||||
|
|||||||
+4
-4
@@ -161,11 +161,11 @@ schema debate). As settled in sprints 04 and 05:
|
|||||||
|
|
||||||
- The **orchestrator** owns what is tracked (each market's two connections,
|
- The **orchestrator** owns what is tracked (each market's two connections,
|
||||||
the timers) and decides: placing, moving, splitting, and recycling one
|
the timers) and decides: placing, moving, splitting, and recycling one
|
||||||
connection at a time, about hourly, and sooner a Kalshi connection whose
|
connection at a time, about hourly; a Kalshi connection whose mean
|
||||||
mean lateness stays above twice everyone's (sprint 07, LATENCY.md). It
|
lateness stays above twice everyone's takes the next turn, in place of
|
||||||
never blocks on HTTP.
|
the oldest (sprints 07 and 08, LATENCY.md). It never blocks on HTTP.
|
||||||
- The two **fetchers** only page the exchange's market listing on request
|
- The two **fetchers** only page the exchange's market listing on request
|
||||||
and stream what they read.
|
and send on what they read.
|
||||||
- A **connection** reads, stamps the receive time, parses, and routes each
|
- A **connection** reads, stamps the receive time, parses, and routes each
|
||||||
event to the book thread of its market. Every symbol is on two connections,
|
event to the book thread of its market. Every symbol is on two connections,
|
||||||
chosen independently. One silent for 30 s reconnects.
|
chosen independently. One silent for 30 s reconnects.
|
||||||
|
|||||||
+19
-4
@@ -91,11 +91,17 @@ abstraction is *extracted* once a second exchange exists, not designed up front.
|
|||||||
The source port can't be chosen here (a NAT rewrites it); first after
|
The source port can't be chosen here (a NAT rewrites it); first after
|
||||||
maintenance wasn't tried.
|
maintenance wasn't tried.
|
||||||
|
|
||||||
8. **Polish + submission** — *not started*
|
8. **Polish + submission** — *done (2026-11-12)*
|
||||||
Final review, README for the solution (the template's README still says the
|
A review of the whole codebase and its fixes, a run anyone can watch
|
||||||
helpers are "demonstrated in `main.rs`", whose demo M1 replaced),
|
(`scripts/watch.sh`, `show`), the README, final runs of both
|
||||||
packaging/handoff for the challenge.
|
exchanges, and the handoff as a public repo on git.pband.ch.
|
||||||
|
Sprint: [../roadmap/08. ready to submit.md](../roadmap/08.%20ready%20to%20submit.md).
|
||||||
*Ends:* ready to submit.
|
*Ends:* ready to submit.
|
||||||
|
Result: Kalshi takes RSA keys too; the slow recycle, which the hourly
|
||||||
|
one nearly always pre-empted, now takes its turn and acted twice in 20
|
||||||
|
live minutes; a market tracked again no longer fails `reconcile`; 19
|
||||||
|
surviving mutants got their tests. Both final runs reconciled, one
|
||||||
|
through an outage of about two minutes (sprint 08's Outcome).
|
||||||
|
|
||||||
## Why this order
|
## Why this order
|
||||||
|
|
||||||
@@ -155,3 +161,12 @@ abstraction is *extracted* once a second exchange exists, not designed up front.
|
|||||||
- Judging the connections' speed at the book threads' wall-clock minute,
|
- Judging the connections' speed at the book threads' wall-clock minute,
|
||||||
not on the orchestrator's own timer (a judgment near the crossing sees
|
not on the orchestrator's own timer (a judgment near the crossing sees
|
||||||
partial samples; sprint 07's review).
|
partial samples; sprint 07's review).
|
||||||
|
- Kalshi: a resync asked from a connection the symbol has since left stays
|
||||||
|
asked there while the other connection is down (sprint 08's review; rare,
|
||||||
|
and the resync count grows each minute).
|
||||||
|
- Paging Kalshi's listing in parallel: on 2026-11-12 the demo served a page
|
||||||
|
of 1,000 markets in 1.7 s, so the first scan took 7.5 minutes (under one
|
||||||
|
in sprint 06).
|
||||||
|
- Connecting to each of the exchange's addresses in turn, not only the
|
||||||
|
first the lookup returns (a host with IPv6 DNS but no IPv6 route would
|
||||||
|
never connect; sprint 08's review).
|
||||||
@@ -0,0 +1,425 @@
|
|||||||
|
# 08. Ready to submit
|
||||||
|
|
||||||
|
**Status: done (2026-11-12)**
|
||||||
|
|
||||||
|
M8, the last milestone: the collector handed to Voluntas in a state a
|
||||||
|
reviewer can read, run and check on their own. A review of the whole
|
||||||
|
codebase (not one sprint's diff) and its fixes; one command that runs the
|
||||||
|
collector and shows what it does; a README for the solution in place of
|
||||||
|
the template's; a final live run of each exchange, reconciled, whose
|
||||||
|
figures the README quotes; and the handoff as a public repo.
|
||||||
|
|
||||||
|
## The watched run
|
||||||
|
|
||||||
|
`scripts/watch.sh [EXCHANGE] [MINUTES]`, defaults `polymarket` (no key) and
|
||||||
|
3 minutes (two minute summaries; Polymarket tracks about 200,000 markets
|
||||||
|
within its first minute, `../runs/poly10.log`). It builds release, runs the
|
||||||
|
collector under `timeout -s INT`, and writes the output and the full log
|
||||||
|
to a gitignored `watch/`. On the terminal, a curated view of the log:
|
||||||
|
|
||||||
|
- each minute, the orchestrator's line as is, and the 16 book threads'
|
||||||
|
lines summed into one (changes written, duplicates, rescued, resyncs,
|
||||||
|
losses, subscription gaps; their counts are cumulative, "so far");
|
||||||
|
- between them, the notable events only: a connection lost, recycled
|
||||||
|
(hourly or slow), split, silent; the startup's connections as one count,
|
||||||
|
not a line each. The Snapshot cause lines (3,300 of 3,952 lines in a
|
||||||
|
10-minute Polymarket run) stay in the file.
|
||||||
|
|
||||||
|
Ctrl-C stops the run early and cleanly: the script goes on to the checks
|
||||||
|
(how Ctrl-C reaches `timeout` and the collector from a script is
|
||||||
|
**assumed**; a quick try without a terminal didn't deliver the signal, so
|
||||||
|
the first task tries it in one). Then it runs `reconcile` on the output
|
||||||
|
and the log, and `show` on the output.
|
||||||
|
|
||||||
|
On the workspace: `rc shell voluntas`, then `cd voluntas/repo &&
|
||||||
|
scripts/watch.sh` (`rc shell`'s interactive use is assumed from
|
||||||
|
`rc shell exec`). Kalshi needs `KALSHI_KEY_ID_FILE` and
|
||||||
|
`KALSHI_PRIVATE_KEY_FILE` exported; without them the collector stops
|
||||||
|
before connecting, naming the variable (`src/kalshi/sign.rs:26`). The
|
||||||
|
script adds nothing for keys.
|
||||||
|
|
||||||
|
## show
|
||||||
|
|
||||||
|
`src/bin/show.rs`: `show OUTPUT [SYMBOL]`, reading through the template's
|
||||||
|
`AlignedSbeReader` as `src/reconcile.rs` does. With no symbol, the one
|
||||||
|
with the most Deltas in the file. It prints that symbol's first Snapshot
|
||||||
|
(top 5 levels a side, its receive time), its next 20 Deltas (receive time,
|
||||||
|
side, price, new quantity; 0 shown as deleted), then the top 5 levels a
|
||||||
|
side of the book folded from them, in exchange units through
|
||||||
|
`book::format_fixed` (`src/book.rs:46`). A Polymarket symbol is its token
|
||||||
|
id: the output holds no market name. An unknown symbol is an error.
|
||||||
|
|
||||||
|
## The review
|
||||||
|
|
||||||
|
The whole codebase, not a diff: `src/`, `scripts/`, the tests, by five
|
||||||
|
`reviewer` subagents:
|
||||||
|
|
||||||
|
- **brief**, in place of a spec reviewer: the brief checked line by line
|
||||||
|
against the code (every live symbol, MVEs excluded on Kalshi, new and
|
||||||
|
gone symbols, Deltas to stdout and a Snapshot each minute, SIGINT closes
|
||||||
|
the websockets and the file, each extension);
|
||||||
|
- **smells**, read as a stranger: leftovers of superseded designs (lanes,
|
||||||
|
the template's demo), comments that need sprint history (19 citations
|
||||||
|
of `roadmap/0N` or "sprint 0N" in `src/`), the two exchanges
|
||||||
|
inconsistent with each other;
|
||||||
|
- **bugs**, **tests**, **security**, as usual; sprint 07's 11 post-review
|
||||||
|
fixes, never reviewed a second time, included.
|
||||||
|
|
||||||
|
Findings are sorted with the user: fix now (a task below, with its test),
|
||||||
|
or a limit or a "next" in the README.
|
||||||
|
|
||||||
|
## The README
|
||||||
|
|
||||||
|
Replaces the template's, for the Voluntas reader, in this order: what the
|
||||||
|
collector does, with the line on how it was built (below); quick start
|
||||||
|
(`scripts/watch.sh`); build and run per exchange, Kalshi's two `export`
|
||||||
|
lines, and that Kalshi ran only against the demo; checking the output
|
||||||
|
(`reconcile` with the log); the final runs' figures; the design in one
|
||||||
|
diagram (`docs/project.md`'s pipeline); each extension and where it is
|
||||||
|
met (ADRs, `LATENCY.md`); the limits (the crash ceiling, the subscription
|
||||||
|
gap when a symbol opens, Kalshi production never run); next (a short
|
||||||
|
pick of the roadmap's possible features); what to read first
|
||||||
|
(`docs/project.md`, `docs/adr/`, `LATENCY.md`, `docs/notes.md`, and
|
||||||
|
`GLOSSARY.md`, `CLAUDE.md`) and that `roadmap/` holds each sprint's plan
|
||||||
|
and outcome; an "Output format" section (the template's
|
||||||
|
`AlignedSbeWriter` unchanged, 64 MiB chunks, the schema's path). The
|
||||||
|
order as built supersedes the first one (the runs after the extensions,
|
||||||
|
the line on how it was built last): the runs read as evidence for the
|
||||||
|
design that follows, and the line belongs with what the project is. The
|
||||||
|
line:
|
||||||
|
|
||||||
|
> Built with Claude Code. I set the architecture (the threading model,
|
||||||
|
> redundancy, the exchange seam) and settled every design decision;
|
||||||
|
> Claude proposed the options, wrote the code and ran the experiments.
|
||||||
|
> Each decision and its reasons are in `roadmap/` and `docs/adr/`.
|
||||||
|
|
||||||
|
## Decisions
|
||||||
|
|
||||||
|
- **The watched run is a script over the existing log**, not pipeline
|
||||||
|
code. Rejected: a status line in the collector (counts across threads,
|
||||||
|
new pipeline code); a TUI.
|
||||||
|
- **The data is shown, not only counted** (`show`), after the run: the
|
||||||
|
writer emits 64 MiB chunks, so a live stream of the output would come
|
||||||
|
in bursts.
|
||||||
|
- **One script for everything**: the final runs and the README's quick
|
||||||
|
start are `scripts/watch.sh`, so the grader runs what made the
|
||||||
|
README's figures. The workspace's `../runs/run.sh` is retired.
|
||||||
|
- **Everything ships unchanged** (`docs/`, `roadmap/`, `CLAUDE.md`,
|
||||||
|
`GLOSSARY.md`, `LATENCY.md`); the README says what to read first.
|
||||||
|
Rejected: stripping the process files; the work being AI-assisted is
|
||||||
|
said openly instead.
|
||||||
|
- **Sprint citations in comments**: a file path stays (`roadmap/` ships);
|
||||||
|
a reference that leans on history is rewritten to say what the code
|
||||||
|
does, or to cite the ADR or `docs/notes.md`.
|
||||||
|
- **The final runs** pass `reconcile` (every mismatch explained, no
|
||||||
|
orphan Delta), load and peak memory recorded (`VmHWM` from `/proc`);
|
||||||
|
no comparison with earlier sprints' runs (done in their sprints; the
|
||||||
|
load differs from day to day).
|
||||||
|
- **The template's README text**: its stale lines go ("read `main.rs`",
|
||||||
|
"as demonstrated in `main.rs`"); the rest becomes "Output format".
|
||||||
|
- **Delivery: a public repo on the user's Gitea (git.pband.ch), not on
|
||||||
|
GitHub.** The brief says nothing on how (it links only the template,
|
||||||
|
public, 25 forks). Public, so the link opens with no account; on
|
||||||
|
GitHub it would publish a solution to the challenge where other
|
||||||
|
candidates search, which the user doesn't want. Rejected: a GitHub
|
||||||
|
fork (listed under the template), a GitHub repo of our own (found by
|
||||||
|
search), a private repo (an invite to a handle not known), a PR to the
|
||||||
|
template (public). Full history, the template's 3 commits first, so a
|
||||||
|
diff against it shows what was added. Pushed over SSH (which reaches
|
||||||
|
Gitea from here), with the user's go. Supersedes "a public repo of the
|
||||||
|
user's own on GitHub", at the user's request during the build.
|
||||||
|
- **Kalshi's RSA keys are taken** (from the review): RSA-PSS-SHA256 with
|
||||||
|
`ring`, PKCS#1 or PKCS#8, told from Ed25519 by the parsed key; tested
|
||||||
|
on RFC 9500's published test key (no private key of our own in a public
|
||||||
|
repo; `*.pem` is gitignored globally); checked
|
||||||
|
against `openssl`. Supersedes sprint 06's "Ed25519 only", because a
|
||||||
|
grader's key from Kalshi may well be RSA.
|
||||||
|
- **A slow connection takes the hourly recycle's next turn** (from the
|
||||||
|
review), in place of being recycled at once: the slow recycle waited for
|
||||||
|
the same moment as the hourly one, which nearly always took it first.
|
||||||
|
One path; the mark is logged when made. Supersedes sprint 07's "recycled
|
||||||
|
at once, ahead of the hourly order". `LATENCY.md`'s "no connection was
|
||||||
|
slow" in 20 live minutes rested on the starved count: reworded.
|
||||||
|
- **The final runs, as they went.** Polymarket's 10 minutes met an outage
|
||||||
|
of about two minutes (177 of 178 connections found silent at 16:50:17,
|
||||||
|
so since about 16:49:47; back from 16:51:38 to 16:51:59): kept in the
|
||||||
|
README, as it shows the recovery; 138,950 of its 140,129 losses (the
|
||||||
|
log's and `reconcile`'s) are those two minutes. Kalshi's 10-minute run spent 7.5
|
||||||
|
of them on the first scan (the demo's listing at 1.7 s a page) and saw
|
||||||
|
74 normal closes by the demo, never seen before; an A/B of `main`'s
|
||||||
|
build against the branch's, 10 minutes each, saw none on either, so
|
||||||
|
the closes were the demo's. The README's Kalshi run is 20 minutes,
|
||||||
|
long enough for the slow judgment (from a connection's 10th minute):
|
||||||
|
it recycled two. Supersedes "kalshi-demo 10". Logs in `../runs/`.
|
||||||
|
- **ShellCheck in CI**: `uvx --from shellcheck-py==0.10.0.1 shellcheck`
|
||||||
|
over `scripts/*.sh` (tried: 0.10.0, `scripts/ci.sh` clean).
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
|
||||||
|
- [x] Ctrl-C through the script, tried in a terminal: `timeout -s INT`
|
||||||
|
and the collector stop cleanly and the script goes on to the
|
||||||
|
checks; the script's handling settled from what happens.
|
||||||
|
- [x] `scripts/watch.sh`: build, bounded run, `watch/` (gitignored), the
|
||||||
|
curated view, then `reconcile` and `show`.
|
||||||
|
- [x] The view's summed book-thread line checked once by hand against
|
||||||
|
the 16 raw lines of the same minute.
|
||||||
|
- [x] ShellCheck in `scripts/ci.sh`.
|
||||||
|
- [x] `src/bin/show.rs`.
|
||||||
|
- [x] Test (`show`): on an output written by `AlignedSbeWriter` in the
|
||||||
|
test, the symbol with the most Deltas is picked; a symbol named is
|
||||||
|
used; its Deltas come in order and the folded book is right; an
|
||||||
|
unknown symbol is an error.
|
||||||
|
- [x] The review: five reviewers over the whole codebase; findings
|
||||||
|
sorted with the user, the fix-now ones added here as tasks.
|
||||||
|
- [x] The citations in comments the review lists, rewritten.
|
||||||
|
- [x] A short run, stopped with Ctrl-C partway: the output flushes and
|
||||||
|
reconciles.
|
||||||
|
- [x] The final runs, `scripts/watch.sh polymarket 10` and
|
||||||
|
`scripts/watch.sh kalshi-demo 20`, from the branch with every fix
|
||||||
|
in, load checked before each; both reconcile; load and peak
|
||||||
|
memory recorded.
|
||||||
|
- [x] `README.md`, its figures from those runs.
|
||||||
|
- [x] `docs/roadmap.md` (M8 done) and `CLAUDE.md` (watch, show) brought
|
||||||
|
up to date.
|
||||||
|
- [x] The stale branches `feat/full-coverage` and
|
||||||
|
`feat/redundant-connections` deleted.
|
||||||
|
- [ ] After the squash into `main`, with the user's go: the public repo
|
||||||
|
created on git.pband.ch and pushed, and its link opened logged
|
||||||
|
out.
|
||||||
|
|
||||||
|
From the review (2026-11-12, at `73a1551`):
|
||||||
|
|
||||||
|
- [x] Kalshi's RSA keys (PKCS#1 PEM, as Kalshi generates them), signed
|
||||||
|
RSA-PSS-SHA256 with `ring` (the scheme assumed until checked in
|
||||||
|
Kalshi's docs); Ed25519 kept. Test: an RSA signature verifies (with
|
||||||
|
`ring`; checked once by hand with `openssl`); a key of neither kind
|
||||||
|
is refused.
|
||||||
|
- [x] Usage names the two `KALSHI_*` variables; `--help` and `-h` print
|
||||||
|
it; the missing-key error names both variables.
|
||||||
|
- [x] The first scan is logged: when it starts, and every 10 s the
|
||||||
|
markets listed so far; `watch.awk` shows them.
|
||||||
|
- [x] A slow connection takes the hourly recycle's next turn, in place of
|
||||||
|
the oldest: one recycle path. Test: with the hourly recycle running,
|
||||||
|
a connection slow 3 minutes is the next recycled.
|
||||||
|
- [x] A symbol tracked again after its market was removed: its first
|
||||||
|
Snapshot logs a cause, tracked again, which `reconcile` reads. Test:
|
||||||
|
remove, add, the Snapshot reconciles with the log.
|
||||||
|
- [x] `watch.sh` exits non-zero when the collector or `reconcile` fails.
|
||||||
|
- [x] `book::add` checks for overflow; an overflow counts as a loss and
|
||||||
|
resyncs. Test.
|
||||||
|
- [x] Test: `orchestrator::run` offline against the local listing and
|
||||||
|
websocket servers; on stop, each socket gets a Close and the output
|
||||||
|
reconciles.
|
||||||
|
- [x] Tests: a command sent on an open connection reaches the server; a
|
||||||
|
1013 close becomes a split.
|
||||||
|
- [x] Tests for the review's surviving mutants: Kalshi's rule
|
||||||
|
(`src/kalshi/feed.rs` 94, 114, 215, 285), the shell
|
||||||
|
(`src/core/feed.rs` 483, 539, 589, 683, 687), `show` (several asks,
|
||||||
|
the 20-Delta cap, the first ladder), `reconcile` (orphan Deltas, one
|
||||||
|
cause line explains one Snapshot), the socket (a dead book thread
|
||||||
|
stops it, the PING).
|
||||||
|
- [x] Test: a sample log through `watch.awk`, from `scripts/ci.sh`.
|
||||||
|
- [x] `a_silent_connection_reconnects_with_its_tokens` fails instead of
|
||||||
|
hanging (a timeout on the server's accept).
|
||||||
|
- [x] "ponytail:" becomes "Ceiling:" (4 places); "lag" for lateness and
|
||||||
|
"stream" reworded (or "stream" in `GLOSSARY.md`); the 4 history
|
||||||
|
citations rewritten; `Cargo.toml`'s comments, the `CHUNK`
|
||||||
|
constants' names, `impl RecentChange`'s place, "the server" in
|
||||||
|
`watch.sh`.
|
||||||
|
|
||||||
|
From the review of `26b4d67`:
|
||||||
|
|
||||||
|
- [x] `watch.sh` records the peak thread count, not the last sample.
|
||||||
|
- [x] README: Polymarket's threads (one per connection plus about 20,
|
||||||
|
derived from the Kalshi run: 137 = 116 + 21, labelled so); losses
|
||||||
|
140,129 (the log's and `reconcile`'s), the same in this file; the
|
||||||
|
first reconnect at 16:51:38; 177 of the 178 connections; the outage
|
||||||
|
"about two minutes" (silence is noticed after 30 s), the roadmap
|
||||||
|
too; Kalshi's scan at about 1.2 s a page; "every sign of a loss
|
||||||
|
resyncs, and the count includes some that weren't"; "system load";
|
||||||
|
each term defined at its first use; `reconcile`'s Delta count only;
|
||||||
|
the unwrapped line.
|
||||||
|
- [x] This file: the RSA task says the `openssl` check was by hand; the
|
||||||
|
README section's order follows the README; "Done when" names the
|
||||||
|
20-minute Kalshi run.
|
||||||
|
- [x] Test: a "tracked again" line explains a mismatch in `reconcile`.
|
||||||
|
- [x] Test: an overflowing Kalshi change asks a resync and removes the
|
||||||
|
level.
|
||||||
|
- [x] The view prints a minute's sum on the first line 2 s or more after
|
||||||
|
its first book line, too (a book thread with nothing logs none).
|
||||||
|
Test: a minute missing one thread is printed before the
|
||||||
|
orchestrator's line.
|
||||||
|
- [x] "Next to recycle" is logged when the mark is new only; the slow log
|
||||||
|
text in one place. Test.
|
||||||
|
- [x] Test: the `reconcile` binary fails on an unexplained mismatch.
|
||||||
|
- [x] Test: a mark is cleared when a later judgment finds the connection
|
||||||
|
no longer slow.
|
||||||
|
- [x] Test: `--help` prints the usage and exits 0.
|
||||||
|
- [x] The two timing-dependent tests poll until their condition, under a
|
||||||
|
timeout, in place of a fixed sleep.
|
||||||
|
- [x] `gamma.rs`'s "stream what they read"; `orchestrator.rs:60`'s
|
||||||
|
unwrapped doc comment; the Kalshi test's hand-built `Market` uses
|
||||||
|
`market()`.
|
||||||
|
|
||||||
|
From the review of `d94f683`:
|
||||||
|
|
||||||
|
- [x] The view's window is 10 s, to the millisecond and across midnight:
|
||||||
|
a minute's book-thread lines spread 1.6 s in the final Polymarket
|
||||||
|
run, and the 2 s rule, in whole seconds, could split its totals.
|
||||||
|
Tests: a thread 1.6 s late, live and final; across midnight.
|
||||||
|
- [x] "Next to recycle" said once per connection while it stays slow,
|
||||||
|
even when two marked connections swap ranks. Test.
|
||||||
|
- [x] README: silent between 16:50:17 and 16:50:25; back from 16:51:37;
|
||||||
|
recycling explained before the slow recycle; a line wrapped.
|
||||||
|
- [x] The `reconcile` binary's test matches `unexplained: 1 at `, not the
|
||||||
|
symbol alone.
|
||||||
|
|
||||||
|
Each test is seen to fail once (the code it protects removed, then put
|
||||||
|
back).
|
||||||
|
|
||||||
|
## Review of `d94f683` (2026-11-12)
|
||||||
|
|
||||||
|
The 7 commits after the second review's tasks, one reviewer on the five
|
||||||
|
questions. Fix now: the four tasks above. No: the test of an unknown
|
||||||
|
exchange (unasked, kept: cheap and useful); the 1013 test's copy of
|
||||||
|
`client` (changing `client` touches every socket test); `output(orphan,
|
||||||
|
mismatched)`'s two flags (a test helper). Assumed, written so in
|
||||||
|
`watch.awk` before: "they come within milliseconds", which the final
|
||||||
|
run's log contradicted.
|
||||||
|
|
||||||
|
## Review of `26b4d67` (2026-11-12)
|
||||||
|
|
||||||
|
The diff since `73a1551`, five reviewers; none of the findings dropped.
|
||||||
|
Fix now: the tasks above. Otherwise:
|
||||||
|
|
||||||
|
- **Spec:** no: the Polymarket run predating the view's fix (`src/`
|
||||||
|
unchanged since; the sums recomputed on the log); "sprint 08's Outcome"
|
||||||
|
(written at the close); the README's longer reading list.
|
||||||
|
- **Tests:** no: `watch.sh`'s exit status (needs the live exchange or a
|
||||||
|
fake collector; tried by hand); "neither kind" with an EC key (garbage
|
||||||
|
covers the refusal); the first-scan lines typed in `test_watch.sh`
|
||||||
|
(`CLAUDE.md` says change both).
|
||||||
|
- **Smells:** no: `Symbol::again` carrying the exchange's name
|
||||||
|
(documented); `64 << 20` in the library's tests (the binary's constant
|
||||||
|
isn't visible there); `run_tests` serving Kalshi locally (real sockets).
|
||||||
|
- **Security:** no vulnerability.
|
||||||
|
|
||||||
|
## Review of `73a1551` (2026-11-12)
|
||||||
|
|
||||||
|
The whole codebase, five reviewers (brief, bugs, tests, smells,
|
||||||
|
security); the tests reviewer ran 40 mutants, 19 survived. 2 findings
|
||||||
|
dropped (files since committed; a 429 that recovered). Fix now: the
|
||||||
|
tasks above. Otherwise:
|
||||||
|
|
||||||
|
- **Brief:** quiet judged against the twin connection, a `Rule` beyond
|
||||||
|
encode/decode, first one wins in place of a source switch: the README
|
||||||
|
explains each with its ADR. No: `/dev/stdout` truncating a `>>`.
|
||||||
|
- **Bugs:** later (`docs/roadmap.md`): a Kalshi resync left asked on a
|
||||||
|
connection the symbol left; only the first resolved address tried. No:
|
||||||
|
Polymarket changes held at stop, at most 1 s (a README limit); the
|
||||||
|
probe's exit 0 when every connection drops.
|
||||||
|
- **Smells:** no: the two fetchers, `Rule::Symbol`, the socket's data
|
||||||
|
clumps, `on_book`'s length, the exchanges' decoding and construction.
|
||||||
|
- **Tests, as built:** every new test seen red on its mutant, but one:
|
||||||
|
removing `drop(outputs)` at the top of `orchestrator::run` changes
|
||||||
|
nothing a test can see (`outputs` drops when `run` returns, which is
|
||||||
|
after `threads.join()`); kept, as it says when the writer may end.
|
||||||
|
- **Security:** no secret in the tree or history. No: unbounded channels
|
||||||
|
(never dropping is the design; a README limit), raw exchange text in
|
||||||
|
the log, the template's `unsafe-libyaml`. Kept, the user's call: the
|
||||||
|
commit email and `CLAUDE.md`'s DNS lines in the public history.
|
||||||
|
|
||||||
|
## Not in this sprint
|
||||||
|
|
||||||
|
The trading simulator of `docs/evaluation.md` (its fill model is a project
|
||||||
|
of its own; reconcile, lateness and rescued counts already measure the
|
||||||
|
claims); the roadmap's possible features (the README's "next");
|
||||||
|
containerized packaging; a Kalshi production run (no key).
|
||||||
|
|
||||||
|
## Done when
|
||||||
|
|
||||||
|
- The review's findings are sorted; the fix-now ones are fixed and
|
||||||
|
tested.
|
||||||
|
- `scripts/watch.sh` and `show` work; Ctrl-C was tried.
|
||||||
|
- Both final runs (Polymarket 10 minutes, Kalshi's demo 20) reconcile,
|
||||||
|
load and memory recorded.
|
||||||
|
- `README.md` is written, its figures from those runs.
|
||||||
|
- The stale branches are deleted.
|
||||||
|
- `scripts/ci.sh` is green, ShellCheck included.
|
||||||
|
|
||||||
|
After the squash:
|
||||||
|
|
||||||
|
- The public repo is created on git.pband.ch and pushed, with the user's
|
||||||
|
go, and its link works logged out.
|
||||||
|
|
||||||
|
## Outcome
|
||||||
|
|
||||||
|
The collector is ready to hand over. `scripts/watch.sh` runs it with a live
|
||||||
|
view, then `reconcile` and `show` (one symbol's book as text); the README
|
||||||
|
replaces the template's; three reviews (the whole codebase, then each round
|
||||||
|
of fixes) and their fixes are in; both final runs reconciled.
|
||||||
|
|
||||||
|
- **Fixed, from the whole-codebase review:** Kalshi takes RSA keys too
|
||||||
|
(RSA-PSS-SHA256, PKCS#1 or PKCS#8, told by the parsed key); the slow
|
||||||
|
recycle, which the hourly one nearly always took first, now marks the
|
||||||
|
connection and takes the next turn, and acted twice in the 20-minute
|
||||||
|
Kalshi run; a market tracked again says so, so `reconcile` doesn't fail
|
||||||
|
on it; `Book::add` checks for overflow; usage, `--help` and the
|
||||||
|
missing-key error name both key variables; the first scan says how far
|
||||||
|
it got. 19 surviving mutants got their tests, and an offline run of the
|
||||||
|
whole orchestrator proves SIGINT closes every socket and the output
|
||||||
|
reconciles.
|
||||||
|
- **Final runs:** Polymarket, 10 minutes, 199,531 markets on 178
|
||||||
|
connections, 14.4 million Deltas, through an outage of about two minutes
|
||||||
|
that it recovered from on its own (138,950 of its 140,129 losses).
|
||||||
|
Kalshi's demo, 20 minutes, 260,865 markets on 116 connections, 4.1
|
||||||
|
million Deltas, 12 losses, two slow connections recycled.
|
||||||
|
|
||||||
|
**Diverged from the plan**, each written into this file when it
|
||||||
|
happened: delivery on the user's Gitea, not GitHub (the user's call, so as
|
||||||
|
not to publish a solution where candidates search); RSA keys taken
|
||||||
|
(sprint 06 had Ed25519 only), tested on RFC 9500's published key since
|
||||||
|
`*.pem` is ignored globally; the slow recycle takes the hourly turn, not
|
||||||
|
recycled at once; the Kalshi final run is 20 minutes, not 10 (the demo's
|
||||||
|
listing took 5 to 7.5 minutes to page); the README's sections in another
|
||||||
|
order; `LATENCY.md`'s "no connection was slow" withdrawn, since it rested
|
||||||
|
on the starved count.
|
||||||
|
|
||||||
|
**Left out**, and where it went: a Kalshi resync left asked on a
|
||||||
|
connection the symbol left; connecting to every resolved address;
|
||||||
|
paging Kalshi's listing in parallel (`docs/roadmap.md`, possible
|
||||||
|
features). Unexplained, recorded: the Polymarket outage's cause, and the
|
||||||
|
74 normal closes by Kalshi's demo in one run (an A/B ruled out our code).
|
||||||
|
|
||||||
|
**Assumed**, not shown: that the outage was on the network's side ("no
|
||||||
|
route to host"); that RSA keys work against Kalshi itself (tried offline
|
||||||
|
only, against `openssl`); `rc shell`'s interactive use; Polymarket's
|
||||||
|
thread count, about 200, derived from the Kalshi run's.
|
||||||
|
|
||||||
|
**Reviewed** at `73a1551` (the whole codebase), `26b4d67` and `d94f683`;
|
||||||
|
the 3 commits after `d94f683` are its fixes, each with a test seen to
|
||||||
|
fail, and had no further review.
|
||||||
|
|
||||||
|
### Retrospective
|
||||||
|
|
||||||
|
- **Went well:** reviewing the whole codebase, not the sprint's diff, found
|
||||||
|
a starved mechanism that had made a sprint 07 claim false; a sprint diff
|
||||||
|
never would have. Rerunning one key mutant of each subagent's tests, and
|
||||||
|
an A/B against `main`'s build, kept their claims and the demo's closes
|
||||||
|
honest. The user's watchable run turned out to be the best test of the
|
||||||
|
view: each live run found something.
|
||||||
|
- **Went wrong:** my own instruments misled me four times: `head` cut the
|
||||||
|
mutation output so a red test looked green, `$?` after a pipe reported
|
||||||
|
the pipe's last command, the view summed part of a minute, and a count
|
||||||
|
grouped by 10 s gave a reconnect time 8 s early. The README's first
|
||||||
|
figures came from the view and from memory, not from `reconcile` and the
|
||||||
|
log, and two reviews corrected them. The view's 2 s window rested on
|
||||||
|
"they come within milliseconds", never measured. Three review rounds
|
||||||
|
where one careful round of fixes would have done.
|
||||||
|
- **Tested enough?** The fixes are, by removal. Not: `watch.sh`'s exit
|
||||||
|
status and its sampling (live only), RSA against Kalshi, the "tracked
|
||||||
|
again" log line itself (stderr), the first-scan lines' wording against
|
||||||
|
the view (typed in the test).
|
||||||
|
- **Next time:** take every quoted figure from the tool that checks it
|
||||||
|
(`reconcile`, the log), never from a summary of it; check an exit status
|
||||||
|
without a pipe; measure a timing before building a rule on it.
|
||||||
@@ -13,3 +13,8 @@ cargo test
|
|||||||
uvx ruff@0.15.0 format --check scripts/latency
|
uvx ruff@0.15.0 format --check scripts/latency
|
||||||
uvx ruff@0.15.0 check scripts/latency
|
uvx ruff@0.15.0 check scripts/latency
|
||||||
python3 -m unittest discover -s scripts/latency
|
python3 -m unittest discover -s scripts/latency
|
||||||
|
|
||||||
|
# The shell scripts: ShellCheck pinned, run through uvx as ruff is.
|
||||||
|
uvx --from shellcheck-py==0.10.0.1 shellcheck scripts/*.sh
|
||||||
|
# The view of watch.sh, on a sample log.
|
||||||
|
sh scripts/test_watch.sh
|
||||||
Executable
+157
@@ -0,0 +1,157 @@
|
|||||||
|
#!/bin/sh
|
||||||
|
# Pipes a sample log of each exchange through scripts/watch.awk, live and
|
||||||
|
# with -v final=1, and compares what it prints with what the view must show;
|
||||||
|
# exits non-zero on a difference. The lines are worded as the collector
|
||||||
|
# writes them: the book threads' minute lines (log_minute in
|
||||||
|
# src/polymarket/feed.rs and src/kalshi/feed.rs), the orchestrator's
|
||||||
|
# (src/core/orchestrator.rs), the connections' (src/core/socket.rs) and a
|
||||||
|
# Snapshot's cause (src/cause.rs).
|
||||||
|
set -eu
|
||||||
|
cd "$(dirname "$0")/.."
|
||||||
|
|
||||||
|
failed=0
|
||||||
|
# check NAME LOG EXPECTED [AWK OPTION...]
|
||||||
|
check() {
|
||||||
|
name=$1 log=$2 expected=$3
|
||||||
|
shift 3
|
||||||
|
actual=$(printf '%s\n' "$log" | awk -v book_threads=2 "$@" -f scripts/watch.awk)
|
||||||
|
if [ "$actual" != "$expected" ]; then
|
||||||
|
printf 'watch.awk, %s: printed\n%s\nexpected\n%s\n' "$name" "$actual" "$expected"
|
||||||
|
failed=1
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
polymarket=$(
|
||||||
|
cat <<'EOF'
|
||||||
|
12:00:00.100 polymarket: orchestrator: first scan, listing the exchange's markets
|
||||||
|
12:00:30.200 polymarket: orchestrator: first scan done, 40 live markets, subscribing
|
||||||
|
12:00:30.300 polymarket: connection 0: connected, 40 symbols
|
||||||
|
12:00:30.310 polymarket: connection 1: connected, 40 symbols
|
||||||
|
12:01:00.001 polymarket: book thread 0: 10 symbols; so far 100 changes written (90 verified), 50 duplicates, 2 rescued, 0 held dropped, 1 chains broken, 3 resyncs, 0 full books behind ours, 1 subscription gaps, 2 losses, 0 switches, 0 quiet subscriptions resubscribed, 0 markets reported quiet again, 4 reconnect books judged (1 losses); duplicates' lateness 1.5 ms at the median, 9.0 ms at p99; this minute 150 changes received, at most 4 events waited
|
||||||
|
12:01:00.002 polymarket: book thread 1: 30 symbols; so far 300 changes written (290 verified), 150 duplicates, 4 rescued, 0 held dropped, 0 chains broken, 1 resyncs, 0 full books behind ours, 0 subscription gaps, 1 losses, 0 switches, 0 quiet subscriptions resubscribed, 0 markets reported quiet again, 0 reconnect books judged (0 losses); duplicates' lateness 2.0 ms at the median, 8.0 ms at p99; this minute 450 changes received, at most 9 events waited
|
||||||
|
12:01:00.050 polymarket: Snapshot of 123 at 1794390060050000: loss
|
||||||
|
12:01:30.400 polymarket: orchestrator: 40 markets on 2 connections; 40 added, 0 removed, 0 moved (0 quiet), 0 splits, 0 recycled (0 slow), 0 silent this minute; busiest connection at 1200 bytes/s
|
||||||
|
12:02:00.001 polymarket: book thread 0: 10 symbols; so far 160 changes written (150 verified), 80 duplicates, 3 rescued, 0 held dropped, 1 chains broken, 3 resyncs, 0 full books behind ours, 1 subscription gaps, 2 losses, 0 switches, 1 quiet subscriptions resubscribed, 0 markets reported quiet again, 4 reconnect books judged (1 losses); duplicates' lateness 1.5 ms at the median, 9.0 ms at p99; this minute 90 changes received, at most 2 events waited
|
||||||
|
12:02:00.002 polymarket: book thread 1: 30 symbols; so far 500 changes written (480 verified), 250 duplicates, 6 rescued, 0 held dropped, 0 chains broken, 2 resyncs, 0 full books behind ours, 0 subscription gaps, 1 losses, 0 switches, 0 quiet subscriptions resubscribed, 0 markets reported quiet again, 0 reconnect books judged (0 losses); duplicates' lateness 2.0 ms at the median, 8.0 ms at p99; this minute 300 changes received, at most 5 events waited
|
||||||
|
12:02:10.000 polymarket: connection 1: recycled
|
||||||
|
12:02:11.000 polymarket: connection 1: connected, 40 symbols
|
||||||
|
12:02:30.400 polymarket: orchestrator: 40 markets on 2 connections; 0 added, 0 removed, 0 moved (0 quiet), 0 splits, 1 recycled (0 slow), 0 silent this minute; busiest connection at 1300 bytes/s
|
||||||
|
EOF
|
||||||
|
)
|
||||||
|
|
||||||
|
check "Polymarket, live" "$polymarket" "$(
|
||||||
|
cat <<'EOF'
|
||||||
|
12:00:00 polymarket: orchestrator: first scan, listing the exchange's markets
|
||||||
|
12:00:30 polymarket: orchestrator: first scan done, 40 live markets, subscribing
|
||||||
|
12:00:30 connections opening (counted in the minute line)
|
||||||
|
12:01 books (2 threads): 40 symbols, 400 changes written, 200 duplicates, 6 rescued, 4 resyncs, 1 subscription gaps, 3 losses
|
||||||
|
12:01:30 40 markets on 2 connections; 40 added, 0 removed, 0 moved (0 quiet), 0 splits, 0 recycled (0 slow), 0 silent this minute; busiest connection at 1200 bytes/s; 2 connected (0 refused, retried), 1 Snapshots with a cause
|
||||||
|
12:02 books (2 threads): 40 symbols, 660 changes written, 330 duplicates, 9 rescued, 5 resyncs, 1 subscription gaps, 3 losses
|
||||||
|
12:02:10 polymarket: connection 1: recycled
|
||||||
|
12:02:30 40 markets on 2 connections; 0 added, 0 removed, 0 moved (0 quiet), 0 splits, 1 recycled (0 slow), 0 silent this minute; busiest connection at 1300 bytes/s; 1 connected (0 refused, retried), 0 Snapshots with a cause
|
||||||
|
EOF
|
||||||
|
)"
|
||||||
|
check "Polymarket, final" "$polymarket" \
|
||||||
|
"12:02 books (2 threads): 40 symbols, 660 changes written, 330 duplicates, 9 rescued, 5 resyncs, 1 subscription gaps, 3 losses" \
|
||||||
|
-v final=1
|
||||||
|
|
||||||
|
# A minute with one of its two book threads' lines: printed with the first
|
||||||
|
# line 10 s after it, before the orchestrator's.
|
||||||
|
missing=$(
|
||||||
|
cat <<'EOF'
|
||||||
|
12:01:00.001 polymarket: book thread 0: 10 symbols; so far 100 changes written (90 verified), 50 duplicates, 2 rescued, 0 held dropped, 1 chains broken, 3 resyncs, 0 full books behind ours, 1 subscription gaps, 2 losses, 0 switches, 0 quiet subscriptions resubscribed, 0 markets reported quiet again, 4 reconnect books judged (1 losses); duplicates' lateness 1.5 ms at the median, 9.0 ms at p99; this minute 150 changes received, at most 4 events waited
|
||||||
|
12:01:00.500 polymarket: connection 1: recycled
|
||||||
|
12:01:30.400 polymarket: orchestrator: 40 markets on 2 connections; 0 added, 0 removed, 0 moved (0 quiet), 0 splits, 1 recycled (0 slow), 0 silent this minute; busiest connection at 1200 bytes/s
|
||||||
|
EOF
|
||||||
|
)
|
||||||
|
check "a book thread missing" "$missing" "$(
|
||||||
|
cat <<'EOF'
|
||||||
|
12:01:00 polymarket: connection 1: recycled
|
||||||
|
12:01 books (1 threads): 10 symbols, 100 changes written, 50 duplicates, 2 rescued, 3 resyncs, 1 subscription gaps, 2 losses
|
||||||
|
12:01:30 40 markets on 2 connections; 0 added, 0 removed, 0 moved (0 quiet), 0 splits, 1 recycled (0 slow), 0 silent this minute; busiest connection at 1200 bytes/s; 0 connected (0 refused, retried), 0 Snapshots with a cause
|
||||||
|
EOF
|
||||||
|
)"
|
||||||
|
|
||||||
|
# A thread's line 1.6 s after the first, past a line between them: still
|
||||||
|
# the same minute's, summed whole.
|
||||||
|
late=$(
|
||||||
|
cat <<'EOF'
|
||||||
|
12:01:00.075 polymarket: book thread 0: 10 symbols; so far 100 changes written (90 verified), 50 duplicates, 2 rescued, 0 held dropped, 1 chains broken, 3 resyncs, 0 full books behind ours, 1 subscription gaps, 2 losses, 0 switches, 0 quiet subscriptions resubscribed, 0 markets reported quiet again, 4 reconnect books judged (1 losses); duplicates' lateness 1.5 ms at the median, 9.0 ms at p99; this minute 150 changes received, at most 4 events waited
|
||||||
|
12:01:01.600 polymarket: connection 1: recycled
|
||||||
|
12:01:01.672 polymarket: book thread 1: 30 symbols; so far 300 changes written (290 verified), 150 duplicates, 4 rescued, 0 held dropped, 0 chains broken, 1 resyncs, 0 full books behind ours, 0 subscription gaps, 1 losses, 0 switches, 0 quiet subscriptions resubscribed, 0 markets reported quiet again, 0 reconnect books judged (0 losses); duplicates' lateness 2.0 ms at the median, 8.0 ms at p99; this minute 450 changes received, at most 9 events waited
|
||||||
|
EOF
|
||||||
|
)
|
||||||
|
check "a thread 1.6 s late" "$late" "$(
|
||||||
|
cat <<'EOF'
|
||||||
|
12:01:01 polymarket: connection 1: recycled
|
||||||
|
12:01 books (2 threads): 40 symbols, 400 changes written, 200 duplicates, 6 rescued, 4 resyncs, 1 subscription gaps, 3 losses
|
||||||
|
EOF
|
||||||
|
)"
|
||||||
|
check "a thread 1.6 s late, final" "$late" \
|
||||||
|
"12:01 books (2 threads): 40 symbols, 400 changes written, 200 duplicates, 6 rescued, 4 resyncs, 1 subscription gaps, 3 losses" \
|
||||||
|
-v final=1
|
||||||
|
|
||||||
|
# Across midnight: a minute begun at 23:59 still ends 10 s on.
|
||||||
|
midnight=$(
|
||||||
|
cat <<'EOF'
|
||||||
|
23:59:00.001 polymarket: book thread 0: 10 symbols; so far 100 changes written (90 verified), 50 duplicates, 2 rescued, 0 held dropped, 1 chains broken, 3 resyncs, 0 full books behind ours, 1 subscription gaps, 2 losses, 0 switches, 0 quiet subscriptions resubscribed, 0 markets reported quiet again, 4 reconnect books judged (1 losses); duplicates' lateness 1.5 ms at the median, 9.0 ms at p99; this minute 150 changes received, at most 4 events waited
|
||||||
|
00:00:30.000 polymarket: connection 1: recycled
|
||||||
|
EOF
|
||||||
|
)
|
||||||
|
check "across midnight" "$midnight" "$(
|
||||||
|
cat <<'EOF'
|
||||||
|
23:59 books (1 threads): 10 symbols, 100 changes written, 50 duplicates, 2 rescued, 3 resyncs, 1 subscription gaps, 2 losses
|
||||||
|
00:00:30 polymarket: connection 1: recycled
|
||||||
|
EOF
|
||||||
|
)"
|
||||||
|
|
||||||
|
# A line between a minute's book threads: the minute is still summed whole.
|
||||||
|
interleaved=$(
|
||||||
|
cat <<'EOF'
|
||||||
|
12:01:00.001 polymarket: book thread 0: 10 symbols; so far 100 changes written (90 verified), 50 duplicates, 2 rescued, 0 held dropped, 1 chains broken, 3 resyncs, 0 full books behind ours, 1 subscription gaps, 2 losses, 0 switches, 0 quiet subscriptions resubscribed, 0 markets reported quiet again, 4 reconnect books judged (1 losses); duplicates' lateness 1.5 ms at the median, 9.0 ms at p99; this minute 150 changes received, at most 4 events waited
|
||||||
|
12:01:00.002 polymarket: connection 1: recycled
|
||||||
|
12:01:00.003 polymarket: book thread 1: 30 symbols; so far 300 changes written (290 verified), 150 duplicates, 4 rescued, 0 held dropped, 0 chains broken, 1 resyncs, 0 full books behind ours, 0 subscription gaps, 1 losses, 0 switches, 0 quiet subscriptions resubscribed, 0 markets reported quiet again, 0 reconnect books judged (0 losses); duplicates' lateness 2.0 ms at the median, 8.0 ms at p99; this minute 450 changes received, at most 9 events waited
|
||||||
|
EOF
|
||||||
|
)
|
||||||
|
check "a line between book threads" "$interleaved" "$(
|
||||||
|
cat <<'EOF'
|
||||||
|
12:01:00 polymarket: connection 1: recycled
|
||||||
|
12:01 books (2 threads): 40 symbols, 400 changes written, 200 duplicates, 6 rescued, 4 resyncs, 1 subscription gaps, 3 losses
|
||||||
|
EOF
|
||||||
|
)"
|
||||||
|
|
||||||
|
kalshi=$(
|
||||||
|
cat <<'EOF'
|
||||||
|
12:00:00.100 kalshi: orchestrator: first scan, listing the exchange's markets
|
||||||
|
12:00:10.100 kalshi: orchestrator: first scan, 1000 live markets listed so far
|
||||||
|
12:00:20.200 kalshi: orchestrator: first scan done, 1500 live markets, subscribing
|
||||||
|
12:00:20.300 kalshi: connection 0: connected, 1500 symbols
|
||||||
|
12:00:20.310 kalshi: connection 1: connect failed: handshake refused: 429 Too Many Requests
|
||||||
|
12:00:21.320 kalshi: connection 1: connected, 1500 symbols
|
||||||
|
12:01:00.001 kalshi: book thread 0: 700 symbols; so far 100 changes written, 90 duplicates, 1 before their connection's book, 0 ahead of the book, 3 fresh books (1 losses), 0 below zero, 2 resyncs, 0 quiet subscriptions resubscribed, 0 markets reported quiet again; this minute 190 changes received, at most 3 events waited
|
||||||
|
12:01:00.002 kalshi: book thread 1: 800 symbols; so far 200 changes written, 190 duplicates, 0 before their connection's book, 1 ahead of the book, 2 fresh books (0 losses), 1 below zero, 1 resyncs, 0 quiet subscriptions resubscribed, 0 markets reported quiet again; this minute 390 changes received, at most 6 events waited
|
||||||
|
12:01:00.030 kalshi: Snapshot of KXA at 1794390060030000: loss (its clock 12, ours 11)
|
||||||
|
12:01:20.400 kalshi: orchestrator: 1500 markets on 2 connections; 1500 added, 0 removed, 0 moved (0 quiet), 0 splits, 0 recycled (0 slow), 0 silent this minute; busiest connection at 9000 bytes/s
|
||||||
|
12:02:00.001 kalshi: book thread 0: 700 symbols; so far 150 changes written, 140 duplicates, 1 before their connection's book, 0 ahead of the book, 3 fresh books (1 losses), 0 below zero, 2 resyncs, 0 quiet subscriptions resubscribed, 0 markets reported quiet again; this minute 100 changes received, at most 2 events waited
|
||||||
|
12:02:00.002 kalshi: book thread 1: 800 symbols; so far 260 changes written, 250 duplicates, 0 before their connection's book, 1 ahead of the book, 2 fresh books (0 losses), 1 below zero, 1 resyncs, 0 quiet subscriptions resubscribed, 0 markets reported quiet again; this minute 120 changes received, at most 4 events waited
|
||||||
|
12:02:20.400 kalshi: orchestrator: 1500 markets on 2 connections; 0 added, 0 removed, 0 moved (0 quiet), 0 splits, 0 recycled (0 slow), 0 silent this minute; busiest connection at 8000 bytes/s
|
||||||
|
EOF
|
||||||
|
)
|
||||||
|
|
||||||
|
check "Kalshi, live" "$kalshi" "$(
|
||||||
|
cat <<'EOF'
|
||||||
|
12:00:00 kalshi: orchestrator: first scan, listing the exchange's markets
|
||||||
|
12:00:10 kalshi: orchestrator: first scan, 1000 live markets listed so far
|
||||||
|
12:00:20 kalshi: orchestrator: first scan done, 1500 live markets, subscribing
|
||||||
|
12:00:20 connections opening (counted in the minute line)
|
||||||
|
12:01 books (2 threads): 1500 symbols, 300 changes written, 280 duplicates, 3 resyncs, 1 losses
|
||||||
|
12:01:20 1500 markets on 2 connections; 1500 added, 0 removed, 0 moved (0 quiet), 0 splits, 0 recycled (0 slow), 0 silent this minute; busiest connection at 9000 bytes/s; 2 connected (1 refused, retried), 1 Snapshots with a cause
|
||||||
|
12:02 books (2 threads): 1500 symbols, 410 changes written, 390 duplicates, 3 resyncs, 1 losses
|
||||||
|
12:02:20 1500 markets on 2 connections; 0 added, 0 removed, 0 moved (0 quiet), 0 splits, 0 recycled (0 slow), 0 silent this minute; busiest connection at 8000 bytes/s; 0 connected (0 refused, retried), 0 Snapshots with a cause
|
||||||
|
EOF
|
||||||
|
)"
|
||||||
|
check "Kalshi, final" "$kalshi" \
|
||||||
|
"12:02 books (2 threads): 1500 symbols, 410 changes written, 390 duplicates, 3 resyncs, 1 losses" \
|
||||||
|
-v final=1
|
||||||
|
|
||||||
|
exit "$failed"
|
||||||
@@ -0,0 +1,107 @@
|
|||||||
|
# The view of scripts/watch.sh: reads the collector's log and prints, each
|
||||||
|
# minute, the orchestrator's line and the book threads' lines summed into one,
|
||||||
|
# and between them only the notable events. The Snapshot cause lines and the
|
||||||
|
# connections coming up are counted, not shown.
|
||||||
|
#
|
||||||
|
# With -v final=1 it prints nothing live, only the last minute's sums: the
|
||||||
|
# run's totals, since the book threads count "so far". A minute's sum is
|
||||||
|
# printed once all its book threads' lines are in (-v book_threads=N, 16 by
|
||||||
|
# default, src/core/mod.rs's BOOK_THREADS), or with the first other line 10 s
|
||||||
|
# or more after the minute's first, or when the next minute begins. A book
|
||||||
|
# thread that received nothing logs no line; the others' come within a few
|
||||||
|
# seconds of the minute (1.6 s apart at most in a 10-minute Polymarket run,
|
||||||
|
# 2026-11-12), and other lines often fall between them.
|
||||||
|
|
||||||
|
# The book threads' counters shown, in this order; a name missing from an
|
||||||
|
# exchange's line (Kalshi has no "rescued") is left out.
|
||||||
|
BEGIN {
|
||||||
|
if (book_threads == "") book_threads = 16
|
||||||
|
shown = "symbols|changes written|duplicates|rescued|resyncs|subscription gaps|losses"
|
||||||
|
count = split(shown, names, "|")
|
||||||
|
}
|
||||||
|
|
||||||
|
function flush( line, i) {
|
||||||
|
if (threads == 0) return
|
||||||
|
line = minute " books (" threads " threads):"
|
||||||
|
for (i = 1; i <= count; i++)
|
||||||
|
if (names[i] in sum) line = line " " sum[names[i]] " " names[i] ","
|
||||||
|
sub(/,$/, "", line)
|
||||||
|
last = line
|
||||||
|
event(line)
|
||||||
|
threads = 0
|
||||||
|
split("", sum)
|
||||||
|
}
|
||||||
|
|
||||||
|
# Printed at once: the view is read while the run goes on, and mawk
|
||||||
|
# buffers its output otherwise.
|
||||||
|
function event(text) {
|
||||||
|
if (final) return
|
||||||
|
print text
|
||||||
|
fflush()
|
||||||
|
}
|
||||||
|
|
||||||
|
{
|
||||||
|
time = substr($1, 1, 8)
|
||||||
|
text = substr($0, index($0, " ") + 1)
|
||||||
|
# To the millisecond: $1 is HH:MM:SS.mmm.
|
||||||
|
seconds = substr($1, 1, 2) * 3600 + substr($1, 4, 2) * 60 + substr($1, 7)
|
||||||
|
}
|
||||||
|
|
||||||
|
/: book thread [0-9]+: / {
|
||||||
|
this = substr(time, 1, 5)
|
||||||
|
if (threads > 0 && this != minute) flush()
|
||||||
|
if (threads == 0) first_at = seconds
|
||||||
|
minute = this
|
||||||
|
threads++
|
||||||
|
body = text
|
||||||
|
sub(/; this minute .*/, "", body)
|
||||||
|
split("", seen)
|
||||||
|
# Each "N name": a count and its name, up to the next comma, semicolon
|
||||||
|
# or parenthesis. A name is summed once a line (its first occurrence).
|
||||||
|
while (match(body, /[0-9]+ [a-z][a-z' ]*[a-z]/)) {
|
||||||
|
pair = substr(body, RSTART, RLENGTH)
|
||||||
|
body = substr(body, RSTART + RLENGTH)
|
||||||
|
n = substr(pair, 1, index(pair, " ") - 1)
|
||||||
|
name = substr(pair, index(pair, " ") + 1)
|
||||||
|
sub(/^so far /, "", name)
|
||||||
|
if (name in seen) continue
|
||||||
|
seen[name] = 1
|
||||||
|
sum[name] += n
|
||||||
|
}
|
||||||
|
if (threads == book_threads) flush()
|
||||||
|
next
|
||||||
|
}
|
||||||
|
|
||||||
|
# A minute with a book thread's line missing: printed once it is clearly over
|
||||||
|
# (a day's seconds added back across midnight).
|
||||||
|
threads > 0 && (seconds - first_at + 86400) % 86400 >= 10 { flush() }
|
||||||
|
|
||||||
|
/: orchestrator: [0-9]+ markets on/ {
|
||||||
|
line = text
|
||||||
|
sub(/^[a-z-]+: orchestrator: /, "", line)
|
||||||
|
event(time " " line "; " up + 0 " connected (" refused + 0 " refused, retried), " snapshots + 0 " Snapshots with a cause")
|
||||||
|
up = 0
|
||||||
|
refused = 0
|
||||||
|
snapshots = 0
|
||||||
|
next
|
||||||
|
}
|
||||||
|
|
||||||
|
/: connection [0-9]+: connected, / {
|
||||||
|
if (!opened) event(time " connections opening (counted in the minute line)")
|
||||||
|
opened = 1
|
||||||
|
up++
|
||||||
|
next
|
||||||
|
}
|
||||||
|
# Kalshi refuses some of ~100 handshakes opened at once (429); each retries.
|
||||||
|
/: connect failed: handshake refused: 429/ { refused++; next }
|
||||||
|
/: Snapshot of / { snapshots++; next }
|
||||||
|
/ changes go unchecked$| tick size changed, resync$| unreadable / { next }
|
||||||
|
|
||||||
|
# Everything else is an event worth seeing: a connection lost, recycled,
|
||||||
|
# split, moved off when quiet, a refused handshake, a failed fetch, the end.
|
||||||
|
{ event(time " " text) }
|
||||||
|
|
||||||
|
END {
|
||||||
|
flush()
|
||||||
|
if (final && last != "") print last
|
||||||
|
}
|
||||||
Executable
+101
@@ -0,0 +1,101 @@
|
|||||||
|
#!/bin/sh
|
||||||
|
# watch.sh [EXCHANGE] [MINUTES]: runs the collector and shows what it does.
|
||||||
|
# EXCHANGE is polymarket (the default, no key needed), kalshi or kalshi-demo
|
||||||
|
# (KALSHI_KEY_ID_FILE and KALSHI_PRIVATE_KEY_FILE exported); MINUTES
|
||||||
|
# defaults to 3. Ctrl-C stops the run early, cleanly.
|
||||||
|
#
|
||||||
|
# The output and the full log go to watch/; the terminal shows, each minute,
|
||||||
|
# the orchestrator's line and the book threads' summed (scripts/watch.awk),
|
||||||
|
# and the notable events between. Then the output is checked against the
|
||||||
|
# log (reconcile) and one symbol is shown as text (show). It exits non-zero
|
||||||
|
# when the collector failed or the output doesn't reconcile.
|
||||||
|
set -eu
|
||||||
|
cd "$(dirname "$0")/.."
|
||||||
|
exchange=${1:-polymarket}
|
||||||
|
minutes=${2:-3}
|
||||||
|
case $exchange in
|
||||||
|
polymarket | kalshi | kalshi-demo) ;;
|
||||||
|
*)
|
||||||
|
echo "usage: scripts/watch.sh [polymarket|kalshi|kalshi-demo] [MINUTES]" >&2
|
||||||
|
exit 2
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
echo "building (cargo build --release)"
|
||||||
|
cargo build --release --quiet
|
||||||
|
mkdir -p watch
|
||||||
|
out=watch/$exchange.sbe log=watch/$exchange.log memory=watch/$exchange.memory
|
||||||
|
rm -f "$out" "$log" "$memory"
|
||||||
|
: >"$log"
|
||||||
|
|
||||||
|
echo "load before the run: $(cut -d' ' -f1-3 /proc/loadavg) (1, 5, 15 min)"
|
||||||
|
echo "collecting $exchange for $minutes minutes into $out (log: $log); Ctrl-C stops it"
|
||||||
|
echo "the markets are listed first (a minute or two), then subscribed"
|
||||||
|
timeout -s INT "${minutes}m" ./target/release/engineering-challenge "$exchange" "$out" 2>>"$log" &
|
||||||
|
run=$!
|
||||||
|
# The view ends by itself once the run is over and the log read to its end.
|
||||||
|
# mawk reads a pipe a buffer at a time unless interactive: the view would
|
||||||
|
# show nothing until the end.
|
||||||
|
live=
|
||||||
|
if awk -W version 2>/dev/null | grep -q mawk; then live='-W interactive'; fi
|
||||||
|
# shellcheck disable=SC2086 # $live is empty or two words, both wanted.
|
||||||
|
tail -n +1 -F --pid="$run" "$log" 2>/dev/null | awk $live -f scripts/watch.awk &
|
||||||
|
view=$!
|
||||||
|
# A job started with & ignores Ctrl-C (sh without job control), so the
|
||||||
|
# script passes it on: timeout hands SIGINT to the collector, which stops
|
||||||
|
# cleanly, and the script carries on to the checks. The view, also a job,
|
||||||
|
# keeps showing the end.
|
||||||
|
trap 'echo; echo "stopping (Ctrl-C)"; kill -INT "$run" 2>/dev/null || true' INT
|
||||||
|
|
||||||
|
# The collector's peak memory (VmHWM, the kernel's own peak) and most
|
||||||
|
# threads, sampled every 10 s while it runs, from /proc: /usr/bin/time isn't
|
||||||
|
# installed everywhere. The last sample may fall in the shutdown, when the
|
||||||
|
# connections' threads are gone: the most seen is kept.
|
||||||
|
(
|
||||||
|
sleep 1
|
||||||
|
pid=$(pgrep -P "$run" || true)
|
||||||
|
most=0
|
||||||
|
while [ -n "$pid" ] && kill -0 "$pid" 2>/dev/null; do
|
||||||
|
proc=$(cat "/proc/$pid/status" 2>/dev/null || true)
|
||||||
|
peak=$(printf '%s\n' "$proc" | awk '/^VmHWM:/ {print $2}')
|
||||||
|
threads=$(printf '%s\n' "$proc" | awk '/^Threads:/ {print $2}')
|
||||||
|
if [ -n "$threads" ] && [ "$threads" -gt "$most" ]; then most=$threads; fi
|
||||||
|
if [ -n "$peak" ]; then echo "$peak kB, at most $most threads" >"$memory"; fi
|
||||||
|
sleep 10
|
||||||
|
done
|
||||||
|
) &
|
||||||
|
|
||||||
|
# wait returns early when Ctrl-C interrupts it: wait again until the
|
||||||
|
# collector has flushed and exited.
|
||||||
|
status=0
|
||||||
|
while kill -0 "$run" 2>/dev/null; do
|
||||||
|
wait "$run" && status=0 || status=$?
|
||||||
|
done
|
||||||
|
trap - INT
|
||||||
|
wait "$view" || true
|
||||||
|
# 124: stopped by timeout, at the end or on Ctrl-C (passed on by it).
|
||||||
|
failed=0
|
||||||
|
case $status in
|
||||||
|
0 | 124 | 130) ;;
|
||||||
|
*)
|
||||||
|
echo "the collector failed (exit $status); its log: $log"
|
||||||
|
failed=1
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
if [ ! -e "$out" ]; then
|
||||||
|
echo "no output was written"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo
|
||||||
|
echo "== totals at the last minute"
|
||||||
|
awk -v final=1 -f scripts/watch.awk "$log"
|
||||||
|
if [ -s "$memory" ]; then echo "peak memory $(cat "$memory")"; fi
|
||||||
|
echo
|
||||||
|
echo "== the output against the log (reconcile)"
|
||||||
|
./target/release/reconcile "$out" "$log" || failed=1
|
||||||
|
echo
|
||||||
|
echo "== one symbol, as text (show)"
|
||||||
|
# An output stopped before any book is empty: said, not a failure.
|
||||||
|
./target/release/show "$out" || true
|
||||||
|
exit "$failed"
|
||||||
@@ -0,0 +1,15 @@
|
|||||||
|
//! `show OUTPUT [SYMBOL]`: one symbol's first Snapshot, the Deltas after it
|
||||||
|
//! and the book they fold into, as text (`src/show.rs`).
|
||||||
|
|
||||||
|
fn main() -> anyhow::Result<()> {
|
||||||
|
let mut args = std::env::args().skip(1);
|
||||||
|
let path = args
|
||||||
|
.next()
|
||||||
|
.ok_or_else(|| anyhow::anyhow!("usage: show OUTPUT [SYMBOL]"))?;
|
||||||
|
let symbol = args.next();
|
||||||
|
print!(
|
||||||
|
"{}",
|
||||||
|
engineering_challenge::show::show(path, symbol.as_deref())?
|
||||||
|
);
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
+24
-9
@@ -78,15 +78,21 @@ impl Book {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/// Moves a level's quantity by `delta`, which can be negative, and returns
|
/// Moves a level's quantity by `delta`, which can be negative, and returns
|
||||||
/// the sum. A level can't hold less than nothing: a negative sum removes
|
/// the sum. A level can't hold less than nothing, nor more than an `i64`:
|
||||||
/// it, and tells the caller the book was wrong.
|
/// either sum removes it and returns `None`, which tells the caller the
|
||||||
pub fn add(&mut self, side: Side, price: i64, delta: i64) -> i64 {
|
/// book was wrong.
|
||||||
|
pub fn add(&mut self, side: Side, price: i64, delta: i64) -> Option<i64> {
|
||||||
let levels = match side {
|
let levels = match side {
|
||||||
Side::Buy => &self.bids,
|
Side::Buy => &self.bids,
|
||||||
Side::Sell => &self.asks,
|
Side::Sell => &self.asks,
|
||||||
};
|
};
|
||||||
let quantity = levels.get(&price).copied().unwrap_or(0) + delta;
|
let quantity = levels
|
||||||
self.set(side, price, quantity.max(0));
|
.get(&price)
|
||||||
|
.copied()
|
||||||
|
.unwrap_or(0)
|
||||||
|
.checked_add(delta)
|
||||||
|
.filter(|&quantity| quantity >= 0);
|
||||||
|
self.set(side, price, quantity.unwrap_or(0));
|
||||||
quantity
|
quantity
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -173,12 +179,21 @@ mod tests {
|
|||||||
#[test]
|
#[test]
|
||||||
fn adding_moves_a_level_and_never_leaves_it_below_zero() {
|
fn adding_moves_a_level_and_never_leaves_it_below_zero() {
|
||||||
let mut book = Book::default();
|
let mut book = Book::default();
|
||||||
assert_eq!(book.add(Side::Sell, 600_000, 5), 5);
|
assert_eq!(book.add(Side::Sell, 600_000, 5), Some(5));
|
||||||
assert_eq!(book.add(Side::Sell, 600_000, -2), 3);
|
assert_eq!(book.add(Side::Sell, 600_000, -2), Some(3));
|
||||||
assert_eq!(book.asks, BTreeMap::from([(600_000, 3)]));
|
assert_eq!(book.asks, BTreeMap::from([(600_000, 3)]));
|
||||||
assert_eq!(book.add(Side::Sell, 600_000, -3), 0);
|
assert_eq!(book.add(Side::Sell, 600_000, -3), Some(0));
|
||||||
assert!(book.asks.is_empty());
|
assert!(book.asks.is_empty());
|
||||||
assert_eq!(book.add(Side::Buy, 400_000, -1), -1);
|
assert_eq!(book.add(Side::Buy, 400_000, -1), None);
|
||||||
|
assert!(book.bids.is_empty());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_sum_past_the_largest_quantity_removes_the_level() {
|
||||||
|
let mut book = Book::default();
|
||||||
|
assert_eq!(book.add(Side::Buy, 400_000, i64::MAX), Some(i64::MAX));
|
||||||
|
assert_eq!(book.bids, BTreeMap::from([(400_000, i64::MAX)]));
|
||||||
|
assert_eq!(book.add(Side::Buy, 400_000, 1), None);
|
||||||
assert!(book.bids.is_empty());
|
assert!(book.bids.is_empty());
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
+10
-1
@@ -17,16 +17,25 @@ pub enum Cause {
|
|||||||
/// The connection followed changed: the new one's book, which may hold
|
/// The connection followed changed: the new one's book, which may hold
|
||||||
/// changes the old one never delivered.
|
/// changes the old one never delivered.
|
||||||
Switch,
|
Switch,
|
||||||
|
/// Its market was removed and is tracked again: its first book since,
|
||||||
|
/// against the last one written before.
|
||||||
|
TrackedAgain,
|
||||||
}
|
}
|
||||||
|
|
||||||
impl Cause {
|
impl Cause {
|
||||||
const ALL: [Cause; 3] = [Cause::Loss, Cause::SubscriptionGap, Cause::Switch];
|
const ALL: [Cause; 4] = [
|
||||||
|
Cause::Loss,
|
||||||
|
Cause::SubscriptionGap,
|
||||||
|
Cause::Switch,
|
||||||
|
Cause::TrackedAgain,
|
||||||
|
];
|
||||||
|
|
||||||
fn name(self) -> &'static str {
|
fn name(self) -> &'static str {
|
||||||
match self {
|
match self {
|
||||||
Cause::Loss => "loss",
|
Cause::Loss => "loss",
|
||||||
Cause::SubscriptionGap => "subscription gap",
|
Cause::SubscriptionGap => "subscription gap",
|
||||||
Cause::Switch => "switch",
|
Cause::Switch => "switch",
|
||||||
|
Cause::TrackedAgain => "tracked again",
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
+29
-9
@@ -6,13 +6,14 @@
|
|||||||
//! subscription gone quiet on one connection (roadmap/05). What an event does
|
//! subscription gone quiet on one connection (roadmap/05). What an event does
|
||||||
//! to a book, how a loss is found and when to resync is the rule's.
|
//! to a book, how a loss is found and when to resync is the rule's.
|
||||||
|
|
||||||
use std::collections::{BTreeMap, BTreeSet, HashMap};
|
use std::collections::{BTreeMap, BTreeSet, HashMap, HashSet};
|
||||||
use std::time::Duration;
|
use std::time::Duration;
|
||||||
|
|
||||||
use chrono::{DateTime, DurationRound, TimeDelta, Utc};
|
use chrono::{DateTime, DurationRound, TimeDelta, Utc};
|
||||||
use crossbeam::channel::{Receiver, Sender, never, select};
|
use crossbeam::channel::{Receiver, Sender, never, select};
|
||||||
use engineering_challenge::Side;
|
use engineering_challenge::Side;
|
||||||
use engineering_challenge::book::{Book, Output};
|
use engineering_challenge::book::{Book, Output};
|
||||||
|
use engineering_challenge::cause::{self, Cause};
|
||||||
use engineering_challenge::util::dtos::challenge_orderbook::PriceLevelUpdateDomain;
|
use engineering_challenge::util::dtos::challenge_orderbook::PriceLevelUpdateDomain;
|
||||||
|
|
||||||
use super::socket::Command;
|
use super::socket::Command;
|
||||||
@@ -24,9 +25,9 @@ pub(crate) const MIN_MEASURED: TimeDelta = TimeDelta::seconds(30);
|
|||||||
/// How often a book thread whose rule asks for it wakes for the rule's timer.
|
/// How often a book thread whose rule asks for it wakes for the rule's timer.
|
||||||
const WAKE_EVERY: Duration = Duration::from_millis(10);
|
const WAKE_EVERY: Duration = Duration::from_millis(10);
|
||||||
/// A connection that delivered none of a symbol's changes in a minute while
|
/// A connection that delivered none of a symbol's changes in a minute while
|
||||||
/// another delivered at least this many is quiet for it (sprint 05: crossed
|
/// another delivered at least this many is quiet for it (crossed once in 21
|
||||||
/// once in 21 live minutes; at 1, copies landing in the next minute would
|
/// live minutes, docs/notes.md; at 1, copies landing in the next minute
|
||||||
/// count).
|
/// would count).
|
||||||
const QUIET_AGAINST: u32 = 3;
|
const QUIET_AGAINST: u32 = 3;
|
||||||
|
|
||||||
/// What the book thread does with an event or a command: write output, ask a
|
/// What the book thread does with an event or a command: write output, ask a
|
||||||
@@ -222,6 +223,10 @@ pub struct Symbol<S> {
|
|||||||
/// The receive time of its last Delta or Snapshot: a later Delta is
|
/// The receive time of its last Delta or Snapshot: a later Delta is
|
||||||
/// stamped no earlier, so replaying the output by time keeps its order.
|
/// stamped no earlier, so replaying the output by time keeps its order.
|
||||||
pub last_written: DateTime<Utc>,
|
pub last_written: DateTime<Utc>,
|
||||||
|
/// The exchange's name while its market, removed before, is tracked
|
||||||
|
/// again and no Snapshot was written since: the first says so (its
|
||||||
|
/// book can't be told from the one written before it left).
|
||||||
|
pub again: Option<&'static str>,
|
||||||
pub rule: S,
|
pub rule: S,
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -238,6 +243,7 @@ impl<S> Symbol<S> {
|
|||||||
.collect(),
|
.collect(),
|
||||||
quiet_on: None,
|
quiet_on: None,
|
||||||
last_written: DateTime::UNIX_EPOCH,
|
last_written: DateTime::UNIX_EPOCH,
|
||||||
|
again: None,
|
||||||
rule,
|
rule,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -281,6 +287,12 @@ impl<S> Symbol<S> {
|
|||||||
/// The Snapshot of its book, received at `received_at`.
|
/// The Snapshot of its book, received at `received_at`.
|
||||||
pub fn snapshot(&mut self, name: &str, received_at: DateTime<Utc>) -> Output {
|
pub fn snapshot(&mut self, name: &str, received_at: DateTime<Utc>) -> Output {
|
||||||
self.last_written = self.last_written.max(received_at);
|
self.last_written = self.last_written.max(received_at);
|
||||||
|
if let Some(exchange) = self.again.take() {
|
||||||
|
log!(
|
||||||
|
"{exchange}: {}",
|
||||||
|
cause::line(name, received_at, Cause::TrackedAgain, "")
|
||||||
|
);
|
||||||
|
}
|
||||||
self.book
|
self.book
|
||||||
.snapshot(name, received_at, self.exchange_timestamp)
|
.snapshot(name, received_at, self.exchange_timestamp)
|
||||||
}
|
}
|
||||||
@@ -378,6 +390,11 @@ pub struct Shell<S> {
|
|||||||
pub peak_waiting: usize,
|
pub peak_waiting: usize,
|
||||||
pub quiet: u64,
|
pub quiet: u64,
|
||||||
pub quiet_reported: u64,
|
pub quiet_reported: u64,
|
||||||
|
/// The symbols of the markets removed: one tracked again says so in its
|
||||||
|
/// first Snapshot. Ceiling: it grows with every market closed in the
|
||||||
|
/// run (tens of bytes a symbol, a few thousand a day); forgetting them
|
||||||
|
/// after a day would bound it.
|
||||||
|
pub removed: HashSet<String>,
|
||||||
}
|
}
|
||||||
|
|
||||||
impl<S> Shell<S> {
|
impl<S> Shell<S> {
|
||||||
@@ -429,6 +446,7 @@ impl<R: Rule> Feed<R> {
|
|||||||
peak_waiting: 0,
|
peak_waiting: 0,
|
||||||
quiet: 0,
|
quiet: 0,
|
||||||
quiet_reported: 0,
|
quiet_reported: 0,
|
||||||
|
removed: HashSet::new(),
|
||||||
},
|
},
|
||||||
rule,
|
rule,
|
||||||
}
|
}
|
||||||
@@ -446,10 +464,11 @@ impl<R: Rule> Feed<R> {
|
|||||||
let names = market.symbols().to_vec();
|
let names = market.symbols().to_vec();
|
||||||
for name in &names {
|
for name in &names {
|
||||||
let rule = self.rule.symbol(market);
|
let rule = self.rule.symbol(market);
|
||||||
self.shell.symbols.insert(
|
let mut symbol = Symbol::new(market.id(), connections, now, rule);
|
||||||
name.clone(),
|
if self.shell.removed.remove(name) {
|
||||||
Symbol::new(market.id(), connections, now, rule),
|
symbol.again = Some(self.shell.exchange);
|
||||||
);
|
}
|
||||||
|
self.shell.symbols.insert(name.clone(), symbol);
|
||||||
}
|
}
|
||||||
let tracked = Tracked {
|
let tracked = Tracked {
|
||||||
names: names.clone(),
|
names: names.clone(),
|
||||||
@@ -535,6 +554,7 @@ impl<R: Rule> Feed<R> {
|
|||||||
for name in market.symbols() {
|
for name in market.symbols() {
|
||||||
if let Some(symbol) = self.shell.symbols.remove(name) {
|
if let Some(symbol) = self.shell.symbols.remove(name) {
|
||||||
connections = symbol.connections;
|
connections = symbol.connections;
|
||||||
|
self.shell.removed.insert(name.clone());
|
||||||
}
|
}
|
||||||
self.rule.on_removed(name);
|
self.rule.on_removed(name);
|
||||||
}
|
}
|
||||||
@@ -700,7 +720,7 @@ impl<R: Rule> Feed<R> {
|
|||||||
/// A book thread: runs until every connection and the orchestrator have
|
/// A book thread: runs until every connection and the orchestrator have
|
||||||
/// dropped its channels, then drops its senders so the writer thread finishes
|
/// dropped its channels, then drops its senders so the writer thread finishes
|
||||||
/// once every book thread has.
|
/// once every book thread has.
|
||||||
// ponytail: unbounded channels never block a socket read nor drop an event;
|
// Ceiling: unbounded channels never block a socket read nor drop an event;
|
||||||
// the ceiling is memory if a consumer stays behind. The minute log's peak of
|
// the ceiling is memory if a consumer stays behind. The minute log's peak of
|
||||||
// events waiting makes that visible; bounded channels are a constructor change
|
// events waiting makes that visible; bounded channels are a constructor change
|
||||||
// if it ever matters.
|
// if it ever matters.
|
||||||
|
|||||||
+1
-1
@@ -18,7 +18,7 @@ use crossbeam::channel::{Receiver, Sender};
|
|||||||
use feed::Rule;
|
use feed::Rule;
|
||||||
|
|
||||||
/// How many book threads own the books. One handled ~30,000 frames a second
|
/// How many book threads own the books. One handled ~30,000 frames a second
|
||||||
/// in sprint 02.
|
/// when measured (roadmap/04).
|
||||||
pub const BOOK_THREADS: usize = 16;
|
pub const BOOK_THREADS: usize = 16;
|
||||||
|
|
||||||
/// The book thread that owns a market's books, from a hash of its id.
|
/// The book thread that owns a market's books, from a hash of its id.
|
||||||
|
|||||||
+395
-63
@@ -44,6 +44,8 @@ const MOVED_STAYS: Duration = Duration::from_secs(10 * 60);
|
|||||||
const SPLIT_WITHIN: Duration = Duration::from_secs(10 * 60);
|
const SPLIT_WITHIN: Duration = Duration::from_secs(10 * 60);
|
||||||
const RECENT_EVERY: Duration = Duration::from_secs(60);
|
const RECENT_EVERY: Duration = Duration::from_secs(60);
|
||||||
const RESCAN_EVERY: Duration = Duration::from_secs(60 * 60);
|
const RESCAN_EVERY: Duration = Duration::from_secs(60 * 60);
|
||||||
|
/// How often the first scan says how far it got.
|
||||||
|
const SCAN_NOTE_EVERY: Duration = Duration::from_secs(10);
|
||||||
/// The listing's indexing delay: each recent check reaches this far before
|
/// The listing's indexing delay: each recent check reaches this far before
|
||||||
/// the last successful one. New markets were listed up to 5 minutes after
|
/// the last successful one. New markets were listed up to 5 minutes after
|
||||||
/// their creation (docs/notes.md); known markets are skipped, so the margin
|
/// their creation (docs/notes.md); known markets are skipped, so the margin
|
||||||
@@ -54,9 +56,10 @@ const RECENT_MARGIN: TimeDelta = TimeDelta::minutes(15);
|
|||||||
const RECYCLE_EVERY: Duration = Duration::from_secs(60 * 60);
|
const RECYCLE_EVERY: Duration = Duration::from_secs(60 * 60);
|
||||||
/// A connection slow (among the slowest quarter by mean lateness, and more
|
/// A connection slow (among the slowest quarter by mean lateness, and more
|
||||||
/// than [`SLOW_FACTOR`] times the mean of every copy judged) this many
|
/// than [`SLOW_FACTOR`] times the mean of every copy judged) this many
|
||||||
/// minutes in a row is recycled ahead of its turn: the next one is drawn
|
/// minutes in a row takes the next recycle's turn, in place of the oldest:
|
||||||
/// afresh. Kalshi's demo kept the 2 fastest of 16 connections 28% and 42%
|
/// its replacement's speed is drawn afresh. Kalshi's demo kept the 2 fastest
|
||||||
/// lower at p99 (LATENCY.md); only Kalshi's rule reports lateness. The mean,
|
/// of 16 connections 28% and 42% lower at p99 (LATENCY.md); only Kalshi's
|
||||||
|
/// rule reports lateness. The mean,
|
||||||
/// not the median: with two copies of each change, a connection first more
|
/// not the median: with two copies of each change, a connection first more
|
||||||
/// than half the time has a median of 0 (roadmap/07).
|
/// than half the time has a median of 0 (roadmap/07).
|
||||||
const SLOW_MINUTES: u32 = 3;
|
const SLOW_MINUTES: u32 = 3;
|
||||||
@@ -123,6 +126,9 @@ pub fn run<E: Exchange>(
|
|||||||
let mut fetcher_died = false;
|
let mut fetcher_died = false;
|
||||||
let mut next_rescan = Instant::now();
|
let mut next_rescan = Instant::now();
|
||||||
let mut next_recent = Instant::now() + RECENT_EVERY;
|
let mut next_recent = Instant::now() + RECENT_EVERY;
|
||||||
|
// The first scan pages the whole listing (a minute or two): said as it
|
||||||
|
// goes, so a run that has written nothing yet doesn't look hung.
|
||||||
|
let mut next_scan_note = Instant::now() + SCAN_NOTE_EVERY;
|
||||||
|
|
||||||
while !stop.load(Ordering::Relaxed) {
|
while !stop.load(Ordering::Relaxed) {
|
||||||
// Only a panic ends a fetcher while this runs, even one in the middle
|
// Only a panic ends a fetcher while this runs, even one in the middle
|
||||||
@@ -136,6 +142,16 @@ pub fn run<E: Exchange>(
|
|||||||
}
|
}
|
||||||
let now = Instant::now();
|
let now = Instant::now();
|
||||||
let mut commands = Vec::new();
|
let mut commands = Vec::new();
|
||||||
|
if let Some(waiting) = &first_scan
|
||||||
|
&& scanning
|
||||||
|
&& now >= next_scan_note
|
||||||
|
{
|
||||||
|
log!(
|
||||||
|
"{name}: orchestrator: first scan, {} live markets listed so far",
|
||||||
|
waiting.len()
|
||||||
|
);
|
||||||
|
next_scan_note = now + SCAN_NOTE_EVERY;
|
||||||
|
}
|
||||||
if now >= next_rescan {
|
if now >= next_rescan {
|
||||||
if !scanning {
|
if !scanning {
|
||||||
_ = rescan.send(Request::Rescan);
|
_ = rescan.send(Request::Rescan);
|
||||||
@@ -155,15 +171,11 @@ pub fn run<E: Exchange>(
|
|||||||
let moves = tracker.rebalance(now);
|
let moves = tracker.rebalance(now);
|
||||||
moved += moves.len();
|
moved += moves.len();
|
||||||
commands.extend(moves);
|
commands.extend(moves);
|
||||||
if let Some((connection, mean)) = tracker.judge_speeds(now) {
|
if let Some(marked) = tracker.judge_speeds(now) {
|
||||||
log!(
|
log!(
|
||||||
"{name}: orchestrator: connection {connection} among the slowest for \
|
"{name}: orchestrator: {}, next to recycle",
|
||||||
{SLOW_MINUTES} minutes (mean lateness {:.1} ms), recycled",
|
slow_line(marked)
|
||||||
mean as f64 / 1_000.0
|
|
||||||
);
|
);
|
||||||
threads.recycle(connection);
|
|
||||||
recycled += 1;
|
|
||||||
slow_recycled += 1;
|
|
||||||
}
|
}
|
||||||
let busiest = tracker.connections.iter().map(|one| one.load).max();
|
let busiest = tracker.connections.iter().map(|one| one.load).max();
|
||||||
log!(
|
log!(
|
||||||
@@ -181,7 +193,14 @@ pub fn run<E: Exchange>(
|
|||||||
if first_scan.is_none() && queue.is_empty() {
|
if first_scan.is_none() && queue.is_empty() {
|
||||||
tracker.start_recycling(now);
|
tracker.start_recycling(now);
|
||||||
}
|
}
|
||||||
if let Some(connection) = tracker.recycle(now) {
|
if let Some((connection, slow)) = tracker.recycle(now) {
|
||||||
|
if let Some(mean) = slow {
|
||||||
|
log!(
|
||||||
|
"{name}: orchestrator: {}, recycled",
|
||||||
|
slow_line((connection, mean))
|
||||||
|
);
|
||||||
|
slow_recycled += 1;
|
||||||
|
}
|
||||||
threads.recycle(connection);
|
threads.recycle(connection);
|
||||||
recycled += 1;
|
recycled += 1;
|
||||||
}
|
}
|
||||||
@@ -203,6 +222,10 @@ pub fn run<E: Exchange>(
|
|||||||
recv(findings) -> finding => match finding {
|
recv(findings) -> finding => match finding {
|
||||||
Ok(Finding::ScanStarted) => {
|
Ok(Finding::ScanStarted) => {
|
||||||
tracker.scan_started();
|
tracker.scan_started();
|
||||||
|
if first_scan.is_some() {
|
||||||
|
log!("{name}: orchestrator: first scan, listing the exchange's markets");
|
||||||
|
next_scan_note = Instant::now() + SCAN_NOTE_EVERY;
|
||||||
|
}
|
||||||
// A first scan that failed starts over.
|
// A first scan that failed starts over.
|
||||||
if let Some(waiting) = &mut first_scan {
|
if let Some(waiting) = &mut first_scan {
|
||||||
waiting.clear();
|
waiting.clear();
|
||||||
@@ -224,6 +247,10 @@ pub fn run<E: Exchange>(
|
|||||||
scanned = true;
|
scanned = true;
|
||||||
match first_scan.take() {
|
match first_scan.take() {
|
||||||
Some(markets) => {
|
Some(markets) => {
|
||||||
|
log!(
|
||||||
|
"{name}: orchestrator: first scan done, {} live markets, subscribing",
|
||||||
|
markets.len()
|
||||||
|
);
|
||||||
let adds = tracker.deal(markets);
|
let adds = tracker.deal(markets);
|
||||||
added += adds.len();
|
added += adds.len();
|
||||||
commands.extend(adds);
|
commands.extend(adds);
|
||||||
@@ -466,6 +493,21 @@ struct Tracker<M> {
|
|||||||
recycling: Option<usize>,
|
recycling: Option<usize>,
|
||||||
/// The book threads' lateness samples since the last judgment.
|
/// The book threads' lateness samples since the last judgment.
|
||||||
lateness: Lateness,
|
lateness: Lateness,
|
||||||
|
/// The connection judged slow, with its mean lateness (µs): it takes the
|
||||||
|
/// next recycle's turn in place of the oldest.
|
||||||
|
slow: Option<(usize, i64)>,
|
||||||
|
/// The connections whose mark was logged, while they stay slow: two that
|
||||||
|
/// swap ranks are each said once.
|
||||||
|
said: HashSet<usize>,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A slow connection, as the log says it: its number and mean lateness (µs).
|
||||||
|
fn slow_line((connection, mean): (usize, i64)) -> String {
|
||||||
|
format!(
|
||||||
|
"connection {connection} among the slowest for {SLOW_MINUTES} minutes (mean lateness \
|
||||||
|
{:.1} ms)",
|
||||||
|
mean as f64 / 1_000.0
|
||||||
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
impl<M: Market> Tracker<M> {
|
impl<M: Market> Tracker<M> {
|
||||||
@@ -479,6 +521,8 @@ impl<M: Market> Tracker<M> {
|
|||||||
next_recycle: None,
|
next_recycle: None,
|
||||||
recycling: None,
|
recycling: None,
|
||||||
lateness: Lateness::new(),
|
lateness: Lateness::new(),
|
||||||
|
slow: None,
|
||||||
|
said: HashSet::new(),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -602,7 +646,7 @@ impl<M: Market> Tracker<M> {
|
|||||||
.collect();
|
.collect();
|
||||||
over.sort_by_key(|&number| Reverse(self.connections[number].load));
|
over.sort_by_key(|&number| Reverse(self.connections[number].load));
|
||||||
for from in over {
|
for from in over {
|
||||||
// ponytail: a scan of every market per connection over budget,
|
// Ceiling: a scan of every market per connection over budget,
|
||||||
// fine at a few hundred thousand once a minute; per-connection
|
// fine at a few hundred thousand once a minute; per-connection
|
||||||
// market sets if not.
|
// market sets if not.
|
||||||
let mut busiest: Vec<(String, u64)> = self
|
let mut busiest: Vec<(String, u64)> = self
|
||||||
@@ -715,6 +759,11 @@ impl<M: Market> Tracker<M> {
|
|||||||
if self.recycling == Some(connection) {
|
if self.recycling == Some(connection) {
|
||||||
self.recycling = None;
|
self.recycling = None;
|
||||||
}
|
}
|
||||||
|
// A fresh connection: what was judged slow is gone with the old one.
|
||||||
|
if self.slow.is_some_and(|(slow, _)| slow == connection) {
|
||||||
|
self.slow = None;
|
||||||
|
}
|
||||||
|
self.said.remove(&connection);
|
||||||
let settled = now + self.recycle_interval();
|
let settled = now + self.recycle_interval();
|
||||||
if let Some(next) = &mut self.next_recycle {
|
if let Some(next) = &mut self.next_recycle {
|
||||||
*next = (*next).max(settled);
|
*next = (*next).max(settled);
|
||||||
@@ -738,11 +787,12 @@ impl<M: Market> Tracker<M> {
|
|||||||
RECYCLE_EVERY / self.connections.len().max(1) as u32
|
RECYCLE_EVERY / self.connections.len().max(1) as u32
|
||||||
}
|
}
|
||||||
|
|
||||||
/// The connection to recycle now, if one is due: the one up the longest,
|
/// The connection to recycle now, if one is due: the one judged slow if
|
||||||
/// once the last recycled is up again, and only while every connection is
|
/// there is one (with its mean lateness), else the one up the longest;
|
||||||
/// up and has been for an interval, so no market loses both of its
|
/// once the last recycled is up again, and only while every connection
|
||||||
|
/// is up and has been for an interval, so no market loses both of its
|
||||||
/// connections at once.
|
/// connections at once.
|
||||||
fn recycle(&mut self, now: Instant) -> Option<usize> {
|
fn recycle(&mut self, now: Instant) -> Option<(usize, Option<i64>)> {
|
||||||
let due = self.next_recycle.is_some_and(|next| now >= next);
|
let due = self.next_recycle.is_some_and(|next| now >= next);
|
||||||
if !due
|
if !due
|
||||||
|| self.recycling.is_some()
|
|| self.recycling.is_some()
|
||||||
@@ -750,11 +800,17 @@ impl<M: Market> Tracker<M> {
|
|||||||
{
|
{
|
||||||
return None;
|
return None;
|
||||||
}
|
}
|
||||||
let oldest = (0..self.connections.len())
|
let (connection, slow) = match self.slow.take() {
|
||||||
.min_by_key(|&number| self.connections[number].connected_at)?;
|
Some((slow, mean)) => (slow, Some(mean)),
|
||||||
self.recycling = Some(oldest);
|
None => (
|
||||||
|
(0..self.connections.len())
|
||||||
|
.min_by_key(|&number| self.connections[number].connected_at)?,
|
||||||
|
None,
|
||||||
|
),
|
||||||
|
};
|
||||||
|
self.recycling = Some(connection);
|
||||||
self.next_recycle = Some(now + self.recycle_interval());
|
self.next_recycle = Some(now + self.recycle_interval());
|
||||||
Some(oldest)
|
Some((connection, slow))
|
||||||
}
|
}
|
||||||
|
|
||||||
/// A book thread's lateness samples, by connection.
|
/// A book thread's lateness samples, by connection.
|
||||||
@@ -768,10 +824,10 @@ impl<M: Market> Tracker<M> {
|
|||||||
/// since the last time; those of the slowest 1 in [`SLOW_SHARE`] more
|
/// since the last time; those of the slowest 1 in [`SLOW_SHARE`] more
|
||||||
/// than [`SLOW_FACTOR`] times the mean of every judged copy count a slow
|
/// than [`SLOW_FACTOR`] times the mean of every judged copy count a slow
|
||||||
/// minute, the others start over. The slowest of those slow for
|
/// minute, the others start over. The slowest of those slow for
|
||||||
/// [`SLOW_MINUTES`] is recycled now, with its mean (µs), under
|
/// [`SLOW_MINUTES`] is marked, with its mean (µs), to take the next
|
||||||
/// [`Tracker::recycle`]'s
|
/// recycle's turn ([`Tracker::recycle`]): one path, one at a time.
|
||||||
/// rule: one at a time, every connection up, none up for less than an
|
/// Returns the mark when it is new: a connection marked again while it
|
||||||
/// interval.
|
/// stays slow, waiting for its turn, is said once.
|
||||||
fn judge_speeds(&mut self, now: Instant) -> Option<(usize, i64)> {
|
fn judge_speeds(&mut self, now: Instant) -> Option<(usize, i64)> {
|
||||||
let samples = std::mem::take(&mut self.lateness);
|
let samples = std::mem::take(&mut self.lateness);
|
||||||
let speeds: Vec<(usize, i64, i64)> = samples
|
let speeds: Vec<(usize, i64, i64)> = samples
|
||||||
@@ -814,22 +870,15 @@ impl<M: Market> Tracker<M> {
|
|||||||
0
|
0
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
// As for the hourly recycle: one at a time, every connection up, and
|
let connections = &self.connections;
|
||||||
// the last to come up past its opening books.
|
self.said
|
||||||
let settled = self
|
.retain(|&connection| connections[connection].slow_minutes > 0);
|
||||||
.connections
|
self.slow = slow
|
||||||
.iter()
|
.iter()
|
||||||
.filter_map(|state| state.connected_at)
|
.copied()
|
||||||
.max()
|
.find(|&(connection, _)| self.connections[connection].slow_minutes >= SLOW_MINUTES);
|
||||||
.is_none_or(|last| now.duration_since(last) >= self.recycle_interval());
|
self.slow
|
||||||
if self.recycling.is_some() || self.connections.iter().any(|state| !state.up) || !settled {
|
.filter(|&(connection, _)| self.said.insert(connection))
|
||||||
return None;
|
|
||||||
}
|
|
||||||
let &(connection, speed) = slow
|
|
||||||
.iter()
|
|
||||||
.find(|&&(connection, _)| self.connections[connection].slow_minutes >= SLOW_MINUTES)?;
|
|
||||||
self.recycling = Some(connection);
|
|
||||||
Some((connection, speed))
|
|
||||||
}
|
}
|
||||||
|
|
||||||
fn scan_started(&mut self) {
|
fn scan_started(&mut self) {
|
||||||
@@ -1189,7 +1238,7 @@ mod tests {
|
|||||||
tracker.start_recycling(at(10));
|
tracker.start_recycling(at(10));
|
||||||
// One every hour / 3 connections.
|
// One every hour / 3 connections.
|
||||||
assert_eq!(tracker.recycle(at(1_209)), None);
|
assert_eq!(tracker.recycle(at(1_209)), None);
|
||||||
assert_eq!(tracker.recycle(at(1_210)), Some(2));
|
assert_eq!(tracker.recycle(at(1_210)), Some((2, None)));
|
||||||
// The next waits for the recycled one, whose Down may not even have
|
// The next waits for the recycled one, whose Down may not even have
|
||||||
// come yet, to be up again...
|
// come yet, to be up again...
|
||||||
assert_eq!(tracker.recycle(at(2_410)), None);
|
assert_eq!(tracker.recycle(at(2_410)), None);
|
||||||
@@ -1198,7 +1247,7 @@ mod tests {
|
|||||||
tracker.up(2, at(2_411));
|
tracker.up(2, at(2_411));
|
||||||
// ...and for its opening books: an interval more.
|
// ...and for its opening books: an interval more.
|
||||||
assert_eq!(tracker.recycle(at(3_610)), None);
|
assert_eq!(tracker.recycle(at(3_610)), None);
|
||||||
assert_eq!(tracker.recycle(at(3_611)), Some(0));
|
assert_eq!(tracker.recycle(at(3_611)), Some((0, None)));
|
||||||
tracker.down(0);
|
tracker.down(0);
|
||||||
tracker.up(0, at(3_612));
|
tracker.up(0, at(3_612));
|
||||||
// A connection down for its own reasons holds every recycle, and its
|
// A connection down for its own reasons holds every recycle, and its
|
||||||
@@ -1207,7 +1256,7 @@ mod tests {
|
|||||||
assert_eq!(tracker.recycle(at(6_000)), None);
|
assert_eq!(tracker.recycle(at(6_000)), None);
|
||||||
tracker.up(2, at(6_001));
|
tracker.up(2, at(6_001));
|
||||||
assert_eq!(tracker.recycle(at(7_200)), None);
|
assert_eq!(tracker.recycle(at(7_200)), None);
|
||||||
assert_eq!(tracker.recycle(at(7_201)), Some(1));
|
assert_eq!(tracker.recycle(at(7_201)), Some((1, None)));
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Twelve connections, up from `start`.
|
/// Twelve connections, up from `start`.
|
||||||
@@ -1236,36 +1285,119 @@ mod tests {
|
|||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn a_connection_among_the_slowest_three_minutes_running_is_recycled() {
|
fn a_connection_among_the_slowest_three_minutes_running_takes_the_next_recycle() {
|
||||||
let start = Instant::now();
|
let start = Instant::now();
|
||||||
let at = |seconds: u64| start + Duration::from_secs(seconds);
|
let at = |seconds: u64| start + Duration::from_secs(seconds);
|
||||||
let mut tracker = twelve_up(start);
|
let mut tracker = twelve_up(start);
|
||||||
// The slowest quarter is three of twelve. 3 and 7 are slow every
|
// The hourly recycle runs: one every 300 s (an hour / 12), the
|
||||||
// minute, 5 for two minutes and then not.
|
// first due at 600.
|
||||||
|
tracker.start_recycling(at(300));
|
||||||
|
// The slowest quarter is three of twelve, then two of the eleven
|
||||||
|
// judged once 0 is fresh. 3 and 7 are slow every minute, 5 only in
|
||||||
|
// the first.
|
||||||
minute(&mut tracker, &[(3, 90), (5, 50), (7, 40)], 100);
|
minute(&mut tracker, &[(3, 90), (5, 50), (7, 40)], 100);
|
||||||
assert_eq!(tracker.judge_speeds(at(601)), None);
|
assert_eq!(tracker.judge_speeds(at(601)), None);
|
||||||
minute(&mut tracker, &[(3, 90), (5, 50), (7, 40)], 100);
|
// The hourly turn takes the oldest (all equal: the first).
|
||||||
|
assert_eq!(tracker.recycle(at(601)), Some((0, None)));
|
||||||
|
tracker.down(0);
|
||||||
|
tracker.up(0, at(603));
|
||||||
|
minute(&mut tracker, &[(3, 90), (7, 40), (5, 35)], 100);
|
||||||
assert_eq!(tracker.judge_speeds(at(661)), None);
|
assert_eq!(tracker.judge_speeds(at(661)), None);
|
||||||
// 3 and 7 both reach three minutes: one recycle a minute, the slowest.
|
// 3 and 7 both reach three minutes: the slowest is marked.
|
||||||
minute(&mut tracker, &[(3, 90), (7, 40), (9, 30)], 100);
|
minute(&mut tracker, &[(3, 90), (7, 40), (9, 30)], 100);
|
||||||
assert_eq!(tracker.judge_speeds(at(721)), Some((3, 90_000)));
|
assert_eq!(tracker.judge_speeds(at(721)), Some((3, 90_000)));
|
||||||
assert_eq!(tracker.connections[5].slow_minutes, 0);
|
assert_eq!(tracker.connections[5].slow_minutes, 0);
|
||||||
// While 3 recycles, 7 stays slow but isn't recycled.
|
// Not before the turn (0 came up at 603: due at 903)...
|
||||||
minute(&mut tracker, &[(3, 90), (7, 40), (9, 30)], 100);
|
assert_eq!(tracker.recycle(at(902)), None);
|
||||||
assert_eq!(tracker.judge_speeds(at(781)), None);
|
// ...then the turn is 3's, not the oldest's.
|
||||||
assert_eq!(tracker.connections[7].slow_minutes, 4);
|
assert_eq!(tracker.recycle(at(903)), Some((3, Some(90_000))));
|
||||||
|
// While 3 recycles, nothing else is.
|
||||||
|
assert_eq!(tracker.recycle(at(1_300)), None);
|
||||||
tracker.down(3);
|
tracker.down(3);
|
||||||
tracker.up(3, at(790));
|
tracker.up(3, at(1_301));
|
||||||
// 3 is fresh: slow, but not judged. 7 waits, as the hourly recycle
|
// 3 is fresh: not judged, its slow minutes start over; 7, still
|
||||||
// would, for 3's opening books: an interval (an hour / 12) after it
|
// slow, takes the next turn, 300 s after 3 came up.
|
||||||
// came up.
|
|
||||||
minute(&mut tracker, &[(3, 90), (7, 40), (9, 30)], 100);
|
minute(&mut tracker, &[(3, 90), (7, 40), (9, 30)], 100);
|
||||||
assert_eq!(tracker.judge_speeds(at(841)), None);
|
assert_eq!(tracker.judge_speeds(at(1_302)), Some((7, 40_000)));
|
||||||
assert_eq!(tracker.connections[3].slow_minutes, 0);
|
assert_eq!(tracker.connections[3].slow_minutes, 0);
|
||||||
minute(&mut tracker, &[(3, 90), (7, 40), (9, 30)], 100);
|
assert_eq!(tracker.recycle(at(1_600)), None);
|
||||||
assert_eq!(tracker.judge_speeds(at(1_089)), None);
|
assert_eq!(tracker.recycle(at(1_601)), Some((7, Some(40_000))));
|
||||||
minute(&mut tracker, &[(3, 90), (7, 40), (9, 30)], 100);
|
// Then the hourly order again.
|
||||||
assert_eq!(tracker.judge_speeds(at(1_090)), Some((7, 40_000)));
|
tracker.down(7);
|
||||||
|
tracker.up(7, at(1_602));
|
||||||
|
assert_eq!(tracker.recycle(at(1_902)), Some((1, None)));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_mark_is_said_once_and_cleared_once_the_connection_is_no_longer_slow() {
|
||||||
|
let start = Instant::now();
|
||||||
|
let at = |seconds: u64| start + Duration::from_secs(seconds);
|
||||||
|
let mut tracker = twelve_up(start);
|
||||||
|
for seconds in [601, 661] {
|
||||||
|
minute(&mut tracker, &[(3, 90)], 100);
|
||||||
|
assert_eq!(tracker.judge_speeds(at(seconds)), None);
|
||||||
|
}
|
||||||
|
minute(&mut tracker, &[(3, 90)], 100);
|
||||||
|
assert_eq!(tracker.judge_speeds(at(721)), Some((3, 90_000)));
|
||||||
|
// Still slow, its turn not come: marked still, but not said again.
|
||||||
|
minute(&mut tracker, &[(3, 90)], 100);
|
||||||
|
assert_eq!(tracker.judge_speeds(at(781)), None);
|
||||||
|
assert_eq!(tracker.slow, Some((3, 90_000)));
|
||||||
|
// As fast as the others before its turn: the mark goes, and the
|
||||||
|
// turn is the oldest's.
|
||||||
|
minute(&mut tracker, &[], 100);
|
||||||
|
assert_eq!(tracker.judge_speeds(at(841)), None);
|
||||||
|
assert_eq!(tracker.slow, None);
|
||||||
|
tracker.start_recycling(at(841));
|
||||||
|
assert_eq!(tracker.recycle(at(1_141)), Some((0, None)));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn two_marked_connections_swapping_ranks_are_each_said_once() {
|
||||||
|
let start = Instant::now();
|
||||||
|
let at = |seconds: u64| start + Duration::from_secs(seconds);
|
||||||
|
let mut tracker = twelve_up(start);
|
||||||
|
for seconds in [601, 661] {
|
||||||
|
minute(&mut tracker, &[(3, 90), (7, 40)], 100);
|
||||||
|
assert_eq!(tracker.judge_speeds(at(seconds)), None);
|
||||||
|
}
|
||||||
|
minute(&mut tracker, &[(3, 90), (7, 40)], 100);
|
||||||
|
assert_eq!(tracker.judge_speeds(at(721)), Some((3, 90_000)));
|
||||||
|
// 7 overtakes 3: the mark moves to 7, said once.
|
||||||
|
minute(&mut tracker, &[(3, 40), (7, 90)], 100);
|
||||||
|
assert_eq!(tracker.judge_speeds(at(781)), Some((7, 90_000)));
|
||||||
|
// Back to 3, still slow since it was said: not again.
|
||||||
|
minute(&mut tracker, &[(3, 90), (7, 40)], 100);
|
||||||
|
assert_eq!(tracker.judge_speeds(at(841)), None);
|
||||||
|
assert_eq!(tracker.slow, Some((3, 90_000)));
|
||||||
|
// 3 fast for a minute, then slow 3 minutes again: said anew.
|
||||||
|
minute(&mut tracker, &[(7, 40)], 100);
|
||||||
|
tracker.judge_speeds(at(901));
|
||||||
|
for seconds in [961, 1_021] {
|
||||||
|
minute(&mut tracker, &[(3, 90), (7, 40)], 100);
|
||||||
|
tracker.judge_speeds(at(seconds));
|
||||||
|
}
|
||||||
|
minute(&mut tracker, &[(3, 90), (7, 40)], 100);
|
||||||
|
assert_eq!(tracker.judge_speeds(at(1_081)), Some((3, 90_000)));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_slow_mark_is_dropped_when_its_connection_comes_up_fresh() {
|
||||||
|
let start = Instant::now();
|
||||||
|
let at = |seconds: u64| start + Duration::from_secs(seconds);
|
||||||
|
let mut tracker = twelve_up(start);
|
||||||
|
for seconds in [601, 661, 721] {
|
||||||
|
minute(&mut tracker, &[(3, 90), (7, 40)], 100);
|
||||||
|
tracker.judge_speeds(at(seconds));
|
||||||
|
}
|
||||||
|
assert_eq!(tracker.slow, Some((3, 90_000)));
|
||||||
|
// 3 reconnects for its own reasons before its turn: renewed, so
|
||||||
|
// the turn goes back to the oldest.
|
||||||
|
tracker.down(3);
|
||||||
|
tracker.up(3, at(730));
|
||||||
|
assert_eq!(tracker.slow, None);
|
||||||
|
tracker.start_recycling(at(730));
|
||||||
|
assert_eq!(tracker.recycle(at(1_030)), Some((0, None)));
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
@@ -1326,13 +1458,16 @@ mod tests {
|
|||||||
minute(&mut tracker, &[(3, 90), (5, 50)], 99);
|
minute(&mut tracker, &[(3, 90), (5, 50)], 99);
|
||||||
assert_eq!(tracker.judge_speeds(at(seconds)), None);
|
assert_eq!(tracker.judge_speeds(at(seconds)), None);
|
||||||
}
|
}
|
||||||
// A connection down holds every recycle, as the hourly one's rule.
|
// A connection down holds the slow one's turn, as every recycle.
|
||||||
let mut tracker = twelve_up(start);
|
let mut tracker = twelve_up(start);
|
||||||
|
tracker.start_recycling(start);
|
||||||
tracker.down(0);
|
tracker.down(0);
|
||||||
for seconds in [601, 661, 721, 781] {
|
for seconds in [601, 661, 721] {
|
||||||
minute(&mut tracker, &[(3, 90), (5, 50)], 100);
|
minute(&mut tracker, &[(3, 90), (5, 50)], 100);
|
||||||
assert_eq!(tracker.judge_speeds(at(seconds)), None);
|
tracker.judge_speeds(at(seconds));
|
||||||
}
|
}
|
||||||
|
assert_eq!(tracker.slow, Some((3, 90_000)));
|
||||||
|
assert_eq!(tracker.recycle(at(1_000)), None);
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
@@ -1447,3 +1582,200 @@ mod tests {
|
|||||||
assert_eq!(tracker.markets.len(), 1);
|
assert_eq!(tracker.markets.len(), 1);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// [`run`] end to end, offline: Kalshi's listing and websocket served
|
||||||
|
/// locally, the writer as `main` runs it.
|
||||||
|
#[cfg(test)]
|
||||||
|
mod run_tests {
|
||||||
|
use std::io::{BufRead, BufReader, Write};
|
||||||
|
use std::net::{TcpListener, TcpStream};
|
||||||
|
use std::sync::Arc;
|
||||||
|
use std::sync::atomic::{AtomicBool, Ordering};
|
||||||
|
use std::thread;
|
||||||
|
use std::time::Duration;
|
||||||
|
|
||||||
|
use crossbeam::channel::{Sender, unbounded};
|
||||||
|
use tungstenite::Message;
|
||||||
|
|
||||||
|
use super::SUBSCRIPTIONS;
|
||||||
|
use crate::kalshi::{Kalshi, sign};
|
||||||
|
|
||||||
|
const TICKERS: [&str; 2] = ["KXA", "KXB"];
|
||||||
|
|
||||||
|
/// A local listing: the tickers open, nothing unopened.
|
||||||
|
fn listing() -> String {
|
||||||
|
let listener = TcpListener::bind("127.0.0.1:0").unwrap();
|
||||||
|
let url = format!("http://{}", listener.local_addr().unwrap());
|
||||||
|
thread::spawn(move || {
|
||||||
|
for stream in listener.incoming() {
|
||||||
|
let stream = stream.unwrap();
|
||||||
|
let mut reader = BufReader::new(&stream);
|
||||||
|
let mut request = String::new();
|
||||||
|
reader.read_line(&mut request).unwrap();
|
||||||
|
// The headers, read whole so the close doesn't reset them.
|
||||||
|
let mut line = String::new();
|
||||||
|
while reader.read_line(&mut line).unwrap() > 2 {
|
||||||
|
line.clear();
|
||||||
|
}
|
||||||
|
let markets: Vec<String> = TICKERS
|
||||||
|
.iter()
|
||||||
|
.map(|ticker| format!(r#"{{"ticker":"{ticker}","status":"active"}}"#))
|
||||||
|
.collect();
|
||||||
|
let open = request.contains("status=open") && !request.contains("cursor=");
|
||||||
|
let markets = if open {
|
||||||
|
markets.join(",")
|
||||||
|
} else {
|
||||||
|
String::new()
|
||||||
|
};
|
||||||
|
let body = format!(r#"{{"markets":[{markets}],"cursor":""}}"#);
|
||||||
|
write!(
|
||||||
|
&stream,
|
||||||
|
"HTTP/1.1 200 OK\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}",
|
||||||
|
body.len()
|
||||||
|
)
|
||||||
|
.unwrap();
|
||||||
|
}
|
||||||
|
});
|
||||||
|
url
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A local websocket: each connection answers every ticker it subscribes
|
||||||
|
/// with a book and a change, the same on every connection. `books` hears
|
||||||
|
/// of each ticker served, `closed` whether each connection was closed
|
||||||
|
/// with a Close frame.
|
||||||
|
fn websocket(books: Sender<()>, closed: Sender<bool>) -> String {
|
||||||
|
let listener = TcpListener::bind("127.0.0.1:0").unwrap();
|
||||||
|
let url = format!("ws://{}/trade-api/ws/v2", listener.local_addr().unwrap());
|
||||||
|
thread::spawn(move || {
|
||||||
|
for stream in listener.incoming() {
|
||||||
|
let (books, closed) = (books.clone(), closed.clone());
|
||||||
|
thread::spawn(move || serve(stream.unwrap(), &books, &closed));
|
||||||
|
}
|
||||||
|
});
|
||||||
|
url
|
||||||
|
}
|
||||||
|
|
||||||
|
fn serve(stream: TcpStream, books: &Sender<()>, closed: &Sender<bool>) {
|
||||||
|
let mut socket = tungstenite::accept(stream).unwrap();
|
||||||
|
let mut seq = 0;
|
||||||
|
loop {
|
||||||
|
match socket.read() {
|
||||||
|
Ok(Message::Text(text)) => {
|
||||||
|
let command: serde_json::Value = serde_json::from_str(&text).unwrap();
|
||||||
|
if command["cmd"] != "subscribe" {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if seq == 0 {
|
||||||
|
let subscribed = r#"{"type":"subscribed","id":1,"msg":{"channel":"orderbook_delta","sid":1}}"#;
|
||||||
|
socket.send(Message::text(subscribed)).unwrap();
|
||||||
|
}
|
||||||
|
for ticker in command["params"]["market_tickers"].as_array().unwrap() {
|
||||||
|
let ticker = ticker.as_str().unwrap();
|
||||||
|
let book = format!(
|
||||||
|
r#"{{"type":"orderbook_snapshot","sid":1,"seq":{},"msg":{{"market_ticker":"{ticker}","market_id":"x","clock":10,"yes_dollars_fp":[["0.4000","5.00"]]}},"sending_ts_ms":1794422918369}}"#,
|
||||||
|
seq + 1
|
||||||
|
);
|
||||||
|
let change = format!(
|
||||||
|
r#"{{"type":"orderbook_delta","sid":1,"seq":{},"msg":{{"market_ticker":"{ticker}","market_id":"x","clock":11,"price_dollars":"0.4000","delta_fp":"1.00","side":"yes","ts_ms":1794422918370}}}}"#,
|
||||||
|
seq + 2
|
||||||
|
);
|
||||||
|
seq += 2;
|
||||||
|
socket.send(Message::text(book)).unwrap();
|
||||||
|
socket.send(Message::text(change)).unwrap();
|
||||||
|
_ = books.send(());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Ok(Message::Close(_)) => {
|
||||||
|
// The reply goes out with the next read.
|
||||||
|
while socket.read().is_ok() {}
|
||||||
|
_ = closed.send(true);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
Ok(_) => {}
|
||||||
|
Err(_) => {
|
||||||
|
_ = closed.send(false);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_run_subscribes_writes_and_stops_on_the_flag_closing_every_socket() {
|
||||||
|
let (books_tx, books) = unbounded();
|
||||||
|
let (closed_tx, closed) = unbounded();
|
||||||
|
let kalshi = Kalshi {
|
||||||
|
websocket: websocket(books_tx, closed_tx),
|
||||||
|
listing: listing(),
|
||||||
|
key: sign::test_key(),
|
||||||
|
};
|
||||||
|
let path =
|
||||||
|
std::env::temp_dir().join(format!("orchestrator-run-{}.sbe", std::process::id()));
|
||||||
|
let (outputs, outputs_rx) = unbounded();
|
||||||
|
// Between the run and the writer: says when every change has gone
|
||||||
|
// through, so the test stops then, not after a guessed sleep.
|
||||||
|
let (relayed, writer_rx) = unbounded();
|
||||||
|
let (all_deltas_tx, all_deltas) = unbounded();
|
||||||
|
thread::spawn(move || {
|
||||||
|
let mut deltas = 0;
|
||||||
|
for output in outputs_rx {
|
||||||
|
if matches!(output, engineering_challenge::book::Output::Delta(_)) {
|
||||||
|
deltas += 1;
|
||||||
|
if deltas == TICKERS.len() {
|
||||||
|
_ = all_deltas_tx.send(());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
_ = relayed.send(output);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
let (written_tx, written) = unbounded();
|
||||||
|
let writer_path = path.to_str().unwrap().to_string();
|
||||||
|
thread::spawn(move || {
|
||||||
|
let result = crate::write(&writer_path, writer_rx).map_err(|error| error.to_string());
|
||||||
|
written_tx.send(result).unwrap();
|
||||||
|
});
|
||||||
|
let stop = Arc::new(AtomicBool::new(false));
|
||||||
|
let run_stop = Arc::clone(&stop);
|
||||||
|
let (failed_tx, failed) = unbounded();
|
||||||
|
thread::spawn(move || failed_tx.send(super::run(Arc::new(kalshi), outputs, &run_stop)));
|
||||||
|
|
||||||
|
// Each ticker served on each of its connections.
|
||||||
|
for _ in 0..TICKERS.len() * SUBSCRIPTIONS {
|
||||||
|
books
|
||||||
|
.recv_timeout(Duration::from_secs(10))
|
||||||
|
.expect("every subscription served");
|
||||||
|
}
|
||||||
|
// Their events reach the book threads, and the writer.
|
||||||
|
all_deltas
|
||||||
|
.recv_timeout(Duration::from_secs(10))
|
||||||
|
.expect("every change written");
|
||||||
|
stop.store(true, Ordering::Relaxed);
|
||||||
|
// Timed, so a run that never ends fails rather than hangs.
|
||||||
|
let failed = failed
|
||||||
|
.recv_timeout(Duration::from_secs(10))
|
||||||
|
.expect("the run returns once stopped");
|
||||||
|
assert_eq!(failed, 0);
|
||||||
|
let written = written
|
||||||
|
.recv_timeout(Duration::from_secs(10))
|
||||||
|
.expect("the writer ends once the run has");
|
||||||
|
let closes: Vec<bool> = (0..SUBSCRIPTIONS)
|
||||||
|
.map(|_| closed.recv_timeout(Duration::from_secs(5)).unwrap())
|
||||||
|
.collect();
|
||||||
|
assert_eq!(
|
||||||
|
closes, [true; SUBSCRIPTIONS],
|
||||||
|
"a Close frame on every socket"
|
||||||
|
);
|
||||||
|
// No other connection.
|
||||||
|
assert!(closed.recv_timeout(Duration::from_millis(200)).is_err());
|
||||||
|
written.unwrap();
|
||||||
|
let report = engineering_challenge::reconcile::reconcile(&path, None);
|
||||||
|
std::fs::remove_file(&path).unwrap();
|
||||||
|
let report = report.unwrap();
|
||||||
|
// A first book per ticker, and one per minute crossed; each change
|
||||||
|
// once, the other connection's copy a duplicate.
|
||||||
|
assert!(report.snapshots >= TICKERS.len(), "{report:?}");
|
||||||
|
assert_eq!(report.deltas, TICKERS.len(), "{report:?}");
|
||||||
|
assert_eq!(report.orphan_deltas, 0);
|
||||||
|
assert_eq!(report.mismatches, []);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -31,6 +31,7 @@ const CLOSE_TIMEOUT: Duration = Duration::from_secs(2);
|
|||||||
/// Nothing received this long, not even a keepalive's answer or a ping: the
|
/// Nothing received this long, not even a keepalive's answer or a ping: the
|
||||||
/// connection is silent, and reconnects.
|
/// connection is silent, and reconnects.
|
||||||
pub const SILENT_AFTER: Duration = Duration::from_secs(30);
|
pub const SILENT_AFTER: Duration = Duration::from_secs(30);
|
||||||
|
|
||||||
/// What a book thread asks a connection for.
|
/// What a book thread asks a connection for.
|
||||||
#[derive(Debug, PartialEq, Eq)]
|
#[derive(Debug, PartialEq, Eq)]
|
||||||
pub enum Command {
|
pub enum Command {
|
||||||
|
|||||||
+174
-5
@@ -29,7 +29,7 @@ const BOOK: u32 = u32::MAX;
|
|||||||
/// Changes remembered per symbol for their second copy's lateness: two
|
/// Changes remembered per symbol for their second copy's lateness: two
|
||||||
/// connections deliver a market's changes 2 ms apart at the median, 344 ms
|
/// connections deliver a market's changes 2 ms apart at the median, 344 ms
|
||||||
/// at worst (docs/notes.md), far fewer changes than this.
|
/// at worst (docs/notes.md), far fewer changes than this.
|
||||||
// ponytail: a second copy more than 128 changes behind goes unmeasured, and
|
// Ceiling: a second copy more than 128 changes behind goes unmeasured, and
|
||||||
// the lookup is linear; a map by position, emptied as the book passes,
|
// the lookup is linear; a map by position, emptied as the book passes,
|
||||||
// would lift both.
|
// would lift both.
|
||||||
const RECENT: usize = 128;
|
const RECENT: usize = 128;
|
||||||
@@ -205,8 +205,11 @@ impl Clock {
|
|||||||
.recent
|
.recent
|
||||||
.push_back((position, connection, received_at));
|
.push_back((position, connection, received_at));
|
||||||
let quantity = symbol.book.add(change.side, change.price, change.delta);
|
let quantity = symbol.book.add(change.side, change.price, change.delta);
|
||||||
let quantity = if quantity < 0 {
|
let quantity = if let Some(quantity) = quantity {
|
||||||
// A level can't hold less than nothing: ours was wrong.
|
quantity
|
||||||
|
} else {
|
||||||
|
// A level can't hold less than nothing (nor more than an i64):
|
||||||
|
// ours was wrong.
|
||||||
self.counts.below_zero += 1;
|
self.counts.below_zero += 1;
|
||||||
log!(
|
log!(
|
||||||
"kalshi: {ticker}: a change takes a level below zero, resync on connection \
|
"kalshi: {ticker}: a change takes a level below zero, resync on connection \
|
||||||
@@ -217,8 +220,6 @@ impl Clock {
|
|||||||
self.counts.resyncs += 1;
|
self.counts.resyncs += 1;
|
||||||
}
|
}
|
||||||
0
|
0
|
||||||
} else {
|
|
||||||
quantity
|
|
||||||
};
|
};
|
||||||
symbol.exchange_timestamp = change.at;
|
symbol.exchange_timestamp = change.at;
|
||||||
self.counts.written += 1;
|
self.counts.written += 1;
|
||||||
@@ -677,6 +678,20 @@ mod tests {
|
|||||||
assert_eq!(deltas(&after)[0].quantity, 10_000_000);
|
assert_eq!(deltas(&after)[0].quantity, 10_000_000);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// A change that would take a level past an i64: the book is wrong, as
|
||||||
|
/// below zero.
|
||||||
|
#[test]
|
||||||
|
fn a_level_past_the_largest_quantity_resyncs() {
|
||||||
|
let mut feed = both_on("M");
|
||||||
|
let level = feed.shell.symbols["M"].book.bids[&400_000];
|
||||||
|
assert!(level > 0);
|
||||||
|
let effects = feed.from(0, 3, change("M", 11, Side::Buy, 400_000, i64::MAX));
|
||||||
|
assert_eq!(deltas(&effects)[0].quantity, 0);
|
||||||
|
assert_eq!(resyncs(&effects), [(0, "M")]);
|
||||||
|
assert_eq!(feed.shell.symbols["M"].book.bids.get(&400_000), None);
|
||||||
|
assert_eq!(feed.rule.counts.below_zero, 1);
|
||||||
|
}
|
||||||
|
|
||||||
/// The answer to a resync can be behind what the other connection has
|
/// The answer to a resync can be behind what the other connection has
|
||||||
/// delivered: the book goes back to it, and only a stream that continues
|
/// delivered: the book goes back to it, and only a stream that continues
|
||||||
/// from there is taken until the other fits again.
|
/// from there is taken until the other fits again.
|
||||||
@@ -778,6 +793,140 @@ mod tests {
|
|||||||
assert_eq!(snapshots(&feed.tick(minute + TimeDelta::minutes(2))), 1);
|
assert_eq!(snapshots(&feed.tick(minute + TimeDelta::minutes(2))), 1);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
fn unsubscribes(effects: &[Effect]) -> Vec<usize> {
|
||||||
|
effects
|
||||||
|
.iter()
|
||||||
|
.filter_map(|effect| match effect {
|
||||||
|
Effect::Send(connection, Command::Unsubscribe(_)) => Some(*connection),
|
||||||
|
_ => None,
|
||||||
|
})
|
||||||
|
.collect()
|
||||||
|
}
|
||||||
|
|
||||||
|
fn market(ticker: &str) -> Market {
|
||||||
|
Market {
|
||||||
|
ticker: ticker.into(),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// `both_on`'s market moved from connection 0 to 2, the move ended by
|
||||||
|
/// connection 2's book: connection 0 left it.
|
||||||
|
fn moved_off_zero() -> Feed {
|
||||||
|
let mut feed = both_on("M");
|
||||||
|
feed.move_market(&market("M"), 0, 2, at(3));
|
||||||
|
let ended = feed.from(2, 4, book("M", 10, &[(400_000, 5_000_000)], &[]));
|
||||||
|
assert_eq!(unsubscribes(&ended), [0]);
|
||||||
|
feed
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_book_from_a_connection_the_symbol_left_counts_for_nothing() {
|
||||||
|
let mut feed = moved_off_zero();
|
||||||
|
// Still on its way when it left, past ours and different.
|
||||||
|
let late = feed.from(0, 5, book("M", 20, &[(400_000, 1_000_000)], &[]));
|
||||||
|
assert!(late.is_empty(), "{late:?}");
|
||||||
|
assert_eq!(feed.rule.counts.losses, 0);
|
||||||
|
assert_eq!(feed.shell.symbols["M"].book.bids[&400_000], 5_000_000);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_change_from_a_connection_the_symbol_left_has_no_place() {
|
||||||
|
let mut feed = moved_off_zero();
|
||||||
|
let late = feed.from(0, 5, change("M", 11, Side::Buy, 400_000, 1_000_000));
|
||||||
|
assert!(deltas(&late).is_empty(), "{late:?}");
|
||||||
|
assert_eq!(feed.rule.counts.unplaced, 1);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The answer to a resync can be our very book: the resync is over, not
|
||||||
|
/// asked again every minute.
|
||||||
|
#[test]
|
||||||
|
fn an_answer_equal_to_ours_ends_the_resync() {
|
||||||
|
let mut feed = both_on("M");
|
||||||
|
feed.on_received(Received::Down(0));
|
||||||
|
feed.on_received(Received::Down(1));
|
||||||
|
feed.on_received(Received::Up(0));
|
||||||
|
feed.on_received(Received::Up(1));
|
||||||
|
let minute = next_minute(at(10));
|
||||||
|
assert_eq!(resyncs(&feed.tick(minute)), [(0, "M")]);
|
||||||
|
let answer = feed.from(0, 70_000, book("M", 10, &[(400_000, 5_000_000)], &[]));
|
||||||
|
assert!(answer.is_empty(), "{answer:?}");
|
||||||
|
assert_eq!(feed.shell.symbols["M"].rule.asked, None);
|
||||||
|
assert!(resyncs(&feed.tick(minute + TimeDelta::minutes(1))).is_empty());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_second_level_below_zero_while_a_resync_is_asked_asks_no_other() {
|
||||||
|
let mut feed = both_on("M");
|
||||||
|
let first = feed.from(0, 3, change("M", 11, Side::Buy, 400_000, -6_000_000));
|
||||||
|
assert_eq!(resyncs(&first), [(0, "M")]);
|
||||||
|
let second = feed.from(0, 4, change("M", 12, Side::Sell, 600_000, -1_000_000));
|
||||||
|
assert_eq!(deltas(&second)[0].quantity, 0);
|
||||||
|
assert!(resyncs(&second).is_empty(), "{second:?}");
|
||||||
|
assert_eq!(feed.rule.counts.below_zero, 2);
|
||||||
|
assert_eq!(feed.rule.counts.resyncs, 1);
|
||||||
|
}
|
||||||
|
|
||||||
|
// The shell's moves and quiet watch (src/core/feed.rs), through Kalshi's
|
||||||
|
// rule, the simpler one.
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_move_ends_on_the_new_connections_book_not_on_its_change() {
|
||||||
|
let mut feed = both_on("M");
|
||||||
|
feed.move_market(&market("M"), 0, 2, at(3));
|
||||||
|
let change = feed.from(2, 4, change("M", 11, Side::Buy, 400_000, 1_000_000));
|
||||||
|
assert!(unsubscribes(&change).is_empty(), "{change:?}");
|
||||||
|
let ended = feed.from(2, 5, book("M", 10, &[(400_000, 5_000_000)], &[]));
|
||||||
|
assert_eq!(unsubscribes(&ended), [0]);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_second_move_while_one_is_under_way_ends_the_first() {
|
||||||
|
let mut feed = both_on("M");
|
||||||
|
feed.move_market(&market("M"), 0, 2, at(3));
|
||||||
|
let second = feed.move_market(&market("M"), 1, 3, at(4));
|
||||||
|
assert_eq!(unsubscribes(&second), [0]);
|
||||||
|
assert!(second.iter().any(|effect| matches!(
|
||||||
|
effect,
|
||||||
|
Effect::Send(3, Command::Subscribe(tickers)) if tickers == &["M".to_string()]
|
||||||
|
)));
|
||||||
|
let mut on = feed.shell.symbols["M"].connections.clone();
|
||||||
|
on.sort();
|
||||||
|
assert_eq!(on, [1, 2, 3]);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Quiet on connection 1, resubscribed; a minute delivering on both;
|
||||||
|
/// quiet again: resubscribed again, not reported for a move.
|
||||||
|
#[test]
|
||||||
|
fn a_good_minute_between_two_quiet_ones_forgets_the_first() {
|
||||||
|
let mut feed = both_on("M");
|
||||||
|
let minute = |n: i64| next_minute(at(10)) + TimeDelta::minutes(n);
|
||||||
|
feed.tick(minute(0));
|
||||||
|
let mut clock = 10;
|
||||||
|
let mut changes = |feed: &mut Feed, n: i64, on: &[usize]| {
|
||||||
|
for step in 0..3 {
|
||||||
|
clock += 1;
|
||||||
|
for &connection in on {
|
||||||
|
let millis = (minute(n) - at(0)).num_milliseconds() + 10_000 + step;
|
||||||
|
feed.from(connection, millis, change("M", clock, Side::Buy, 1, 1));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
};
|
||||||
|
changes(&mut feed, 0, &[0]);
|
||||||
|
assert_eq!(resyncs(&feed.tick(minute(1))), [(1, "M")]);
|
||||||
|
changes(&mut feed, 1, &[0, 1]);
|
||||||
|
assert!(resyncs(&feed.tick(minute(2))).is_empty());
|
||||||
|
changes(&mut feed, 2, &[0]);
|
||||||
|
let again = feed.tick(minute(3));
|
||||||
|
assert_eq!(resyncs(&again), [(1, "M")]);
|
||||||
|
assert!(
|
||||||
|
!again
|
||||||
|
.iter()
|
||||||
|
.any(|effect| matches!(effect, Effect::Report(feed::Report::Quiet { .. }))),
|
||||||
|
"{again:?}"
|
||||||
|
);
|
||||||
|
assert_eq!(feed.shell.quiet, 2);
|
||||||
|
}
|
||||||
|
|
||||||
/// The demo's recording on two connections (tests/fixtures/README.md).
|
/// The demo's recording on two connections (tests/fixtures/README.md).
|
||||||
fn recording() -> Vec<(DateTime<Utc>, usize, Event)> {
|
fn recording() -> Vec<(DateTime<Utc>, usize, Event)> {
|
||||||
#[derive(Deserialize)]
|
#[derive(Deserialize)]
|
||||||
@@ -871,6 +1020,26 @@ mod tests {
|
|||||||
assert!(two_sided >= 10, "{two_sided}");
|
assert!(two_sided >= 10, "{two_sided}");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// A market removed and tracked again: its first book's Snapshot says
|
||||||
|
/// so, once.
|
||||||
|
#[test]
|
||||||
|
fn a_market_tracked_again_says_so_in_its_first_snapshot() {
|
||||||
|
let mut feed = feed_of(&["A"], &[0]);
|
||||||
|
let first = feed.from(0, 1, book("A", 1, &[(400_000, 1_000_000)], &[]));
|
||||||
|
assert_eq!(snapshots(&first), 1);
|
||||||
|
assert_eq!(feed.shell.symbols["A"].again, None);
|
||||||
|
let a = market("A");
|
||||||
|
feed.remove_market(&a);
|
||||||
|
feed.add_market(&a, &[0], at(10));
|
||||||
|
assert_eq!(feed.shell.symbols["A"].again, Some("kalshi"));
|
||||||
|
let back = feed.from(0, 20, book("A", 9, &[(500_000, 2_000_000)], &[]));
|
||||||
|
assert_eq!(snapshots(&back), 1);
|
||||||
|
assert_eq!(feed.shell.symbols["A"].again, None);
|
||||||
|
// A market new to the run is not "again".
|
||||||
|
feed.add_market(&market("B"), &[0], at(20));
|
||||||
|
assert_eq!(feed.shell.symbols["B"].again, None);
|
||||||
|
}
|
||||||
|
|
||||||
/// The book and writer threads as main runs them, fed the recording:
|
/// The book and writer threads as main runs them, fed the recording:
|
||||||
/// every Snapshot is the one before it with the Deltas between.
|
/// every Snapshot is the one before it with the Deltas between.
|
||||||
#[test]
|
#[test]
|
||||||
|
|||||||
@@ -1,6 +1,7 @@
|
|||||||
//! Kalshi's market listing: which markets are live, or about to be. Two fetcher threads page
|
//! Kalshi's market listing: which markets are live, or about to be. Two
|
||||||
//! it on the orchestrator's request and stream what they read; they hold no
|
//! fetcher threads page it on the orchestrator's request and send on what
|
||||||
//! state and no timer, as Gamma's (roadmap/06). Read without a key.
|
//! they read; they hold no state and no timer, as Gamma's (roadmap/06). Read
|
||||||
|
//! without a key.
|
||||||
|
|
||||||
use std::time::Duration;
|
use std::time::Duration;
|
||||||
|
|
||||||
|
|||||||
+6
-5
@@ -15,8 +15,9 @@ use tungstenite::client::IntoClientRequest;
|
|||||||
use crate::core::{self, Finding, Request, Undecoded};
|
use crate::core::{self, Finding, Request, Undecoded};
|
||||||
use sign::Key;
|
use sign::Key;
|
||||||
|
|
||||||
/// Tickers per subscribe or unsubscribe message.
|
/// Tickers per subscribe or unsubscribe message, as Polymarket's (whose
|
||||||
const CHUNK: usize = 500;
|
/// limit is known, docs/notes.md); Kalshi documents none.
|
||||||
|
const PER_MESSAGE: usize = 500;
|
||||||
/// Errors after which the subscription is gone (Kalshi's docs: a channel
|
/// Errors after which the subscription is gone (Kalshi's docs: a channel
|
||||||
/// error, a subscription buffer overflow): the connection starts over.
|
/// error, a subscription buffer overflow): the connection starts over.
|
||||||
const TERMINAL_ERRORS: [u32; 2] = [10, 25];
|
const TERMINAL_ERRORS: [u32; 2] = [10, 25];
|
||||||
@@ -74,7 +75,7 @@ impl core::Exchange for Kalshi {
|
|||||||
|
|
||||||
fn request(&self) -> anyhow::Result<tungstenite::handshake::client::Request> {
|
fn request(&self) -> anyhow::Result<tungstenite::handshake::client::Request> {
|
||||||
let mut request = self.websocket.as_str().into_client_request()?;
|
let mut request = self.websocket.as_str().into_client_request()?;
|
||||||
for (name, value) in self.key.headers(Utc::now().timestamp_millis()) {
|
for (name, value) in self.key.headers(Utc::now().timestamp_millis())? {
|
||||||
request.headers_mut().insert(name, value.parse()?);
|
request.headers_mut().insert(name, value.parse()?);
|
||||||
}
|
}
|
||||||
Ok(request)
|
Ok(request)
|
||||||
@@ -125,7 +126,7 @@ impl core::Session for Session {
|
|||||||
/// default is announced to flip.
|
/// default is announced to flip.
|
||||||
fn subscribe(&mut self, tickers: &[&String]) -> Vec<String> {
|
fn subscribe(&mut self, tickers: &[&String]) -> Vec<String> {
|
||||||
tickers
|
tickers
|
||||||
.chunks(CHUNK)
|
.chunks(PER_MESSAGE)
|
||||||
.map(|chunk| {
|
.map(|chunk| {
|
||||||
let params = json!({
|
let params = json!({
|
||||||
"channels": ["orderbook_delta"],
|
"channels": ["orderbook_delta"],
|
||||||
@@ -143,7 +144,7 @@ impl core::Session for Session {
|
|||||||
// unsubscribe sent before its answer names it so.
|
// unsubscribe sent before its answer names it so.
|
||||||
let sid = self.sid.unwrap_or(1);
|
let sid = self.sid.unwrap_or(1);
|
||||||
tickers
|
tickers
|
||||||
.chunks(CHUNK)
|
.chunks(PER_MESSAGE)
|
||||||
.map(|chunk| {
|
.map(|chunk| {
|
||||||
let params = json!({
|
let params = json!({
|
||||||
"sids": [sid],
|
"sids": [sid],
|
||||||
|
|||||||
+191
-25
@@ -1,11 +1,13 @@
|
|||||||
//! The key that signs Kalshi's websocket handshake (docs/notes.md, Kalshi's
|
//! The key that signs Kalshi's websocket handshake (docs/notes.md, Kalshi's
|
||||||
//! websocket): an Ed25519 key, read once from the files that the environment
|
//! websocket): Ed25519, or RSA signed with RSA-PSS and SHA-256, the two
|
||||||
//! names. Neither file's content is ever logged.
|
//! kinds Kalshi takes; read once from the files that the environment names.
|
||||||
|
//! Neither file's content is ever logged.
|
||||||
|
|
||||||
use std::path::Path;
|
use std::path::Path;
|
||||||
|
|
||||||
use base64::Engine;
|
use base64::Engine;
|
||||||
use ring::signature::Ed25519KeyPair;
|
use ring::rand::SystemRandom;
|
||||||
|
use ring::signature::{Ed25519KeyPair, RSA_PSS_SHA256, RsaKeyPair};
|
||||||
use rustls_pki_types::PrivateKeyDer;
|
use rustls_pki_types::PrivateKeyDer;
|
||||||
use rustls_pki_types::pem::PemObject;
|
use rustls_pki_types::pem::PemObject;
|
||||||
|
|
||||||
@@ -18,15 +20,33 @@ const SIGNED_PATH: &str = "GET/trade-api/ws/v2";
|
|||||||
|
|
||||||
pub struct Key {
|
pub struct Key {
|
||||||
id: String,
|
id: String,
|
||||||
pair: Ed25519KeyPair,
|
pair: Pair,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Kalshi tells the two apart by the public key it holds; the PEM's header
|
||||||
|
/// doesn't say (an RSA key can be PKCS#8 too), so the parsed key does.
|
||||||
|
enum Pair {
|
||||||
|
Ed25519(Ed25519KeyPair),
|
||||||
|
Rsa(RsaKeyPair),
|
||||||
}
|
}
|
||||||
|
|
||||||
impl Key {
|
impl Key {
|
||||||
/// The key from the files [`KEY_ID_FILE`] and [`PRIVATE_KEY_FILE`] name.
|
/// The key from the files [`KEY_ID_FILE`] and [`PRIVATE_KEY_FILE`] name.
|
||||||
pub fn from_env() -> anyhow::Result<Self> {
|
pub fn from_env() -> anyhow::Result<Self> {
|
||||||
let path = |variable: &str| {
|
Self::from_variables(|name| std::env::var_os(name))
|
||||||
std::env::var_os(variable)
|
}
|
||||||
.ok_or_else(|| anyhow::anyhow!("{variable} is not set: it names a file of the key"))
|
|
||||||
|
/// [`Self::from_env`], the variables read through `variable`.
|
||||||
|
fn from_variables(
|
||||||
|
variable: impl Fn(&str) -> Option<std::ffi::OsString>,
|
||||||
|
) -> anyhow::Result<Self> {
|
||||||
|
let path = |name: &str| {
|
||||||
|
variable(name).ok_or_else(|| {
|
||||||
|
anyhow::anyhow!(
|
||||||
|
"{name} is not set: Kalshi needs {KEY_ID_FILE} and {PRIVATE_KEY_FILE}, \
|
||||||
|
the files of its key id and its private key"
|
||||||
|
)
|
||||||
|
})
|
||||||
};
|
};
|
||||||
Self::from_files(
|
Self::from_files(
|
||||||
path(KEY_ID_FILE)?.as_ref(),
|
path(KEY_ID_FILE)?.as_ref(),
|
||||||
@@ -37,7 +57,8 @@ impl Key {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
/// The key id in `id_file`, the private key (a PKCS#8 PEM) in `key_file`.
|
/// The key id in `id_file`, the private key in `key_file`: a PEM,
|
||||||
|
/// Ed25519 in PKCS#8, or RSA in PKCS#1 (as Kalshi gives it) or PKCS#8.
|
||||||
pub fn from_files(id_file: &Path, key_file: &Path) -> anyhow::Result<Self> {
|
pub fn from_files(id_file: &Path, key_file: &Path) -> anyhow::Result<Self> {
|
||||||
let id = std::fs::read_to_string(id_file)
|
let id = std::fs::read_to_string(id_file)
|
||||||
.map_err(|error| anyhow::anyhow!("{}: {error}", id_file.display()))?
|
.map_err(|error| anyhow::anyhow!("{}: {error}", id_file.display()))?
|
||||||
@@ -50,28 +71,54 @@ impl Key {
|
|||||||
// error quotes what it read, so it is dropped.
|
// error quotes what it read, so it is dropped.
|
||||||
let der = PrivateKeyDer::from_pem_slice(&pem)
|
let der = PrivateKeyDer::from_pem_slice(&pem)
|
||||||
.map_err(|_| anyhow::anyhow!("{}: not a PEM private key", key_file.display()))?;
|
.map_err(|_| anyhow::anyhow!("{}: not a PEM private key", key_file.display()))?;
|
||||||
let PrivateKeyDer::Pkcs8(der) = der else {
|
let pair = match &der {
|
||||||
anyhow::bail!(
|
PrivateKeyDer::Pkcs8(der) => {
|
||||||
"{}: not a PKCS#8 key; Kalshi's Ed25519 keys are",
|
Ed25519KeyPair::from_pkcs8_maybe_unchecked(der.secret_pkcs8_der())
|
||||||
key_file.display()
|
.map(Pair::Ed25519)
|
||||||
);
|
.ok()
|
||||||
|
.or_else(|| {
|
||||||
|
RsaKeyPair::from_pkcs8(der.secret_pkcs8_der())
|
||||||
|
.ok()
|
||||||
|
.map(Pair::Rsa)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
PrivateKeyDer::Pkcs1(der) => RsaKeyPair::from_der(der.secret_pkcs1_der())
|
||||||
|
.ok()
|
||||||
|
.map(Pair::Rsa),
|
||||||
|
_ => None,
|
||||||
};
|
};
|
||||||
let pair = Ed25519KeyPair::from_pkcs8_maybe_unchecked(der.secret_pkcs8_der())
|
let pair = pair.ok_or_else(|| {
|
||||||
.map_err(|_| anyhow::anyhow!("{}: not an Ed25519 key", key_file.display()))?;
|
anyhow::anyhow!("{}: neither an Ed25519 nor an RSA key", key_file.display())
|
||||||
|
})?;
|
||||||
Ok(Self { id, pair })
|
Ok(Self { id, pair })
|
||||||
}
|
}
|
||||||
|
|
||||||
/// The handshake's three headers at `millis` since the epoch.
|
/// The handshake's three headers at `millis` since the epoch.
|
||||||
pub fn headers(&self, millis: i64) -> [(&'static str, String); 3] {
|
pub fn headers(&self, millis: i64) -> anyhow::Result<[(&'static str, String); 3]> {
|
||||||
let signature = self.pair.sign(format!("{millis}{SIGNED_PATH}").as_bytes());
|
let signed = format!("{millis}{SIGNED_PATH}");
|
||||||
[
|
let signature = match &self.pair {
|
||||||
|
Pair::Ed25519(pair) => pair.sign(signed.as_bytes()).as_ref().to_vec(),
|
||||||
|
Pair::Rsa(pair) => {
|
||||||
|
// Salted: a new signature each time, from the system's random.
|
||||||
|
let mut signature = vec![0; pair.public().modulus_len()];
|
||||||
|
pair.sign(
|
||||||
|
&RSA_PSS_SHA256,
|
||||||
|
&SystemRandom::new(),
|
||||||
|
signed.as_bytes(),
|
||||||
|
&mut signature,
|
||||||
|
)
|
||||||
|
.map_err(|_| anyhow::anyhow!("the RSA signature failed"))?;
|
||||||
|
signature
|
||||||
|
}
|
||||||
|
};
|
||||||
|
Ok([
|
||||||
("KALSHI-ACCESS-KEY", self.id.clone()),
|
("KALSHI-ACCESS-KEY", self.id.clone()),
|
||||||
("KALSHI-ACCESS-TIMESTAMP", millis.to_string()),
|
("KALSHI-ACCESS-TIMESTAMP", millis.to_string()),
|
||||||
(
|
(
|
||||||
"KALSHI-ACCESS-SIGNATURE",
|
"KALSHI-ACCESS-SIGNATURE",
|
||||||
base64::engine::general_purpose::STANDARD.encode(signature.as_ref()),
|
base64::engine::general_purpose::STANDARD.encode(signature),
|
||||||
),
|
),
|
||||||
]
|
])
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -92,7 +139,9 @@ pub fn test_key() -> Key {
|
|||||||
};
|
};
|
||||||
Key {
|
Key {
|
||||||
id: "a-key-id".into(),
|
id: "a-key-id".into(),
|
||||||
pair: Ed25519KeyPair::from_pkcs8_maybe_unchecked(der.secret_pkcs8_der()).unwrap(),
|
pair: Pair::Ed25519(
|
||||||
|
Ed25519KeyPair::from_pkcs8_maybe_unchecked(der.secret_pkcs8_der()).unwrap(),
|
||||||
|
),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -120,7 +169,7 @@ mod tests {
|
|||||||
&write(&dir, "key.pem", TEST_KEY),
|
&write(&dir, "key.pem", TEST_KEY),
|
||||||
)
|
)
|
||||||
.unwrap();
|
.unwrap();
|
||||||
let [id, time, signature] = key.headers(1_669_149_841_234);
|
let [id, time, signature] = key.headers(1_669_149_841_234).unwrap();
|
||||||
assert_eq!(id, ("KALSHI-ACCESS-KEY", "a-key-id".to_string()));
|
assert_eq!(id, ("KALSHI-ACCESS-KEY", "a-key-id".to_string()));
|
||||||
assert_eq!(
|
assert_eq!(
|
||||||
time,
|
time,
|
||||||
@@ -143,10 +192,122 @@ mod tests {
|
|||||||
.verify(b"1669149841234GET/trade-api/ws/v2", &bytes)
|
.verify(b"1669149841234GET/trade-api/ws/v2", &bytes)
|
||||||
.unwrap();
|
.unwrap();
|
||||||
// Ed25519 is deterministic: the same time, the same signature.
|
// Ed25519 is deterministic: the same time, the same signature.
|
||||||
assert_eq!(key.headers(1_669_149_841_234)[2], signature);
|
assert_eq!(key.headers(1_669_149_841_234).unwrap()[2], signature);
|
||||||
std::fs::remove_dir_all(dir).unwrap();
|
std::fs::remove_dir_all(dir).unwrap();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// RFC 9500's "testRSA2048" (section 2.1), a key published for tests,
|
||||||
|
/// in PKCS#1 as Kalshi gives it.
|
||||||
|
const RSA_PKCS1: &str = "-----BEGIN RSA PRIVATE KEY-----
|
||||||
|
MIIEowIBAAKCAQEAsPnoGUOnrpiSqt4XynxA+HRP7S+BSObI6qJ7fQAVSPtRkqso
|
||||||
|
tWxQYLEYzNEx5ZSHTGypibVsJylvCfuToDTfMul8b/CZjP2Ob0LdpYrNH6l5hvFE
|
||||||
|
89FU1nZQF15oVLOpUgA7wGiHuEVawrGfey92UE68mOyUVXGweJIVDdxqdMoPvNNU
|
||||||
|
l86BU02vlBiESxOuox+dWmuVV7vfYZ79Toh/LUK43YvJh+rhv4nKuF7iHjVjBd9s
|
||||||
|
B6iDjj70HFldzOQ9r8SRI+9NirupPTkF5AKNe6kUhKJ1luB7S27ZkvB3tSTT3P59
|
||||||
|
3VVJvnzOjaA1z6Cz+4+eRvcysqhrRgFlwI9TEwIDAQABAoIBAEEYiyDP29vCzx/+
|
||||||
|
dS3LqnI5BjUuJhXUnc6AWX/PCgVAO+8A+gZRgvct7PtZb0sM6P9ZcLrweomlGezI
|
||||||
|
FrL0/6xQaa8bBr/ve/a8155OgcjFo6fZEw3Dz7ra5fbSiPmu4/b/kvrg+Br1l77J
|
||||||
|
aun6uUAs1f5B9wW+vbR7tzbT/mxaUeDiBzKpe15GwcvbJtdIVMa2YErtRjc1/5B2
|
||||||
|
BGVXyvlJv0SIlcIEMsHgnAFOp1ZgQ08aDzvilLq8XVMOahAhP1O2A3X8hKdXPyrx
|
||||||
|
IVWE9bS9ptTo+eF6eNl+d7htpKGEZHUxinoQpWEBTv+iOoHsVunkEJ3vjLP3lyI/
|
||||||
|
fY0NQ1ECgYEA3RBXAjgvIys2gfU3keImF8e/TprLge1I2vbWmV2j6rZCg5r/AS0u
|
||||||
|
pii5CvJ5/T5vfJPNgPBy8B/yRDs+6PJO1GmnlhOkG9JAIPkv0RBZvR0PMBtbp6nT
|
||||||
|
Y3yo1lwamBVBfY6rc0sLTzosZh2aGoLzrHNMQFMGaauORzBFpY5lU50CgYEAzPHl
|
||||||
|
u5DI6Xgep1vr8QvCUuEesCOgJg8Yh1UqVoY/SmQh6MYAv1I9bLGwrb3WW/7kqIoD
|
||||||
|
fj0aQV5buVZI2loMomtU9KY5SFIsPV+JuUpy7/+VE01ZQM5FdY8wiYCQiVZYju9X
|
||||||
|
Wz5LxMNoz+gT7pwlLCsC4N+R8aoBk404aF1gum8CgYAJ7VTq7Zj4TFV7Soa/T1eE
|
||||||
|
k9y8a+kdoYk3BASpCHJ29M5R2KEA7YV9wrBklHTz8VzSTFTbKHEQ5W5csAhoL5Fo
|
||||||
|
qoHzFFi3Qx7MHESQb9qHyolHEMNx6QdsHUn7rlEnaTTyrXh3ifQtD6C0yTmFXUIS
|
||||||
|
CW9wKApOrnyKJ9nI0HcuZQKBgQCMtoV6e9VGX4AEfpuHvAAnMYQFgeBiYTkBKltQ
|
||||||
|
XwozhH63uMMomUmtSG87Sz1TmrXadjAhy8gsG6I0pWaN7QgBuFnzQ/HOkwTm+qKw
|
||||||
|
AsrZt4zeXNwsH7QXHEJCFnCmqw9QzEoZTrNtHJHpNboBuVnYcoueZEJrP8OnUG3r
|
||||||
|
UjmopwKBgAqB2KYYMUqAOvYcBnEfLDmyZv9BTVNHbR2lKkMYqv5LlvDaBxVfilE0
|
||||||
|
2riO4p6BaAdvzXjKeRrGNEKoHNBpOSfYCOM16NjL8hIZB1CaV3WbT5oY+jp7Mzd5
|
||||||
|
7d56RZOE+ERK2uz/7JX9VSsM/LbH9pJibd4e8mikDS9ntciqOH/3
|
||||||
|
-----END RSA PRIVATE KEY-----
|
||||||
|
";
|
||||||
|
/// The same key in PKCS#8 (`openssl pkcs8 -topk8 -nocrypt`).
|
||||||
|
const RSA_PKCS8: &str = "-----BEGIN PRIVATE KEY-----
|
||||||
|
MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCw+egZQ6eumJKq
|
||||||
|
3hfKfED4dE/tL4FI5sjqont9ABVI+1GSqyi1bFBgsRjM0THllIdMbKmJtWwnKW8J
|
||||||
|
+5OgNN8y6Xxv8JmM/Y5vQt2lis0fqXmG8UTz0VTWdlAXXmhUs6lSADvAaIe4RVrC
|
||||||
|
sZ97L3ZQTryY7JRVcbB4khUN3Gp0yg+801SXzoFTTa+UGIRLE66jH51aa5VXu99h
|
||||||
|
nv1OiH8tQrjdi8mH6uG/icq4XuIeNWMF32wHqIOOPvQcWV3M5D2vxJEj702Ku6k9
|
||||||
|
OQXkAo17qRSEonWW4HtLbtmS8He1JNPc/n3dVUm+fM6NoDXPoLP7j55G9zKyqGtG
|
||||||
|
AWXAj1MTAgMBAAECggEAQRiLIM/b28LPH/51LcuqcjkGNS4mFdSdzoBZf88KBUA7
|
||||||
|
7wD6BlGC9y3s+1lvSwzo/1lwuvB6iaUZ7MgWsvT/rFBprxsGv+979rzXnk6ByMWj
|
||||||
|
p9kTDcPPutrl9tKI+a7j9v+S+uD4GvWXvslq6fq5QCzV/kH3Bb69tHu3NtP+bFpR
|
||||||
|
4OIHMql7XkbBy9sm10hUxrZgSu1GNzX/kHYEZVfK+Um/RIiVwgQyweCcAU6nVmBD
|
||||||
|
TxoPO+KUurxdUw5qECE/U7YDdfyEp1c/KvEhVYT1tL2m1Oj54Xp42X53uG2koYRk
|
||||||
|
dTGKehClYQFO/6I6gexW6eQQne+Ms/eXIj99jQ1DUQKBgQDdEFcCOC8jKzaB9TeR
|
||||||
|
4iYXx79OmsuB7Uja9taZXaPqtkKDmv8BLS6mKLkK8nn9Pm98k82A8HLwH/JEOz7o
|
||||||
|
8k7UaaeWE6Qb0kAg+S/REFm9HQ8wG1unqdNjfKjWXBqYFUF9jqtzSwtPOixmHZoa
|
||||||
|
gvOsc0xAUwZpq45HMEWljmVTnQKBgQDM8eW7kMjpeB6nW+vxC8JS4R6wI6AmDxiH
|
||||||
|
VSpWhj9KZCHoxgC/Uj1ssbCtvdZb/uSoigN+PRpBXlu5VkjaWgyia1T0pjlIUiw9
|
||||||
|
X4m5SnLv/5UTTVlAzkV1jzCJgJCJVliO71dbPkvEw2jP6BPunCUsKwLg35HxqgGT
|
||||||
|
jThoXWC6bwKBgAntVOrtmPhMVXtKhr9PV4ST3Lxr6R2hiTcEBKkIcnb0zlHYoQDt
|
||||||
|
hX3CsGSUdPPxXNJMVNsocRDlblywCGgvkWiqgfMUWLdDHswcRJBv2ofKiUcQw3Hp
|
||||||
|
B2wdSfuuUSdpNPKteHeJ9C0PoLTJOYVdQhIJb3AoCk6ufIon2cjQdy5lAoGBAIy2
|
||||||
|
hXp71UZfgAR+m4e8ACcxhAWB4GJhOQEqW1BfCjOEfre4wyiZSa1IbztLPVOatdp2
|
||||||
|
MCHLyCwbojSlZo3tCAG4WfND8c6TBOb6orACytm3jN5c3CwftBccQkIWcKarD1DM
|
||||||
|
ShlOs20ckek1ugG5Wdhyi55kQms/w6dQbetSOainAoGACoHYphgxSoA69hwGcR8s
|
||||||
|
ObJm/0FNU0dtHaUqQxiq/kuW8NoHFV+KUTTauI7inoFoB2/NeMp5GsY0Qqgc0Gk5
|
||||||
|
J9gI4zXo2MvyEhkHUJpXdZtPmhj6OnszN3nt3npFk4T4REra7P/slf1VKwz8tsf2
|
||||||
|
kmJt3h7yaKQNL2e1yKo4f/c=
|
||||||
|
-----END PRIVATE KEY-----
|
||||||
|
";
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn signs_with_an_rsa_key_in_pss_and_sha256() {
|
||||||
|
let dir = scratch("rsa");
|
||||||
|
for (name, pem) in [("pkcs1.pem", RSA_PKCS1), ("pkcs8.pem", RSA_PKCS8)] {
|
||||||
|
let key = Key::from_files(&write(&dir, "id", "an-rsa-key\n"), &write(&dir, name, pem))
|
||||||
|
.unwrap();
|
||||||
|
let Pair::Rsa(pair) = &key.pair else {
|
||||||
|
panic!("{name}: read as Ed25519");
|
||||||
|
};
|
||||||
|
let [id, _, signature] = key.headers(1_669_149_841_234).unwrap();
|
||||||
|
assert_eq!(id.1, "an-rsa-key");
|
||||||
|
let bytes = base64::engine::general_purpose::STANDARD
|
||||||
|
.decode(&signature.1)
|
||||||
|
.unwrap();
|
||||||
|
let public = ring::signature::UnparsedPublicKey::new(
|
||||||
|
&ring::signature::RSA_PSS_2048_8192_SHA256,
|
||||||
|
pair.public().as_ref(),
|
||||||
|
);
|
||||||
|
public
|
||||||
|
.verify(b"1669149841234GET/trade-api/ws/v2", &bytes)
|
||||||
|
.unwrap();
|
||||||
|
// Not PKCS#1 v1.5, which Kalshi doesn't take.
|
||||||
|
let pkcs1v15 = ring::signature::UnparsedPublicKey::new(
|
||||||
|
&ring::signature::RSA_PKCS1_2048_8192_SHA256,
|
||||||
|
pair.public().as_ref(),
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
pkcs1v15
|
||||||
|
.verify(b"1669149841234GET/trade-api/ws/v2", &bytes)
|
||||||
|
.is_err()
|
||||||
|
);
|
||||||
|
}
|
||||||
|
std::fs::remove_dir_all(dir).unwrap();
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_missing_variable_names_both() {
|
||||||
|
let error = Key::from_variables(|_| None).err().unwrap().to_string();
|
||||||
|
assert!(
|
||||||
|
error.starts_with("KALSHI_KEY_ID_FILE is not set"),
|
||||||
|
"{error}"
|
||||||
|
);
|
||||||
|
assert!(error.contains("KALSHI_PRIVATE_KEY_FILE"), "{error}");
|
||||||
|
let only_id = |name: &str| (name == KEY_ID_FILE).then(|| "id".into());
|
||||||
|
let error = Key::from_variables(only_id).err().unwrap().to_string();
|
||||||
|
assert!(
|
||||||
|
error.starts_with("KALSHI_PRIVATE_KEY_FILE is not set"),
|
||||||
|
"{error}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn a_missing_or_wrong_file_says_which_and_never_what_it_holds() {
|
fn a_missing_or_wrong_file_says_which_and_never_what_it_holds() {
|
||||||
let dir = scratch("errors");
|
let dir = scratch("errors");
|
||||||
@@ -170,13 +331,18 @@ mod tests {
|
|||||||
"{unreadable}"
|
"{unreadable}"
|
||||||
);
|
);
|
||||||
|
|
||||||
// An RSA key in PKCS#1, as Kalshi generates its RSA keys: refused.
|
// An RSA header over what isn't an RSA key: refused, its content
|
||||||
|
// unquoted.
|
||||||
let rsa = write(
|
let rsa = write(
|
||||||
&dir,
|
&dir,
|
||||||
"rsa.pem",
|
"rsa.pem",
|
||||||
"-----BEGIN RSA PRIVATE KEY-----\nMAA=\n-----END RSA PRIVATE KEY-----\n",
|
"-----BEGIN RSA PRIVATE KEY-----\nMAA=\n-----END RSA PRIVATE KEY-----\n",
|
||||||
);
|
);
|
||||||
assert!(error(&id, &rsa).contains("not a PKCS#8 key"));
|
let refused = error(&id, &rsa);
|
||||||
|
assert!(
|
||||||
|
refused.ends_with("neither an Ed25519 nor an RSA key"),
|
||||||
|
"{refused}"
|
||||||
|
);
|
||||||
|
|
||||||
let empty = write(&dir, "empty-id", "\n");
|
let empty = write(&dir, "empty-id", "\n");
|
||||||
let key = write(&dir, "key.pem", TEST_KEY);
|
let key = write(&dir, "key.pem", TEST_KEY);
|
||||||
|
|||||||
@@ -1,6 +1,7 @@
|
|||||||
pub mod book;
|
pub mod book;
|
||||||
pub mod cause;
|
pub mod cause;
|
||||||
pub mod reconcile;
|
pub mod reconcile;
|
||||||
|
pub mod show;
|
||||||
pub mod util;
|
pub mod util;
|
||||||
|
|
||||||
mod side;
|
mod side;
|
||||||
|
|||||||
+19
-7
@@ -29,13 +29,25 @@ mod polymarket;
|
|||||||
use engineering_challenge::book::Output;
|
use engineering_challenge::book::Output;
|
||||||
|
|
||||||
/// `AlignedSbeReader::open` reads nothing else.
|
/// `AlignedSbeReader::open` reads nothing else.
|
||||||
const CHUNK: usize = 64 << 20;
|
const OUTPUT_CHUNK: usize = 64 << 20;
|
||||||
|
|
||||||
const USAGE: &str = "usage: engineering-challenge <polymarket|kalshi|kalshi-demo> [OUTPUT], or latency <polymarket|kalshi|kalshi-demo> OUTPUT";
|
const USAGE: &str = "\
|
||||||
|
usage: engineering-challenge EXCHANGE [OUTPUT]
|
||||||
|
EXCHANGE polymarket (needs no key), kalshi, or kalshi-demo (Kalshi's demo)
|
||||||
|
OUTPUT the SBE output, stdout by default; the log goes to stderr
|
||||||
|
Kalshi's key: KALSHI_KEY_ID_FILE and KALSHI_PRIVATE_KEY_FILE name the files
|
||||||
|
of its id and its private key (a PEM, Ed25519 or RSA).
|
||||||
|
SIGINT, SIGTERM or SIGHUP stops it and flushes the output.
|
||||||
|
or: engineering-challenge latency EXCHANGE OUT.csv, the latency probe
|
||||||
|
(LATENCY.md); SIGINT stops it.";
|
||||||
|
|
||||||
fn main() -> anyhow::Result<()> {
|
fn main() -> anyhow::Result<()> {
|
||||||
let mut args = std::env::args().skip(1);
|
let mut args = std::env::args().skip(1);
|
||||||
let exchange = args.next().ok_or_else(|| anyhow::anyhow!(USAGE))?;
|
let exchange = args.next().ok_or_else(|| anyhow::anyhow!(USAGE))?;
|
||||||
|
if exchange == "--help" || exchange == "-h" {
|
||||||
|
println!("{USAGE}");
|
||||||
|
return Ok(());
|
||||||
|
}
|
||||||
if exchange == "latency" {
|
if exchange == "latency" {
|
||||||
return latency::run(args);
|
return latency::run(args);
|
||||||
}
|
}
|
||||||
@@ -53,7 +65,7 @@ fn main() -> anyhow::Result<()> {
|
|||||||
output,
|
output,
|
||||||
),
|
),
|
||||||
"kalshi-demo" => collect(kalshi::Kalshi::demo(kalshi::sign::Key::from_env()?), output),
|
"kalshi-demo" => collect(kalshi::Kalshi::demo(kalshi::sign::Key::from_env()?), output),
|
||||||
_ => anyhow::bail!("unknown exchange {exchange:?}; {USAGE}"),
|
_ => anyhow::bail!("unknown exchange {exchange:?}\n{USAGE}"),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -87,12 +99,12 @@ fn collect<E: core::Exchange>(exchange: E, output: String) -> anyhow::Result<()>
|
|||||||
|
|
||||||
/// The writer thread: writes outputs in arrival order until every book
|
/// The writer thread: writes outputs in arrival order until every book
|
||||||
/// thread is gone, then flushes the last chunk.
|
/// thread is gone, then flushes the last chunk.
|
||||||
// ponytail: the chunk being filled lives only in memory until it is full, so
|
// Ceiling: the chunk being filled lives only in memory until it is full, so
|
||||||
// a hard crash loses it; flushing more often pads every flush to 64 MiB.
|
// a hard crash loses it; flushing more often pads every flush to 64 MiB.
|
||||||
// Kept as a ceiling in M4 (roadmap/04): writing to a file of our own and
|
// Accepted for redundancy's sake too (docs/project.md, Constraints): writing
|
||||||
// copying it into chunks would lift it.
|
// to a file of our own and copying it into chunks would lift it.
|
||||||
fn write(path: &str, outputs: Receiver<Output>) -> anyhow::Result<()> {
|
fn write(path: &str, outputs: Receiver<Output>) -> anyhow::Result<()> {
|
||||||
let mut writer = AlignedSbeWriter::create(path, CHUNK)?;
|
let mut writer = AlignedSbeWriter::create(path, OUTPUT_CHUNK)?;
|
||||||
for output in outputs {
|
for output in outputs {
|
||||||
match output {
|
match output {
|
||||||
Output::Snapshot(snapshot) => writer.write_msg(&snapshot)?,
|
Output::Snapshot(snapshot) => writer.write_msg(&snapshot)?,
|
||||||
|
|||||||
+76
-19
@@ -28,7 +28,7 @@ use engineering_challenge::book::{Book, Output, format_fixed, parse_fixed};
|
|||||||
/// so is a fresh book behind our stream.
|
/// so is a fresh book behind our stream.
|
||||||
const RECENT_CHANGES: usize = 128;
|
const RECENT_CHANGES: usize = 128;
|
||||||
/// How long a change may be held while another connection might still deliver
|
/// How long a change may be held while another connection might still deliver
|
||||||
/// the one before it: the second delivery lagged up to 0.7 s.
|
/// the one before it: the second delivery was up to 0.7 s late.
|
||||||
const HOLD: TimeDelta = TimeDelta::seconds(1);
|
const HOLD: TimeDelta = TimeDelta::seconds(1);
|
||||||
/// How long once no other connection can: only the rest of a batch is still to
|
/// How long once no other connection can: only the rest of a batch is still to
|
||||||
/// come, from the same connection (40 ms at worst).
|
/// come, from the same connection (40 ms at worst).
|
||||||
@@ -91,12 +91,6 @@ impl Held {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
impl RecentChange {
|
|
||||||
fn is_copy(&self, change: &Held) -> bool {
|
|
||||||
(self.hash, self.side, self.price, self.quantity) == change.key()
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// A change as applied, with the exchange's hash of the book after it.
|
/// A change as applied, with the exchange's hash of the book after it.
|
||||||
struct RecentChange {
|
struct RecentChange {
|
||||||
side: Side,
|
side: Side,
|
||||||
@@ -115,6 +109,12 @@ struct RecentChange {
|
|||||||
counted: bool,
|
counted: bool,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
impl RecentChange {
|
||||||
|
fn is_copy(&self, change: &Held) -> bool {
|
||||||
|
(self.hash, self.side, self.price, self.quantity) == change.key()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// A symbol's state for the rule.
|
/// A symbol's state for the rule.
|
||||||
pub struct Chained {
|
pub struct Chained {
|
||||||
state: State,
|
state: State,
|
||||||
@@ -524,7 +524,7 @@ impl Chain {
|
|||||||
};
|
};
|
||||||
if reconnect {
|
if reconnect {
|
||||||
self.counts.reconnect_books += 1;
|
self.counts.reconnect_books += 1;
|
||||||
// The exchange's opening books lag by minutes (docs/notes.md): one
|
// The exchange's opening books trail by minutes (docs/notes.md): one
|
||||||
// that is ours, or a state we passed, leaves ours, and the hash
|
// that is ours, or a state we passed, leaves ours, and the hash
|
||||||
// inputs, as they are.
|
// inputs, as they are.
|
||||||
let brought = symbol
|
let brought = symbol
|
||||||
@@ -537,7 +537,7 @@ impl Chain {
|
|||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
// Our chain holds up to our last change, so a book no newer than
|
// Our chain holds up to our last change, so a book no newer than
|
||||||
// it holds nothing we missed, however far back it lags: further
|
// it holds nothing we missed, however far behind it is: further
|
||||||
// than the changes we remember, too.
|
// than the changes we remember, too.
|
||||||
if exchange_timestamp <= symbol.exchange_timestamp || symbol.was_ours(&book) {
|
if exchange_timestamp <= symbol.exchange_timestamp || symbol.was_ours(&book) {
|
||||||
self.counts.books_behind += 1;
|
self.counts.books_behind += 1;
|
||||||
@@ -583,7 +583,7 @@ impl Chain {
|
|||||||
let mut book = book;
|
let mut book = book;
|
||||||
let book_time = exchange_timestamp;
|
let book_time = exchange_timestamp;
|
||||||
let mut exchange_timestamp = exchange_timestamp;
|
let mut exchange_timestamp = exchange_timestamp;
|
||||||
// The book a resync brings can lag the changes already applied, with a
|
// The book a resync brings can trail the changes already applied, with a
|
||||||
// newer timestamp or last trade price than its levels (docs/notes.md).
|
// newer timestamp or last trade price than its levels (docs/notes.md).
|
||||||
// It is behind if it is ours as it was a few changes ago: that past
|
// It is behind if it is ours as it was a few changes ago: that past
|
||||||
// state was the exchange's own, and the changes since are what it
|
// state was the exchange's own, and the changes since are what it
|
||||||
@@ -621,7 +621,8 @@ impl Chain {
|
|||||||
Some(Cause::Loss) => self.counts.losses += 1,
|
Some(Cause::Loss) => self.counts.losses += 1,
|
||||||
Some(Cause::SubscriptionGap) => self.counts.subscription_gaps += 1,
|
Some(Cause::SubscriptionGap) => self.counts.subscription_gaps += 1,
|
||||||
Some(Cause::Switch) => self.counts.switches += 1,
|
Some(Cause::Switch) => self.counts.switches += 1,
|
||||||
None => {}
|
// Not found here: the shell says it, at the Snapshot.
|
||||||
|
Some(Cause::TrackedAgain) | None => {}
|
||||||
}
|
}
|
||||||
symbol.book = book;
|
symbol.book = book;
|
||||||
symbol.exchange_timestamp = exchange_timestamp;
|
symbol.exchange_timestamp = exchange_timestamp;
|
||||||
@@ -633,11 +634,7 @@ impl Chain {
|
|||||||
// Our changes before this book no longer lead to ours: rewinding
|
// Our changes before this book no longer lead to ours: rewinding
|
||||||
// through them would make up states that never existed.
|
// through them would make up states that never existed.
|
||||||
symbol.rule.recent_changes.clear();
|
symbol.rule.recent_changes.clear();
|
||||||
let snapshot = symbol
|
effects.push(Effect::Write(symbol.snapshot(&token, received_at)));
|
||||||
.book
|
|
||||||
.snapshot(&token, received_at, exchange_timestamp);
|
|
||||||
effects.push(Effect::Write(snapshot));
|
|
||||||
symbol.last_written = symbol.last_written.max(received_at);
|
|
||||||
if let Some(cause) = cause {
|
if let Some(cause) = cause {
|
||||||
let found = if reconnect {
|
let found = if reconnect {
|
||||||
", found by a reconnect"
|
", found by a reconnect"
|
||||||
@@ -1370,7 +1367,7 @@ mod tests {
|
|||||||
(token.clone(), free)
|
(token.clone(), free)
|
||||||
}
|
}
|
||||||
|
|
||||||
// --- one connection: sprint 02's rules still hold ---
|
// --- one connection: the hash check alone ---
|
||||||
|
|
||||||
/// Split batches and trades make checks wait or fail with nothing lost:
|
/// Split batches and trades make checks wait or fail with nothing lost:
|
||||||
/// the recording must replay without a single loss.
|
/// the recording must replay without a single loss.
|
||||||
@@ -1980,6 +1977,38 @@ mod tests {
|
|||||||
assert_eq!(feed.rule.counts.rescued, rescued + 1);
|
assert_eq!(feed.rule.counts.rescued, rescued + 1);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// A market removed while a change of it waits: the book thread stops
|
||||||
|
/// waking for it.
|
||||||
|
#[test]
|
||||||
|
fn a_market_removed_while_a_change_waits_no_longer_wakes_the_book_thread() {
|
||||||
|
let (mut feed, token, book, free, millis, at) = two_connection_symbol();
|
||||||
|
let symbol = &feed.shell.symbols[&token];
|
||||||
|
let (_, after_x) = changes_frame(
|
||||||
|
&token,
|
||||||
|
symbol,
|
||||||
|
&book,
|
||||||
|
&[(Side::Buy, free, 1_000_000)],
|
||||||
|
millis,
|
||||||
|
);
|
||||||
|
let (y, _) = changes_frame(
|
||||||
|
&token,
|
||||||
|
symbol,
|
||||||
|
&after_x,
|
||||||
|
&[(Side::Buy, free + 1_000, 2_000_000)],
|
||||||
|
millis + 1,
|
||||||
|
);
|
||||||
|
assert!(feed.on_frame_from(0, at, &y).is_empty());
|
||||||
|
assert!(feed.rule.wakes_soon());
|
||||||
|
let markets = fixture_markets();
|
||||||
|
let market = markets
|
||||||
|
.iter()
|
||||||
|
.find(|market| market.tokens.contains(&token))
|
||||||
|
.unwrap();
|
||||||
|
feed.remove_market(market);
|
||||||
|
assert!(!feed.rule.waiting.contains(&token));
|
||||||
|
assert!(!feed.rule.wakes_soon());
|
||||||
|
}
|
||||||
|
|
||||||
/// A burst on a connection that missed the change before it: the held
|
/// A burst on a connection that missed the change before it: the held
|
||||||
/// changes stop at the cap, the chain breaks, and it is all quick.
|
/// changes stop at the cap, the chain breaks, and it is all quick.
|
||||||
#[test]
|
#[test]
|
||||||
@@ -2276,10 +2305,13 @@ mod tests {
|
|||||||
assert_eq!(feed.shell.quiet, 1);
|
assert_eq!(feed.shell.quiet, 1);
|
||||||
assert_eq!(feed.shell.symbols[&token].rule.state, State::Verified);
|
assert_eq!(feed.shell.symbols[&token].rule.state, State::Verified);
|
||||||
// The book the resubscribe brings back is not asked for: ignored.
|
// The book the resubscribe brings back is not asked for: ignored.
|
||||||
|
// Nor judged as a reconnect's: the subscription there is renewed.
|
||||||
|
let judged = feed.rule.counts.reconnect_books;
|
||||||
let book = feed.shell.symbols[&token].book.clone();
|
let book = feed.shell.symbols[&token].book.clone();
|
||||||
let back = fresh_book(&token, &feed.shell.symbols[&token], &book);
|
let back = fresh_book(&token, &feed.shell.symbols[&token], &book);
|
||||||
let effects = feed.on_frame_from(1, minute(2) + TimeDelta::seconds(1), &back);
|
let effects = feed.on_frame_from(1, minute(2) + TimeDelta::seconds(1), &back);
|
||||||
assert!(snapshot_symbols(&effects).is_empty());
|
assert!(snapshot_symbols(&effects).is_empty());
|
||||||
|
assert_eq!(feed.rule.counts.reconnect_books, judged);
|
||||||
|
|
||||||
// Still quiet the minute after: reported for a move, not resubscribed.
|
// Still quiet the minute after: reported for a move, not resubscribed.
|
||||||
changes(&mut feed, &token, &[0], minute(2), free, 3);
|
changes(&mut feed, &token, &[0], minute(2), free, 3);
|
||||||
@@ -2385,14 +2417,14 @@ mod tests {
|
|||||||
let effects = reconnect_with(&mut feed, &token, &ours, later);
|
let effects = reconnect_with(&mut feed, &token, &ours, later);
|
||||||
assert!(snapshot_symbols(&effects).is_empty());
|
assert!(snapshot_symbols(&effects).is_empty());
|
||||||
assert_eq!(feed.rule.counts.reconnect_books, 1);
|
assert_eq!(feed.rule.counts.reconnect_books, 1);
|
||||||
// The server's opening books lag: a state we passed is no loss.
|
// The server's opening books trail: a state we passed is no loss.
|
||||||
let behind = feed.rule.counts.books_behind;
|
let behind = feed.rule.counts.books_behind;
|
||||||
let effects = reconnect_with(&mut feed, &token, &before, later + TimeDelta::seconds(1));
|
let effects = reconnect_with(&mut feed, &token, &before, later + TimeDelta::seconds(1));
|
||||||
assert!(snapshot_symbols(&effects).is_empty());
|
assert!(snapshot_symbols(&effects).is_empty());
|
||||||
assert_eq!(feed.rule.counts.reconnect_books, 2);
|
assert_eq!(feed.rule.counts.reconnect_books, 2);
|
||||||
assert_eq!(feed.rule.counts.books_behind, behind + 1);
|
assert_eq!(feed.rule.counts.books_behind, behind + 1);
|
||||||
assert_eq!(feed.shell.symbols[&token].book, ours);
|
assert_eq!(feed.shell.symbols[&token].book, ours);
|
||||||
// It lags further back than the changes we remember: no newer than
|
// It trails further back than the changes we remember: no newer than
|
||||||
// our last change, it still holds nothing we missed.
|
// our last change, it still holds nothing we missed.
|
||||||
let symbol = feed.shell.symbols.get_mut(&token).unwrap();
|
let symbol = feed.shell.symbols.get_mut(&token).unwrap();
|
||||||
symbol.rule.recent_changes.clear();
|
symbol.rule.recent_changes.clear();
|
||||||
@@ -2719,6 +2751,31 @@ mod tests {
|
|||||||
assert_eq!(snapshot_symbols(&effects).len(), 38 * 2);
|
assert_eq!(snapshot_symbols(&effects).len(), 38 * 2);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Removed, then tracked again: its first books' Snapshots say so
|
||||||
|
/// (the shell's flag, cleared once written).
|
||||||
|
#[test]
|
||||||
|
fn a_market_tracked_again_says_so_in_its_first_snapshot() {
|
||||||
|
let frames = fixture_frames();
|
||||||
|
let markets = fixture_markets();
|
||||||
|
let mut feed = feed_of(&markets);
|
||||||
|
replay(&mut feed, &frames[..1]);
|
||||||
|
feed.remove_market(&markets[0]);
|
||||||
|
feed.add_market(&markets[0], &[0], frames[100].0);
|
||||||
|
let tokens = &markets[0].tokens;
|
||||||
|
assert!(
|
||||||
|
tokens
|
||||||
|
.iter()
|
||||||
|
.all(|token| feed.shell.symbols[token].again.is_some())
|
||||||
|
);
|
||||||
|
let effects = feed.on_frame(frames[100].0, &opening_books(&frames, &markets[0]));
|
||||||
|
assert_eq!(snapshot_symbols(&effects).len(), tokens.len());
|
||||||
|
assert!(
|
||||||
|
tokens
|
||||||
|
.iter()
|
||||||
|
.all(|token| feed.shell.symbols[token].again.is_none())
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn each_minute_reports_the_bytes_per_second_of_every_market() {
|
fn each_minute_reports_the_bytes_per_second_of_every_market() {
|
||||||
let minute = "2026-11-10T16:00:00Z".parse::<DateTime<Utc>>().unwrap();
|
let minute = "2026-11-10T16:00:00Z".parse::<DateTime<Utc>>().unwrap();
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
//! Gamma, Polymarket's market listing: which markets are live. Two fetcher
|
//! Gamma, Polymarket's market listing: which markets are live. Two fetcher
|
||||||
//! threads page it on the orchestrator's request and stream what they read;
|
//! threads page it on the orchestrator's request and send on what they
|
||||||
//! they hold no state and no timer (roadmap/03).
|
//! read; they hold no state and no timer (roadmap/03).
|
||||||
|
|
||||||
use std::time::Duration;
|
use std::time::Duration;
|
||||||
|
|
||||||
|
|||||||
@@ -15,7 +15,7 @@ use crate::core::{self, Finding, Request, Undecoded};
|
|||||||
pub const MARKET_CHANNEL_URL: &str = "wss://ws-subscriptions-clob.polymarket.com/ws/market";
|
pub const MARKET_CHANNEL_URL: &str = "wss://ws-subscriptions-clob.polymarket.com/ws/market";
|
||||||
/// Tokens per subscribe or unsubscribe message: a first message of more than
|
/// Tokens per subscribe or unsubscribe message: a first message of more than
|
||||||
/// 500 gets no opening books (docs/notes.md).
|
/// 500 gets no opening books (docs/notes.md).
|
||||||
const CHUNK: usize = 500;
|
const PER_MESSAGE: usize = 500;
|
||||||
|
|
||||||
/// A live market: its symbols and the two inputs of their book hash that the
|
/// A live market: its symbols and the two inputs of their book hash that the
|
||||||
/// websocket doesn't carry (docs/notes.md).
|
/// websocket doesn't carry (docs/notes.md).
|
||||||
@@ -78,7 +78,7 @@ impl Session {
|
|||||||
/// connection.
|
/// connection.
|
||||||
fn operations(operation: &str, tokens: &[&String]) -> Vec<String> {
|
fn operations(operation: &str, tokens: &[&String]) -> Vec<String> {
|
||||||
tokens
|
tokens
|
||||||
.chunks(CHUNK)
|
.chunks(PER_MESSAGE)
|
||||||
.map(|chunk| json!({"assets_ids": chunk, "operation": operation}).to_string())
|
.map(|chunk| json!({"assets_ids": chunk, "operation": operation}).to_string())
|
||||||
.collect()
|
.collect()
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -2,22 +2,48 @@
|
|||||||
//! Polymarket's codec.
|
//! Polymarket's codec.
|
||||||
|
|
||||||
use std::collections::BTreeSet;
|
use std::collections::BTreeSet;
|
||||||
|
use std::io::ErrorKind;
|
||||||
use std::net::{TcpListener, TcpStream};
|
use std::net::{TcpListener, TcpStream};
|
||||||
use std::sync::Arc;
|
use std::sync::Arc;
|
||||||
use std::sync::atomic::{AtomicBool, Ordering};
|
use std::sync::atomic::{AtomicBool, Ordering};
|
||||||
use std::thread;
|
use std::thread;
|
||||||
use std::time::Duration;
|
use std::time::{Duration, Instant};
|
||||||
|
|
||||||
use chrono::Utc;
|
use chrono::Utc;
|
||||||
use crossbeam::channel::{Sender, unbounded};
|
use crossbeam::channel::{Sender, unbounded};
|
||||||
use serde_json::json;
|
use serde_json::json;
|
||||||
|
use tungstenite::protocol::CloseFrame;
|
||||||
|
use tungstenite::protocol::frame::coding::CloseCode;
|
||||||
use tungstenite::{Message, WebSocket};
|
use tungstenite::{Message, WebSocket};
|
||||||
|
|
||||||
use super::{CHUNK, Polymarket, Session};
|
use super::{PER_MESSAGE, Polymarket, Session};
|
||||||
use crate::core::Event;
|
use crate::core::Event;
|
||||||
use crate::core::socket::{Command, Notice, Routed, SILENT_AFTER, apply, route, run};
|
use crate::core::socket::{Command, Notice, PING_EVERY, Routed, SILENT_AFTER, apply, route, run};
|
||||||
use crate::core::{BOOK_THREADS, Received, book_thread};
|
use crate::core::{BOOK_THREADS, Received, book_thread};
|
||||||
|
|
||||||
|
/// The next connection to `listener`, within 5 s: a client that never
|
||||||
|
/// connects (again) fails the test rather than hangs it.
|
||||||
|
fn accept(listener: &TcpListener) -> TcpStream {
|
||||||
|
listener.set_nonblocking(true).unwrap();
|
||||||
|
let deadline = Instant::now() + Duration::from_secs(5);
|
||||||
|
loop {
|
||||||
|
match listener.accept() {
|
||||||
|
Ok((stream, _)) => {
|
||||||
|
stream.set_nonblocking(false).unwrap();
|
||||||
|
stream
|
||||||
|
.set_read_timeout(Some(Duration::from_secs(5)))
|
||||||
|
.unwrap();
|
||||||
|
return stream;
|
||||||
|
}
|
||||||
|
Err(error) if error.kind() == ErrorKind::WouldBlock => {
|
||||||
|
assert!(Instant::now() < deadline, "no connection within 5 s");
|
||||||
|
thread::sleep(Duration::from_millis(10));
|
||||||
|
}
|
||||||
|
Err(error) => panic!("accept: {error}"),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// The tokens each message of one connection subscribed, after the first.
|
/// The tokens each message of one connection subscribed, after the first.
|
||||||
fn subscribed(socket: &mut WebSocket<TcpStream>, tokens: usize) -> Vec<Vec<String>> {
|
fn subscribed(socket: &mut WebSocket<TcpStream>, tokens: usize) -> Vec<Vec<String>> {
|
||||||
let first = socket.read().unwrap().into_text().unwrap();
|
let first = socket.read().unwrap().into_text().unwrap();
|
||||||
@@ -83,7 +109,7 @@ fn a_reconnect_subscribes_every_token_again_in_chunks() {
|
|||||||
let mut tokens = tokens;
|
let mut tokens = tokens;
|
||||||
tokens.sort();
|
tokens.sort();
|
||||||
for chunks in connections {
|
for chunks in connections {
|
||||||
assert!(chunks.iter().all(|chunk| chunk.len() <= CHUNK));
|
assert!(chunks.iter().all(|chunk| chunk.len() <= PER_MESSAGE));
|
||||||
let mut subscribed = chunks.concat();
|
let mut subscribed = chunks.concat();
|
||||||
subscribed.sort();
|
subscribed.sort();
|
||||||
assert_eq!(subscribed, tokens);
|
assert_eq!(subscribed, tokens);
|
||||||
@@ -102,11 +128,7 @@ fn serving_twice(
|
|||||||
let server = thread::spawn(move || {
|
let server = thread::spawn(move || {
|
||||||
(0..2)
|
(0..2)
|
||||||
.map(|_| {
|
.map(|_| {
|
||||||
let (stream, _) = listener.accept().unwrap();
|
let mut socket = tungstenite::accept(accept(&listener)).unwrap();
|
||||||
stream
|
|
||||||
.set_read_timeout(Some(Duration::from_secs(5)))
|
|
||||||
.unwrap();
|
|
||||||
let mut socket = tungstenite::accept(stream).unwrap();
|
|
||||||
let chunks = subscribed(&mut socket, total);
|
let chunks = subscribed(&mut socket, total);
|
||||||
serve(&mut socket);
|
serve(&mut socket);
|
||||||
chunks
|
chunks
|
||||||
@@ -270,3 +292,146 @@ fn commands_update_the_token_set_and_say_what_to_send() {
|
|||||||
assert!(apply(&mut session, &mut tokens, &Command::Resubscribe("a".into())).is_empty());
|
assert!(apply(&mut session, &mut tokens, &Command::Resubscribe("a".into())).is_empty());
|
||||||
assert_eq!(tokens.into_iter().collect::<Vec<_>>(), ["b"]);
|
assert_eq!(tokens.into_iter().collect::<Vec<_>>(), ["b"]);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// A local server for one connection of one token: once subscribed, `serve`
|
||||||
|
/// handles it and sends what it found on the channel returned.
|
||||||
|
fn serving_once<T: Send + 'static>(
|
||||||
|
serve: impl FnOnce(&mut WebSocket<TcpStream>, &Sender<T>) + Send + 'static,
|
||||||
|
) -> (String, crossbeam::channel::Receiver<T>) {
|
||||||
|
let listener = TcpListener::bind("127.0.0.1:0").unwrap();
|
||||||
|
let url = format!("ws://{}/ws/market", listener.local_addr().unwrap());
|
||||||
|
let (found, found_rx) = unbounded();
|
||||||
|
thread::spawn(move || {
|
||||||
|
let mut socket = tungstenite::accept(accept(&listener)).unwrap();
|
||||||
|
subscribed(&mut socket, 1);
|
||||||
|
serve(&mut socket, &found);
|
||||||
|
while socket.read().is_ok() {}
|
||||||
|
});
|
||||||
|
(url, found_rx)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Every other socket test queues its Subscribe before the connection is up.
|
||||||
|
#[test]
|
||||||
|
fn a_command_sent_on_an_open_connection_reaches_the_server() {
|
||||||
|
let (url, sent) = serving_once(|socket, sent| {
|
||||||
|
sent.send(String::new()).unwrap();
|
||||||
|
// The keepalive comes only after 10 s: the next text is the command.
|
||||||
|
let text = socket.read().unwrap().into_text().unwrap();
|
||||||
|
sent.send(text.to_string()).unwrap();
|
||||||
|
});
|
||||||
|
client(url, vec!["a".into()], SILENT_AFTER, |commands| {
|
||||||
|
// Subscribed: the connection is open.
|
||||||
|
sent.recv_timeout(Duration::from_secs(5)).unwrap();
|
||||||
|
commands.send(Command::Subscribe(vec!["b".into()])).unwrap();
|
||||||
|
let text = sent.recv_timeout(Duration::from_secs(5)).unwrap();
|
||||||
|
let message: serde_json::Value = serde_json::from_str(&text).unwrap();
|
||||||
|
assert_eq!(
|
||||||
|
message,
|
||||||
|
json!({"assets_ids": ["b"], "operation": "subscribe"})
|
||||||
|
);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_close_with_1013_tells_the_orchestrator_the_connection_is_too_slow() {
|
||||||
|
let (url, closed) = serving_once(|socket, closed| {
|
||||||
|
let frame = CloseFrame {
|
||||||
|
code: CloseCode::Again,
|
||||||
|
reason: "".into(),
|
||||||
|
};
|
||||||
|
socket.close(Some(frame)).unwrap();
|
||||||
|
closed.send(()).unwrap();
|
||||||
|
});
|
||||||
|
// As `client` does, but waiting for the notices themselves, each under
|
||||||
|
// a timeout, rather than a sleep before reading them.
|
||||||
|
let (events_tx, _events_rx) = unbounded();
|
||||||
|
let (commands_tx, commands_rx) = unbounded();
|
||||||
|
commands_tx
|
||||||
|
.send(Command::Subscribe(vec!["a".into()]))
|
||||||
|
.unwrap();
|
||||||
|
let stop = Arc::new(AtomicBool::new(false));
|
||||||
|
let client_stop = Arc::clone(&stop);
|
||||||
|
let (notices, notices_rx) = unbounded();
|
||||||
|
let client = thread::spawn(move || {
|
||||||
|
run(
|
||||||
|
&Polymarket { url },
|
||||||
|
0,
|
||||||
|
&[events_tx],
|
||||||
|
&commands_rx,
|
||||||
|
¬ices,
|
||||||
|
SILENT_AFTER,
|
||||||
|
&client_stop,
|
||||||
|
)
|
||||||
|
});
|
||||||
|
closed.recv_timeout(Duration::from_secs(5)).unwrap();
|
||||||
|
let first: Vec<Notice> = (0..3)
|
||||||
|
.map(|_| notices_rx.recv_timeout(Duration::from_secs(5)).unwrap())
|
||||||
|
.collect();
|
||||||
|
stop.store(true, Ordering::Relaxed);
|
||||||
|
client.join().unwrap();
|
||||||
|
assert_eq!(first, [Notice::Up(0), Notice::TooSlow(0), Notice::Down(0)]);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_connection_sends_its_keepalive_every_ten_seconds() {
|
||||||
|
let (url, received) = serving_once(|socket, received| {
|
||||||
|
let stream = socket.get_ref();
|
||||||
|
stream
|
||||||
|
.set_read_timeout(Some(PING_EVERY + Duration::from_secs(5)))
|
||||||
|
.unwrap();
|
||||||
|
let text = socket.read().unwrap().into_text().unwrap();
|
||||||
|
received.send(text.to_string()).unwrap();
|
||||||
|
});
|
||||||
|
client(url, vec!["a".into()], SILENT_AFTER, |_| {
|
||||||
|
let text = received
|
||||||
|
.recv_timeout(PING_EVERY + Duration::from_secs(5))
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(text, "PING");
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Only a panic ends a book thread before its connections: the run stops.
|
||||||
|
#[test]
|
||||||
|
fn a_book_thread_gone_stops_the_connection_and_the_run() {
|
||||||
|
#[derive(serde::Deserialize)]
|
||||||
|
struct Line {
|
||||||
|
text: String,
|
||||||
|
}
|
||||||
|
let path = concat!(
|
||||||
|
env!("CARGO_MANIFEST_DIR"),
|
||||||
|
"/tests/fixtures/polymarket-frames.jsonl"
|
||||||
|
);
|
||||||
|
let recording = std::fs::read_to_string(path).unwrap();
|
||||||
|
let line: Line = serde_json::from_str(recording.lines().next().unwrap()).unwrap();
|
||||||
|
let (url, _sent) = serving_once(move |socket, sent| {
|
||||||
|
socket.send(Message::text(line.text)).unwrap();
|
||||||
|
sent.send(()).unwrap();
|
||||||
|
});
|
||||||
|
let (book_threads, receivers): (Vec<_>, Vec<_>) =
|
||||||
|
(0..BOOK_THREADS).map(|_| unbounded()).unzip();
|
||||||
|
drop(receivers);
|
||||||
|
let (commands_tx, commands) = unbounded();
|
||||||
|
commands_tx
|
||||||
|
.send(Command::Subscribe(vec!["a".into()]))
|
||||||
|
.unwrap();
|
||||||
|
let (notices, _notices_rx) = unbounded();
|
||||||
|
let stop = Arc::new(AtomicBool::new(false));
|
||||||
|
let client_stop = Arc::clone(&stop);
|
||||||
|
let (ended, ended_rx) = unbounded();
|
||||||
|
thread::spawn(move || {
|
||||||
|
run(
|
||||||
|
&Polymarket { url },
|
||||||
|
0,
|
||||||
|
&book_threads,
|
||||||
|
&commands,
|
||||||
|
¬ices,
|
||||||
|
SILENT_AFTER,
|
||||||
|
&client_stop,
|
||||||
|
);
|
||||||
|
ended.send(()).unwrap();
|
||||||
|
});
|
||||||
|
ended_rx
|
||||||
|
.recv_timeout(Duration::from_secs(5))
|
||||||
|
.expect("the connection stops by itself");
|
||||||
|
assert!(stop.load(Ordering::Relaxed), "and stops the run");
|
||||||
|
}
|
||||||
+99
-6
@@ -99,10 +99,11 @@ mod tests {
|
|||||||
use crate::util::dtos::challenge_orderbook::{PriceLevelUpdateDomain, SnapshotBidsEntryDomain};
|
use crate::util::dtos::challenge_orderbook::{PriceLevelUpdateDomain, SnapshotBidsEntryDomain};
|
||||||
|
|
||||||
/// An output with a symbol's Snapshot, a Delta, then a Snapshot that
|
/// An output with a symbol's Snapshot, a Delta, then a Snapshot that
|
||||||
/// misses it; and a log line for that second Snapshot when `explained`.
|
/// misses it; and a log line with `cause` for that second Snapshot when
|
||||||
fn reconcile_with(explained: bool) -> Report {
|
/// `explained`.
|
||||||
|
fn reconcile_with(explained: bool, cause: Cause) -> Report {
|
||||||
let dir = std::env::temp_dir();
|
let dir = std::env::temp_dir();
|
||||||
let id = format!("{}-{explained}", std::process::id());
|
let id = format!("{}-{explained}-{cause}", std::process::id()).replace(' ', "-");
|
||||||
let (output, log) = (
|
let (output, log) = (
|
||||||
dir.join(format!("reconcile-{id}.sbe")),
|
dir.join(format!("reconcile-{id}.sbe")),
|
||||||
dir.join(format!("reconcile-{id}.log")),
|
dir.join(format!("reconcile-{id}.log")),
|
||||||
@@ -132,7 +133,7 @@ mod tests {
|
|||||||
.unwrap();
|
.unwrap();
|
||||||
writer.write_msg(&snapshot(at(2), 3_000_000)).unwrap();
|
writer.write_msg(&snapshot(at(2), 3_000_000)).unwrap();
|
||||||
writer.flush().unwrap();
|
writer.flush().unwrap();
|
||||||
let line = cause::line("1", at(if explained { 2 } else { 3 }), Cause::Loss, "");
|
let line = cause::line("1", at(if explained { 2 } else { 3 }), cause, "");
|
||||||
std::fs::write(&log, format!("12:00:00.000 polymarket: {line}\n")).unwrap();
|
std::fs::write(&log, format!("12:00:00.000 polymarket: {line}\n")).unwrap();
|
||||||
let report = reconcile(&output, Some(&log)).unwrap();
|
let report = reconcile(&output, Some(&log)).unwrap();
|
||||||
std::fs::remove_file(output).unwrap();
|
std::fs::remove_file(output).unwrap();
|
||||||
@@ -140,14 +141,106 @@ mod tests {
|
|||||||
report
|
report
|
||||||
}
|
}
|
||||||
|
|
||||||
|
fn at(seconds: i64) -> DateTime<Utc> {
|
||||||
|
DateTime::from_timestamp(1_794_390_000 + seconds, 0).unwrap()
|
||||||
|
}
|
||||||
|
|
||||||
|
fn snapshot(received_at: DateTime<Utc>, quantity: i64) -> SnapshotDomain {
|
||||||
|
SnapshotDomain {
|
||||||
|
received_at,
|
||||||
|
exchange_timestamp: received_at,
|
||||||
|
bids: vec![SnapshotBidsEntryDomain {
|
||||||
|
price_level: 500_000,
|
||||||
|
quantity,
|
||||||
|
}],
|
||||||
|
asks: Vec::new(),
|
||||||
|
symbol: "1".to_string(),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn delta(received_at: DateTime<Utc>, quantity: i64) -> PriceLevelUpdateDomain {
|
||||||
|
PriceLevelUpdateDomain {
|
||||||
|
backtest_timestamp: received_at,
|
||||||
|
exchange_timestamp: received_at,
|
||||||
|
price_level: 500_000,
|
||||||
|
quantity,
|
||||||
|
side: Side::Buy,
|
||||||
|
symbol: "1".to_string(),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The report on the output `write` writes, with `log` as the log;
|
||||||
|
/// `name` keeps parallel tests apart.
|
||||||
|
fn reconciled(name: &str, write: impl FnOnce(&mut AlignedSbeWriter), log: &str) -> Report {
|
||||||
|
let dir = std::env::temp_dir();
|
||||||
|
let id = format!("{}-{name}", std::process::id());
|
||||||
|
let (output, log_path) = (
|
||||||
|
dir.join(format!("reconcile-{id}.sbe")),
|
||||||
|
dir.join(format!("reconcile-{id}.log")),
|
||||||
|
);
|
||||||
|
let mut writer = AlignedSbeWriter::create(output.to_str().unwrap(), 64 << 20).unwrap();
|
||||||
|
write(&mut writer);
|
||||||
|
writer.flush().unwrap();
|
||||||
|
std::fs::write(&log_path, log).unwrap();
|
||||||
|
let report = reconcile(&output, Some(&log_path)).unwrap();
|
||||||
|
std::fs::remove_file(output).unwrap();
|
||||||
|
std::fs::remove_file(log_path).unwrap();
|
||||||
|
report
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_delta_before_its_symbols_first_snapshot_is_an_orphan() {
|
||||||
|
let report = reconciled(
|
||||||
|
"orphan",
|
||||||
|
|writer| {
|
||||||
|
writer.write_msg(&delta(at(0), 2_000_000)).unwrap();
|
||||||
|
writer.write_msg(&snapshot(at(1), 1_000_000)).unwrap();
|
||||||
|
writer.write_msg(&delta(at(2), 3_000_000)).unwrap();
|
||||||
|
},
|
||||||
|
"",
|
||||||
|
);
|
||||||
|
assert_eq!((report.deltas, report.orphan_deltas), (2, 1));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Two Snapshots of one symbol stamped alike, both mismatched: the cause
|
||||||
|
/// line explains one of them only.
|
||||||
|
#[test]
|
||||||
|
fn a_cause_line_explains_one_snapshot_only() {
|
||||||
|
let line = cause::line("1", at(2), Cause::Loss, "");
|
||||||
|
let report = reconciled(
|
||||||
|
"once",
|
||||||
|
|writer| {
|
||||||
|
writer.write_msg(&snapshot(at(0), 1_000_000)).unwrap();
|
||||||
|
writer.write_msg(&delta(at(1), 2_000_000)).unwrap();
|
||||||
|
writer.write_msg(&snapshot(at(2), 3_000_000)).unwrap();
|
||||||
|
writer.write_msg(&delta(at(2), 4_000_000)).unwrap();
|
||||||
|
writer.write_msg(&snapshot(at(2), 5_000_000)).unwrap();
|
||||||
|
},
|
||||||
|
&format!("12:00:00.000 kalshi: {line}\n"),
|
||||||
|
);
|
||||||
|
assert_eq!(report.mismatches.len(), 2);
|
||||||
|
assert_eq!(report.explained, HashMap::from([(Cause::Loss, 1)]));
|
||||||
|
assert_eq!(report.unexplained, [("1".to_string(), at(2))]);
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn a_mismatch_with_its_log_line_is_explained_and_one_without_is_not() {
|
fn a_mismatch_with_its_log_line_is_explained_and_one_without_is_not() {
|
||||||
let report = reconcile_with(true);
|
let report = reconcile_with(true, Cause::Loss);
|
||||||
assert_eq!(report.mismatches.len(), 1);
|
assert_eq!(report.mismatches.len(), 1);
|
||||||
assert_eq!(report.explained, HashMap::from([(Cause::Loss, 1)]));
|
assert_eq!(report.explained, HashMap::from([(Cause::Loss, 1)]));
|
||||||
assert!(report.unexplained.is_empty());
|
assert!(report.unexplained.is_empty());
|
||||||
let report = reconcile_with(false);
|
let report = reconcile_with(false, Cause::Loss);
|
||||||
assert_eq!(report.unexplained.len(), 1);
|
assert_eq!(report.unexplained.len(), 1);
|
||||||
assert!(report.explained.is_empty());
|
assert!(report.explained.is_empty());
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// A market tracked again: its first Snapshot since differs from the one
|
||||||
|
/// written before it left, and its log line says why.
|
||||||
|
#[test]
|
||||||
|
fn a_market_tracked_again_explains_its_first_snapshot() {
|
||||||
|
let report = reconcile_with(true, Cause::TrackedAgain);
|
||||||
|
assert_eq!(report.mismatches.len(), 1);
|
||||||
|
assert_eq!(report.explained, HashMap::from([(Cause::TrackedAgain, 1)]));
|
||||||
|
assert!(report.unexplained.is_empty());
|
||||||
|
}
|
||||||
}
|
}
|
||||||
+350
@@ -0,0 +1,350 @@
|
|||||||
|
//! A look at the output as text: one symbol's first Snapshot, the Deltas
|
||||||
|
//! after it, and the book they fold into. With no symbol given, the one with
|
||||||
|
//! the most Deltas in the file.
|
||||||
|
|
||||||
|
use std::collections::HashMap;
|
||||||
|
use std::fmt::Write;
|
||||||
|
use std::path::Path;
|
||||||
|
|
||||||
|
use chrono::{DateTime, Utc};
|
||||||
|
|
||||||
|
use crate::Side;
|
||||||
|
use crate::book::{Book, format_fixed};
|
||||||
|
use crate::util::dtos::challenge_orderbook::{AnyMessage, PriceLevelUpdateDomain, SnapshotDomain};
|
||||||
|
use crate::util::{AlignedSbeReader, OrderbookSchema};
|
||||||
|
|
||||||
|
/// The Deltas shown after the Snapshot.
|
||||||
|
const DELTAS: usize = 20;
|
||||||
|
/// The levels shown on each side of a book.
|
||||||
|
const LEVELS: usize = 5;
|
||||||
|
|
||||||
|
/// The text for `symbol` in the output at `path`, or for its busiest symbol.
|
||||||
|
pub fn show(path: impl AsRef<Path>, symbol: Option<&str>) -> anyhow::Result<String> {
|
||||||
|
let path = path.as_ref();
|
||||||
|
let mut counts: HashMap<String, (usize, usize)> = HashMap::new();
|
||||||
|
each(path, |message| {
|
||||||
|
let (symbol, is_snapshot) = match message {
|
||||||
|
Message::Snapshot(snapshot) => (snapshot.symbol, true),
|
||||||
|
Message::Delta(delta) => (delta.symbol, false),
|
||||||
|
};
|
||||||
|
let (snapshots, deltas) = counts.entry(symbol).or_default();
|
||||||
|
if is_snapshot {
|
||||||
|
*snapshots += 1;
|
||||||
|
} else {
|
||||||
|
*deltas += 1;
|
||||||
|
}
|
||||||
|
})?;
|
||||||
|
let symbol = match symbol {
|
||||||
|
Some(symbol) if counts.contains_key(symbol) => symbol.to_string(),
|
||||||
|
Some(symbol) => anyhow::bail!("{symbol} is not in {}", path.display()),
|
||||||
|
// Ties go to the smallest symbol, so the pick doesn't depend on hashing.
|
||||||
|
None => counts
|
||||||
|
.iter()
|
||||||
|
.max_by(|a, b| a.1.1.cmp(&b.1.1).then(b.0.cmp(a.0)))
|
||||||
|
.map(|(symbol, _)| symbol.clone())
|
||||||
|
.ok_or_else(|| anyhow::anyhow!("{} holds no message", path.display()))?,
|
||||||
|
};
|
||||||
|
let (snapshots, deltas) = counts[&symbol];
|
||||||
|
|
||||||
|
let mut first: Option<SnapshotDomain> = None;
|
||||||
|
let mut after: Vec<PriceLevelUpdateDomain> = Vec::new();
|
||||||
|
each(path, |message| match message {
|
||||||
|
Message::Snapshot(snapshot) if snapshot.symbol == symbol && first.is_none() => {
|
||||||
|
first = Some(snapshot)
|
||||||
|
}
|
||||||
|
Message::Delta(delta)
|
||||||
|
if delta.symbol == symbol && first.is_some() && after.len() < DELTAS =>
|
||||||
|
{
|
||||||
|
after.push(delta)
|
||||||
|
}
|
||||||
|
_ => {}
|
||||||
|
})?;
|
||||||
|
let first = first.ok_or_else(|| anyhow::anyhow!("{symbol} has no Snapshot"))?;
|
||||||
|
|
||||||
|
let mut text = String::new();
|
||||||
|
writeln!(
|
||||||
|
text,
|
||||||
|
"{symbol}: Snapshots {snapshots}, Deltas {deltas}, in {}",
|
||||||
|
path.display()
|
||||||
|
)?;
|
||||||
|
let mut book = Book {
|
||||||
|
bids: first
|
||||||
|
.bids
|
||||||
|
.iter()
|
||||||
|
.map(|level| (level.price_level, level.quantity))
|
||||||
|
.collect(),
|
||||||
|
asks: first
|
||||||
|
.asks
|
||||||
|
.iter()
|
||||||
|
.map(|level| (level.price_level, level.quantity))
|
||||||
|
.collect(),
|
||||||
|
};
|
||||||
|
writeln!(
|
||||||
|
text,
|
||||||
|
"\nIts first Snapshot, received {} (exchange time {}):",
|
||||||
|
time(first.received_at),
|
||||||
|
time(first.exchange_timestamp)
|
||||||
|
)?;
|
||||||
|
ladder(&mut text, &book)?;
|
||||||
|
writeln!(text, "\nThe {} Deltas after it:", after.len())?;
|
||||||
|
for delta in &after {
|
||||||
|
let side = match delta.side {
|
||||||
|
Side::Buy => "bid",
|
||||||
|
Side::Sell => "ask",
|
||||||
|
};
|
||||||
|
let quantity = match delta.quantity {
|
||||||
|
0 => "deleted".to_string(),
|
||||||
|
quantity => format!("now {}", format_fixed(quantity)),
|
||||||
|
};
|
||||||
|
writeln!(
|
||||||
|
text,
|
||||||
|
" {} {side} {:>8} {quantity}",
|
||||||
|
time(delta.backtest_timestamp),
|
||||||
|
format_fixed(delta.price_level)
|
||||||
|
)?;
|
||||||
|
book.set(delta.side, delta.price_level, delta.quantity);
|
||||||
|
}
|
||||||
|
writeln!(text, "\nThe book they fold into:")?;
|
||||||
|
ladder(&mut text, &book)?;
|
||||||
|
Ok(text)
|
||||||
|
}
|
||||||
|
|
||||||
|
enum Message {
|
||||||
|
Snapshot(SnapshotDomain),
|
||||||
|
Delta(PriceLevelUpdateDomain),
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Calls `f` with every message of the output at `path`, in order.
|
||||||
|
fn each(path: &Path, mut f: impl FnMut(Message)) -> anyhow::Result<()> {
|
||||||
|
let mut reader = AlignedSbeReader::<OrderbookSchema>::open(path)?;
|
||||||
|
while let Some(message) = reader.read_msg()? {
|
||||||
|
f(match message {
|
||||||
|
AnyMessage::Snapshot(decoder) => {
|
||||||
|
Message::Snapshot(SnapshotDomain::try_from_decoder(decoder)?)
|
||||||
|
}
|
||||||
|
AnyMessage::PriceLevelUpdate(decoder) => {
|
||||||
|
Message::Delta(PriceLevelUpdateDomain::try_from_decoder(decoder)?)
|
||||||
|
}
|
||||||
|
_ => anyhow::bail!("unknown message in the output"),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The best `LEVELS` asks above the best `LEVELS` bids, prices descending.
|
||||||
|
fn ladder(text: &mut String, book: &Book) -> std::fmt::Result {
|
||||||
|
let asks: Vec<_> = book.asks.iter().take(LEVELS).collect();
|
||||||
|
for (price, quantity) in asks.into_iter().rev() {
|
||||||
|
writeln!(
|
||||||
|
text,
|
||||||
|
" ask {:>8} {}",
|
||||||
|
format_fixed(*price),
|
||||||
|
format_fixed(*quantity)
|
||||||
|
)?;
|
||||||
|
}
|
||||||
|
writeln!(text, " ----")?;
|
||||||
|
for (price, quantity) in book.bids.iter().rev().take(LEVELS) {
|
||||||
|
writeln!(
|
||||||
|
text,
|
||||||
|
" bid {:>8} {}",
|
||||||
|
format_fixed(*price),
|
||||||
|
format_fixed(*quantity)
|
||||||
|
)?;
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
fn time(at: DateTime<Utc>) -> String {
|
||||||
|
at.format("%H:%M:%S%.6f").to_string()
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
use crate::util::AlignedSbeWriter;
|
||||||
|
use crate::util::dtos::challenge_orderbook::{
|
||||||
|
SnapshotAsksEntryDomain, SnapshotBidsEntryDomain,
|
||||||
|
};
|
||||||
|
|
||||||
|
const AT: i64 = 1_794_390_000;
|
||||||
|
|
||||||
|
fn at(seconds: i64) -> DateTime<Utc> {
|
||||||
|
DateTime::from_timestamp(AT + seconds, 0).unwrap()
|
||||||
|
}
|
||||||
|
|
||||||
|
fn snapshot(symbol: &str, seconds: i64) -> SnapshotDomain {
|
||||||
|
SnapshotDomain {
|
||||||
|
received_at: at(seconds),
|
||||||
|
exchange_timestamp: at(seconds),
|
||||||
|
bids: vec![
|
||||||
|
SnapshotBidsEntryDomain {
|
||||||
|
price_level: 400_000,
|
||||||
|
quantity: 10_000_000,
|
||||||
|
},
|
||||||
|
SnapshotBidsEntryDomain {
|
||||||
|
price_level: 500_000,
|
||||||
|
quantity: 20_000_000,
|
||||||
|
},
|
||||||
|
],
|
||||||
|
asks: vec![SnapshotAsksEntryDomain {
|
||||||
|
price_level: 600_000,
|
||||||
|
quantity: 30_000_000,
|
||||||
|
}],
|
||||||
|
symbol: symbol.to_string(),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn delta(
|
||||||
|
symbol: &str,
|
||||||
|
seconds: i64,
|
||||||
|
side: Side,
|
||||||
|
price: i64,
|
||||||
|
quantity: i64,
|
||||||
|
) -> PriceLevelUpdateDomain {
|
||||||
|
PriceLevelUpdateDomain {
|
||||||
|
backtest_timestamp: at(seconds),
|
||||||
|
exchange_timestamp: at(seconds),
|
||||||
|
price_level: price,
|
||||||
|
quantity,
|
||||||
|
side,
|
||||||
|
symbol: symbol.to_string(),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// "quiet" has one Delta, "busy" three: a new bid, a deleted bid, an ask
|
||||||
|
/// changed. `name` keeps parallel tests apart.
|
||||||
|
fn output(name: &str) -> std::path::PathBuf {
|
||||||
|
let path = std::env::temp_dir().join(format!("show-{}-{name}.sbe", std::process::id()));
|
||||||
|
let mut writer = AlignedSbeWriter::create(path.to_str().unwrap(), 64 << 20).unwrap();
|
||||||
|
writer.write_msg(&snapshot("quiet", 0)).unwrap();
|
||||||
|
writer.write_msg(&snapshot("busy", 0)).unwrap();
|
||||||
|
writer
|
||||||
|
.write_msg(&delta("busy", 1, Side::Buy, 550_000, 5_000_000))
|
||||||
|
.unwrap();
|
||||||
|
writer
|
||||||
|
.write_msg(&delta("quiet", 2, Side::Buy, 400_000, 1_000_000))
|
||||||
|
.unwrap();
|
||||||
|
writer
|
||||||
|
.write_msg(&delta("busy", 3, Side::Buy, 400_000, 0))
|
||||||
|
.unwrap();
|
||||||
|
writer
|
||||||
|
.write_msg(&delta("busy", 4, Side::Sell, 600_000, 7_000_000))
|
||||||
|
.unwrap();
|
||||||
|
writer.flush().unwrap();
|
||||||
|
path
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn shows_the_busiest_symbol_its_deltas_in_order_and_the_folded_book() {
|
||||||
|
let path = output("busiest");
|
||||||
|
let text = show(&path, None).unwrap();
|
||||||
|
std::fs::remove_file(&path).unwrap();
|
||||||
|
assert!(
|
||||||
|
text.starts_with("busy: Snapshots 1, Deltas 3, in"),
|
||||||
|
"{text}"
|
||||||
|
);
|
||||||
|
let deltas = text.split("The 3 Deltas after it:\n").nth(1).unwrap();
|
||||||
|
let deltas: Vec<_> = deltas.lines().take(3).map(str::trim).collect();
|
||||||
|
assert_eq!(
|
||||||
|
deltas,
|
||||||
|
[
|
||||||
|
"09:40:01.000000 bid 0.55 now 5",
|
||||||
|
"09:40:03.000000 bid 0.4 deleted",
|
||||||
|
"09:40:04.000000 ask 0.6 now 7",
|
||||||
|
]
|
||||||
|
);
|
||||||
|
let folded = text.split("The book they fold into:\n").nth(1).unwrap();
|
||||||
|
let folded: Vec<_> = folded.lines().map(str::trim).collect();
|
||||||
|
assert_eq!(
|
||||||
|
folded,
|
||||||
|
[
|
||||||
|
"ask 0.6 7",
|
||||||
|
"----",
|
||||||
|
"bid 0.55 5",
|
||||||
|
"bid 0.5 20"
|
||||||
|
]
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// "deep": a Delta before its first Snapshot, a Snapshot of 7 asks and 7
|
||||||
|
/// bids, a second Snapshot, then 25 Deltas.
|
||||||
|
#[test]
|
||||||
|
fn shows_the_first_snapshots_best_levels_and_the_first_20_deltas_after_it() {
|
||||||
|
let path = std::env::temp_dir().join(format!("show-{}-deep.sbe", std::process::id()));
|
||||||
|
let mut writer = AlignedSbeWriter::create(path.to_str().unwrap(), 64 << 20).unwrap();
|
||||||
|
writer
|
||||||
|
.write_msg(&delta("deep", 0, Side::Buy, 100_000, 1_000_000))
|
||||||
|
.unwrap();
|
||||||
|
let mut deep = snapshot("deep", 1);
|
||||||
|
deep.asks = (1..=7)
|
||||||
|
.map(|level| SnapshotAsksEntryDomain {
|
||||||
|
price_level: 500_000 + level * 10_000,
|
||||||
|
quantity: level * 1_000_000,
|
||||||
|
})
|
||||||
|
.collect();
|
||||||
|
deep.bids = (1..=7)
|
||||||
|
.map(|level| SnapshotBidsEntryDomain {
|
||||||
|
price_level: 500_000 - level * 10_000,
|
||||||
|
quantity: level * 1_000_000,
|
||||||
|
})
|
||||||
|
.collect();
|
||||||
|
writer.write_msg(&deep).unwrap();
|
||||||
|
writer.write_msg(&snapshot("deep", 2)).unwrap();
|
||||||
|
for second in 3..28 {
|
||||||
|
writer
|
||||||
|
.write_msg(&delta("deep", second, Side::Buy, 100_000, 2_000_000))
|
||||||
|
.unwrap();
|
||||||
|
}
|
||||||
|
writer.flush().unwrap();
|
||||||
|
let text = show(&path, Some("deep")).unwrap();
|
||||||
|
std::fs::remove_file(&path).unwrap();
|
||||||
|
|
||||||
|
assert!(
|
||||||
|
text.starts_with("deep: Snapshots 2, Deltas 26, in"),
|
||||||
|
"{text}"
|
||||||
|
);
|
||||||
|
let first = text.split("09:40:01.000000):\n").nth(1).unwrap();
|
||||||
|
let first: Vec<_> = first.lines().take(11).map(str::trim).collect();
|
||||||
|
assert_eq!(
|
||||||
|
first,
|
||||||
|
[
|
||||||
|
"ask 0.55 5",
|
||||||
|
"ask 0.54 4",
|
||||||
|
"ask 0.53 3",
|
||||||
|
"ask 0.52 2",
|
||||||
|
"ask 0.51 1",
|
||||||
|
"----",
|
||||||
|
"bid 0.49 1",
|
||||||
|
"bid 0.48 2",
|
||||||
|
"bid 0.47 3",
|
||||||
|
"bid 0.46 4",
|
||||||
|
"bid 0.45 5",
|
||||||
|
]
|
||||||
|
);
|
||||||
|
let deltas = text.split("The 20 Deltas after it:\n").nth(1).unwrap();
|
||||||
|
let deltas: Vec<_> = deltas.lines().take_while(|line| !line.is_empty()).collect();
|
||||||
|
assert_eq!(deltas.len(), 20, "{text}");
|
||||||
|
assert!(deltas[0].contains("09:40:03.000000"), "{text}");
|
||||||
|
assert!(deltas[19].contains("09:40:22.000000"), "{text}");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn shows_the_symbol_named() {
|
||||||
|
let path = output("named");
|
||||||
|
let text = show(&path, Some("quiet")).unwrap();
|
||||||
|
std::fs::remove_file(&path).unwrap();
|
||||||
|
assert!(
|
||||||
|
text.starts_with("quiet: Snapshots 1, Deltas 1, in"),
|
||||||
|
"{text}"
|
||||||
|
);
|
||||||
|
assert!(text.contains("The 1 Deltas after it:"), "{text}");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn an_unknown_symbol_is_an_error() {
|
||||||
|
let path = output("unknown");
|
||||||
|
let shown = show(&path, Some("absent"));
|
||||||
|
std::fs::remove_file(&path).unwrap();
|
||||||
|
assert!(shown.unwrap_err().to_string().contains("absent is not in"));
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,35 @@
|
|||||||
|
//! The collector's command line, run as a user would.
|
||||||
|
|
||||||
|
use std::process::Command;
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn help_prints_the_usage_and_exits_0() {
|
||||||
|
for flag in ["--help", "-h"] {
|
||||||
|
let output = Command::new(env!("CARGO_BIN_EXE_engineering-challenge"))
|
||||||
|
.arg(flag)
|
||||||
|
.output()
|
||||||
|
.unwrap();
|
||||||
|
let text = String::from_utf8(output.stdout).unwrap();
|
||||||
|
assert!(output.status.success(), "{flag}: {text}");
|
||||||
|
assert!(
|
||||||
|
text.starts_with("usage: engineering-challenge EXCHANGE"),
|
||||||
|
"{flag}: {text}"
|
||||||
|
);
|
||||||
|
assert!(text.contains("KALSHI_PRIVATE_KEY_FILE"), "{flag}: {text}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn an_unknown_exchange_fails_with_the_usage() {
|
||||||
|
let output = Command::new(env!("CARGO_BIN_EXE_engineering-challenge"))
|
||||||
|
.arg("kraken")
|
||||||
|
.output()
|
||||||
|
.unwrap();
|
||||||
|
let text = String::from_utf8(output.stderr).unwrap();
|
||||||
|
assert!(!output.status.success());
|
||||||
|
assert!(text.contains("unknown exchange \"kraken\""), "{text}");
|
||||||
|
assert!(
|
||||||
|
text.contains("usage: engineering-challenge EXCHANGE"),
|
||||||
|
"{text}"
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -0,0 +1,94 @@
|
|||||||
|
//! The `reconcile` binary's exit status: what `scripts/watch.sh` relies on.
|
||||||
|
|
||||||
|
use std::process::Command;
|
||||||
|
|
||||||
|
use chrono::DateTime;
|
||||||
|
use engineering_challenge::Side;
|
||||||
|
use engineering_challenge::util::AlignedSbeWriter;
|
||||||
|
use engineering_challenge::util::dtos::challenge_orderbook::{
|
||||||
|
PriceLevelUpdateDomain, SnapshotBidsEntryDomain, SnapshotDomain,
|
||||||
|
};
|
||||||
|
|
||||||
|
/// An output of a symbol's Snapshot and a Delta, after an orphan Delta of
|
||||||
|
/// another symbol if `orphan`; then, if `mismatched`, a Snapshot of the
|
||||||
|
/// symbol that the Delta doesn't lead to.
|
||||||
|
fn output(orphan: bool, mismatched: bool) -> std::path::PathBuf {
|
||||||
|
let at = DateTime::from_timestamp(1_794_390_000, 0).unwrap();
|
||||||
|
let delta = |symbol: &str| PriceLevelUpdateDomain {
|
||||||
|
backtest_timestamp: at,
|
||||||
|
exchange_timestamp: at,
|
||||||
|
price_level: 500_000,
|
||||||
|
quantity: 2_000_000,
|
||||||
|
side: Side::Buy,
|
||||||
|
symbol: symbol.to_string(),
|
||||||
|
};
|
||||||
|
let path = std::env::temp_dir().join(format!(
|
||||||
|
"reconcile-exit-{}-{orphan}-{mismatched}.sbe",
|
||||||
|
std::process::id()
|
||||||
|
));
|
||||||
|
let mut writer = AlignedSbeWriter::create(path.to_str().unwrap(), 64 << 20).unwrap();
|
||||||
|
if orphan {
|
||||||
|
writer.write_msg(&delta("2")).unwrap();
|
||||||
|
}
|
||||||
|
writer
|
||||||
|
.write_msg(&SnapshotDomain {
|
||||||
|
received_at: at,
|
||||||
|
exchange_timestamp: at,
|
||||||
|
bids: vec![SnapshotBidsEntryDomain {
|
||||||
|
price_level: 500_000,
|
||||||
|
quantity: 1_000_000,
|
||||||
|
}],
|
||||||
|
asks: Vec::new(),
|
||||||
|
symbol: "1".to_string(),
|
||||||
|
})
|
||||||
|
.unwrap();
|
||||||
|
writer.write_msg(&delta("1")).unwrap();
|
||||||
|
if mismatched {
|
||||||
|
writer
|
||||||
|
.write_msg(&SnapshotDomain {
|
||||||
|
received_at: at,
|
||||||
|
exchange_timestamp: at,
|
||||||
|
bids: vec![SnapshotBidsEntryDomain {
|
||||||
|
price_level: 500_000,
|
||||||
|
quantity: 3_000_000,
|
||||||
|
}],
|
||||||
|
asks: Vec::new(),
|
||||||
|
symbol: "1".to_string(),
|
||||||
|
})
|
||||||
|
.unwrap();
|
||||||
|
}
|
||||||
|
writer.flush().unwrap();
|
||||||
|
path
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn reconcile_fails_on_an_orphan_delta_or_an_unexplained_mismatch_and_passes_otherwise() {
|
||||||
|
let status = |orphan: bool, mismatched: bool| {
|
||||||
|
let path = output(orphan, mismatched);
|
||||||
|
let output = Command::new(env!("CARGO_BIN_EXE_reconcile"))
|
||||||
|
.arg(&path)
|
||||||
|
.output()
|
||||||
|
.unwrap();
|
||||||
|
std::fs::remove_file(&path).unwrap();
|
||||||
|
(
|
||||||
|
output.status.success(),
|
||||||
|
String::from_utf8(output.stdout).unwrap(),
|
||||||
|
)
|
||||||
|
};
|
||||||
|
let (passed, text) = status(false, false);
|
||||||
|
assert!(passed, "{text}");
|
||||||
|
assert_eq!(
|
||||||
|
text,
|
||||||
|
"1 snapshots, 1 deltas, 0 orphan deltas, 0 mismatches\n"
|
||||||
|
);
|
||||||
|
let (passed, text) = status(true, false);
|
||||||
|
assert!(!passed, "{text}");
|
||||||
|
assert!(text.contains("1 orphan deltas"), "{text}");
|
||||||
|
// A Snapshot no log line explains (no log given): it fails too.
|
||||||
|
let (passed, text) = status(false, true);
|
||||||
|
assert!(!passed, "{text}");
|
||||||
|
assert!(
|
||||||
|
text.contains("1 mismatches") && text.contains("unexplained: 1 at "),
|
||||||
|
"{text}"
|
||||||
|
);
|
||||||
|
}
|
||||||
Reference in new issue
Block a user