feat(latency): latency experiments, LATENCY.md, Kalshi's slow-connection recycle

Extension 4. `engineering-challenge latency EXCHANGE OUT.csv` opens 16
connections to the same busy markets and writes every copy of every change;
`scripts/latency/analyse.py` gives each connection's speed, whether it
persists, what keeping the fastest would gain, lateness by address, and lag
with the clock's offset. LATENCY.md writes up four runs: on Polymarket speed
persists but is worth under a millisecond, at the median only; on Kalshi's
demo keeping the fastest pair lowers p99 by 28% and 42%, the threshold
agreed before the runs.

So the collector now recycles a Kalshi connection whose mean lateness stays,
3 minutes running, in the slowest quarter and above twice everyone's, ahead
of its hourly turn. Kalshi's rule measures each duplicate's lateness; the
orchestrator judges once a minute. A duplicate's lag in Polymarket's minute
line is now its "lateness".

Roadmap milestone 7, roadmap/07.
This commit is contained in:
pascal committed 2026-11-12 15:05:25 +01:00
1 parent 1a5bbcd602
commit 515e85941d
22 files changed
+2132 -55

No files matched your search

+1
View File
@@ -1 +1,2 @@
/target/
__pycache__/
+16 -2
View File
@@ -20,10 +20,11 @@ outside this repo.
- `GLOSSARY.md`: the terms (symbol, connection, load, first one wins,
duplicate, held, rescued, follow, resync, loss, subscription gap, split,
quiet, resubscribe, silent, recycle, sequence number, clock, Snapshot,
Delta).
Delta, lateness, lag, speed).
- `src/core/`: what is the same for every exchange (ADR 0004).
`orchestrator.rs`: what is tracked, which connections carry each market,
their balancing and their recycling; `socket.rs`: a connection (read,
their balancing and their recycling (hourly, and sooner for a Kalshi
connection that stays among the slowest); `socket.rs`: a connection (read,
stamp, decode, route; reconnects when dropped, silent, lost a message or
recycled); `feed.rs`: a book thread's shell (Snapshots, moves, the quiet
watch) and the `Rule` trait; `mod.rs`: the `Exchange`, `Session`, `Market`
@@ -38,6 +39,11 @@ outside this repo.
`src/bin/reconcile.rs`: the output check, reading the log's cause lines
(`src/cause.rs`). `tests/fixtures/`: recordings of the live feeds that the
tests replay.
- `src/latency.rs` + `scripts/latency/analyse.py`: the latency probe
(`engineering-challenge latency EXCHANGE OUT.csv`, until SIGINT: one CSV
line per copy of a change on each of 16 connections) and its analysis
(speed, persistence, racing, by address, lag); `LATENCY.md` is the
writeup (roadmap/07).
- `src/util/` and `sbe_schemas/`: the template's SBE machinery, used as given so
output stays readable by the template's `AlignedSbeReader`. Our code is the
rest of `src/`.
@@ -102,6 +108,14 @@ and let the user settle it. Claude owns the data model.
- **Live data beats the docs**: Polymarket's market channel has rules its
docs don't state (`docs/notes.md`); check a protocol assumption on a
recording of the real feed (`tests/fixtures/record.py`) before relying on it.
- **The workspace can't choose what the exchange sees**: a NAT rewrites
every connection's source port (local 41000 left as 36996, then as 37086
a minute later; `curl --local-port P https://ifconfig.me/all.json`), and
there is no IPv6 route (connections to an IPv6 address fail with "Network
is unreachable"). The probe uses IPv4 addresses only.
- **Python checks run through `uvx`**: no ruff or pyflakes is installed;
`scripts/ci.sh` runs `uvx ruff@0.15.0` (config in
`scripts/latency/ruff.toml`).
## Workflow differences
Generated
+27
View File
@@ -314,6 +314,8 @@ dependencies = [
"serde_json",
"serde_yaml_ng",
"sha1",
"sntpc",
"sntpc-net-std",
"tungstenite",
"ureq",
]
@@ -657,6 +659,12 @@ dependencies = [
"syn 2.0.119",
]
[[package]]
name = "miniloop"
version = "0.4.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ad14b72c9fa5a663d069c582d576e5336995e3fafe7d0965d2bba3db1f98c40c"
[[package]]
name = "miniz_oxide"
version = "0.8.9"
@@ -998,6 +1006,25 @@ version = "1.16.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5b3dc8af474f516a851ff4bd12db780f948b9250ad37211e4eec0bccea54e01b"
[[package]]
name = "sntpc"
version = "0.11.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "294278b134cd7d06d7f22eca970aebe38707864c080f66547f9ad26327534af3"
dependencies = [
"cfg-if",
"miniloop",
]
[[package]]
name = "sntpc-net-std"
version = "1.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "335b2e533977c45a8c909c2297c659942aee18f2319aae29ee6038271d8d0131"
dependencies = [
"sntpc",
]
[[package]]
name = "stable_deref_trait"
version = "1.2.1"
+2
View File
@@ -24,6 +24,8 @@ rustls-pki-types = { version = "1", features = ["std"] }
serde = { version = "1", features = ["derive"] }
serde_json = "1"
sha1 = "0.11"
sntpc = { version = "0.11", features = ["sync", "std"] }
sntpc-net-std = "1.3"
tungstenite = { version = "0.30", features = ["rustls-tls-webpki-roots"] }
ureq = "3"
+13
View File
@@ -64,6 +64,19 @@ _Avoid_: Source, primary, standby, failover (no connection has a rank)
**Duplicate**:
A change another connection already delivered.
**Lateness**:
How long after a change's first copy a connection delivered it; 0 for the
connection that delivered first. Needs no clock.
_Avoid_: Lag, delay
**Lag**:
A change's receive time minus the exchange's timestamp for it. It carries
the local clock's offset.
_Avoid_: Latency (the general idea, not a number)
**Speed**:
A connection's median lateness over a minute.
**Verified**:
Said of a symbol whose book gives the exchange's book hash after every
change. It stops being verified when a change fits on no connection in time.
+211
View File
@@ -0,0 +1,211 @@
# Latency experiments
Extension 4 of the challenge: ways to get the exchanges' changes sooner,
tried on Polymarket and Kalshi. Every trick the brief names (connecting
first after maintenance, spamming connections until a fast one comes,
changing the TCP source port) rests on one question: **is a connection that
is fast now still fast later?** If its speed is noise from message to
message, no choice of connection pays. That is what was measured, from a
server in Switzerland, on 2026-11-12.
## Method
`engineering-challenge latency EXCHANGE OUT.csv` (`src/latency.rs`) opens
16 connections to the same 100 busy markets, spread evenly over the
websocket host's addresses, and writes one line per copy of a change: which
connection delivered it, from which address, and when (stamped right after
the socket read, as the collector does). `scripts/latency/analyse.py`
matches the copies of each change across connections.
- **Lateness**: how long after a change's first copy a connection
delivered it. It compares receive times on one machine, so it needs no
clock. A connection's **speed** is its median lateness over a minute.
- **Lag**: receive time minus the exchange's timestamp. It carries the
local clock's error, so the probe asks `time.cloudflare.com` (SNTP) for
the offset at the start and end of each run: −0.25 to +0.69 ms, with
3 to 4 ms round trips. The clock is not what limits the figures below.
- **Markets**: on Polymarket the 100 most traded over 24 hours (Gamma's
`volume24hr`), 200 tokens. Kalshi's demo reports no traded amount, so
the markets that changed most in a minute are kept: in the first run
among the listing's first 30,000 open ones, in the second among every
open one (267,691), after the listing's order had moved and the first
30,000 held one market that changed.
- **Runs**: 10 minutes each, alternated (Polymarket, Kalshi, Polymarket,
Kalshi), the server's load between 0.4 and 2.7 on 12 cores before each.
Kalshi is its demo: no production key. The demo's servers are in AWS
us-west-2 (Oregon) and production's in us-east-2 (Ohio) (AWS's published
ranges), so how far the demo's figures hold for production is unknown.
- **Agreed before the runs**: the collector changes only if, on an exchange,
speed persists (rank correlation of the connections' speeds from one
minute to the next above about 0.5, read on the run's mean) and keeping
the 2 fastest connections
of the first minute lowers the p99 of the pair's lateness (the sooner of
its two copies, as first one wins takes it) by at least 20% against a
pair at random, in both runs.
## Does a connection's speed persist?
| Run | Changes | Rank correlation, minute to minute (mean) | Speeds in a minute |
|---|---|---|---|
| Polymarket 1 | 450,096 | 0.91 to 0.99 (0.96) | 0.2 to 2.5 ms |
| Polymarket 2 | 541,933 | 0.96 to 1.00 (0.98) | 0.04 to 3.3 ms |
| Kalshi demo 1 | 540,574 | 0.47 to 0.86 (0.66) | 0 to 177 ms |
| Kalshi demo 2 | 600,046 | 0.67 to 0.91 (0.78) | 0.04 to 136 ms |
**On Polymarket, yes, almost perfectly.** A connection keeps its place for
the whole run: in the second run connection 6 was 0.04 to 0.1 ms behind the
first copy in 7 of its 9 minutes, connection 3 about 3 ms in all 9. But the
spread is a few milliseconds, and the slow moments are shared: the p99 of
lateness is 25 to 54 ms on every address alike, and the 4 fastest
connections of the first minute had a p99 barely below the others' after
it (32 against 43 ms in the first run, 21 against 22 in the second), bursts
that delay all 16 at once. Speed seems to be decided when a connection opens (four
connections to each address, fast and slow ones on all four), and it can
change mid-way: connection 11 of the first run went from 0.35 to 1 ms in
its fifth minute.
**On Kalshi's demo, partly.** Connections differ by tens of milliseconds
up to 177, and the ranks hold less well (0.66 and 0.78; two minutes of the
first run fell to 0.47 and 0.49, so the threshold is met on the mean, not
in every minute). Some connections settle at
0 for minutes (12 from its third minute in the first run): they became the
one that delivers first.
## Racing: keep the fastest
Choosing after the fact is what opening many connections and keeping the
fast ones would do, provided speed is set when a connection opens; the
persistence above says it largely is. The 2 fastest of the first minute,
against a pair at random (all 120 pairs, averaged), over the minutes
after (8, and 9 in Kalshi's second run: the first run's stop was timed from
its start, so its listing and count left 9 min 52 s of probing; the
second's came 10 minutes after its count, 10 min 2 s):
| Run | Kept pair, median / p99 | Pair at random, median / p99 | p99 lower by |
|---|---|---|---|
| Polymarket 1 | 0.20 / 8.79 ms | 0.60 / 12.43 ms | 29% |
| Polymarket 2 | 0.00 / 13.96 ms | 0.95 / 13.38 ms | −4% |
| Kalshi demo 1 | 20.15 / 233.59 ms | 28.91 / 324.56 ms | 28% |
| Kalshi demo 2 | 0.16 / 164.25 ms | 9.24 / 283.20 ms | 42% |
No change was missed by any pair. **Polymarket: racing works at the median
and not in the tail.** The kept pair is ahead of a random one by under a
millisecond every time, but the p99 gain vanished in the second run: the
tail is the bursts every connection shares. The threshold is not met, and
the collector stays as it is: two connections per market, chosen without
regard to speed. **Kalshi's demo: racing works in both.** The p99 falls
by 28% and 42%, the median from 29 to 20 ms and from 9 to 0.2 ms: there,
a connection's speed is worth choosing, and the threshold is met. Whether
production behaves alike is unknown.
## Does the address matter?
Polymarket's websocket host resolves to 4 CloudFront addresses (and 4 IPv6
ones, unreachable from this server), Kalshi demo's to 2. Lateness by
address, all copies of a run:
| Run | Medians by address | p99s by address |
|---|---|---|
| Polymarket 1 | 0.90 to 1.04 ms | 33 to 54 ms |
| Polymarket 2 | 1.08 to 1.98 ms | 25 to 36 ms |
| Kalshi demo 1 | 40.6 and 45.2 ms | 503 and 548 ms |
| Kalshi demo 2 | 16.5 and 23.9 ms | 406 and 418 ms |
The address matters much less than the connection. On Polymarket,
`143.204.55.29` had the lowest median in both runs, by 0.1 and 0.5 ms, but
every address carried both fast and slow connections, and the spread inside
an address (0.3 to 2.5 ms in the first run) is larger than between them.
On Kalshi's demo the faster of the two addresses changed between runs.
Whatever sets a connection's speed is behind the address, assumed to be
the CDN's or the load balancer's backend, chosen per connection. The collector connects only to the first address its lookup
returns (`src/core/socket.rs`); spreading over all of them would change
nothing measured here.
## Lag: how far behind the exchange
The first copy's lag, corrected by the clock's offset:
| Run | Median | p99 |
|---|---|---|
| Polymarket 1 | 13.70 ms | 20.21 ms |
| Polymarket 2 | 14.55 ms | 25.37 ms |
| Kalshi demo 1 | 94.03 ms | 285.20 ms |
| Kalshi demo 2 | 106.31 ms | 382.30 ms |
These include the exchange's own time between stamping a change and sending
it, which can't be told apart from the network from here. Kalshi's demo in
Oregon is about 80 ms further than Polymarket, which is one hop to a
CloudFront edge (where Polymarket's own servers are was not checked). On
Polymarket that is far above what racing wins (under a millisecond). On
Kalshi's demo racing wins more in the tail (90 and 119 ms at p99, as much
as the median lag), but at the median it wins 9 ms, a tenth of the lag.
## Not tried, and why
- **First connection after maintenance.** Kalshi's docs announce
maintenance on Thursdays from 3:00 to 5:00 AM ET (08:00 to 10:00 UTC in
winter); the sprint ran after one. The test: run the probe at the
window's end and an hour later, and compare the speeds of the connections
opened first. Polymarket announces no window.
- **The TCP source port.** It can't be chosen from here: a NAT between the
server and the internet rewrites every connection's port (local 41000
left as 36996, and as 37086 a minute later). Testing it needs a host with
a public address: open connections from fixed ports, reconnect from the
same ones, and see whether each port's speed returns. Given that speed
does not follow the address, a load balancer hashing on the port is the
likeliest reason it could.
- **Location.** Assumed the largest gain by far: lag is 14 ms to Polymarket
and 94 ms to Kalshi's demo from Switzerland, against a millisecond won by
racing. A server in the exchange's AWS region (us-east-2 for Kalshi
production) would remove most of the lag; not tried, it needs a second
machine.
- **IPv6.** This server has no IPv6 route. From a host with one, the
probe would keep Polymarket's 4 IPv6 addresses in its round-robin (it
leaves them out) and the address table would say whether they differ.
- **Kernel tuning** (busy polling, receive buffers, pinning threads). The
receive time is stamped at the read, the machine's load was low, and
nothing above points at the kernel. The test: the same probe run with
`SO_BUSY_POLL` set and each connection's thread pinned, alternated with
runs without, comparing the first copies' lag on an idle machine.
- **Our own added latency**, from the read to the output write: not
measured. It doesn't change when a change is stamped. The test: stamp
each output again as it is written, and report the difference.
- **Kalshi production**: no key. The probe runs on it as `latency kalshi`
with production's two key files.
## Verdict
- **Connections differ, and keep their speed**: strongly on Polymarket
(rank correlation 0.96 and 0.98), less on Kalshi's demo (0.66 and 0.78).
Nothing we could choose at connect sets it (the address doesn't): it is
drawn per connection, mostly when it opens, though it can shift mid-way
(Kalshi's connection 4 went from 75 to 3 ms over its second run).
- **On Polymarket it is worth under a millisecond.** Keeping the fastest
pair wins at the median, not in the tail, where bursts delay every
connection alike. The collector keeps its connections as they are.
- **On Kalshi's demo it is worth tens of milliseconds**, in the tail too
(p99 28% and 42% lower with the fastest pair kept). This met the
threshold agreed before the runs, so the collector now recycles a Kalshi
connection that stays slow for 3 minutes running, ahead of its hourly
turn: its replacement's speed is drawn afresh. Each duplicate tells how
late its connection was against the other's copy (`src/kalshi/feed.rs`);
once a minute the orchestrator ranks the connections by their mean
lateness, and one is slow if it is in the slowest quarter and more than
twice the mean of all (`src/core/orchestrator.rs`). The mean, not the
median: with two copies of each change, a connection first more than
half the time has a median of 0. Without the factor of two, a slowest
quarter always exists: a first version, by median alone, recycled 6
connections in 20 minutes, one a minute, connections no slower than the
rest. **In 20 minutes of the collector on the demo (270,000 markets on
120 connections, load 5.7 at the start), no connection was slow by that
rule**, so none was recycled for it; the hourly recycle went on (33), and
the output reconciled (25 mismatches, each a logged loss). The probe's 16
connections, on the 100 busiest markets, differed far more than the
collector's; assumed to be because each of the collector's shares its
markets with many others and is judged against all their copies, which
evens the means out. So the mechanism is tested but was not seen
to fire live, and its gain on the collector was not measured; how the
demo's figures carry to production, which runs elsewhere, is unknown.
- **Distance weighs more**: 14 ms of lag to Polymarket, about 100 ms to
Kalshi's demo, against a median gain from racing under 1 ms and of 9 ms.
Colocation is the lever; it was out of reach here.
+18
View File
@@ -313,6 +313,24 @@ is the demo's.
(2026-11-11, 19:29 UTC).
- **Maintenance (docs):** Thursdays 3:00 to 5:00 AM ET; connections drop.
## Latency, from the server (observed 2026-11-12)
- **The server's clock is synced**: SNTP against `time.cloudflare.com` read
−0.25 to +0.69 ms, 3 to 4 ms round trips (sprint 07's probe).
- **What the exchange sees isn't ours to choose**: a NAT rewrites every
connection's source port (not even stable for one local port), and there
is no IPv6 route; Polymarket's websocket resolves to 4 IPv6 addresses
beside its 4 IPv4 ones (CloudFront).
- **Where the hosts are** (AWS's published ranges): Kalshi production's
websocket in us-east-2, the demo's two addresses in us-west-2; lag about
14 ms to Polymarket, about 100 ms to Kalshi's demo.
- **Polymarket repeats a change on one connection**, rarely: 22 of 29,922
changes came twice on every connection in a minute.
- **Kalshi's listing order moves**: its first 30,000 open markets held the
98 busiest at 11:12 and none of them at 11:50. Pick by counting every
market, not by the listing's first pages. The demo reports
`volume_24h_fp` 0.00 (267,691 open markets, a full listing in 75 s).
## Clock accuracy bounds time accuracy
`received_at` comes from the local clock, so it carries the clock's error. On
2026-11-04 the dev machine ran ~30 ms slow (`sntp`: +30.6 ms ± 13 ms), which
+5 -2
View File
@@ -150,7 +150,8 @@ schema debate). As settled in sprints 04 and 05:
└──────┬───▲───┘ ──── recycle ────────▶ connections
Add / Move / Remove (a market and │ │ load reports, each minute
its connections), to its thread │ │ (bytes/s per market); markets
│ │ quiet twice on a connection
│ │ quiet twice on a connection;
│ │ lateness per connection (Kalshi)
┌──────▼───┴───┐
connection 0..N ──events───▶ │ book threads │ ──Snapshot | Delta──▶ writer
(read, stamp, parse, route) │ 0..15 │
@@ -160,7 +161,9 @@ schema debate). As settled in sprints 04 and 05:
- The **orchestrator** owns what is tracked (each market's two connections,
the timers) and decides: placing, moving, splitting, and recycling one
connection at a time, about hourly. It never blocks on HTTP.
connection at a time, about hourly, and sooner a Kalshi connection whose
mean lateness stays above twice everyone's (sprint 07, LATENCY.md). It
never blocks on HTTP.
- The two **fetchers** only page the exchange's market listing on request
and stream what they read.
- A **connection** reads, stamps the receive time, parses, and routes each
+22 -5
View File
@@ -77,12 +77,19 @@ abstraction is *extracted* once a second exchange exists, not designed up front.
higher than sprint 05's; Kalshi on its demo (about 245,000 markets on 110
connections), production not run for lack of a key (sprint 06's Outcome).
7. **Extension 4 — latency experiments + writeup** — *not started*
Experiment with faster connections (first-after-maintenance, connection
racing, source-port tricks) on both exchanges; document findings in a
markdown file, with the local clock's offset measured alongside (the dev
machine ran ~30 ms slow; see notes.md).
7. **Extension 4 — latency experiments + writeup** — *done (2026-11-12)*
Is a connection that is fast now still fast later? 16 connections to the
same markets, ranked by how long after a change's first copy each
delivers it (no clock needed), with the clock's offset measured by SNTP;
written up in [../LATENCY.md](../LATENCY.md).
Sprint: [../roadmap/07. latency experiments.md](../roadmap/07.%20latency%20experiments.md).
*Ends:* experiments documented.
Result: speed persists on both exchanges; keeping the fastest wins under
a millisecond on Polymarket (not in the tail), tens of milliseconds on
Kalshi's demo, where the collector now recycles a connection whose mean
lateness stays above twice everyone's (none was, in 20 live minutes).
The source port can't be chosen here (a NAT rewrites it); first after
maintenance wasn't tried.
8. **Polish + submission** — *not started*
Final review, README for the solution (the template's README still says the
@@ -138,3 +145,13 @@ abstraction is *extracted* once a second exchange exists, not designed up front.
- Telling Kalshi's book at a market's close, and a connection's book merely
ahead of the other's stream, from a real loss (sprint 06 logs all three as
a loss; the Snapshot is right either way).
- The first connection after Kalshi's maintenance (Thursdays 08:00 to
10:00 UTC in winter): the probe at the window's end against an hour later
(sprint 07 ran after one).
- The TCP source port's effect on speed, from a host with a public address
(here a NAT rewrites it).
- The slow recycle's gain on Kalshi production, measured on the collector
(sprint 07 measured racing with the probe, on the demo only).
- Judging the connections' speed at the book threads' wall-clock minute,
not on the orchestrator's own timer (a judgment near the crossing sees
partial samples; sprint 07's review).
+361
View File
@@ -0,0 +1,361 @@
# 07. Latency experiments
**Status: done (2026-11-12)**
Extension 4 of the brief: "experimentation into how to [get faster
connections] on various exchanges like Polymarket or Kalshi", written up in a
markdown file. Correctness stays the first priority, so the collector
changes only if the data clears a threshold agreed here. The question under
every trick the brief names (racing connections, the source port, being
first after maintenance) is one: **is a connection that is fast now still
fast later?** If speed is noise per message, no choice of connection pays.
Terms: [../GLOSSARY.md](../GLOSSARY.md) (lateness, lag, speed, duplicate).
## The probe
`src/latency.rs`, a subcommand of the collector (`engineering-challenge
latency <polymarket|kalshi|kalshi-demo> <out.csv>`), kept in the repo
because `LATENCY.md` cites its numbers. It is not the collector: no book,
no rule, no reconnect. A subcommand, not `src/bin/latency.rs` as first
agreed, because what it reuses (`core`, `polymarket`, `kalshi`, `log!`)
belongs to the collector's binary, not the library; moving them was
rejected as a large diff that buys nothing else.
- **Markets:** the same 100 on every connection, few enough that our own
load stays out of the measurement. Polymarket: the 100 live markets with
the highest `volume24hr` (Gamma sorts by it: `order=volume24hr&ascending=false`,
checked), all their tokens. Kalshi: the demo reports `volume_24h_fp` 0.00
(all of the first 1,000 open markets, checked), so the probe counts every
open market's changes for a minute, 30,000 per connection, and keeps the
100 with the most; it stops if fewer than 10 changed. Production would be
picked the same way, one rule for both Kalshi deployments. Supersedes the
pick by traded amount on Kalshi, because the demo has none; and then the
count over the listing's first 30,000 only, because the listing's order
moves: the 98 busiest of 11:12 were past its first 40,000 at 11:50, and
the second Kalshi run counted 1 market that changed.
- **Connections:** 16 per run, one plain thread each, tungstenite over a
`TcpStream` the probe opens to an address it chooses, round-robin over the
exchange's IPv4 addresses (the workspace has no IPv6 route: the smoke
run's connections to Polymarket's IPv6 ones failed with "Network is
unreachable"; Polymarket's websocket has 4 IPv4 ones, the Kalshi demo's 2,
production's 1, checked 2026-11-12; the collector connects only to the
first, `src/core/socket.rs:184`). It reuses `Exchange::request` (Kalshi's
signed handshake), the `Session`'s subscribe, keepalive and decode
(`src/core/mod.rs:83` to `89`), and the collector's connect with the
address given (split out of `src/core/socket.rs`'s `connect`); a refused
handshake (429) is tried again through `core::retried`, 3 tries 1 s apart. The receive time is stamped
right after the read, as in the collector. A connection that drops is
logged and stays down: a run is 10 minutes.
- **Output:** one CSV line per copy of a change, raw: connection, address,
symbol, the change's identity, receive time (µs), exchange timestamp (ms).
The analysis matches copies across connections; the probe computes
nothing.
- **A change's identity** across connections: on Polymarket the symbol and
the change's book hash, side, price and size (the rule's own copy test,
`Held::key`, `src/polymarket/feed.rs:89`); on Kalshi the market,
`clock`, side, price and delta, plus the tuple's occurrence number on its
connection, since 2 to 4 changes can share a clock, some identical
(`docs/notes.md`).
- **Clock:** an SNTP query to `time.cloudflare.com` at the start and end of
each run, logged; lag is reported raw and corrected. `sntpc` 0.11
(`sync`, `std`) with `sntpc-net-std` 1.3, tried: offset +0.5 ms, round
trip 3 ms from the server (2026-11-12).
## The analysis
`scripts/latency/analyse.py`, standard library only (`statistics.correlation`
with `method="ranked"` is Spearman, Python 3.13 here). From a run's CSV:
- each copy's lateness, a server's repeat on one connection counted once;
a change a connection never delivered while it was up (from its first
copy until it went down) counts as missed by it;
- each connection's speed per minute, and its missed count;
- **persistence:** the rank correlation of the 16 speeds between each
minute and the next; and the 4 fastest of minute 1 against the other 12
over the full minutes after it, at the median and p99 of lateness;
- **racing:** the 2 fastest of minute 1 kept, against 2 at random (averaged
over every pair of the 16), over the full minutes after it, at the
median and p99 of lateness, with each pair's missed changes; a run too
thin to compare is said so;
- **by address:** lateness per edge address;
- **lag:** per exchange, raw and corrected by the offset.
## The runs
Two 10-minute runs per exchange, alternated (Polymarket, Kalshi,
Polymarket, Kalshi), `/proc/loadavg` recorded before each and the run put
off while the load is high. The second run repeats the first, which the
adoption threshold needs. Raw CSVs and logs stay out of the repo, in
`../runs/latency/`.
## The writeup
`LATENCY.md` at the repo root, a page or two: the method (the probe,
lateness against lag, the clock's offset), one section per experiment
(question, a small table per exchange, verdict), and the ideas not tried,
each with why and how one would: first after maintenance, the source port,
location, kernel tuning, IPv6, our own added latency. Kalshi's numbers are
labelled the demo's. What was not measured is written "assumed".
## Decisions
- **Lateness, not lag, ranks connections.** Lateness compares copies of one
change on one machine and needs no clock; lag carries the clock's offset
and the exchange's own stamping. Rejected: ranking by lag.
- **Raw CSV, analysis apart.** A new question is asked of a run without
running it again; live runs are what cost time on a shared server.
Rejected: summaries computed by the probe.
- **Racing is computed from the persistence run**, not run as an
open-close loop: choosing after the fact shows the same gain as long as a
connection's speed is set when it connects, which is what persistence
tests. Rejected: a separate racing run.
- **The source port is not tested.** At least one NAT between the workspace
and the internet rewrites it: local ports 40123, 40124 and 41000 to 41003
left as 50990, 53878, 36996, 48950, 51760, 60434, and 41000 again a minute
later as 37086 (`ifconfig.me`, 2026-11-12). The probe binds no port.
Supersedes the scope's "same (address, port) pairs" run.
- **Adoption threshold**, agreed before the data: the collector changes on
an exchange only if speed persists (rank correlation above about 0.5
across the run) and the 2 picked connections' p99 lateness is at least
20% below the random pair's, in both runs. The change would then be to
recycle a connection whose speed stayed among the slowest for several
minutes (the stretch goal). Otherwise the writeup says the threshold was
not met. Met on Kalshi's demo (28% and 42%), not on Polymarket (29%, then
-4%).
- **Recycling the slow connections (Kalshi)**, agreed after the runs, for
showing the loop from data to code and as one more self-correcting
mechanism, knowing the evidence is the demo's: Kalshi's rule keeps the
first copy's receive time and connection of its last 128 positions per
symbol, forgotten when a fresh book replaces ours; a duplicate records
its lateness for its connection, and 0 for the first copy's (the later
stamp is the late one, whichever reached the book thread first); a
third copy, while a market moves, is not measured. Each book thread reports the samples at the minute
(`Report::Lateness`); Polymarket's rule reports none, so nothing changes
there. The orchestrator, once a minute, ranks the connections with at
least 100 samples and up for 10 minutes by their mean lateness; those
of the slowest quarter whose mean is more than twice the mean of every
judged copy count a slow minute, the rest start over; one slow 3
minutes in a row is recycled at once, ahead of the hourly order, at most
one a minute and under the hourly recycle's rule (one at a time, every
connection up, the last up an interval ago). Rejected: building it in
the core for both exchanges, since Polymarket's p99 doesn't follow
speed. The mean, not the median, and the factor of two, supersede the
first design (by median, the slowest quarter alone), because of the
review: a slowest quarter always exists, so equal connections were
recycled about once a minute; and with two copies of each change, a
connection first more than half the time has a median of 0, so a median
says how often it comes second, not how late, and twice the median of
medians is about 0. The mean holds both, and the tail where Kalshi's
gain was. The probe's speed (the median, GLOSSARY.md) is unchanged.
- **Kalshi on the demo.** No production key; the code is the same, only
whether the demo behaves like production is unknown.
- **`lags` becomes `lateness`** in Polymarket's rule (Kalshi's measures
none), with the log text "second copies … late" read as "duplicates'
lateness … at the median"; nothing parses that line (`src/cause.rs`,
`src/reconcile.rs`).
- **Our own added latency** (receive time to output write) is not measured:
the receive time is stamped before any queue, and the brief's extension
is about the connection.
## Tasks
- [x] Rename `lags` to `lateness` in `src/polymarket/feed.rs`, comments
and log text included (Kalshi's rule measures none); the rules' tests
pass unchanged.
- [x] The probe's change identities, for both exchanges.
- [x] Test: a Polymarket change on two connections gets the same identity;
one differing only in its hash does not.
- [x] Test: two identical Kalshi changes under one clock on one connection
get two identities, the same two as on another connection.
- [x] The probe's market pick (Polymarket by `volume24hr`, Kalshi by a
minute's count) and its connections (address round-robin, request,
subscribe, keepalive, stamping, CSV), with the SNTP offset at start
and end.
- [x] A 1-minute smoke run of the probe on each exchange: 16 connections
up, every connection's lines in the CSV, the offset logged.
- [x] `scripts/latency/analyse.py`: lateness, missed, speed per minute,
persistence, racing, by address, lag.
- [x] Test (`scripts/latency/test_analyse.py`, from `scripts/ci.sh`): on a
small CSV worked out by hand, the first copy's lateness is 0, a change
one connection lacks counts as missed by it, and the speeds, ranks
and racing figures are the hand-computed ones.
- [x] The four runs, alternated, load and offset recorded.
- [x] `LATENCY.md`.
- [x] Stretch (threshold met on Kalshi): Kalshi's rule records each
duplicate's lateness and the first copy's 0, reported at the minute.
- [x] Test: a duplicate's lateness is reported for its connection and 0
for the first copy's; a change with one copy reports nothing.
- [x] The tracker's judgment and the orchestrator's recycle of a slow
connection, logged.
- [x] Test: slowest for 3 minutes is recycled; slowest 2 minutes then not
starts over; a connection up less than 10 minutes or with too few
samples is not judged; never one while another recycle is under way.
- [x] A 20-minute Kalshi demo run (a connection is judged from its 10th
minute, recycled after 3 slow ones), reconciled, its log showing the slow
recycles.
- [x] `CLAUDE.md` (the probe, the new terms, the NAT trap) and
`docs/roadmap.md` brought up to date.
From the review of `d722446`:
- [x] A slow recycle waits, as the hourly one does, an interval after the
last connection came up (its opening books), with its test.
- [x] A slow connection is ranked by its mean lateness and recycled only
if that mean is more than twice the mean of every judged copy: equal
connections are left alone, and the hourly recycle isn't starved.
Its test (equal connections, none recycled).
- [x] Kalshi's lateness: a copy stamped before the one applied first
swaps the two (no negative lateness); the remembered positions are
cleared when a fresh book replaces ours. Tests for both.
- [x] Tests: the tracker's mean against samples that differ (a median of
0, a high mean); two book
threads' samples merged; the 128-position cap; judging refused at
599 s.
- [x] Tests: Kalshi's identity on two interleaved markets; Polymarket's
told apart by size.
- [x] The stop below 10 counted markets moved into `busiest`, with its test.
- [x] The probe reuses the Kalshi listing's paging and agent
(`src/kalshi/listing.rs`, made `pub(crate)`).
- [x] `RECENT`'s comment names its upgrade path; renames: `counting` (it
means counting is over), `SLOW_SHARE`'s doc, `struct Copy`,
`Tracker::lateness`.
- [x] The analysis: no traceback on thin runs (4 or fewer connections in
the first minute, a kept pair silent after it, one connection);
tests for the p99 against the maximum (200 values), `report()` on
the hand-worked run (racing's gain and the Missed column), racing's
missed counts, and an offset note with no value.
- [x] `LATENCY.md`: the two Kalshi runs' picks; persistence read on the
mean (minutes at 0.47 and 0.49 said); the slowest 177 ms; the
second run's length explained right (the first's stop counted from
start: 9 min 52 s of probing, the second 10 min 2 s); no "two orders
of magnitude" for Kalshi's p99; "behind the address" marked assumed;
the collector run's load 7.3; how one would try IPv6, kernel tuning
and our own added latency.
- [x] This file: minutes 2 to the end, a missed count (while up) rather
than a share, and the third copy unmeasured, in the Design.
Each test is seen to fail once (the code it protects removed, then put
back).
## Review of `d722446` (2026-11-12)
Five reviewers (spec, bugs, tests, smells, security); 2 findings dropped
(the tests were seen to fail; the memory bound is 128 positions a
symbol). Every one below was decided "fix now" (the tasks above) unless
said otherwise.
- **Bugs:** the slow recycle skipped the hourly one's settling wait; it
never stops (a slowest quarter always exists) and can starve the hourly
one; the analysis crashed on thin runs; lateness could be negative or
measured against a stale position. Later: the judgment runs on the
orchestrator's timer, not the book threads' minute (partial samples make
it more conservative, not unsafe). No: the probe ignores SIGINT during
the listing scan and never notes a silent connection (a tool run by
hand; missed counts were 0 to 5).
- **Tests:** 9 gaps verified one mutant at a time (p99 and the tracker's
median against the maximum, `report()`, samples merged, the cap, racing's
missed, the offset note, Kalshi's identity across markets, Polymarket's
size), the stop below 10 and the 599 s boundary. No: the CSV contract
across the two languages, the IPv4 filter and round-robin, the tie-break,
the loop's wiring (live runs show them).
- **Smells:** the probe's copies of the listing's paging and agent; the
`RECENT` comment; four names. No: `Rule::take_lateness` (a hook on a
trait with two implementations), the analysis's primitive tuples and
repeats, the lateness and median in several places, the probe's match
on the exchange, `latency.rs`'s size.
- **Security:** no vulnerability. No: the docs name the country and the
NAT (part of the findings), unquoted CSV fields (a TLS-authenticated
source), unauthenticated SNTP (only the corrected lag).
- **Spec:** `LATENCY.md`'s figures and claims, the two picks, the mean,
three "how"s, this file's Design. No: the CSV's notes, ruff in CI, the
slow count in the minute line, the listing retry (each serves what was
asked).
## Not in this sprint
Polish and the README (M8); the roadmap's possible features; kernel
tuning; anything needing a second machine, another region or a public
address; the first connection after Kalshi's maintenance (next window
Thursday 2026-11-19, 08:00 to 10:00 UTC), unless the sprint is still open
then.
## Done when
- The probe and the analysis are on the branch, their tests seen to fail
once.
- Both exchanges' two runs are done and analysed, load and offset recorded.
- `LATENCY.md` gives a verdict per experiment and per exchange, Kalshi
labelled the demo.
- The stretch goal is built and tested if the threshold is met; otherwise
the writeup says it wasn't.
- `scripts/ci.sh` is green.
## Outcome
Extension 4 is answered and written up in `LATENCY.md`. `engineering-challenge
latency EXCHANGE OUT.csv` opens 16 connections to the same 100 busy markets
and writes every copy of every change; `scripts/latency/analyse.py` turns a
run into speeds, persistence, racing, lateness by address and lag. Four
10-minute runs, alternated, the clock's offset under 0.7 ms throughout:
| | Persistence (mean) | Racing, p99 lower by | Lag, median |
|---|---|---|---|
| Polymarket | 0.96, 0.98 | 29%, then −4% | 14 ms |
| Kalshi demo | 0.66, 0.78 | 28%, 42% | 94 to 106 ms |
On Polymarket speed persists but is worth under a millisecond, at the
median only: the tail is bursts every connection shares. On Kalshi's demo
it met the agreed threshold, so the collector now recycles a Kalshi
connection whose mean lateness stays, 3 minutes running, in the slowest
quarter and above twice everyone's (`src/kalshi/feed.rs` measures, the
orchestrator judges). In 20 live minutes on the demo (120 connections) no
connection was slow by that rule; the output reconciled (25 mismatches,
each a logged loss).
**Diverged from the plan**, each written into this file when it happened:
the probe is a subcommand, not `src/bin/latency.rs` (what it reuses belongs
to the binary); IPv4 addresses only (no IPv6 route); no source-port run (a
NAT rewrites the port); Kalshi's pick by a minute's count, first over the
listing's first 30,000 markets, then over every open one, after the
listing's order moved and the second run counted one market; the slow
recycle by mean lateness and only past twice the mean, after the review
showed that by median, a slowest quarter alone recycled equal connections
about once a minute.
**Left out**, and where it went: first after maintenance, the source port
from a public host, the slow recycle's gain on Kalshi production, and
judging at the book threads' minute (`docs/roadmap.md`, possible
features); IPv6, kernel tuning and our own added latency (`LATENCY.md`,
not tried, each with how). Kalshi production: no key, as in sprint 06.
**Assumed**, not measured: that what sets a connection's speed is the
backend behind the address; that the collector's connections differ less
than the probe's because each is judged against many partners' copies;
that location would gain the most.
**Reviewed** at `d722446`; the 11 fixes after it are the review's own
tasks, each with a test seen to fail, and had no second review.
### Retrospective
- **Went well:** one question under the brief's three tricks made the
experiments few and the writeup short. A threshold agreed before the
data kept the verdict honest: Polymarket's 29% looked like a win until
the repeat run. Raw CSVs let every new question (the pair's median, the
missed counts) be asked of runs already made. Mutating each test's code
one at a time found holes in my test data four times.
- **Went wrong:** the Kalshi pick depended on the listing's order, which
cost a 10-minute run; a pick should not rest on an order the exchange
doesn't promise. I designed the collector's slow rule on the probe's
median without asking what the median means with two copies of each
change: the review and then the fix found it, and the final rule never
fired live, so its value on the demo is unshown. Three chained shell
commands were refused whole in the worktree session, and once that left
a commit amended without its CI run; caught and redone.
- **Tested enough?** The probe's identities, the analysis and the slow
rule are, by removal. Not: the probe's network paths and the
orchestrator loop's wiring (live runs only), and the slow recycle firing
live.
- **Next time:** before reusing a statistic in a new place, work out its
value on that place's data in one line. A live check of a mechanism
should show it acting, or say plainly that it didn't.
+5
View File
@@ -8,3 +8,8 @@ cd "$(dirname "$0")/.."
cargo fmt --check
cargo clippy --all-targets -- -D warnings
cargo test
# The latency analysis (scripts/latency/): ruff pinned, run through uvx.
uvx ruff@0.15.0 format --check scripts/latency
uvx ruff@0.15.0 check scripts/latency
python3 -m unittest discover -s scripts/latency
+258
View File
@@ -0,0 +1,258 @@
"""The latency probe's CSV, analysed (roadmap/07): lateness, missed changes,
each connection's speed per minute, whether speed persists, what racing
would gain, lateness by address, and lag. Standard library only.
Usage: python3 scripts/latency/analyse.py RUN.csv
Terms (GLOSSARY.md): a copy's lateness is how long after the change's first
copy it came; a connection's speed is its median lateness over a minute;
lag is a receive time minus the exchange's timestamp.
"""
import csv
import itertools
import statistics
import sys
from pathlib import Path
MINUTE_US = 60_000_000
# The fastest of the first minute, against the others (persistence).
TOP = 4
# Connections a symbol has in the collector: first one wins (ADR 0003).
PAIR = 2
class Run:
"""A run's changes, each with every connection's first copy of it."""
def __init__(self, lines):
self.connections = {} # connection -> address
self.down = {} # connection -> when it went down, µs
self.offsets = [] # the clock's offsets, µs
# (symbol, identity) -> [exchange_ms, {connection: received_us}]
self.changes = {}
rows = []
for line in lines:
if line.startswith("#"):
self._note(line[1:].strip())
else:
rows.append(line)
for row in csv.DictReader(rows):
connection = int(row["connection"])
self.connections[connection] = row["address"]
key = (row["symbol"], row["identity"])
change = self.changes.setdefault(key, [int(row["exchange_ms"]), {}])
# A server's repeat on one connection: its first copy counts.
change[1].setdefault(connection, int(row["received_us"]))
self.up_from = {} # connection -> its first copy, µs
for _, copies in self.changes.values():
for connection, received in copies.items():
self.up_from[connection] = min(self.up_from.get(connection, received), received)
def _note(self, note):
fields = note.split(",")
if fields[0] == "down":
self.down[int(fields[1])] = int(fields[2])
elif fields[0] == "offset" and fields[2]:
self.offsets.append(int(fields[2]))
def first(self, change):
return min(change[1].values())
def minutes(self):
"""Each full minute's changes, from the run's first change."""
firsts = [(self.first(change), change) for change in self.changes.values()]
if not firsts:
return []
start = min(first for first, _ in firsts)
last = max(first for first, _ in firsts)
count = (last - start) // MINUTE_US # the last, partial, left out
minutes = [[] for _ in range(count)]
for first, change in firsts:
minute = (first - start) // MINUTE_US
if minute < count:
minutes[minute].append(change)
return minutes
def up(self, connection, at):
"""From its first copy (its subscribe done) until it went down."""
never = float("inf")
return self.up_from.get(connection, never) <= at < self.down.get(connection, never)
def lateness(self, changes, connections):
"""The lateness of the first copy among `connections`, change by
change, behind the first of all; and how many changes none of them
delivered while up."""
lateness, missed = [], 0
for change in changes:
first = self.first(change)
copies = [change[1][c] for c in connections if c in change[1]]
if copies:
lateness.append(min(copies) - first)
elif any(self.up(c, first) for c in connections):
missed += 1
return lateness, missed
def quantile(values, q):
"""As the collector's log computes it: the value at index n * q."""
ordered = sorted(values)
return ordered[min(int(len(ordered) * q), len(ordered) - 1)]
def speeds(run, minute):
"""Each connection's speed over a minute's changes, µs."""
result = {}
for connection in run.connections:
lateness, _ = run.lateness(minute, [connection])
if lateness:
result[connection] = statistics.median(lateness)
return result
def ranked(speeds_by_connection):
return sorted(speeds_by_connection, key=lambda c: (speeds_by_connection[c], c))
def persistence(run, minutes):
"""The rank correlation of the connections' speeds between each minute
and the next."""
per_minute = [speeds(run, minute) for minute in minutes]
correlations = []
for this, following in itertools.pairwise(per_minute):
both = sorted(set(this) & set(following))
if len(both) > 2:
correlations.append(
statistics.correlation(
[this[c] for c in both],
[following[c] for c in both],
method="ranked",
)
)
return correlations
def summary(lateness):
if not lateness:
return None
return statistics.median(lateness), quantile(lateness, 0.99)
def top_against_rest(run, minutes, top=TOP):
"""The `top` fastest of the first minute against the others, over the
minutes after it: (median, p99) of their copies' lateness each."""
order = ranked(speeds(run, minutes[0]))
top, rest = order[:top], order[top:]
later = [change for minute in minutes[1:] for change in minute]
def copies(connections):
lateness = []
for connection in connections:
lateness += run.lateness(later, [connection])[0]
return summary(lateness)
return copies(top), copies(rest)
def racing(run, minutes, pair=PAIR):
"""The `pair` fastest of the first minute kept, against a pair at
random (every pair, averaged), over the minutes after it: the (median,
p99) of the pair's lateness (first one wins: the sooner of its two
copies), and its missed changes. None if too few connections deliver
after the first minute to compare."""
order = ranked(speeds(run, minutes[0]))
later = [change for minute in minutes[1:] for change in minute]
picked_lateness, picked_missed = run.lateness(later, order[:pair])
pairs = [run.lateness(later, list(two)) for two in itertools.combinations(order, pair)]
random = [s for s in (summary(lateness) for lateness, _ in pairs) if s]
picked = summary(picked_lateness)
if len(order) < pair or picked is None or not random:
return None
return (
picked,
picked_missed,
(statistics.mean(m for m, _ in random), statistics.mean(p for _, p in random)),
statistics.mean(missed for _, missed in pairs),
)
def by_address(run, changes):
"""(median, p99) of lateness by address, every copy counted."""
addresses = {}
for connection, address in run.connections.items():
addresses.setdefault(address, []).extend(run.lateness(changes, [connection])[0])
return {address: summary(lateness) for address, lateness in sorted(addresses.items())}
def lag(run, changes):
"""(median, p99) of the first copies' lag, µs, raw and corrected by the
clock's mean offset (positive: the local clock is behind)."""
raw = [run.first(change) - change[0] * 1000 for change in changes]
offset = statistics.mean(run.offsets) if run.offsets else 0
return summary(raw), summary([value + offset for value in raw]), offset
def ms(micros):
return f"{micros / 1000:.2f}"
def report(run):
minutes = run.minutes()
changes = [change for minute in minutes for change in minute]
if not changes:
return f"{len(run.connections)} connections, no change over a full minute"
lines = [
f"{len(run.connections)} connections, {len(changes)} changes "
f"over {len(minutes)} full minutes",
"",
"| Connection | Address | Speed per minute (ms) | Missed |",
"|---|---|---|---|",
]
per_minute = [speeds(run, minute) for minute in minutes]
for connection, address in sorted(run.connections.items()):
row = " ".join(ms(s[connection]) if connection in s else "-" for s in per_minute)
_, missed = run.lateness(changes, [connection])
lines.append(f"| {connection} | {address} | {row} | {missed} |")
if len(minutes) >= 2:
correlations = persistence(run, minutes)
if correlations:
each = " ".join(f"{c:.2f}" for c in correlations)
mean = statistics.mean(correlations)
lines += ["", f"Persistence: rank correlation minute to minute {each}, mean {mean:.2f}"]
top, rest = top_against_rest(run, minutes)
if top and rest:
lines.append(
f"The {TOP} fastest of minute 1 after it: {ms(top[0])} ms median, "
f"{ms(top[1])} ms p99; the others: {ms(rest[0])} ms, {ms(rest[1])} ms"
)
else:
lines.append(f"The {TOP} fastest of minute 1 against the others: too few connections")
raced = racing(run, minutes)
if raced:
picked, picked_missed, random, random_missed = raced
gain = (random[1] - picked[1]) / random[1] if random[1] else 0
lines.append(
f"Racing: the {PAIR} fastest of minute 1, {ms(picked[0])} ms median, "
f"{ms(picked[1])} ms p99, {picked_missed} missed; a pair at random, "
f"{ms(random[0])} ms, {ms(random[1])} ms, {random_missed:.1f} missed; "
f"p99 {gain:.0%} lower"
)
else:
lines.append("Racing: too few connections deliver after minute 1")
lines += ["", "| Address | Median (ms) | p99 (ms) |", "|---|---|---|"]
for address, lateness in by_address(run, changes).items():
cells = f"{ms(lateness[0])} | {ms(lateness[1])}" if lateness else "- | -"
lines.append(f"| {address} | {cells} |")
raw, corrected, offset = lag(run, changes)
lines += [
"",
f"Lag of the first copies: raw {ms(raw[0])} ms median, {ms(raw[1])} ms p99; "
f"corrected by the clock's offset ({ms(offset)} ms): "
f"{ms(corrected[0])} ms, {ms(corrected[1])} ms",
]
return "\n".join(lines)
if __name__ == "__main__":
with Path(sys.argv[1]).open() as file:
print(report(Run(file)))
+15
View File
@@ -0,0 +1,15 @@
# The analysis scripts' lint: every rule, less those that don't fit two
# standalone scripts.
line-length = 100
[lint]
select = ["ALL"]
ignore = [
"D", # docstrings: written where they say something, not on every function
"ANN", # type annotations: two scripts, no type checker run
"T201", # print: the report is printed
"PLR2004", # magic values: the tests compare hand-computed numbers
"PT009", # unittest's assertEqual: the tests use unittest, no pytest
"INP001", # no package: scripts run directly
"COM812", # trailing commas: conflicts with the formatter
]
+203
View File
@@ -0,0 +1,203 @@
"""analyse.py on runs small enough to work out by hand.
Run from `scripts/ci.sh`: `python3 -m unittest discover -s scripts/latency`.
"""
import unittest
import analyse
S = 1_000_000_000_000 # the run's first change, µs
MIN = analyse.MINUTE_US
ADDRESS = {0: "A", 1: "B", 2: "A"}
def csv(copies, notes=()):
"""`copies`: (connection, change, received_us, exchange_ms)."""
lines = ["connection,address,symbol,identity,received_us,exchange_ms"]
lines += [f"# {note}" for note in notes]
for connection, change, received, exchange_ms in copies:
lines.append(f"{connection},{ADDRESS[connection]},s,{change},{received},{exchange_ms}")
return [line + "\n" for line in lines]
# Minute 1: c1 and c2 (connection 2 misses c2); minute 2: c3 and c4; c5 only
# ends minute 2, its own minute partial. Lateness in µs:
# conn 0 conn 1 conn 2
# c1 0 1000 3000
# c2 0 2000 -
# c3 1000 0 4000
# c4 3000 0 5000
RUN = csv(
[
(0, "c1", S, (S - 10_000) // 1000),
(1, "c1", S + 1000, 0),
(2, "c1", S + 3000, 0),
# A server's repeat on connection 1: its first copy counts.
(1, "c1", S + 50_000, 0),
(0, "c2", S + 10_000_000, 0),
(1, "c2", S + 10_002_000, 0),
(1, "c3", S + MIN, 0),
(0, "c3", S + MIN + 1000, 0),
(2, "c3", S + MIN + 4000, 0),
(1, "c4", S + MIN + 10_000_000, 0),
(0, "c4", S + MIN + 10_003_000, 0),
(2, "c4", S + MIN + 10_005_000, 0),
(0, "c5", S + 2 * MIN + 5_000_000, 0),
(1, "c5", S + 2 * MIN + 5_000_000, 0),
(2, "c5", S + 2 * MIN + 5_000_000, 0),
],
notes=["offset,start,400,3000", "offset,end,600,3000"],
)
class Analyse(unittest.TestCase):
def setUp(self):
self.run = analyse.Run(RUN)
self.minutes = self.run.minutes()
def test_full_minutes_only(self):
self.assertEqual([len(minute) for minute in self.minutes], [2, 2])
def test_the_first_copy_is_not_late_and_a_repeat_is_ignored(self):
lateness, _ = self.run.lateness(self.minutes[0], [0])
self.assertEqual(lateness, [0, 0])
lateness, _ = self.run.lateness(self.minutes[0], [1])
self.assertEqual(lateness, [1000, 2000])
def test_a_change_a_connection_lacks_is_missed_by_it(self):
changes = self.minutes[0] + self.minutes[1]
self.assertEqual([self.run.lateness(changes, [c])[1] for c in (0, 1, 2)], [0, 0, 1])
def test_speeds_and_their_persistence(self):
self.assertEqual(analyse.speeds(self.run, self.minutes[0]), {0: 0, 1: 1500, 2: 3000})
self.assertEqual(analyse.speeds(self.run, self.minutes[1]), {0: 2000, 1: 0, 2: 4500})
# Ranks (1, 2, 3) then (2, 1, 3).
self.assertEqual(analyse.persistence(self.run, self.minutes), [0.5])
def test_the_fastest_of_minute_one_against_the_others(self):
top, rest = analyse.top_against_rest(self.run, self.minutes, top=1)
self.assertEqual(top, (2000, 3000))
# [0, 0, 4000, 5000]: the median between, p99 the last.
self.assertEqual(rest, (2000, 5000))
def test_racing(self):
# Kept: connections 0 and 1, the sooner copy 0 for c3 and c4. At
# random: (0, 1) [0, 0], (0, 2) [1000, 3000] (median 2000, p99 3000),
# (1, 2) [0, 0].
self.assertEqual(
analyse.racing(self.run, self.minutes, pair=2), ((0, 0), 0, (2000 / 3, 1000), 0)
)
def test_by_address(self):
changes = self.minutes[0] + self.minutes[1]
self.assertEqual(
analyse.by_address(self.run, changes),
# A: [0, 0, 1000, 3000] and [3000, 4000, 5000]; B: [1000, 2000, 0, 0].
{"A": (3000, 5000), "B": (500, 2000)},
)
def test_lag_raw_and_corrected_by_the_mean_offset(self):
raw, corrected, offset = analyse.lag(self.run, self.minutes[0][:1])
self.assertEqual((raw, corrected, offset), ((10_000, 10_000), (10_500, 10_500), 500))
def test_speed_is_the_median_and_racing_keeps_the_fastest(self):
# Minute 1: lateness 0, 0, 0 on connection 0; 0, 0, 3000 on 1 (its
# median 0, its mean 1000); 2000 each on 2. Order 0, 1, 2, so the
# pair kept is (0, 1). Minute 2: connection 1 is now the slowest.
copies = []
for n, late in enumerate([0, 0, 3000]):
at = S + n * 1_000_000
copies += [(0, f"a{n}", at, 0), (1, f"a{n}", at + late, 0), (2, f"a{n}", at + 2000, 0)]
copies += [
(0, "b", S + MIN, 0),
(1, "b", S + MIN + 9000, 0),
(2, "b", S + MIN + 1000, 0),
(0, "end", S + 2 * MIN + 1, 0),
]
run = analyse.Run(csv(copies))
minutes = run.minutes()
self.assertEqual(analyse.speeds(run, minutes[0]), {0: 0, 1: 0, 2: 2000})
# Kept (0, 1): 0. At random: (0, 1) 0, (0, 2) 0, (1, 2) 1000.
picked, _, random, _ = analyse.racing(run, minutes, pair=2)
self.assertEqual((picked, random), ((0, 0), (1000 / 3, 1000 / 3)))
def test_p99_is_not_the_maximum(self):
self.assertEqual(analyse.quantile(range(200), 0.99), 198)
def test_the_report_of_the_hand_worked_run(self):
report = analyse.report(self.run)
# Connection 2's speeds, and the change it missed.
self.assertIn("| 2 | A | 3.00 4.50 | 1 |", report)
self.assertIn("rank correlation minute to minute 0.50, mean 0.50", report)
# Three connections: no 4 fastest against the others.
self.assertIn("The 4 fastest of minute 1 against the others: too few connections", report)
# Kept pair p99 0, pairs at random 1000 on average: 100% lower.
self.assertIn("a pair at random, 0.67 ms, 1.00 ms, 0.0 missed; p99 100% lower", report)
def test_racing_counts_what_a_pair_missed(self):
# Minute 1 ranks 0, 1, 2; after it, one change only connection 2
# delivers: missed by the kept pair (0, 1), by one pair in three.
run = analyse.Run(
csv(
[
(0, "a", S, 0),
(1, "a", S + 1000, 0),
(2, "a", S + 2000, 0),
(2, "b", S + MIN, 0),
(0, "c", S + MIN + 1000, 0),
(1, "c", S + MIN + 1000, 0),
(2, "c", S + MIN + 1000, 0),
(0, "end", S + 2 * MIN + 1, 0),
]
)
)
_, picked_missed, _, random_missed = analyse.racing(run, run.minutes(), pair=2)
self.assertEqual((picked_missed, random_missed), (1, 1 / 3))
def test_an_offset_the_probe_could_not_measure_is_left_out(self):
run = analyse.Run(csv([], notes=["offset,start,,", "offset,end,600,3000"]))
self.assertEqual(run.offsets, [600])
def test_thin_runs_are_reported_without_a_traceback(self):
# One connection over two minutes: nothing to compare.
alone = [(0, "a", S, 0), (0, "b", S + MIN, 0), (0, "end", S + 2 * MIN + 1, 0)]
report = analyse.report(analyse.Run(csv(alone)))
self.assertIn("Racing: too few connections deliver after minute 1", report)
# The kept pair, and every other, silent after minute 1 but one.
silent = [
(0, "a", S, 0),
(1, "a", S + 1000, 0),
(2, "a", S + 2000, 0),
(2, "end", S + 2 * MIN + 1, 0),
]
analyse.report(analyse.Run(csv(silent)))
def test_a_run_with_no_change_is_reported_as_such(self):
self.assertEqual(
analyse.report(analyse.Run(csv([]))), "0 connections, no change over a full minute"
)
def test_a_connection_is_missing_only_while_up(self):
run = analyse.Run(
csv(
[
# c1 before connection 2's first copy, c3 after it went
# down: neither is missed by it.
(0, "c1", S, 0),
(2, "c2", S + 1000, 0),
(0, "c2", S + 1000, 0),
(0, "c3", S + 3000, 0),
(0, "c4", S + MIN + 1, 0),
],
notes=[f"down,2,{S + 2000}"],
)
)
self.assertEqual(run.lateness(run.minutes()[0], [2])[1], 0)
# Without the note, c3 is missed.
run.down.clear()
self.assertEqual(run.lateness(run.minutes()[0], [2])[1], 1)
if __name__ == "__main__":
unittest.main()
+17
View File
@@ -48,8 +48,15 @@ pub enum Report {
market: String,
connection: usize,
},
/// Each connection's lateness samples over the minute, µs (roadmap/07):
/// the orchestrator recycles a connection that stays among the slowest.
Lateness(Lateness),
}
/// Connection → its copies' lateness this minute, µs: how long after the
/// first copy of a change it delivered its own (0 for the first).
pub type Lateness = HashMap<usize, Vec<i64>>;
/// Each market's load (market id → bytes per second on one connection),
/// every market of the book thread measured for at least [`MIN_MEASURED`].
pub type Loads = HashMap<String, u64>;
@@ -148,6 +155,12 @@ pub trait Rule: Send + 'static {
/// Logs the minute line, the shell's figures in it.
fn log_minute(&mut self, minute: &Minute);
/// The connections' lateness since the last minute, for the orchestrator;
/// none from a rule that doesn't judge its connections' speed.
fn take_lateness(&mut self) -> Lateness {
Lateness::new()
}
}
/// The shell's figures for the minute line.
@@ -622,6 +635,10 @@ impl<R: Rule> Feed<R> {
}
}
let mut effects = vec![Effect::Report(Report::Loads(loads))];
let lateness = self.rule.take_lateness();
if !lateness.is_empty() {
effects.push(Effect::Report(Report::Lateness(lateness)));
}
let Some(received_at) = shell.last_received_at else {
return effects;
};
+237 -5
View File
@@ -14,7 +14,7 @@ use chrono::{DateTime, TimeDelta, Utc};
use crossbeam::channel::{Sender, select, unbounded};
use engineering_challenge::book::Output;
use super::feed::{self, BookCommand, Feed, Loads, Report};
use super::feed::{self, BookCommand, Feed, Lateness, Loads, Report};
use super::socket::{self, Command, Notice};
use super::{BOOK_THREADS, Exchange, Finding, Market, Received, Request, book_thread};
@@ -52,6 +52,22 @@ const RECENT_MARGIN: TimeDelta = TimeDelta::minutes(15);
/// Each connection is recycled about this often, one at a time, to shed what
/// accumulates on the exchange's side of it.
const RECYCLE_EVERY: Duration = Duration::from_secs(60 * 60);
/// A connection slow (among the slowest quarter by mean lateness, and more
/// than [`SLOW_FACTOR`] times the mean of every copy judged) this many
/// minutes in a row is recycled ahead of its turn: the next one is drawn
/// afresh. Kalshi's demo kept the 2 fastest of 16 connections 28% and 42%
/// lower at p99 (LATENCY.md); only Kalshi's rule reports lateness. The mean,
/// not the median: with two copies of each change, a connection first more
/// than half the time has a median of 0 (roadmap/07).
const SLOW_MINUTES: u32 = 3;
/// The slowest quarter is looked at: 1 in this many connections judged.
const SLOW_SHARE: usize = 4;
/// Equal connections are left alone: a slowest quarter always exists.
const SLOW_FACTOR: i64 = 2;
/// A connection is judged once up this long, past its opening books, and
/// with this many samples in the minute.
const JUDGED_AFTER: Duration = Duration::from_secs(10 * 60);
const JUDGED_SAMPLES: usize = 100;
/// So SIGINT is seen while no timer is due.
const WAKE_EVERY: Duration = Duration::from_secs(1);
@@ -98,7 +114,7 @@ pub fn run<E: Exchange>(
let mut queue: VecDeque<BookCommand<E::Market>> = VecDeque::new();
let mut next_deal = Instant::now();
let (mut added, mut removed, mut moved, mut splits) = (0, 0, 0, 0);
let (mut recycled, mut quiet_moves, mut silent) = (0, 0, 0);
let (mut recycled, mut slow_recycled, mut quiet_moves, mut silent) = (0, 0, 0, 0);
// The minute checks cover the first scan from its start.
let mut recent_since = Utc::now() - RECENT_MARGIN;
let mut recent_asked_at: Option<DateTime<Utc>> = None;
@@ -139,17 +155,27 @@ pub fn run<E: Exchange>(
let moves = tracker.rebalance(now);
moved += moves.len();
commands.extend(moves);
if let Some((connection, mean)) = tracker.judge_speeds(now) {
log!(
"{name}: orchestrator: connection {connection} among the slowest for \
{SLOW_MINUTES} minutes (mean lateness {:.1} ms), recycled",
mean as f64 / 1_000.0
);
threads.recycle(connection);
recycled += 1;
slow_recycled += 1;
}
let busiest = tracker.connections.iter().map(|one| one.load).max();
log!(
"{name}: orchestrator: {} markets on {} connections; {added} added, {removed} \
removed, {moved} moved ({quiet_moves} quiet), {splits} splits, {recycled} recycled, \
{silent} silent this minute; busiest connection at {} bytes/s",
removed, {moved} moved ({quiet_moves} quiet), {splits} splits, {recycled} recycled \
({slow_recycled} slow), {silent} silent this minute; busiest connection at {} bytes/s",
tracker.markets.len(),
tracker.connections.len(),
busiest.unwrap_or(0)
);
(added, removed, moved, splits) = (0, 0, 0, 0);
(recycled, quiet_moves, silent) = (0, 0, 0);
(recycled, slow_recycled, quiet_moves, silent) = (0, 0, 0, 0);
}
// Recycling starts once the startup deal is sent.
if first_scan.is_none() && queue.is_empty() {
@@ -237,6 +263,7 @@ pub fn run<E: Exchange>(
},
recv(reports) -> report => match report {
Ok(Report::Loads(loads)) => tracker.report(&loads),
Ok(Report::Lateness(samples)) => tracker.add_lateness(samples),
Ok(Report::Quiet { market, connection }) => {
if let Some(command) = tracker.quiet(&market, connection, Instant::now()) {
log!(
@@ -418,6 +445,8 @@ struct ConnectionState {
up: bool,
/// When it last came up: the one longest ago is recycled next.
connected_at: Option<Instant>,
/// Minutes in a row among the slowest.
slow_minutes: u32,
}
/// Which connections carry each market and what it weighs, and which markets
@@ -435,6 +464,8 @@ struct Tracker<M> {
next_recycle: Option<Instant>,
/// The connection recycled and not yet up again.
recycling: Option<usize>,
/// The book threads' lateness samples since the last judgment.
lateness: Lateness,
}
impl<M: Market> Tracker<M> {
@@ -447,6 +478,7 @@ impl<M: Market> Tracker<M> {
random: Random(seed | 1),
next_recycle: None,
recycling: None,
lateness: Lateness::new(),
}
}
@@ -725,6 +757,81 @@ impl<M: Market> Tracker<M> {
Some(oldest)
}
/// A book thread's lateness samples, by connection.
fn add_lateness(&mut self, samples: Lateness) {
for (connection, samples) in samples {
self.lateness.entry(connection).or_default().extend(samples);
}
}
/// Once a minute: ranks the connections judged by their mean lateness
/// since the last time; those of the slowest 1 in [`SLOW_SHARE`] more
/// than [`SLOW_FACTOR`] times the mean of every judged copy count a slow
/// minute, the others start over. The slowest of those slow for
/// [`SLOW_MINUTES`] is recycled now, with its mean (µs), under
/// [`Tracker::recycle`]'s
/// rule: one at a time, every connection up, none up for less than an
/// interval.
fn judge_speeds(&mut self, now: Instant) -> Option<(usize, i64)> {
let samples = std::mem::take(&mut self.lateness);
let speeds: Vec<(usize, i64, i64)> = samples
.into_iter()
.filter(|(connection, samples)| {
samples.len() >= JUDGED_SAMPLES
&& self.connections.get(*connection).is_some_and(|state| {
state.up
&& state
.connected_at
.is_some_and(|at| now.duration_since(at) >= JUDGED_AFTER)
})
})
.map(|(connection, samples)| {
let sum: i64 = samples.iter().sum();
(connection, sum, samples.len() as i64)
})
.collect();
let (sum, count) = speeds
.iter()
.fold((0, 0), |(sum, count), &(_, s, n)| (sum + s, count + n));
let overall = sum.checked_div(count).unwrap_or(0);
let mut speeds: Vec<(usize, i64)> = speeds
.into_iter()
.map(|(connection, sum, count)| (connection, sum / count))
.collect();
// The slowest first; ties by number, so a judgment doesn't depend on
// the map's order.
speeds.sort_by_key(|&(connection, mean)| (Reverse(mean), connection));
let quarter = speeds.len() / SLOW_SHARE;
let slow: Vec<(usize, i64)> = speeds[..quarter]
.iter()
.copied()
.filter(|&(_, mean)| mean > SLOW_FACTOR * overall)
.collect();
for (number, state) in self.connections.iter_mut().enumerate() {
state.slow_minutes = if slow.iter().any(|&(connection, _)| connection == number) {
state.slow_minutes + 1
} else {
0
};
}
// As for the hourly recycle: one at a time, every connection up, and
// the last to come up past its opening books.
let settled = self
.connections
.iter()
.filter_map(|state| state.connected_at)
.max()
.is_none_or(|last| now.duration_since(last) >= self.recycle_interval());
if self.recycling.is_some() || self.connections.iter().any(|state| !state.up) || !settled {
return None;
}
let &(connection, speed) = slow
.iter()
.find(|&&(connection, _)| self.connections[connection].slow_minutes >= SLOW_MINUTES)?;
self.recycling = Some(connection);
Some((connection, speed))
}
fn scan_started(&mut self) {
self.unseen = Some(self.markets.keys().cloned().collect());
}
@@ -1103,6 +1210,131 @@ mod tests {
assert_eq!(tracker.recycle(at(7_201)), Some(1));
}
/// Twelve connections, up from `start`.
fn twelve_up(start: Instant) -> Tracker {
let mut tracker = tracker();
tracker.connections = vec![ConnectionState::default(); 12];
for connection in 0..12 {
tracker.up(connection, start);
}
tracker
}
/// A minute's samples, `count` per connection: 1 ms each, but for those
/// `slow` names, at their ms.
fn minute(tracker: &mut Tracker, slow: &[(usize, i64)], count: usize) {
let samples = (0..12)
.map(|connection| {
let ms = slow
.iter()
.find(|&&(slow, _)| slow == connection)
.map_or(1, |&(_, ms)| ms);
(connection, vec![ms * 1_000; count])
})
.collect();
tracker.add_lateness(samples);
}
#[test]
fn a_connection_among_the_slowest_three_minutes_running_is_recycled() {
let start = Instant::now();
let at = |seconds: u64| start + Duration::from_secs(seconds);
let mut tracker = twelve_up(start);
// The slowest quarter is three of twelve. 3 and 7 are slow every
// minute, 5 for two minutes and then not.
minute(&mut tracker, &[(3, 90), (5, 50), (7, 40)], 100);
assert_eq!(tracker.judge_speeds(at(601)), None);
minute(&mut tracker, &[(3, 90), (5, 50), (7, 40)], 100);
assert_eq!(tracker.judge_speeds(at(661)), None);
// 3 and 7 both reach three minutes: one recycle a minute, the slowest.
minute(&mut tracker, &[(3, 90), (7, 40), (9, 30)], 100);
assert_eq!(tracker.judge_speeds(at(721)), Some((3, 90_000)));
assert_eq!(tracker.connections[5].slow_minutes, 0);
// While 3 recycles, 7 stays slow but isn't recycled.
minute(&mut tracker, &[(3, 90), (7, 40), (9, 30)], 100);
assert_eq!(tracker.judge_speeds(at(781)), None);
assert_eq!(tracker.connections[7].slow_minutes, 4);
tracker.down(3);
tracker.up(3, at(790));
// 3 is fresh: slow, but not judged. 7 waits, as the hourly recycle
// would, for 3's opening books: an interval (an hour / 12) after it
// came up.
minute(&mut tracker, &[(3, 90), (7, 40), (9, 30)], 100);
assert_eq!(tracker.judge_speeds(at(841)), None);
assert_eq!(tracker.connections[3].slow_minutes, 0);
minute(&mut tracker, &[(3, 90), (7, 40), (9, 30)], 100);
assert_eq!(tracker.judge_speeds(at(1_089)), None);
minute(&mut tracker, &[(3, 90), (7, 40), (9, 30)], 100);
assert_eq!(tracker.judge_speeds(at(1_090)), Some((7, 40_000)));
}
#[test]
fn the_mean_of_every_book_threads_samples_judges() {
let start = Instant::now();
let at = |seconds: u64| start + Duration::from_secs(seconds);
let mut tracker = twelve_up(start);
for seconds in [601, 661, 721] {
// Two book threads report, 50 samples each for every connection.
// Connection 3's are 0 but for 10 copies 1 s late: a median of
// 0, a mean of 100 ms; the others' 10 ms each.
for thread in 0..2 {
let mut samples: Lateness = (0..12).map(|c| (c, vec![10_000; 50])).collect();
let late = if thread == 0 { 10 } else { 0 };
let mut three = vec![1_000_000; late];
three.resize(50, 0);
samples.insert(3, three);
tracker.add_lateness(samples);
}
let judged = tracker.judge_speeds(at(seconds));
assert_eq!(judged, (seconds == 721).then_some((3, 100_000)));
}
}
#[test]
fn equal_connections_are_left_alone() {
let start = Instant::now();
let at = |seconds: u64| start + Duration::from_secs(seconds);
let mut tracker = twelve_up(start);
// A slowest quarter exists every minute (3, 7 and 9, at 15 to 20 ms
// against 10), but none is more than twice the mean of all.
for seconds in [601, 661, 721, 781] {
let mut samples: Lateness = (0..12).map(|c| (c, vec![10_000; 100])).collect();
for (connection, ms) in [(3, 20), (7, 18), (9, 15)] {
samples.insert(connection, vec![ms * 1_000; 100]);
}
tracker.add_lateness(samples);
assert_eq!(tracker.judge_speeds(at(seconds)), None);
}
assert_eq!(tracker.connections[3].slow_minutes, 0);
}
#[test]
fn a_connection_is_judged_only_up_long_enough_and_with_enough_samples() {
let start = Instant::now();
let at = |seconds: u64| start + Duration::from_secs(seconds);
// Too early: slow minutes before the tenth say nothing, to the
// second.
let mut tracker = twelve_up(start);
for seconds in [420, 480, 540, 599] {
minute(&mut tracker, &[(3, 90), (5, 50)], 100);
assert_eq!(tracker.judge_speeds(at(seconds)), None);
assert_eq!(tracker.connections[3].slow_minutes, 0);
}
// Too few samples: never judged, so the others share the slowest.
let mut tracker = twelve_up(start);
for seconds in [601, 661, 721] {
minute(&mut tracker, &[(3, 90), (5, 50)], 99);
assert_eq!(tracker.judge_speeds(at(seconds)), None);
}
// A connection down holds every recycle, as the hourly one's rule.
let mut tracker = twelve_up(start);
tracker.down(0);
for seconds in [601, 661, 721, 781] {
minute(&mut tracker, &[(3, 90), (5, 50)], 100);
assert_eq!(tracker.judge_speeds(at(seconds)), None);
}
}
#[test]
fn a_market_quiet_again_moves_off_that_connection_unless_it_moved_lately() {
let mut tracker = tracker();
+24 -13
View File
@@ -6,7 +6,7 @@
use std::collections::BTreeSet;
use std::io::ErrorKind;
use std::net::{TcpStream, ToSocketAddrs};
use std::net::{SocketAddr, TcpStream, ToSocketAddrs};
use std::sync::atomic::{AtomicBool, Ordering};
use std::thread;
use std::time::{Duration, Instant};
@@ -25,7 +25,7 @@ const CONNECT_TIMEOUT: Duration = Duration::from_secs(10);
/// so the reads and the writes below (keepalive, commands, close) share this
/// thread.
const READ_TIMEOUT: Duration = Duration::from_millis(200);
const PING_EVERY: Duration = Duration::from_secs(10);
pub(crate) const PING_EVERY: Duration = Duration::from_secs(10);
const RECONNECT_AFTER: Duration = Duration::from_secs(1);
const CLOSE_TIMEOUT: Duration = Duration::from_secs(2);
/// Nothing received this long, not even a keepalive's answer or a ping: the
@@ -56,7 +56,7 @@ pub enum Notice {
Silent(usize),
}
type Socket = WebSocket<MaybeTlsStream<TcpStream>>;
pub(crate) type Socket = WebSocket<MaybeTlsStream<TcpStream>>;
/// Runs until `stop` is set or a book thread is gone, reconnecting (and
/// subscribing its symbols again) whenever the connection drops, falls silent
@@ -183,10 +183,29 @@ fn connect<E: Exchange>(
.to_socket_addrs()?
.next()
.ok_or_else(|| anyhow::anyhow!("{host} has no address"))?;
let mut socket = open(request, address)?;
for message in session.opening() {
socket.send(Message::text(message))?;
}
if !symbols.is_empty() {
for message in session.subscribe(&symbols.iter().collect::<Vec<_>>()) {
socket.send(Message::text(message))?;
}
}
Ok(socket)
}
/// The TCP connect to `address` and the TLS + websocket handshake of
/// `request`, each bounded by [`CONNECT_TIMEOUT`]; reads then time out after
/// [`READ_TIMEOUT`].
pub(crate) fn open(
request: tungstenite::handshake::client::Request,
address: SocketAddr,
) -> anyhow::Result<Socket> {
let stream = TcpStream::connect_timeout(&address, CONNECT_TIMEOUT)?;
stream.set_read_timeout(Some(CONNECT_TIMEOUT))?;
stream.set_write_timeout(Some(CONNECT_TIMEOUT))?;
let (mut socket, _response) = tungstenite::client_tls(request, stream).map_err(|error| {
let (socket, _response) = tungstenite::client_tls(request, stream).map_err(|error| {
// A refusal's status says why: 401 for a key the exchange refuses.
match error {
tungstenite::HandshakeError::Failure(tungstenite::Error::Http(response)) => {
@@ -201,14 +220,6 @@ fn connect<E: Exchange>(
_ => anyhow::bail!("unexpected TLS stream"),
};
stream.set_read_timeout(Some(READ_TIMEOUT))?;
for message in session.opening() {
socket.send(Message::text(message))?;
}
if !symbols.is_empty() {
for message in session.subscribe(&symbols.iter().collect::<Vec<_>>()) {
socket.send(Message::text(message))?;
}
}
Ok(socket)
}
@@ -392,6 +403,6 @@ fn close<S: Session>(
/// A read timeout shows up as `WouldBlock` on macOS and `TimedOut` elsewhere;
/// tungstenite keeps a partly read frame, so the next read resumes it.
fn timed_out(error: &std::io::Error) -> bool {
pub(crate) fn timed_out(error: &std::io::Error) -> bool {
matches!(error.kind(), ErrorKind::WouldBlock | ErrorKind::TimedOut)
}
+128 -2
View File
@@ -9,7 +9,7 @@
//! A connection that lost a message reconnects (its `seq`, in
//! `src/kalshi/mod.rs`); the book then comes fresh.
use std::collections::BTreeMap;
use std::collections::{BTreeMap, VecDeque};
use chrono::{DateTime, Utc};
use engineering_challenge::book::Book;
@@ -17,7 +17,7 @@ use engineering_challenge::cause::{self, Cause};
use super::Market;
use super::wire::{BookMessage, ChangeMessage, Event};
use crate::core::feed::{self, Effect, Minute, Rule, Shell, is_up};
use crate::core::feed::{self, Effect, Lateness, Minute, Rule, Shell, is_up};
use crate::core::socket::Command;
/// A clock, and a rank among the changes of that clock.
@@ -26,6 +26,14 @@ type Position = (u64, u32);
/// A book's rank: it holds every change of its clock.
const BOOK: u32 = u32::MAX;
/// Changes remembered per symbol for their second copy's lateness: two
/// connections deliver a market's changes 2 ms apart at the median, 344 ms
/// at worst (docs/notes.md), far fewer changes than this.
// ponytail: a second copy more than 128 changes behind goes unmeasured, and
// the lookup is linear; a map by position, emptied as the book passes,
// would lift both.
const RECENT: usize = 128;
/// A symbol's state for the rule.
#[derive(Debug, Default)]
pub struct Clocked {
@@ -37,6 +45,9 @@ pub struct Clocked {
on: BTreeMap<usize, Position>,
/// The connection a fresh book was asked from: its answer replaces ours.
asked: Option<usize>,
/// Its last [`RECENT`] changes applied: where, from which connection,
/// when; a change's second copy takes its out.
recent: VecDeque<(Position, usize, DateTime<Utc>)>,
}
type Symbol = feed::Symbol<Clocked>;
@@ -64,6 +75,7 @@ struct Counts {
#[derive(Default)]
pub struct Clock {
counts: Counts,
lateness: Lateness,
}
impl Clock {
@@ -113,6 +125,9 @@ impl Clock {
symbol.book = book;
symbol.exchange_timestamp = message.sent_at;
symbol.rule.at = Some(position);
// A position before it may come again, from a stream behind: it would
// be measured against a copy from before the book.
symbol.rule.recent.clear();
// A first book answers whichever connection it was asked from.
if asked || cause.is_none() {
symbol.rule.asked = None;
@@ -154,6 +169,22 @@ impl Clock {
match symbol.rule.at {
Some(at) if position <= at => {
self.counts.duplicates += 1;
let recent = &mut symbol.rule.recent;
let index = recent.iter().position(|&(at, ..)| at == position);
if let Some((_, first, first_at)) = index.and_then(|index| recent.remove(index))
&& let Some(late) = (received_at - first_at).num_microseconds()
{
// The book thread applies copies in the order they reach
// it, not in the order they were stamped: the later
// stamp is the late one.
let (late_on, first_on) = if late < 0 {
(first, connection)
} else {
(connection, first)
};
self.lateness.entry(late_on).or_default().push(late.abs());
self.lateness.entry(first_on).or_default().push(0);
}
return;
}
// Its stream was already past the book, which went back to an
@@ -166,6 +197,13 @@ impl Clock {
_ => {}
}
symbol.rule.at = Some(position);
if symbol.rule.recent.len() == RECENT {
symbol.rule.recent.pop_front();
}
symbol
.rule
.recent
.push_back((position, connection, received_at));
let quantity = symbol.book.add(change.side, change.price, change.delta);
let quantity = if quantity < 0 {
// A level can't hold less than nothing: ours was wrong.
@@ -316,6 +354,10 @@ impl Rule for Clock {
}
}
fn take_lateness(&mut self) -> Lateness {
std::mem::take(&mut self.lateness)
}
fn log_minute(&mut self, minute: &Minute) {
let counts = &self.counts;
log!(
@@ -462,6 +504,90 @@ mod tests {
assert_eq!(feed.rule.counts.duplicates, 1);
}
#[test]
fn a_duplicate_reports_its_lateness_and_the_first_copy_zero_at_the_minute() {
let mut feed = both_on("M");
feed.from(1, 5, change("M", 11, Side::Buy, 400_000, 2_000_000));
feed.from(0, 9, change("M", 11, Side::Buy, 400_000, 2_000_000));
// Only one connection delivers it: no copy to measure against.
feed.from(1, 20, change("M", 12, Side::Buy, 400_000, 1_000_000));
let reported: Vec<_> = feed
.tick(at(0) + TimeDelta::minutes(2))
.into_iter()
.filter_map(|effect| match effect {
Effect::Report(feed::Report::Lateness(lateness)) => Some(lateness),
_ => None,
})
.collect();
assert_eq!(reported, [HashMap::from([(0, vec![4_000]), (1, vec![0])])]);
// Taken: the next minute starts empty.
assert!(feed.rule.take_lateness().is_empty());
}
#[test]
fn the_copy_stamped_later_is_the_late_one_whichever_is_applied_first() {
let mut feed = both_on("M");
// Connection 1's copy reaches the book thread first, but connection
// 0's was stamped 4 ms before it.
feed.from(1, 9, change("M", 11, Side::Buy, 400_000, 2_000_000));
feed.from(0, 5, change("M", 11, Side::Buy, 400_000, 2_000_000));
assert_eq!(
feed.rule.take_lateness(),
HashMap::from([(0, vec![0]), (1, vec![4_000])])
);
}
#[test]
fn a_fresh_book_forgets_the_copies_before_it() {
let mut feed = both_on("M");
feed.from(1, 5, change("M", 11, Side::Buy, 400_000, 2_000_000));
// A fresh book past ours, and different: it replaces ours.
let replaced = feed.from(0, 6, book("M", 12, &[(400_000, 1_000_000)], &[]));
assert_eq!(snapshots(&replaced), 1);
// A change behind it, from a stream behind, an hour later: a
// duplicate, not measured against the copy of before the book.
feed.from(0, 3_600_000, change("M", 11, Side::Buy, 400_000, 2_000_000));
assert_eq!(feed.rule.counts.duplicates, 1);
assert!(feed.rule.take_lateness().is_empty());
}
#[test]
fn a_symbol_remembers_its_last_128_changes_only() {
let mut feed = feed_of(&["M"], &[0]);
feed.from(0, 1, book("M", 10, &[(400_000, 5_000_000)], &[]));
// One connection: no change has a second copy to take it out.
for clock in 11..211 {
feed.from(0, 2, change("M", clock, Side::Buy, 400_000, 1_000));
}
let recent = &feed.shell.symbols["M"].rule.recent;
assert_eq!(recent.len(), RECENT);
// The oldest went first.
assert_eq!(
recent.front().map(|&(position, ..)| position),
Some((83, 1))
);
}
#[test]
fn a_third_copy_while_a_market_moves_is_not_measured() {
let mut feed = feed_of(&["M"], &[0, 1, 2]);
for connection in [0, 1, 2] {
feed.from(connection, 1, book("M", 10, &[(400_000, 5_000_000)], &[]));
}
for (connection, millis) in [(1, 5), (0, 9), (2, 30)] {
feed.from(
connection,
millis,
change("M", 11, Side::Buy, 400_000, 2_000_000),
);
}
// The first copy counts once, against the second.
assert_eq!(
feed.rule.take_lateness(),
HashMap::from([(0, vec![4_000]), (1, vec![0])])
);
}
#[test]
fn every_change_of_one_clock_is_applied_once() {
let mut feed = both_on("M");
+17 -5
View File
@@ -25,11 +25,7 @@ const OPENING_AHEAD: TimeDelta = TimeDelta::hours(3);
/// A fetcher thread: answers each request until the orchestrator is gone.
pub fn run(listing: &str, requests: Receiver<Request>, findings: Sender<Finding<Market>>) {
let agent = ureq::Agent::new_with_config(
ureq::Agent::config_builder()
.timeout_global(Some(TIMEOUT))
.build(),
);
let agent = agent();
let markets = format!("{listing}/markets");
for request in requests {
let sent = match request {
@@ -42,6 +38,22 @@ pub fn run(listing: &str, requests: Receiver<Request>, findings: Sender<Finding<
}
}
fn agent() -> ureq::Agent {
ureq::Agent::new_with_config(
ureq::Agent::config_builder()
.timeout_global(Some(TIMEOUT))
.build(),
)
}
/// Every open market's ticker, in the listing's order (the latency probe's
/// candidates, roadmap/07).
pub(crate) fn open_tickers(listing: &str) -> anyhow::Result<Vec<String>> {
let query = [("status", "open".to_string()), mve_excluded()];
let markets = every_page(&agent(), &format!("{listing}/markets"), &query)?;
Ok(markets.into_iter().map(|market| market.ticker).collect())
}
/// `Err` once the orchestrator is gone.
type Sent = Result<(), crossbeam::channel::SendError<Finding<Market>>>;
+524
View File
@@ -0,0 +1,524 @@
//! The latency probe (roadmap/07): [`CONNECTIONS`] connections to the same
//! busy markets, spread over the exchange's addresses, each copy of each
//! change written as a CSV line: which connection delivered it, from which
//! address, when. It computes nothing: `scripts/latency/analyse.py` matches
//! the copies across connections. No book, no rule, no reconnect: a
//! connection that drops stays down, and the CSV says when.
//! Usage: `engineering-challenge latency EXCHANGE OUTPUT`, until SIGINT.
use std::collections::{BTreeMap, BTreeSet, HashMap};
use std::fs::File;
use std::io::{BufWriter, Write};
use std::net::{SocketAddr, ToSocketAddrs, UdpSocket};
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
use std::thread;
use std::time::{Duration, Instant};
use chrono::Utc;
use crossbeam::channel::{Sender, unbounded};
use engineering_challenge::Side;
use serde::Deserialize;
use tungstenite::Message;
use crate::core::socket::{self, PING_EVERY};
use crate::core::{Exchange, Session, Undecoded, retried};
use crate::kalshi::{self, Kalshi};
use crate::polymarket::{self, Polymarket};
/// Connections per run: enough to rank, few enough to share the server.
const CONNECTIONS: usize = 16;
/// Markets every connection carries: few, so our own load stays out of the
/// measurement.
const MARKETS: usize = 100;
/// Kalshi's demo reports no traded amount: every open market's changes are
/// counted for [`COUNTING`], this many on each connection (30,000 met no
/// limit, docs/notes.md).
const COUNTED_PER_CONNECTION: usize = 30_000;
const COUNTING: Duration = Duration::from_secs(60);
/// Fewer markets changed while counting: the run would measure nothing.
const ENOUGH_MARKETS: usize = 10;
/// A refused handshake (429 when many open at once) is tried again.
const CONNECT_TRIES: u32 = 3;
const CONNECT_RETRY_AFTER: Duration = Duration::from_secs(1);
const SNTP_SERVER: &str = "time.cloudflare.com:123";
const GAMMA_MARKETS: &str = "https://gamma-api.polymarket.com/markets";
const HTTP_TIMEOUT: Duration = Duration::from_secs(30);
const USAGE: &str = "usage: engineering-challenge latency <polymarket|kalshi|kalshi-demo> OUTPUT";
/// One copy of a change, as a connection delivered it.
#[derive(Debug, Clone, PartialEq, Eq)]
struct ChangeCopy {
symbol: String,
/// The same on every connection for one change, different for two.
identity: String,
exchange_ms: i64,
}
/// How an exchange's events become copies.
trait Copies: Exchange {
/// What a connection remembers to tell identical changes apart.
type Seen: Default;
fn copies(seen: &mut Self::Seen, event: &Self::Event) -> Vec<ChangeCopy>;
}
impl Copies for Polymarket {
type Seen = ();
/// A price change's entries, each told apart by its symbol and the book
/// hash after it, with its side, price and size: the rule's own copy
/// test (`Held::key`).
fn copies(_: &mut (), event: &polymarket::wire::Event) -> Vec<ChangeCopy> {
let polymarket::wire::Event::PriceChange(message) = event else {
return Vec::new();
};
let Ok(exchange_ms) = message.timestamp.parse() else {
return Vec::new();
};
message
.price_changes
.iter()
.map(|change| ChangeCopy {
symbol: change.asset_id.clone(),
identity: format!(
"{}/{}/{}/{}",
change.hash, change.side, change.price, change.size
),
exchange_ms,
})
.collect()
}
}
/// A change's side, price and delta.
type Content = (Side, i64, i64);
/// By market: its last clock, and how often each change came under it.
#[derive(Default)]
struct KalshiSeen(HashMap<String, (u64, HashMap<Content, u32>)>);
impl Copies for Kalshi {
type Seen = KalshiSeen;
/// A change, told apart by its market, clock, side, price and delta, and
/// by how many such came before it under that clock on this connection:
/// 2 to 4 changes can share a clock, some identical (docs/notes.md).
fn copies(seen: &mut KalshiSeen, event: &kalshi::wire::Event) -> Vec<ChangeCopy> {
let kalshi::wire::Event::Change(change) = event else {
return Vec::new();
};
let (clock, counts) = seen.0.entry(change.ticker.clone()).or_default();
// A market's clock never goes back: the older ones' counts are done.
if *clock != change.clock {
*clock = change.clock;
counts.clear();
}
let count = counts
.entry((change.side, change.price, change.delta))
.or_default();
*count += 1;
vec![ChangeCopy {
symbol: change.ticker.clone(),
identity: format!(
"{}/{}/{}/{}/{}",
change.clock, change.side, change.price, change.delta, count
),
exchange_ms: change.at.timestamp_millis(),
}]
}
}
/// What the writer gets.
enum Line {
Copy {
connection: usize,
address: SocketAddr,
received_us: i64,
copy: ChangeCopy,
},
/// A line starting with `#`: the clock's offset, a connection down.
Note(String),
}
pub fn run(mut args: impl Iterator<Item = String>) -> anyhow::Result<()> {
let exchange = args.next().ok_or_else(|| anyhow::anyhow!(USAGE))?;
let output = args.next().ok_or_else(|| anyhow::anyhow!(USAGE))?;
let stop = Arc::new(AtomicBool::new(false));
let on_sigint = Arc::clone(&stop);
ctrlc::set_handler(move || on_sigint.store(true, Ordering::Relaxed))?;
match exchange.as_str() {
"polymarket" => {
let polymarket = Polymarket {
url: polymarket::MARKET_CHANNEL_URL.to_string(),
};
probe(&polymarket, polymarket_symbols()?, &output, &stop)
}
"kalshi" | "kalshi-demo" => {
let key = kalshi::sign::Key::from_env()?;
let kalshi = if exchange == "kalshi" {
Kalshi::production(key)
} else {
Kalshi::demo(key)
};
let symbols = kalshi_symbols(&kalshi, &stop)?;
probe(&kalshi, symbols, &output, &stop)
}
_ => anyhow::bail!("unknown exchange {exchange:?}; {USAGE}"),
}
}
/// Runs the connections until `stop`, writing every copy to `output`.
fn probe<E: Copies>(
exchange: &E,
symbols: Vec<String>,
output: &str,
stop: &AtomicBool,
) -> anyhow::Result<()> {
let addresses = addresses(exchange)?;
log!(
"latency: {} symbols on {CONNECTIONS} connections over {addresses:?}",
symbols.len()
);
let mut out = BufWriter::new(File::create(output)?);
let (lines, received) = unbounded();
let writer = thread::spawn(move || -> anyhow::Result<()> {
writeln!(
out,
"connection,address,symbol,identity,received_us,exchange_ms"
)?;
for line in received {
match line {
Line::Copy {
connection,
address,
received_us,
copy,
} => writeln!(
out,
"{connection},{address},{},{},{received_us},{}",
copy.symbol, copy.identity, copy.exchange_ms
)?,
Line::Note(note) => writeln!(out, "# {note}")?,
}
}
out.flush()?;
Ok(())
});
note_offset(&lines, "start");
thread::scope(|scope| {
for number in 0..CONNECTIONS {
let address = addresses[number % addresses.len()];
let (symbols, lines) = (&symbols, lines.clone());
scope.spawn(move || connection(exchange, number, address, symbols, stop, &lines));
}
});
note_offset(&lines, "end");
drop(lines);
writer
.join()
.map_err(|_| anyhow::anyhow!("the writer thread panicked"))??;
log!("latency: output closed");
Ok(())
}
/// Every IPv4 address the websocket's host resolves to, in a fixed order.
/// IPv6 ones are left out: the workspace has no IPv6 route ("Network is
/// unreachable", 2026-11-12).
fn addresses<E: Exchange>(exchange: &E) -> anyhow::Result<Vec<SocketAddr>> {
let request = exchange.request()?;
let host = request
.uri()
.host()
.ok_or_else(|| anyhow::anyhow!("{} has no host", request.uri()))?;
let port = request.uri().port_u16().unwrap_or(443);
let addresses: BTreeSet<SocketAddr> = (host, port)
.to_socket_addrs()?
.filter(SocketAddr::is_ipv4)
.collect();
anyhow::ensure!(!addresses.is_empty(), "{host} has no IPv4 address");
Ok(addresses.into_iter().collect())
}
/// One connection, to `address`, until `stop` or until it drops.
fn connection<E: Copies>(
exchange: &E,
number: usize,
address: SocketAddr,
symbols: &[String],
stop: &AtomicBool,
lines: &Sender<Line>,
) {
let name = E::NAME;
let down = |why: String| {
log!("{name}: latency connection {number} to {address}: {why}");
let at = Utc::now().timestamp_micros();
_ = lines.send(Line::Note(format!("down,{number},{at}")));
};
let opened = retried(CONNECT_TRIES, CONNECT_RETRY_AFTER, || {
socket::open(exchange.request()?, address)
});
let mut socket = match opened {
Ok(socket) => socket,
Err(error) => return down(format!("connect failed: {error}")),
};
let mut session = exchange.session();
let symbols: Vec<&String> = symbols.iter().collect();
for message in [session.opening(), session.subscribe(&symbols)].concat() {
if let Err(error) = socket.send(Message::text(message)) {
return down(format!("subscribe failed: {error}"));
}
}
log!("{name}: latency connection {number} to {address}: connected");
let mut seen = E::Seen::default();
let mut last_ping = Instant::now();
while !stop.load(Ordering::Relaxed) {
let message = socket.read();
// Stamped as soon as the frame is off the wire, as the collector does.
let received_us = Utc::now().timestamp_micros();
match message {
Ok(Message::Text(text)) => match session.decode(text.as_str()) {
Ok(events) => {
for copy in events.iter().flat_map(|event| E::copies(&mut seen, event)) {
let line = Line::Copy {
connection: number,
address,
received_us,
copy,
};
if lines.send(line).is_err() {
return;
}
}
}
Err(Undecoded::Unreadable(error)) => {
log!("{name}: latency connection {number}: unreadable frame ({error})")
}
Err(Undecoded::Lost(why)) => return down(format!("lost a message: {why}")),
},
Ok(Message::Close(frame)) => return down(format!("closed by the server: {frame:?}")),
Ok(_) => {}
Err(tungstenite::Error::Io(error)) if socket::timed_out(&error) => {}
Err(error) => return down(format!("connection lost: {error}")),
}
if last_ping.elapsed() >= PING_EVERY {
if let Some(keepalive) = session.keepalive()
&& let Err(error) = socket.send(Message::text(keepalive))
{
return down(format!("keepalive failed: {error}"));
}
last_ping = Instant::now();
}
}
_ = socket.close(None);
_ = socket.flush();
}
/// The local clock's offset from [`SNTP_SERVER`], as a note: `offset,<when>,
/// <offset µs>,<round trip µs>`; a failed query is noted and logged, not
/// fatal.
fn note_offset(lines: &Sender<Line>, when: &str) {
let note = match offset() {
Ok((offset, roundtrip)) => {
log!("latency: clock offset {offset} µs (round trip {roundtrip} µs)");
format!("offset,{when},{offset},{roundtrip}")
}
Err(error) => {
log!("latency: clock offset unknown: {error}");
format!("offset,{when},,")
}
};
_ = lines.send(Line::Note(note));
}
fn offset() -> anyhow::Result<(i64, u64)> {
let socket = UdpSocket::bind("0.0.0.0:0")?;
socket.set_read_timeout(Some(Duration::from_secs(2)))?;
let server = SNTP_SERVER
.to_socket_addrs()?
.find(SocketAddr::is_ipv4)
.ok_or_else(|| anyhow::anyhow!("{SNTP_SERVER} has no IPv4 address"))?;
let context = sntpc::NtpContext::new(sntpc::StdTimestampGen::default());
let result = sntpc::sync::get_time(
server,
&sntpc_net_std::UdpSocketWrapper::new(socket),
context,
)
.map_err(|error| anyhow::anyhow!("SNTP: {error:?}"))?;
Ok((result.offset(), result.roundtrip()))
}
fn agent() -> ureq::Agent {
ureq::Agent::config_builder()
.timeout_global(Some(HTTP_TIMEOUT))
.build()
.into()
}
/// The tokens of the [`MARKETS`] live markets most traded over 24 hours.
fn polymarket_symbols() -> anyhow::Result<Vec<String>> {
#[derive(Deserialize)]
#[serde(rename_all = "camelCase")]
struct GammaMarket {
/// A JSON array inside a string.
clob_token_ids: Option<String>,
}
let limit = MARKETS.to_string();
let body = agent()
.get(GAMMA_MARKETS)
.query("closed", "false")
.query("active", "true")
.query("order", "volume24hr")
.query("ascending", "false")
.query("limit", &limit)
.call()?
.body_mut()
.read_to_string()?;
let markets: Vec<GammaMarket> = serde_json::from_str(&body)?;
let tokens: Vec<String> = markets
.iter()
.filter_map(|market| market.clob_token_ids.as_deref())
.filter_map(|ids| serde_json::from_str::<Vec<String>>(ids).ok())
.flatten()
.collect();
anyhow::ensure!(!tokens.is_empty(), "Gamma listed no tokens");
Ok(tokens)
}
/// The [`MARKETS`] markets with the most changes in [`COUNTING`], among the
/// listing's open ones. Not the listing's first pages: its order moves (the
/// 98 busiest of 11:12 were past its first 40,000 at 11:50, 2026-11-12).
fn kalshi_symbols(kalshi: &Kalshi, stop: &AtomicBool) -> anyhow::Result<Vec<String>> {
let candidates = kalshi::listing::open_tickers(&kalshi.listing)?;
let chunks: Vec<&[String]> = candidates.chunks(COUNTED_PER_CONNECTION).collect();
log!(
"latency: counting the changes of {} Kalshi markets on {} connections for {} s",
candidates.len(),
chunks.len(),
COUNTING.as_secs()
);
let addresses = addresses(kalshi)?;
let counted = AtomicBool::new(false);
let (lines, received) = unbounded();
let mut changes: BTreeMap<String, u64> = BTreeMap::new();
thread::scope(|scope| {
for (number, chunk) in chunks.iter().enumerate() {
let (address, counted, lines) = (addresses[number % addresses.len()], &counted, &lines);
scope.spawn(move || connection(kalshi, number, address, chunk, counted, lines));
}
let deadline = Instant::now() + COUNTING;
while Instant::now() < deadline && !stop.load(Ordering::Relaxed) {
for line in received.try_iter() {
if let Line::Copy { copy, .. } = line {
*changes.entry(copy.symbol).or_default() += 1;
}
}
thread::sleep(Duration::from_millis(100));
}
counted.store(true, Ordering::Relaxed);
});
busiest(changes, MARKETS)
}
/// The `n` symbols with the most changes, the most first; an error if fewer
/// than [`ENOUGH_MARKETS`] changed at all: the run would measure nothing.
fn busiest(changes: BTreeMap<String, u64>, n: usize) -> anyhow::Result<Vec<String>> {
anyhow::ensure!(
changes.len() >= ENOUGH_MARKETS,
"only {} markets changed in {} s: nothing to measure",
changes.len(),
COUNTING.as_secs()
);
let mut ranked: Vec<(String, u64)> = changes.into_iter().collect();
ranked.sort_by_key(|(_, changes)| std::cmp::Reverse(*changes));
Ok(ranked
.into_iter()
.take(n)
.map(|(symbol, _)| symbol)
.collect())
}
#[cfg(test)]
mod tests {
use super::*;
fn polymarket(text: &str) -> Vec<ChangeCopy> {
polymarket::wire::decode(text)
.unwrap()
.iter()
.flat_map(|event| Polymarket::copies(&mut (), event))
.collect()
}
fn price_change(hash: &str, size: &str) -> String {
format!(
r#"{{"market":"0xm","price_changes":[{{"asset_id":"a","price":"0.29","size":"{size}","side":"SELL","hash":"{hash}","best_bid":"0.18","best_ask":"0.19"}}],"timestamp":"1793802284739","event_type":"price_change"}}"#
)
}
#[test]
fn a_polymarket_change_has_one_identity_on_every_connection_its_hash_and_size_tell_two_apart() {
let first = polymarket(&price_change("2d0c", "2306.33"));
assert_eq!(first.len(), 1);
assert_eq!(first[0].symbol, "a");
assert_eq!(first[0].exchange_ms, 1_793_802_284_739);
// The same frame on another connection: the same copy.
assert_eq!(polymarket(&price_change("2d0c", "2306.33")), first);
// Only the hash, or only the size, differs: another change.
for other in [price_change("c000", "2306.33"), price_change("2d0c", "10")] {
assert_ne!(polymarket(&other)[0].identity, first[0].identity);
}
}
fn kalshi_change(seq: u64, ticker: &str, clock: u64) -> String {
format!(
r#"{{"type":"orderbook_delta","sid":1,"seq":{seq},"msg":{{"market_ticker":"{ticker}","market_id":"01a1","clock":{clock},"price_dollars":"0.8100","delta_fp":"-1.50","side":"no","ts":"2026-11-11T18:36:37.012Z","ts_ms":1794422197012}},"sending_ts_ms":1794422197023}}"#
)
}
/// One connection's copies of `frames`, each a (ticker, clock), in order.
fn kalshi(frames: &[(&str, u64)]) -> Vec<String> {
let mut session = kalshi::Session::default();
let mut seen = KalshiSeen::default();
(1..)
.zip(frames)
.flat_map(|(seq, &(ticker, clock))| {
session.decode(&kalshi_change(seq, ticker, clock)).unwrap()
})
.flat_map(|event| Kalshi::copies(&mut seen, &event))
.map(|copy| copy.identity)
.collect()
}
#[test]
fn identical_kalshi_changes_under_one_clock_are_two_changes_on_every_connection() {
// Two identical changes under clock 7, then the same again under 8.
let one = kalshi(&[("M", 7), ("M", 7), ("M", 8)]);
assert_eq!(one.len(), 3);
assert_ne!(one[0], one[1]);
assert_ne!(one[2], one[0]);
// A new clock counts from one again.
assert!(one[2].ends_with("/1"), "{}", one[2]);
// Another connection: the same three changes.
assert_eq!(kalshi(&[("M", 7), ("M", 7), ("M", 8)]), one);
}
#[test]
fn each_kalshi_market_counts_its_identical_changes_on_its_own() {
// Another market's change between M's two identical ones.
let copies = kalshi(&[("M", 7), ("N", 9), ("M", 7)]);
assert_ne!(copies[0], copies[2]);
assert!(copies[2].ends_with("/2"), "{}", copies[2]);
}
#[test]
fn the_busiest_come_first_and_too_few_stop_the_run() {
// Ten markets, "m9" with the most changes, "m8" next.
let mut changes: BTreeMap<String, u64> = (0..10).map(|n| (format!("m{n}"), n)).collect();
assert_eq!(busiest(changes.clone(), 2).unwrap(), ["m9", "m8"]);
// Nine: nothing to measure.
changes.remove("m0");
assert!(busiest(changes, 2).is_err());
}
}
+7 -2
View File
@@ -4,7 +4,8 @@
//! resync), all writing through one writer (SBE).
//! Usage: `engineering-challenge EXCHANGE [OUTPUT]`, EXCHANGE one of
//! `polymarket`, `kalshi` or `kalshi-demo`, OUTPUT defaulting to stdout; logs
//! go to stderr.
//! go to stderr. `engineering-challenge latency ...` runs the latency probe
//! instead (`src/latency.rs`).
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
@@ -22,6 +23,7 @@ macro_rules! log {
mod core;
mod kalshi;
mod latency;
mod polymarket;
use engineering_challenge::book::Output;
@@ -29,11 +31,14 @@ use engineering_challenge::book::Output;
/// `AlignedSbeReader::open` reads nothing else.
const CHUNK: usize = 64 << 20;
const USAGE: &str = "usage: engineering-challenge <polymarket|kalshi|kalshi-demo> [OUTPUT]";
const USAGE: &str = "usage: engineering-challenge <polymarket|kalshi|kalshi-demo> [OUTPUT], or latency <polymarket|kalshi|kalshi-demo> OUTPUT";
fn main() -> anyhow::Result<()> {
let mut args = std::env::args().skip(1);
let exchange = args.next().ok_or_else(|| anyhow::anyhow!(USAGE))?;
if exchange == "latency" {
return latency::run(args);
}
let output = args.next().unwrap_or_else(|| "/dev/stdout".to_string());
// Before anything starts: a key that can't be read stops the run here.
match exchange.as_str() {
+21 -19
View File
@@ -105,7 +105,7 @@ struct RecentChange {
/// The level's quantity before it (0 if absent): what undoes it.
previous: i64,
hash: Hash,
/// When its first copy came: a later copy's lag is measured from it.
/// When its first copy came: a later copy's lateness is measured from it.
received_at: DateTime<Utc>,
/// How many copies came: one, of a symbol on two connections, is a change
/// the other missed.
@@ -123,7 +123,7 @@ pub struct Chained {
/// By connection: the changes it delivered that don't fit yet.
held: BTreeMap<usize, Vec<Held>>,
/// When the book last moved: held changes older than this, on a
/// connection whose changes still don't fit, are that connection's lag.
/// connection whose changes still don't fit, are late on that connection.
advanced_at: DateTime<Utc>,
/// The connection a fresh book was asked from, and whether it is written
/// whatever it holds: the connection followed changed.
@@ -332,7 +332,7 @@ pub struct Chain {
/// The symbols with held changes: their ceilings are checked.
waiting: BTreeSet<String>,
/// How long after its first copy each duplicate came this minute, µs.
lags: Vec<i64>,
lateness: Vec<i64>,
}
impl Chain {
@@ -730,7 +730,7 @@ impl Chain {
change,
received_at,
&mut self.counts,
&mut self.lags,
&mut self.lateness,
effects,
);
if symbol.rule.held.is_empty() {
@@ -746,7 +746,7 @@ impl Chain {
/// A verified symbol's change: dropped if the book has it, else held on
/// its connection until a tail of its held changes fits; each fit may let
/// another connection's held changes fit in turn.
// The rule's fields it needs (the counts, the lags), borrowed apart from
// The rule's fields it needs (the counts, the lateness), borrowed apart from
// the symbol, which its callers hold out of the shell.
#[allow(clippy::too_many_arguments)]
fn first_one_wins(
@@ -756,7 +756,7 @@ impl Chain {
change: Held,
now: DateTime<Utc>,
counts: &mut Counts,
lags: &mut Vec<i64>,
lateness: &mut Vec<i64>,
effects: &mut Vec<Effect>,
) {
if let Some(first) = symbol
@@ -767,8 +767,8 @@ impl Chain {
{
counts.duplicates += 1;
first.copies = first.copies.saturating_add(1);
if let Some(lag) = (change.received_at - first.received_at).num_microseconds() {
lags.push(lag);
if let Some(late) = (change.received_at - first.received_at).num_microseconds() {
lateness.push(late);
}
return;
}
@@ -952,11 +952,11 @@ impl Rule for Chain {
}
fn log_minute(&mut self, minute: &Minute) {
self.lags.sort_unstable();
let lag = |quantile: f64| {
let index = ((self.lags.len() as f64 * quantile) as usize)
.min(self.lags.len().saturating_sub(1));
self.lags
self.lateness.sort_unstable();
let late = |quantile: f64| {
let index = ((self.lateness.len() as f64 * quantile) as usize)
.min(self.lateness.len().saturating_sub(1));
self.lateness
.get(index)
.map_or(0.0, |&micros| micros as f64 / 1_000.0)
};
@@ -965,8 +965,8 @@ impl Rule for Chain {
"polymarket: book thread {}: {} symbols; so far {} changes written ({} verified), {} duplicates, \
{} rescued, {} held dropped, {} chains broken, {} resyncs, {} full books behind ours, \
{} subscription gaps, {} losses, {} switches, {} quiet subscriptions resubscribed, \
{} markets reported quiet again, {} reconnect books judged ({} losses); second copies \
{:.1} ms late at the median, {:.1} ms at p99; this minute {} changes received, at most \
{} markets reported quiet again, {} reconnect books judged ({} losses); duplicates' \
lateness {:.1} ms at the median, {:.1} ms at p99; this minute {} changes received, at most \
{} events waited",
minute.number,
minute.symbols,
@@ -985,12 +985,12 @@ impl Rule for Chain {
minute.quiet_reported,
counts.reconnect_books,
counts.reconnect_losses,
lag(0.5),
lag(0.99),
late(0.5),
late(0.99),
minute.received,
minute.peak_waiting
);
self.lags.clear();
self.lateness.clear();
}
}
@@ -1709,7 +1709,7 @@ mod tests {
both.rule.counts
);
assert_eq!(both.rule.counts.losses, 0);
assert!(!both.rule.lags.is_empty() || both.rule.counts.duplicates == 0);
assert!(!both.rule.lateness.is_empty() || both.rule.counts.duplicates == 0);
}
/// The first copy's receive time is the one written.
@@ -2711,6 +2711,8 @@ mod tests {
Effect::Send(..) => false,
Effect::Report(Report::Loads(loads)) => loads.contains_key(&markets[0].condition_id),
Effect::Report(Report::Quiet { market, .. }) => *market == markets[0].condition_id,
// Polymarket's rule reports none.
Effect::Report(Report::Lateness(_)) => true,
};
assert!(!effects.iter().any(mentions));
// The minute Snapshots of the 38 others, twice.