This repository has been archived on 2026-11-16. You can view files and clone it. You cannot open issues or pull requests or push a commit.
Files
orderbook-collector/roadmap/07. latency experiments.md
pascal 515e85941d feat(latency): latency experiments, LATENCY.md, Kalshi's slow-connection recycle
Extension 4. `engineering-challenge latency EXCHANGE OUT.csv` opens 16
connections to the same busy markets and writes every copy of every change;
`scripts/latency/analyse.py` gives each connection's speed, whether it
persists, what keeping the fastest would gain, lateness by address, and lag
with the clock's offset. LATENCY.md writes up four runs: on Polymarket speed
persists but is worth under a millisecond, at the median only; on Kalshi's
demo keeping the fastest pair lowers p99 by 28% and 42%, the threshold
agreed before the runs.

So the collector now recycles a Kalshi connection whose mean lateness stays,
3 minutes running, in the slowest quarter and above twice everyone's, ahead
of its hourly turn. Kalshi's rule measures each duplicate's lateness; the
orchestrator judges once a minute. A duplicate's lag in Polymarket's minute
line is now its "lateness".

Roadmap milestone 7, roadmap/07.
2026-11-12 15:05:25 +01:00

362 lines
20 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# 07. Latency experiments
**Status: done (2026-11-12)**
Extension 4 of the brief: "experimentation into how to [get faster
connections] on various exchanges like Polymarket or Kalshi", written up in a
markdown file. Correctness stays the first priority, so the collector
changes only if the data clears a threshold agreed here. The question under
every trick the brief names (racing connections, the source port, being
first after maintenance) is one: **is a connection that is fast now still
fast later?** If speed is noise per message, no choice of connection pays.
Terms: [../GLOSSARY.md](../GLOSSARY.md) (lateness, lag, speed, duplicate).
## The probe
`src/latency.rs`, a subcommand of the collector (`engineering-challenge
latency <polymarket|kalshi|kalshi-demo> <out.csv>`), kept in the repo
because `LATENCY.md` cites its numbers. It is not the collector: no book,
no rule, no reconnect. A subcommand, not `src/bin/latency.rs` as first
agreed, because what it reuses (`core`, `polymarket`, `kalshi`, `log!`)
belongs to the collector's binary, not the library; moving them was
rejected as a large diff that buys nothing else.
- **Markets:** the same 100 on every connection, few enough that our own
load stays out of the measurement. Polymarket: the 100 live markets with
the highest `volume24hr` (Gamma sorts by it: `order=volume24hr&ascending=false`,
checked), all their tokens. Kalshi: the demo reports `volume_24h_fp` 0.00
(all of the first 1,000 open markets, checked), so the probe counts every
open market's changes for a minute, 30,000 per connection, and keeps the
100 with the most; it stops if fewer than 10 changed. Production would be
picked the same way, one rule for both Kalshi deployments. Supersedes the
pick by traded amount on Kalshi, because the demo has none; and then the
count over the listing's first 30,000 only, because the listing's order
moves: the 98 busiest of 11:12 were past its first 40,000 at 11:50, and
the second Kalshi run counted 1 market that changed.
- **Connections:** 16 per run, one plain thread each, tungstenite over a
`TcpStream` the probe opens to an address it chooses, round-robin over the
exchange's IPv4 addresses (the workspace has no IPv6 route: the smoke
run's connections to Polymarket's IPv6 ones failed with "Network is
unreachable"; Polymarket's websocket has 4 IPv4 ones, the Kalshi demo's 2,
production's 1, checked 2026-11-12; the collector connects only to the
first, `src/core/socket.rs:184`). It reuses `Exchange::request` (Kalshi's
signed handshake), the `Session`'s subscribe, keepalive and decode
(`src/core/mod.rs:83` to `89`), and the collector's connect with the
address given (split out of `src/core/socket.rs`'s `connect`); a refused
handshake (429) is tried again through `core::retried`, 3 tries 1 s apart. The receive time is stamped
right after the read, as in the collector. A connection that drops is
logged and stays down: a run is 10 minutes.
- **Output:** one CSV line per copy of a change, raw: connection, address,
symbol, the change's identity, receive time (µs), exchange timestamp (ms).
The analysis matches copies across connections; the probe computes
nothing.
- **A change's identity** across connections: on Polymarket the symbol and
the change's book hash, side, price and size (the rule's own copy test,
`Held::key`, `src/polymarket/feed.rs:89`); on Kalshi the market,
`clock`, side, price and delta, plus the tuple's occurrence number on its
connection, since 2 to 4 changes can share a clock, some identical
(`docs/notes.md`).
- **Clock:** an SNTP query to `time.cloudflare.com` at the start and end of
each run, logged; lag is reported raw and corrected. `sntpc` 0.11
(`sync`, `std`) with `sntpc-net-std` 1.3, tried: offset +0.5 ms, round
trip 3 ms from the server (2026-11-12).
## The analysis
`scripts/latency/analyse.py`, standard library only (`statistics.correlation`
with `method="ranked"` is Spearman, Python 3.13 here). From a run's CSV:
- each copy's lateness, a server's repeat on one connection counted once;
a change a connection never delivered while it was up (from its first
copy until it went down) counts as missed by it;
- each connection's speed per minute, and its missed count;
- **persistence:** the rank correlation of the 16 speeds between each
minute and the next; and the 4 fastest of minute 1 against the other 12
over the full minutes after it, at the median and p99 of lateness;
- **racing:** the 2 fastest of minute 1 kept, against 2 at random (averaged
over every pair of the 16), over the full minutes after it, at the
median and p99 of lateness, with each pair's missed changes; a run too
thin to compare is said so;
- **by address:** lateness per edge address;
- **lag:** per exchange, raw and corrected by the offset.
## The runs
Two 10-minute runs per exchange, alternated (Polymarket, Kalshi,
Polymarket, Kalshi), `/proc/loadavg` recorded before each and the run put
off while the load is high. The second run repeats the first, which the
adoption threshold needs. Raw CSVs and logs stay out of the repo, in
`../runs/latency/`.
## The writeup
`LATENCY.md` at the repo root, a page or two: the method (the probe,
lateness against lag, the clock's offset), one section per experiment
(question, a small table per exchange, verdict), and the ideas not tried,
each with why and how one would: first after maintenance, the source port,
location, kernel tuning, IPv6, our own added latency. Kalshi's numbers are
labelled the demo's. What was not measured is written "assumed".
## Decisions
- **Lateness, not lag, ranks connections.** Lateness compares copies of one
change on one machine and needs no clock; lag carries the clock's offset
and the exchange's own stamping. Rejected: ranking by lag.
- **Raw CSV, analysis apart.** A new question is asked of a run without
running it again; live runs are what cost time on a shared server.
Rejected: summaries computed by the probe.
- **Racing is computed from the persistence run**, not run as an
open-close loop: choosing after the fact shows the same gain as long as a
connection's speed is set when it connects, which is what persistence
tests. Rejected: a separate racing run.
- **The source port is not tested.** At least one NAT between the workspace
and the internet rewrites it: local ports 40123, 40124 and 41000 to 41003
left as 50990, 53878, 36996, 48950, 51760, 60434, and 41000 again a minute
later as 37086 (`ifconfig.me`, 2026-11-12). The probe binds no port.
Supersedes the scope's "same (address, port) pairs" run.
- **Adoption threshold**, agreed before the data: the collector changes on
an exchange only if speed persists (rank correlation above about 0.5
across the run) and the 2 picked connections' p99 lateness is at least
20% below the random pair's, in both runs. The change would then be to
recycle a connection whose speed stayed among the slowest for several
minutes (the stretch goal). Otherwise the writeup says the threshold was
not met. Met on Kalshi's demo (28% and 42%), not on Polymarket (29%, then
-4%).
- **Recycling the slow connections (Kalshi)**, agreed after the runs, for
showing the loop from data to code and as one more self-correcting
mechanism, knowing the evidence is the demo's: Kalshi's rule keeps the
first copy's receive time and connection of its last 128 positions per
symbol, forgotten when a fresh book replaces ours; a duplicate records
its lateness for its connection, and 0 for the first copy's (the later
stamp is the late one, whichever reached the book thread first); a
third copy, while a market moves, is not measured. Each book thread reports the samples at the minute
(`Report::Lateness`); Polymarket's rule reports none, so nothing changes
there. The orchestrator, once a minute, ranks the connections with at
least 100 samples and up for 10 minutes by their mean lateness; those
of the slowest quarter whose mean is more than twice the mean of every
judged copy count a slow minute, the rest start over; one slow 3
minutes in a row is recycled at once, ahead of the hourly order, at most
one a minute and under the hourly recycle's rule (one at a time, every
connection up, the last up an interval ago). Rejected: building it in
the core for both exchanges, since Polymarket's p99 doesn't follow
speed. The mean, not the median, and the factor of two, supersede the
first design (by median, the slowest quarter alone), because of the
review: a slowest quarter always exists, so equal connections were
recycled about once a minute; and with two copies of each change, a
connection first more than half the time has a median of 0, so a median
says how often it comes second, not how late, and twice the median of
medians is about 0. The mean holds both, and the tail where Kalshi's
gain was. The probe's speed (the median, GLOSSARY.md) is unchanged.
- **Kalshi on the demo.** No production key; the code is the same, only
whether the demo behaves like production is unknown.
- **`lags` becomes `lateness`** in Polymarket's rule (Kalshi's measures
none), with the log text "second copies … late" read as "duplicates'
lateness … at the median"; nothing parses that line (`src/cause.rs`,
`src/reconcile.rs`).
- **Our own added latency** (receive time to output write) is not measured:
the receive time is stamped before any queue, and the brief's extension
is about the connection.
## Tasks
- [x] Rename `lags` to `lateness` in `src/polymarket/feed.rs`, comments
and log text included (Kalshi's rule measures none); the rules' tests
pass unchanged.
- [x] The probe's change identities, for both exchanges.
- [x] Test: a Polymarket change on two connections gets the same identity;
one differing only in its hash does not.
- [x] Test: two identical Kalshi changes under one clock on one connection
get two identities, the same two as on another connection.
- [x] The probe's market pick (Polymarket by `volume24hr`, Kalshi by a
minute's count) and its connections (address round-robin, request,
subscribe, keepalive, stamping, CSV), with the SNTP offset at start
and end.
- [x] A 1-minute smoke run of the probe on each exchange: 16 connections
up, every connection's lines in the CSV, the offset logged.
- [x] `scripts/latency/analyse.py`: lateness, missed, speed per minute,
persistence, racing, by address, lag.
- [x] Test (`scripts/latency/test_analyse.py`, from `scripts/ci.sh`): on a
small CSV worked out by hand, the first copy's lateness is 0, a change
one connection lacks counts as missed by it, and the speeds, ranks
and racing figures are the hand-computed ones.
- [x] The four runs, alternated, load and offset recorded.
- [x] `LATENCY.md`.
- [x] Stretch (threshold met on Kalshi): Kalshi's rule records each
duplicate's lateness and the first copy's 0, reported at the minute.
- [x] Test: a duplicate's lateness is reported for its connection and 0
for the first copy's; a change with one copy reports nothing.
- [x] The tracker's judgment and the orchestrator's recycle of a slow
connection, logged.
- [x] Test: slowest for 3 minutes is recycled; slowest 2 minutes then not
starts over; a connection up less than 10 minutes or with too few
samples is not judged; never one while another recycle is under way.
- [x] A 20-minute Kalshi demo run (a connection is judged from its 10th
minute, recycled after 3 slow ones), reconciled, its log showing the slow
recycles.
- [x] `CLAUDE.md` (the probe, the new terms, the NAT trap) and
`docs/roadmap.md` brought up to date.
From the review of `d722446`:
- [x] A slow recycle waits, as the hourly one does, an interval after the
last connection came up (its opening books), with its test.
- [x] A slow connection is ranked by its mean lateness and recycled only
if that mean is more than twice the mean of every judged copy: equal
connections are left alone, and the hourly recycle isn't starved.
Its test (equal connections, none recycled).
- [x] Kalshi's lateness: a copy stamped before the one applied first
swaps the two (no negative lateness); the remembered positions are
cleared when a fresh book replaces ours. Tests for both.
- [x] Tests: the tracker's mean against samples that differ (a median of
0, a high mean); two book
threads' samples merged; the 128-position cap; judging refused at
599 s.
- [x] Tests: Kalshi's identity on two interleaved markets; Polymarket's
told apart by size.
- [x] The stop below 10 counted markets moved into `busiest`, with its test.
- [x] The probe reuses the Kalshi listing's paging and agent
(`src/kalshi/listing.rs`, made `pub(crate)`).
- [x] `RECENT`'s comment names its upgrade path; renames: `counting` (it
means counting is over), `SLOW_SHARE`'s doc, `struct Copy`,
`Tracker::lateness`.
- [x] The analysis: no traceback on thin runs (4 or fewer connections in
the first minute, a kept pair silent after it, one connection);
tests for the p99 against the maximum (200 values), `report()` on
the hand-worked run (racing's gain and the Missed column), racing's
missed counts, and an offset note with no value.
- [x] `LATENCY.md`: the two Kalshi runs' picks; persistence read on the
mean (minutes at 0.47 and 0.49 said); the slowest 177 ms; the
second run's length explained right (the first's stop counted from
start: 9 min 52 s of probing, the second 10 min 2 s); no "two orders
of magnitude" for Kalshi's p99; "behind the address" marked assumed;
the collector run's load 7.3; how one would try IPv6, kernel tuning
and our own added latency.
- [x] This file: minutes 2 to the end, a missed count (while up) rather
than a share, and the third copy unmeasured, in the Design.
Each test is seen to fail once (the code it protects removed, then put
back).
## Review of `d722446` (2026-11-12)
Five reviewers (spec, bugs, tests, smells, security); 2 findings dropped
(the tests were seen to fail; the memory bound is 128 positions a
symbol). Every one below was decided "fix now" (the tasks above) unless
said otherwise.
- **Bugs:** the slow recycle skipped the hourly one's settling wait; it
never stops (a slowest quarter always exists) and can starve the hourly
one; the analysis crashed on thin runs; lateness could be negative or
measured against a stale position. Later: the judgment runs on the
orchestrator's timer, not the book threads' minute (partial samples make
it more conservative, not unsafe). No: the probe ignores SIGINT during
the listing scan and never notes a silent connection (a tool run by
hand; missed counts were 0 to 5).
- **Tests:** 9 gaps verified one mutant at a time (p99 and the tracker's
median against the maximum, `report()`, samples merged, the cap, racing's
missed, the offset note, Kalshi's identity across markets, Polymarket's
size), the stop below 10 and the 599 s boundary. No: the CSV contract
across the two languages, the IPv4 filter and round-robin, the tie-break,
the loop's wiring (live runs show them).
- **Smells:** the probe's copies of the listing's paging and agent; the
`RECENT` comment; four names. No: `Rule::take_lateness` (a hook on a
trait with two implementations), the analysis's primitive tuples and
repeats, the lateness and median in several places, the probe's match
on the exchange, `latency.rs`'s size.
- **Security:** no vulnerability. No: the docs name the country and the
NAT (part of the findings), unquoted CSV fields (a TLS-authenticated
source), unauthenticated SNTP (only the corrected lag).
- **Spec:** `LATENCY.md`'s figures and claims, the two picks, the mean,
three "how"s, this file's Design. No: the CSV's notes, ruff in CI, the
slow count in the minute line, the listing retry (each serves what was
asked).
## Not in this sprint
Polish and the README (M8); the roadmap's possible features; kernel
tuning; anything needing a second machine, another region or a public
address; the first connection after Kalshi's maintenance (next window
Thursday 2026-11-19, 08:00 to 10:00 UTC), unless the sprint is still open
then.
## Done when
- The probe and the analysis are on the branch, their tests seen to fail
once.
- Both exchanges' two runs are done and analysed, load and offset recorded.
- `LATENCY.md` gives a verdict per experiment and per exchange, Kalshi
labelled the demo.
- The stretch goal is built and tested if the threshold is met; otherwise
the writeup says it wasn't.
- `scripts/ci.sh` is green.
## Outcome
Extension 4 is answered and written up in `LATENCY.md`. `engineering-challenge
latency EXCHANGE OUT.csv` opens 16 connections to the same 100 busy markets
and writes every copy of every change; `scripts/latency/analyse.py` turns a
run into speeds, persistence, racing, lateness by address and lag. Four
10-minute runs, alternated, the clock's offset under 0.7 ms throughout:
| | Persistence (mean) | Racing, p99 lower by | Lag, median |
|---|---|---|---|
| Polymarket | 0.96, 0.98 | 29%, then −4% | 14 ms |
| Kalshi demo | 0.66, 0.78 | 28%, 42% | 94 to 106 ms |
On Polymarket speed persists but is worth under a millisecond, at the
median only: the tail is bursts every connection shares. On Kalshi's demo
it met the agreed threshold, so the collector now recycles a Kalshi
connection whose mean lateness stays, 3 minutes running, in the slowest
quarter and above twice everyone's (`src/kalshi/feed.rs` measures, the
orchestrator judges). In 20 live minutes on the demo (120 connections) no
connection was slow by that rule; the output reconciled (25 mismatches,
each a logged loss).
**Diverged from the plan**, each written into this file when it happened:
the probe is a subcommand, not `src/bin/latency.rs` (what it reuses belongs
to the binary); IPv4 addresses only (no IPv6 route); no source-port run (a
NAT rewrites the port); Kalshi's pick by a minute's count, first over the
listing's first 30,000 markets, then over every open one, after the
listing's order moved and the second run counted one market; the slow
recycle by mean lateness and only past twice the mean, after the review
showed that by median, a slowest quarter alone recycled equal connections
about once a minute.
**Left out**, and where it went: first after maintenance, the source port
from a public host, the slow recycle's gain on Kalshi production, and
judging at the book threads' minute (`docs/roadmap.md`, possible
features); IPv6, kernel tuning and our own added latency (`LATENCY.md`,
not tried, each with how). Kalshi production: no key, as in sprint 06.
**Assumed**, not measured: that what sets a connection's speed is the
backend behind the address; that the collector's connections differ less
than the probe's because each is judged against many partners' copies;
that location would gain the most.
**Reviewed** at `d722446`; the 11 fixes after it are the review's own
tasks, each with a test seen to fail, and had no second review.
### Retrospective
- **Went well:** one question under the brief's three tricks made the
experiments few and the writeup short. A threshold agreed before the
data kept the verdict honest: Polymarket's 29% looked like a win until
the repeat run. Raw CSVs let every new question (the pair's median, the
missed counts) be asked of runs already made. Mutating each test's code
one at a time found holes in my test data four times.
- **Went wrong:** the Kalshi pick depended on the listing's order, which
cost a 10-minute run; a pick should not rest on an order the exchange
doesn't promise. I designed the collector's slow rule on the probe's
median without asking what the median means with two copies of each
change: the review and then the fix found it, and the final rule never
fired live, so its value on the demo is unshown. Three chained shell
commands were refused whole in the worktree session, and once that left
a commit amended without its CI run; caught and redone.
- **Tested enough?** The probe's identities, the analysis and the slow
rule are, by removal. Not: the probe's network paths and the
orchestrator loop's wiring (live runs only), and the slow recycle firing
live.
- **Next time:** before reusing a statistic in a new place, work out its
value on that place's data in one line. A live check of a mechanism
should show it acting, or say plainly that it didn't.