Archived
Extension 4. `engineering-challenge latency EXCHANGE OUT.csv` opens 16 connections to the same busy markets and writes every copy of every change; `scripts/latency/analyse.py` gives each connection's speed, whether it persists, what keeping the fastest would gain, lateness by address, and lag with the clock's offset. LATENCY.md writes up four runs: on Polymarket speed persists but is worth under a millisecond, at the median only; on Kalshi's demo keeping the fastest pair lowers p99 by 28% and 42%, the threshold agreed before the runs. So the collector now recycles a Kalshi connection whose mean lateness stays, 3 minutes running, in the slowest quarter and above twice everyone's, ahead of its hourly turn. Kalshi's rule measures each duplicate's lateness; the orchestrator judges once a minute. A duplicate's lag in Polymarket's minute line is now its "lateness". Roadmap milestone 7, roadmap/07.
362 lines
20 KiB
Markdown
362 lines
20 KiB
Markdown
# 07. Latency experiments
|
||
|
||
**Status: done (2026-11-12)**
|
||
|
||
Extension 4 of the brief: "experimentation into how to [get faster
|
||
connections] on various exchanges like Polymarket or Kalshi", written up in a
|
||
markdown file. Correctness stays the first priority, so the collector
|
||
changes only if the data clears a threshold agreed here. The question under
|
||
every trick the brief names (racing connections, the source port, being
|
||
first after maintenance) is one: **is a connection that is fast now still
|
||
fast later?** If speed is noise per message, no choice of connection pays.
|
||
Terms: [../GLOSSARY.md](../GLOSSARY.md) (lateness, lag, speed, duplicate).
|
||
|
||
## The probe
|
||
|
||
`src/latency.rs`, a subcommand of the collector (`engineering-challenge
|
||
latency <polymarket|kalshi|kalshi-demo> <out.csv>`), kept in the repo
|
||
because `LATENCY.md` cites its numbers. It is not the collector: no book,
|
||
no rule, no reconnect. A subcommand, not `src/bin/latency.rs` as first
|
||
agreed, because what it reuses (`core`, `polymarket`, `kalshi`, `log!`)
|
||
belongs to the collector's binary, not the library; moving them was
|
||
rejected as a large diff that buys nothing else.
|
||
|
||
- **Markets:** the same 100 on every connection, few enough that our own
|
||
load stays out of the measurement. Polymarket: the 100 live markets with
|
||
the highest `volume24hr` (Gamma sorts by it: `order=volume24hr&ascending=false`,
|
||
checked), all their tokens. Kalshi: the demo reports `volume_24h_fp` 0.00
|
||
(all of the first 1,000 open markets, checked), so the probe counts every
|
||
open market's changes for a minute, 30,000 per connection, and keeps the
|
||
100 with the most; it stops if fewer than 10 changed. Production would be
|
||
picked the same way, one rule for both Kalshi deployments. Supersedes the
|
||
pick by traded amount on Kalshi, because the demo has none; and then the
|
||
count over the listing's first 30,000 only, because the listing's order
|
||
moves: the 98 busiest of 11:12 were past its first 40,000 at 11:50, and
|
||
the second Kalshi run counted 1 market that changed.
|
||
- **Connections:** 16 per run, one plain thread each, tungstenite over a
|
||
`TcpStream` the probe opens to an address it chooses, round-robin over the
|
||
exchange's IPv4 addresses (the workspace has no IPv6 route: the smoke
|
||
run's connections to Polymarket's IPv6 ones failed with "Network is
|
||
unreachable"; Polymarket's websocket has 4 IPv4 ones, the Kalshi demo's 2,
|
||
production's 1, checked 2026-11-12; the collector connects only to the
|
||
first, `src/core/socket.rs:184`). It reuses `Exchange::request` (Kalshi's
|
||
signed handshake), the `Session`'s subscribe, keepalive and decode
|
||
(`src/core/mod.rs:83` to `89`), and the collector's connect with the
|
||
address given (split out of `src/core/socket.rs`'s `connect`); a refused
|
||
handshake (429) is tried again through `core::retried`, 3 tries 1 s apart. The receive time is stamped
|
||
right after the read, as in the collector. A connection that drops is
|
||
logged and stays down: a run is 10 minutes.
|
||
- **Output:** one CSV line per copy of a change, raw: connection, address,
|
||
symbol, the change's identity, receive time (µs), exchange timestamp (ms).
|
||
The analysis matches copies across connections; the probe computes
|
||
nothing.
|
||
- **A change's identity** across connections: on Polymarket the symbol and
|
||
the change's book hash, side, price and size (the rule's own copy test,
|
||
`Held::key`, `src/polymarket/feed.rs:89`); on Kalshi the market,
|
||
`clock`, side, price and delta, plus the tuple's occurrence number on its
|
||
connection, since 2 to 4 changes can share a clock, some identical
|
||
(`docs/notes.md`).
|
||
- **Clock:** an SNTP query to `time.cloudflare.com` at the start and end of
|
||
each run, logged; lag is reported raw and corrected. `sntpc` 0.11
|
||
(`sync`, `std`) with `sntpc-net-std` 1.3, tried: offset +0.5 ms, round
|
||
trip 3 ms from the server (2026-11-12).
|
||
|
||
## The analysis
|
||
|
||
`scripts/latency/analyse.py`, standard library only (`statistics.correlation`
|
||
with `method="ranked"` is Spearman, Python 3.13 here). From a run's CSV:
|
||
|
||
- each copy's lateness, a server's repeat on one connection counted once;
|
||
a change a connection never delivered while it was up (from its first
|
||
copy until it went down) counts as missed by it;
|
||
- each connection's speed per minute, and its missed count;
|
||
- **persistence:** the rank correlation of the 16 speeds between each
|
||
minute and the next; and the 4 fastest of minute 1 against the other 12
|
||
over the full minutes after it, at the median and p99 of lateness;
|
||
- **racing:** the 2 fastest of minute 1 kept, against 2 at random (averaged
|
||
over every pair of the 16), over the full minutes after it, at the
|
||
median and p99 of lateness, with each pair's missed changes; a run too
|
||
thin to compare is said so;
|
||
- **by address:** lateness per edge address;
|
||
- **lag:** per exchange, raw and corrected by the offset.
|
||
|
||
## The runs
|
||
|
||
Two 10-minute runs per exchange, alternated (Polymarket, Kalshi,
|
||
Polymarket, Kalshi), `/proc/loadavg` recorded before each and the run put
|
||
off while the load is high. The second run repeats the first, which the
|
||
adoption threshold needs. Raw CSVs and logs stay out of the repo, in
|
||
`../runs/latency/`.
|
||
|
||
## The writeup
|
||
|
||
`LATENCY.md` at the repo root, a page or two: the method (the probe,
|
||
lateness against lag, the clock's offset), one section per experiment
|
||
(question, a small table per exchange, verdict), and the ideas not tried,
|
||
each with why and how one would: first after maintenance, the source port,
|
||
location, kernel tuning, IPv6, our own added latency. Kalshi's numbers are
|
||
labelled the demo's. What was not measured is written "assumed".
|
||
|
||
## Decisions
|
||
|
||
- **Lateness, not lag, ranks connections.** Lateness compares copies of one
|
||
change on one machine and needs no clock; lag carries the clock's offset
|
||
and the exchange's own stamping. Rejected: ranking by lag.
|
||
- **Raw CSV, analysis apart.** A new question is asked of a run without
|
||
running it again; live runs are what cost time on a shared server.
|
||
Rejected: summaries computed by the probe.
|
||
- **Racing is computed from the persistence run**, not run as an
|
||
open-close loop: choosing after the fact shows the same gain as long as a
|
||
connection's speed is set when it connects, which is what persistence
|
||
tests. Rejected: a separate racing run.
|
||
- **The source port is not tested.** At least one NAT between the workspace
|
||
and the internet rewrites it: local ports 40123, 40124 and 41000 to 41003
|
||
left as 50990, 53878, 36996, 48950, 51760, 60434, and 41000 again a minute
|
||
later as 37086 (`ifconfig.me`, 2026-11-12). The probe binds no port.
|
||
Supersedes the scope's "same (address, port) pairs" run.
|
||
- **Adoption threshold**, agreed before the data: the collector changes on
|
||
an exchange only if speed persists (rank correlation above about 0.5
|
||
across the run) and the 2 picked connections' p99 lateness is at least
|
||
20% below the random pair's, in both runs. The change would then be to
|
||
recycle a connection whose speed stayed among the slowest for several
|
||
minutes (the stretch goal). Otherwise the writeup says the threshold was
|
||
not met. Met on Kalshi's demo (28% and 42%), not on Polymarket (29%, then
|
||
-4%).
|
||
- **Recycling the slow connections (Kalshi)**, agreed after the runs, for
|
||
showing the loop from data to code and as one more self-correcting
|
||
mechanism, knowing the evidence is the demo's: Kalshi's rule keeps the
|
||
first copy's receive time and connection of its last 128 positions per
|
||
symbol, forgotten when a fresh book replaces ours; a duplicate records
|
||
its lateness for its connection, and 0 for the first copy's (the later
|
||
stamp is the late one, whichever reached the book thread first); a
|
||
third copy, while a market moves, is not measured. Each book thread reports the samples at the minute
|
||
(`Report::Lateness`); Polymarket's rule reports none, so nothing changes
|
||
there. The orchestrator, once a minute, ranks the connections with at
|
||
least 100 samples and up for 10 minutes by their mean lateness; those
|
||
of the slowest quarter whose mean is more than twice the mean of every
|
||
judged copy count a slow minute, the rest start over; one slow 3
|
||
minutes in a row is recycled at once, ahead of the hourly order, at most
|
||
one a minute and under the hourly recycle's rule (one at a time, every
|
||
connection up, the last up an interval ago). Rejected: building it in
|
||
the core for both exchanges, since Polymarket's p99 doesn't follow
|
||
speed. The mean, not the median, and the factor of two, supersede the
|
||
first design (by median, the slowest quarter alone), because of the
|
||
review: a slowest quarter always exists, so equal connections were
|
||
recycled about once a minute; and with two copies of each change, a
|
||
connection first more than half the time has a median of 0, so a median
|
||
says how often it comes second, not how late, and twice the median of
|
||
medians is about 0. The mean holds both, and the tail where Kalshi's
|
||
gain was. The probe's speed (the median, GLOSSARY.md) is unchanged.
|
||
- **Kalshi on the demo.** No production key; the code is the same, only
|
||
whether the demo behaves like production is unknown.
|
||
- **`lags` becomes `lateness`** in Polymarket's rule (Kalshi's measures
|
||
none), with the log text "second copies … late" read as "duplicates'
|
||
lateness … at the median"; nothing parses that line (`src/cause.rs`,
|
||
`src/reconcile.rs`).
|
||
- **Our own added latency** (receive time to output write) is not measured:
|
||
the receive time is stamped before any queue, and the brief's extension
|
||
is about the connection.
|
||
|
||
## Tasks
|
||
|
||
- [x] Rename `lags` to `lateness` in `src/polymarket/feed.rs`, comments
|
||
and log text included (Kalshi's rule measures none); the rules' tests
|
||
pass unchanged.
|
||
- [x] The probe's change identities, for both exchanges.
|
||
- [x] Test: a Polymarket change on two connections gets the same identity;
|
||
one differing only in its hash does not.
|
||
- [x] Test: two identical Kalshi changes under one clock on one connection
|
||
get two identities, the same two as on another connection.
|
||
- [x] The probe's market pick (Polymarket by `volume24hr`, Kalshi by a
|
||
minute's count) and its connections (address round-robin, request,
|
||
subscribe, keepalive, stamping, CSV), with the SNTP offset at start
|
||
and end.
|
||
- [x] A 1-minute smoke run of the probe on each exchange: 16 connections
|
||
up, every connection's lines in the CSV, the offset logged.
|
||
- [x] `scripts/latency/analyse.py`: lateness, missed, speed per minute,
|
||
persistence, racing, by address, lag.
|
||
- [x] Test (`scripts/latency/test_analyse.py`, from `scripts/ci.sh`): on a
|
||
small CSV worked out by hand, the first copy's lateness is 0, a change
|
||
one connection lacks counts as missed by it, and the speeds, ranks
|
||
and racing figures are the hand-computed ones.
|
||
- [x] The four runs, alternated, load and offset recorded.
|
||
- [x] `LATENCY.md`.
|
||
- [x] Stretch (threshold met on Kalshi): Kalshi's rule records each
|
||
duplicate's lateness and the first copy's 0, reported at the minute.
|
||
- [x] Test: a duplicate's lateness is reported for its connection and 0
|
||
for the first copy's; a change with one copy reports nothing.
|
||
- [x] The tracker's judgment and the orchestrator's recycle of a slow
|
||
connection, logged.
|
||
- [x] Test: slowest for 3 minutes is recycled; slowest 2 minutes then not
|
||
starts over; a connection up less than 10 minutes or with too few
|
||
samples is not judged; never one while another recycle is under way.
|
||
- [x] A 20-minute Kalshi demo run (a connection is judged from its 10th
|
||
minute, recycled after 3 slow ones), reconciled, its log showing the slow
|
||
recycles.
|
||
- [x] `CLAUDE.md` (the probe, the new terms, the NAT trap) and
|
||
`docs/roadmap.md` brought up to date.
|
||
|
||
From the review of `d722446`:
|
||
|
||
- [x] A slow recycle waits, as the hourly one does, an interval after the
|
||
last connection came up (its opening books), with its test.
|
||
- [x] A slow connection is ranked by its mean lateness and recycled only
|
||
if that mean is more than twice the mean of every judged copy: equal
|
||
connections are left alone, and the hourly recycle isn't starved.
|
||
Its test (equal connections, none recycled).
|
||
- [x] Kalshi's lateness: a copy stamped before the one applied first
|
||
swaps the two (no negative lateness); the remembered positions are
|
||
cleared when a fresh book replaces ours. Tests for both.
|
||
- [x] Tests: the tracker's mean against samples that differ (a median of
|
||
0, a high mean); two book
|
||
threads' samples merged; the 128-position cap; judging refused at
|
||
599 s.
|
||
- [x] Tests: Kalshi's identity on two interleaved markets; Polymarket's
|
||
told apart by size.
|
||
- [x] The stop below 10 counted markets moved into `busiest`, with its test.
|
||
- [x] The probe reuses the Kalshi listing's paging and agent
|
||
(`src/kalshi/listing.rs`, made `pub(crate)`).
|
||
- [x] `RECENT`'s comment names its upgrade path; renames: `counting` (it
|
||
means counting is over), `SLOW_SHARE`'s doc, `struct Copy`,
|
||
`Tracker::lateness`.
|
||
- [x] The analysis: no traceback on thin runs (4 or fewer connections in
|
||
the first minute, a kept pair silent after it, one connection);
|
||
tests for the p99 against the maximum (200 values), `report()` on
|
||
the hand-worked run (racing's gain and the Missed column), racing's
|
||
missed counts, and an offset note with no value.
|
||
- [x] `LATENCY.md`: the two Kalshi runs' picks; persistence read on the
|
||
mean (minutes at 0.47 and 0.49 said); the slowest 177 ms; the
|
||
second run's length explained right (the first's stop counted from
|
||
start: 9 min 52 s of probing, the second 10 min 2 s); no "two orders
|
||
of magnitude" for Kalshi's p99; "behind the address" marked assumed;
|
||
the collector run's load 7.3; how one would try IPv6, kernel tuning
|
||
and our own added latency.
|
||
- [x] This file: minutes 2 to the end, a missed count (while up) rather
|
||
than a share, and the third copy unmeasured, in the Design.
|
||
|
||
Each test is seen to fail once (the code it protects removed, then put
|
||
back).
|
||
|
||
## Review of `d722446` (2026-11-12)
|
||
|
||
Five reviewers (spec, bugs, tests, smells, security); 2 findings dropped
|
||
(the tests were seen to fail; the memory bound is 128 positions a
|
||
symbol). Every one below was decided "fix now" (the tasks above) unless
|
||
said otherwise.
|
||
|
||
- **Bugs:** the slow recycle skipped the hourly one's settling wait; it
|
||
never stops (a slowest quarter always exists) and can starve the hourly
|
||
one; the analysis crashed on thin runs; lateness could be negative or
|
||
measured against a stale position. Later: the judgment runs on the
|
||
orchestrator's timer, not the book threads' minute (partial samples make
|
||
it more conservative, not unsafe). No: the probe ignores SIGINT during
|
||
the listing scan and never notes a silent connection (a tool run by
|
||
hand; missed counts were 0 to 5).
|
||
- **Tests:** 9 gaps verified one mutant at a time (p99 and the tracker's
|
||
median against the maximum, `report()`, samples merged, the cap, racing's
|
||
missed, the offset note, Kalshi's identity across markets, Polymarket's
|
||
size), the stop below 10 and the 599 s boundary. No: the CSV contract
|
||
across the two languages, the IPv4 filter and round-robin, the tie-break,
|
||
the loop's wiring (live runs show them).
|
||
- **Smells:** the probe's copies of the listing's paging and agent; the
|
||
`RECENT` comment; four names. No: `Rule::take_lateness` (a hook on a
|
||
trait with two implementations), the analysis's primitive tuples and
|
||
repeats, the lateness and median in several places, the probe's match
|
||
on the exchange, `latency.rs`'s size.
|
||
- **Security:** no vulnerability. No: the docs name the country and the
|
||
NAT (part of the findings), unquoted CSV fields (a TLS-authenticated
|
||
source), unauthenticated SNTP (only the corrected lag).
|
||
- **Spec:** `LATENCY.md`'s figures and claims, the two picks, the mean,
|
||
three "how"s, this file's Design. No: the CSV's notes, ruff in CI, the
|
||
slow count in the minute line, the listing retry (each serves what was
|
||
asked).
|
||
|
||
## Not in this sprint
|
||
|
||
Polish and the README (M8); the roadmap's possible features; kernel
|
||
tuning; anything needing a second machine, another region or a public
|
||
address; the first connection after Kalshi's maintenance (next window
|
||
Thursday 2026-11-19, 08:00 to 10:00 UTC), unless the sprint is still open
|
||
then.
|
||
|
||
## Done when
|
||
|
||
- The probe and the analysis are on the branch, their tests seen to fail
|
||
once.
|
||
- Both exchanges' two runs are done and analysed, load and offset recorded.
|
||
- `LATENCY.md` gives a verdict per experiment and per exchange, Kalshi
|
||
labelled the demo.
|
||
- The stretch goal is built and tested if the threshold is met; otherwise
|
||
the writeup says it wasn't.
|
||
- `scripts/ci.sh` is green.
|
||
|
||
## Outcome
|
||
|
||
Extension 4 is answered and written up in `LATENCY.md`. `engineering-challenge
|
||
latency EXCHANGE OUT.csv` opens 16 connections to the same 100 busy markets
|
||
and writes every copy of every change; `scripts/latency/analyse.py` turns a
|
||
run into speeds, persistence, racing, lateness by address and lag. Four
|
||
10-minute runs, alternated, the clock's offset under 0.7 ms throughout:
|
||
|
||
| | Persistence (mean) | Racing, p99 lower by | Lag, median |
|
||
|---|---|---|---|
|
||
| Polymarket | 0.96, 0.98 | 29%, then −4% | 14 ms |
|
||
| Kalshi demo | 0.66, 0.78 | 28%, 42% | 94 to 106 ms |
|
||
|
||
On Polymarket speed persists but is worth under a millisecond, at the
|
||
median only: the tail is bursts every connection shares. On Kalshi's demo
|
||
it met the agreed threshold, so the collector now recycles a Kalshi
|
||
connection whose mean lateness stays, 3 minutes running, in the slowest
|
||
quarter and above twice everyone's (`src/kalshi/feed.rs` measures, the
|
||
orchestrator judges). In 20 live minutes on the demo (120 connections) no
|
||
connection was slow by that rule; the output reconciled (25 mismatches,
|
||
each a logged loss).
|
||
|
||
**Diverged from the plan**, each written into this file when it happened:
|
||
the probe is a subcommand, not `src/bin/latency.rs` (what it reuses belongs
|
||
to the binary); IPv4 addresses only (no IPv6 route); no source-port run (a
|
||
NAT rewrites the port); Kalshi's pick by a minute's count, first over the
|
||
listing's first 30,000 markets, then over every open one, after the
|
||
listing's order moved and the second run counted one market; the slow
|
||
recycle by mean lateness and only past twice the mean, after the review
|
||
showed that by median, a slowest quarter alone recycled equal connections
|
||
about once a minute.
|
||
|
||
**Left out**, and where it went: first after maintenance, the source port
|
||
from a public host, the slow recycle's gain on Kalshi production, and
|
||
judging at the book threads' minute (`docs/roadmap.md`, possible
|
||
features); IPv6, kernel tuning and our own added latency (`LATENCY.md`,
|
||
not tried, each with how). Kalshi production: no key, as in sprint 06.
|
||
|
||
**Assumed**, not measured: that what sets a connection's speed is the
|
||
backend behind the address; that the collector's connections differ less
|
||
than the probe's because each is judged against many partners' copies;
|
||
that location would gain the most.
|
||
|
||
**Reviewed** at `d722446`; the 11 fixes after it are the review's own
|
||
tasks, each with a test seen to fail, and had no second review.
|
||
|
||
### Retrospective
|
||
|
||
- **Went well:** one question under the brief's three tricks made the
|
||
experiments few and the writeup short. A threshold agreed before the
|
||
data kept the verdict honest: Polymarket's 29% looked like a win until
|
||
the repeat run. Raw CSVs let every new question (the pair's median, the
|
||
missed counts) be asked of runs already made. Mutating each test's code
|
||
one at a time found holes in my test data four times.
|
||
- **Went wrong:** the Kalshi pick depended on the listing's order, which
|
||
cost a 10-minute run; a pick should not rest on an order the exchange
|
||
doesn't promise. I designed the collector's slow rule on the probe's
|
||
median without asking what the median means with two copies of each
|
||
change: the review and then the fix found it, and the final rule never
|
||
fired live, so its value on the demo is unshown. Three chained shell
|
||
commands were refused whole in the worktree session, and once that left
|
||
a commit amended without its CI run; caught and redone.
|
||
- **Tested enough?** The probe's identities, the analysis and the slow
|
||
rule are, by removal. Not: the probe's network paths and the
|
||
orchestrator loop's wiring (live runs only), and the slow recycle firing
|
||
live.
|
||
- **Next time:** before reusing a statistic in a new place, work out its
|
||
value on that place's data in one line. A live check of a mechanism
|
||
should show it acting, or say plainly that it didn't.
|