This repository has been archived on 2026-11-16. You can view files and clone it. You cannot open issues or pull requests or push a commit.
Files
orderbook-collector/roadmap/07. latency experiments.md
pascal 515e85941d feat(latency): latency experiments, LATENCY.md, Kalshi's slow-connection recycle
Extension 4. `engineering-challenge latency EXCHANGE OUT.csv` opens 16
connections to the same busy markets and writes every copy of every change;
`scripts/latency/analyse.py` gives each connection's speed, whether it
persists, what keeping the fastest would gain, lateness by address, and lag
with the clock's offset. LATENCY.md writes up four runs: on Polymarket speed
persists but is worth under a millisecond, at the median only; on Kalshi's
demo keeping the fastest pair lowers p99 by 28% and 42%, the threshold
agreed before the runs.

So the collector now recycles a Kalshi connection whose mean lateness stays,
3 minutes running, in the slowest quarter and above twice everyone's, ahead
of its hourly turn. Kalshi's rule measures each duplicate's lateness; the
orchestrator judges once a minute. A duplicate's lag in Polymarket's minute
line is now its "lateness".

Roadmap milestone 7, roadmap/07.
2026-11-12 15:05:25 +01:00

20 KiB
Raw Permalink Blame History

07. Latency experiments

Status: done (2026-11-12)

Extension 4 of the brief: "experimentation into how to [get faster connections] on various exchanges like Polymarket or Kalshi", written up in a markdown file. Correctness stays the first priority, so the collector changes only if the data clears a threshold agreed here. The question under every trick the brief names (racing connections, the source port, being first after maintenance) is one: is a connection that is fast now still fast later? If speed is noise per message, no choice of connection pays. Terms: ../GLOSSARY.md (lateness, lag, speed, duplicate).

The probe

src/latency.rs, a subcommand of the collector (engineering-challenge latency <polymarket|kalshi|kalshi-demo> <out.csv>), kept in the repo because LATENCY.md cites its numbers. It is not the collector: no book, no rule, no reconnect. A subcommand, not src/bin/latency.rs as first agreed, because what it reuses (core, polymarket, kalshi, log!) belongs to the collector's binary, not the library; moving them was rejected as a large diff that buys nothing else.

  • Markets: the same 100 on every connection, few enough that our own load stays out of the measurement. Polymarket: the 100 live markets with the highest volume24hr (Gamma sorts by it: order=volume24hr&ascending=false, checked), all their tokens. Kalshi: the demo reports volume_24h_fp 0.00 (all of the first 1,000 open markets, checked), so the probe counts every open market's changes for a minute, 30,000 per connection, and keeps the 100 with the most; it stops if fewer than 10 changed. Production would be picked the same way, one rule for both Kalshi deployments. Supersedes the pick by traded amount on Kalshi, because the demo has none; and then the count over the listing's first 30,000 only, because the listing's order moves: the 98 busiest of 11:12 were past its first 40,000 at 11:50, and the second Kalshi run counted 1 market that changed.
  • Connections: 16 per run, one plain thread each, tungstenite over a TcpStream the probe opens to an address it chooses, round-robin over the exchange's IPv4 addresses (the workspace has no IPv6 route: the smoke run's connections to Polymarket's IPv6 ones failed with "Network is unreachable"; Polymarket's websocket has 4 IPv4 ones, the Kalshi demo's 2, production's 1, checked 2026-11-12; the collector connects only to the first, src/core/socket.rs:184). It reuses Exchange::request (Kalshi's signed handshake), the Session's subscribe, keepalive and decode (src/core/mod.rs:83 to 89), and the collector's connect with the address given (split out of src/core/socket.rs's connect); a refused handshake (429) is tried again through core::retried, 3 tries 1 s apart. The receive time is stamped right after the read, as in the collector. A connection that drops is logged and stays down: a run is 10 minutes.
  • Output: one CSV line per copy of a change, raw: connection, address, symbol, the change's identity, receive time (µs), exchange timestamp (ms). The analysis matches copies across connections; the probe computes nothing.
  • A change's identity across connections: on Polymarket the symbol and the change's book hash, side, price and size (the rule's own copy test, Held::key, src/polymarket/feed.rs:89); on Kalshi the market, clock, side, price and delta, plus the tuple's occurrence number on its connection, since 2 to 4 changes can share a clock, some identical (docs/notes.md).
  • Clock: an SNTP query to time.cloudflare.com at the start and end of each run, logged; lag is reported raw and corrected. sntpc 0.11 (sync, std) with sntpc-net-std 1.3, tried: offset +0.5 ms, round trip 3 ms from the server (2026-11-12).

The analysis

scripts/latency/analyse.py, standard library only (statistics.correlation with method="ranked" is Spearman, Python 3.13 here). From a run's CSV:

  • each copy's lateness, a server's repeat on one connection counted once; a change a connection never delivered while it was up (from its first copy until it went down) counts as missed by it;
  • each connection's speed per minute, and its missed count;
  • persistence: the rank correlation of the 16 speeds between each minute and the next; and the 4 fastest of minute 1 against the other 12 over the full minutes after it, at the median and p99 of lateness;
  • racing: the 2 fastest of minute 1 kept, against 2 at random (averaged over every pair of the 16), over the full minutes after it, at the median and p99 of lateness, with each pair's missed changes; a run too thin to compare is said so;
  • by address: lateness per edge address;
  • lag: per exchange, raw and corrected by the offset.

The runs

Two 10-minute runs per exchange, alternated (Polymarket, Kalshi, Polymarket, Kalshi), /proc/loadavg recorded before each and the run put off while the load is high. The second run repeats the first, which the adoption threshold needs. Raw CSVs and logs stay out of the repo, in ../runs/latency/.

The writeup

LATENCY.md at the repo root, a page or two: the method (the probe, lateness against lag, the clock's offset), one section per experiment (question, a small table per exchange, verdict), and the ideas not tried, each with why and how one would: first after maintenance, the source port, location, kernel tuning, IPv6, our own added latency. Kalshi's numbers are labelled the demo's. What was not measured is written "assumed".

Decisions

  • Lateness, not lag, ranks connections. Lateness compares copies of one change on one machine and needs no clock; lag carries the clock's offset and the exchange's own stamping. Rejected: ranking by lag.
  • Raw CSV, analysis apart. A new question is asked of a run without running it again; live runs are what cost time on a shared server. Rejected: summaries computed by the probe.
  • Racing is computed from the persistence run, not run as an open-close loop: choosing after the fact shows the same gain as long as a connection's speed is set when it connects, which is what persistence tests. Rejected: a separate racing run.
  • The source port is not tested. At least one NAT between the workspace and the internet rewrites it: local ports 40123, 40124 and 41000 to 41003 left as 50990, 53878, 36996, 48950, 51760, 60434, and 41000 again a minute later as 37086 (ifconfig.me, 2026-11-12). The probe binds no port. Supersedes the scope's "same (address, port) pairs" run.
  • Adoption threshold, agreed before the data: the collector changes on an exchange only if speed persists (rank correlation above about 0.5 across the run) and the 2 picked connections' p99 lateness is at least 20% below the random pair's, in both runs. The change would then be to recycle a connection whose speed stayed among the slowest for several minutes (the stretch goal). Otherwise the writeup says the threshold was not met. Met on Kalshi's demo (28% and 42%), not on Polymarket (29%, then -4%).
  • Recycling the slow connections (Kalshi), agreed after the runs, for showing the loop from data to code and as one more self-correcting mechanism, knowing the evidence is the demo's: Kalshi's rule keeps the first copy's receive time and connection of its last 128 positions per symbol, forgotten when a fresh book replaces ours; a duplicate records its lateness for its connection, and 0 for the first copy's (the later stamp is the late one, whichever reached the book thread first); a third copy, while a market moves, is not measured. Each book thread reports the samples at the minute (Report::Lateness); Polymarket's rule reports none, so nothing changes there. The orchestrator, once a minute, ranks the connections with at least 100 samples and up for 10 minutes by their mean lateness; those of the slowest quarter whose mean is more than twice the mean of every judged copy count a slow minute, the rest start over; one slow 3 minutes in a row is recycled at once, ahead of the hourly order, at most one a minute and under the hourly recycle's rule (one at a time, every connection up, the last up an interval ago). Rejected: building it in the core for both exchanges, since Polymarket's p99 doesn't follow speed. The mean, not the median, and the factor of two, supersede the first design (by median, the slowest quarter alone), because of the review: a slowest quarter always exists, so equal connections were recycled about once a minute; and with two copies of each change, a connection first more than half the time has a median of 0, so a median says how often it comes second, not how late, and twice the median of medians is about 0. The mean holds both, and the tail where Kalshi's gain was. The probe's speed (the median, GLOSSARY.md) is unchanged.
  • Kalshi on the demo. No production key; the code is the same, only whether the demo behaves like production is unknown.
  • lags becomes lateness in Polymarket's rule (Kalshi's measures none), with the log text "second copies … late" read as "duplicates' lateness … at the median"; nothing parses that line (src/cause.rs, src/reconcile.rs).
  • Our own added latency (receive time to output write) is not measured: the receive time is stamped before any queue, and the brief's extension is about the connection.

Tasks

  • Rename lags to lateness in src/polymarket/feed.rs, comments and log text included (Kalshi's rule measures none); the rules' tests pass unchanged.
  • The probe's change identities, for both exchanges.
  • Test: a Polymarket change on two connections gets the same identity; one differing only in its hash does not.
  • Test: two identical Kalshi changes under one clock on one connection get two identities, the same two as on another connection.
  • The probe's market pick (Polymarket by volume24hr, Kalshi by a minute's count) and its connections (address round-robin, request, subscribe, keepalive, stamping, CSV), with the SNTP offset at start and end.
  • A 1-minute smoke run of the probe on each exchange: 16 connections up, every connection's lines in the CSV, the offset logged.
  • scripts/latency/analyse.py: lateness, missed, speed per minute, persistence, racing, by address, lag.
  • Test (scripts/latency/test_analyse.py, from scripts/ci.sh): on a small CSV worked out by hand, the first copy's lateness is 0, a change one connection lacks counts as missed by it, and the speeds, ranks and racing figures are the hand-computed ones.
  • The four runs, alternated, load and offset recorded.
  • LATENCY.md.
  • Stretch (threshold met on Kalshi): Kalshi's rule records each duplicate's lateness and the first copy's 0, reported at the minute.
  • Test: a duplicate's lateness is reported for its connection and 0 for the first copy's; a change with one copy reports nothing.
  • The tracker's judgment and the orchestrator's recycle of a slow connection, logged.
  • Test: slowest for 3 minutes is recycled; slowest 2 minutes then not starts over; a connection up less than 10 minutes or with too few samples is not judged; never one while another recycle is under way.
  • A 20-minute Kalshi demo run (a connection is judged from its 10th minute, recycled after 3 slow ones), reconciled, its log showing the slow recycles.
  • CLAUDE.md (the probe, the new terms, the NAT trap) and docs/roadmap.md brought up to date.

From the review of d722446:

  • A slow recycle waits, as the hourly one does, an interval after the last connection came up (its opening books), with its test.
  • A slow connection is ranked by its mean lateness and recycled only if that mean is more than twice the mean of every judged copy: equal connections are left alone, and the hourly recycle isn't starved. Its test (equal connections, none recycled).
  • Kalshi's lateness: a copy stamped before the one applied first swaps the two (no negative lateness); the remembered positions are cleared when a fresh book replaces ours. Tests for both.
  • Tests: the tracker's mean against samples that differ (a median of 0, a high mean); two book threads' samples merged; the 128-position cap; judging refused at 599 s.
  • Tests: Kalshi's identity on two interleaved markets; Polymarket's told apart by size.
  • The stop below 10 counted markets moved into busiest, with its test.
  • The probe reuses the Kalshi listing's paging and agent (src/kalshi/listing.rs, made pub(crate)).
  • RECENT's comment names its upgrade path; renames: counting (it means counting is over), SLOW_SHARE's doc, struct Copy, Tracker::lateness.
  • The analysis: no traceback on thin runs (4 or fewer connections in the first minute, a kept pair silent after it, one connection); tests for the p99 against the maximum (200 values), report() on the hand-worked run (racing's gain and the Missed column), racing's missed counts, and an offset note with no value.
  • LATENCY.md: the two Kalshi runs' picks; persistence read on the mean (minutes at 0.47 and 0.49 said); the slowest 177 ms; the second run's length explained right (the first's stop counted from start: 9 min 52 s of probing, the second 10 min 2 s); no "two orders of magnitude" for Kalshi's p99; "behind the address" marked assumed; the collector run's load 7.3; how one would try IPv6, kernel tuning and our own added latency.
  • This file: minutes 2 to the end, a missed count (while up) rather than a share, and the third copy unmeasured, in the Design.

Each test is seen to fail once (the code it protects removed, then put back).

Review of d722446 (2026-11-12)

Five reviewers (spec, bugs, tests, smells, security); 2 findings dropped (the tests were seen to fail; the memory bound is 128 positions a symbol). Every one below was decided "fix now" (the tasks above) unless said otherwise.

  • Bugs: the slow recycle skipped the hourly one's settling wait; it never stops (a slowest quarter always exists) and can starve the hourly one; the analysis crashed on thin runs; lateness could be negative or measured against a stale position. Later: the judgment runs on the orchestrator's timer, not the book threads' minute (partial samples make it more conservative, not unsafe). No: the probe ignores SIGINT during the listing scan and never notes a silent connection (a tool run by hand; missed counts were 0 to 5).
  • Tests: 9 gaps verified one mutant at a time (p99 and the tracker's median against the maximum, report(), samples merged, the cap, racing's missed, the offset note, Kalshi's identity across markets, Polymarket's size), the stop below 10 and the 599 s boundary. No: the CSV contract across the two languages, the IPv4 filter and round-robin, the tie-break, the loop's wiring (live runs show them).
  • Smells: the probe's copies of the listing's paging and agent; the RECENT comment; four names. No: Rule::take_lateness (a hook on a trait with two implementations), the analysis's primitive tuples and repeats, the lateness and median in several places, the probe's match on the exchange, latency.rs's size.
  • Security: no vulnerability. No: the docs name the country and the NAT (part of the findings), unquoted CSV fields (a TLS-authenticated source), unauthenticated SNTP (only the corrected lag).
  • Spec: LATENCY.md's figures and claims, the two picks, the mean, three "how"s, this file's Design. No: the CSV's notes, ruff in CI, the slow count in the minute line, the listing retry (each serves what was asked).

Not in this sprint

Polish and the README (M8); the roadmap's possible features; kernel tuning; anything needing a second machine, another region or a public address; the first connection after Kalshi's maintenance (next window Thursday 2026-11-19, 08:00 to 10:00 UTC), unless the sprint is still open then.

Done when

  • The probe and the analysis are on the branch, their tests seen to fail once.
  • Both exchanges' two runs are done and analysed, load and offset recorded.
  • LATENCY.md gives a verdict per experiment and per exchange, Kalshi labelled the demo.
  • The stretch goal is built and tested if the threshold is met; otherwise the writeup says it wasn't.
  • scripts/ci.sh is green.

Outcome

Extension 4 is answered and written up in LATENCY.md. engineering-challenge latency EXCHANGE OUT.csv opens 16 connections to the same 100 busy markets and writes every copy of every change; scripts/latency/analyse.py turns a run into speeds, persistence, racing, lateness by address and lag. Four 10-minute runs, alternated, the clock's offset under 0.7 ms throughout:

Persistence (mean) Racing, p99 lower by Lag, median
Polymarket 0.96, 0.98 29%, then −4% 14 ms
Kalshi demo 0.66, 0.78 28%, 42% 94 to 106 ms

On Polymarket speed persists but is worth under a millisecond, at the median only: the tail is bursts every connection shares. On Kalshi's demo it met the agreed threshold, so the collector now recycles a Kalshi connection whose mean lateness stays, 3 minutes running, in the slowest quarter and above twice everyone's (src/kalshi/feed.rs measures, the orchestrator judges). In 20 live minutes on the demo (120 connections) no connection was slow by that rule; the output reconciled (25 mismatches, each a logged loss).

Diverged from the plan, each written into this file when it happened: the probe is a subcommand, not src/bin/latency.rs (what it reuses belongs to the binary); IPv4 addresses only (no IPv6 route); no source-port run (a NAT rewrites the port); Kalshi's pick by a minute's count, first over the listing's first 30,000 markets, then over every open one, after the listing's order moved and the second run counted one market; the slow recycle by mean lateness and only past twice the mean, after the review showed that by median, a slowest quarter alone recycled equal connections about once a minute.

Left out, and where it went: first after maintenance, the source port from a public host, the slow recycle's gain on Kalshi production, and judging at the book threads' minute (docs/roadmap.md, possible features); IPv6, kernel tuning and our own added latency (LATENCY.md, not tried, each with how). Kalshi production: no key, as in sprint 06.

Assumed, not measured: that what sets a connection's speed is the backend behind the address; that the collector's connections differ less than the probe's because each is judged against many partners' copies; that location would gain the most.

Reviewed at d722446; the 11 fixes after it are the review's own tasks, each with a test seen to fail, and had no second review.

Retrospective

  • Went well: one question under the brief's three tricks made the experiments few and the writeup short. A threshold agreed before the data kept the verdict honest: Polymarket's 29% looked like a win until the repeat run. Raw CSVs let every new question (the pair's median, the missed counts) be asked of runs already made. Mutating each test's code one at a time found holes in my test data four times.
  • Went wrong: the Kalshi pick depended on the listing's order, which cost a 10-minute run; a pick should not rest on an order the exchange doesn't promise. I designed the collector's slow rule on the probe's median without asking what the median means with two copies of each change: the review and then the fix found it, and the final rule never fired live, so its value on the demo is unshown. Three chained shell commands were refused whole in the worktree session, and once that left a commit amended without its CI run; caught and redone.
  • Tested enough? The probe's identities, the analysis and the slow rule are, by removal. Not: the probe's network paths and the orchestrator loop's wiring (live runs only), and the slow recycle firing live.
  • Next time: before reusing a statistic in a new place, work out its value on that place's data in one line. A live check of a mechanism should show it acting, or say plainly that it didn't.