# Latency experiments Extension 4 of the challenge: ways to get the exchanges' changes sooner, tried on Polymarket and Kalshi. Every trick the brief names (connecting first after maintenance, spamming connections until a fast one comes, changing the TCP source port) rests on one question: **is a connection that is fast now still fast later?** If its speed is noise from message to message, no choice of connection pays. That is what was measured, from a server in Switzerland, on 2026-11-12. ## Method `engineering-challenge latency EXCHANGE OUT.csv` (`src/latency.rs`) opens 16 connections to the same 100 busy markets, spread evenly over the websocket host's addresses, and writes one line per copy of a change: which connection delivered it, from which address, and when (stamped right after the socket read, as the collector does). `scripts/latency/analyse.py` matches the copies of each change across connections. - **Lateness**: how long after a change's first copy a connection delivered it. It compares receive times on one machine, so it needs no clock. A connection's **speed** is its median lateness over a minute. - **Lag**: receive time minus the exchange's timestamp. It carries the local clock's error, so the probe asks `time.cloudflare.com` (SNTP) for the offset at the start and end of each run: −0.25 to +0.69 ms, with 3 to 4 ms round trips. The clock is not what limits the figures below. - **Markets**: on Polymarket the 100 most traded over 24 hours (Gamma's `volume24hr`), 200 tokens. Kalshi's demo reports no traded amount, so the markets that changed most in a minute are kept: in the first run among the listing's first 30,000 open ones, in the second among every open one (267,691), after the listing's order had moved and the first 30,000 held one market that changed. - **Runs**: 10 minutes each, alternated (Polymarket, Kalshi, Polymarket, Kalshi), the server's load between 0.4 and 2.7 on 12 cores before each. Kalshi is its demo: no production key. The demo's servers are in AWS us-west-2 (Oregon) and production's in us-east-2 (Ohio) (AWS's published ranges), so how far the demo's figures hold for production is unknown. - **Agreed before the runs**: the collector changes only if, on an exchange, speed persists (rank correlation of the connections' speeds from one minute to the next above about 0.5, read on the run's mean) and keeping the 2 fastest connections of the first minute lowers the p99 of the pair's lateness (the sooner of its two copies, as first one wins takes it) by at least 20% against a pair at random, in both runs. ## Does a connection's speed persist? | Run | Changes | Rank correlation, minute to minute (mean) | Speeds in a minute | |---|---|---|---| | Polymarket 1 | 450,096 | 0.91 to 0.99 (0.96) | 0.2 to 2.5 ms | | Polymarket 2 | 541,933 | 0.96 to 1.00 (0.98) | 0.04 to 3.3 ms | | Kalshi demo 1 | 540,574 | 0.47 to 0.86 (0.66) | 0 to 177 ms | | Kalshi demo 2 | 600,046 | 0.67 to 0.91 (0.78) | 0.04 to 136 ms | **On Polymarket, yes, almost perfectly.** A connection keeps its place for the whole run: in the second run connection 6 was 0.04 to 0.1 ms behind the first copy in 7 of its 9 minutes, connection 3 about 3 ms in all 9. But the spread is a few milliseconds, and the slow moments are shared: the p99 of lateness is 25 to 54 ms on every address alike, and the 4 fastest connections of the first minute had a p99 barely below the others' after it (32 against 43 ms in the first run, 21 against 22 in the second), bursts that delay all 16 at once. Speed seems to be decided when a connection opens (four connections to each address, fast and slow ones on all four), and it can change mid-way: connection 11 of the first run went from 0.35 to 1 ms in its fifth minute. **On Kalshi's demo, partly.** Connections differ by tens of milliseconds up to 177, and the ranks hold less well (0.66 and 0.78; two minutes of the first run fell to 0.47 and 0.49, so the threshold is met on the mean, not in every minute). Some connections settle at 0 for minutes (12 from its third minute in the first run): they became the one that delivers first. ## Racing: keep the fastest Choosing after the fact is what opening many connections and keeping the fast ones would do, provided speed is set when a connection opens; the persistence above says it largely is. The 2 fastest of the first minute, against a pair at random (all 120 pairs, averaged), over the minutes after (8, and 9 in Kalshi's second run: the first run's stop was timed from its start, so its listing and count left 9 min 52 s of probing; the second's came 10 minutes after its count, 10 min 2 s): | Run | Kept pair, median / p99 | Pair at random, median / p99 | p99 lower by | |---|---|---|---| | Polymarket 1 | 0.20 / 8.79 ms | 0.60 / 12.43 ms | 29% | | Polymarket 2 | 0.00 / 13.96 ms | 0.95 / 13.38 ms | −4% | | Kalshi demo 1 | 20.15 / 233.59 ms | 28.91 / 324.56 ms | 28% | | Kalshi demo 2 | 0.16 / 164.25 ms | 9.24 / 283.20 ms | 42% | No change was missed by any pair. **Polymarket: racing works at the median and not in the tail.** The kept pair is ahead of a random one by under a millisecond every time, but the p99 gain vanished in the second run: the tail is the bursts every connection shares. The threshold is not met, and the collector stays as it is: two connections per market, chosen without regard to speed. **Kalshi's demo: racing works in both.** The p99 falls by 28% and 42%, the median from 29 to 20 ms and from 9 to 0.2 ms: there, a connection's speed is worth choosing, and the threshold is met. Whether production behaves alike is unknown. ## Does the address matter? Polymarket's websocket host resolves to 4 CloudFront addresses (and 4 IPv6 ones, unreachable from this server), Kalshi demo's to 2. Lateness by address, all copies of a run: | Run | Medians by address | p99s by address | |---|---|---| | Polymarket 1 | 0.90 to 1.04 ms | 33 to 54 ms | | Polymarket 2 | 1.08 to 1.98 ms | 25 to 36 ms | | Kalshi demo 1 | 40.6 and 45.2 ms | 503 and 548 ms | | Kalshi demo 2 | 16.5 and 23.9 ms | 406 and 418 ms | The address matters much less than the connection. On Polymarket, `143.204.55.29` had the lowest median in both runs, by 0.1 and 0.5 ms, but every address carried both fast and slow connections, and the spread inside an address (0.3 to 2.5 ms in the first run) is larger than between them. On Kalshi's demo the faster of the two addresses changed between runs. Whatever sets a connection's speed is behind the address, assumed to be the CDN's or the load balancer's backend, chosen per connection. The collector connects only to the first address its lookup returns (`src/core/socket.rs`); spreading over all of them would change nothing measured here. ## Lag: how far behind the exchange The first copy's lag, corrected by the clock's offset: | Run | Median | p99 | |---|---|---| | Polymarket 1 | 13.70 ms | 20.21 ms | | Polymarket 2 | 14.55 ms | 25.37 ms | | Kalshi demo 1 | 94.03 ms | 285.20 ms | | Kalshi demo 2 | 106.31 ms | 382.30 ms | These include the exchange's own time between stamping a change and sending it, which can't be told apart from the network from here. Kalshi's demo in Oregon is about 80 ms further than Polymarket, which is one hop to a CloudFront edge (where Polymarket's own servers are was not checked). On Polymarket that is far above what racing wins (under a millisecond). On Kalshi's demo racing wins more in the tail (90 and 119 ms at p99, as much as the median lag), but at the median it wins 9 ms, a tenth of the lag. ## Not tried, and why - **First connection after maintenance.** Kalshi's docs announce maintenance on Thursdays from 3:00 to 5:00 AM ET (08:00 to 10:00 UTC in winter); the sprint ran after one. The test: run the probe at the window's end and an hour later, and compare the speeds of the connections opened first. Polymarket announces no window. - **The TCP source port.** It can't be chosen from here: a NAT between the server and the internet rewrites every connection's port (local 41000 left as 36996, and as 37086 a minute later). Testing it needs a host with a public address: open connections from fixed ports, reconnect from the same ones, and see whether each port's speed returns. Given that speed does not follow the address, a load balancer hashing on the port is the likeliest reason it could. - **Location.** Assumed the largest gain by far: lag is 14 ms to Polymarket and 94 ms to Kalshi's demo from Switzerland, against a millisecond won by racing. A server in the exchange's AWS region (us-east-2 for Kalshi production) would remove most of the lag; not tried, it needs a second machine. - **IPv6.** This server has no IPv6 route. From a host with one, the probe would keep Polymarket's 4 IPv6 addresses in its round-robin (it leaves them out) and the address table would say whether they differ. - **Kernel tuning** (busy polling, receive buffers, pinning threads). The receive time is stamped at the read, the machine's load was low, and nothing above points at the kernel. The test: the same probe run with `SO_BUSY_POLL` set and each connection's thread pinned, alternated with runs without, comparing the first copies' lag on an idle machine. - **Our own added latency**, from the read to the output write: not measured. It doesn't change when a change is stamped. The test: stamp each output again as it is written, and report the difference. - **Kalshi production**: no key. The probe runs on it as `latency kalshi` with production's two key files. ## Verdict - **Connections differ, and keep their speed**: strongly on Polymarket (rank correlation 0.96 and 0.98), less on Kalshi's demo (0.66 and 0.78). Nothing we could choose at connect sets it (the address doesn't): it is drawn per connection, mostly when it opens, though it can shift mid-way (Kalshi's connection 4 went from 75 to 3 ms over its second run). - **On Polymarket it is worth under a millisecond.** Keeping the fastest pair wins at the median, not in the tail, where bursts delay every connection alike. The collector keeps its connections as they are. - **On Kalshi's demo it is worth tens of milliseconds**, in the tail too (p99 28% and 42% lower with the fastest pair kept). This met the threshold agreed before the runs, so the collector now recycles a Kalshi connection that stays slow for 3 minutes running at the hourly recycle's next turn (one every 30 s or so), in place of the oldest: its replacement's speed is drawn afresh. Each duplicate tells how late its connection was against the other's copy (`src/kalshi/feed.rs`); once a minute the orchestrator ranks the connections by their mean lateness, and one is slow if it is in the slowest quarter and more than twice the mean of all (`src/core/orchestrator.rs`). The mean, not the median: with two copies of each change, a connection first more than half the time has a median of 0. Without the factor of two, a slowest quarter always exists: a first version, by median alone, recycled 6 connections in 20 minutes, one a minute, connections no slower than the rest. **In 20 minutes of the collector on the demo (270,000 markets on 120 connections, load 5.7 at the start), none was recycled for it**; the hourly recycle went on (33), and the output reconciled (25 mismatches, each a logged loss). That run doesn't say whether one was judged slow: a later review found the slow recycle waited for the same moment as the hourly one, which nearly always took it first, and only the recycle was logged. Now the slow one takes the hourly recycle's turn, and the judgment is logged on its own. **In 20 minutes on the demo after that fix** (260,865 markets on 116 connections, 2026-11-12), two connections were judged slow and recycled: one at a mean lateness of 246 ms, one at 80 ms. The probe's 16 connections, on the 100 busiest markets, differed far more than the collector's; assumed to be because each of the collector's shares its markets with many others and is judged against all their copies, which evens the means out. So the mechanism is tested but was not seen to fire live, and its gain on the collector was not measured; how the demo's figures carry to production, which runs elsewhere, is unknown. - **Distance weighs more**: 14 ms of lag to Polymarket, about 100 ms to Kalshi's demo, against a median gain from racing under 1 ms and of 9 ms. Colocation is the lever; it was out of reach here.