Archived
Extension 4. `engineering-challenge latency EXCHANGE OUT.csv` opens 16 connections to the same busy markets and writes every copy of every change; `scripts/latency/analyse.py` gives each connection's speed, whether it persists, what keeping the fastest would gain, lateness by address, and lag with the clock's offset. LATENCY.md writes up four runs: on Polymarket speed persists but is worth under a millisecond, at the median only; on Kalshi's demo keeping the fastest pair lowers p99 by 28% and 42%, the threshold agreed before the runs. So the collector now recycles a Kalshi connection whose mean lateness stays, 3 minutes running, in the slowest quarter and above twice everyone's, ahead of its hourly turn. Kalshi's rule measures each duplicate's lateness; the orchestrator judges once a minute. A duplicate's lag in Polymarket's minute line is now its "lateness". Roadmap milestone 7, roadmap/07.
212 lines
12 KiB
Markdown
212 lines
12 KiB
Markdown
# Latency experiments
|
||
|
||
Extension 4 of the challenge: ways to get the exchanges' changes sooner,
|
||
tried on Polymarket and Kalshi. Every trick the brief names (connecting
|
||
first after maintenance, spamming connections until a fast one comes,
|
||
changing the TCP source port) rests on one question: **is a connection that
|
||
is fast now still fast later?** If its speed is noise from message to
|
||
message, no choice of connection pays. That is what was measured, from a
|
||
server in Switzerland, on 2026-11-12.
|
||
|
||
## Method
|
||
|
||
`engineering-challenge latency EXCHANGE OUT.csv` (`src/latency.rs`) opens
|
||
16 connections to the same 100 busy markets, spread evenly over the
|
||
websocket host's addresses, and writes one line per copy of a change: which
|
||
connection delivered it, from which address, and when (stamped right after
|
||
the socket read, as the collector does). `scripts/latency/analyse.py`
|
||
matches the copies of each change across connections.
|
||
|
||
- **Lateness**: how long after a change's first copy a connection
|
||
delivered it. It compares receive times on one machine, so it needs no
|
||
clock. A connection's **speed** is its median lateness over a minute.
|
||
- **Lag**: receive time minus the exchange's timestamp. It carries the
|
||
local clock's error, so the probe asks `time.cloudflare.com` (SNTP) for
|
||
the offset at the start and end of each run: −0.25 to +0.69 ms, with
|
||
3 to 4 ms round trips. The clock is not what limits the figures below.
|
||
- **Markets**: on Polymarket the 100 most traded over 24 hours (Gamma's
|
||
`volume24hr`), 200 tokens. Kalshi's demo reports no traded amount, so
|
||
the markets that changed most in a minute are kept: in the first run
|
||
among the listing's first 30,000 open ones, in the second among every
|
||
open one (267,691), after the listing's order had moved and the first
|
||
30,000 held one market that changed.
|
||
- **Runs**: 10 minutes each, alternated (Polymarket, Kalshi, Polymarket,
|
||
Kalshi), the server's load between 0.4 and 2.7 on 12 cores before each.
|
||
Kalshi is its demo: no production key. The demo's servers are in AWS
|
||
us-west-2 (Oregon) and production's in us-east-2 (Ohio) (AWS's published
|
||
ranges), so how far the demo's figures hold for production is unknown.
|
||
- **Agreed before the runs**: the collector changes only if, on an exchange,
|
||
speed persists (rank correlation of the connections' speeds from one
|
||
minute to the next above about 0.5, read on the run's mean) and keeping
|
||
the 2 fastest connections
|
||
of the first minute lowers the p99 of the pair's lateness (the sooner of
|
||
its two copies, as first one wins takes it) by at least 20% against a
|
||
pair at random, in both runs.
|
||
|
||
## Does a connection's speed persist?
|
||
|
||
| Run | Changes | Rank correlation, minute to minute (mean) | Speeds in a minute |
|
||
|---|---|---|---|
|
||
| Polymarket 1 | 450,096 | 0.91 to 0.99 (0.96) | 0.2 to 2.5 ms |
|
||
| Polymarket 2 | 541,933 | 0.96 to 1.00 (0.98) | 0.04 to 3.3 ms |
|
||
| Kalshi demo 1 | 540,574 | 0.47 to 0.86 (0.66) | 0 to 177 ms |
|
||
| Kalshi demo 2 | 600,046 | 0.67 to 0.91 (0.78) | 0.04 to 136 ms |
|
||
|
||
**On Polymarket, yes, almost perfectly.** A connection keeps its place for
|
||
the whole run: in the second run connection 6 was 0.04 to 0.1 ms behind the
|
||
first copy in 7 of its 9 minutes, connection 3 about 3 ms in all 9. But the
|
||
spread is a few milliseconds, and the slow moments are shared: the p99 of
|
||
lateness is 25 to 54 ms on every address alike, and the 4 fastest
|
||
connections of the first minute had a p99 barely below the others' after
|
||
it (32 against 43 ms in the first run, 21 against 22 in the second), bursts
|
||
that delay all 16 at once. Speed seems to be decided when a connection opens (four
|
||
connections to each address, fast and slow ones on all four), and it can
|
||
change mid-way: connection 11 of the first run went from 0.35 to 1 ms in
|
||
its fifth minute.
|
||
|
||
**On Kalshi's demo, partly.** Connections differ by tens of milliseconds
|
||
up to 177, and the ranks hold less well (0.66 and 0.78; two minutes of the
|
||
first run fell to 0.47 and 0.49, so the threshold is met on the mean, not
|
||
in every minute). Some connections settle at
|
||
0 for minutes (12 from its third minute in the first run): they became the
|
||
one that delivers first.
|
||
|
||
## Racing: keep the fastest
|
||
|
||
Choosing after the fact is what opening many connections and keeping the
|
||
fast ones would do, provided speed is set when a connection opens; the
|
||
persistence above says it largely is. The 2 fastest of the first minute,
|
||
against a pair at random (all 120 pairs, averaged), over the minutes
|
||
after (8, and 9 in Kalshi's second run: the first run's stop was timed from
|
||
its start, so its listing and count left 9 min 52 s of probing; the
|
||
second's came 10 minutes after its count, 10 min 2 s):
|
||
|
||
| Run | Kept pair, median / p99 | Pair at random, median / p99 | p99 lower by |
|
||
|---|---|---|---|
|
||
| Polymarket 1 | 0.20 / 8.79 ms | 0.60 / 12.43 ms | 29% |
|
||
| Polymarket 2 | 0.00 / 13.96 ms | 0.95 / 13.38 ms | −4% |
|
||
| Kalshi demo 1 | 20.15 / 233.59 ms | 28.91 / 324.56 ms | 28% |
|
||
| Kalshi demo 2 | 0.16 / 164.25 ms | 9.24 / 283.20 ms | 42% |
|
||
|
||
No change was missed by any pair. **Polymarket: racing works at the median
|
||
and not in the tail.** The kept pair is ahead of a random one by under a
|
||
millisecond every time, but the p99 gain vanished in the second run: the
|
||
tail is the bursts every connection shares. The threshold is not met, and
|
||
the collector stays as it is: two connections per market, chosen without
|
||
regard to speed. **Kalshi's demo: racing works in both.** The p99 falls
|
||
by 28% and 42%, the median from 29 to 20 ms and from 9 to 0.2 ms: there,
|
||
a connection's speed is worth choosing, and the threshold is met. Whether
|
||
production behaves alike is unknown.
|
||
|
||
## Does the address matter?
|
||
|
||
Polymarket's websocket host resolves to 4 CloudFront addresses (and 4 IPv6
|
||
ones, unreachable from this server), Kalshi demo's to 2. Lateness by
|
||
address, all copies of a run:
|
||
|
||
| Run | Medians by address | p99s by address |
|
||
|---|---|---|
|
||
| Polymarket 1 | 0.90 to 1.04 ms | 33 to 54 ms |
|
||
| Polymarket 2 | 1.08 to 1.98 ms | 25 to 36 ms |
|
||
| Kalshi demo 1 | 40.6 and 45.2 ms | 503 and 548 ms |
|
||
| Kalshi demo 2 | 16.5 and 23.9 ms | 406 and 418 ms |
|
||
|
||
The address matters much less than the connection. On Polymarket,
|
||
`143.204.55.29` had the lowest median in both runs, by 0.1 and 0.5 ms, but
|
||
every address carried both fast and slow connections, and the spread inside
|
||
an address (0.3 to 2.5 ms in the first run) is larger than between them.
|
||
On Kalshi's demo the faster of the two addresses changed between runs.
|
||
Whatever sets a connection's speed is behind the address, assumed to be
|
||
the CDN's or the load balancer's backend, chosen per connection. The collector connects only to the first address its lookup
|
||
returns (`src/core/socket.rs`); spreading over all of them would change
|
||
nothing measured here.
|
||
|
||
## Lag: how far behind the exchange
|
||
|
||
The first copy's lag, corrected by the clock's offset:
|
||
|
||
| Run | Median | p99 |
|
||
|---|---|---|
|
||
| Polymarket 1 | 13.70 ms | 20.21 ms |
|
||
| Polymarket 2 | 14.55 ms | 25.37 ms |
|
||
| Kalshi demo 1 | 94.03 ms | 285.20 ms |
|
||
| Kalshi demo 2 | 106.31 ms | 382.30 ms |
|
||
|
||
These include the exchange's own time between stamping a change and sending
|
||
it, which can't be told apart from the network from here. Kalshi's demo in
|
||
Oregon is about 80 ms further than Polymarket, which is one hop to a
|
||
CloudFront edge (where Polymarket's own servers are was not checked). On
|
||
Polymarket that is far above what racing wins (under a millisecond). On
|
||
Kalshi's demo racing wins more in the tail (90 and 119 ms at p99, as much
|
||
as the median lag), but at the median it wins 9 ms, a tenth of the lag.
|
||
|
||
## Not tried, and why
|
||
|
||
- **First connection after maintenance.** Kalshi's docs announce
|
||
maintenance on Thursdays from 3:00 to 5:00 AM ET (08:00 to 10:00 UTC in
|
||
winter); the sprint ran after one. The test: run the probe at the
|
||
window's end and an hour later, and compare the speeds of the connections
|
||
opened first. Polymarket announces no window.
|
||
- **The TCP source port.** It can't be chosen from here: a NAT between the
|
||
server and the internet rewrites every connection's port (local 41000
|
||
left as 36996, and as 37086 a minute later). Testing it needs a host with
|
||
a public address: open connections from fixed ports, reconnect from the
|
||
same ones, and see whether each port's speed returns. Given that speed
|
||
does not follow the address, a load balancer hashing on the port is the
|
||
likeliest reason it could.
|
||
- **Location.** Assumed the largest gain by far: lag is 14 ms to Polymarket
|
||
and 94 ms to Kalshi's demo from Switzerland, against a millisecond won by
|
||
racing. A server in the exchange's AWS region (us-east-2 for Kalshi
|
||
production) would remove most of the lag; not tried, it needs a second
|
||
machine.
|
||
- **IPv6.** This server has no IPv6 route. From a host with one, the
|
||
probe would keep Polymarket's 4 IPv6 addresses in its round-robin (it
|
||
leaves them out) and the address table would say whether they differ.
|
||
- **Kernel tuning** (busy polling, receive buffers, pinning threads). The
|
||
receive time is stamped at the read, the machine's load was low, and
|
||
nothing above points at the kernel. The test: the same probe run with
|
||
`SO_BUSY_POLL` set and each connection's thread pinned, alternated with
|
||
runs without, comparing the first copies' lag on an idle machine.
|
||
- **Our own added latency**, from the read to the output write: not
|
||
measured. It doesn't change when a change is stamped. The test: stamp
|
||
each output again as it is written, and report the difference.
|
||
- **Kalshi production**: no key. The probe runs on it as `latency kalshi`
|
||
with production's two key files.
|
||
|
||
## Verdict
|
||
|
||
- **Connections differ, and keep their speed**: strongly on Polymarket
|
||
(rank correlation 0.96 and 0.98), less on Kalshi's demo (0.66 and 0.78).
|
||
Nothing we could choose at connect sets it (the address doesn't): it is
|
||
drawn per connection, mostly when it opens, though it can shift mid-way
|
||
(Kalshi's connection 4 went from 75 to 3 ms over its second run).
|
||
- **On Polymarket it is worth under a millisecond.** Keeping the fastest
|
||
pair wins at the median, not in the tail, where bursts delay every
|
||
connection alike. The collector keeps its connections as they are.
|
||
- **On Kalshi's demo it is worth tens of milliseconds**, in the tail too
|
||
(p99 28% and 42% lower with the fastest pair kept). This met the
|
||
threshold agreed before the runs, so the collector now recycles a Kalshi
|
||
connection that stays slow for 3 minutes running, ahead of its hourly
|
||
turn: its replacement's speed is drawn afresh. Each duplicate tells how
|
||
late its connection was against the other's copy (`src/kalshi/feed.rs`);
|
||
once a minute the orchestrator ranks the connections by their mean
|
||
lateness, and one is slow if it is in the slowest quarter and more than
|
||
twice the mean of all (`src/core/orchestrator.rs`). The mean, not the
|
||
median: with two copies of each change, a connection first more than
|
||
half the time has a median of 0. Without the factor of two, a slowest
|
||
quarter always exists: a first version, by median alone, recycled 6
|
||
connections in 20 minutes, one a minute, connections no slower than the
|
||
rest. **In 20 minutes of the collector on the demo (270,000 markets on
|
||
120 connections, load 5.7 at the start), no connection was slow by that
|
||
rule**, so none was recycled for it; the hourly recycle went on (33), and
|
||
the output reconciled (25 mismatches, each a logged loss). The probe's 16
|
||
connections, on the 100 busiest markets, differed far more than the
|
||
collector's; assumed to be because each of the collector's shares its
|
||
markets with many others and is judged against all their copies, which
|
||
evens the means out. So the mechanism is tested but was not seen
|
||
to fire live, and its gain on the collector was not measured; how the
|
||
demo's figures carry to production, which runs elsewhere, is unknown.
|
||
- **Distance weighs more**: 14 ms of lag to Polymarket, about 100 ms to
|
||
Kalshi's demo, against a median gain from racing under 1 ms and of 9 ms.
|
||
Colocation is the lever; it was out of reach here.
|