Archived
The README said the quiet rule holds for the quietest markets (it can't judge one under 3 changes a minute) and that all four extensions were done (two of the brief's latency ideas were not tried); LATENCY.md said the slow recycle was not seen to fire live, two lines after the run where it did. Limits gains the process going down, the quiet rule's blind spots and Kalshi's skipped-change case. CLAUDE.md loses the home network's details and the server's key paths, of no use to a reader of the public repo. scripts/test_watch.sh no longer puts a here-document inside `$( )`: bash 3.2, which is sh on macOS, stopped at an apostrophe there and scripts/ci.sh ended on a syntax error.
220 lines
12 KiB
Markdown
220 lines
12 KiB
Markdown
# Latency experiments
|
||
|
||
Extension 4 of the challenge: ways to get the exchanges' changes sooner,
|
||
tried on Polymarket and Kalshi's demo. Every trick the brief names (connecting
|
||
first after maintenance, spamming connections until a fast one comes,
|
||
changing the TCP source port) rests on one question: **is a connection that
|
||
is fast now still fast later?** If its speed is noise from message to
|
||
message, no choice of connection pays. That is what was measured, from a
|
||
server in Switzerland, on 2026-11-12.
|
||
|
||
## Method
|
||
|
||
`engineering-challenge latency EXCHANGE OUT.csv` (`src/latency.rs`) opens
|
||
16 connections to the same 100 busy markets, spread evenly over the
|
||
websocket host's addresses, and writes one line per copy of a change: which
|
||
connection delivered it, from which address, and when (stamped right after
|
||
the socket read, as the collector does). `scripts/latency/analyse.py`
|
||
matches the copies of each change across connections.
|
||
|
||
- **Lateness**: how long after a change's first copy a connection
|
||
delivered it. It compares receive times on one machine, so it needs no
|
||
clock. A connection's **speed** is its median lateness over a minute.
|
||
- **Lag**: receive time minus the exchange's timestamp. It carries the
|
||
local clock's error, so the probe asks `time.cloudflare.com` (SNTP) for
|
||
the offset at the start and end of each run: −0.25 to +0.69 ms, with
|
||
3 to 4 ms round trips. The clock is not what limits the figures below.
|
||
- **Markets**: on Polymarket the 100 most traded over 24 hours (Gamma's
|
||
`volume24hr`), 200 tokens. Kalshi's demo reports no traded amount, so
|
||
the markets that changed most in a minute are kept: in the first run
|
||
among the listing's first 30,000 open ones, in the second among every
|
||
open one (267,691), after the listing's order had moved and the first
|
||
30,000 held one market that changed.
|
||
- **Runs**: 10 minutes each, alternated (Polymarket, Kalshi, Polymarket,
|
||
Kalshi), the server's load between 0.4 and 2.7 on 12 cores before each.
|
||
Kalshi is its demo: no production key. The demo's servers are in AWS
|
||
us-west-2 (Oregon) and production's in us-east-2 (Ohio) (AWS's published
|
||
ranges), so how far the demo's figures hold for production is unknown.
|
||
- **Agreed before the runs**: the collector changes only if, on an exchange,
|
||
speed persists (rank correlation of the connections' speeds from one
|
||
minute to the next above about 0.5, read on the run's mean) and keeping
|
||
the 2 fastest connections
|
||
of the first minute lowers the p99 of the pair's lateness (the sooner of
|
||
its two copies, as first one wins takes it) by at least 20% against a
|
||
pair at random, in both runs.
|
||
|
||
## Does a connection's speed persist?
|
||
|
||
| Run | Changes | Rank correlation, minute to minute (mean) | Speeds in a minute |
|
||
|---|---|---|---|
|
||
| Polymarket 1 | 450,096 | 0.91 to 0.99 (0.96) | 0.2 to 2.5 ms |
|
||
| Polymarket 2 | 541,933 | 0.96 to 1.00 (0.98) | 0.04 to 3.3 ms |
|
||
| Kalshi demo 1 | 540,574 | 0.47 to 0.86 (0.66) | 0 to 177 ms |
|
||
| Kalshi demo 2 | 600,046 | 0.67 to 0.91 (0.78) | 0.04 to 136 ms |
|
||
|
||
**On Polymarket, yes, almost perfectly.** A connection keeps its place for
|
||
the whole run: in the second run connection 6 was 0.04 to 0.1 ms behind the
|
||
first copy in 7 of its 9 minutes, connection 3 about 3 ms in all 9. But the
|
||
spread is a few milliseconds, and the slow moments are shared: the p99 of
|
||
lateness is 25 to 54 ms on every address alike, and the 4 fastest
|
||
connections of the first minute had a p99 barely below the others' after
|
||
it (32 against 43 ms in the first run, 21 against 22 in the second), bursts
|
||
that delay all 16 at once. Speed seems to be decided when a connection opens (four
|
||
connections to each address, fast and slow ones on all four), and it can
|
||
change mid-way: connection 11 of the first run went from 0.35 to 1 ms in
|
||
its fifth minute.
|
||
|
||
**On Kalshi's demo, partly.** Connections differ by tens of milliseconds
|
||
up to 177, and the ranks hold less well (0.66 and 0.78; two minutes of the
|
||
first run fell to 0.47 and 0.49, so the threshold is met on the mean, not
|
||
in every minute). Some connections settle at
|
||
0 for minutes (12 from its third minute in the first run): they became the
|
||
one that delivers first.
|
||
|
||
## Racing: keep the fastest
|
||
|
||
Choosing after the fact is what opening many connections and keeping the
|
||
fast ones would do, provided speed is set when a connection opens; the
|
||
persistence above says it largely is. The 2 fastest of the first minute,
|
||
against a pair at random (all 120 pairs, averaged), over the minutes
|
||
after (8, and 9 in Kalshi's second run: the first run's stop was timed from
|
||
its start, so its listing and count left 9 min 52 s of probing; the
|
||
second's came 10 minutes after its count, 10 min 2 s):
|
||
|
||
| Run | Kept pair, median / p99 | Pair at random, median / p99 | p99 lower by |
|
||
|---|---|---|---|
|
||
| Polymarket 1 | 0.20 / 8.79 ms | 0.60 / 12.43 ms | 29% |
|
||
| Polymarket 2 | 0.00 / 13.96 ms | 0.95 / 13.38 ms | −4% |
|
||
| Kalshi demo 1 | 20.15 / 233.59 ms | 28.91 / 324.56 ms | 28% |
|
||
| Kalshi demo 2 | 0.16 / 164.25 ms | 9.24 / 283.20 ms | 42% |
|
||
|
||
No change was missed by any pair. **Polymarket: racing works at the median
|
||
and not in the tail.** The kept pair is ahead of a random one by under a
|
||
millisecond every time, but the p99 gain vanished in the second run: the
|
||
tail is the bursts every connection shares. The threshold is not met, and
|
||
the collector stays as it is: two connections per market, chosen without
|
||
regard to speed. **Kalshi's demo: racing works in both.** The p99 falls
|
||
by 28% and 42%, the median from 29 to 20 ms and from 9 to 0.2 ms: there,
|
||
a connection's speed is worth choosing, and the threshold is met. Whether
|
||
production behaves alike is unknown.
|
||
|
||
## Does the address matter?
|
||
|
||
Polymarket's websocket host resolves to 4 CloudFront addresses (and 4 IPv6
|
||
ones, unreachable from this server), Kalshi demo's to 2. Lateness by
|
||
address, all copies of a run:
|
||
|
||
| Run | Medians by address | p99s by address |
|
||
|---|---|---|
|
||
| Polymarket 1 | 0.90 to 1.04 ms | 33 to 54 ms |
|
||
| Polymarket 2 | 1.08 to 1.98 ms | 25 to 36 ms |
|
||
| Kalshi demo 1 | 40.6 and 45.2 ms | 503 and 548 ms |
|
||
| Kalshi demo 2 | 16.5 and 23.9 ms | 406 and 418 ms |
|
||
|
||
The address matters much less than the connection. On Polymarket,
|
||
`143.204.55.29` had the lowest median in both runs, by 0.1 and 0.5 ms, but
|
||
every address carried both fast and slow connections, and the spread inside
|
||
an address (0.3 to 2.5 ms in the first run) is larger than between them.
|
||
On Kalshi's demo the faster of the two addresses changed between runs.
|
||
Whatever sets a connection's speed is behind the address, assumed to be
|
||
the CDN's or the load balancer's backend, chosen per connection. The collector connects only to the first address its lookup
|
||
returns (`src/core/socket.rs`); spreading over all of them would change
|
||
nothing measured here.
|
||
|
||
## Lag: how far behind the exchange
|
||
|
||
The first copy's lag, corrected by the clock's offset:
|
||
|
||
| Run | Median | p99 |
|
||
|---|---|---|
|
||
| Polymarket 1 | 13.70 ms | 20.21 ms |
|
||
| Polymarket 2 | 14.55 ms | 25.37 ms |
|
||
| Kalshi demo 1 | 94.03 ms | 285.20 ms |
|
||
| Kalshi demo 2 | 106.31 ms | 382.30 ms |
|
||
|
||
These include the exchange's own time between stamping a change and sending
|
||
it, which can't be told apart from the network from here. Kalshi's demo in
|
||
Oregon is about 80 ms further than Polymarket, which is one hop to a
|
||
CloudFront edge (where Polymarket's own servers are was not checked). On
|
||
Polymarket that is far above what racing wins (under a millisecond). On
|
||
Kalshi's demo racing wins more in the tail (90 and 119 ms at p99, as much
|
||
as the median lag), but at the median it wins 9 ms, a tenth of the lag.
|
||
|
||
## Not tried, and why
|
||
|
||
- **First connection after maintenance.** Kalshi's docs announce
|
||
maintenance on Thursdays from 3:00 to 5:00 AM ET (08:00 to 10:00 UTC in
|
||
winter); the sprint ran after one. The test: run the probe at the
|
||
window's end and an hour later, and compare the speeds of the connections
|
||
opened first. Polymarket announces no window.
|
||
- **The TCP source port.** It can't be chosen from here: a NAT between the
|
||
server and the internet rewrites every connection's port (local 41000
|
||
left as 36996, and as 37086 a minute later). Testing it needs a host with
|
||
a public address: open connections from fixed ports, reconnect from the
|
||
same ones, and see whether each port's speed returns. Given that speed
|
||
does not follow the address, a load balancer hashing on the port is the
|
||
likeliest reason it could.
|
||
- **Location.** Assumed the largest gain by far: lag is 14 ms to Polymarket
|
||
and 94 ms to Kalshi's demo from Switzerland, against a millisecond won by
|
||
racing. A server in the exchange's AWS region (us-east-2 for Kalshi
|
||
production) would remove most of the lag; not tried, it needs a second
|
||
machine.
|
||
- **IPv6.** This server has no IPv6 route. From a host with one, the
|
||
probe would keep Polymarket's 4 IPv6 addresses in its round-robin (it
|
||
leaves them out) and the address table would say whether they differ.
|
||
- **Kernel tuning** (busy polling, receive buffers, pinning threads). The
|
||
receive time is stamped at the read, the machine's load was low, and
|
||
nothing above points at the kernel. The test: the same probe run with
|
||
`SO_BUSY_POLL` set and each connection's thread pinned, alternated with
|
||
runs without, comparing the first copies' lag on an idle machine.
|
||
- **Our own added latency**, from the read to the output write: not
|
||
measured. It doesn't change when a change is stamped. The test: stamp
|
||
each output again as it is written, and report the difference.
|
||
- **Kalshi production**: no key. The probe runs on it as `latency kalshi`
|
||
with production's two key files.
|
||
|
||
## Verdict
|
||
|
||
- **Connections differ, and keep their speed**: strongly on Polymarket
|
||
(rank correlation 0.96 and 0.98), less on Kalshi's demo (0.66 and 0.78).
|
||
The one thing we could choose at connect, the address, doesn't set it
|
||
(the source port could not be tried): it is drawn per connection, mostly when it opens, though it can shift mid-way
|
||
(Kalshi's connection 4 went from 75 to 3 ms over its second run).
|
||
- **On Polymarket it is worth under a millisecond.** Keeping the fastest
|
||
pair wins at the median, not in the tail, where bursts delay every
|
||
connection alike. The collector keeps its connections as they are.
|
||
- **On Kalshi's demo it is worth tens of milliseconds**, in the tail too
|
||
(p99 28% and 42% lower with the fastest pair kept). This met the
|
||
threshold agreed before the runs, so the collector now recycles a Kalshi
|
||
connection that stays slow for 3 minutes running at the hourly
|
||
recycle's next turn (one every 30 s or so), in place of the oldest: its
|
||
replacement's speed is drawn afresh. Each duplicate tells how
|
||
late its connection was against the other's copy (`src/kalshi/feed.rs`);
|
||
once a minute the orchestrator ranks the connections by their mean
|
||
lateness, and one is slow if it is in the slowest quarter and more than
|
||
twice the mean of all (`src/core/orchestrator.rs`). The mean, not the
|
||
median: with two copies of each change, a connection first more than
|
||
half the time has a median of 0. Without the factor of two, a slowest
|
||
quarter always exists: a first version, by median alone, recycled 6
|
||
connections in 20 minutes, one a minute, connections no slower than the
|
||
rest. **In 20 minutes of the collector on the demo (270,000 markets on
|
||
120 connections, load 5.7 at the start), none was recycled for it**; the
|
||
hourly recycle went on (33), and the output reconciled (25 mismatches,
|
||
each a logged loss). That run doesn't say whether one was judged slow: a
|
||
later review found the slow recycle waited for the same moment as the
|
||
hourly one, which nearly always took it first, and only the recycle was
|
||
logged. Now the slow one takes the hourly recycle's turn, and the
|
||
judgment is logged on its own. **In 20 minutes on the demo after that
|
||
fix** (260,865 markets on 116 connections, 2026-11-12), two connections
|
||
were judged slow and recycled: one at a mean lateness of 246 ms, one at
|
||
80 ms. The probe's 16
|
||
connections, on the 100 busiest markets, differed far more than the
|
||
collector's; assumed to be because each of the collector's shares its
|
||
markets with many others and is judged against all their copies, which
|
||
evens the means out. So the mechanism is tested and was seen to fire
|
||
live, but its gain on the collector was not measured; how the
|
||
demo's figures carry to production, which runs elsewhere, is unknown.
|
||
- **Distance weighs more**: 14 ms of lag to Polymarket, about 100 ms to
|
||
Kalshi's demo, against a median gain from racing under 1 ms and of 9 ms.
|
||
Colocation is the lever; it was out of reach here.
|