This repository has been archived on 2026-11-16. You can view files and clone it. You cannot open issues or pull requests or push a commit.
Files
pascal 00202bbc51 docs: correct the README's claims, state the limits, and CI on macOS
The README said the quiet rule holds for the quietest markets (it can't
judge one under 3 changes a minute) and that all four extensions were
done (two of the brief's latency ideas were not tried); LATENCY.md said
the slow recycle was not seen to fire live, two lines after the run where
it did. Limits gains the process going down, the quiet rule's blind
spots and Kalshi's skipped-change case.

CLAUDE.md loses the home network's details and the server's key paths,
of no use to a reader of the public repo.

scripts/test_watch.sh no longer puts a here-document inside `$( )`:
bash 3.2, which is sh on macOS, stopped at an apostrophe there and
scripts/ci.sh ended on a syntax error.
2026-11-12 23:17:10 +01:00

220 lines
12 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Latency experiments
Extension 4 of the challenge: ways to get the exchanges' changes sooner,
tried on Polymarket and Kalshi's demo. Every trick the brief names (connecting
first after maintenance, spamming connections until a fast one comes,
changing the TCP source port) rests on one question: **is a connection that
is fast now still fast later?** If its speed is noise from message to
message, no choice of connection pays. That is what was measured, from a
server in Switzerland, on 2026-11-12.
## Method
`engineering-challenge latency EXCHANGE OUT.csv` (`src/latency.rs`) opens
16 connections to the same 100 busy markets, spread evenly over the
websocket host's addresses, and writes one line per copy of a change: which
connection delivered it, from which address, and when (stamped right after
the socket read, as the collector does). `scripts/latency/analyse.py`
matches the copies of each change across connections.
- **Lateness**: how long after a change's first copy a connection
delivered it. It compares receive times on one machine, so it needs no
clock. A connection's **speed** is its median lateness over a minute.
- **Lag**: receive time minus the exchange's timestamp. It carries the
local clock's error, so the probe asks `time.cloudflare.com` (SNTP) for
the offset at the start and end of each run: −0.25 to +0.69 ms, with
3 to 4 ms round trips. The clock is not what limits the figures below.
- **Markets**: on Polymarket the 100 most traded over 24 hours (Gamma's
`volume24hr`), 200 tokens. Kalshi's demo reports no traded amount, so
the markets that changed most in a minute are kept: in the first run
among the listing's first 30,000 open ones, in the second among every
open one (267,691), after the listing's order had moved and the first
30,000 held one market that changed.
- **Runs**: 10 minutes each, alternated (Polymarket, Kalshi, Polymarket,
Kalshi), the server's load between 0.4 and 2.7 on 12 cores before each.
Kalshi is its demo: no production key. The demo's servers are in AWS
us-west-2 (Oregon) and production's in us-east-2 (Ohio) (AWS's published
ranges), so how far the demo's figures hold for production is unknown.
- **Agreed before the runs**: the collector changes only if, on an exchange,
speed persists (rank correlation of the connections' speeds from one
minute to the next above about 0.5, read on the run's mean) and keeping
the 2 fastest connections
of the first minute lowers the p99 of the pair's lateness (the sooner of
its two copies, as first one wins takes it) by at least 20% against a
pair at random, in both runs.
## Does a connection's speed persist?
| Run | Changes | Rank correlation, minute to minute (mean) | Speeds in a minute |
|---|---|---|---|
| Polymarket 1 | 450,096 | 0.91 to 0.99 (0.96) | 0.2 to 2.5 ms |
| Polymarket 2 | 541,933 | 0.96 to 1.00 (0.98) | 0.04 to 3.3 ms |
| Kalshi demo 1 | 540,574 | 0.47 to 0.86 (0.66) | 0 to 177 ms |
| Kalshi demo 2 | 600,046 | 0.67 to 0.91 (0.78) | 0.04 to 136 ms |
**On Polymarket, yes, almost perfectly.** A connection keeps its place for
the whole run: in the second run connection 6 was 0.04 to 0.1 ms behind the
first copy in 7 of its 9 minutes, connection 3 about 3 ms in all 9. But the
spread is a few milliseconds, and the slow moments are shared: the p99 of
lateness is 25 to 54 ms on every address alike, and the 4 fastest
connections of the first minute had a p99 barely below the others' after
it (32 against 43 ms in the first run, 21 against 22 in the second), bursts
that delay all 16 at once. Speed seems to be decided when a connection opens (four
connections to each address, fast and slow ones on all four), and it can
change mid-way: connection 11 of the first run went from 0.35 to 1 ms in
its fifth minute.
**On Kalshi's demo, partly.** Connections differ by tens of milliseconds
up to 177, and the ranks hold less well (0.66 and 0.78; two minutes of the
first run fell to 0.47 and 0.49, so the threshold is met on the mean, not
in every minute). Some connections settle at
0 for minutes (12 from its third minute in the first run): they became the
one that delivers first.
## Racing: keep the fastest
Choosing after the fact is what opening many connections and keeping the
fast ones would do, provided speed is set when a connection opens; the
persistence above says it largely is. The 2 fastest of the first minute,
against a pair at random (all 120 pairs, averaged), over the minutes
after (8, and 9 in Kalshi's second run: the first run's stop was timed from
its start, so its listing and count left 9 min 52 s of probing; the
second's came 10 minutes after its count, 10 min 2 s):
| Run | Kept pair, median / p99 | Pair at random, median / p99 | p99 lower by |
|---|---|---|---|
| Polymarket 1 | 0.20 / 8.79 ms | 0.60 / 12.43 ms | 29% |
| Polymarket 2 | 0.00 / 13.96 ms | 0.95 / 13.38 ms | −4% |
| Kalshi demo 1 | 20.15 / 233.59 ms | 28.91 / 324.56 ms | 28% |
| Kalshi demo 2 | 0.16 / 164.25 ms | 9.24 / 283.20 ms | 42% |
No change was missed by any pair. **Polymarket: racing works at the median
and not in the tail.** The kept pair is ahead of a random one by under a
millisecond every time, but the p99 gain vanished in the second run: the
tail is the bursts every connection shares. The threshold is not met, and
the collector stays as it is: two connections per market, chosen without
regard to speed. **Kalshi's demo: racing works in both.** The p99 falls
by 28% and 42%, the median from 29 to 20 ms and from 9 to 0.2 ms: there,
a connection's speed is worth choosing, and the threshold is met. Whether
production behaves alike is unknown.
## Does the address matter?
Polymarket's websocket host resolves to 4 CloudFront addresses (and 4 IPv6
ones, unreachable from this server), Kalshi demo's to 2. Lateness by
address, all copies of a run:
| Run | Medians by address | p99s by address |
|---|---|---|
| Polymarket 1 | 0.90 to 1.04 ms | 33 to 54 ms |
| Polymarket 2 | 1.08 to 1.98 ms | 25 to 36 ms |
| Kalshi demo 1 | 40.6 and 45.2 ms | 503 and 548 ms |
| Kalshi demo 2 | 16.5 and 23.9 ms | 406 and 418 ms |
The address matters much less than the connection. On Polymarket,
`143.204.55.29` had the lowest median in both runs, by 0.1 and 0.5 ms, but
every address carried both fast and slow connections, and the spread inside
an address (0.3 to 2.5 ms in the first run) is larger than between them.
On Kalshi's demo the faster of the two addresses changed between runs.
Whatever sets a connection's speed is behind the address, assumed to be
the CDN's or the load balancer's backend, chosen per connection. The collector connects only to the first address its lookup
returns (`src/core/socket.rs`); spreading over all of them would change
nothing measured here.
## Lag: how far behind the exchange
The first copy's lag, corrected by the clock's offset:
| Run | Median | p99 |
|---|---|---|
| Polymarket 1 | 13.70 ms | 20.21 ms |
| Polymarket 2 | 14.55 ms | 25.37 ms |
| Kalshi demo 1 | 94.03 ms | 285.20 ms |
| Kalshi demo 2 | 106.31 ms | 382.30 ms |
These include the exchange's own time between stamping a change and sending
it, which can't be told apart from the network from here. Kalshi's demo in
Oregon is about 80 ms further than Polymarket, which is one hop to a
CloudFront edge (where Polymarket's own servers are was not checked). On
Polymarket that is far above what racing wins (under a millisecond). On
Kalshi's demo racing wins more in the tail (90 and 119 ms at p99, as much
as the median lag), but at the median it wins 9 ms, a tenth of the lag.
## Not tried, and why
- **First connection after maintenance.** Kalshi's docs announce
maintenance on Thursdays from 3:00 to 5:00 AM ET (08:00 to 10:00 UTC in
winter); the sprint ran after one. The test: run the probe at the
window's end and an hour later, and compare the speeds of the connections
opened first. Polymarket announces no window.
- **The TCP source port.** It can't be chosen from here: a NAT between the
server and the internet rewrites every connection's port (local 41000
left as 36996, and as 37086 a minute later). Testing it needs a host with
a public address: open connections from fixed ports, reconnect from the
same ones, and see whether each port's speed returns. Given that speed
does not follow the address, a load balancer hashing on the port is the
likeliest reason it could.
- **Location.** Assumed the largest gain by far: lag is 14 ms to Polymarket
and 94 ms to Kalshi's demo from Switzerland, against a millisecond won by
racing. A server in the exchange's AWS region (us-east-2 for Kalshi
production) would remove most of the lag; not tried, it needs a second
machine.
- **IPv6.** This server has no IPv6 route. From a host with one, the
probe would keep Polymarket's 4 IPv6 addresses in its round-robin (it
leaves them out) and the address table would say whether they differ.
- **Kernel tuning** (busy polling, receive buffers, pinning threads). The
receive time is stamped at the read, the machine's load was low, and
nothing above points at the kernel. The test: the same probe run with
`SO_BUSY_POLL` set and each connection's thread pinned, alternated with
runs without, comparing the first copies' lag on an idle machine.
- **Our own added latency**, from the read to the output write: not
measured. It doesn't change when a change is stamped. The test: stamp
each output again as it is written, and report the difference.
- **Kalshi production**: no key. The probe runs on it as `latency kalshi`
with production's two key files.
## Verdict
- **Connections differ, and keep their speed**: strongly on Polymarket
(rank correlation 0.96 and 0.98), less on Kalshi's demo (0.66 and 0.78).
The one thing we could choose at connect, the address, doesn't set it
(the source port could not be tried): it is drawn per connection, mostly when it opens, though it can shift mid-way
(Kalshi's connection 4 went from 75 to 3 ms over its second run).
- **On Polymarket it is worth under a millisecond.** Keeping the fastest
pair wins at the median, not in the tail, where bursts delay every
connection alike. The collector keeps its connections as they are.
- **On Kalshi's demo it is worth tens of milliseconds**, in the tail too
(p99 28% and 42% lower with the fastest pair kept). This met the
threshold agreed before the runs, so the collector now recycles a Kalshi
connection that stays slow for 3 minutes running at the hourly
recycle's next turn (one every 30 s or so), in place of the oldest: its
replacement's speed is drawn afresh. Each duplicate tells how
late its connection was against the other's copy (`src/kalshi/feed.rs`);
once a minute the orchestrator ranks the connections by their mean
lateness, and one is slow if it is in the slowest quarter and more than
twice the mean of all (`src/core/orchestrator.rs`). The mean, not the
median: with two copies of each change, a connection first more than
half the time has a median of 0. Without the factor of two, a slowest
quarter always exists: a first version, by median alone, recycled 6
connections in 20 minutes, one a minute, connections no slower than the
rest. **In 20 minutes of the collector on the demo (270,000 markets on
120 connections, load 5.7 at the start), none was recycled for it**; the
hourly recycle went on (33), and the output reconciled (25 mismatches,
each a logged loss). That run doesn't say whether one was judged slow: a
later review found the slow recycle waited for the same moment as the
hourly one, which nearly always took it first, and only the recycle was
logged. Now the slow one takes the hourly recycle's turn, and the
judgment is logged on its own. **In 20 minutes on the demo after that
fix** (260,865 markets on 116 connections, 2026-11-12), two connections
were judged slow and recycled: one at a mean lateness of 246 ms, one at
80 ms. The probe's 16
connections, on the 100 busiest markets, differed far more than the
collector's; assumed to be because each of the collector's shares its
markets with many others and is judged against all their copies, which
evens the means out. So the mechanism is tested and was seen to fire
live, but its gain on the collector was not measured; how the
demo's figures carry to production, which runs elsewhere, is unknown.
- **Distance weighs more**: 14 ms of lag to Polymarket, about 100 ms to
Kalshi's demo, against a median gain from racing under 1 ms and of 9 ms.
Colocation is the lever; it was out of reach here.