feat(latency): latency experiments, LATENCY.md, Kalshi's slow-connection recycle

Extension 4. `engineering-challenge latency EXCHANGE OUT.csv` opens 16
connections to the same busy markets and writes every copy of every change;
`scripts/latency/analyse.py` gives each connection's speed, whether it
persists, what keeping the fastest would gain, lateness by address, and lag
with the clock's offset. LATENCY.md writes up four runs: on Polymarket speed
persists but is worth under a millisecond, at the median only; on Kalshi's
demo keeping the fastest pair lowers p99 by 28% and 42%, the threshold
agreed before the runs.

So the collector now recycles a Kalshi connection whose mean lateness stays,
3 minutes running, in the slowest quarter and above twice everyone's, ahead
of its hourly turn. Kalshi's rule measures each duplicate's lateness; the
orchestrator judges once a minute. A duplicate's lag in Polymarket's minute
line is now its "lateness".

Roadmap milestone 7, roadmap/07.
This commit is contained in:
pascal committed 2026-11-12 15:05:25 +01:00
1 parent 1a5bbcd602
commit 515e85941d
22 files changed
+2132 -55

No files matched your search

+18
View File
@@ -313,6 +313,24 @@ is the demo's.
(2026-11-11, 19:29 UTC).
- **Maintenance (docs):** Thursdays 3:00 to 5:00 AM ET; connections drop.
## Latency, from the server (observed 2026-11-12)
- **The server's clock is synced**: SNTP against `time.cloudflare.com` read
−0.25 to +0.69 ms, 3 to 4 ms round trips (sprint 07's probe).
- **What the exchange sees isn't ours to choose**: a NAT rewrites every
connection's source port (not even stable for one local port), and there
is no IPv6 route; Polymarket's websocket resolves to 4 IPv6 addresses
beside its 4 IPv4 ones (CloudFront).
- **Where the hosts are** (AWS's published ranges): Kalshi production's
websocket in us-east-2, the demo's two addresses in us-west-2; lag about
14 ms to Polymarket, about 100 ms to Kalshi's demo.
- **Polymarket repeats a change on one connection**, rarely: 22 of 29,922
changes came twice on every connection in a minute.
- **Kalshi's listing order moves**: its first 30,000 open markets held the
98 busiest at 11:12 and none of them at 11:50. Pick by counting every
market, not by the listing's first pages. The demo reports
`volume_24h_fp` 0.00 (267,691 open markets, a full listing in 75 s).
## Clock accuracy bounds time accuracy
`received_at` comes from the local clock, so it carries the clock's error. On
2026-11-04 the dev machine ran ~30 ms slow (`sntp`: +30.6 ms ± 13 ms), which