feat(latency): latency experiments, LATENCY.md, Kalshi's slow-connection recycle

Extension 4. `engineering-challenge latency EXCHANGE OUT.csv` opens 16
connections to the same busy markets and writes every copy of every change;
`scripts/latency/analyse.py` gives each connection's speed, whether it
persists, what keeping the fastest would gain, lateness by address, and lag
with the clock's offset. LATENCY.md writes up four runs: on Polymarket speed
persists but is worth under a millisecond, at the median only; on Kalshi's
demo keeping the fastest pair lowers p99 by 28% and 42%, the threshold
agreed before the runs.

So the collector now recycles a Kalshi connection whose mean lateness stays,
3 minutes running, in the slowest quarter and above twice everyone's, ahead
of its hourly turn. Kalshi's rule measures each duplicate's lateness; the
orchestrator judges once a minute. A duplicate's lag in Polymarket's minute
line is now its "lateness".

Roadmap milestone 7, roadmap/07.
This commit is contained in:
pascal committed 2026-11-12 15:05:25 +01:00
1 parent 1a5bbcd602
commit 515e85941d
22 files changed
+2132 -55

No files matched your search

+5 -2
View File
@@ -150,7 +150,8 @@ schema debate). As settled in sprints 04 and 05:
└──────┬───▲───┘ ──── recycle ────────▶ connections
Add / Move / Remove (a market and │ │ load reports, each minute
its connections), to its thread │ │ (bytes/s per market); markets
│ │ quiet twice on a connection
│ │ quiet twice on a connection;
│ │ lateness per connection (Kalshi)
┌──────▼───┴───┐
connection 0..N ──events───▶ │ book threads │ ──Snapshot | Delta──▶ writer
(read, stamp, parse, route) │ 0..15 │
@@ -160,7 +161,9 @@ schema debate). As settled in sprints 04 and 05:
- The **orchestrator** owns what is tracked (each market's two connections,
the timers) and decides: placing, moving, splitting, and recycling one
connection at a time, about hourly. It never blocks on HTTP.
connection at a time, about hourly, and sooner a Kalshi connection whose
mean lateness stays above twice everyone's (sprint 07, LATENCY.md). It
never blocks on HTTP.
- The two **fetchers** only page the exchange's market listing on request
and stream what they read.
- A **connection** reads, stamps the receive time, parses, and routes each