feat(latency): latency experiments, LATENCY.md, Kalshi's slow-connection recycle

Extension 4. `engineering-challenge latency EXCHANGE OUT.csv` opens 16
connections to the same busy markets and writes every copy of every change;
`scripts/latency/analyse.py` gives each connection's speed, whether it
persists, what keeping the fastest would gain, lateness by address, and lag
with the clock's offset. LATENCY.md writes up four runs: on Polymarket speed
persists but is worth under a millisecond, at the median only; on Kalshi's
demo keeping the fastest pair lowers p99 by 28% and 42%, the threshold
agreed before the runs.

So the collector now recycles a Kalshi connection whose mean lateness stays,
3 minutes running, in the slowest quarter and above twice everyone's, ahead
of its hourly turn. Kalshi's rule measures each duplicate's lateness; the
orchestrator judges once a minute. A duplicate's lag in Polymarket's minute
line is now its "lateness".

Roadmap milestone 7, roadmap/07.
This commit is contained in:
pascal committed 2026-11-12 15:05:25 +01:00
1 parent 1a5bbcd602
commit 515e85941d
22 files changed
+2132 -55

No files matched your search

+16 -2
View File
@@ -20,10 +20,11 @@ outside this repo.
- `GLOSSARY.md`: the terms (symbol, connection, load, first one wins,
duplicate, held, rescued, follow, resync, loss, subscription gap, split,
quiet, resubscribe, silent, recycle, sequence number, clock, Snapshot,
Delta).
Delta, lateness, lag, speed).
- `src/core/`: what is the same for every exchange (ADR 0004).
`orchestrator.rs`: what is tracked, which connections carry each market,
their balancing and their recycling; `socket.rs`: a connection (read,
their balancing and their recycling (hourly, and sooner for a Kalshi
connection that stays among the slowest); `socket.rs`: a connection (read,
stamp, decode, route; reconnects when dropped, silent, lost a message or
recycled); `feed.rs`: a book thread's shell (Snapshots, moves, the quiet
watch) and the `Rule` trait; `mod.rs`: the `Exchange`, `Session`, `Market`
@@ -38,6 +39,11 @@ outside this repo.
`src/bin/reconcile.rs`: the output check, reading the log's cause lines
(`src/cause.rs`). `tests/fixtures/`: recordings of the live feeds that the
tests replay.
- `src/latency.rs` + `scripts/latency/analyse.py`: the latency probe
(`engineering-challenge latency EXCHANGE OUT.csv`, until SIGINT: one CSV
line per copy of a change on each of 16 connections) and its analysis
(speed, persistence, racing, by address, lag); `LATENCY.md` is the
writeup (roadmap/07).
- `src/util/` and `sbe_schemas/`: the template's SBE machinery, used as given so
output stays readable by the template's `AlignedSbeReader`. Our code is the
rest of `src/`.
@@ -102,6 +108,14 @@ and let the user settle it. Claude owns the data model.
- **Live data beats the docs**: Polymarket's market channel has rules its
docs don't state (`docs/notes.md`); check a protocol assumption on a
recording of the real feed (`tests/fixtures/record.py`) before relying on it.
- **The workspace can't choose what the exchange sees**: a NAT rewrites
every connection's source port (local 41000 left as 36996, then as 37086
a minute later; `curl --local-port P https://ifconfig.me/all.json`), and
there is no IPv6 route (connections to an IPv6 address fail with "Network
is unreachable"). The probe uses IPv4 addresses only.
- **Python checks run through `uvx`**: no ruff or pyflakes is installed;
`scripts/ci.sh` runs `uvx ruff@0.15.0` (config in
`scripts/latency/ruff.toml`).
## Workflow differences