Archived
feat(latency): latency experiments, LATENCY.md, Kalshi's slow-connection recycle
Extension 4. `engineering-challenge latency EXCHANGE OUT.csv` opens 16 connections to the same busy markets and writes every copy of every change; `scripts/latency/analyse.py` gives each connection's speed, whether it persists, what keeping the fastest would gain, lateness by address, and lag with the clock's offset. LATENCY.md writes up four runs: on Polymarket speed persists but is worth under a millisecond, at the median only; on Kalshi's demo keeping the fastest pair lowers p99 by 28% and 42%, the threshold agreed before the runs. So the collector now recycles a Kalshi connection whose mean lateness stays, 3 minutes running, in the slowest quarter and above twice everyone's, ahead of its hourly turn. Kalshi's rule measures each duplicate's lateness; the orchestrator judges once a minute. A duplicate's lag in Polymarket's minute line is now its "lateness". Roadmap milestone 7, roadmap/07.
This commit is contained in:
22 files changed
+2132
-55
No files matched your search
@@ -20,10 +20,11 @@ outside this repo.
|
||||
- `GLOSSARY.md`: the terms (symbol, connection, load, first one wins,
|
||||
duplicate, held, rescued, follow, resync, loss, subscription gap, split,
|
||||
quiet, resubscribe, silent, recycle, sequence number, clock, Snapshot,
|
||||
Delta).
|
||||
Delta, lateness, lag, speed).
|
||||
- `src/core/`: what is the same for every exchange (ADR 0004).
|
||||
`orchestrator.rs`: what is tracked, which connections carry each market,
|
||||
their balancing and their recycling; `socket.rs`: a connection (read,
|
||||
their balancing and their recycling (hourly, and sooner for a Kalshi
|
||||
connection that stays among the slowest); `socket.rs`: a connection (read,
|
||||
stamp, decode, route; reconnects when dropped, silent, lost a message or
|
||||
recycled); `feed.rs`: a book thread's shell (Snapshots, moves, the quiet
|
||||
watch) and the `Rule` trait; `mod.rs`: the `Exchange`, `Session`, `Market`
|
||||
@@ -38,6 +39,11 @@ outside this repo.
|
||||
`src/bin/reconcile.rs`: the output check, reading the log's cause lines
|
||||
(`src/cause.rs`). `tests/fixtures/`: recordings of the live feeds that the
|
||||
tests replay.
|
||||
- `src/latency.rs` + `scripts/latency/analyse.py`: the latency probe
|
||||
(`engineering-challenge latency EXCHANGE OUT.csv`, until SIGINT: one CSV
|
||||
line per copy of a change on each of 16 connections) and its analysis
|
||||
(speed, persistence, racing, by address, lag); `LATENCY.md` is the
|
||||
writeup (roadmap/07).
|
||||
- `src/util/` and `sbe_schemas/`: the template's SBE machinery, used as given so
|
||||
output stays readable by the template's `AlignedSbeReader`. Our code is the
|
||||
rest of `src/`.
|
||||
@@ -102,6 +108,14 @@ and let the user settle it. Claude owns the data model.
|
||||
- **Live data beats the docs**: Polymarket's market channel has rules its
|
||||
docs don't state (`docs/notes.md`); check a protocol assumption on a
|
||||
recording of the real feed (`tests/fixtures/record.py`) before relying on it.
|
||||
- **The workspace can't choose what the exchange sees**: a NAT rewrites
|
||||
every connection's source port (local 41000 left as 36996, then as 37086
|
||||
a minute later; `curl --local-port P https://ifconfig.me/all.json`), and
|
||||
there is no IPv6 route (connections to an IPv6 address fail with "Network
|
||||
is unreachable"). The probe uses IPv4 addresses only.
|
||||
- **Python checks run through `uvx`**: no ruff or pyflakes is installed;
|
||||
`scripts/ci.sh` runs `uvx ruff@0.15.0` (config in
|
||||
`scripts/latency/ruff.toml`).
|
||||
|
||||
## Workflow differences
|
||||
|
||||
|
||||
Reference in new issue
Block a user