feat(polymarket): track every live market over load-balanced lanes

An orchestrator keeps the tracked set equal to Gamma's live markets (a scan at
startup and every hour, new and closed markets every minute, reaching back 15
minutes for Gamma's indexing delay) and spreads them over lanes, each a
connection and a book thread, all writing through the one writer. Lanes report
each market's load in bytes per second; the orchestrator puts new markets on
the least-loaded lane and moves the busiest markets off lanes over a 300 KB/s
budget, busiest lane first. Differing books our own subscribe explains are
subscription gaps, not losses, and every first book after a move is logged,
so reconcile's mismatches are all accounted for in the log. A 75-minute live
run tracked ~195k markets (~390k symbols) in 89 lanes.
This commit is contained in:
pascal committed 2026-11-11 06:28:25 +00:00
1 parent adb7fee8d8
commit 8114a51dc2
18 files changed
+2826 -320

No files matched your search

+8 -2
View File
@@ -1,10 +1,16 @@
# Test fixtures
A recording of Polymarket's market channel, replayed by the tests in
`src/polymarket/feed.rs` (no network in tests).
`src/polymarket/feed.rs`, and Gamma pages read by `src/polymarket/gamma.rs`
(no network in tests).
- `gamma-markets.json`: Gamma's top 20 markets by 24 h volume on 2026-11-04,
trimmed to the fields the collector reads.
trimmed to the fields the collector reads. `conditionId`, `active`, `closed`
and `acceptingOrders` were added in sprint 03 (the ids from the recorded
books' `market`).
- `gamma-created.json`, `gamma-closed.json`: one `/markets/keyset` page each
(`limit=10`) of the newest live markets by `createdAt` and the latest closed
by `closedTime`, 2026-11-04 ~17:00 UTC, trimmed the same way.
- `polymarket-frames.jsonl`: the first 90 s (14:24:44 to 14:26:14 UTC, so two
minute boundaries) of the market channel for those markets' 40 tokens, one
line per frame: `{"received_at_us": <receive time, µs>, "text": <frame>}`.