feat(polymarket): track every live market over load-balanced lanes

An orchestrator keeps the tracked set equal to Gamma's live markets (a scan at
startup and every hour, new and closed markets every minute, reaching back 15
minutes for Gamma's indexing delay) and spreads them over lanes, each a
connection and a book thread, all writing through the one writer. Lanes report
each market's load in bytes per second; the orchestrator puts new markets on
the least-loaded lane and moves the busiest markets off lanes over a 300 KB/s
budget, busiest lane first. Differing books our own subscribe explains are
subscription gaps, not losses, and every first book after a move is logged,
so reconcile's mismatches are all accounted for in the log. A 75-minute live
run tracked ~195k markets (~390k symbols) in 89 lanes.
This commit is contained in:
pascal committed 2026-11-11 06:28:25 +00:00
1 parent adb7fee8d8
commit 8114a51dc2
18 files changed
+2826 -320

No files matched your search

+29 -6
View File
@@ -15,6 +15,12 @@ The unit that has one order book: on Polymarket a token (one outcome of a
market), named by its token id.
_Avoid_: Asset, instrument, ticker (for Polymarket)
**Live**:
Said of a market (and its symbols) the exchange lists as open for trading on
its order book. A market stops being live when it closes or stops accepting
orders, not at its end date.
_Avoid_: Active (Gamma's flag, only one input of being live)
**Book hash**:
The exchange's fingerprint of a symbol's full book state, sent with every
full book and every change.
@@ -26,6 +32,21 @@ The collector's in-memory order book for one symbol: its bid and ask price
levels, each with its absolute quantity.
_Avoid_: Order book state, depth
**Lane**:
A set of symbols whose books one book thread owns, fed by its own
connections: one path from the wire to the writer. Every live symbol is in
exactly one lane.
_Avoid_: Shard (a technique, not this thing), group, partition
**Load**:
How much a market weighs on its connection: the bytes per second of its
frames. A lane's load is the sum of its markets'.
_Avoid_: Activity, volume (the exchange's traded amount, unrelated)
**Connection**:
One websocket to the exchange, carrying the symbols of one lane.
_Avoid_: Socket (for the concept; fine for the code), stream
**Receive time**:
When a message came off the socket, by the local clock.
_Avoid_: Arrival time, local timestamp
@@ -37,14 +58,16 @@ _Avoid_: Gap (implies sequence numbers)
**Resync**:
Asking the exchange for a fresh full book of a symbol and replacing ours with
it, whenever our book can't be verified. Our book was wrong (a loss) only if
the fresh one differs.
it, whenever our book can't be verified. Our book was wrong only if the fresh
one differs: a loss, or a subscription gap.
_Avoid_: Re-snapshot, rebuild, refresh
**Resync gap**:
The changes a symbol's resync itself keeps from us: the exchange holds them in
the fresh book but never sends them as changes.
_Avoid_: calling it a loss (the state stays right; only those updates are missing)
**Subscription gap**:
The changes our own subscribe keeps from us, whether it opens, resyncs or
moves a symbol: the exchange holds them in the book it sends but never sends
them as changes.
_Avoid_: Resync gap (the older, narrower name); calling it a loss (the state
stays right; only those updates are missing)
## Output