feat(polymarket): track every live market over load-balanced lanes

An orchestrator keeps the tracked set equal to Gamma's live markets (a scan at
startup and every hour, new and closed markets every minute, reaching back 15
minutes for Gamma's indexing delay) and spreads them over lanes, each a
connection and a book thread, all writing through the one writer. Lanes report
each market's load in bytes per second; the orchestrator puts new markets on
the least-loaded lane and moves the busiest markets off lanes over a 300 KB/s
budget, busiest lane first. Differing books our own subscribe explains are
subscription gaps, not losses, and every first book after a move is logged,
so reconcile's mismatches are all accounted for in the log. A 75-minute live
run tracked ~195k markets (~390k symbols) in 89 lanes.
This commit is contained in:
pascal committed 2026-11-11 06:28:25 +00:00
1 parent adb7fee8d8
commit 8114a51dc2
18 files changed
+2826 -320

No files matched your search

+39
View File
@@ -0,0 +1,39 @@
# 0002. Lanes and an orchestrator
## Context
Every live Polymarket symbol is ~476k tokens (2026-11-04), an estimated
10–25k frames/s, near the ~30k frames/s one book thread did in sprint 02. One
connection can carry 50,000 tokens, but the server closes a connection it
can't send to fast enough (1013, "slow consumer"), so connections must be
many and read by threads that do nothing else. The set of live symbols
changes by ~20k markets a day, and Gamma, the only complete source, takes
minutes to scan (~2 from the collector).
## Decision
Symbols are spread over **lanes**: each lane is a connection and a book
thread of its own, within two limits (a byte budget for its load, and 5,000
tokens), and every lane's book thread writes to the one writer. An **orchestrator** thread is the single
owner of what is tracked: the market → lane map, the lanes and the timers. It
adds a new live market to the least-loaded lane, opening one when all are
full, removes markets that are gone, and moves the busiest markets off a lane
over its budget, from the load each lane reports every minute. Two fetcher threads only fetch Gamma (the
full scan; the new and closed markets since a time) and stream what they read
to it.
## Why
A lane shards the book state by construction: no routing stage between
connections and book threads, nothing shared, and redundancy (M4) adds
connections to a lane rather than reshaping the pipeline. A router to book
shards would decouple connection and book thread counts, which nothing needs
yet. The orchestrator keeps every decision in one thread that never blocks on
HTTP, which is where M4's source switching and M5's recycling will live; the
fetchers stay stateless so a scan never holds up the minute check.
Trade-off: a move unsubscribes a market on one lane before subscribing it on
the other, so it costs a few ms of its changes (a subscription gap); moving it
without that gap means two lanes writing one symbol, which is M4's source
switching. Moves are rate-limited so balancing doesn't churn. Balancing by
token count alone was tried first: the first live run had one lane closed by
the server six times, its markets created together and busy together.