Archived
An orchestrator keeps the tracked set equal to Gamma's live markets (a scan at startup and every hour, new and closed markets every minute, reaching back 15 minutes for Gamma's indexing delay) and spreads them over lanes, each a connection and a book thread, all writing through the one writer. Lanes report each market's load in bytes per second; the orchestrator puts new markets on the least-loaded lane and moves the busiest markets off lanes over a 300 KB/s budget, busiest lane first. Differing books our own subscribe explains are subscription gaps, not losses, and every first book after a move is logged, so reconcile's mismatches are all accounted for in the log. A 75-minute live run tracked ~195k markets (~390k symbols) in 89 lanes.
40 lines
2.1 KiB
Markdown
40 lines
2.1 KiB
Markdown
# 0002. Lanes and an orchestrator
|
||
|
||
## Context
|
||
|
||
Every live Polymarket symbol is ~476k tokens (2026-11-04), an estimated
|
||
10–25k frames/s, near the ~30k frames/s one book thread did in sprint 02. One
|
||
connection can carry 50,000 tokens, but the server closes a connection it
|
||
can't send to fast enough (1013, "slow consumer"), so connections must be
|
||
many and read by threads that do nothing else. The set of live symbols
|
||
changes by ~20k markets a day, and Gamma, the only complete source, takes
|
||
minutes to scan (~2 from the collector).
|
||
|
||
## Decision
|
||
|
||
Symbols are spread over **lanes**: each lane is a connection and a book
|
||
thread of its own, within two limits (a byte budget for its load, and 5,000
|
||
tokens), and every lane's book thread writes to the one writer. An **orchestrator** thread is the single
|
||
owner of what is tracked: the market → lane map, the lanes and the timers. It
|
||
adds a new live market to the least-loaded lane, opening one when all are
|
||
full, removes markets that are gone, and moves the busiest markets off a lane
|
||
over its budget, from the load each lane reports every minute. Two fetcher threads only fetch Gamma (the
|
||
full scan; the new and closed markets since a time) and stream what they read
|
||
to it.
|
||
|
||
## Why
|
||
|
||
A lane shards the book state by construction: no routing stage between
|
||
connections and book threads, nothing shared, and redundancy (M4) adds
|
||
connections to a lane rather than reshaping the pipeline. A router to book
|
||
shards would decouple connection and book thread counts, which nothing needs
|
||
yet. The orchestrator keeps every decision in one thread that never blocks on
|
||
HTTP, which is where M4's source switching and M5's recycling will live; the
|
||
fetchers stay stateless so a scan never holds up the minute check.
|
||
Trade-off: a move unsubscribes a market on one lane before subscribing it on
|
||
the other, so it costs a few ms of its changes (a subscription gap); moving it
|
||
without that gap means two lanes writing one symbol, which is M4's source
|
||
switching. Moves are rate-limited so balancing doesn't churn. Balancing by
|
||
token count alone was tried first: the first live run had one lane closed by
|
||
the server six times, its markets created together and busy together.
|