Archived
Every live market is subscribed on two connections chosen at random, and its book takes each change from whichever connection delivers it first: the book hash says whether a change is a duplicate, the next one, or one to hold until the change before it arrives. A change one connection misses costs nothing, each Delta carries the earliest receive time, and a connection that drops is covered by the other. When a change fits on no connection in time, the book follows one connection unchecked and asks the other for a fresh book. Connections now decode and route events to 16 book threads, independent of them; a move leaves its old connection only once the new one delivers, so it costs no change; a connection closed with 1013 twice within ten minutes is split. reconcile reads the collector's log and fails only on a mismatch no cause line explains. SIGTERM and SIGHUP stop the collector cleanly. Breaking: lanes are gone (connections and book threads are separate), and reconcile takes the log as a second argument.
44 lines
2.3 KiB
Markdown
44 lines
2.3 KiB
Markdown
# 0002. Lanes and an orchestrator
|
||
|
||
*Superseded in part by [0003](0003-first-one-wins.md) (sprint 04): the lanes are
|
||
gone, connections and book threads being separate sets; the orchestrator and
|
||
the fetchers stand.*
|
||
|
||
## Context
|
||
|
||
Every live Polymarket symbol is ~476k tokens (2026-11-04), an estimated
|
||
10–25k frames/s, near the ~30k frames/s one book thread did in sprint 02. One
|
||
connection can carry 50,000 tokens, but the server closes a connection it
|
||
can't send to fast enough (1013, "slow consumer"), so connections must be
|
||
many and read by threads that do nothing else. The set of live symbols
|
||
changes by ~20k markets a day, and Gamma, the only complete source, takes
|
||
minutes to scan (~2 from the collector).
|
||
|
||
## Decision
|
||
|
||
Symbols are spread over **lanes**: each lane is a connection and a book
|
||
thread of its own, within two limits (a byte budget for its load, and 5,000
|
||
tokens), and every lane's book thread writes to the one writer. An **orchestrator** thread is the single
|
||
owner of what is tracked: the market → lane map, the lanes and the timers. It
|
||
adds a new live market to the least-loaded lane, opening one when all are
|
||
full, removes markets that are gone, and moves the busiest markets off a lane
|
||
over its budget, from the load each lane reports every minute. Two fetcher threads only fetch Gamma (the
|
||
full scan; the new and closed markets since a time) and stream what they read
|
||
to it.
|
||
|
||
## Why
|
||
|
||
A lane shards the book state by construction: no routing stage between
|
||
connections and book threads, nothing shared, and redundancy (M4) adds
|
||
connections to a lane rather than reshaping the pipeline. A router to book
|
||
shards would decouple connection and book thread counts, which nothing needs
|
||
yet. The orchestrator keeps every decision in one thread that never blocks on
|
||
HTTP, which is where M4's source switching and M5's recycling will live; the
|
||
fetchers stay stateless so a scan never holds up the minute check.
|
||
Trade-off: a move unsubscribes a market on one lane before subscribing it on
|
||
the other, so it costs a few ms of its changes (a subscription gap); moving it
|
||
without that gap means two lanes writing one symbol, which is M4's source
|
||
switching. Moves are rate-limited so balancing doesn't churn. Balancing by
|
||
token count alone was tried first: the first live run had one lane closed by
|
||
the server six times, its markets created together and busy together.
|