This repository has been archived on 2026-11-16. You can view files and clone it. You cannot open issues or pull requests or push a commit.
Files
orderbook-collector/docs/adr/0002-lanes-and-orchestrator.md
pascal 02747b882f feat(polymarket)!: two connections per market, first one wins
Every live market is subscribed on two connections chosen at random, and its
book takes each change from whichever connection delivers it first: the book
hash says whether a change is a duplicate, the next one, or one to hold until
the change before it arrives. A change one connection misses costs nothing,
each Delta carries the earliest receive time, and a connection that drops is
covered by the other. When a change fits on no connection in time, the book
follows one connection unchecked and asks the other for a fresh book.
Connections now decode and route events to 16 book threads, independent of
them; a move leaves its old connection only once the new one delivers, so it
costs no change; a connection closed with 1013 twice within ten minutes is
split. reconcile reads the collector's log and fails only on a mismatch no
cause line explains. SIGTERM and SIGHUP stop the collector cleanly.

Breaking: lanes are gone (connections and book threads are separate), and
reconcile takes the log as a second argument.
2026-11-11 15:18:48 +00:00

2.3 KiB
Raw Permalink Blame History

0002. Lanes and an orchestrator

Superseded in part by 0003 (sprint 04): the lanes are gone, connections and book threads being separate sets; the orchestrator and the fetchers stand.

Context

Every live Polymarket symbol is ~476k tokens (2026-11-04), an estimated 10–25k frames/s, near the ~30k frames/s one book thread did in sprint 02. One connection can carry 50,000 tokens, but the server closes a connection it can't send to fast enough (1013, "slow consumer"), so connections must be many and read by threads that do nothing else. The set of live symbols changes by ~20k markets a day, and Gamma, the only complete source, takes minutes to scan (~2 from the collector).

Decision

Symbols are spread over lanes: each lane is a connection and a book thread of its own, within two limits (a byte budget for its load, and 5,000 tokens), and every lane's book thread writes to the one writer. An orchestrator thread is the single owner of what is tracked: the market → lane map, the lanes and the timers. It adds a new live market to the least-loaded lane, opening one when all are full, removes markets that are gone, and moves the busiest markets off a lane over its budget, from the load each lane reports every minute. Two fetcher threads only fetch Gamma (the full scan; the new and closed markets since a time) and stream what they read to it.

Why

A lane shards the book state by construction: no routing stage between connections and book threads, nothing shared, and redundancy (M4) adds connections to a lane rather than reshaping the pipeline. A router to book shards would decouple connection and book thread counts, which nothing needs yet. The orchestrator keeps every decision in one thread that never blocks on HTTP, which is where M4's source switching and M5's recycling will live; the fetchers stay stateless so a scan never holds up the minute check. Trade-off: a move unsubscribes a market on one lane before subscribing it on the other, so it costs a few ms of its changes (a subscription gap); moving it without that gap means two lanes writing one symbol, which is M4's source switching. Moves are rate-limited so balancing doesn't churn. Balancing by token count alone was tried first: the first live run had one lane closed by the server six times, its markets created together and busy together.