This repository has been archived on 2026-11-16. You can view files and clone it. You cannot open issues or pull requests or push a commit.
Files
orderbook-collector/docs/adr/0002-lanes-and-orchestrator.md
T
pascal 02747b882f feat(polymarket)!: two connections per market, first one wins
Every live market is subscribed on two connections chosen at random, and its
book takes each change from whichever connection delivers it first: the book
hash says whether a change is a duplicate, the next one, or one to hold until
the change before it arrives. A change one connection misses costs nothing,
each Delta carries the earliest receive time, and a connection that drops is
covered by the other. When a change fits on no connection in time, the book
follows one connection unchecked and asks the other for a fresh book.
Connections now decode and route events to 16 book threads, independent of
them; a move leaves its old connection only once the new one delivers, so it
costs no change; a connection closed with 1013 twice within ten minutes is
split. reconcile reads the collector's log and fails only on a mismatch no
cause line explains. SIGTERM and SIGHUP stop the collector cleanly.

Breaking: lanes are gone (connections and book threads are separate), and
reconcile takes the log as a second argument.
2026-11-11 15:18:48 +00:00

44 lines
2.3 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# 0002. Lanes and an orchestrator
*Superseded in part by [0003](0003-first-one-wins.md) (sprint 04): the lanes are
gone, connections and book threads being separate sets; the orchestrator and
the fetchers stand.*
## Context
Every live Polymarket symbol is ~476k tokens (2026-11-04), an estimated
10–25k frames/s, near the ~30k frames/s one book thread did in sprint 02. One
connection can carry 50,000 tokens, but the server closes a connection it
can't send to fast enough (1013, "slow consumer"), so connections must be
many and read by threads that do nothing else. The set of live symbols
changes by ~20k markets a day, and Gamma, the only complete source, takes
minutes to scan (~2 from the collector).
## Decision
Symbols are spread over **lanes**: each lane is a connection and a book
thread of its own, within two limits (a byte budget for its load, and 5,000
tokens), and every lane's book thread writes to the one writer. An **orchestrator** thread is the single
owner of what is tracked: the market → lane map, the lanes and the timers. It
adds a new live market to the least-loaded lane, opening one when all are
full, removes markets that are gone, and moves the busiest markets off a lane
over its budget, from the load each lane reports every minute. Two fetcher threads only fetch Gamma (the
full scan; the new and closed markets since a time) and stream what they read
to it.
## Why
A lane shards the book state by construction: no routing stage between
connections and book threads, nothing shared, and redundancy (M4) adds
connections to a lane rather than reshaping the pipeline. A router to book
shards would decouple connection and book thread counts, which nothing needs
yet. The orchestrator keeps every decision in one thread that never blocks on
HTTP, which is where M4's source switching and M5's recycling will live; the
fetchers stay stateless so a scan never holds up the minute check.
Trade-off: a move unsubscribes a market on one lane before subscribing it on
the other, so it costs a few ms of its changes (a subscription gap); moving it
without that gap means two lanes writing one symbol, which is M4's source
switching. Moves are rate-limited so balancing doesn't churn. Balancing by
token count alone was tried first: the first live run had one lane closed by
the server six times, its markets created together and busy together.