feat(polymarket): track every live market over load-balanced lanes

An orchestrator keeps the tracked set equal to Gamma's live markets (a scan at
startup and every hour, new and closed markets every minute, reaching back 15
minutes for Gamma's indexing delay) and spreads them over lanes, each a
connection and a book thread, all writing through the one writer. Lanes report
each market's load in bytes per second; the orchestrator puts new markets on
the least-loaded lane and moves the busiest markets off lanes over a 300 KB/s
budget, busiest lane first. Differing books our own subscribe explains are
subscription gaps, not losses, and every first book after a move is logged,
so reconcile's mismatches are all accounted for in the log. A 75-minute live
run tracked ~195k markets (~390k symbols) in 89 lanes.
This commit is contained in:
pascal committed 2026-11-11 06:28:25 +00:00
1 parent adb7fee8d8
commit 8114a51dc2
18 files changed
+2826 -320

No files matched your search

+39
View File
@@ -0,0 +1,39 @@
# 0002. Lanes and an orchestrator
## Context
Every live Polymarket symbol is ~476k tokens (2026-11-04), an estimated
10–25k frames/s, near the ~30k frames/s one book thread did in sprint 02. One
connection can carry 50,000 tokens, but the server closes a connection it
can't send to fast enough (1013, "slow consumer"), so connections must be
many and read by threads that do nothing else. The set of live symbols
changes by ~20k markets a day, and Gamma, the only complete source, takes
minutes to scan (~2 from the collector).
## Decision
Symbols are spread over **lanes**: each lane is a connection and a book
thread of its own, within two limits (a byte budget for its load, and 5,000
tokens), and every lane's book thread writes to the one writer. An **orchestrator** thread is the single
owner of what is tracked: the market → lane map, the lanes and the timers. It
adds a new live market to the least-loaded lane, opening one when all are
full, removes markets that are gone, and moves the busiest markets off a lane
over its budget, from the load each lane reports every minute. Two fetcher threads only fetch Gamma (the
full scan; the new and closed markets since a time) and stream what they read
to it.
## Why
A lane shards the book state by construction: no routing stage between
connections and book threads, nothing shared, and redundancy (M4) adds
connections to a lane rather than reshaping the pipeline. A router to book
shards would decouple connection and book thread counts, which nothing needs
yet. The orchestrator keeps every decision in one thread that never blocks on
HTTP, which is where M4's source switching and M5's recycling will live; the
fetchers stay stateless so a scan never holds up the minute check.
Trade-off: a move unsubscribes a market on one lane before subscribing it on
the other, so it costs a few ms of its changes (a subscription gap); moving it
without that gap means two lanes writing one symbol, which is M4's source
switching. Moves are rate-limited so balancing doesn't churn. Balancing by
token count alone was tried first: the first live run had one lane closed by
the server six times, its markets created together and busy together.
+71 -5
View File
@@ -100,15 +100,81 @@ From a 60 s run of the M1 spike on one market, both tokens subscribed, and the
answers `PONG`. The spike survived 60 s without it, but it is required.
- **Server restarts:** the server can close the socket with code `Away`,
reason `server shutting down` (seen once in about 8 minutes of connections).
- **`last_trade_price`:** `asset_id`, `price`, `size`, `side`, `fee_rate_bps`,
`timestamp`, `transaction_hash` (one per trade, for one token).
- **Not observed:** `tick_size_change` and the custom-feature events. The docs
now show these only in an SDK's wrapped form (`topic` / `payload`,
camelCase), not the raw wire shape above.
- **`last_trade_price`:** `market`, `asset_id`, `price`, `size`, `side`,
`fee_rate_bps`, `timestamp`, `transaction_hash` (one per trade, for one
token).
- **Not observed:** `market_resolved`, in 5 minutes with
`custom_feature_enabled` on 10 tokens. The docs now show it only in an
SDK's wrapped form (`topic` / `payload`, camelCase), not the raw wire
shape above.
- **`new_market` is broadcast** (with `custom_feature_enabled`, 2026-11-04):
every market created, subscribed or not (69 in 5 minutes), with
`condition_id`, `assets_ids` / `clob_token_ids`, `question`, `active`
(`false` for markets created ahead), but no `order_min_size` or `neg_risk`,
which the book hash needs.
- **Tokens per connection (2026-11-04):** no cap seen: one connection served
50,000 tokens. But a first `{"type":"market"}` message with more than 500
tokens gets no opening books (1,000 to 10,000 tried; changes still flow),
while `subscribe` operations of 500 or 1,000 get theirs. A connection opened
with `"assets_ids": []` stays up and serves a later `subscribe`.
- **Slow consumers are closed:** a connection the server can't send to fast
enough is closed with code 1013, reason `slow consumer: send buffer full`
(3 of 5 test connections, with a Python reader parsing every frame at up to
~10,000 frames/s). The collector's reader, which only stamps, got it too:
six times in 8 minutes on the one connection carrying ~5,000 changes a
second (NFL quarter spreads, little traded); none on the others, the next
at ~1,300. No rate is safe (2026-11-10, server, 5 ms from the edge): four
connections ramped on random live tokens were closed at 0.7, 2.1, 2.3 and
5.1 MB/s (minute averages of frame bytes), with subscribes of 5,000 or 500
tokens alike. 5,000 random tokens now carry ~380 frames/s, ~200 KB/s.
Closes come together: 5 of 20 connections (~1 MB/s each) were closed
within 45 s of each other, one at 0.46 MB/s.
- **A subscribe hides a few ms of changes** (smoke run, 2026-11-04): after
subscribing a market on a busy connection, the first check of its opening
book often failed, and the fresh book differed from ours at levels none of
our changes had touched, though the opening book matched its own hash. The
server never sent those changes: like the resync's unsubscribe/subscribe,
any subscribe opens this gap (glossary: subscription gap).
- **`tick_size_change`** (recorded 2026-11-10 on every live token: 62 markets
in 10 minutes, all `0.01` to `0.001`): one object per token, both tokens of
the market, each sent twice on the same connection, identical:
`{"market", "asset_id", "old_tick_size", "new_tick_size", "timestamp",
"event_type": "tick_size_change"}`.
- **Load (2026-11-04):** a random 5,000 of the live tokens gave 133 frames/s
and 266 changes/s; their books had a median of 5 levels (mean 10.9, p90 29,
max 136), 15% empty. The top 50,000 by volume gave ~10,000 frames/s.
- **REST from scripts:** `clob.polymarket.com` answers Python's `urllib` with
403 (curl and `ureq` get 200), most likely Cloudflare filtering by user agent.
Fetch with curl in throwaway scripts.
## Gamma, the market listing (observed 2026-11-04)
- **Size:** 238,308 markets `active` and not `closed`; 238,201 of them with
`enableOrderBook` and `acceptingOrders` (~476k tokens). 32,092 still accept
orders past their `endDate`, waiting to be resolved: the end date doesn't
mean closed. ~20k markets are created a day, mostly crypto "Up or Down"
markets created a day ahead and accepting orders at once.
- **Pages of 100**, whatever `limit` asks. `/markets` refuses offsets past
2,000 ("use /markets/keyset"); `/markets/keyset` pages with `after_cursor`
(the `next_cursor` of the last page) and has no such limit. A full keyset
scan of the live markets took 6 min 41 s.
- **Sorting works, date filters don't:** `order=createdAt&ascending=false`
(newest first) and `closed=true&order=closedTime&ascending=false` sort both
endpoints, and keyset pages keep the order across cursors. `created_at_min`
and `closed_time_min` are ignored. 2–20 markets were created and 10–18
closed a minute; more than 420 a minute changed `updatedAt`.
- **`conditionId`** is the websocket's `market`; `clobTokenIds` (a JSON array
inside a string) are its symbols.
- **Null hash inputs:** a null `orderMinSize` (31 live markets) hashes as
`"0"`, what REST's `/book` reports; a null `negRisk` (230) as `false`.
- **Scan speed:** the full keyset scan took 6 min 41 s from a Python script
spawning curl per page, ~2 minutes from the collector (`ureq`, one agent),
under a minute from the server (2026-11-10).
- **Indexing delay (2026-11-10):** new markets appear in the newest-first
listing 219 to 296 s after their `createdAt` (8 markets, polled every
20 s): a check reaching back 5 minutes misses some. 195,130 markets were
live that evening.
## Clock accuracy bounds time accuracy
`received_at` comes from the local clock, so it carries the clock's error. On
2026-11-04 the dev machine ran ~30 ms slow (`sntp`: +30.6 ms ± 13 ms), which
+36 -19
View File
@@ -42,8 +42,10 @@ The interacting decisions it is made of:
([ADR 0001](adr/0001-concurrency-model.md)).
- **Stamping point:** in the socket-read path, immediately, so as little as
possible sits between the wire and the receive timestamp.
- **Book ownership:** shard symbols so each book is owned by exactly one worker
(lock-free) rather than a shared, locked map.
- **Book ownership:** settled in sprint 03 — symbols are spread over lanes,
each a connection and a book thread of its own, so each book is owned by
exactly one thread; an orchestrator assigns them
([ADR 0002](adr/0002-lanes-and-orchestrator.md)).
- **Backpressure:** bounded vs unbounded channels — the core correctness/perf
tension. Drop-on-full loses data (violates completeness); block adds latency
and fills socket buffers. This decision is the bet in miniature.
@@ -130,25 +132,37 @@ and drop from the global workflow for a job-challenge repo.
- Any exchange beyond Polymarket and Kalshi.
- Persistence/replay infrastructure beyond what the SBE output already gives.
## Pipeline, first sketch
## Pipeline
The primary thing we debate sprint by sprint (this project's equivalent of the
schema debate). Rough shape:
schema debate). As settled in sprint 03:
```
discovery ──▶ subscription manager ──▶ N socket readers (stamp receive time here)
│
route by symbol
▼
per-symbol book workers (fold, loss-detect, dedup sources)
│
▼
single output writer (SBE sink)
rescan fetcher ──findings──▶ ┌──────────────┐ ◀──findings── recent fetcher
(full Gamma scan) ◀─request─│ orchestrator │──request──▶ (new + closed since T)
└──┬────────▲──┘
Add / MoveIn / Remove, to │ │ load reports, each minute
the market's lane │ │ (bytes/s per market)
┌──────────────────────┼────────┴─────────────────┐
▼ ▼ ▼
lane 0: book thread ◀─frames─ connection lane 1: ... lane N: ...
│ └─subscribe / unsubscribe / resync─▶┘
└──────────── Snapshot | Delta ─────────────▶ writer thread (one)
```
Each stage and the channels between them are open until settled in a sprint.
Settled so far: OS threads, shared-nothing, no mutexes
([ADR 0001](adr/0001-concurrency-model.md)).
- The **orchestrator** owns what is tracked (market → lane, the lanes, the
timers) and decides; it never blocks on HTTP, so later coordination
(switching sources in M4, recycling in M5) lives there too.
- The two **fetchers** only page Gamma on request and stream what they read.
- Each **lane** is a connection (stamps receive time, sends subscriptions and
resyncs) and a book thread (fold, check, resync, Snapshots); M4 gives a lane
several connections and has its book thread choose the trusted source.
- One **writer** owns the output.
Settled: OS threads, shared-nothing, no mutexes
([ADR 0001](adr/0001-concurrency-model.md)); lanes and the orchestrator
([ADR 0002](adr/0002-lanes-and-orchestrator.md)). Details:
[sprint 03](../roadmap/03.%20full%20coverage.md).
## Data model, first sketch
@@ -167,7 +181,13 @@ Not a DB — an in-memory domain plus the fixed SBE output schema. Main entities
- **Book** (one per symbol): bids and asks as ordered price→quantity maps, plus
the last loss-detection marker seen, last exchange timestamp, and the id of the
source currently trusted (for redundancy).
- **Connection** (one websocket): the set of symbols it carries.
- **Connection** (one websocket): the set of symbols it carries, those of its
lane.
- **Lane**: a set of symbols, the book thread that owns their books, and the
connections that feed it. Live markets are assigned to lanes by
`conditionId`; Polymarket's lifecycle signal is Gamma (settled in sprint 03:
a market is live while active, not closed, with an order book and accepting
orders).
- **Source / redundancy:** several connections may feed the same symbol; one is
the trusted source at a time.
@@ -181,9 +201,6 @@ trusted source is chosen and switched, and emitting a fresh snapshot on switch).
- **Loss detection on Kalshi.** Its sequence semantics: checked in M6.
(Polymarket: settled in M2, by its book hash.)
- **Symbol lifecycle signal.** What marks a market "no longer trading"?
Polymarket's `new_market` / `market_resolved` events (with
`custom_feature_enabled`) are candidates; settle in M3.
- **Redundancy source-selection.** How to pick/switch the trusted source
(freshness, message rate, which source is ahead). Designed in the redundancy
sprint.
+33 -22
View File
@@ -30,33 +30,35 @@ abstraction is *extracted* once a second exchange exists, not designed up front.
Sprint: [../roadmap/02. book pipeline.md](../roadmap/02.%20book%20pipeline.md).
*Ends:* correct SBE output for a symbol subset; tests green.
3. **Full coverage: discovery + sharding (base challenge complete)** — *not started*
Discovery with diffing (subscribe new, unsubscribe gone) — REST polling, or
Polymarket's `new_market` / `market_resolved` events — using subscribe /
unsubscribe on open connections; load-balance all live regular-market symbols
across N sockets.
**Opening questions (from M2):** what "live" means, and so how many symbols:
Gamma lists at least 100,000 markets `active` and not `closed` (2026-11-04,
count capped), at least 13,000 of them traded in the last 24 h, against the
40 symbols M2 runs. At that size the minute Snapshots alone (~2 KB each)
would be ~0.5 TB a day for 200,000 symbols, mostly unchanged books; and
memory (~10 KB a book), sockets per connection, resyncs per minute and the
book thread's throughput (M2: ~30,000 frames/s on one thread, 43 frames/s
of load) all need measuring. Gamma pages past 10,000 markets only through
`/markets/keyset` with `after_cursor`.
3. **Full coverage: discovery + lanes (base challenge complete)** — *done (2026-11-11)*
Every live symbol (~476k tokens, 2026-11-04) spread over lanes, each a
connection and a book thread of its own; an orchestrator keeps the tracked
set equal to Gamma's live set (a scan at startup and every hour, new and
closed markets every minute) through subscribe / unsubscribe on open
connections.
Sprint: [../roadmap/03. full coverage.md](../roadmap/03.%20full%20coverage.md).
*Ends:* every live Polymarket symbol tracked — base challenge satisfied for
one exchange.
4. **Extension 1 — redundant websockets** — *not started*
Multiple connections per symbol; choose a trusted source and switch on
loss/quiet/delay; emit a fresh snapshot on every source switch.
Multiple connections per symbol (several connections in a lane, its book
thread choosing among them, the orchestrator coordinating); choose a
trusted source and switch on loss/quiet/delay; emit a fresh snapshot on
every source switch. Recovery from errors, with the lane budget scaling
itself down when connections still get 1013 closes (no rate is safe:
[notes.md](notes.md), so a budget set too high must correct itself).
**Async re-evaluation checkpoint:** before building this, reassess whether the
socket-management complexity (reconnect/timeout/select, redundancy) warrants
swapping the thread-only runtime for tokio (see ADR 0001).
The lane that holds markets busy together (the 5- and 15-minute crypto
markets: closed 16 times in 75 minutes, in bursts a minute's load
average doesn't show) is the case to beat.
Also: surviving a hard crash without losing the writer's in-memory chunk,
and closing the resync gap (the few ms of a symbol's changes the server
doesn't send between our unsubscribe and subscribe), both accepted as
ceilings in M2.
a move's Remove reaching the old lane before the new lane's first
Snapshot, the memory growth seen in sprint 03 (~90 MB every 10 minutes
over 75 minutes), and closing the subscription gap (the few ms of a
symbol's changes the server doesn't send between our unsubscribe and
subscribe); the crash chunk and the gap were accepted as ceilings in M2.
*Ends:* redundancy demonstrably reduces data loss; a Snapshot on every source
switch.
@@ -106,7 +108,16 @@ abstraction is *extracted* once a second exchange exists, not designed up front.
- Observability: metrics, structured logging, dashboards.
- Config-driven symbol filtering and per-exchange tuning.
- Containerized packaging.
- Counting a reconnect's differing books as losses rather than resync gaps
(a reconnect marker through the frames channel; M2 labels them gaps).
- A `reconcile` that reads the collector's log, to tell resync gaps from
- Counting a reconnect's differing books as losses rather than subscription
gaps when the symbol was resyncing or not yet verified since its opening
book (a reconnect marker through the frames channel; they are labelled
gaps).
- A `reconcile` that reads the collector's log, to tell subscription gaps from
losses in its own report.
- Polymarket's `new_market` websocket event as a faster discovery signal (it
is broadcast, but lacks `orderMinSize` / `negRisk`, so Gamma is still
needed; `market_resolved` was never seen).
- Closing lanes that drain.
- Fewer Snapshots for unchanged books (the brief asks for every symbol every
minute: ~130 MB a minute at full coverage).
- Re-reading a market's `orderMinSize` / `negRisk` if they can change.