Archived
feat(polymarket): track every live market over load-balanced lanes
An orchestrator keeps the tracked set equal to Gamma's live markets (a scan at startup and every hour, new and closed markets every minute, reaching back 15 minutes for Gamma's indexing delay) and spreads them over lanes, each a connection and a book thread, all writing through the one writer. Lanes report each market's load in bytes per second; the orchestrator puts new markets on the least-loaded lane and moves the busiest markets off lanes over a 300 KB/s budget, busiest lane first. Differing books our own subscribe explains are subscription gaps, not losses, and every first book after a move is logged, so reconcile's mismatches are all accounted for in the log. A 75-minute live run tracked ~195k markets (~390k symbols) in 89 lanes.
This commit is contained in:
18 files changed
+2826
-320
No files matched your search
@@ -0,0 +1,39 @@
|
||||
# 0002. Lanes and an orchestrator
|
||||
|
||||
## Context
|
||||
|
||||
Every live Polymarket symbol is ~476k tokens (2026-11-04), an estimated
|
||||
10–25k frames/s, near the ~30k frames/s one book thread did in sprint 02. One
|
||||
connection can carry 50,000 tokens, but the server closes a connection it
|
||||
can't send to fast enough (1013, "slow consumer"), so connections must be
|
||||
many and read by threads that do nothing else. The set of live symbols
|
||||
changes by ~20k markets a day, and Gamma, the only complete source, takes
|
||||
minutes to scan (~2 from the collector).
|
||||
|
||||
## Decision
|
||||
|
||||
Symbols are spread over **lanes**: each lane is a connection and a book
|
||||
thread of its own, within two limits (a byte budget for its load, and 5,000
|
||||
tokens), and every lane's book thread writes to the one writer. An **orchestrator** thread is the single
|
||||
owner of what is tracked: the market → lane map, the lanes and the timers. It
|
||||
adds a new live market to the least-loaded lane, opening one when all are
|
||||
full, removes markets that are gone, and moves the busiest markets off a lane
|
||||
over its budget, from the load each lane reports every minute. Two fetcher threads only fetch Gamma (the
|
||||
full scan; the new and closed markets since a time) and stream what they read
|
||||
to it.
|
||||
|
||||
## Why
|
||||
|
||||
A lane shards the book state by construction: no routing stage between
|
||||
connections and book threads, nothing shared, and redundancy (M4) adds
|
||||
connections to a lane rather than reshaping the pipeline. A router to book
|
||||
shards would decouple connection and book thread counts, which nothing needs
|
||||
yet. The orchestrator keeps every decision in one thread that never blocks on
|
||||
HTTP, which is where M4's source switching and M5's recycling will live; the
|
||||
fetchers stay stateless so a scan never holds up the minute check.
|
||||
Trade-off: a move unsubscribes a market on one lane before subscribing it on
|
||||
the other, so it costs a few ms of its changes (a subscription gap); moving it
|
||||
without that gap means two lanes writing one symbol, which is M4's source
|
||||
switching. Moves are rate-limited so balancing doesn't churn. Balancing by
|
||||
token count alone was tried first: the first live run had one lane closed by
|
||||
the server six times, its markets created together and busy together.
|
||||
+71
-5
@@ -100,15 +100,81 @@ From a 60 s run of the M1 spike on one market, both tokens subscribed, and the
|
||||
answers `PONG`. The spike survived 60 s without it, but it is required.
|
||||
- **Server restarts:** the server can close the socket with code `Away`,
|
||||
reason `server shutting down` (seen once in about 8 minutes of connections).
|
||||
- **`last_trade_price`:** `asset_id`, `price`, `size`, `side`, `fee_rate_bps`,
|
||||
`timestamp`, `transaction_hash` (one per trade, for one token).
|
||||
- **Not observed:** `tick_size_change` and the custom-feature events. The docs
|
||||
now show these only in an SDK's wrapped form (`topic` / `payload`,
|
||||
camelCase), not the raw wire shape above.
|
||||
- **`last_trade_price`:** `market`, `asset_id`, `price`, `size`, `side`,
|
||||
`fee_rate_bps`, `timestamp`, `transaction_hash` (one per trade, for one
|
||||
token).
|
||||
- **Not observed:** `market_resolved`, in 5 minutes with
|
||||
`custom_feature_enabled` on 10 tokens. The docs now show it only in an
|
||||
SDK's wrapped form (`topic` / `payload`, camelCase), not the raw wire
|
||||
shape above.
|
||||
- **`new_market` is broadcast** (with `custom_feature_enabled`, 2026-11-04):
|
||||
every market created, subscribed or not (69 in 5 minutes), with
|
||||
`condition_id`, `assets_ids` / `clob_token_ids`, `question`, `active`
|
||||
(`false` for markets created ahead), but no `order_min_size` or `neg_risk`,
|
||||
which the book hash needs.
|
||||
- **Tokens per connection (2026-11-04):** no cap seen: one connection served
|
||||
50,000 tokens. But a first `{"type":"market"}` message with more than 500
|
||||
tokens gets no opening books (1,000 to 10,000 tried; changes still flow),
|
||||
while `subscribe` operations of 500 or 1,000 get theirs. A connection opened
|
||||
with `"assets_ids": []` stays up and serves a later `subscribe`.
|
||||
- **Slow consumers are closed:** a connection the server can't send to fast
|
||||
enough is closed with code 1013, reason `slow consumer: send buffer full`
|
||||
(3 of 5 test connections, with a Python reader parsing every frame at up to
|
||||
~10,000 frames/s). The collector's reader, which only stamps, got it too:
|
||||
six times in 8 minutes on the one connection carrying ~5,000 changes a
|
||||
second (NFL quarter spreads, little traded); none on the others, the next
|
||||
at ~1,300. No rate is safe (2026-11-10, server, 5 ms from the edge): four
|
||||
connections ramped on random live tokens were closed at 0.7, 2.1, 2.3 and
|
||||
5.1 MB/s (minute averages of frame bytes), with subscribes of 5,000 or 500
|
||||
tokens alike. 5,000 random tokens now carry ~380 frames/s, ~200 KB/s.
|
||||
Closes come together: 5 of 20 connections (~1 MB/s each) were closed
|
||||
within 45 s of each other, one at 0.46 MB/s.
|
||||
- **A subscribe hides a few ms of changes** (smoke run, 2026-11-04): after
|
||||
subscribing a market on a busy connection, the first check of its opening
|
||||
book often failed, and the fresh book differed from ours at levels none of
|
||||
our changes had touched, though the opening book matched its own hash. The
|
||||
server never sent those changes: like the resync's unsubscribe/subscribe,
|
||||
any subscribe opens this gap (glossary: subscription gap).
|
||||
- **`tick_size_change`** (recorded 2026-11-10 on every live token: 62 markets
|
||||
in 10 minutes, all `0.01` to `0.001`): one object per token, both tokens of
|
||||
the market, each sent twice on the same connection, identical:
|
||||
`{"market", "asset_id", "old_tick_size", "new_tick_size", "timestamp",
|
||||
"event_type": "tick_size_change"}`.
|
||||
- **Load (2026-11-04):** a random 5,000 of the live tokens gave 133 frames/s
|
||||
and 266 changes/s; their books had a median of 5 levels (mean 10.9, p90 29,
|
||||
max 136), 15% empty. The top 50,000 by volume gave ~10,000 frames/s.
|
||||
- **REST from scripts:** `clob.polymarket.com` answers Python's `urllib` with
|
||||
403 (curl and `ureq` get 200), most likely Cloudflare filtering by user agent.
|
||||
Fetch with curl in throwaway scripts.
|
||||
|
||||
## Gamma, the market listing (observed 2026-11-04)
|
||||
|
||||
- **Size:** 238,308 markets `active` and not `closed`; 238,201 of them with
|
||||
`enableOrderBook` and `acceptingOrders` (~476k tokens). 32,092 still accept
|
||||
orders past their `endDate`, waiting to be resolved: the end date doesn't
|
||||
mean closed. ~20k markets are created a day, mostly crypto "Up or Down"
|
||||
markets created a day ahead and accepting orders at once.
|
||||
- **Pages of 100**, whatever `limit` asks. `/markets` refuses offsets past
|
||||
2,000 ("use /markets/keyset"); `/markets/keyset` pages with `after_cursor`
|
||||
(the `next_cursor` of the last page) and has no such limit. A full keyset
|
||||
scan of the live markets took 6 min 41 s.
|
||||
- **Sorting works, date filters don't:** `order=createdAt&ascending=false`
|
||||
(newest first) and `closed=true&order=closedTime&ascending=false` sort both
|
||||
endpoints, and keyset pages keep the order across cursors. `created_at_min`
|
||||
and `closed_time_min` are ignored. 2–20 markets were created and 10–18
|
||||
closed a minute; more than 420 a minute changed `updatedAt`.
|
||||
- **`conditionId`** is the websocket's `market`; `clobTokenIds` (a JSON array
|
||||
inside a string) are its symbols.
|
||||
- **Null hash inputs:** a null `orderMinSize` (31 live markets) hashes as
|
||||
`"0"`, what REST's `/book` reports; a null `negRisk` (230) as `false`.
|
||||
- **Scan speed:** the full keyset scan took 6 min 41 s from a Python script
|
||||
spawning curl per page, ~2 minutes from the collector (`ureq`, one agent),
|
||||
under a minute from the server (2026-11-10).
|
||||
- **Indexing delay (2026-11-10):** new markets appear in the newest-first
|
||||
listing 219 to 296 s after their `createdAt` (8 markets, polled every
|
||||
20 s): a check reaching back 5 minutes misses some. 195,130 markets were
|
||||
live that evening.
|
||||
|
||||
## Clock accuracy bounds time accuracy
|
||||
`received_at` comes from the local clock, so it carries the clock's error. On
|
||||
2026-11-04 the dev machine ran ~30 ms slow (`sntp`: +30.6 ms ± 13 ms), which
|
||||
|
||||
+36
-19
@@ -42,8 +42,10 @@ The interacting decisions it is made of:
|
||||
([ADR 0001](adr/0001-concurrency-model.md)).
|
||||
- **Stamping point:** in the socket-read path, immediately, so as little as
|
||||
possible sits between the wire and the receive timestamp.
|
||||
- **Book ownership:** shard symbols so each book is owned by exactly one worker
|
||||
(lock-free) rather than a shared, locked map.
|
||||
- **Book ownership:** settled in sprint 03 — symbols are spread over lanes,
|
||||
each a connection and a book thread of its own, so each book is owned by
|
||||
exactly one thread; an orchestrator assigns them
|
||||
([ADR 0002](adr/0002-lanes-and-orchestrator.md)).
|
||||
- **Backpressure:** bounded vs unbounded channels — the core correctness/perf
|
||||
tension. Drop-on-full loses data (violates completeness); block adds latency
|
||||
and fills socket buffers. This decision is the bet in miniature.
|
||||
@@ -130,25 +132,37 @@ and drop from the global workflow for a job-challenge repo.
|
||||
- Any exchange beyond Polymarket and Kalshi.
|
||||
- Persistence/replay infrastructure beyond what the SBE output already gives.
|
||||
|
||||
## Pipeline, first sketch
|
||||
## Pipeline
|
||||
|
||||
The primary thing we debate sprint by sprint (this project's equivalent of the
|
||||
schema debate). Rough shape:
|
||||
schema debate). As settled in sprint 03:
|
||||
|
||||
```
|
||||
discovery ──▶ subscription manager ──▶ N socket readers (stamp receive time here)
|
||||
│
|
||||
route by symbol
|
||||
▼
|
||||
per-symbol book workers (fold, loss-detect, dedup sources)
|
||||
│
|
||||
▼
|
||||
single output writer (SBE sink)
|
||||
rescan fetcher ──findings──▶ ┌──────────────┐ ◀──findings── recent fetcher
|
||||
(full Gamma scan) ◀─request─│ orchestrator │──request──▶ (new + closed since T)
|
||||
└──┬────────▲──┘
|
||||
Add / MoveIn / Remove, to │ │ load reports, each minute
|
||||
the market's lane │ │ (bytes/s per market)
|
||||
┌──────────────────────┼────────┴─────────────────┐
|
||||
▼ ▼ ▼
|
||||
lane 0: book thread ◀─frames─ connection lane 1: ... lane N: ...
|
||||
│ └─subscribe / unsubscribe / resync─▶┘
|
||||
└──────────── Snapshot | Delta ─────────────▶ writer thread (one)
|
||||
```
|
||||
|
||||
Each stage and the channels between them are open until settled in a sprint.
|
||||
Settled so far: OS threads, shared-nothing, no mutexes
|
||||
([ADR 0001](adr/0001-concurrency-model.md)).
|
||||
- The **orchestrator** owns what is tracked (market → lane, the lanes, the
|
||||
timers) and decides; it never blocks on HTTP, so later coordination
|
||||
(switching sources in M4, recycling in M5) lives there too.
|
||||
- The two **fetchers** only page Gamma on request and stream what they read.
|
||||
- Each **lane** is a connection (stamps receive time, sends subscriptions and
|
||||
resyncs) and a book thread (fold, check, resync, Snapshots); M4 gives a lane
|
||||
several connections and has its book thread choose the trusted source.
|
||||
- One **writer** owns the output.
|
||||
|
||||
Settled: OS threads, shared-nothing, no mutexes
|
||||
([ADR 0001](adr/0001-concurrency-model.md)); lanes and the orchestrator
|
||||
([ADR 0002](adr/0002-lanes-and-orchestrator.md)). Details:
|
||||
[sprint 03](../roadmap/03.%20full%20coverage.md).
|
||||
|
||||
## Data model, first sketch
|
||||
|
||||
@@ -167,7 +181,13 @@ Not a DB — an in-memory domain plus the fixed SBE output schema. Main entities
|
||||
- **Book** (one per symbol): bids and asks as ordered price→quantity maps, plus
|
||||
the last loss-detection marker seen, last exchange timestamp, and the id of the
|
||||
source currently trusted (for redundancy).
|
||||
- **Connection** (one websocket): the set of symbols it carries.
|
||||
- **Connection** (one websocket): the set of symbols it carries, those of its
|
||||
lane.
|
||||
- **Lane**: a set of symbols, the book thread that owns their books, and the
|
||||
connections that feed it. Live markets are assigned to lanes by
|
||||
`conditionId`; Polymarket's lifecycle signal is Gamma (settled in sprint 03:
|
||||
a market is live while active, not closed, with an order book and accepting
|
||||
orders).
|
||||
- **Source / redundancy:** several connections may feed the same symbol; one is
|
||||
the trusted source at a time.
|
||||
|
||||
@@ -181,9 +201,6 @@ trusted source is chosen and switched, and emitting a fresh snapshot on switch).
|
||||
|
||||
- **Loss detection on Kalshi.** Its sequence semantics: checked in M6.
|
||||
(Polymarket: settled in M2, by its book hash.)
|
||||
- **Symbol lifecycle signal.** What marks a market "no longer trading"?
|
||||
Polymarket's `new_market` / `market_resolved` events (with
|
||||
`custom_feature_enabled`) are candidates; settle in M3.
|
||||
- **Redundancy source-selection.** How to pick/switch the trusted source
|
||||
(freshness, message rate, which source is ahead). Designed in the redundancy
|
||||
sprint.
|
||||
+33
-22
@@ -30,33 +30,35 @@ abstraction is *extracted* once a second exchange exists, not designed up front.
|
||||
Sprint: [../roadmap/02. book pipeline.md](../roadmap/02.%20book%20pipeline.md).
|
||||
*Ends:* correct SBE output for a symbol subset; tests green.
|
||||
|
||||
3. **Full coverage: discovery + sharding (base challenge complete)** — *not started*
|
||||
Discovery with diffing (subscribe new, unsubscribe gone) — REST polling, or
|
||||
Polymarket's `new_market` / `market_resolved` events — using subscribe /
|
||||
unsubscribe on open connections; load-balance all live regular-market symbols
|
||||
across N sockets.
|
||||
**Opening questions (from M2):** what "live" means, and so how many symbols:
|
||||
Gamma lists at least 100,000 markets `active` and not `closed` (2026-11-04,
|
||||
count capped), at least 13,000 of them traded in the last 24 h, against the
|
||||
40 symbols M2 runs. At that size the minute Snapshots alone (~2 KB each)
|
||||
would be ~0.5 TB a day for 200,000 symbols, mostly unchanged books; and
|
||||
memory (~10 KB a book), sockets per connection, resyncs per minute and the
|
||||
book thread's throughput (M2: ~30,000 frames/s on one thread, 43 frames/s
|
||||
of load) all need measuring. Gamma pages past 10,000 markets only through
|
||||
`/markets/keyset` with `after_cursor`.
|
||||
3. **Full coverage: discovery + lanes (base challenge complete)** — *done (2026-11-11)*
|
||||
Every live symbol (~476k tokens, 2026-11-04) spread over lanes, each a
|
||||
connection and a book thread of its own; an orchestrator keeps the tracked
|
||||
set equal to Gamma's live set (a scan at startup and every hour, new and
|
||||
closed markets every minute) through subscribe / unsubscribe on open
|
||||
connections.
|
||||
Sprint: [../roadmap/03. full coverage.md](../roadmap/03.%20full%20coverage.md).
|
||||
*Ends:* every live Polymarket symbol tracked — base challenge satisfied for
|
||||
one exchange.
|
||||
|
||||
4. **Extension 1 — redundant websockets** — *not started*
|
||||
Multiple connections per symbol; choose a trusted source and switch on
|
||||
loss/quiet/delay; emit a fresh snapshot on every source switch.
|
||||
Multiple connections per symbol (several connections in a lane, its book
|
||||
thread choosing among them, the orchestrator coordinating); choose a
|
||||
trusted source and switch on loss/quiet/delay; emit a fresh snapshot on
|
||||
every source switch. Recovery from errors, with the lane budget scaling
|
||||
itself down when connections still get 1013 closes (no rate is safe:
|
||||
[notes.md](notes.md), so a budget set too high must correct itself).
|
||||
**Async re-evaluation checkpoint:** before building this, reassess whether the
|
||||
socket-management complexity (reconnect/timeout/select, redundancy) warrants
|
||||
swapping the thread-only runtime for tokio (see ADR 0001).
|
||||
The lane that holds markets busy together (the 5- and 15-minute crypto
|
||||
markets: closed 16 times in 75 minutes, in bursts a minute's load
|
||||
average doesn't show) is the case to beat.
|
||||
Also: surviving a hard crash without losing the writer's in-memory chunk,
|
||||
and closing the resync gap (the few ms of a symbol's changes the server
|
||||
doesn't send between our unsubscribe and subscribe), both accepted as
|
||||
ceilings in M2.
|
||||
a move's Remove reaching the old lane before the new lane's first
|
||||
Snapshot, the memory growth seen in sprint 03 (~90 MB every 10 minutes
|
||||
over 75 minutes), and closing the subscription gap (the few ms of a
|
||||
symbol's changes the server doesn't send between our unsubscribe and
|
||||
subscribe); the crash chunk and the gap were accepted as ceilings in M2.
|
||||
*Ends:* redundancy demonstrably reduces data loss; a Snapshot on every source
|
||||
switch.
|
||||
|
||||
@@ -106,7 +108,16 @@ abstraction is *extracted* once a second exchange exists, not designed up front.
|
||||
- Observability: metrics, structured logging, dashboards.
|
||||
- Config-driven symbol filtering and per-exchange tuning.
|
||||
- Containerized packaging.
|
||||
- Counting a reconnect's differing books as losses rather than resync gaps
|
||||
(a reconnect marker through the frames channel; M2 labels them gaps).
|
||||
- A `reconcile` that reads the collector's log, to tell resync gaps from
|
||||
- Counting a reconnect's differing books as losses rather than subscription
|
||||
gaps when the symbol was resyncing or not yet verified since its opening
|
||||
book (a reconnect marker through the frames channel; they are labelled
|
||||
gaps).
|
||||
- A `reconcile` that reads the collector's log, to tell subscription gaps from
|
||||
losses in its own report.
|
||||
- Polymarket's `new_market` websocket event as a faster discovery signal (it
|
||||
is broadcast, but lacks `orderMinSize` / `negRisk`, so Gamma is still
|
||||
needed; `market_resolved` was never seen).
|
||||
- Closing lanes that drain.
|
||||
- Fewer Snapshots for unchanged books (the brief asks for every symbol every
|
||||
minute: ~130 MB a minute at full coverage).
|
||||
- Re-reading a market's `orderMinSize` / `negRisk` if they can change.
|
||||
Reference in new issue
Block a user