Archived
feat(polymarket): track every live market over load-balanced lanes
An orchestrator keeps the tracked set equal to Gamma's live markets (a scan at startup and every hour, new and closed markets every minute, reaching back 15 minutes for Gamma's indexing delay) and spreads them over lanes, each a connection and a book thread, all writing through the one writer. Lanes report each market's load in bytes per second; the orchestrator puts new markets on the least-loaded lane and moves the busiest markets off lanes over a 300 KB/s budget, busiest lane first. Differing books our own subscribe explains are subscription gaps, not losses, and every first book after a move is logged, so reconcile's mismatches are all accounted for in the log. A 75-minute live run tracked ~195k markets (~390k symbols) in 89 lanes.
This commit is contained in:
18 files changed
+2826
-320
No files matched your search
+71
-5
@@ -100,15 +100,81 @@ From a 60 s run of the M1 spike on one market, both tokens subscribed, and the
|
||||
answers `PONG`. The spike survived 60 s without it, but it is required.
|
||||
- **Server restarts:** the server can close the socket with code `Away`,
|
||||
reason `server shutting down` (seen once in about 8 minutes of connections).
|
||||
- **`last_trade_price`:** `asset_id`, `price`, `size`, `side`, `fee_rate_bps`,
|
||||
`timestamp`, `transaction_hash` (one per trade, for one token).
|
||||
- **Not observed:** `tick_size_change` and the custom-feature events. The docs
|
||||
now show these only in an SDK's wrapped form (`topic` / `payload`,
|
||||
camelCase), not the raw wire shape above.
|
||||
- **`last_trade_price`:** `market`, `asset_id`, `price`, `size`, `side`,
|
||||
`fee_rate_bps`, `timestamp`, `transaction_hash` (one per trade, for one
|
||||
token).
|
||||
- **Not observed:** `market_resolved`, in 5 minutes with
|
||||
`custom_feature_enabled` on 10 tokens. The docs now show it only in an
|
||||
SDK's wrapped form (`topic` / `payload`, camelCase), not the raw wire
|
||||
shape above.
|
||||
- **`new_market` is broadcast** (with `custom_feature_enabled`, 2026-11-04):
|
||||
every market created, subscribed or not (69 in 5 minutes), with
|
||||
`condition_id`, `assets_ids` / `clob_token_ids`, `question`, `active`
|
||||
(`false` for markets created ahead), but no `order_min_size` or `neg_risk`,
|
||||
which the book hash needs.
|
||||
- **Tokens per connection (2026-11-04):** no cap seen: one connection served
|
||||
50,000 tokens. But a first `{"type":"market"}` message with more than 500
|
||||
tokens gets no opening books (1,000 to 10,000 tried; changes still flow),
|
||||
while `subscribe` operations of 500 or 1,000 get theirs. A connection opened
|
||||
with `"assets_ids": []` stays up and serves a later `subscribe`.
|
||||
- **Slow consumers are closed:** a connection the server can't send to fast
|
||||
enough is closed with code 1013, reason `slow consumer: send buffer full`
|
||||
(3 of 5 test connections, with a Python reader parsing every frame at up to
|
||||
~10,000 frames/s). The collector's reader, which only stamps, got it too:
|
||||
six times in 8 minutes on the one connection carrying ~5,000 changes a
|
||||
second (NFL quarter spreads, little traded); none on the others, the next
|
||||
at ~1,300. No rate is safe (2026-11-10, server, 5 ms from the edge): four
|
||||
connections ramped on random live tokens were closed at 0.7, 2.1, 2.3 and
|
||||
5.1 MB/s (minute averages of frame bytes), with subscribes of 5,000 or 500
|
||||
tokens alike. 5,000 random tokens now carry ~380 frames/s, ~200 KB/s.
|
||||
Closes come together: 5 of 20 connections (~1 MB/s each) were closed
|
||||
within 45 s of each other, one at 0.46 MB/s.
|
||||
- **A subscribe hides a few ms of changes** (smoke run, 2026-11-04): after
|
||||
subscribing a market on a busy connection, the first check of its opening
|
||||
book often failed, and the fresh book differed from ours at levels none of
|
||||
our changes had touched, though the opening book matched its own hash. The
|
||||
server never sent those changes: like the resync's unsubscribe/subscribe,
|
||||
any subscribe opens this gap (glossary: subscription gap).
|
||||
- **`tick_size_change`** (recorded 2026-11-10 on every live token: 62 markets
|
||||
in 10 minutes, all `0.01` to `0.001`): one object per token, both tokens of
|
||||
the market, each sent twice on the same connection, identical:
|
||||
`{"market", "asset_id", "old_tick_size", "new_tick_size", "timestamp",
|
||||
"event_type": "tick_size_change"}`.
|
||||
- **Load (2026-11-04):** a random 5,000 of the live tokens gave 133 frames/s
|
||||
and 266 changes/s; their books had a median of 5 levels (mean 10.9, p90 29,
|
||||
max 136), 15% empty. The top 50,000 by volume gave ~10,000 frames/s.
|
||||
- **REST from scripts:** `clob.polymarket.com` answers Python's `urllib` with
|
||||
403 (curl and `ureq` get 200), most likely Cloudflare filtering by user agent.
|
||||
Fetch with curl in throwaway scripts.
|
||||
|
||||
## Gamma, the market listing (observed 2026-11-04)
|
||||
|
||||
- **Size:** 238,308 markets `active` and not `closed`; 238,201 of them with
|
||||
`enableOrderBook` and `acceptingOrders` (~476k tokens). 32,092 still accept
|
||||
orders past their `endDate`, waiting to be resolved: the end date doesn't
|
||||
mean closed. ~20k markets are created a day, mostly crypto "Up or Down"
|
||||
markets created a day ahead and accepting orders at once.
|
||||
- **Pages of 100**, whatever `limit` asks. `/markets` refuses offsets past
|
||||
2,000 ("use /markets/keyset"); `/markets/keyset` pages with `after_cursor`
|
||||
(the `next_cursor` of the last page) and has no such limit. A full keyset
|
||||
scan of the live markets took 6 min 41 s.
|
||||
- **Sorting works, date filters don't:** `order=createdAt&ascending=false`
|
||||
(newest first) and `closed=true&order=closedTime&ascending=false` sort both
|
||||
endpoints, and keyset pages keep the order across cursors. `created_at_min`
|
||||
and `closed_time_min` are ignored. 2–20 markets were created and 10–18
|
||||
closed a minute; more than 420 a minute changed `updatedAt`.
|
||||
- **`conditionId`** is the websocket's `market`; `clobTokenIds` (a JSON array
|
||||
inside a string) are its symbols.
|
||||
- **Null hash inputs:** a null `orderMinSize` (31 live markets) hashes as
|
||||
`"0"`, what REST's `/book` reports; a null `negRisk` (230) as `false`.
|
||||
- **Scan speed:** the full keyset scan took 6 min 41 s from a Python script
|
||||
spawning curl per page, ~2 minutes from the collector (`ureq`, one agent),
|
||||
under a minute from the server (2026-11-10).
|
||||
- **Indexing delay (2026-11-10):** new markets appear in the newest-first
|
||||
listing 219 to 296 s after their `createdAt` (8 markets, polled every
|
||||
20 s): a check reaching back 5 minutes misses some. 195,130 markets were
|
||||
live that evening.
|
||||
|
||||
## Clock accuracy bounds time accuracy
|
||||
`received_at` comes from the local clock, so it carries the clock's error. On
|
||||
2026-11-04 the dev machine ran ~30 ms slow (`sntp`: +30.6 ms ± 13 ms), which
|
||||
|
||||
Reference in new issue
Block a user