This repository has been archived on 2026-11-16. You can view files and clone it. You cannot open issues or pull requests or push a commit.
Files
orderbook-collector/docs/notes.md
T
pascal 480ee3ad2d feat(polymarket): subscription monitoring and connection recycling
The exchange can fail without telling us, and first one wins hides a
subscription gone quiet on one of a symbol's two connections.

- A connection that receives nothing for 30 s, not even the keepalive's
  answer, reconnects.
- Each book thread counts a symbol's changes per connection every minute:
  a subscription with none against three or more on the other is
  resubscribed, and moved to another connection if it stays quiet.
- The orchestrator recycles one connection at a time, each about hourly,
  and never while another connection is down or just back.
- A reconnect's opening book of a symbol unchanged for 10 s is judged like
  a resync's answer: one newer than ours that differs shows a symbol quiet
  on both connections, written as a Snapshot and logged as a loss.
2026-11-11 17:42:27 +00:00

14 KiB
Raw Blame History

Challenge notes

Deltas are absolute, not incremental

A delta carries the new total quantity at a price level, not a change to it. quantity: 400 means "this level is now 400," not "+400". Overwrite the level; never add. quantity: 0 means delete the level. (Schema confirms: "absolute available quantity".)

A missed message means a wrong book: resync it

If a message is lost, the in-memory book is silently wrong and cannot be patched — throw it away and resync that symbol from a fresh full book. This is the "completeness" requirement made mechanical. How a loss is detected depends on the exchange: a sequence-number gap where one exists (Kalshi, to verify in M6), something else where it doesn't — Polymarket has no sequence number, but its book hash does the job (below). The challenge brief prescribes no mechanism; it only asks for complete data.

Polymarket market channel (observed 2026-11-04)

From a 60 s run of the M1 spike on one market, both tokens subscribed, and the market channel docs (read 2026-11-04).

  • Endpoint: wss://ws-subscriptions-clob.polymarket.com/ws/market, public, no auth. Subscribe: {"type":"market","assets_ids":["<token id>", ...]}. On an open connection, {"assets_ids":[...],"operation":"subscribe"} (or "unsubscribe") changes the token set without reconnecting.
  • custom_feature_enabled: true in the subscribe message adds best_bid_ask, new_market and market_resolved events — candidate signals for symbol discovery (M3), possibly replacing REST polling.
  • Symbol = token. A market (market, a 0x… condition id) has two tokens (asset_id: Yes, No), each with its own book. The two books are exact mirrors: Yes had only bids (best 0.999), No only asks (best 0.001), and every change arrives as a pair in one message — Yes BUY p with No SELL 1−p, same size.
  • Frames: on subscribe, one frame holding a JSON array of book messages (one per token); afterwards, one object per frame. A frame's text can end with a newline.
  • Full books also arrive mid-stream. After a trade the server sent, all with one timestamp: the price_change, then a full book object for each token, then a last_trade_price — a natural resync point.
  • book (snapshot): event_type, market, asset_id, timestamp, hash, bids, asks, tick_size, last_trade_price. bids/asks are lists of {price, size}, bids sorted ascending and asks descending by price, so the best level of each side comes last. Its timestamp is the time of the book's last change, not the send time: 19 s old on receipt. Books sent mid-stream omit tick_size and last_trade_price.
  • price_change (delta): event_type, market, timestamp, price_changes[], each entry {asset_id, price, size, side, hash, best_bid, best_ask}. side is BUY (bid) or SELL (ask); size is the absolute level size (docs: "0" removes the level).
  • Numbers are decimal strings ("0.77", "26350.28"): parse them exactly into fixed point (1_000_000 = 1), never through a float. Prices and sizes are canonical (no trailing zeros: every one in 40 REST books, 2026-11-04); last_trade_price is not ("0.080").
  • Timestamps are milliseconds since the epoch (the schema stores microseconds).
  • No sequence number, confirmed by the docs: none, and no option enables one, for the market channel (the seq in Polymarket's docs belongs to PolyBolt, a separate price feed). The docs describe no gap detection or resync either.
  • The book hash identifies the full book state, and we can reproduce it (found 2026-11-04; the docs don't describe it). It is the SHA-1, as 40 lowercase hex chars, of this compact JSON (no spaces), keys in this order: {"market","asset_id","timestamp","hash":"","bids","asks","min_order_size","tick_size","neg_risk","last_trade_price"} — the REST /book response with hash emptied. Bids ascending, asks descending, prices and sizes as the exchange writes them. On a change, the timestamp is the change message's. min_order_size ("5") and neg_risk (true) are not on the websocket: Gamma's orderMinSize / negRisk give them (matched REST in 40 of 40 markets). Checked live: 14 books and 182 changes on 6 tokens, every one matched after folding the changes into our own book. A change's hash therefore also equals that of the next book.
  • One update can span several frames (seen in a 90 s recording of 40 tokens): price_change frames with the same timestamp, one change each for the same token, all carrying the hash of the book after the whole batch. Check a hash once its batch is over.
  • last_trade_price belongs to the market: both tokens of a market hash the same value (20 of 20 markets). A trade changes it, and the hashes use the new value up to ~680 ms before the last_trade_price event arrives; that event names one token, and the hashed value was its price (16 cases), its complement 1 − price (8 cases), or neither (1 case, a trade that probably filled at two prices). So a hash check fails after most trades with nothing lost; a fresh book (from a resync) carries the right value.
  • Resync on the open connection: unsubscribe then subscribe for a token makes the server send its full book at once. That book can lag the changes already sent by a few ms: in a 2-minute run, 14 of 14 fresh books that differed from ours had the hash of one of our own recent changes, i.e. a state we had already passed. Its timestamp and last_trade_price can be newer than its levels: in a 15-minute run, 16 of 27 fresh books that differed from ours equalled an earlier state of ours by levels, while their hash matched none of our changes (two of them with a timestamp ending in .000, as if cut to the second). So recognise a lagging book by its levels. It can also be ahead: between handling the unsubscribe and the subscribe, the server sends no changes for the token, so the book holds changes we never receive (6 of 302 resyncs in a 10-minute run, 1 to 48 ms ahead of our last change; 39 others were behind). A subscribe for a token already subscribed is answered with the plain text NO NEW ASSETS (not JSON).
  • Keepalive (docs): send the text frame PING every 10 s; the server answers PONG. The spike survived 60 s without it, but it is required.
  • Server restarts: the server can close the socket with code Away, reason server shutting down (seen once in about 8 minutes of connections).
  • last_trade_price: market, asset_id, price, size, side, fee_rate_bps, timestamp, transaction_hash (one per trade, for one token).
  • Not observed: market_resolved, in 5 minutes with custom_feature_enabled on 10 tokens. The docs now show it only in an SDK's wrapped form (topic / payload, camelCase), not the raw wire shape above.
  • new_market is broadcast (with custom_feature_enabled, 2026-11-04): every market created, subscribed or not (69 in 5 minutes), with condition_id, assets_ids / clob_token_ids, question, active (false for markets created ahead), but no order_min_size or neg_risk, which the book hash needs.
  • Tokens per connection (2026-11-04): no cap seen: one connection served 50,000 tokens. But a first {"type":"market"} message with more than 500 tokens gets no opening books (1,000 to 10,000 tried; changes still flow), while subscribe operations of 500 or 1,000 get theirs. A connection opened with "assets_ids": [] stays up and serves a later subscribe.
  • Slow consumers are closed: a connection the server can't send to fast enough is closed with code 1013, reason slow consumer: send buffer full (3 of 5 test connections, with a Python reader parsing every frame at up to ~10,000 frames/s). The collector's reader, which only stamps, got it too: six times in 8 minutes on the one connection carrying ~5,000 changes a second (NFL quarter spreads, little traded); none on the others, the next at ~1,300. No rate is safe (2026-11-10, server, 5 ms from the edge): four connections ramped on random live tokens were closed at 0.7, 2.1, 2.3 and 5.1 MB/s (minute averages of frame bytes), with subscribes of 5,000 or 500 tokens alike. 5,000 random tokens now carry ~380 frames/s, ~200 KB/s. Closes come together: 5 of 20 connections (~1 MB/s each) were closed within 45 s of each other, one at 0.46 MB/s.
  • A subscribe hides a few ms of changes (smoke run, 2026-11-04): after subscribing a market on a busy connection, the first check of its opening book often failed, and the fresh book differed from ours at levels none of our changes had touched, though the opening book matched its own hash. The server never sent those changes: like the resync's unsubscribe/subscribe, any subscribe opens this gap (glossary: subscription gap).
  • tick_size_change (recorded 2026-11-10 on every live token: 62 markets in 10 minutes, all 0.01 to 0.001): one object per token, both tokens of the market, each sent twice on the same connection, identical: {"market", "asset_id", "old_tick_size", "new_tick_size", "timestamp", "event_type": "tick_size_change"}.
  • Load (2026-11-04): a random 5,000 of the live tokens gave 133 frames/s and 266 changes/s; their books had a median of 5 levels (mean 10.9, p90 29, max 136), 15% empty. The top 50,000 by volume gave ~10,000 frames/s.
  • REST from scripts: clob.polymarket.com answers Python's urllib with 403 (curl and ureq get 200), most likely Cloudflare filtering by user agent. Fetch with curl in throwaway scripts.

Two connections on the same symbols (observed 2026-11-11)

  • Same changes, same order: two connections subscribed to the same 8,000 tokens for 335 s delivered the same 616,708 changes, none on only one, in the same order per token. So the book hash orders changes across connections too.
  • The second delivery's lag: 0.5 ms at the median, over 10 ms for 6.5% of changes, over 100 ms for 1%, 706 ms at worst; up to 82 changes of its token behind. Neither connection leads (one was first 34% of the time).
  • Batches: 95% of updates are one change. The others come one change per frame, all with the hash of the whole batch, within 1 ms (40 ms at worst).
  • The server repeats some changes on one connection (104 in that recording, the same on both): harmless, quantities being absolute.
  • Connections carrying the same tokens are closed together: two ramped on the same tokens got their 1013 close 3 s apart. Over 20 minutes on 48,038 tokens with two subscriptions each, twins in creation order had both subscriptions down for 33,345 token-seconds, twins dealt at random 3,787, two independent random deals 272.
  • 300 connections opened at once from one address all stayed up.
  • A connection whose every subscribed market has resolved is closed by the server, code 1000, reason all subscribed assets resolved (seen on a connection left with one market, 2026-11-11).
  • A change's timestamp never goes back within a token on a connection (0 of 660,084 changes on each of the two): one older than the last applied is a late copy.

Quiet subscriptions and reconnects (observed 2026-11-11)

From a 25-minute live run on every live market, two connections each, one connection recycled every 60 s (sprint 05).

  • Quiet subscriptions are rare: in 21 minutes, one market once had 3 or more changes on one connection and none on the other; none had 10. Both tokens of a market go one-sided together.
  • A reconnect's opening book can lag by minutes: for symbols unchanged for 10 s or more, 514 of 19,379 differed from ours, 512 of them equal to one of our earlier states (1 to 4 levels behind, some over 10 minutes), none with a newer timestamp. Books of symbols never changed all matched (42,096). Under 10 s, some ran ahead of our stream.
  • With the monitoring on (30 minutes, 107 recycles): 26 subscriptions resubscribed for being quiet, none quiet again the next minute; of 335,488 reconnect books of unchanged symbols, 7,240 fresh books in all were behind ours; no connection silent for 30 s.
  • A reconnect takes ~1.2 s from close to up (1 s of it our wait); the opening books of ~4,500 tokens come 0.3 s later on most connections, 20 s on one; 2 of 4,490 never came on another.

Gamma, the market listing (observed 2026-11-04)

  • Size: 238,308 markets active and not closed; 238,201 of them with enableOrderBook and acceptingOrders (~476k tokens). 32,092 still accept orders past their endDate, waiting to be resolved: the end date doesn't mean closed. ~20k markets are created a day, mostly crypto "Up or Down" markets created a day ahead and accepting orders at once.
  • Pages of 100, whatever limit asks. /markets refuses offsets past 2,000 ("use /markets/keyset"); /markets/keyset pages with after_cursor (the next_cursor of the last page) and has no such limit. A full keyset scan of the live markets took 6 min 41 s.
  • Sorting works, date filters don't: order=createdAt&ascending=false (newest first) and closed=true&order=closedTime&ascending=false sort both endpoints, and keyset pages keep the order across cursors. created_at_min and closed_time_min are ignored. 2–20 markets were created and 10–18 closed a minute; more than 420 a minute changed updatedAt.
  • conditionId is the websocket's market; clobTokenIds (a JSON array inside a string) are its symbols.
  • Null hash inputs: a null orderMinSize (31 live markets) hashes as "0", what REST's /book reports; a null negRisk (230) as false.
  • Scan speed: the full keyset scan took 6 min 41 s from a Python script spawning curl per page, ~2 minutes from the collector (ureq, one agent), under a minute from the server (2026-11-10).
  • Indexing delay (2026-11-10): new markets appear in the newest-first listing 219 to 296 s after their createdAt (8 markets, polled every 20 s): a check reaching back 5 minutes misses some. 195,130 markets were live that evening.

Clock accuracy bounds time accuracy

received_at comes from the local clock, so it carries the clock's error. On 2026-11-04 the dev machine ran ~30 ms slow (sntp: +30.6 ms ± 13 ms), which made raw received_at − exchange timestamp read −14 to +41 ms; corrected, the real lag was ~+16 to +71 ms. Latency figures need a synced clock, or the offset measured and reported alongside them.