Archived
Every live market is subscribed on two connections chosen at random, and its book takes each change from whichever connection delivers it first: the book hash says whether a change is a duplicate, the next one, or one to hold until the change before it arrives. A change one connection misses costs nothing, each Delta carries the earliest receive time, and a connection that drops is covered by the other. When a change fits on no connection in time, the book follows one connection unchecked and asks the other for a fresh book. Connections now decode and route events to 16 book threads, independent of them; a move leaves its old connection only once the new one delivers, so it costs no change; a connection closed with 1013 twice within ten minutes is split. reconcile reads the collector's log and fails only on a mismatch no cause line explains. SIGTERM and SIGHUP stop the collector cleanly. Breaking: lanes are gone (connections and book threads are separate), and reconcile takes the log as a second argument.
210 lines
13 KiB
Markdown
210 lines
13 KiB
Markdown
# Challenge notes
|
||
|
||
## Deltas are absolute, not incremental
|
||
A delta carries the **new total quantity** at a price level, not a change to it.
|
||
`quantity: 400` means "this level is now 400," not "+400". Overwrite the level;
|
||
never add. `quantity: 0` means delete the level. (Schema confirms: "absolute
|
||
available quantity".)
|
||
|
||
## A missed message means a wrong book: resync it
|
||
If a message is lost, the in-memory book is silently wrong and cannot be
|
||
patched — throw it away and resync that symbol from a fresh full book. This is the
|
||
"completeness" requirement made mechanical. *How* a loss is detected depends on
|
||
the exchange: a sequence-number gap where one exists (Kalshi, to verify in M6),
|
||
something else where it doesn't — Polymarket has no sequence number, but its
|
||
book hash does the job (below).
|
||
The challenge brief prescribes no mechanism; it only asks for complete data.
|
||
|
||
## Polymarket market channel (observed 2026-11-04)
|
||
|
||
From a 60 s run of the M1 spike on one market, both tokens subscribed, and the
|
||
[market channel docs](https://docs.polymarket.com/market-data/websocket/market-channel.md)
|
||
(read 2026-11-04).
|
||
|
||
- **Endpoint:** `wss://ws-subscriptions-clob.polymarket.com/ws/market`, public,
|
||
no auth. Subscribe: `{"type":"market","assets_ids":["<token id>", ...]}`.
|
||
On an open connection, `{"assets_ids":[...],"operation":"subscribe"}` (or
|
||
`"unsubscribe"`) changes the token set without reconnecting.
|
||
- **`custom_feature_enabled: true`** in the subscribe message adds
|
||
`best_bid_ask`, `new_market` and `market_resolved` events — candidate signals
|
||
for symbol discovery (M3), possibly replacing REST polling.
|
||
- **Symbol = token.** A market (`market`, a `0x…` condition id) has two tokens
|
||
(`asset_id`: Yes, No), each with its own book. The two books are **exact
|
||
mirrors**: Yes had only bids (best 0.999), No only asks (best 0.001), and
|
||
every change arrives as a pair in one message — Yes `BUY p` with No `SELL 1−p`,
|
||
same size.
|
||
- **Frames:** on subscribe, one frame holding a JSON **array** of `book`
|
||
messages (one per token); afterwards, one **object** per frame. A frame's text
|
||
can end with a newline.
|
||
- **Full books also arrive mid-stream.** After a trade the server sent, all with
|
||
one timestamp: the `price_change`, then a full `book` object for each token,
|
||
then a `last_trade_price` — a natural resync point.
|
||
- **`book`** (snapshot): `event_type`, `market`, `asset_id`, `timestamp`,
|
||
`hash`, `bids`, `asks`, `tick_size`, `last_trade_price`. `bids`/`asks` are lists
|
||
of `{price, size}`, bids sorted **ascending** and asks **descending** by
|
||
price, so the best level of each side comes *last*. Its `timestamp` is the
|
||
time of the book's **last change**, not the send time: 19 s old on receipt.
|
||
Books sent mid-stream omit `tick_size` and `last_trade_price`.
|
||
- **`price_change`** (delta): `event_type`, `market`, `timestamp`,
|
||
`price_changes[]`, each entry `{asset_id, price, size, side, hash, best_bid,
|
||
best_ask}`. `side` is `BUY` (bid) or `SELL` (ask); `size` is the absolute
|
||
level size (docs: `"0"` removes the level).
|
||
- **Numbers are decimal strings** (`"0.77"`, `"26350.28"`): parse them exactly
|
||
into fixed point (`1_000_000` = 1), never through a float. Prices and sizes
|
||
are canonical (no trailing zeros: every one in 40 REST books, 2026-11-04);
|
||
`last_trade_price` is not (`"0.080"`).
|
||
- **Timestamps are milliseconds** since the epoch (the schema stores
|
||
microseconds).
|
||
- **No sequence number**, confirmed by the docs: none, and no option enables
|
||
one, for the market channel (the `seq` in Polymarket's docs belongs to
|
||
PolyBolt, a separate price feed). The docs describe no gap detection or
|
||
resync either.
|
||
- **The book hash identifies the full book state, and we can reproduce it**
|
||
(found 2026-11-04; the docs don't describe it). It is the SHA-1, as 40 lowercase
|
||
hex chars, of this compact JSON (no spaces), keys in this order:
|
||
`{"market","asset_id","timestamp","hash":"","bids","asks","min_order_size","tick_size","neg_risk","last_trade_price"}`
|
||
— the REST `/book` response with `hash` emptied. Bids ascending, asks
|
||
descending, prices and sizes as the exchange writes them. On a change, the
|
||
`timestamp` is the change message's. `min_order_size` (`"5"`) and `neg_risk`
|
||
(`true`) are not on the websocket: Gamma's `orderMinSize` / `negRisk` give
|
||
them (matched REST in 40 of 40 markets). Checked live: 14 books and 182
|
||
changes on 6 tokens, every one matched after folding the changes into our
|
||
own book. A change's `hash` therefore also equals that of the next `book`.
|
||
- **One update can span several frames** (seen in a 90 s recording of 40
|
||
tokens): `price_change` frames with the same `timestamp`, one change each
|
||
for the same token, all carrying the hash of the book after the *whole*
|
||
batch. Check a hash once its batch is over.
|
||
- **`last_trade_price` belongs to the market**: both tokens of a market hash
|
||
the same value (20 of 20 markets). A trade changes it, and the hashes use the
|
||
new value up to ~680 ms before the `last_trade_price` event arrives; that
|
||
event names one token, and the hashed value was its `price` (16 cases), its
|
||
complement 1 − `price` (8 cases), or neither (1 case, a trade that probably
|
||
filled at two prices). So a hash check fails after most trades with nothing
|
||
lost; a fresh book (from a resync) carries the right value.
|
||
- **Resync on the open connection:** `unsubscribe` then `subscribe` for a token
|
||
makes the server send its full `book` at once. That book can lag the changes
|
||
already sent by a few ms: in a 2-minute run, 14 of 14 fresh books that
|
||
differed from ours had the hash of one of our own recent changes, i.e. a
|
||
state we had already passed. Its timestamp and `last_trade_price` can be
|
||
newer than its levels: in a 15-minute run, 16 of 27 fresh books that
|
||
differed from ours equalled an earlier state of ours by levels, while their
|
||
hash matched none of our changes (two of them with a timestamp ending in
|
||
`.000`, as if cut to the second). So recognise a lagging book by its levels.
|
||
It can also be *ahead*: between handling the
|
||
`unsubscribe` and the `subscribe`, the server sends no changes for the
|
||
token, so the book holds changes we never receive (6 of 302 resyncs in a
|
||
10-minute run, 1 to 48 ms ahead of our last change; 39 others were behind). A `subscribe` for a token
|
||
already subscribed is answered with the plain text `NO NEW ASSETS` (not
|
||
JSON).
|
||
- **Keepalive (docs):** send the text frame `PING` every 10 s; the server
|
||
answers `PONG`. The spike survived 60 s without it, but it is required.
|
||
- **Server restarts:** the server can close the socket with code `Away`,
|
||
reason `server shutting down` (seen once in about 8 minutes of connections).
|
||
- **`last_trade_price`:** `market`, `asset_id`, `price`, `size`, `side`,
|
||
`fee_rate_bps`, `timestamp`, `transaction_hash` (one per trade, for one
|
||
token).
|
||
- **Not observed:** `market_resolved`, in 5 minutes with
|
||
`custom_feature_enabled` on 10 tokens. The docs now show it only in an
|
||
SDK's wrapped form (`topic` / `payload`, camelCase), not the raw wire
|
||
shape above.
|
||
- **`new_market` is broadcast** (with `custom_feature_enabled`, 2026-11-04):
|
||
every market created, subscribed or not (69 in 5 minutes), with
|
||
`condition_id`, `assets_ids` / `clob_token_ids`, `question`, `active`
|
||
(`false` for markets created ahead), but no `order_min_size` or `neg_risk`,
|
||
which the book hash needs.
|
||
- **Tokens per connection (2026-11-04):** no cap seen: one connection served
|
||
50,000 tokens. But a first `{"type":"market"}` message with more than 500
|
||
tokens gets no opening books (1,000 to 10,000 tried; changes still flow),
|
||
while `subscribe` operations of 500 or 1,000 get theirs. A connection opened
|
||
with `"assets_ids": []` stays up and serves a later `subscribe`.
|
||
- **Slow consumers are closed:** a connection the server can't send to fast
|
||
enough is closed with code 1013, reason `slow consumer: send buffer full`
|
||
(3 of 5 test connections, with a Python reader parsing every frame at up to
|
||
~10,000 frames/s). The collector's reader, which only stamps, got it too:
|
||
six times in 8 minutes on the one connection carrying ~5,000 changes a
|
||
second (NFL quarter spreads, little traded); none on the others, the next
|
||
at ~1,300. No rate is safe (2026-11-10, server, 5 ms from the edge): four
|
||
connections ramped on random live tokens were closed at 0.7, 2.1, 2.3 and
|
||
5.1 MB/s (minute averages of frame bytes), with subscribes of 5,000 or 500
|
||
tokens alike. 5,000 random tokens now carry ~380 frames/s, ~200 KB/s.
|
||
Closes come together: 5 of 20 connections (~1 MB/s each) were closed
|
||
within 45 s of each other, one at 0.46 MB/s.
|
||
- **A subscribe hides a few ms of changes** (smoke run, 2026-11-04): after
|
||
subscribing a market on a busy connection, the first check of its opening
|
||
book often failed, and the fresh book differed from ours at levels none of
|
||
our changes had touched, though the opening book matched its own hash. The
|
||
server never sent those changes: like the resync's unsubscribe/subscribe,
|
||
any subscribe opens this gap (glossary: subscription gap).
|
||
- **`tick_size_change`** (recorded 2026-11-10 on every live token: 62 markets
|
||
in 10 minutes, all `0.01` to `0.001`): one object per token, both tokens of
|
||
the market, each sent twice on the same connection, identical:
|
||
`{"market", "asset_id", "old_tick_size", "new_tick_size", "timestamp",
|
||
"event_type": "tick_size_change"}`.
|
||
- **Load (2026-11-04):** a random 5,000 of the live tokens gave 133 frames/s
|
||
and 266 changes/s; their books had a median of 5 levels (mean 10.9, p90 29,
|
||
max 136), 15% empty. The top 50,000 by volume gave ~10,000 frames/s.
|
||
- **REST from scripts:** `clob.polymarket.com` answers Python's `urllib` with
|
||
403 (curl and `ureq` get 200), most likely Cloudflare filtering by user agent.
|
||
Fetch with curl in throwaway scripts.
|
||
|
||
## Two connections on the same symbols (observed 2026-11-11)
|
||
|
||
- **Same changes, same order**: two connections subscribed to the same 8,000
|
||
tokens for 335 s delivered the same 616,708 changes, none on only one, in
|
||
the same order per token. So the book hash orders changes across
|
||
connections too.
|
||
- **The second delivery's lag**: 0.5 ms at the median, over 10 ms for 6.5% of
|
||
changes, over 100 ms for 1%, 706 ms at worst; up to 82 changes of its token
|
||
behind. Neither connection leads (one was first 34% of the time).
|
||
- **Batches**: 95% of updates are one change. The others come one change per
|
||
frame, all with the hash of the whole batch, within 1 ms (40 ms at worst).
|
||
- **The server repeats some changes** on one connection (104 in that
|
||
recording, the same on both): harmless, quantities being absolute.
|
||
- **Connections carrying the same tokens are closed together**: two ramped on
|
||
the same tokens got their 1013 close 3 s apart. Over 20 minutes on 48,038
|
||
tokens with two subscriptions each, twins in creation order had both
|
||
subscriptions down for 33,345 token-seconds, twins dealt at random 3,787,
|
||
two independent random deals 272.
|
||
- **300 connections** opened at once from one address all stayed up.
|
||
- **A connection whose every subscribed market has resolved is closed** by the
|
||
server, code 1000, reason `all subscribed assets resolved` (seen on a
|
||
connection left with one market, 2026-11-11).
|
||
- **A change's `timestamp` never goes back within a token** on a connection
|
||
(0 of 660,084 changes on each of the two):
|
||
one older than the last applied is a late copy.
|
||
|
||
## Gamma, the market listing (observed 2026-11-04)
|
||
|
||
- **Size:** 238,308 markets `active` and not `closed`; 238,201 of them with
|
||
`enableOrderBook` and `acceptingOrders` (~476k tokens). 32,092 still accept
|
||
orders past their `endDate`, waiting to be resolved: the end date doesn't
|
||
mean closed. ~20k markets are created a day, mostly crypto "Up or Down"
|
||
markets created a day ahead and accepting orders at once.
|
||
- **Pages of 100**, whatever `limit` asks. `/markets` refuses offsets past
|
||
2,000 ("use /markets/keyset"); `/markets/keyset` pages with `after_cursor`
|
||
(the `next_cursor` of the last page) and has no such limit. A full keyset
|
||
scan of the live markets took 6 min 41 s.
|
||
- **Sorting works, date filters don't:** `order=createdAt&ascending=false`
|
||
(newest first) and `closed=true&order=closedTime&ascending=false` sort both
|
||
endpoints, and keyset pages keep the order across cursors. `created_at_min`
|
||
and `closed_time_min` are ignored. 2–20 markets were created and 10–18
|
||
closed a minute; more than 420 a minute changed `updatedAt`.
|
||
- **`conditionId`** is the websocket's `market`; `clobTokenIds` (a JSON array
|
||
inside a string) are its symbols.
|
||
- **Null hash inputs:** a null `orderMinSize` (31 live markets) hashes as
|
||
`"0"`, what REST's `/book` reports; a null `negRisk` (230) as `false`.
|
||
- **Scan speed:** the full keyset scan took 6 min 41 s from a Python script
|
||
spawning curl per page, ~2 minutes from the collector (`ureq`, one agent),
|
||
under a minute from the server (2026-11-10).
|
||
- **Indexing delay (2026-11-10):** new markets appear in the newest-first
|
||
listing 219 to 296 s after their `createdAt` (8 markets, polled every
|
||
20 s): a check reaching back 5 minutes misses some. 195,130 markets were
|
||
live that evening.
|
||
|
||
## Clock accuracy bounds time accuracy
|
||
`received_at` comes from the local clock, so it carries the clock's error. On
|
||
2026-11-04 the dev machine ran ~30 ms slow (`sntp`: +30.6 ms ± 13 ms), which
|
||
made raw `received_at − exchange timestamp` read −14 to +41 ms; corrected, the
|
||
real lag was ~+16 to +71 ms. Latency figures need a synced clock, or the offset
|
||
measured and reported alongside them.
|