Archived
feat(polymarket): hash-verified order books for a subset of symbols
Collects the order books of Polymarket's 20 busiest markets (40 symbols) into the template's SBE format: a Delta per level change, plus Snapshots on subscribe, every minute, and whenever a fresh book corrects ours. Every change is checked against Polymarket's own book hash; a failed check resyncs the symbol on the open socket, and the log counts losses apart from the few ms our own resync hides. Runs until SIGINT, which closes the socket and flushes the output; reconnects when the server drops it. A `reconcile` binary checks an output file against itself.
This commit is contained in:
22 files changed
+6423
-84
No files matched your search
+57
-16
@@ -6,12 +6,13 @@ A delta carries the **new total quantity** at a price level, not a change to it.
|
||||
never add. `quantity: 0` means delete the level. (Schema confirms: "absolute
|
||||
available quantity".)
|
||||
|
||||
## A missed message means a wrong book: rebuild it
|
||||
## A missed message means a wrong book: resync it
|
||||
If a message is lost, the in-memory book is silently wrong and cannot be
|
||||
patched — throw it away and re-snapshot that symbol. This is the
|
||||
patched — throw it away and resync that symbol from a fresh full book. This is the
|
||||
"completeness" requirement made mechanical. *How* a loss is detected depends on
|
||||
the exchange: a sequence-number gap where one exists (Kalshi, to verify in M6),
|
||||
something else where it doesn't — Polymarket has no sequence number (below).
|
||||
something else where it doesn't — Polymarket has no sequence number, but its
|
||||
book hash does the job (below).
|
||||
The challenge brief prescribes no mechanism; it only asks for complete data.
|
||||
|
||||
## Polymarket market channel (observed 2026-11-04)
|
||||
@@ -40,33 +41,73 @@ From a 60 s run of the M1 spike on one market, both tokens subscribed, and the
|
||||
then a `last_trade_price` — a natural resync point.
|
||||
- **`book`** (snapshot): `event_type`, `market`, `asset_id`, `timestamp`,
|
||||
`hash`, `bids`, `asks`, `tick_size`, `last_trade_price`. `bids`/`asks` are lists
|
||||
of `{price, size}` sorted **ascending by price**, so the best bid is the
|
||||
*last* bid. Its `timestamp` is the time of the book's **last change**, not the
|
||||
send time: 19 s old on receipt.
|
||||
of `{price, size}`, bids sorted **ascending** and asks **descending** by
|
||||
price, so the best level of each side comes *last*. Its `timestamp` is the
|
||||
time of the book's **last change**, not the send time: 19 s old on receipt.
|
||||
Books sent mid-stream omit `tick_size` and `last_trade_price`.
|
||||
- **`price_change`** (delta): `event_type`, `market`, `timestamp`,
|
||||
`price_changes[]`, each entry `{asset_id, price, size, side, hash, best_bid,
|
||||
best_ask}`. `side` is `BUY` (bid) or `SELL` (ask); `size` is the absolute
|
||||
level size (docs: `"0"` removes the level).
|
||||
- **Numbers are decimal strings** (`"0.77"`, `"26350.28"`): parse them exactly
|
||||
into fixed point (`1_000_000` = 1), never through a float.
|
||||
into fixed point (`1_000_000` = 1), never through a float. Prices and sizes
|
||||
are canonical (no trailing zeros: every one in 40 REST books, 2026-11-04);
|
||||
`last_trade_price` is not (`"0.080"`).
|
||||
- **Timestamps are milliseconds** since the epoch (the schema stores
|
||||
microseconds).
|
||||
- **No sequence number**, confirmed by the docs: none, and no option enables
|
||||
one, for the market channel (the `seq` in Polymarket's docs belongs to
|
||||
PolyBolt, a separate price feed). The docs describe no gap detection or
|
||||
resync either. Candidates for loss detection, to settle in M2:
|
||||
- the `hash` (40 hex chars, SHA-1 length) on every book and every change. A
|
||||
change's `hash` equals that of the `book` sent right after it for the same
|
||||
token, so it identifies the **full book state after the change**: a full
|
||||
check if we can reproduce it from our own book. The docs don't say how it is
|
||||
computed (M2's opening question);
|
||||
- `best_bid`/`best_ask` on every change — a cheap top-of-book check after
|
||||
applying it.
|
||||
resync either.
|
||||
- **The book hash identifies the full book state, and we can reproduce it**
|
||||
(found 2026-11-04; the docs don't describe it). It is the SHA-1, as 40 lowercase
|
||||
hex chars, of this compact JSON (no spaces), keys in this order:
|
||||
`{"market","asset_id","timestamp","hash":"","bids","asks","min_order_size","tick_size","neg_risk","last_trade_price"}`
|
||||
— the REST `/book` response with `hash` emptied. Bids ascending, asks
|
||||
descending, prices and sizes as the exchange writes them. On a change, the
|
||||
`timestamp` is the change message's. `min_order_size` (`"5"`) and `neg_risk`
|
||||
(`true`) are not on the websocket: Gamma's `orderMinSize` / `negRisk` give
|
||||
them (matched REST in 40 of 40 markets). Checked live: 14 books and 182
|
||||
changes on 6 tokens, every one matched after folding the changes into our
|
||||
own book. A change's `hash` therefore also equals that of the next `book`.
|
||||
- **One update can span several frames** (seen in a 90 s recording of 40
|
||||
tokens): `price_change` frames with the same `timestamp`, one change each
|
||||
for the same token, all carrying the hash of the book after the *whole*
|
||||
batch. Check a hash once its batch is over.
|
||||
- **`last_trade_price` belongs to the market**: both tokens of a market hash
|
||||
the same value (20 of 20 markets). A trade changes it, and the hashes use the
|
||||
new value up to ~680 ms before the `last_trade_price` event arrives; that
|
||||
event names one token, and the hashed value was its `price` (16 cases), its
|
||||
complement 1 − `price` (8 cases), or neither (1 case, a trade that probably
|
||||
filled at two prices). So a hash check fails after most trades with nothing
|
||||
lost; a fresh book (from a resync) carries the right value.
|
||||
- **Resync on the open connection:** `unsubscribe` then `subscribe` for a token
|
||||
makes the server send its full `book` at once. That book can lag the changes
|
||||
already sent by a few ms: in a 2-minute run, 14 of 14 fresh books that
|
||||
differed from ours had the hash of one of our own recent changes, i.e. a
|
||||
state we had already passed. Its timestamp and `last_trade_price` can be
|
||||
newer than its levels: in a 15-minute run, 16 of 27 fresh books that
|
||||
differed from ours equalled an earlier state of ours by levels, while their
|
||||
hash matched none of our changes (two of them with a timestamp ending in
|
||||
`.000`, as if cut to the second). So recognise a lagging book by its levels.
|
||||
It can also be *ahead*: between handling the
|
||||
`unsubscribe` and the `subscribe`, the server sends no changes for the
|
||||
token, so the book holds changes we never receive (6 of 302 resyncs in a
|
||||
10-minute run, 1 to 48 ms ahead of our last change; 39 others were behind). A `subscribe` for a token
|
||||
already subscribed is answered with the plain text `NO NEW ASSETS` (not
|
||||
JSON).
|
||||
- **Keepalive (docs):** send the text frame `PING` every 10 s; the server
|
||||
answers `PONG`. The spike survived 60 s without it, but it is required.
|
||||
- **Server restarts:** the server can close the socket with code `Away`,
|
||||
reason `server shutting down` (seen once in about 8 minutes of connections).
|
||||
- **`last_trade_price`:** `asset_id`, `price`, `size`, `side`, `fee_rate_bps`,
|
||||
`timestamp`, `transaction_hash` (one per trade, for one token).
|
||||
- **Not observed:** `tick_size_change` and the custom-feature events.
|
||||
- **Not observed:** `tick_size_change` and the custom-feature events. The docs
|
||||
now show these only in an SDK's wrapped form (`topic` / `payload`,
|
||||
camelCase), not the raw wire shape above.
|
||||
- **REST from scripts:** `clob.polymarket.com` answers Python's `urllib` with
|
||||
403 (curl and `ureq` get 200), most likely Cloudflare filtering by user agent.
|
||||
Fetch with curl in throwaway scripts.
|
||||
|
||||
## Clock accuracy bounds time accuracy
|
||||
`received_at` comes from the local clock, so it carries the clock's error. On
|
||||
|
||||
Reference in new issue
Block a user