Archived
feat(polymarket): hash-verified order books for a subset of symbols
Collects the order books of Polymarket's 20 busiest markets (40 symbols) into the template's SBE format: a Delta per level change, plus Snapshots on subscribe, every minute, and whenever a fresh book corrects ours. Every change is checked against Polymarket's own book hash; a failed check resyncs the symbol on the open socket, and the log counts losses apart from the few ms our own resync hides. Runs until SIGINT, which closes the socket and flushes the output; reconnects when the server drops it. A `reconcile` binary checks an output file against itself.
This commit is contained in:
22 files changed
+6423
-84
No files matched your search
+1
-1
@@ -25,7 +25,7 @@ The two claims split apart:
|
||||
|
||||
- **Completeness is cleanly measurable on recorded data.** Treat our output as
|
||||
ground truth, make a lossy copy (drop a % of messages, skip some
|
||||
re-snapshots), run the same strategy on both, compare P&L. The gap = cost of
|
||||
resyncs), run the same strategy on both, compare P&L. The gap = cost of
|
||||
dropped data. No live trading needed.
|
||||
- **Latency is NOT measurable from recorded data.** Collection latency is baked
|
||||
into a recording; a backtest can't show "we'd have been a few ms faster." A
|
||||
|
||||
+57
-16
@@ -6,12 +6,13 @@ A delta carries the **new total quantity** at a price level, not a change to it.
|
||||
never add. `quantity: 0` means delete the level. (Schema confirms: "absolute
|
||||
available quantity".)
|
||||
|
||||
## A missed message means a wrong book: rebuild it
|
||||
## A missed message means a wrong book: resync it
|
||||
If a message is lost, the in-memory book is silently wrong and cannot be
|
||||
patched — throw it away and re-snapshot that symbol. This is the
|
||||
patched — throw it away and resync that symbol from a fresh full book. This is the
|
||||
"completeness" requirement made mechanical. *How* a loss is detected depends on
|
||||
the exchange: a sequence-number gap where one exists (Kalshi, to verify in M6),
|
||||
something else where it doesn't — Polymarket has no sequence number (below).
|
||||
something else where it doesn't — Polymarket has no sequence number, but its
|
||||
book hash does the job (below).
|
||||
The challenge brief prescribes no mechanism; it only asks for complete data.
|
||||
|
||||
## Polymarket market channel (observed 2026-11-04)
|
||||
@@ -40,33 +41,73 @@ From a 60 s run of the M1 spike on one market, both tokens subscribed, and the
|
||||
then a `last_trade_price` — a natural resync point.
|
||||
- **`book`** (snapshot): `event_type`, `market`, `asset_id`, `timestamp`,
|
||||
`hash`, `bids`, `asks`, `tick_size`, `last_trade_price`. `bids`/`asks` are lists
|
||||
of `{price, size}` sorted **ascending by price**, so the best bid is the
|
||||
*last* bid. Its `timestamp` is the time of the book's **last change**, not the
|
||||
send time: 19 s old on receipt.
|
||||
of `{price, size}`, bids sorted **ascending** and asks **descending** by
|
||||
price, so the best level of each side comes *last*. Its `timestamp` is the
|
||||
time of the book's **last change**, not the send time: 19 s old on receipt.
|
||||
Books sent mid-stream omit `tick_size` and `last_trade_price`.
|
||||
- **`price_change`** (delta): `event_type`, `market`, `timestamp`,
|
||||
`price_changes[]`, each entry `{asset_id, price, size, side, hash, best_bid,
|
||||
best_ask}`. `side` is `BUY` (bid) or `SELL` (ask); `size` is the absolute
|
||||
level size (docs: `"0"` removes the level).
|
||||
- **Numbers are decimal strings** (`"0.77"`, `"26350.28"`): parse them exactly
|
||||
into fixed point (`1_000_000` = 1), never through a float.
|
||||
into fixed point (`1_000_000` = 1), never through a float. Prices and sizes
|
||||
are canonical (no trailing zeros: every one in 40 REST books, 2026-11-04);
|
||||
`last_trade_price` is not (`"0.080"`).
|
||||
- **Timestamps are milliseconds** since the epoch (the schema stores
|
||||
microseconds).
|
||||
- **No sequence number**, confirmed by the docs: none, and no option enables
|
||||
one, for the market channel (the `seq` in Polymarket's docs belongs to
|
||||
PolyBolt, a separate price feed). The docs describe no gap detection or
|
||||
resync either. Candidates for loss detection, to settle in M2:
|
||||
- the `hash` (40 hex chars, SHA-1 length) on every book and every change. A
|
||||
change's `hash` equals that of the `book` sent right after it for the same
|
||||
token, so it identifies the **full book state after the change**: a full
|
||||
check if we can reproduce it from our own book. The docs don't say how it is
|
||||
computed (M2's opening question);
|
||||
- `best_bid`/`best_ask` on every change — a cheap top-of-book check after
|
||||
applying it.
|
||||
resync either.
|
||||
- **The book hash identifies the full book state, and we can reproduce it**
|
||||
(found 2026-11-04; the docs don't describe it). It is the SHA-1, as 40 lowercase
|
||||
hex chars, of this compact JSON (no spaces), keys in this order:
|
||||
`{"market","asset_id","timestamp","hash":"","bids","asks","min_order_size","tick_size","neg_risk","last_trade_price"}`
|
||||
— the REST `/book` response with `hash` emptied. Bids ascending, asks
|
||||
descending, prices and sizes as the exchange writes them. On a change, the
|
||||
`timestamp` is the change message's. `min_order_size` (`"5"`) and `neg_risk`
|
||||
(`true`) are not on the websocket: Gamma's `orderMinSize` / `negRisk` give
|
||||
them (matched REST in 40 of 40 markets). Checked live: 14 books and 182
|
||||
changes on 6 tokens, every one matched after folding the changes into our
|
||||
own book. A change's `hash` therefore also equals that of the next `book`.
|
||||
- **One update can span several frames** (seen in a 90 s recording of 40
|
||||
tokens): `price_change` frames with the same `timestamp`, one change each
|
||||
for the same token, all carrying the hash of the book after the *whole*
|
||||
batch. Check a hash once its batch is over.
|
||||
- **`last_trade_price` belongs to the market**: both tokens of a market hash
|
||||
the same value (20 of 20 markets). A trade changes it, and the hashes use the
|
||||
new value up to ~680 ms before the `last_trade_price` event arrives; that
|
||||
event names one token, and the hashed value was its `price` (16 cases), its
|
||||
complement 1 − `price` (8 cases), or neither (1 case, a trade that probably
|
||||
filled at two prices). So a hash check fails after most trades with nothing
|
||||
lost; a fresh book (from a resync) carries the right value.
|
||||
- **Resync on the open connection:** `unsubscribe` then `subscribe` for a token
|
||||
makes the server send its full `book` at once. That book can lag the changes
|
||||
already sent by a few ms: in a 2-minute run, 14 of 14 fresh books that
|
||||
differed from ours had the hash of one of our own recent changes, i.e. a
|
||||
state we had already passed. Its timestamp and `last_trade_price` can be
|
||||
newer than its levels: in a 15-minute run, 16 of 27 fresh books that
|
||||
differed from ours equalled an earlier state of ours by levels, while their
|
||||
hash matched none of our changes (two of them with a timestamp ending in
|
||||
`.000`, as if cut to the second). So recognise a lagging book by its levels.
|
||||
It can also be *ahead*: between handling the
|
||||
`unsubscribe` and the `subscribe`, the server sends no changes for the
|
||||
token, so the book holds changes we never receive (6 of 302 resyncs in a
|
||||
10-minute run, 1 to 48 ms ahead of our last change; 39 others were behind). A `subscribe` for a token
|
||||
already subscribed is answered with the plain text `NO NEW ASSETS` (not
|
||||
JSON).
|
||||
- **Keepalive (docs):** send the text frame `PING` every 10 s; the server
|
||||
answers `PONG`. The spike survived 60 s without it, but it is required.
|
||||
- **Server restarts:** the server can close the socket with code `Away`,
|
||||
reason `server shutting down` (seen once in about 8 minutes of connections).
|
||||
- **`last_trade_price`:** `asset_id`, `price`, `size`, `side`, `fee_rate_bps`,
|
||||
`timestamp`, `transaction_hash` (one per trade, for one token).
|
||||
- **Not observed:** `tick_size_change` and the custom-feature events.
|
||||
- **Not observed:** `tick_size_change` and the custom-feature events. The docs
|
||||
now show these only in an SDK's wrapped form (`topic` / `payload`,
|
||||
camelCase), not the raw wire shape above.
|
||||
- **REST from scripts:** `clob.polymarket.com` answers Python's `urllib` with
|
||||
403 (curl and `ureq` get 200), most likely Cloudflare filtering by user agent.
|
||||
Fetch with curl in throwaway scripts.
|
||||
|
||||
## Clock accuracy bounds time accuracy
|
||||
`received_at` comes from the local clock, so it carries the clock's error. On
|
||||
|
||||
+17
-16
@@ -18,7 +18,7 @@ A running service that, for a chosen exchange:
|
||||
(subscribe to new, unsubscribe from gone);
|
||||
- maintains a correct, complete, time-accurate order book per symbol;
|
||||
- emits SBE `PriceLevelUpdate` deltas continuously and a full `Snapshot` per
|
||||
symbol every minute, on subscribe, and after any re-snapshot;
|
||||
symbol every minute, on subscribe, and after any resync;
|
||||
- shuts down cleanly on SIGINT (close sockets, flush/close output).
|
||||
|
||||
Full target (all four extensions): redundant websockets, subscription
|
||||
@@ -61,11 +61,11 @@ must guarantee:
|
||||
1. **Receive time stamped at the socket edge**, before any queue or processing.
|
||||
2. **A per-symbol book folded from absolute deltas** (overwrite the level;
|
||||
quantity 0 deletes it).
|
||||
3. **Loss detection → discard-and-re-snapshot** (a lost message means the book
|
||||
is silently corrupt; rebuilding is the only safe response). How a loss is
|
||||
detected depends on the exchange: a sequence gap where one exists; Polymarket
|
||||
has none, so its `hash` or best bid/ask instead
|
||||
([notes.md](notes.md)) — settled in M2.
|
||||
3. **Loss detection → resync** (a lost message means the book is silently
|
||||
corrupt; replacing it with a fresh full book is the only safe response). How
|
||||
a loss is detected depends on the exchange: a sequence gap where one exists;
|
||||
on Polymarket, which has none, recomputing its book hash after every change
|
||||
(settled in M2; the formula is in [notes.md](notes.md)).
|
||||
|
||||
These are proven first, before redundancy or scale.
|
||||
|
||||
@@ -92,7 +92,7 @@ function, symbols shardable later) are also the simplest, clearest design.
|
||||
not.** A channel, a function boundary, or a `trait` with two real impls keeps
|
||||
the door open; the clever internals are what we refuse.
|
||||
- **Never simplified away, even in V1:** receive-time stamping at the socket
|
||||
edge, and loss → re-snapshot. These are correctness, not readability tradeoffs.
|
||||
edge, and loss → resync. These are correctness, not readability tradeoffs.
|
||||
- **If a corner is genuinely cut for speed later** (a global lock, an O(n²)
|
||||
scan), mark it with a comment naming the ceiling and the upgrade path, so the
|
||||
debt is visible rather than silently clever.
|
||||
@@ -109,10 +109,14 @@ Their operational form, for writing code, is in the project
|
||||
modules under `src/`. `website/` stays separate as the brief.
|
||||
- **Scope of markets:** regular markets only; MVEs excluded; websocket exchanges
|
||||
only.
|
||||
- **Output destination** (settled in sprint 01): `AlignedSbeWriter` takes a file
|
||||
path, and stdout is just the path `/dev/stdout` (or a tiny writer tweak), so
|
||||
one code path covers both. Chunks must be 64 MiB: `AlignedSbeReader::open`
|
||||
reads nothing else.
|
||||
- **Output destination** (settled in sprints 01 and 02): `AlignedSbeWriter`
|
||||
takes a file path; the program's only argument is that path, default
|
||||
`/dev/stdout`, and logs go to stderr. No rotation: one output per run
|
||||
(rotation can't apply to stdout; a consumer that wants files rotates them).
|
||||
Chunks must be 64 MiB: `AlignedSbeReader::open` reads nothing else.
|
||||
- **Crash ceiling** (accepted in sprint 02): the writer holds the current
|
||||
64 MiB chunk in memory until it is full, so a hard crash loses it; flushing
|
||||
more often pads each flush to 64 MiB. Taken up with extension 1 (M4).
|
||||
|
||||
## Workflow differences
|
||||
|
||||
@@ -175,14 +179,11 @@ trusted source is chosen and switched, and emitting a fresh snapshot on switch).
|
||||
|
||||
## Open questions
|
||||
|
||||
- **Loss detection per exchange.** Polymarket has no sequence number: can we
|
||||
reproduce its book `hash`, or do we check best bid/ask after each change? M2's
|
||||
opening question. Kalshi's sequence semantics: checked in M6.
|
||||
- **Loss detection on Kalshi.** Its sequence semantics: checked in M6.
|
||||
(Polymarket: settled in M2, by its book hash.)
|
||||
- **Symbol lifecycle signal.** What marks a market "no longer trading"?
|
||||
Polymarket's `new_market` / `market_resolved` events (with
|
||||
`custom_feature_enabled`) are candidates; settle in M3.
|
||||
- **File rotation policy.** When to roll to a new output file (size, time), given
|
||||
64 MiB chunks. Resolve in M2 with the output writer.
|
||||
- **Redundancy source-selection.** How to pick/switch the trusted source
|
||||
(freshness, message rate, which source is ahead). Designed in the redundancy
|
||||
sprint.
|
||||
+27
-9
@@ -20,14 +20,14 @@ abstraction is *extracted* once a second exchange exists, not designed up front.
|
||||
Connect to Polymarket and log raw messages.
|
||||
*Ends:* CI green; raw Polymarket messages flowing.
|
||||
|
||||
2. **Single-source book pipeline (Polymarket, subset of symbols)** — *not started*
|
||||
The central bet in its first real form: normalize events → per-symbol book
|
||||
(fold absolute deltas) → loss-detect → re-snapshot → SBE output (deltas +
|
||||
snapshot every minute, on subscribe, on re-snapshot) → receive-time stamping
|
||||
→ clean SIGINT, with the `PING` keepalive every 10 s. Polymarket has no
|
||||
sequence number, so loss detection (its book `hash`, or best bid/ask) is the
|
||||
opening question. One/few sockets, a fixed subset of symbols. Deterministic
|
||||
replay tests + snapshot↔delta reconciliation.
|
||||
2. **Single-source book pipeline (Polymarket, subset of symbols)** — *done (2026-11-04)*
|
||||
The central bet in its first real form: parse → per-symbol book (fold
|
||||
absolute deltas) → loss-detect → resync → SBE output (deltas + snapshot
|
||||
every minute, on subscribe, after a resync) → receive-time stamping → clean
|
||||
SIGINT, with the `PING` keepalive every 10 s. Loss detection: Polymarket's
|
||||
book hash, reproduced after every change. One socket, a fixed subset of
|
||||
symbols. Deterministic replay tests + snapshot↔delta reconciliation.
|
||||
Sprint: [../roadmap/02. book pipeline.md](../roadmap/02.%20book%20pipeline.md).
|
||||
*Ends:* correct SBE output for a symbol subset; tests green.
|
||||
|
||||
3. **Full coverage: discovery + sharding (base challenge complete)** — *not started*
|
||||
@@ -35,6 +35,15 @@ abstraction is *extracted* once a second exchange exists, not designed up front.
|
||||
Polymarket's `new_market` / `market_resolved` events — using subscribe /
|
||||
unsubscribe on open connections; load-balance all live regular-market symbols
|
||||
across N sockets.
|
||||
**Opening questions (from M2):** what "live" means, and so how many symbols:
|
||||
Gamma lists at least 100,000 markets `active` and not `closed` (2026-11-04,
|
||||
count capped), at least 13,000 of them traded in the last 24 h, against the
|
||||
40 symbols M2 runs. At that size the minute Snapshots alone (~2 KB each)
|
||||
would be ~0.5 TB a day for 200,000 symbols, mostly unchanged books; and
|
||||
memory (~10 KB a book), sockets per connection, resyncs per minute and the
|
||||
book thread's throughput (M2: ~30,000 frames/s on one thread, 43 frames/s
|
||||
of load) all need measuring. Gamma pages past 10,000 markets only through
|
||||
`/markets/keyset` with `after_cursor`.
|
||||
*Ends:* every live Polymarket symbol tracked — base challenge satisfied for
|
||||
one exchange.
|
||||
|
||||
@@ -44,7 +53,12 @@ abstraction is *extracted* once a second exchange exists, not designed up front.
|
||||
**Async re-evaluation checkpoint:** before building this, reassess whether the
|
||||
socket-management complexity (reconnect/timeout/select, redundancy) warrants
|
||||
swapping the thread-only runtime for tokio (see ADR 0001).
|
||||
*Ends:* redundancy demonstrably reduces data loss; source-switch re-snapshots.
|
||||
Also: surviving a hard crash without losing the writer's in-memory chunk,
|
||||
and closing the resync gap (the few ms of a symbol's changes the server
|
||||
doesn't send between our unsubscribe and subscribe), both accepted as
|
||||
ceilings in M2.
|
||||
*Ends:* redundancy demonstrably reduces data loss; a Snapshot on every source
|
||||
switch.
|
||||
|
||||
5. **Extension 2 — subscription monitoring** — *not started*
|
||||
Per-symbol activity monitoring (data points per second/minute); unsubscribe +
|
||||
@@ -92,3 +106,7 @@ abstraction is *extracted* once a second exchange exists, not designed up front.
|
||||
- Observability: metrics, structured logging, dashboards.
|
||||
- Config-driven symbol filtering and per-exchange tuning.
|
||||
- Containerized packaging.
|
||||
- Counting a reconnect's differing books as losses rather than resync gaps
|
||||
(a reconnect marker through the frames channel; M2 labels them gaps).
|
||||
- A `reconcile` that reads the collector's log, to tell resync gaps from
|
||||
losses in its own report.
|
||||
Reference in new issue
Block a user