Archived
feat(polymarket): hash-verified order books for a subset of symbols
Collects the order books of Polymarket's 20 busiest markets (40 symbols) into the template's SBE format: a Delta per level change, plus Snapshots on subscribe, every minute, and whenever a fresh book corrects ours. Every change is checked against Polymarket's own book hash; a failed check resyncs the symbol on the open socket, and the log counts losses apart from the few ms our own resync hides. Runs until SIGINT, which closes the socket and flushes the output; reconnects when the server drops it. A `reconcile` binary checks an output file against itself.
This commit is contained in:
22 files changed
+6423
-84
No files matched your search
+27
-9
@@ -20,14 +20,14 @@ abstraction is *extracted* once a second exchange exists, not designed up front.
|
||||
Connect to Polymarket and log raw messages.
|
||||
*Ends:* CI green; raw Polymarket messages flowing.
|
||||
|
||||
2. **Single-source book pipeline (Polymarket, subset of symbols)** — *not started*
|
||||
The central bet in its first real form: normalize events → per-symbol book
|
||||
(fold absolute deltas) → loss-detect → re-snapshot → SBE output (deltas +
|
||||
snapshot every minute, on subscribe, on re-snapshot) → receive-time stamping
|
||||
→ clean SIGINT, with the `PING` keepalive every 10 s. Polymarket has no
|
||||
sequence number, so loss detection (its book `hash`, or best bid/ask) is the
|
||||
opening question. One/few sockets, a fixed subset of symbols. Deterministic
|
||||
replay tests + snapshot↔delta reconciliation.
|
||||
2. **Single-source book pipeline (Polymarket, subset of symbols)** — *done (2026-11-04)*
|
||||
The central bet in its first real form: parse → per-symbol book (fold
|
||||
absolute deltas) → loss-detect → resync → SBE output (deltas + snapshot
|
||||
every minute, on subscribe, after a resync) → receive-time stamping → clean
|
||||
SIGINT, with the `PING` keepalive every 10 s. Loss detection: Polymarket's
|
||||
book hash, reproduced after every change. One socket, a fixed subset of
|
||||
symbols. Deterministic replay tests + snapshot↔delta reconciliation.
|
||||
Sprint: [../roadmap/02. book pipeline.md](../roadmap/02.%20book%20pipeline.md).
|
||||
*Ends:* correct SBE output for a symbol subset; tests green.
|
||||
|
||||
3. **Full coverage: discovery + sharding (base challenge complete)** — *not started*
|
||||
@@ -35,6 +35,15 @@ abstraction is *extracted* once a second exchange exists, not designed up front.
|
||||
Polymarket's `new_market` / `market_resolved` events — using subscribe /
|
||||
unsubscribe on open connections; load-balance all live regular-market symbols
|
||||
across N sockets.
|
||||
**Opening questions (from M2):** what "live" means, and so how many symbols:
|
||||
Gamma lists at least 100,000 markets `active` and not `closed` (2026-11-04,
|
||||
count capped), at least 13,000 of them traded in the last 24 h, against the
|
||||
40 symbols M2 runs. At that size the minute Snapshots alone (~2 KB each)
|
||||
would be ~0.5 TB a day for 200,000 symbols, mostly unchanged books; and
|
||||
memory (~10 KB a book), sockets per connection, resyncs per minute and the
|
||||
book thread's throughput (M2: ~30,000 frames/s on one thread, 43 frames/s
|
||||
of load) all need measuring. Gamma pages past 10,000 markets only through
|
||||
`/markets/keyset` with `after_cursor`.
|
||||
*Ends:* every live Polymarket symbol tracked — base challenge satisfied for
|
||||
one exchange.
|
||||
|
||||
@@ -44,7 +53,12 @@ abstraction is *extracted* once a second exchange exists, not designed up front.
|
||||
**Async re-evaluation checkpoint:** before building this, reassess whether the
|
||||
socket-management complexity (reconnect/timeout/select, redundancy) warrants
|
||||
swapping the thread-only runtime for tokio (see ADR 0001).
|
||||
*Ends:* redundancy demonstrably reduces data loss; source-switch re-snapshots.
|
||||
Also: surviving a hard crash without losing the writer's in-memory chunk,
|
||||
and closing the resync gap (the few ms of a symbol's changes the server
|
||||
doesn't send between our unsubscribe and subscribe), both accepted as
|
||||
ceilings in M2.
|
||||
*Ends:* redundancy demonstrably reduces data loss; a Snapshot on every source
|
||||
switch.
|
||||
|
||||
5. **Extension 2 — subscription monitoring** — *not started*
|
||||
Per-symbol activity monitoring (data points per second/minute); unsubscribe +
|
||||
@@ -92,3 +106,7 @@ abstraction is *extracted* once a second exchange exists, not designed up front.
|
||||
- Observability: metrics, structured logging, dashboards.
|
||||
- Config-driven symbol filtering and per-exchange tuning.
|
||||
- Containerized packaging.
|
||||
- Counting a reconnect's differing books as losses rather than resync gaps
|
||||
(a reconnect marker through the frames channel; M2 labels them gaps).
|
||||
- A `reconcile` that reads the collector's log, to tell resync gaps from
|
||||
losses in its own report.
|
||||
Reference in new issue
Block a user