feat(polymarket): hash-verified order books for a subset of symbols

Collects the order books of Polymarket's 20 busiest markets (40 symbols)
into the template's SBE format: a Delta per level change, plus Snapshots on
subscribe, every minute, and whenever a fresh book corrects ours. Every
change is checked against Polymarket's own book hash; a failed check
resyncs the symbol on the open socket, and the log counts losses apart from
the few ms our own resync hides. Runs until SIGINT, which closes the socket
and flushes the output; reconnects when the server drops it. A `reconcile`
binary checks an output file against itself.
This commit is contained in:
pascal committed 2026-11-04 17:10:17 +01:00
1 parent 0e687e7eba
commit adb7fee8d8
22 files changed
+6423 -84

No files matched your search

+17 -16
View File
@@ -18,7 +18,7 @@ A running service that, for a chosen exchange:
(subscribe to new, unsubscribe from gone);
- maintains a correct, complete, time-accurate order book per symbol;
- emits SBE `PriceLevelUpdate` deltas continuously and a full `Snapshot` per
symbol every minute, on subscribe, and after any re-snapshot;
symbol every minute, on subscribe, and after any resync;
- shuts down cleanly on SIGINT (close sockets, flush/close output).
Full target (all four extensions): redundant websockets, subscription
@@ -61,11 +61,11 @@ must guarantee:
1. **Receive time stamped at the socket edge**, before any queue or processing.
2. **A per-symbol book folded from absolute deltas** (overwrite the level;
quantity 0 deletes it).
3. **Loss detection → discard-and-re-snapshot** (a lost message means the book
is silently corrupt; rebuilding is the only safe response). How a loss is
detected depends on the exchange: a sequence gap where one exists; Polymarket
has none, so its `hash` or best bid/ask instead
([notes.md](notes.md)) — settled in M2.
3. **Loss detection → resync** (a lost message means the book is silently
corrupt; replacing it with a fresh full book is the only safe response). How
a loss is detected depends on the exchange: a sequence gap where one exists;
on Polymarket, which has none, recomputing its book hash after every change
(settled in M2; the formula is in [notes.md](notes.md)).
These are proven first, before redundancy or scale.
@@ -92,7 +92,7 @@ function, symbols shardable later) are also the simplest, clearest design.
not.** A channel, a function boundary, or a `trait` with two real impls keeps
the door open; the clever internals are what we refuse.
- **Never simplified away, even in V1:** receive-time stamping at the socket
edge, and loss → re-snapshot. These are correctness, not readability tradeoffs.
edge, and loss → resync. These are correctness, not readability tradeoffs.
- **If a corner is genuinely cut for speed later** (a global lock, an O(n²)
scan), mark it with a comment naming the ceiling and the upgrade path, so the
debt is visible rather than silently clever.
@@ -109,10 +109,14 @@ Their operational form, for writing code, is in the project
modules under `src/`. `website/` stays separate as the brief.
- **Scope of markets:** regular markets only; MVEs excluded; websocket exchanges
only.
- **Output destination** (settled in sprint 01): `AlignedSbeWriter` takes a file
path, and stdout is just the path `/dev/stdout` (or a tiny writer tweak), so
one code path covers both. Chunks must be 64 MiB: `AlignedSbeReader::open`
reads nothing else.
- **Output destination** (settled in sprints 01 and 02): `AlignedSbeWriter`
takes a file path; the program's only argument is that path, default
`/dev/stdout`, and logs go to stderr. No rotation: one output per run
(rotation can't apply to stdout; a consumer that wants files rotates them).
Chunks must be 64 MiB: `AlignedSbeReader::open` reads nothing else.
- **Crash ceiling** (accepted in sprint 02): the writer holds the current
64 MiB chunk in memory until it is full, so a hard crash loses it; flushing
more often pads each flush to 64 MiB. Taken up with extension 1 (M4).
## Workflow differences
@@ -175,14 +179,11 @@ trusted source is chosen and switched, and emitting a fresh snapshot on switch).
## Open questions
- **Loss detection per exchange.** Polymarket has no sequence number: can we
reproduce its book `hash`, or do we check best bid/ask after each change? M2's
opening question. Kalshi's sequence semantics: checked in M6.
- **Loss detection on Kalshi.** Its sequence semantics: checked in M6.
(Polymarket: settled in M2, by its book hash.)
- **Symbol lifecycle signal.** What marks a market "no longer trading"?
Polymarket's `new_market` / `market_resolved` events (with
`custom_feature_enabled`) are candidates; settle in M3.
- **File rotation policy.** When to roll to a new output file (size, time), given
64 MiB chunks. Resolve in M2 with the output writer.
- **Redundancy source-selection.** How to pick/switch the trusted source
(freshness, message rate, which source is ahead). Designed in the redundancy
sprint.