This repository has been archived on 2026-11-16. You can view files and clone it. You cannot open issues or pull requests or push a commit.
Files
pascal adb7fee8d8 feat(polymarket): hash-verified order books for a subset of symbols
Collects the order books of Polymarket's 20 busiest markets (40 symbols)
into the template's SBE format: a Delta per level change, plus Snapshots on
subscribe, every minute, and whenever a fresh book corrects ours. Every
change is checked against Polymarket's own book hash; a failed check
resyncs the symbol on the open socket, and the log counts losses apart from
the few ms our own resync hides. Runs until SIGINT, which closes the socket
and flushes the output; reconnects when the server drops it. A `reconcile`
binary checks an output file against itself.
2026-11-04 17:10:17 +01:00

408 lines
23 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# 02. Single-source book pipeline
**Status: done (2026-11-04)**
The central bet in its first real form, on one Polymarket socket and a fixed
subset of symbols: stamp each frame at the socket, fold changes into a book per
symbol, detect any loss through Polymarket's book hash and resync, and write
Deltas plus Snapshots (on subscribe, every minute, after a resync) with the
template's SBE writer, until SIGINT closes everything cleanly. Design:
[../docs/project.md](../docs/project.md) (central bet, pipeline sketch),
[../docs/notes.md](../docs/notes.md) (protocol facts),
[../GLOSSARY.md](../GLOSSARY.md) (terms), ADR 0001 (threads, shared-nothing).
## Pipeline
Three threads, joined by unbounded `crossbeam` channels:
```
socket thread ──(receive time, frame text)──▶ book thread ──(Snapshot | Delta)──▶ writer thread
▲ │
└──────────── resync requests ───────────────┘
```
- **Socket thread**: connects, subscribes to every symbol, reads. The receive
time is stamped right after `read()` returns, before anything else. The
socket has a short read timeout (200 ms) so the loop wakes up to send `PING`
every 10 s, to send the unsubscribe / subscribe pairs the book thread asks for,
and to see the SIGINT flag. It doesn't parse frames.
- **Book thread**: parses frames, folds them into books, checks hashes, decides
resyncs, emits a Delta per applied level change and Snapshots. It waits on
the channel with a timeout so it also wakes for the minute Snapshots.
- **Writer thread**: owns the `AlignedSbeWriter` (64 MiB chunks) and writes what
it receives, in order.
Why the read timeout and not a fourth thread for writes: tungstenite 0.30 has no
`split`, and `read()` takes `&mut self` and itself writes (pong replies, the
close handshake), so read and write can't live on two threads without a mutex.
The timeout was tried live: 1,081 timeouts in 4 minutes, every frame intact,
23 PINGs answered by 23 PONGs.
## Data model
A **symbol** is a Polymarket token, named in the output by its token id; both
tokens of a market are tracked, as two symbols. Per symbol, the book thread
keeps:
- `bids` and `asks`: `BTreeMap<i64, i64>`, price → quantity in fixed point
(1,000,000 = 1), parsed exactly from the decimal strings, never through a
float. More than 6 decimals is a protocol error for that symbol: it resyncs.
- the other inputs of the book hash, as the exchange's exact strings:
`market`, `tick_size`, `last_trade_price`, `min_order_size`, `neg_risk`.
- the last exchange timestamp (a Snapshot's `exchange_timestamp`).
- a state: `AwaitingBook` (no book yet since subscribing: changes are
dropped), `Live` (every change checked), `Resyncing` (a fresh book is
asked for: changes are applied and written unchecked until it comes), or
`Unverifiable` (see below).
- the hash of a batch not yet checked, and the last 64 changes with their
hashes (see below).
The subset: at startup, Gamma's `/markets` top 20 by 24 h volume (`active`,
`closed=false`, order book enabled), which gives the token ids
(`clobTokenIds`, a JSON array inside a string), `orderMinSize` and `negRisk`.
40 symbols (fewer if one of the 20 has no order book). Fixed for the run;
discovery is M3.
Output fields: a Delta's `backtest_timestamp` and a Snapshot's `received_at`
are both the receive time. A minute Snapshot carries the receive time of the
last frame the book thread applied, not the wall clock: the book is exact as of
that frame, while frames still queued aren't in it yet.
No `NormalizedEvent` yet: Polymarket's messages parse into Polymarket types,
the book code knows nothing of Polymarket, and the hash check is Polymarket
code. The shared seam is extracted in M6 from two exchanges.
## Loss detection and resync
Checked live on 6 symbols for 2 minutes (14 books, 182 changes, all matched):
the book hash is the SHA-1 of the compact JSON
```
{"market","asset_id","timestamp","hash":"","bids","asks","min_order_size","tick_size","neg_risk","last_trade_price"}
```
in that key order, with bids ascending and asks descending by price, prices and
sizes as canonical decimals (true of every price and size in 40 books checked),
and, for a change, the change message's `timestamp`. Full books sent
mid-stream omit `tick_size` and `last_trade_price`: the last known values are
used.
Two things the 2-minute check didn't show, found in the 90 s test recording
(40 symbols, 3,850 changes; `docs/notes.md`):
- **Batches**: one update can arrive split over several frames, all carrying
the hash of the book after the whole batch. A change's hash is therefore
checked when its batch is over: right away if it matches, else when the
symbol's next change carries a different hash (before applying it).
- **The last trade price can't be known in time**: trades change it, the hash
picks up the new value up to ~680 ms before the `last_trade_price` event,
and that event gives it, its complement (1 − p), or neither. So after a
trade the check fails although nothing was lost; only a fresh book (which
carries the last trade price) tells the two apart.
The rules:
1. **A complete full book checks itself** (one that carries
`last_trade_price`: on subscribe and after a resync): hash it as received.
A mismatch means our formula or metadata is wrong for that symbol, not a
loss: log it loudly, mark the symbol `Unverifiable`, and keep writing its
Deltas, unchecked. No resync, which would loop forever.
2. **A failed check asks for a fresh book** (a change, after its batch, or a
mid-stream book, whose last trade price may be stale): the symbol goes
`Resyncing`, the book thread asks the socket thread to unsubscribe and
subscribe it again (tried live: the fresh book arrives at once), and its
changes keep being applied and written, unchecked, until that book arrives.
A symbol still `Resyncing` at a wall-clock minute is asked for again, so a
lost answer doesn't leave it unchecked for the rest of the run. A full book
that can't be read is a protocol error: the symbol resyncs too (it keeps
`AwaitingBook` if it had no book yet).
3. **A full book replaces ours** (on subscribe, mid-stream after trades, after
a resync or a reconnect). It is written as a Snapshot if it is the symbol's
first, or if its levels differ from ours: that is a **loss**, logged and
counted. A fresh book with the same levels (the usual case after a trade)
writes nothing. Two timings of a resync's fresh book need care:
- **Behind our stream**: it can lag the changes already sent by a few ms
(live: 14 of 14 differing books in 2 minutes), and its timestamp or last
trade price can be newer than its levels, so its hash may match none of
our changes. Checked in order, against the symbol's last 64 changes:
1. if it equals our book as it was after one of them (ours rewound,
each change undone back to the quantity it replaced), it is behind:
ours is kept, its tick size and last trade price are taken, nothing
is written. That past state of ours was the exchange's own, and the
changes since are what it sent us. (Comparing the book *plus our
later changes* with ours instead would be unsound: a change we lost
at a price we had changed before is overwritten by that earlier
change, and the wrong book passes.)
2. else, if its hash is that of one of our changes (the latest
included), it is the exchange's state after that change and ours is
wrong (a loss): the changes received since are applied to it, and it
replaces ours.
- **Ahead of our stream (a resync gap)**: between the server handling our
`unsubscribe` and our `subscribe`, it sends no changes for that symbol,
so the fresh book holds changes we never got. A fresh book answering
our resync that isn't behind, whose levels differ and whose exchange
time is at or after our last change, is counted as a **resync gap**, not
a loss: its Snapshot corrects the
state, the Deltas inside the gap are lost. M4's redundant sockets can
close it.
4. **`tick_size_change`** resyncs the symbol: the fresh book carries the new
tick size. Its wire shape was never observed (the docs now show an SDK's
wrapped form), and this needs only `event_type` and `asset_id`.
5. **`last_trade_price` events are ignored**: the hash takes the new price
before the event and not always the event's value; the fresh book after
the failed check carries the right one. Trades aren't in the schema.
`PONG` and `NO NEW ASSETS` (the server's answer to a subscribe for a symbol
already subscribed) are skipped, as are unknown event types; any other frame
that isn't readable JSON is logged and skipped.
## Output, shutdown and reconnect
- **Destination**: the program's only argument is the output path, default
`/dev/stdout`. Logs go to stderr. No rotation: one output per run.
- **Minute Snapshots**: every symbol with a book (not `AwaitingBook`), at each
wall-clock minute.
- **SIGINT**: `ctrlc` sets a flag; the socket thread sees it within 200 ms
when connected, within 10 s while connecting (the TCP connect and the TLS
handshake have a 10 s timeout), and 1 s while waiting to reconnect. It
closes the websocket and drops its channel; the book thread drains and drops
its own; the writer drains, flushes the last chunk and closes the file. Each
stage stops because the previous one did.
- **Reconnect**: when the server closes the socket or it errors, the socket
thread waits 1 s, connects again and subscribes every symbol; the fresh
books then go through rule 3. Seen once in about 8 minutes of test
connections: a close with "server shutting down". Ceiling: a symbol that
was `Resyncing` when the connection dropped has its reconnect book counted
as a resync gap rather than a loss (the state and output are right either
way; only the count is mislabelled, next to the logged reconnect).
- **Crash ceiling**: `AlignedSbeWriter` keeps the current 64 MiB chunk in memory
and writes it only when full (`src/util/aligned_sbe_writer.rs:48`) or on
`flush`, which pads it to 64 MiB with zeros (`:68-70`). A hard crash loses
the unwritten chunk, hours of data at 40 symbols; flushing every minute would
write ~90 GB a day of padding. Accepted for M2; the brief lists "we go down
unexpectedly" under extension 1, so M4 takes it up.
## Libraries
New: `serde_json` (parsing), `ureq` 3 (Gamma; tried: it reaches Gamma and the
CLOB, and its TLS uses `ring`, already locked, with no `aws-lc-rs`), `sha1` 0.11
(RustCrypto), `ctrlc` 3.
## Decisions
- **Loss detection by the book hash, after every change.** It identifies the
full book state, so a check after each change catches any loss at once.
*Rejected:* comparing best bid / ask, which misses losses deeper in the book.
- **Resync by unsubscribe + subscribe on the open socket.** The fresh book
arrives at once, on the stream we're already reading. *Rejected:* REST
`/book` (a second path whose timing relative to the stream is unknown).
- **A failed check asks for a fresh book but keeps applying changes; a loss is
a fresh book that differs** (supersedes "a mismatch is a loss, changes are
dropped until the fresh book", because the recording showed checks failing
after trades with nothing lost: the hash includes a last trade price we
can't know in time). After a trade it costs one book on the socket and
nothing in the output; after a real loss the fresh book's Snapshot corrects
the output. *Rejected:* guessing the last trade price from the trade event
(its value, its complement, the batch's prices), a heuristic over
undocumented behaviour that already missed a case in 90 s; dropping changes
and writing a Snapshot on every trade; checking only best bid / ask, which
misses deeper losses.
- **A full book whose hash we saw on a recent change is brought forward with
the changes received since, then compared** (supersedes "a book whose
levels differ is a loss", because live, fresh books lagged our stream by a
few ms; and supersedes "keep ours", agreed in between, because our book may
be the wrong one). The hash identifies the full state, so the test is
exact. *Rejected:* comparing exchange timestamps (ties at the same
millisecond were common); REST `/book` (may lag the same way, and is a
second path); keeping our book.
- **"Behind" is recognised by levels first, by hash second** (supersedes the
hash match alone, because live, fresh books were behind by levels while
carrying a newer timestamp or last trade price: both losses of a 15-minute
run and 14 books counted as resync gaps, which then replaced our newer book
with a stale one). The hash match stays second: when our book is the wrong
one, it still brings the fresh book forward to the right state at once.
The level test rewinds ours (supersedes "the book plus our later changes
equals ours", found unsound at the close review: a lost change at a price
we had changed before is masked). *Rejected:* levels only (a loss would
first write the stale book); hashes only (the misclassifications above).
- **A requested resync is asked again at each minute while unanswered; the
connect and TLS handshake time out after 10 s** (both found at the close
review: a lost answer left a symbol unchecked for the rest of the run, and
SIGINT could hang on a stalled connect). *Rejected:* a per-symbol resync
timer (more state for the same effect at this scale).
- **A resync gap is counted apart from losses and accepted in M2.** Our own
unsubscribe / subscribe opens a few ms in which the server sends us no
changes for the symbol; the fresh book's Snapshot keeps the state right.
*Rejected:* resyncing through a short-lived second connection (M4's
machinery, and its book could be ahead of our stream, a case not handled);
fewer resyncs by guessing the last trade price (rejected above).
- **A change's hash is checked when its batch is over.** One update can span
frames that all carry its final hash. *Rejected:* checking after each
change, which fails mid-batch.
- **A self-check on full books, `Unverifiable` on failure.** Separates "our
formula is wrong" from "we lost a message". *Rejected:* resyncing on any
mismatch, which loops forever on a formula error.
- **Levels as fixed-point integers, other hash inputs as exact strings.** The
output needs integers, and integer keys make `"0.5"` and `"0.50"` one level;
`last_trade_price` is not canonical (`"0.080"` seen), so it stays a string.
*Rejected:* keying levels by the exchange's strings.
- **Both tokens of a market are symbols.** The brief asks for a book per symbol,
and consumers shouldn't need to know Polymarket mirrors them. *Rejected:* one
token per market (half the output).
- **Three threads, socket / book / writer.** The socket thread only reads and
stamps; the writer survives a book thread crash and still flushes, and its
64 MiB writes don't pause the book thread; M3 needs it anyway. *Rejected:*
two threads (book thread owning the writer), and four (no `split` in
tungstenite without a mutex).
- **Unbounded channels.** Blocking stalls the socket read and hurts time
accuracy; dropping breaks completeness. Their ceiling (memory, if a consumer
stays behind) is named in a comment, and the book thread's minute log
reports the most frames that waited for it that minute (supersedes "logs the
queue length past 10,000", whose check could only run when the queue was
empty). Switching to bounded is a constructor change. *Rejected:* bounded,
for now.
- **Subset from Gamma at startup.** A hardcoded list goes stale in days (M1's
did), and M3 needs the HTTP client anyway. *Rejected:* hardcoded token ids.
- **Simple reconnect in M2.** A collector that dies at each server restart
isn't one, and rule 3 already handles the fresh books. *Rejected:* exit on
disconnect until M4, which keeps redundancy.
- **Output to a path, default stdout, no rotation.** The brief says stdout;
rotation can't apply to it. *Rejected:* rotating files by size or time.
- **`sha1` (RustCrypto).** The usual choice, and faster: 1.8 µs per book
against 4.6 µs for `sha1_smol` (measured, ~5 KB). *Rejected:* `sha1_smol`,
`ring`'s SHA-1.
## Tests
No network in tests. A fixture of a few minutes of real frames for the subset,
each with its receive time, plus the Gamma metadata the hash needs, recorded
once by a throwaway script and committed (a few MB at most; its size is
checked when recorded).
- **Replay**: the fixture replays with no loss and no `Unverifiable` symbol,
split batches and trades included.
- **Induced loss**: the replay with one frame dropped: a resync of that
symbol is requested where the full replay requests none, and a fresh book
that differs is written as a Snapshot and counted as a loss.
- **Self-check**: a full book with wrong metadata marks the symbol
`Unverifiable` and requests no resync.
- **Reconciliation**: book and writer threads run on the fixture into a temp
file (`std::env::temp_dir()`), read back with `AlignedSbeReader`; per symbol,
the Deltas applied to a Snapshot equal the next Snapshot.
- **Parsing**: fixed-point edges (`"0"`, integers, 6 decimals, more than 6,
empty sides), non-JSON frames.
## Not in this sprint
- Discovery, the full symbol set, several sockets (M3).
- Redundant sockets, and losing nothing during a reconnect or a resync gap
(M4).
- Activity monitoring, socket recycling (M5).
- Surviving a hard crash without losing the in-memory chunk (M4).
- Kalshi, the generic core (M6); latency work (M7).
## Done when
- `scripts/ci.sh` is green.
- A 10-minute live run on the top 20 markets, ended by SIGINT, leaves an output
that reads back with `AlignedSbeReader` and passes the reconciliation check.
- That run's log shows no `Unverifiable` symbol, and every loss explained
(e.g. by a reconnect); resync gaps are counted. The reconciliation's
mismatches are exactly those losses and gaps.
- `docs/notes.md` corrected (asks descending) and completed (the hash formula,
resync on the open socket, `NO NEW ASSETS`, mid-stream books' missing fields,
the server's restart close); `docs/project.md` records loss detection, output
destination and rotation as settled.
## Outcome
**Built.** Three threads joined by unbounded channels:
- **Socket thread:** connects with a 10 s timeout, stamps each frame's receive
time right after the read, sends `PING`, resync requests and the close on
SIGINT, and reconnects after 1 s.
- **Book thread** (`src/polymarket/feed.rs`): parses, folds into a `Book` per
symbol, checks Polymarket's book hash once each batch is over, resyncs on a
failed check, and writes Deltas and Snapshots (on subscribe, at each
wall-clock minute, when a fresh book changes our state).
- **Writer thread:** owns the template's `AlignedSbeWriter`.
Also: a `reconcile` binary that checks an output against itself; 16 tests on a
90 s recording of the live feed (replay, induced loss, self-check, books
behind and ahead of our stream, the reconciliation through the real book and
writer threads, parsing). The final 10-minute live run on the top 20 markets
(40 symbols), ended by SIGINT: 97,763 changes verified, 266 resyncs, 29 fresh
books behind ours, 4 resync gaps, 0 losses, no `Unverifiable` symbol. Its 4
reconciliation mismatches are exactly the 4 logged gaps, and at most 20
frames waited for the book thread in any minute. CI is green.
**Diverged from the plan, and why.** The plan's loss detection rested on a
2-minute check of 6 symbols. Larger recordings and live runs showed the feed
breaks that model in four ways, each settled with the user and recorded in
Decisions:
1. **Batches.** One update can span several frames carrying the same hash, so
a hash is checked once its batch is over.
2. **The last trade price.** It is a market-wide hash input that changes
before its event, sometimes as the event's complement. A failed check
therefore asks for a fresh book and keeps writing changes ("soft resync"),
rather than calling it a loss.
3. **Fresh books can lag our stream.** They are recognised by levels (ours
rewound), then by hash.
4. **Fresh books can lead our stream.** Our own unsubscribe/subscribe hides a
few ms of changes: a **resync gap**, counted apart from losses and
accepted for M2.
Two more changes came out of the close review:
- **A soundness hole.** "The book plus our later changes" masked a change lost
at a level we had changed before, so the level test now rewinds ours instead.
- **Resyncs no longer wait forever.** An unanswered or unreadable fresh book is
asked for again at each minute.
Smaller changes:
- `book.rs` moved into the library, so `reconcile` uses `Book`.
- The queue warning, which could only fire when the queue was empty, became a
per-minute peak in the stats line.
- `last_trade_price` events are ignored, as their value isn't reliable.
**Left out, and where it went:**
- **Resync gaps and the hard-crash chunk loss:** M4.
- **Scale** (40 symbols here, against ≥100,000 live markets, ≥13,000 traded in
24 h): M3's opening questions in the roadmap.
- **Counting a reconnect's differing books as losses rather than gaps,** and a
`reconcile` that reads the log: Possible features.
- **The rest, recorded as ceilings:**
- a book that stays unreadable is asked for once per answer;
- the name lookup has no timeout;
- the read loop is written twice in `socket.rs`;
- `open_batch` is a pair of Strings.
### Retrospective
- **Went well:**
- Checking every claim against live data: the hash formula and each of the
four surprises above were found by recordings and runs, not assumed.
- Mutation checks (disable the rule, watch its test fail) on every new
loss-detection test.
- Doc-first changes, each settled with the user before code.
- The two-axis close review found a real soundness hole the tests missed.
- **Went wrong, or took too long:**
- The sprint was agreed on a 6-symbol, 2-minute sample, so its central rule
was rewritten four times during implementation.
- My estimate of resyncs per trade was off (28 estimated, 8 measured).
- The queue warning sat where it could never fire.
- The scale of the real target (100,000+ markets against 40 symbols) came up
only when the user asked.
- **Tested enough?** The rules are tested on a real recording and on built
cases. Not covered by tests:
- the reconnect path (seen live once, before this code);
- SIGINT during a connect;
- the book thread's wall-clock tick when no frame comes;
- the minute re-ask and the unreadable-book path;
- `tick_size_change` (never observed);
- a live `Unverifiable` symbol.
- **Next time:**
- During planning, record a realistic sample of the external feed and keep
it as the fixture.
- Count the target scale before fixing a test subset.
- Have the close review hunt for adversarial cases against each correctness
rule.