Archived
feat(polymarket): hash-verified order books for a subset of symbols
Collects the order books of Polymarket's 20 busiest markets (40 symbols) into the template's SBE format: a Delta per level change, plus Snapshots on subscribe, every minute, and whenever a fresh book corrects ours. Every change is checked against Polymarket's own book hash; a failed check resyncs the symbol on the open socket, and the log counts losses apart from the few ms our own resync hides. Runs until SIGINT, which closes the socket and flushes the output; reconnects when the server drops it. A `reconcile` binary checks an output file against itself.
This commit is contained in:
22 files changed
+6423
-84
No files matched your search
@@ -0,0 +1,407 @@
|
||||
# 02. Single-source book pipeline
|
||||
|
||||
**Status: done (2026-11-04)**
|
||||
|
||||
The central bet in its first real form, on one Polymarket socket and a fixed
|
||||
subset of symbols: stamp each frame at the socket, fold changes into a book per
|
||||
symbol, detect any loss through Polymarket's book hash and resync, and write
|
||||
Deltas plus Snapshots (on subscribe, every minute, after a resync) with the
|
||||
template's SBE writer, until SIGINT closes everything cleanly. Design:
|
||||
[../docs/project.md](../docs/project.md) (central bet, pipeline sketch),
|
||||
[../docs/notes.md](../docs/notes.md) (protocol facts),
|
||||
[../GLOSSARY.md](../GLOSSARY.md) (terms), ADR 0001 (threads, shared-nothing).
|
||||
|
||||
## Pipeline
|
||||
|
||||
Three threads, joined by unbounded `crossbeam` channels:
|
||||
|
||||
```
|
||||
socket thread ──(receive time, frame text)──▶ book thread ──(Snapshot | Delta)──▶ writer thread
|
||||
▲ │
|
||||
└──────────── resync requests ───────────────┘
|
||||
```
|
||||
|
||||
- **Socket thread**: connects, subscribes to every symbol, reads. The receive
|
||||
time is stamped right after `read()` returns, before anything else. The
|
||||
socket has a short read timeout (200 ms) so the loop wakes up to send `PING`
|
||||
every 10 s, to send the unsubscribe / subscribe pairs the book thread asks for,
|
||||
and to see the SIGINT flag. It doesn't parse frames.
|
||||
- **Book thread**: parses frames, folds them into books, checks hashes, decides
|
||||
resyncs, emits a Delta per applied level change and Snapshots. It waits on
|
||||
the channel with a timeout so it also wakes for the minute Snapshots.
|
||||
- **Writer thread**: owns the `AlignedSbeWriter` (64 MiB chunks) and writes what
|
||||
it receives, in order.
|
||||
|
||||
Why the read timeout and not a fourth thread for writes: tungstenite 0.30 has no
|
||||
`split`, and `read()` takes `&mut self` and itself writes (pong replies, the
|
||||
close handshake), so read and write can't live on two threads without a mutex.
|
||||
The timeout was tried live: 1,081 timeouts in 4 minutes, every frame intact,
|
||||
23 PINGs answered by 23 PONGs.
|
||||
|
||||
## Data model
|
||||
|
||||
A **symbol** is a Polymarket token, named in the output by its token id; both
|
||||
tokens of a market are tracked, as two symbols. Per symbol, the book thread
|
||||
keeps:
|
||||
|
||||
- `bids` and `asks`: `BTreeMap<i64, i64>`, price → quantity in fixed point
|
||||
(1,000,000 = 1), parsed exactly from the decimal strings, never through a
|
||||
float. More than 6 decimals is a protocol error for that symbol: it resyncs.
|
||||
- the other inputs of the book hash, as the exchange's exact strings:
|
||||
`market`, `tick_size`, `last_trade_price`, `min_order_size`, `neg_risk`.
|
||||
- the last exchange timestamp (a Snapshot's `exchange_timestamp`).
|
||||
- a state: `AwaitingBook` (no book yet since subscribing: changes are
|
||||
dropped), `Live` (every change checked), `Resyncing` (a fresh book is
|
||||
asked for: changes are applied and written unchecked until it comes), or
|
||||
`Unverifiable` (see below).
|
||||
- the hash of a batch not yet checked, and the last 64 changes with their
|
||||
hashes (see below).
|
||||
|
||||
The subset: at startup, Gamma's `/markets` top 20 by 24 h volume (`active`,
|
||||
`closed=false`, order book enabled), which gives the token ids
|
||||
(`clobTokenIds`, a JSON array inside a string), `orderMinSize` and `negRisk`.
|
||||
40 symbols (fewer if one of the 20 has no order book). Fixed for the run;
|
||||
discovery is M3.
|
||||
|
||||
Output fields: a Delta's `backtest_timestamp` and a Snapshot's `received_at`
|
||||
are both the receive time. A minute Snapshot carries the receive time of the
|
||||
last frame the book thread applied, not the wall clock: the book is exact as of
|
||||
that frame, while frames still queued aren't in it yet.
|
||||
|
||||
No `NormalizedEvent` yet: Polymarket's messages parse into Polymarket types,
|
||||
the book code knows nothing of Polymarket, and the hash check is Polymarket
|
||||
code. The shared seam is extracted in M6 from two exchanges.
|
||||
|
||||
## Loss detection and resync
|
||||
|
||||
Checked live on 6 symbols for 2 minutes (14 books, 182 changes, all matched):
|
||||
the book hash is the SHA-1 of the compact JSON
|
||||
|
||||
```
|
||||
{"market","asset_id","timestamp","hash":"","bids","asks","min_order_size","tick_size","neg_risk","last_trade_price"}
|
||||
```
|
||||
|
||||
in that key order, with bids ascending and asks descending by price, prices and
|
||||
sizes as canonical decimals (true of every price and size in 40 books checked),
|
||||
and, for a change, the change message's `timestamp`. Full books sent
|
||||
mid-stream omit `tick_size` and `last_trade_price`: the last known values are
|
||||
used.
|
||||
|
||||
Two things the 2-minute check didn't show, found in the 90 s test recording
|
||||
(40 symbols, 3,850 changes; `docs/notes.md`):
|
||||
|
||||
- **Batches**: one update can arrive split over several frames, all carrying
|
||||
the hash of the book after the whole batch. A change's hash is therefore
|
||||
checked when its batch is over: right away if it matches, else when the
|
||||
symbol's next change carries a different hash (before applying it).
|
||||
- **The last trade price can't be known in time**: trades change it, the hash
|
||||
picks up the new value up to ~680 ms before the `last_trade_price` event,
|
||||
and that event gives it, its complement (1 − p), or neither. So after a
|
||||
trade the check fails although nothing was lost; only a fresh book (which
|
||||
carries the last trade price) tells the two apart.
|
||||
|
||||
The rules:
|
||||
|
||||
1. **A complete full book checks itself** (one that carries
|
||||
`last_trade_price`: on subscribe and after a resync): hash it as received.
|
||||
A mismatch means our formula or metadata is wrong for that symbol, not a
|
||||
loss: log it loudly, mark the symbol `Unverifiable`, and keep writing its
|
||||
Deltas, unchecked. No resync, which would loop forever.
|
||||
2. **A failed check asks for a fresh book** (a change, after its batch, or a
|
||||
mid-stream book, whose last trade price may be stale): the symbol goes
|
||||
`Resyncing`, the book thread asks the socket thread to unsubscribe and
|
||||
subscribe it again (tried live: the fresh book arrives at once), and its
|
||||
changes keep being applied and written, unchecked, until that book arrives.
|
||||
A symbol still `Resyncing` at a wall-clock minute is asked for again, so a
|
||||
lost answer doesn't leave it unchecked for the rest of the run. A full book
|
||||
that can't be read is a protocol error: the symbol resyncs too (it keeps
|
||||
`AwaitingBook` if it had no book yet).
|
||||
3. **A full book replaces ours** (on subscribe, mid-stream after trades, after
|
||||
a resync or a reconnect). It is written as a Snapshot if it is the symbol's
|
||||
first, or if its levels differ from ours: that is a **loss**, logged and
|
||||
counted. A fresh book with the same levels (the usual case after a trade)
|
||||
writes nothing. Two timings of a resync's fresh book need care:
|
||||
- **Behind our stream**: it can lag the changes already sent by a few ms
|
||||
(live: 14 of 14 differing books in 2 minutes), and its timestamp or last
|
||||
trade price can be newer than its levels, so its hash may match none of
|
||||
our changes. Checked in order, against the symbol's last 64 changes:
|
||||
1. if it equals our book as it was after one of them (ours rewound,
|
||||
each change undone back to the quantity it replaced), it is behind:
|
||||
ours is kept, its tick size and last trade price are taken, nothing
|
||||
is written. That past state of ours was the exchange's own, and the
|
||||
changes since are what it sent us. (Comparing the book *plus our
|
||||
later changes* with ours instead would be unsound: a change we lost
|
||||
at a price we had changed before is overwritten by that earlier
|
||||
change, and the wrong book passes.)
|
||||
2. else, if its hash is that of one of our changes (the latest
|
||||
included), it is the exchange's state after that change and ours is
|
||||
wrong (a loss): the changes received since are applied to it, and it
|
||||
replaces ours.
|
||||
- **Ahead of our stream (a resync gap)**: between the server handling our
|
||||
`unsubscribe` and our `subscribe`, it sends no changes for that symbol,
|
||||
so the fresh book holds changes we never got. A fresh book answering
|
||||
our resync that isn't behind, whose levels differ and whose exchange
|
||||
time is at or after our last change, is counted as a **resync gap**, not
|
||||
a loss: its Snapshot corrects the
|
||||
state, the Deltas inside the gap are lost. M4's redundant sockets can
|
||||
close it.
|
||||
4. **`tick_size_change`** resyncs the symbol: the fresh book carries the new
|
||||
tick size. Its wire shape was never observed (the docs now show an SDK's
|
||||
wrapped form), and this needs only `event_type` and `asset_id`.
|
||||
5. **`last_trade_price` events are ignored**: the hash takes the new price
|
||||
before the event and not always the event's value; the fresh book after
|
||||
the failed check carries the right one. Trades aren't in the schema.
|
||||
|
||||
`PONG` and `NO NEW ASSETS` (the server's answer to a subscribe for a symbol
|
||||
already subscribed) are skipped, as are unknown event types; any other frame
|
||||
that isn't readable JSON is logged and skipped.
|
||||
|
||||
## Output, shutdown and reconnect
|
||||
|
||||
- **Destination**: the program's only argument is the output path, default
|
||||
`/dev/stdout`. Logs go to stderr. No rotation: one output per run.
|
||||
- **Minute Snapshots**: every symbol with a book (not `AwaitingBook`), at each
|
||||
wall-clock minute.
|
||||
- **SIGINT**: `ctrlc` sets a flag; the socket thread sees it within 200 ms
|
||||
when connected, within 10 s while connecting (the TCP connect and the TLS
|
||||
handshake have a 10 s timeout), and 1 s while waiting to reconnect. It
|
||||
closes the websocket and drops its channel; the book thread drains and drops
|
||||
its own; the writer drains, flushes the last chunk and closes the file. Each
|
||||
stage stops because the previous one did.
|
||||
- **Reconnect**: when the server closes the socket or it errors, the socket
|
||||
thread waits 1 s, connects again and subscribes every symbol; the fresh
|
||||
books then go through rule 3. Seen once in about 8 minutes of test
|
||||
connections: a close with "server shutting down". Ceiling: a symbol that
|
||||
was `Resyncing` when the connection dropped has its reconnect book counted
|
||||
as a resync gap rather than a loss (the state and output are right either
|
||||
way; only the count is mislabelled, next to the logged reconnect).
|
||||
- **Crash ceiling**: `AlignedSbeWriter` keeps the current 64 MiB chunk in memory
|
||||
and writes it only when full (`src/util/aligned_sbe_writer.rs:48`) or on
|
||||
`flush`, which pads it to 64 MiB with zeros (`:68-70`). A hard crash loses
|
||||
the unwritten chunk, hours of data at 40 symbols; flushing every minute would
|
||||
write ~90 GB a day of padding. Accepted for M2; the brief lists "we go down
|
||||
unexpectedly" under extension 1, so M4 takes it up.
|
||||
|
||||
## Libraries
|
||||
|
||||
New: `serde_json` (parsing), `ureq` 3 (Gamma; tried: it reaches Gamma and the
|
||||
CLOB, and its TLS uses `ring`, already locked, with no `aws-lc-rs`), `sha1` 0.11
|
||||
(RustCrypto), `ctrlc` 3.
|
||||
|
||||
## Decisions
|
||||
|
||||
- **Loss detection by the book hash, after every change.** It identifies the
|
||||
full book state, so a check after each change catches any loss at once.
|
||||
*Rejected:* comparing best bid / ask, which misses losses deeper in the book.
|
||||
- **Resync by unsubscribe + subscribe on the open socket.** The fresh book
|
||||
arrives at once, on the stream we're already reading. *Rejected:* REST
|
||||
`/book` (a second path whose timing relative to the stream is unknown).
|
||||
- **A failed check asks for a fresh book but keeps applying changes; a loss is
|
||||
a fresh book that differs** (supersedes "a mismatch is a loss, changes are
|
||||
dropped until the fresh book", because the recording showed checks failing
|
||||
after trades with nothing lost: the hash includes a last trade price we
|
||||
can't know in time). After a trade it costs one book on the socket and
|
||||
nothing in the output; after a real loss the fresh book's Snapshot corrects
|
||||
the output. *Rejected:* guessing the last trade price from the trade event
|
||||
(its value, its complement, the batch's prices), a heuristic over
|
||||
undocumented behaviour that already missed a case in 90 s; dropping changes
|
||||
and writing a Snapshot on every trade; checking only best bid / ask, which
|
||||
misses deeper losses.
|
||||
- **A full book whose hash we saw on a recent change is brought forward with
|
||||
the changes received since, then compared** (supersedes "a book whose
|
||||
levels differ is a loss", because live, fresh books lagged our stream by a
|
||||
few ms; and supersedes "keep ours", agreed in between, because our book may
|
||||
be the wrong one). The hash identifies the full state, so the test is
|
||||
exact. *Rejected:* comparing exchange timestamps (ties at the same
|
||||
millisecond were common); REST `/book` (may lag the same way, and is a
|
||||
second path); keeping our book.
|
||||
- **"Behind" is recognised by levels first, by hash second** (supersedes the
|
||||
hash match alone, because live, fresh books were behind by levels while
|
||||
carrying a newer timestamp or last trade price: both losses of a 15-minute
|
||||
run and 14 books counted as resync gaps, which then replaced our newer book
|
||||
with a stale one). The hash match stays second: when our book is the wrong
|
||||
one, it still brings the fresh book forward to the right state at once.
|
||||
The level test rewinds ours (supersedes "the book plus our later changes
|
||||
equals ours", found unsound at the close review: a lost change at a price
|
||||
we had changed before is masked). *Rejected:* levels only (a loss would
|
||||
first write the stale book); hashes only (the misclassifications above).
|
||||
- **A requested resync is asked again at each minute while unanswered; the
|
||||
connect and TLS handshake time out after 10 s** (both found at the close
|
||||
review: a lost answer left a symbol unchecked for the rest of the run, and
|
||||
SIGINT could hang on a stalled connect). *Rejected:* a per-symbol resync
|
||||
timer (more state for the same effect at this scale).
|
||||
- **A resync gap is counted apart from losses and accepted in M2.** Our own
|
||||
unsubscribe / subscribe opens a few ms in which the server sends us no
|
||||
changes for the symbol; the fresh book's Snapshot keeps the state right.
|
||||
*Rejected:* resyncing through a short-lived second connection (M4's
|
||||
machinery, and its book could be ahead of our stream, a case not handled);
|
||||
fewer resyncs by guessing the last trade price (rejected above).
|
||||
- **A change's hash is checked when its batch is over.** One update can span
|
||||
frames that all carry its final hash. *Rejected:* checking after each
|
||||
change, which fails mid-batch.
|
||||
- **A self-check on full books, `Unverifiable` on failure.** Separates "our
|
||||
formula is wrong" from "we lost a message". *Rejected:* resyncing on any
|
||||
mismatch, which loops forever on a formula error.
|
||||
- **Levels as fixed-point integers, other hash inputs as exact strings.** The
|
||||
output needs integers, and integer keys make `"0.5"` and `"0.50"` one level;
|
||||
`last_trade_price` is not canonical (`"0.080"` seen), so it stays a string.
|
||||
*Rejected:* keying levels by the exchange's strings.
|
||||
- **Both tokens of a market are symbols.** The brief asks for a book per symbol,
|
||||
and consumers shouldn't need to know Polymarket mirrors them. *Rejected:* one
|
||||
token per market (half the output).
|
||||
- **Three threads, socket / book / writer.** The socket thread only reads and
|
||||
stamps; the writer survives a book thread crash and still flushes, and its
|
||||
64 MiB writes don't pause the book thread; M3 needs it anyway. *Rejected:*
|
||||
two threads (book thread owning the writer), and four (no `split` in
|
||||
tungstenite without a mutex).
|
||||
- **Unbounded channels.** Blocking stalls the socket read and hurts time
|
||||
accuracy; dropping breaks completeness. Their ceiling (memory, if a consumer
|
||||
stays behind) is named in a comment, and the book thread's minute log
|
||||
reports the most frames that waited for it that minute (supersedes "logs the
|
||||
queue length past 10,000", whose check could only run when the queue was
|
||||
empty). Switching to bounded is a constructor change. *Rejected:* bounded,
|
||||
for now.
|
||||
- **Subset from Gamma at startup.** A hardcoded list goes stale in days (M1's
|
||||
did), and M3 needs the HTTP client anyway. *Rejected:* hardcoded token ids.
|
||||
- **Simple reconnect in M2.** A collector that dies at each server restart
|
||||
isn't one, and rule 3 already handles the fresh books. *Rejected:* exit on
|
||||
disconnect until M4, which keeps redundancy.
|
||||
- **Output to a path, default stdout, no rotation.** The brief says stdout;
|
||||
rotation can't apply to it. *Rejected:* rotating files by size or time.
|
||||
- **`sha1` (RustCrypto).** The usual choice, and faster: 1.8 µs per book
|
||||
against 4.6 µs for `sha1_smol` (measured, ~5 KB). *Rejected:* `sha1_smol`,
|
||||
`ring`'s SHA-1.
|
||||
|
||||
## Tests
|
||||
|
||||
No network in tests. A fixture of a few minutes of real frames for the subset,
|
||||
each with its receive time, plus the Gamma metadata the hash needs, recorded
|
||||
once by a throwaway script and committed (a few MB at most; its size is
|
||||
checked when recorded).
|
||||
|
||||
- **Replay**: the fixture replays with no loss and no `Unverifiable` symbol,
|
||||
split batches and trades included.
|
||||
- **Induced loss**: the replay with one frame dropped: a resync of that
|
||||
symbol is requested where the full replay requests none, and a fresh book
|
||||
that differs is written as a Snapshot and counted as a loss.
|
||||
- **Self-check**: a full book with wrong metadata marks the symbol
|
||||
`Unverifiable` and requests no resync.
|
||||
- **Reconciliation**: book and writer threads run on the fixture into a temp
|
||||
file (`std::env::temp_dir()`), read back with `AlignedSbeReader`; per symbol,
|
||||
the Deltas applied to a Snapshot equal the next Snapshot.
|
||||
- **Parsing**: fixed-point edges (`"0"`, integers, 6 decimals, more than 6,
|
||||
empty sides), non-JSON frames.
|
||||
|
||||
## Not in this sprint
|
||||
|
||||
- Discovery, the full symbol set, several sockets (M3).
|
||||
- Redundant sockets, and losing nothing during a reconnect or a resync gap
|
||||
(M4).
|
||||
- Activity monitoring, socket recycling (M5).
|
||||
- Surviving a hard crash without losing the in-memory chunk (M4).
|
||||
- Kalshi, the generic core (M6); latency work (M7).
|
||||
|
||||
## Done when
|
||||
|
||||
- `scripts/ci.sh` is green.
|
||||
- A 10-minute live run on the top 20 markets, ended by SIGINT, leaves an output
|
||||
that reads back with `AlignedSbeReader` and passes the reconciliation check.
|
||||
- That run's log shows no `Unverifiable` symbol, and every loss explained
|
||||
(e.g. by a reconnect); resync gaps are counted. The reconciliation's
|
||||
mismatches are exactly those losses and gaps.
|
||||
- `docs/notes.md` corrected (asks descending) and completed (the hash formula,
|
||||
resync on the open socket, `NO NEW ASSETS`, mid-stream books' missing fields,
|
||||
the server's restart close); `docs/project.md` records loss detection, output
|
||||
destination and rotation as settled.
|
||||
|
||||
## Outcome
|
||||
|
||||
**Built.** Three threads joined by unbounded channels:
|
||||
- **Socket thread:** connects with a 10 s timeout, stamps each frame's receive
|
||||
time right after the read, sends `PING`, resync requests and the close on
|
||||
SIGINT, and reconnects after 1 s.
|
||||
- **Book thread** (`src/polymarket/feed.rs`): parses, folds into a `Book` per
|
||||
symbol, checks Polymarket's book hash once each batch is over, resyncs on a
|
||||
failed check, and writes Deltas and Snapshots (on subscribe, at each
|
||||
wall-clock minute, when a fresh book changes our state).
|
||||
- **Writer thread:** owns the template's `AlignedSbeWriter`.
|
||||
|
||||
Also: a `reconcile` binary that checks an output against itself; 16 tests on a
|
||||
90 s recording of the live feed (replay, induced loss, self-check, books
|
||||
behind and ahead of our stream, the reconciliation through the real book and
|
||||
writer threads, parsing). The final 10-minute live run on the top 20 markets
|
||||
(40 symbols), ended by SIGINT: 97,763 changes verified, 266 resyncs, 29 fresh
|
||||
books behind ours, 4 resync gaps, 0 losses, no `Unverifiable` symbol. Its 4
|
||||
reconciliation mismatches are exactly the 4 logged gaps, and at most 20
|
||||
frames waited for the book thread in any minute. CI is green.
|
||||
|
||||
**Diverged from the plan, and why.** The plan's loss detection rested on a
|
||||
2-minute check of 6 symbols. Larger recordings and live runs showed the feed
|
||||
breaks that model in four ways, each settled with the user and recorded in
|
||||
Decisions:
|
||||
1. **Batches.** One update can span several frames carrying the same hash, so
|
||||
a hash is checked once its batch is over.
|
||||
2. **The last trade price.** It is a market-wide hash input that changes
|
||||
before its event, sometimes as the event's complement. A failed check
|
||||
therefore asks for a fresh book and keeps writing changes ("soft resync"),
|
||||
rather than calling it a loss.
|
||||
3. **Fresh books can lag our stream.** They are recognised by levels (ours
|
||||
rewound), then by hash.
|
||||
4. **Fresh books can lead our stream.** Our own unsubscribe/subscribe hides a
|
||||
few ms of changes: a **resync gap**, counted apart from losses and
|
||||
accepted for M2.
|
||||
|
||||
Two more changes came out of the close review:
|
||||
- **A soundness hole.** "The book plus our later changes" masked a change lost
|
||||
at a level we had changed before, so the level test now rewinds ours instead.
|
||||
- **Resyncs no longer wait forever.** An unanswered or unreadable fresh book is
|
||||
asked for again at each minute.
|
||||
|
||||
Smaller changes:
|
||||
- `book.rs` moved into the library, so `reconcile` uses `Book`.
|
||||
- The queue warning, which could only fire when the queue was empty, became a
|
||||
per-minute peak in the stats line.
|
||||
- `last_trade_price` events are ignored, as their value isn't reliable.
|
||||
|
||||
**Left out, and where it went:**
|
||||
- **Resync gaps and the hard-crash chunk loss:** M4.
|
||||
- **Scale** (40 symbols here, against ≥100,000 live markets, ≥13,000 traded in
|
||||
24 h): M3's opening questions in the roadmap.
|
||||
- **Counting a reconnect's differing books as losses rather than gaps,** and a
|
||||
`reconcile` that reads the log: Possible features.
|
||||
- **The rest, recorded as ceilings:**
|
||||
- a book that stays unreadable is asked for once per answer;
|
||||
- the name lookup has no timeout;
|
||||
- the read loop is written twice in `socket.rs`;
|
||||
- `open_batch` is a pair of Strings.
|
||||
|
||||
### Retrospective
|
||||
|
||||
- **Went well:**
|
||||
- Checking every claim against live data: the hash formula and each of the
|
||||
four surprises above were found by recordings and runs, not assumed.
|
||||
- Mutation checks (disable the rule, watch its test fail) on every new
|
||||
loss-detection test.
|
||||
- Doc-first changes, each settled with the user before code.
|
||||
- The two-axis close review found a real soundness hole the tests missed.
|
||||
- **Went wrong, or took too long:**
|
||||
- The sprint was agreed on a 6-symbol, 2-minute sample, so its central rule
|
||||
was rewritten four times during implementation.
|
||||
- My estimate of resyncs per trade was off (28 estimated, 8 measured).
|
||||
- The queue warning sat where it could never fire.
|
||||
- The scale of the real target (100,000+ markets against 40 symbols) came up
|
||||
only when the user asked.
|
||||
- **Tested enough?** The rules are tested on a real recording and on built
|
||||
cases. Not covered by tests:
|
||||
- the reconnect path (seen live once, before this code);
|
||||
- SIGINT during a connect;
|
||||
- the book thread's wall-clock tick when no frame comes;
|
||||
- the minute re-ask and the unreadable-book path;
|
||||
- `tick_size_change` (never observed);
|
||||
- a live `Unverifiable` symbol.
|
||||
- **Next time:**
|
||||
- During planning, record a realistic sample of the external feed and keep
|
||||
it as the fixture.
|
||||
- Count the target scale before fixing a test subset.
|
||||
- Have the close review hunt for adversarial cases against each correctness
|
||||
rule.
|
||||
Reference in new issue
Block a user