feat(polymarket): subscription monitoring and connection recycling

The exchange can fail without telling us, and first one wins hides a
subscription gone quiet on one of a symbol's two connections.

- A connection that receives nothing for 30 s, not even the keepalive's
  answer, reconnects.
- Each book thread counts a symbol's changes per connection every minute:
  a subscription with none against three or more on the other is
  resubscribed, and moved to another connection if it stays quiet.
- The orchestrator recycles one connection at a time, each about hourly,
  and never while another connection is down or just back.
- A reconnect's opening book of a symbol unchanged for 10 s is judged like
  a resync's answer: one newer than ours that differs shows a symbol quiet
  on both connections, written as a Snapshot and logged as a loss.
This commit is contained in:
pascal committed 2026-11-11 17:42:27 +00:00
1 parent 02747b882f
commit 480ee3ad2d
10 files changed
+1346 -102

No files matched your search

+9 -5
View File
@@ -18,13 +18,14 @@ outside this repo.
loss detection, message shapes). Read before touching exchange parsing.
- `GLOSSARY.md`: the terms (symbol, connection, load, first one wins,
duplicate, held, rescued, follow, resync, loss, subscription gap, split,
Snapshot, Delta).
quiet, resubscribe, silent, recycle, Snapshot, Delta).
- `src/polymarket/orchestrator.rs`: what is tracked, which connections carry
each market, and their balancing; `src/polymarket/gamma.rs`: the fetchers
each market, their balancing and their recycling; `src/polymarket/gamma.rs`: the fetchers
(discovery); `src/polymarket/socket.rs`: a connection (read, stamp, decode,
route); `src/polymarket/wire.rs`: the exchange's messages;
`src/polymarket/feed.rs`: a book thread and its loss rules (the subtle
part); `src/book.rs`: the book and fixed point; `src/reconcile.rs` +
route; reconnects when dropped, silent or recycled);
`src/polymarket/wire.rs`: the exchange's messages;
`src/polymarket/feed.rs`: a book thread, its loss rules (the subtle
part) and its watch for quiet subscriptions; `src/book.rs`: the book and fixed point; `src/reconcile.rs` +
`src/bin/reconcile.rs`: the output check, reading the log's cause lines
(`src/cause.rs`). `tests/fixtures/`: recordings of the live feed, on one and two
connections, that the tests replay.
@@ -73,6 +74,9 @@ and let the user settle it. Claude owns the data model.
"capacity scan"` matches the waiting shell's own command line and never
ends, and `pkill -f PATTERN` can kill the shell running it. Take the PID
(`pgrep -f '^./target/release/...'`, anchored) and loop on `kill -0 $pid`.
- **No `/usr/bin/time` on the server**: for a run's peak memory and thread
count, read `VmHWM` and `Threads` from `/proc/<pid>/status` while it runs
(sprint 05's run script sampled them every 30 s).
- **The server is shared and gets overloaded** (load up to 92 on 12 cores,
2026-11-11): a starved collector falls behind and gets closed with 1013.
Check `cat /proc/loadavg` before a live run whose numbers are compared, and