feat(polymarket): subscription monitoring and connection recycling

The exchange can fail without telling us, and first one wins hides a
subscription gone quiet on one of a symbol's two connections.

- A connection that receives nothing for 30 s, not even the keepalive's
  answer, reconnects.
- Each book thread counts a symbol's changes per connection every minute:
  a subscription with none against three or more on the other is
  resubscribed, and moved to another connection if it stays quiet.
- The orchestrator recycles one connection at a time, each about hourly,
  and never while another connection is down or just back.
- A reconnect's opening book of a symbol unchanged for 10 s is judged like
  a resync's answer: one newer than ours that differs shows a symbol quiet
  on both connections, written as a Snapshot and logged as a loss.
This commit is contained in:
pascal committed 2026-11-11 17:42:27 +00:00
1 parent 02747b882f
commit 480ee3ad2d
10 files changed
+1346 -102

No files matched your search

+8 -1
View File
@@ -55,12 +55,16 @@ abstraction is *extracted* once a second exchange exists, not designed up front.
Result: losses a quarter lower in each pair of alternating runs; losses and
gaps together lower in one pair, even in the other (sprint 04's Outcome).
5. **Extension 2 — subscription monitoring** — *not started*
5. **Extension 2 — subscription monitoring** — *done (2026-11-11)*
Per-symbol activity monitoring (data points per second/minute); unsubscribe +
resubscribe symbols that go too quiet (with two connections, first one wins
hides a subscription gone quiet on one of them: it has to be looked for); periodic full-socket recycle
(disconnect → reconnect → resubscribe) to shed cumulative exchange-side bugs.
Sprint: [../roadmap/05. subscription monitoring.md](../roadmap/05.%20subscription%20monitoring.md).
*Ends:* monitoring + recycling running.
Result: a 30-minute run recycled 107 connections with losses no higher
than sprint 04's; quiet subscriptions are rare (26 resubscribed, none
quiet again) and no connection fell silent (sprint 05's Outcome).
6. **Extension 3 — generic core + Kalshi adapter** — *not started*
Extract the exchange-agnostic core (the `NormalizedEvent` seam) from the
@@ -115,6 +119,9 @@ abstraction is *extracted* once a second exchange exists, not designed up front.
book, when the first one's chain breaks before any change verified.
- An exact rescued count: a change remembered past the window of 128 until
its copies are counted (sprint 04 reports an upper bound).
- Telling a change still in flight from a symbol quiet on both connections,
when a reconnect's book is newer than ours (`maybe`: sprint 05 logs both
as a loss; the Snapshot is right either way).
- Fewer Snapshots for unchanged books (the brief asks for every symbol every
minute: ~130 MB a minute at full coverage).
- Re-reading a market's `orderMinSize` / `negRisk` if they can change.