feat(polymarket): subscription monitoring and connection recycling

The exchange can fail without telling us, and first one wins hides a
subscription gone quiet on one of a symbol's two connections.

- A connection that receives nothing for 30 s, not even the keepalive's
  answer, reconnects.
- Each book thread counts a symbol's changes per connection every minute:
  a subscription with none against three or more on the other is
  resubscribed, and moved to another connection if it stays quiet.
- The orchestrator recycles one connection at a time, each about hourly,
  and never while another connection is down or just back.
- A reconnect's opening book of a symbol unchanged for 10 s is judged like
  a resync's answer: one newer than ours that differs shows a symbol quiet
  on both connections, written as a Snapshot and logged as a loss.
This commit is contained in:
pascal committed 2026-11-11 17:42:27 +00:00
1 parent 02747b882f
commit 480ee3ad2d
10 files changed
+1346 -102

No files matched your search

+21
View File
@@ -173,6 +173,27 @@ From a 60 s run of the M1 spike on one market, both tokens subscribed, and the
(0 of 660,084 changes on each of the two):
one older than the last applied is a late copy.
## Quiet subscriptions and reconnects (observed 2026-11-11)
From a 25-minute live run on every live market, two connections each, one
connection recycled every 60 s (sprint 05).
- **Quiet subscriptions are rare**: in 21 minutes, one market once had 3 or
more changes on one connection and none on the other; none had 10. Both
tokens of a market go one-sided together.
- **A reconnect's opening book can lag by minutes**: for symbols unchanged
for 10 s or more, 514 of 19,379 differed from ours, 512 of them equal to
one of our earlier states (1 to 4 levels behind, some over 10 minutes),
none with a newer timestamp. Books of symbols never changed all matched
(42,096). Under 10 s, some ran ahead of our stream.
- **With the monitoring on** (30 minutes, 107 recycles): 26 subscriptions
resubscribed for being quiet, none quiet again the next minute; of 335,488
reconnect books of unchanged symbols, 7,240 fresh books in all were behind
ours; no connection silent for 30 s.
- **A reconnect takes ~1.2 s** from close to up (1 s of it our wait); the
opening books of ~4,500 tokens come 0.3 s later on most connections, 20 s
on one; 2 of 4,490 never came on another.
## Gamma, the market listing (observed 2026-11-04)
- **Size:** 238,308 markets `active` and not `closed`; 238,201 of them with