feat(polymarket): subscription monitoring and connection recycling

The exchange can fail without telling us, and first one wins hides a
subscription gone quiet on one of a symbol's two connections.

- A connection that receives nothing for 30 s, not even the keepalive's
  answer, reconnects.
- Each book thread counts a symbol's changes per connection every minute:
  a subscription with none against three or more on the other is
  resubscribed, and moved to another connection if it stays quiet.
- The orchestrator recycles one connection at a time, each about hourly,
  and never while another connection is down or just back.
- A reconnect's opening book of a symbol unchanged for 10 s is judged like
  a resync's answer: one newer than ours that differs shows a symbol quiet
  on both connections, written as a Snapshot and logged as a loss.
This commit is contained in:
pascal committed 2026-11-11 17:42:27 +00:00
1 parent 02747b882f
commit 480ee3ad2d
10 files changed
+1346 -102

No files matched your search

+21
View File
@@ -173,6 +173,27 @@ From a 60 s run of the M1 spike on one market, both tokens subscribed, and the
(0 of 660,084 changes on each of the two):
one older than the last applied is a late copy.
## Quiet subscriptions and reconnects (observed 2026-11-11)
From a 25-minute live run on every live market, two connections each, one
connection recycled every 60 s (sprint 05).
- **Quiet subscriptions are rare**: in 21 minutes, one market once had 3 or
more changes on one connection and none on the other; none had 10. Both
tokens of a market go one-sided together.
- **A reconnect's opening book can lag by minutes**: for symbols unchanged
for 10 s or more, 514 of 19,379 differed from ours, 512 of them equal to
one of our earlier states (1 to 4 levels behind, some over 10 minutes),
none with a newer timestamp. Books of symbols never changed all matched
(42,096). Under 10 s, some ran ahead of our stream.
- **With the monitoring on** (30 minutes, 107 recycles): 26 subscriptions
resubscribed for being quiet, none quiet again the next minute; of 335,488
reconnect books of unchanged symbols, 7,240 fresh books in all were behind
ours; no connection silent for 30 s.
- **A reconnect takes ~1.2 s** from close to up (1 s of it our wait); the
opening books of ~4,500 tokens come 0.3 s later on most connections, 20 s
on one; 2 of 4,490 never came on another.
## Gamma, the market listing (observed 2026-11-04)
- **Size:** 238,308 markets `active` and not `closed`; 238,201 of them with
+10 -8
View File
@@ -135,14 +135,15 @@ and drop from the global workflow for a job-challenge repo.
## Pipeline
The primary thing we debate sprint by sprint (this project's equivalent of the
schema debate). As settled in sprint 04:
schema debate). As settled in sprints 04 and 05:
```
rescan fetcher ──findings──▶ ┌──────────────┐ ◀──findings── recent fetcher
│ orchestrator │ ◀── 1013 closes ── connections
└──────┬───▲───┘
│ orchestrator │ ◀── up, down, 1013 ─── connections
└──────┬───▲───┘ ──── recycle ────────▶ connections
Add / Move / Remove (a market and │ │ load reports, each minute
its connections), to its thread │ │ (bytes/s per market)
its connections), to its thread │ │ (bytes/s per market); markets
│ │ quiet twice on a connection
┌──────▼───┴───┐
connection 0..N ──events───▶ │ book threads │ ──Snapshot | Delta──▶ writer
(read, stamp, parse, route) │ 0..15 │
@@ -151,15 +152,16 @@ schema debate). As settled in sprint 04:
```
- The **orchestrator** owns what is tracked (each market's two connections,
the timers) and decides: placing, moving, splitting. It never blocks on
HTTP.
the timers) and decides: placing, moving, splitting, and recycling one
connection at a time, about hourly. It never blocks on HTTP.
- The two **fetchers** only page Gamma on request and stream what they read.
- A **connection** reads, stamps the receive time, parses, and routes each
event to the book thread of its market. Every symbol is on two connections,
chosen independently.
chosen independently. One silent for 30 s reconnects.
- A **book thread** owns its markets' books and applies first one wins: a
change is a duplicate, the next one, or held until it fits (fold, check,
resync, Snapshots as before).
resync, Snapshots as before). It resubscribes a subscription gone quiet on
one connection, and judges a reconnect's books of unchanged symbols.
- One **writer** owns the output.
Settled: OS threads, shared-nothing, no mutexes
+8 -1
View File
@@ -55,12 +55,16 @@ abstraction is *extracted* once a second exchange exists, not designed up front.
Result: losses a quarter lower in each pair of alternating runs; losses and
gaps together lower in one pair, even in the other (sprint 04's Outcome).
5. **Extension 2 — subscription monitoring** — *not started*
5. **Extension 2 — subscription monitoring** — *done (2026-11-11)*
Per-symbol activity monitoring (data points per second/minute); unsubscribe +
resubscribe symbols that go too quiet (with two connections, first one wins
hides a subscription gone quiet on one of them: it has to be looked for); periodic full-socket recycle
(disconnect → reconnect → resubscribe) to shed cumulative exchange-side bugs.
Sprint: [../roadmap/05. subscription monitoring.md](../roadmap/05.%20subscription%20monitoring.md).
*Ends:* monitoring + recycling running.
Result: a 30-minute run recycled 107 connections with losses no higher
than sprint 04's; quiet subscriptions are rare (26 resubscribed, none
quiet again) and no connection fell silent (sprint 05's Outcome).
6. **Extension 3 — generic core + Kalshi adapter** — *not started*
Extract the exchange-agnostic core (the `NormalizedEvent` seam) from the
@@ -115,6 +119,9 @@ abstraction is *extracted* once a second exchange exists, not designed up front.
book, when the first one's chain breaks before any change verified.
- An exact rescued count: a change remembered past the window of 128 until
its copies are counted (sprint 04 reports an upper bound).
- Telling a change still in flight from a symbol quiet on both connections,
when a reconnect's book is newer than ours (`maybe`: sprint 05 logs both
as a loss; the Snapshot is right either way).
- Fewer Snapshots for unchanged books (the brief asks for every symbol every
minute: ~130 MB a minute at full coverage).
- Re-reading a market's `orderMinSize` / `negRisk` if they can change.