Archived
feat(polymarket): subscription monitoring and connection recycling
The exchange can fail without telling us, and first one wins hides a subscription gone quiet on one of a symbol's two connections. - A connection that receives nothing for 30 s, not even the keepalive's answer, reconnects. - Each book thread counts a symbol's changes per connection every minute: a subscription with none against three or more on the other is resubscribed, and moved to another connection if it stays quiet. - The orchestrator recycles one connection at a time, each about hourly, and never while another connection is down or just back. - A reconnect's opening book of a symbol unchanged for 10 s is judged like a resync's answer: one newer than ours that differs shows a symbol quiet on both connections, written as a Snapshot and logged as a loss.
This commit is contained in:
10 files changed
+1346
-102
No files matched your search
@@ -173,6 +173,27 @@ From a 60 s run of the M1 spike on one market, both tokens subscribed, and the
|
||||
(0 of 660,084 changes on each of the two):
|
||||
one older than the last applied is a late copy.
|
||||
|
||||
## Quiet subscriptions and reconnects (observed 2026-11-11)
|
||||
|
||||
From a 25-minute live run on every live market, two connections each, one
|
||||
connection recycled every 60 s (sprint 05).
|
||||
|
||||
- **Quiet subscriptions are rare**: in 21 minutes, one market once had 3 or
|
||||
more changes on one connection and none on the other; none had 10. Both
|
||||
tokens of a market go one-sided together.
|
||||
- **A reconnect's opening book can lag by minutes**: for symbols unchanged
|
||||
for 10 s or more, 514 of 19,379 differed from ours, 512 of them equal to
|
||||
one of our earlier states (1 to 4 levels behind, some over 10 minutes),
|
||||
none with a newer timestamp. Books of symbols never changed all matched
|
||||
(42,096). Under 10 s, some ran ahead of our stream.
|
||||
- **With the monitoring on** (30 minutes, 107 recycles): 26 subscriptions
|
||||
resubscribed for being quiet, none quiet again the next minute; of 335,488
|
||||
reconnect books of unchanged symbols, 7,240 fresh books in all were behind
|
||||
ours; no connection silent for 30 s.
|
||||
- **A reconnect takes ~1.2 s** from close to up (1 s of it our wait); the
|
||||
opening books of ~4,500 tokens come 0.3 s later on most connections, 20 s
|
||||
on one; 2 of 4,490 never came on another.
|
||||
|
||||
## Gamma, the market listing (observed 2026-11-04)
|
||||
|
||||
- **Size:** 238,308 markets `active` and not `closed`; 238,201 of them with
|
||||
|
||||
+10
-8
@@ -135,14 +135,15 @@ and drop from the global workflow for a job-challenge repo.
|
||||
## Pipeline
|
||||
|
||||
The primary thing we debate sprint by sprint (this project's equivalent of the
|
||||
schema debate). As settled in sprint 04:
|
||||
schema debate). As settled in sprints 04 and 05:
|
||||
|
||||
```
|
||||
rescan fetcher ──findings──▶ ┌──────────────┐ ◀──findings── recent fetcher
|
||||
│ orchestrator │ ◀── 1013 closes ── connections
|
||||
└──────┬───▲───┘
|
||||
│ orchestrator │ ◀── up, down, 1013 ─── connections
|
||||
└──────┬───▲───┘ ──── recycle ────────▶ connections
|
||||
Add / Move / Remove (a market and │ │ load reports, each minute
|
||||
its connections), to its thread │ │ (bytes/s per market)
|
||||
its connections), to its thread │ │ (bytes/s per market); markets
|
||||
│ │ quiet twice on a connection
|
||||
┌──────▼───┴───┐
|
||||
connection 0..N ──events───▶ │ book threads │ ──Snapshot | Delta──▶ writer
|
||||
(read, stamp, parse, route) │ 0..15 │
|
||||
@@ -151,15 +152,16 @@ schema debate). As settled in sprint 04:
|
||||
```
|
||||
|
||||
- The **orchestrator** owns what is tracked (each market's two connections,
|
||||
the timers) and decides: placing, moving, splitting. It never blocks on
|
||||
HTTP.
|
||||
the timers) and decides: placing, moving, splitting, and recycling one
|
||||
connection at a time, about hourly. It never blocks on HTTP.
|
||||
- The two **fetchers** only page Gamma on request and stream what they read.
|
||||
- A **connection** reads, stamps the receive time, parses, and routes each
|
||||
event to the book thread of its market. Every symbol is on two connections,
|
||||
chosen independently.
|
||||
chosen independently. One silent for 30 s reconnects.
|
||||
- A **book thread** owns its markets' books and applies first one wins: a
|
||||
change is a duplicate, the next one, or held until it fits (fold, check,
|
||||
resync, Snapshots as before).
|
||||
resync, Snapshots as before). It resubscribes a subscription gone quiet on
|
||||
one connection, and judges a reconnect's books of unchanged symbols.
|
||||
- One **writer** owns the output.
|
||||
|
||||
Settled: OS threads, shared-nothing, no mutexes
|
||||
|
||||
+8
-1
@@ -55,12 +55,16 @@ abstraction is *extracted* once a second exchange exists, not designed up front.
|
||||
Result: losses a quarter lower in each pair of alternating runs; losses and
|
||||
gaps together lower in one pair, even in the other (sprint 04's Outcome).
|
||||
|
||||
5. **Extension 2 — subscription monitoring** — *not started*
|
||||
5. **Extension 2 — subscription monitoring** — *done (2026-11-11)*
|
||||
Per-symbol activity monitoring (data points per second/minute); unsubscribe +
|
||||
resubscribe symbols that go too quiet (with two connections, first one wins
|
||||
hides a subscription gone quiet on one of them: it has to be looked for); periodic full-socket recycle
|
||||
(disconnect → reconnect → resubscribe) to shed cumulative exchange-side bugs.
|
||||
Sprint: [../roadmap/05. subscription monitoring.md](../roadmap/05.%20subscription%20monitoring.md).
|
||||
*Ends:* monitoring + recycling running.
|
||||
Result: a 30-minute run recycled 107 connections with losses no higher
|
||||
than sprint 04's; quiet subscriptions are rare (26 resubscribed, none
|
||||
quiet again) and no connection fell silent (sprint 05's Outcome).
|
||||
|
||||
6. **Extension 3 — generic core + Kalshi adapter** — *not started*
|
||||
Extract the exchange-agnostic core (the `NormalizedEvent` seam) from the
|
||||
@@ -115,6 +119,9 @@ abstraction is *extracted* once a second exchange exists, not designed up front.
|
||||
book, when the first one's chain breaks before any change verified.
|
||||
- An exact rescued count: a change remembered past the window of 128 until
|
||||
its copies are counted (sprint 04 reports an upper bound).
|
||||
- Telling a change still in flight from a symbol quiet on both connections,
|
||||
when a reconnect's book is newer than ours (`maybe`: sprint 05 logs both
|
||||
as a loss; the Snapshot is right either way).
|
||||
- Fewer Snapshots for unchanged books (the brief asks for every symbol every
|
||||
minute: ~130 MB a minute at full coverage).
|
||||
- Re-reading a market's `orderMinSize` / `negRisk` if they can change.
|
||||
Reference in new issue
Block a user