Archived
feat(polymarket): subscription monitoring and connection recycling
The exchange can fail without telling us, and first one wins hides a subscription gone quiet on one of a symbol's two connections. - A connection that receives nothing for 30 s, not even the keepalive's answer, reconnects. - Each book thread counts a symbol's changes per connection every minute: a subscription with none against three or more on the other is resubscribed, and moved to another connection if it stays quiet. - The orchestrator recycles one connection at a time, each about hourly, and never while another connection is down or just back. - A reconnect's opening book of a symbol unchanged for 10 s is judged like a resync's answer: one newer than ours that differs shows a symbol quiet on both connections, written as a Snapshot and logged as a loss.
This commit is contained in:
10 files changed
+1346
-102
No files matched your search
+10
-8
@@ -135,14 +135,15 @@ and drop from the global workflow for a job-challenge repo.
|
||||
## Pipeline
|
||||
|
||||
The primary thing we debate sprint by sprint (this project's equivalent of the
|
||||
schema debate). As settled in sprint 04:
|
||||
schema debate). As settled in sprints 04 and 05:
|
||||
|
||||
```
|
||||
rescan fetcher ──findings──▶ ┌──────────────┐ ◀──findings── recent fetcher
|
||||
│ orchestrator │ ◀── 1013 closes ── connections
|
||||
└──────┬───▲───┘
|
||||
│ orchestrator │ ◀── up, down, 1013 ─── connections
|
||||
└──────┬───▲───┘ ──── recycle ────────▶ connections
|
||||
Add / Move / Remove (a market and │ │ load reports, each minute
|
||||
its connections), to its thread │ │ (bytes/s per market)
|
||||
its connections), to its thread │ │ (bytes/s per market); markets
|
||||
│ │ quiet twice on a connection
|
||||
┌──────▼───┴───┐
|
||||
connection 0..N ──events───▶ │ book threads │ ──Snapshot | Delta──▶ writer
|
||||
(read, stamp, parse, route) │ 0..15 │
|
||||
@@ -151,15 +152,16 @@ schema debate). As settled in sprint 04:
|
||||
```
|
||||
|
||||
- The **orchestrator** owns what is tracked (each market's two connections,
|
||||
the timers) and decides: placing, moving, splitting. It never blocks on
|
||||
HTTP.
|
||||
the timers) and decides: placing, moving, splitting, and recycling one
|
||||
connection at a time, about hourly. It never blocks on HTTP.
|
||||
- The two **fetchers** only page Gamma on request and stream what they read.
|
||||
- A **connection** reads, stamps the receive time, parses, and routes each
|
||||
event to the book thread of its market. Every symbol is on two connections,
|
||||
chosen independently.
|
||||
chosen independently. One silent for 30 s reconnects.
|
||||
- A **book thread** owns its markets' books and applies first one wins: a
|
||||
change is a duplicate, the next one, or held until it fits (fold, check,
|
||||
resync, Snapshots as before).
|
||||
resync, Snapshots as before). It resubscribes a subscription gone quiet on
|
||||
one connection, and judges a reconnect's books of unchanged symbols.
|
||||
- One **writer** owns the output.
|
||||
|
||||
Settled: OS threads, shared-nothing, no mutexes
|
||||
|
||||
Reference in new issue
Block a user