Archived
feat(polymarket): track every live market over load-balanced lanes
An orchestrator keeps the tracked set equal to Gamma's live markets (a scan at startup and every hour, new and closed markets every minute, reaching back 15 minutes for Gamma's indexing delay) and spreads them over lanes, each a connection and a book thread, all writing through the one writer. Lanes report each market's load in bytes per second; the orchestrator puts new markets on the least-loaded lane and moves the busiest markets off lanes over a 300 KB/s budget, busiest lane first. Differing books our own subscribe explains are subscription gaps, not losses, and every first book after a move is logged, so reconcile's mismatches are all accounted for in the log. A 75-minute live run tracked ~195k markets (~390k symbols) in 89 lanes.
This commit is contained in:
18 files changed
+2826
-320
No files matched your search
+36
-19
@@ -42,8 +42,10 @@ The interacting decisions it is made of:
|
||||
([ADR 0001](adr/0001-concurrency-model.md)).
|
||||
- **Stamping point:** in the socket-read path, immediately, so as little as
|
||||
possible sits between the wire and the receive timestamp.
|
||||
- **Book ownership:** shard symbols so each book is owned by exactly one worker
|
||||
(lock-free) rather than a shared, locked map.
|
||||
- **Book ownership:** settled in sprint 03 — symbols are spread over lanes,
|
||||
each a connection and a book thread of its own, so each book is owned by
|
||||
exactly one thread; an orchestrator assigns them
|
||||
([ADR 0002](adr/0002-lanes-and-orchestrator.md)).
|
||||
- **Backpressure:** bounded vs unbounded channels — the core correctness/perf
|
||||
tension. Drop-on-full loses data (violates completeness); block adds latency
|
||||
and fills socket buffers. This decision is the bet in miniature.
|
||||
@@ -130,25 +132,37 @@ and drop from the global workflow for a job-challenge repo.
|
||||
- Any exchange beyond Polymarket and Kalshi.
|
||||
- Persistence/replay infrastructure beyond what the SBE output already gives.
|
||||
|
||||
## Pipeline, first sketch
|
||||
## Pipeline
|
||||
|
||||
The primary thing we debate sprint by sprint (this project's equivalent of the
|
||||
schema debate). Rough shape:
|
||||
schema debate). As settled in sprint 03:
|
||||
|
||||
```
|
||||
discovery ──▶ subscription manager ──▶ N socket readers (stamp receive time here)
|
||||
│
|
||||
route by symbol
|
||||
▼
|
||||
per-symbol book workers (fold, loss-detect, dedup sources)
|
||||
│
|
||||
▼
|
||||
single output writer (SBE sink)
|
||||
rescan fetcher ──findings──▶ ┌──────────────┐ ◀──findings── recent fetcher
|
||||
(full Gamma scan) ◀─request─│ orchestrator │──request──▶ (new + closed since T)
|
||||
└──┬────────▲──┘
|
||||
Add / MoveIn / Remove, to │ │ load reports, each minute
|
||||
the market's lane │ │ (bytes/s per market)
|
||||
┌──────────────────────┼────────┴─────────────────┐
|
||||
▼ ▼ ▼
|
||||
lane 0: book thread ◀─frames─ connection lane 1: ... lane N: ...
|
||||
│ └─subscribe / unsubscribe / resync─▶┘
|
||||
└──────────── Snapshot | Delta ─────────────▶ writer thread (one)
|
||||
```
|
||||
|
||||
Each stage and the channels between them are open until settled in a sprint.
|
||||
Settled so far: OS threads, shared-nothing, no mutexes
|
||||
([ADR 0001](adr/0001-concurrency-model.md)).
|
||||
- The **orchestrator** owns what is tracked (market → lane, the lanes, the
|
||||
timers) and decides; it never blocks on HTTP, so later coordination
|
||||
(switching sources in M4, recycling in M5) lives there too.
|
||||
- The two **fetchers** only page Gamma on request and stream what they read.
|
||||
- Each **lane** is a connection (stamps receive time, sends subscriptions and
|
||||
resyncs) and a book thread (fold, check, resync, Snapshots); M4 gives a lane
|
||||
several connections and has its book thread choose the trusted source.
|
||||
- One **writer** owns the output.
|
||||
|
||||
Settled: OS threads, shared-nothing, no mutexes
|
||||
([ADR 0001](adr/0001-concurrency-model.md)); lanes and the orchestrator
|
||||
([ADR 0002](adr/0002-lanes-and-orchestrator.md)). Details:
|
||||
[sprint 03](../roadmap/03.%20full%20coverage.md).
|
||||
|
||||
## Data model, first sketch
|
||||
|
||||
@@ -167,7 +181,13 @@ Not a DB — an in-memory domain plus the fixed SBE output schema. Main entities
|
||||
- **Book** (one per symbol): bids and asks as ordered price→quantity maps, plus
|
||||
the last loss-detection marker seen, last exchange timestamp, and the id of the
|
||||
source currently trusted (for redundancy).
|
||||
- **Connection** (one websocket): the set of symbols it carries.
|
||||
- **Connection** (one websocket): the set of symbols it carries, those of its
|
||||
lane.
|
||||
- **Lane**: a set of symbols, the book thread that owns their books, and the
|
||||
connections that feed it. Live markets are assigned to lanes by
|
||||
`conditionId`; Polymarket's lifecycle signal is Gamma (settled in sprint 03:
|
||||
a market is live while active, not closed, with an order book and accepting
|
||||
orders).
|
||||
- **Source / redundancy:** several connections may feed the same symbol; one is
|
||||
the trusted source at a time.
|
||||
|
||||
@@ -181,9 +201,6 @@ trusted source is chosen and switched, and emitting a fresh snapshot on switch).
|
||||
|
||||
- **Loss detection on Kalshi.** Its sequence semantics: checked in M6.
|
||||
(Polymarket: settled in M2, by its book hash.)
|
||||
- **Symbol lifecycle signal.** What marks a market "no longer trading"?
|
||||
Polymarket's `new_market` / `market_resolved` events (with
|
||||
`custom_feature_enabled`) are candidates; settle in M3.
|
||||
- **Redundancy source-selection.** How to pick/switch the trusted source
|
||||
(freshness, message rate, which source is ahead). Designed in the redundancy
|
||||
sprint.
|
||||
Reference in new issue
Block a user