Archived
feat(polymarket)!: two connections per market, first one wins
Every live market is subscribed on two connections chosen at random, and its book takes each change from whichever connection delivers it first: the book hash says whether a change is a duplicate, the next one, or one to hold until the change before it arrives. A change one connection misses costs nothing, each Delta carries the earliest receive time, and a connection that drops is covered by the other. When a change fits on no connection in time, the book follows one connection unchecked and asks the other for a fresh book. Connections now decode and route events to 16 book threads, independent of them; a move leaves its old connection only once the new one delivers, so it costs no change; a connection closed with 1013 twice within ten minutes is split. reconcile reads the collector's log and fails only on a mismatch no cause line explains. SIGTERM and SIGHUP stop the collector cleanly. Breaking: lanes are gone (connections and book threads are separate), and reconcile takes the log as a second argument.
This commit is contained in:
24 files changed
+8984
-1214
No files matched your search
+41
-45
@@ -38,22 +38,22 @@ saves the data.
|
||||
The interacting decisions it is made of:
|
||||
|
||||
- **Threading / runtime:** settled in sprint 01 — OS threads + channels,
|
||||
shared-nothing, async re-evaluated at M4
|
||||
shared-nothing; async was re-evaluated at M4 and threads stay
|
||||
([ADR 0001](adr/0001-concurrency-model.md)).
|
||||
- **Stamping point:** in the socket-read path, immediately, so as little as
|
||||
possible sits between the wire and the receive timestamp.
|
||||
- **Book ownership:** settled in sprint 03 — symbols are spread over lanes,
|
||||
each a connection and a book thread of its own, so each book is owned by
|
||||
exactly one thread; an orchestrator assigns them
|
||||
([ADR 0002](adr/0002-lanes-and-orchestrator.md)).
|
||||
- **Book ownership:** each book is owned by exactly one of 16 book threads,
|
||||
chosen from its market's id; an orchestrator decides which connections
|
||||
carry each market ([ADR 0002](adr/0002-lanes-and-orchestrator.md), its
|
||||
lanes superseded by [ADR 0003](adr/0003-first-one-wins.md) in sprint 04).
|
||||
- **Backpressure:** bounded vs unbounded channels — the core correctness/perf
|
||||
tension. Drop-on-full loses data (violates completeness); block adds latency
|
||||
and fills socket buffers. This decision is the bet in miniature.
|
||||
- **Output serialization:** one writer owns the single sink and consumes from
|
||||
all workers.
|
||||
- **Redundancy fit:** multiple sockets feeding one symbol's worker, which
|
||||
dedups / selects by source — the topology must leave room for this from the
|
||||
start.
|
||||
- **Redundancy:** settled in sprint 04 — every symbol on two connections
|
||||
chosen independently, its book taking each change from whichever delivers
|
||||
it first ([ADR 0003](adr/0003-first-one-wins.md)).
|
||||
|
||||
## Requirements the pipeline must satisfy
|
||||
|
||||
@@ -118,7 +118,7 @@ Their operational form, for writing code, is in the project
|
||||
Chunks must be 64 MiB: `AlignedSbeReader::open` reads nothing else.
|
||||
- **Crash ceiling** (accepted in sprint 02): the writer holds the current
|
||||
64 MiB chunk in memory until it is full, so a hard crash loses it; flushing
|
||||
more often pads each flush to 64 MiB. Taken up with extension 1 (M4).
|
||||
more often pads each flush to 64 MiB. Kept as a ceiling in extension 1 (M4).
|
||||
|
||||
## Workflow differences
|
||||
|
||||
@@ -135,34 +135,38 @@ and drop from the global workflow for a job-challenge repo.
|
||||
## Pipeline
|
||||
|
||||
The primary thing we debate sprint by sprint (this project's equivalent of the
|
||||
schema debate). As settled in sprint 03:
|
||||
schema debate). As settled in sprint 04:
|
||||
|
||||
```
|
||||
rescan fetcher ──findings──▶ ┌──────────────┐ ◀──findings── recent fetcher
|
||||
(full Gamma scan) ◀─request─│ orchestrator │──request──▶ (new + closed since T)
|
||||
└──┬────────▲──┘
|
||||
Add / MoveIn / Remove, to │ │ load reports, each minute
|
||||
the market's lane │ │ (bytes/s per market)
|
||||
┌──────────────────────┼────────┴─────────────────┐
|
||||
▼ ▼ ▼
|
||||
lane 0: book thread ◀─frames─ connection lane 1: ... lane N: ...
|
||||
│ └─subscribe / unsubscribe / resync─▶┘
|
||||
└──────────── Snapshot | Delta ─────────────▶ writer thread (one)
|
||||
│ orchestrator │ ◀── 1013 closes ── connections
|
||||
└──────┬───▲───┘
|
||||
Add / Move / Remove (a market and │ │ load reports, each minute
|
||||
its connections), to its thread │ │ (bytes/s per market)
|
||||
┌──────▼───┴───┐
|
||||
connection 0..N ──events───▶ │ book threads │ ──Snapshot | Delta──▶ writer
|
||||
(read, stamp, parse, route) │ 0..15 │
|
||||
▲ └──────┬───────┘
|
||||
└── subscribe / unsubscribe / resync ┘
|
||||
```
|
||||
|
||||
- The **orchestrator** owns what is tracked (market → lane, the lanes, the
|
||||
timers) and decides; it never blocks on HTTP, so later coordination
|
||||
(switching sources in M4, recycling in M5) lives there too.
|
||||
- The **orchestrator** owns what is tracked (each market's two connections,
|
||||
the timers) and decides: placing, moving, splitting. It never blocks on
|
||||
HTTP.
|
||||
- The two **fetchers** only page Gamma on request and stream what they read.
|
||||
- Each **lane** is a connection (stamps receive time, sends subscriptions and
|
||||
resyncs) and a book thread (fold, check, resync, Snapshots); M4 gives a lane
|
||||
several connections and has its book thread choose the trusted source.
|
||||
- A **connection** reads, stamps the receive time, parses, and routes each
|
||||
event to the book thread of its market. Every symbol is on two connections,
|
||||
chosen independently.
|
||||
- A **book thread** owns its markets' books and applies first one wins: a
|
||||
change is a duplicate, the next one, or held until it fits (fold, check,
|
||||
resync, Snapshots as before).
|
||||
- One **writer** owns the output.
|
||||
|
||||
Settled: OS threads, shared-nothing, no mutexes
|
||||
([ADR 0001](adr/0001-concurrency-model.md)); lanes and the orchestrator
|
||||
([ADR 0002](adr/0002-lanes-and-orchestrator.md)). Details:
|
||||
[sprint 03](../roadmap/03.%20full%20coverage.md).
|
||||
([ADR 0001](adr/0001-concurrency-model.md)); the orchestrator
|
||||
([ADR 0002](adr/0002-lanes-and-orchestrator.md)); two connections per symbol,
|
||||
first one wins ([ADR 0003](adr/0003-first-one-wins.md)). Details:
|
||||
[sprint 04](../roadmap/04.%20redundant%20connections.md).
|
||||
|
||||
## Data model, first sketch
|
||||
|
||||
@@ -179,28 +183,20 @@ Not a DB — an in-memory domain plus the fixed SBE output schema. Main entities
|
||||
whatever the exchange offers for loss detection (a sequence number, a book
|
||||
hash), and payload (side/price/quantity, or the full book).
|
||||
- **Book** (one per symbol): bids and asks as ordered price→quantity maps, plus
|
||||
the last loss-detection marker seen, last exchange timestamp, and the id of the
|
||||
source currently trusted (for redundancy).
|
||||
- **Connection** (one websocket): the set of symbols it carries, those of its
|
||||
lane.
|
||||
- **Lane**: a set of symbols, the book thread that owns their books, and the
|
||||
connections that feed it. Live markets are assigned to lanes by
|
||||
`conditionId`; Polymarket's lifecycle signal is Gamma (settled in sprint 03:
|
||||
a market is live while active, not closed, with an order book and accepting
|
||||
orders).
|
||||
- **Source / redundancy:** several connections may feed the same symbol; one is
|
||||
the trusted source at a time.
|
||||
the last loss-detection marker seen and last exchange timestamp. It is fed
|
||||
by the symbol's two connections, first one wins.
|
||||
- **Connection** (one websocket): the set of symbols it carries. Live markets
|
||||
are placed on connections by `conditionId`, both tokens together;
|
||||
Polymarket's lifecycle signal is Gamma (settled in sprint 03: a market is
|
||||
live while active, not closed, with an order book and accepting orders).
|
||||
|
||||
Output schema is fixed by the SBE schema: `Snapshot` and `PriceLevelUpdate`.
|
||||
|
||||
The debate lives in two places: the **NormalizedEvent** shape (both exchanges
|
||||
must map cleanly onto it) and **redundancy reconciliation** (how a symbol's
|
||||
trusted source is chosen and switched, and emitting a fresh snapshot on switch).
|
||||
The debate lives in the **NormalizedEvent** shape (both exchanges must map
|
||||
cleanly onto it). Redundancy was settled in sprint 04: no source is trusted,
|
||||
the first copy of each change wins ([ADR 0003](adr/0003-first-one-wins.md)).
|
||||
|
||||
## Open questions
|
||||
|
||||
- **Loss detection on Kalshi.** Its sequence semantics: checked in M6.
|
||||
(Polymarket: settled in M2, by its book hash.)
|
||||
- **Redundancy source-selection.** How to pick/switch the trusted source
|
||||
(freshness, message rate, which source is ahead). Designed in the redundancy
|
||||
sprint.
|
||||
Reference in new issue
Block a user