feat(polymarket)!: two connections per market, first one wins

Every live market is subscribed on two connections chosen at random, and its
book takes each change from whichever connection delivers it first: the book
hash says whether a change is a duplicate, the next one, or one to hold until
the change before it arrives. A change one connection misses costs nothing,
each Delta carries the earliest receive time, and a connection that drops is
covered by the other. When a change fits on no connection in time, the book
follows one connection unchecked and asks the other for a fresh book.
Connections now decode and route events to 16 book threads, independent of
them; a move leaves its old connection only once the new one delivers, so it
costs no change; a connection closed with 1013 twice within ten minutes is
split. reconcile reads the collector's log and fails only on a mismatch no
cause line explains. SIGTERM and SIGHUP stop the collector cleanly.

Breaking: lanes are gone (connections and book threads are separate), and
reconcile takes the log as a second argument.
This commit is contained in:
pascal committed 2026-11-11 15:18:48 +00:00
1 parent 8114a51dc2
commit 02747b882f
24 files changed
+8984 -1214

No files matched your search

+4
View File
@@ -29,6 +29,10 @@ races by construction, and keeps the runtime pluggable: only the socket layer is
runtime-specific, so a later async swap is localized to that boundary rather than
a pipeline rewrite. Deferring async is therefore safe through the base challenge.
The M4 checkpoint was held in sprint 04 (2026-11-11): threads stay. The new
logic of redundancy sits in the book threads, not in socket handling, and
about 200 threads (175 connections, 16 book threads) are comfortable.
Trade-off: thread-per-socket does not scale to very large socket counts, and the
extension-era socket-management logic (select / timeout / reconnect) is where
threads and async diverge most — hence the explicit M4 re-evaluation point rather
+4
View File
@@ -1,5 +1,9 @@
# 0002. Lanes and an orchestrator
*Superseded in part by [0003](0003-first-one-wins.md) (sprint 04): the lanes are
gone, connections and book threads being separate sets; the orchestrator and
the fetchers stand.*
## Context
Every live Polymarket symbol is ~476k tokens (2026-11-04), an estimated
+47
View File
@@ -0,0 +1,47 @@
# 0003. Two connections per symbol, first one wins
## Context
Extension 1 asks for redundant websockets: connections drop, go quiet, lag,
and get closed by the server (code 1013) when they carry too much. Sprint 03
gave each lane a connection and a book thread of its own
([0002](0002-lanes-and-orchestrator.md)). The obvious next step was a second,
identical connection per lane, one of them the trusted source, with a
Snapshot at each switch, as the brief sketches. Measured on 2026-11-11
([sprint 04](../../roadmap/04.%20redundant%20connections.md)): two connections
on the same symbols deliver the same changes in the same order, 0.5 ms apart
at the median and 100 ms or more for 1% of changes; and two connections
carrying the same symbols are closed together.
## Decision
Every symbol is subscribed on **two connections chosen independently and at
random**, and its one book takes each change from **whichever connection
delivers it first**. Polymarket's book hash, sent with every change, says
where a change belongs: already applied (a duplicate), next (it fits and is
written), or early (held until the change before it arrives). Only when a
change fits on no connection in time does the book thread fall back to
following one connection unchecked and resyncing on the other.
Connections therefore no longer belong to a book thread. A connection thread
stamps, parses and routes each event to the book thread of its market, one of
16, derived from the market's id. The orchestrator of 0002 stays; its lanes
are superseded.
## Why
First one wins gives each Delta the earliest receive time, turns a change one
connection missed into no loss at all, and needs no second book and no
Snapshot per switch. A source and a standby would have kept the source's
receive time and lost the change the source missed. Independent connections
were measured against twins on the same 48,038 symbols for 20 minutes: 272
symbol-seconds with both subscriptions down, against 3,787 for twins dealt at
random and 33,345 for twins in creation order. A move becomes a subscribe and
an unsubscribe with the book left where it is, so it opens no subscription
gap.
Trade-off: the routing step 0002 avoided is back, parsing moves into the
connection thread, and the book thread's rules get subtler (held changes, a
ceiling, a chain declared broken). The design leans on an exchange that lets
the same change be recognised across connections: Polymarket's hash does;
Kalshi's `clock` looks as if it would (to confirm in M6).
+2 -2
View File
@@ -46,8 +46,8 @@ Prefer these; they can't be gamed by strategy luck:
separately our *internal* added latency (edge-stamp → output byte).
- **Completeness:** reconciliation error rate (snapshot↔delta divergence); count
of missed deletes / stale levels under induced loss.
- **Redundancy value (once ext1 exists):** messages the backup socket caught that
the primary missed — literal "data loss avoided."
- **Redundancy value:** changes one connection missed and the other delivered
(rescued) — literal "data loss avoided."
## Recommendation
+26
View File
@@ -147,6 +147,32 @@ From a 60 s run of the M1 spike on one market, both tokens subscribed, and the
403 (curl and `ureq` get 200), most likely Cloudflare filtering by user agent.
Fetch with curl in throwaway scripts.
## Two connections on the same symbols (observed 2026-11-11)
- **Same changes, same order**: two connections subscribed to the same 8,000
tokens for 335 s delivered the same 616,708 changes, none on only one, in
the same order per token. So the book hash orders changes across
connections too.
- **The second delivery's lag**: 0.5 ms at the median, over 10 ms for 6.5% of
changes, over 100 ms for 1%, 706 ms at worst; up to 82 changes of its token
behind. Neither connection leads (one was first 34% of the time).
- **Batches**: 95% of updates are one change. The others come one change per
frame, all with the hash of the whole batch, within 1 ms (40 ms at worst).
- **The server repeats some changes** on one connection (104 in that
recording, the same on both): harmless, quantities being absolute.
- **Connections carrying the same tokens are closed together**: two ramped on
the same tokens got their 1013 close 3 s apart. Over 20 minutes on 48,038
tokens with two subscriptions each, twins in creation order had both
subscriptions down for 33,345 token-seconds, twins dealt at random 3,787,
two independent random deals 272.
- **300 connections** opened at once from one address all stayed up.
- **A connection whose every subscribed market has resolved is closed** by the
server, code 1000, reason `all subscribed assets resolved` (seen on a
connection left with one market, 2026-11-11).
- **A change's `timestamp` never goes back within a token** on a connection
(0 of 660,084 changes on each of the two):
one older than the last applied is a late copy.
## Gamma, the market listing (observed 2026-11-04)
- **Size:** 238,308 markets `active` and not `closed`; 238,201 of them with
+41 -45
View File
@@ -38,22 +38,22 @@ saves the data.
The interacting decisions it is made of:
- **Threading / runtime:** settled in sprint 01 — OS threads + channels,
shared-nothing, async re-evaluated at M4
shared-nothing; async was re-evaluated at M4 and threads stay
([ADR 0001](adr/0001-concurrency-model.md)).
- **Stamping point:** in the socket-read path, immediately, so as little as
possible sits between the wire and the receive timestamp.
- **Book ownership:** settled in sprint 03 — symbols are spread over lanes,
each a connection and a book thread of its own, so each book is owned by
exactly one thread; an orchestrator assigns them
([ADR 0002](adr/0002-lanes-and-orchestrator.md)).
- **Book ownership:** each book is owned by exactly one of 16 book threads,
chosen from its market's id; an orchestrator decides which connections
carry each market ([ADR 0002](adr/0002-lanes-and-orchestrator.md), its
lanes superseded by [ADR 0003](adr/0003-first-one-wins.md) in sprint 04).
- **Backpressure:** bounded vs unbounded channels — the core correctness/perf
tension. Drop-on-full loses data (violates completeness); block adds latency
and fills socket buffers. This decision is the bet in miniature.
- **Output serialization:** one writer owns the single sink and consumes from
all workers.
- **Redundancy fit:** multiple sockets feeding one symbol's worker, which
dedups / selects by source — the topology must leave room for this from the
start.
- **Redundancy:** settled in sprint 04 — every symbol on two connections
chosen independently, its book taking each change from whichever delivers
it first ([ADR 0003](adr/0003-first-one-wins.md)).
## Requirements the pipeline must satisfy
@@ -118,7 +118,7 @@ Their operational form, for writing code, is in the project
Chunks must be 64 MiB: `AlignedSbeReader::open` reads nothing else.
- **Crash ceiling** (accepted in sprint 02): the writer holds the current
64 MiB chunk in memory until it is full, so a hard crash loses it; flushing
more often pads each flush to 64 MiB. Taken up with extension 1 (M4).
more often pads each flush to 64 MiB. Kept as a ceiling in extension 1 (M4).
## Workflow differences
@@ -135,34 +135,38 @@ and drop from the global workflow for a job-challenge repo.
## Pipeline
The primary thing we debate sprint by sprint (this project's equivalent of the
schema debate). As settled in sprint 03:
schema debate). As settled in sprint 04:
```
rescan fetcher ──findings──▶ ┌──────────────┐ ◀──findings── recent fetcher
(full Gamma scan) ◀─request─│ orchestrator │──request──▶ (new + closed since T)
└──┬────────▲──┘
Add / MoveIn / Remove, to │ │ load reports, each minute
the market's lane │ │ (bytes/s per market)
┌──────────────────────┼────────┴─────────────────┐
▼ ▼ ▼
lane 0: book thread ◀─frames─ connection lane 1: ... lane N: ...
│ └─subscribe / unsubscribe / resync─▶┘
└──────────── Snapshot | Delta ─────────────▶ writer thread (one)
│ orchestrator │ ◀── 1013 closes ── connections
└──────┬───▲───┘
Add / Move / Remove (a market and │ │ load reports, each minute
its connections), to its thread │ │ (bytes/s per market)
┌──────▼───┴───┐
connection 0..N ──events───▶ │ book threads │ ──Snapshot | Delta──▶ writer
(read, stamp, parse, route) │ 0..15 │
▲ └──────┬───────┘
└── subscribe / unsubscribe / resync ┘
```
- The **orchestrator** owns what is tracked (market → lane, the lanes, the
timers) and decides; it never blocks on HTTP, so later coordination
(switching sources in M4, recycling in M5) lives there too.
- The **orchestrator** owns what is tracked (each market's two connections,
the timers) and decides: placing, moving, splitting. It never blocks on
HTTP.
- The two **fetchers** only page Gamma on request and stream what they read.
- Each **lane** is a connection (stamps receive time, sends subscriptions and
resyncs) and a book thread (fold, check, resync, Snapshots); M4 gives a lane
several connections and has its book thread choose the trusted source.
- A **connection** reads, stamps the receive time, parses, and routes each
event to the book thread of its market. Every symbol is on two connections,
chosen independently.
- A **book thread** owns its markets' books and applies first one wins: a
change is a duplicate, the next one, or held until it fits (fold, check,
resync, Snapshots as before).
- One **writer** owns the output.
Settled: OS threads, shared-nothing, no mutexes
([ADR 0001](adr/0001-concurrency-model.md)); lanes and the orchestrator
([ADR 0002](adr/0002-lanes-and-orchestrator.md)). Details:
[sprint 03](../roadmap/03.%20full%20coverage.md).
([ADR 0001](adr/0001-concurrency-model.md)); the orchestrator
([ADR 0002](adr/0002-lanes-and-orchestrator.md)); two connections per symbol,
first one wins ([ADR 0003](adr/0003-first-one-wins.md)). Details:
[sprint 04](../roadmap/04.%20redundant%20connections.md).
## Data model, first sketch
@@ -179,28 +183,20 @@ Not a DB — an in-memory domain plus the fixed SBE output schema. Main entities
whatever the exchange offers for loss detection (a sequence number, a book
hash), and payload (side/price/quantity, or the full book).
- **Book** (one per symbol): bids and asks as ordered price→quantity maps, plus
the last loss-detection marker seen, last exchange timestamp, and the id of the
source currently trusted (for redundancy).
- **Connection** (one websocket): the set of symbols it carries, those of its
lane.
- **Lane**: a set of symbols, the book thread that owns their books, and the
connections that feed it. Live markets are assigned to lanes by
`conditionId`; Polymarket's lifecycle signal is Gamma (settled in sprint 03:
a market is live while active, not closed, with an order book and accepting
orders).
- **Source / redundancy:** several connections may feed the same symbol; one is
the trusted source at a time.
the last loss-detection marker seen and last exchange timestamp. It is fed
by the symbol's two connections, first one wins.
- **Connection** (one websocket): the set of symbols it carries. Live markets
are placed on connections by `conditionId`, both tokens together;
Polymarket's lifecycle signal is Gamma (settled in sprint 03: a market is
live while active, not closed, with an order book and accepting orders).
Output schema is fixed by the SBE schema: `Snapshot` and `PriceLevelUpdate`.
The debate lives in two places: the **NormalizedEvent** shape (both exchanges
must map cleanly onto it) and **redundancy reconciliation** (how a symbol's
trusted source is chosen and switched, and emitting a fresh snapshot on switch).
The debate lives in the **NormalizedEvent** shape (both exchanges must map
cleanly onto it). Redundancy was settled in sprint 04: no source is trusted,
the first copy of each change wins ([ADR 0003](adr/0003-first-one-wins.md)).
## Open questions
- **Loss detection on Kalshi.** Its sequence semantics: checked in M6.
(Polymarket: settled in M2, by its book hash.)
- **Redundancy source-selection.** How to pick/switch the trusted source
(freshness, message rate, which source is ahead). Designed in the redundancy
sprint.
+26 -29
View File
@@ -40,31 +40,25 @@ abstraction is *extracted* once a second exchange exists, not designed up front.
*Ends:* every live Polymarket symbol tracked — base challenge satisfied for
one exchange.
4. **Extension 1 — redundant websockets** — *not started*
Multiple connections per symbol (several connections in a lane, its book
thread choosing among them, the orchestrator coordinating); choose a
trusted source and switch on loss/quiet/delay; emit a fresh snapshot on
every source switch. Recovery from errors, with the lane budget scaling
itself down when connections still get 1013 closes (no rate is safe:
[notes.md](notes.md), so a budget set too high must correct itself).
**Async re-evaluation checkpoint:** before building this, reassess whether the
socket-management complexity (reconnect/timeout/select, redundancy) warrants
swapping the thread-only runtime for tokio (see ADR 0001).
The lane that holds markets busy together (the 5- and 15-minute crypto
markets: closed 16 times in 75 minutes, in bursts a minute's load
average doesn't show) is the case to beat.
Also: surviving a hard crash without losing the writer's in-memory chunk,
a move's Remove reaching the old lane before the new lane's first
Snapshot, the memory growth seen in sprint 03 (~90 MB every 10 minutes
over 75 minutes), and closing the subscription gap (the few ms of a
symbol's changes the server doesn't send between our unsubscribe and
subscribe); the crash chunk and the gap were accepted as ceilings in M2.
*Ends:* redundancy demonstrably reduces data loss; a Snapshot on every source
switch.
4. **Extension 1 — redundant websockets** — *done (2026-11-11)*
Every symbol on two connections chosen independently, its book taking each
change from whichever delivers it first (first one wins); connections
separated from book threads; a connection the server keeps closing with
1013 is split, so a budget set too high corrects itself. The async
checkpoint was held: threads stay
([ADR 0001](adr/0001-concurrency-model.md)).
Sprint: [../roadmap/04. redundant connections.md](../roadmap/04.%20redundant%20connections.md).
Left as ceilings: the writer's in-memory chunk on a hard crash, and the
subscription gap when a symbol opens.
*Ends:* redundancy demonstrably reduces data loss: fewer losses and
subscription gaps per million changes with two subscriptions than with one.
Result: losses a quarter lower in each pair of alternating runs; losses and
gaps together lower in one pair, even in the other (sprint 04's Outcome).
5. **Extension 2 — subscription monitoring** — *not started*
Per-symbol activity monitoring (data points per second/minute); unsubscribe +
resubscribe symbols that go too quiet; periodic full-socket recycle
resubscribe symbols that go too quiet (with two connections, first one wins
hides a subscription gone quiet on one of them: it has to be looked for); periodic full-socket recycle
(disconnect → reconnect → resubscribe) to shed cumulative exchange-side bugs.
*Ends:* monitoring + recycling running.
@@ -108,16 +102,19 @@ abstraction is *extracted* once a second exchange exists, not designed up front.
- Observability: metrics, structured logging, dashboards.
- Config-driven symbol filtering and per-exchange tuning.
- Containerized packaging.
- Counting a reconnect's differing books as losses rather than subscription
gaps when the symbol was resyncing or not yet verified since its opening
book (a reconnect marker through the frames channel; they are labelled
gaps).
- A `reconcile` that reads the collector's log, to tell subscription gaps from
losses in its own report.
- Polymarket's `new_market` websocket event as a faster discovery signal (it
is broadcast, but lacks `orderMinSize` / `negRisk`, so Gamma is still
needed; `market_resolved` was never seen).
- Closing lanes that drain.
- Closing connections that drain.
- Verifying a symbol again from the trade event's price after a trade, with
no resync (sprint 04's prototype did it for 15 of 100 broken chains).
- Telling a real loss from a resync book merely ahead of our stream, so the
loss count measures data missed (most of sprint 04's "losses" follow
trades, as often with one connection as with two).
- Closing the opening subscription gap with the second connection's opening
book, when the first one's chain breaks before any change verified.
- An exact rescued count: a change remembered past the window of 128 until
its copies are counted (sprint 04 reports an upper bound).
- Fewer Snapshots for unchanged books (the brief asks for every symbol every
minute: ~130 MB a minute at full coverage).
- Re-reading a market's `orderMinSize` / `negRisk` if they can change.