Archived
feat(polymarket)!: two connections per market, first one wins
Every live market is subscribed on two connections chosen at random, and its book takes each change from whichever connection delivers it first: the book hash says whether a change is a duplicate, the next one, or one to hold until the change before it arrives. A change one connection misses costs nothing, each Delta carries the earliest receive time, and a connection that drops is covered by the other. When a change fits on no connection in time, the book follows one connection unchecked and asks the other for a fresh book. Connections now decode and route events to 16 book threads, independent of them; a move leaves its old connection only once the new one delivers, so it costs no change; a connection closed with 1013 twice within ten minutes is split. reconcile reads the collector's log and fails only on a mismatch no cause line explains. SIGTERM and SIGHUP stop the collector cleanly. Breaking: lanes are gone (connections and book threads are separate), and reconcile takes the log as a second argument.
This commit is contained in:
24 files changed
+8984
-1214
No files matched your search
@@ -29,6 +29,10 @@ races by construction, and keeps the runtime pluggable: only the socket layer is
|
||||
runtime-specific, so a later async swap is localized to that boundary rather than
|
||||
a pipeline rewrite. Deferring async is therefore safe through the base challenge.
|
||||
|
||||
The M4 checkpoint was held in sprint 04 (2026-11-11): threads stay. The new
|
||||
logic of redundancy sits in the book threads, not in socket handling, and
|
||||
about 200 threads (175 connections, 16 book threads) are comfortable.
|
||||
|
||||
Trade-off: thread-per-socket does not scale to very large socket counts, and the
|
||||
extension-era socket-management logic (select / timeout / reconnect) is where
|
||||
threads and async diverge most — hence the explicit M4 re-evaluation point rather
|
||||
|
||||
@@ -1,5 +1,9 @@
|
||||
# 0002. Lanes and an orchestrator
|
||||
|
||||
*Superseded in part by [0003](0003-first-one-wins.md) (sprint 04): the lanes are
|
||||
gone, connections and book threads being separate sets; the orchestrator and
|
||||
the fetchers stand.*
|
||||
|
||||
## Context
|
||||
|
||||
Every live Polymarket symbol is ~476k tokens (2026-11-04), an estimated
|
||||
|
||||
@@ -0,0 +1,47 @@
|
||||
# 0003. Two connections per symbol, first one wins
|
||||
|
||||
## Context
|
||||
|
||||
Extension 1 asks for redundant websockets: connections drop, go quiet, lag,
|
||||
and get closed by the server (code 1013) when they carry too much. Sprint 03
|
||||
gave each lane a connection and a book thread of its own
|
||||
([0002](0002-lanes-and-orchestrator.md)). The obvious next step was a second,
|
||||
identical connection per lane, one of them the trusted source, with a
|
||||
Snapshot at each switch, as the brief sketches. Measured on 2026-11-11
|
||||
([sprint 04](../../roadmap/04.%20redundant%20connections.md)): two connections
|
||||
on the same symbols deliver the same changes in the same order, 0.5 ms apart
|
||||
at the median and 100 ms or more for 1% of changes; and two connections
|
||||
carrying the same symbols are closed together.
|
||||
|
||||
## Decision
|
||||
|
||||
Every symbol is subscribed on **two connections chosen independently and at
|
||||
random**, and its one book takes each change from **whichever connection
|
||||
delivers it first**. Polymarket's book hash, sent with every change, says
|
||||
where a change belongs: already applied (a duplicate), next (it fits and is
|
||||
written), or early (held until the change before it arrives). Only when a
|
||||
change fits on no connection in time does the book thread fall back to
|
||||
following one connection unchecked and resyncing on the other.
|
||||
|
||||
Connections therefore no longer belong to a book thread. A connection thread
|
||||
stamps, parses and routes each event to the book thread of its market, one of
|
||||
16, derived from the market's id. The orchestrator of 0002 stays; its lanes
|
||||
are superseded.
|
||||
|
||||
## Why
|
||||
|
||||
First one wins gives each Delta the earliest receive time, turns a change one
|
||||
connection missed into no loss at all, and needs no second book and no
|
||||
Snapshot per switch. A source and a standby would have kept the source's
|
||||
receive time and lost the change the source missed. Independent connections
|
||||
were measured against twins on the same 48,038 symbols for 20 minutes: 272
|
||||
symbol-seconds with both subscriptions down, against 3,787 for twins dealt at
|
||||
random and 33,345 for twins in creation order. A move becomes a subscribe and
|
||||
an unsubscribe with the book left where it is, so it opens no subscription
|
||||
gap.
|
||||
|
||||
Trade-off: the routing step 0002 avoided is back, parsing moves into the
|
||||
connection thread, and the book thread's rules get subtler (held changes, a
|
||||
ceiling, a chain declared broken). The design leans on an exchange that lets
|
||||
the same change be recognised across connections: Polymarket's hash does;
|
||||
Kalshi's `clock` looks as if it would (to confirm in M6).
|
||||
+2
-2
@@ -46,8 +46,8 @@ Prefer these; they can't be gamed by strategy luck:
|
||||
separately our *internal* added latency (edge-stamp → output byte).
|
||||
- **Completeness:** reconciliation error rate (snapshot↔delta divergence); count
|
||||
of missed deletes / stale levels under induced loss.
|
||||
- **Redundancy value (once ext1 exists):** messages the backup socket caught that
|
||||
the primary missed — literal "data loss avoided."
|
||||
- **Redundancy value:** changes one connection missed and the other delivered
|
||||
(rescued) — literal "data loss avoided."
|
||||
|
||||
## Recommendation
|
||||
|
||||
|
||||
@@ -147,6 +147,32 @@ From a 60 s run of the M1 spike on one market, both tokens subscribed, and the
|
||||
403 (curl and `ureq` get 200), most likely Cloudflare filtering by user agent.
|
||||
Fetch with curl in throwaway scripts.
|
||||
|
||||
## Two connections on the same symbols (observed 2026-11-11)
|
||||
|
||||
- **Same changes, same order**: two connections subscribed to the same 8,000
|
||||
tokens for 335 s delivered the same 616,708 changes, none on only one, in
|
||||
the same order per token. So the book hash orders changes across
|
||||
connections too.
|
||||
- **The second delivery's lag**: 0.5 ms at the median, over 10 ms for 6.5% of
|
||||
changes, over 100 ms for 1%, 706 ms at worst; up to 82 changes of its token
|
||||
behind. Neither connection leads (one was first 34% of the time).
|
||||
- **Batches**: 95% of updates are one change. The others come one change per
|
||||
frame, all with the hash of the whole batch, within 1 ms (40 ms at worst).
|
||||
- **The server repeats some changes** on one connection (104 in that
|
||||
recording, the same on both): harmless, quantities being absolute.
|
||||
- **Connections carrying the same tokens are closed together**: two ramped on
|
||||
the same tokens got their 1013 close 3 s apart. Over 20 minutes on 48,038
|
||||
tokens with two subscriptions each, twins in creation order had both
|
||||
subscriptions down for 33,345 token-seconds, twins dealt at random 3,787,
|
||||
two independent random deals 272.
|
||||
- **300 connections** opened at once from one address all stayed up.
|
||||
- **A connection whose every subscribed market has resolved is closed** by the
|
||||
server, code 1000, reason `all subscribed assets resolved` (seen on a
|
||||
connection left with one market, 2026-11-11).
|
||||
- **A change's `timestamp` never goes back within a token** on a connection
|
||||
(0 of 660,084 changes on each of the two):
|
||||
one older than the last applied is a late copy.
|
||||
|
||||
## Gamma, the market listing (observed 2026-11-04)
|
||||
|
||||
- **Size:** 238,308 markets `active` and not `closed`; 238,201 of them with
|
||||
|
||||
+41
-45
@@ -38,22 +38,22 @@ saves the data.
|
||||
The interacting decisions it is made of:
|
||||
|
||||
- **Threading / runtime:** settled in sprint 01 — OS threads + channels,
|
||||
shared-nothing, async re-evaluated at M4
|
||||
shared-nothing; async was re-evaluated at M4 and threads stay
|
||||
([ADR 0001](adr/0001-concurrency-model.md)).
|
||||
- **Stamping point:** in the socket-read path, immediately, so as little as
|
||||
possible sits between the wire and the receive timestamp.
|
||||
- **Book ownership:** settled in sprint 03 — symbols are spread over lanes,
|
||||
each a connection and a book thread of its own, so each book is owned by
|
||||
exactly one thread; an orchestrator assigns them
|
||||
([ADR 0002](adr/0002-lanes-and-orchestrator.md)).
|
||||
- **Book ownership:** each book is owned by exactly one of 16 book threads,
|
||||
chosen from its market's id; an orchestrator decides which connections
|
||||
carry each market ([ADR 0002](adr/0002-lanes-and-orchestrator.md), its
|
||||
lanes superseded by [ADR 0003](adr/0003-first-one-wins.md) in sprint 04).
|
||||
- **Backpressure:** bounded vs unbounded channels — the core correctness/perf
|
||||
tension. Drop-on-full loses data (violates completeness); block adds latency
|
||||
and fills socket buffers. This decision is the bet in miniature.
|
||||
- **Output serialization:** one writer owns the single sink and consumes from
|
||||
all workers.
|
||||
- **Redundancy fit:** multiple sockets feeding one symbol's worker, which
|
||||
dedups / selects by source — the topology must leave room for this from the
|
||||
start.
|
||||
- **Redundancy:** settled in sprint 04 — every symbol on two connections
|
||||
chosen independently, its book taking each change from whichever delivers
|
||||
it first ([ADR 0003](adr/0003-first-one-wins.md)).
|
||||
|
||||
## Requirements the pipeline must satisfy
|
||||
|
||||
@@ -118,7 +118,7 @@ Their operational form, for writing code, is in the project
|
||||
Chunks must be 64 MiB: `AlignedSbeReader::open` reads nothing else.
|
||||
- **Crash ceiling** (accepted in sprint 02): the writer holds the current
|
||||
64 MiB chunk in memory until it is full, so a hard crash loses it; flushing
|
||||
more often pads each flush to 64 MiB. Taken up with extension 1 (M4).
|
||||
more often pads each flush to 64 MiB. Kept as a ceiling in extension 1 (M4).
|
||||
|
||||
## Workflow differences
|
||||
|
||||
@@ -135,34 +135,38 @@ and drop from the global workflow for a job-challenge repo.
|
||||
## Pipeline
|
||||
|
||||
The primary thing we debate sprint by sprint (this project's equivalent of the
|
||||
schema debate). As settled in sprint 03:
|
||||
schema debate). As settled in sprint 04:
|
||||
|
||||
```
|
||||
rescan fetcher ──findings──▶ ┌──────────────┐ ◀──findings── recent fetcher
|
||||
(full Gamma scan) ◀─request─│ orchestrator │──request──▶ (new + closed since T)
|
||||
└──┬────────▲──┘
|
||||
Add / MoveIn / Remove, to │ │ load reports, each minute
|
||||
the market's lane │ │ (bytes/s per market)
|
||||
┌──────────────────────┼────────┴─────────────────┐
|
||||
▼ ▼ ▼
|
||||
lane 0: book thread ◀─frames─ connection lane 1: ... lane N: ...
|
||||
│ └─subscribe / unsubscribe / resync─▶┘
|
||||
└──────────── Snapshot | Delta ─────────────▶ writer thread (one)
|
||||
│ orchestrator │ ◀── 1013 closes ── connections
|
||||
└──────┬───▲───┘
|
||||
Add / Move / Remove (a market and │ │ load reports, each minute
|
||||
its connections), to its thread │ │ (bytes/s per market)
|
||||
┌──────▼───┴───┐
|
||||
connection 0..N ──events───▶ │ book threads │ ──Snapshot | Delta──▶ writer
|
||||
(read, stamp, parse, route) │ 0..15 │
|
||||
▲ └──────┬───────┘
|
||||
└── subscribe / unsubscribe / resync ┘
|
||||
```
|
||||
|
||||
- The **orchestrator** owns what is tracked (market → lane, the lanes, the
|
||||
timers) and decides; it never blocks on HTTP, so later coordination
|
||||
(switching sources in M4, recycling in M5) lives there too.
|
||||
- The **orchestrator** owns what is tracked (each market's two connections,
|
||||
the timers) and decides: placing, moving, splitting. It never blocks on
|
||||
HTTP.
|
||||
- The two **fetchers** only page Gamma on request and stream what they read.
|
||||
- Each **lane** is a connection (stamps receive time, sends subscriptions and
|
||||
resyncs) and a book thread (fold, check, resync, Snapshots); M4 gives a lane
|
||||
several connections and has its book thread choose the trusted source.
|
||||
- A **connection** reads, stamps the receive time, parses, and routes each
|
||||
event to the book thread of its market. Every symbol is on two connections,
|
||||
chosen independently.
|
||||
- A **book thread** owns its markets' books and applies first one wins: a
|
||||
change is a duplicate, the next one, or held until it fits (fold, check,
|
||||
resync, Snapshots as before).
|
||||
- One **writer** owns the output.
|
||||
|
||||
Settled: OS threads, shared-nothing, no mutexes
|
||||
([ADR 0001](adr/0001-concurrency-model.md)); lanes and the orchestrator
|
||||
([ADR 0002](adr/0002-lanes-and-orchestrator.md)). Details:
|
||||
[sprint 03](../roadmap/03.%20full%20coverage.md).
|
||||
([ADR 0001](adr/0001-concurrency-model.md)); the orchestrator
|
||||
([ADR 0002](adr/0002-lanes-and-orchestrator.md)); two connections per symbol,
|
||||
first one wins ([ADR 0003](adr/0003-first-one-wins.md)). Details:
|
||||
[sprint 04](../roadmap/04.%20redundant%20connections.md).
|
||||
|
||||
## Data model, first sketch
|
||||
|
||||
@@ -179,28 +183,20 @@ Not a DB — an in-memory domain plus the fixed SBE output schema. Main entities
|
||||
whatever the exchange offers for loss detection (a sequence number, a book
|
||||
hash), and payload (side/price/quantity, or the full book).
|
||||
- **Book** (one per symbol): bids and asks as ordered price→quantity maps, plus
|
||||
the last loss-detection marker seen, last exchange timestamp, and the id of the
|
||||
source currently trusted (for redundancy).
|
||||
- **Connection** (one websocket): the set of symbols it carries, those of its
|
||||
lane.
|
||||
- **Lane**: a set of symbols, the book thread that owns their books, and the
|
||||
connections that feed it. Live markets are assigned to lanes by
|
||||
`conditionId`; Polymarket's lifecycle signal is Gamma (settled in sprint 03:
|
||||
a market is live while active, not closed, with an order book and accepting
|
||||
orders).
|
||||
- **Source / redundancy:** several connections may feed the same symbol; one is
|
||||
the trusted source at a time.
|
||||
the last loss-detection marker seen and last exchange timestamp. It is fed
|
||||
by the symbol's two connections, first one wins.
|
||||
- **Connection** (one websocket): the set of symbols it carries. Live markets
|
||||
are placed on connections by `conditionId`, both tokens together;
|
||||
Polymarket's lifecycle signal is Gamma (settled in sprint 03: a market is
|
||||
live while active, not closed, with an order book and accepting orders).
|
||||
|
||||
Output schema is fixed by the SBE schema: `Snapshot` and `PriceLevelUpdate`.
|
||||
|
||||
The debate lives in two places: the **NormalizedEvent** shape (both exchanges
|
||||
must map cleanly onto it) and **redundancy reconciliation** (how a symbol's
|
||||
trusted source is chosen and switched, and emitting a fresh snapshot on switch).
|
||||
The debate lives in the **NormalizedEvent** shape (both exchanges must map
|
||||
cleanly onto it). Redundancy was settled in sprint 04: no source is trusted,
|
||||
the first copy of each change wins ([ADR 0003](adr/0003-first-one-wins.md)).
|
||||
|
||||
## Open questions
|
||||
|
||||
- **Loss detection on Kalshi.** Its sequence semantics: checked in M6.
|
||||
(Polymarket: settled in M2, by its book hash.)
|
||||
- **Redundancy source-selection.** How to pick/switch the trusted source
|
||||
(freshness, message rate, which source is ahead). Designed in the redundancy
|
||||
sprint.
|
||||
+26
-29
@@ -40,31 +40,25 @@ abstraction is *extracted* once a second exchange exists, not designed up front.
|
||||
*Ends:* every live Polymarket symbol tracked — base challenge satisfied for
|
||||
one exchange.
|
||||
|
||||
4. **Extension 1 — redundant websockets** — *not started*
|
||||
Multiple connections per symbol (several connections in a lane, its book
|
||||
thread choosing among them, the orchestrator coordinating); choose a
|
||||
trusted source and switch on loss/quiet/delay; emit a fresh snapshot on
|
||||
every source switch. Recovery from errors, with the lane budget scaling
|
||||
itself down when connections still get 1013 closes (no rate is safe:
|
||||
[notes.md](notes.md), so a budget set too high must correct itself).
|
||||
**Async re-evaluation checkpoint:** before building this, reassess whether the
|
||||
socket-management complexity (reconnect/timeout/select, redundancy) warrants
|
||||
swapping the thread-only runtime for tokio (see ADR 0001).
|
||||
The lane that holds markets busy together (the 5- and 15-minute crypto
|
||||
markets: closed 16 times in 75 minutes, in bursts a minute's load
|
||||
average doesn't show) is the case to beat.
|
||||
Also: surviving a hard crash without losing the writer's in-memory chunk,
|
||||
a move's Remove reaching the old lane before the new lane's first
|
||||
Snapshot, the memory growth seen in sprint 03 (~90 MB every 10 minutes
|
||||
over 75 minutes), and closing the subscription gap (the few ms of a
|
||||
symbol's changes the server doesn't send between our unsubscribe and
|
||||
subscribe); the crash chunk and the gap were accepted as ceilings in M2.
|
||||
*Ends:* redundancy demonstrably reduces data loss; a Snapshot on every source
|
||||
switch.
|
||||
4. **Extension 1 — redundant websockets** — *done (2026-11-11)*
|
||||
Every symbol on two connections chosen independently, its book taking each
|
||||
change from whichever delivers it first (first one wins); connections
|
||||
separated from book threads; a connection the server keeps closing with
|
||||
1013 is split, so a budget set too high corrects itself. The async
|
||||
checkpoint was held: threads stay
|
||||
([ADR 0001](adr/0001-concurrency-model.md)).
|
||||
Sprint: [../roadmap/04. redundant connections.md](../roadmap/04.%20redundant%20connections.md).
|
||||
Left as ceilings: the writer's in-memory chunk on a hard crash, and the
|
||||
subscription gap when a symbol opens.
|
||||
*Ends:* redundancy demonstrably reduces data loss: fewer losses and
|
||||
subscription gaps per million changes with two subscriptions than with one.
|
||||
Result: losses a quarter lower in each pair of alternating runs; losses and
|
||||
gaps together lower in one pair, even in the other (sprint 04's Outcome).
|
||||
|
||||
5. **Extension 2 — subscription monitoring** — *not started*
|
||||
Per-symbol activity monitoring (data points per second/minute); unsubscribe +
|
||||
resubscribe symbols that go too quiet; periodic full-socket recycle
|
||||
resubscribe symbols that go too quiet (with two connections, first one wins
|
||||
hides a subscription gone quiet on one of them: it has to be looked for); periodic full-socket recycle
|
||||
(disconnect → reconnect → resubscribe) to shed cumulative exchange-side bugs.
|
||||
*Ends:* monitoring + recycling running.
|
||||
|
||||
@@ -108,16 +102,19 @@ abstraction is *extracted* once a second exchange exists, not designed up front.
|
||||
- Observability: metrics, structured logging, dashboards.
|
||||
- Config-driven symbol filtering and per-exchange tuning.
|
||||
- Containerized packaging.
|
||||
- Counting a reconnect's differing books as losses rather than subscription
|
||||
gaps when the symbol was resyncing or not yet verified since its opening
|
||||
book (a reconnect marker through the frames channel; they are labelled
|
||||
gaps).
|
||||
- A `reconcile` that reads the collector's log, to tell subscription gaps from
|
||||
losses in its own report.
|
||||
- Polymarket's `new_market` websocket event as a faster discovery signal (it
|
||||
is broadcast, but lacks `orderMinSize` / `negRisk`, so Gamma is still
|
||||
needed; `market_resolved` was never seen).
|
||||
- Closing lanes that drain.
|
||||
- Closing connections that drain.
|
||||
- Verifying a symbol again from the trade event's price after a trade, with
|
||||
no resync (sprint 04's prototype did it for 15 of 100 broken chains).
|
||||
- Telling a real loss from a resync book merely ahead of our stream, so the
|
||||
loss count measures data missed (most of sprint 04's "losses" follow
|
||||
trades, as often with one connection as with two).
|
||||
- Closing the opening subscription gap with the second connection's opening
|
||||
book, when the first one's chain breaks before any change verified.
|
||||
- An exact rescued count: a change remembered past the window of 128 until
|
||||
its copies are counted (sprint 04 reports an upper bound).
|
||||
- Fewer Snapshots for unchanged books (the brief asks for every symbol every
|
||||
minute: ~130 MB a minute at full coverage).
|
||||
- Re-reading a market's `orderMinSize` / `negRisk` if they can change.
|
||||
Reference in new issue
Block a user