feat(polymarket)!: two connections per market, first one wins

Every live market is subscribed on two connections chosen at random, and its
book takes each change from whichever connection delivers it first: the book
hash says whether a change is a duplicate, the next one, or one to hold until
the change before it arrives. A change one connection misses costs nothing,
each Delta carries the earliest receive time, and a connection that drops is
covered by the other. When a change fits on no connection in time, the book
follows one connection unchecked and asks the other for a fresh book.
Connections now decode and route events to 16 book threads, independent of
them; a move leaves its old connection only once the new one delivers, so it
costs no change; a connection closed with 1013 twice within ten minutes is
split. reconcile reads the collector's log and fails only on a mismatch no
cause line explains. SIGTERM and SIGHUP stop the collector cleanly.

Breaking: lanes are gone (connections and book threads are separate), and
reconcile takes the log as a second argument.
This commit is contained in:
pascal committed 2026-11-11 15:18:48 +00:00
1 parent 8114a51dc2
commit 02747b882f
24 files changed
+8941 -1171

No files matched your search

+24 -15
View File
@@ -12,17 +12,22 @@ outside this repo.
- `docs/roadmap.md` and `roadmap/NN. name.md`: milestones and each sprint's - `docs/roadmap.md` and `roadmap/NN. name.md`: milestones and each sprint's
agreed plan. Read the current sprint file before writing code. agreed plan. Read the current sprint file before writing code.
- `docs/adr/`: hard-to-reverse decisions (0001: threads, shared-nothing; - `docs/adr/`: hard-to-reverse decisions (0001: threads, shared-nothing;
0002: lanes and the orchestrator). 0002: the orchestrator, its lanes superseded; 0003: two connections per
symbol, first one wins).
- `docs/notes.md`: domain rules and exchange protocol facts (delta semantics, - `docs/notes.md`: domain rules and exchange protocol facts (delta semantics,
loss detection, message shapes). Read before touching exchange parsing. loss detection, message shapes). Read before touching exchange parsing.
- `GLOSSARY.md`: the terms (symbol, resync, loss, subscription gap, lane, - `GLOSSARY.md`: the terms (symbol, connection, load, first one wins,
load, Snapshot, Delta). duplicate, held, rescued, follow, resync, loss, subscription gap, split,
- `src/polymarket/orchestrator.rs`: what is tracked, the lanes and their Snapshot, Delta).
balancing; `src/polymarket/gamma.rs`: the fetchers (discovery); - `src/polymarket/orchestrator.rs`: what is tracked, which connections carry
`src/polymarket/feed.rs`: a lane's book thread and its loss rules (the each market, and their balancing; `src/polymarket/gamma.rs`: the fetchers
subtle part); `src/polymarket/socket.rs`: a lane's connection; (discovery); `src/polymarket/socket.rs`: a connection (read, stamp, decode,
`src/book.rs`: the book and fixed point; `src/reconcile.rs` + `src/bin/reconcile.rs`: the output route); `src/polymarket/wire.rs`: the exchange's messages;
check. `tests/fixtures/`: a 90 s recording of the live feed the tests replay. `src/polymarket/feed.rs`: a book thread and its loss rules (the subtle
part); `src/book.rs`: the book and fixed point; `src/reconcile.rs` +
`src/bin/reconcile.rs`: the output check, reading the log's cause lines
(`src/cause.rs`). `tests/fixtures/`: recordings of the live feed, on one and two
connections, that the tests replay.
- `src/util/` and `sbe_schemas/`: the template's SBE machinery, used as given so - `src/util/` and `sbe_schemas/`: the template's SBE machinery, used as given so
output stays readable by the template's `AlignedSbeReader`. Our code is the output stays readable by the template's `AlignedSbeReader`. Our code is the
rest of `src/`. rest of `src/`.
@@ -57,17 +62,21 @@ and let the user settle it. Claude owns the data model.
A 403 is not DNS (see `docs/notes.md`, REST from scripts). A 403 is not DNS (see `docs/notes.md`, REST from scripts).
- **Write 64 MiB chunks**: `AlignedSbeReader::open` hardcodes 64 MiB - **Write 64 MiB chunks**: `AlignedSbeReader::open` hardcodes 64 MiB
(`open_with_chunk_size` is private), so any other chunk size is unreadable. (`open_with_chunk_size` is private), so any other chunk size is unreadable.
- **The collector runs until SIGINT, and only SIGINT flushes its output.** - **Only a clean stop flushes the output**: SIGINT, SIGTERM or SIGHUP. SIGKILL
Any other signal (SIGTERM, SIGALRM) leaves the last 64 MiB chunk unwritten. or SIGALRM leaves the last 64 MiB chunk unwritten. Bound a run with GNU
Bound a run with GNU `timeout -s INT`, then check the output: `timeout -s INT`, then check the output against the log:
`timeout -s INT 600 ./target/release/engineering-challenge out.sbe 2> run.log; `timeout -s INT 600 ./target/release/engineering-challenge out.sbe 2> run.log;
./target/release/reconcile out.sbe`. `reconcile` counts subscription gaps ./target/release/reconcile out.sbe run.log`. With the log, `reconcile`
as mismatches too: compare its list with the log's "loss", "subscription pairs each mismatch with its cause line and fails only on one left
gap" and "first book after a move" lines. unexplained.
- **Wait on a PID, never on a `pgrep -f` pattern**: `until ! pgrep -f - **Wait on a PID, never on a `pgrep -f` pattern**: `until ! pgrep -f
"capacity scan"` matches the waiting shell's own command line and never "capacity scan"` matches the waiting shell's own command line and never
ends, and `pkill -f PATTERN` can kill the shell running it. Take the PID ends, and `pkill -f PATTERN` can kill the shell running it. Take the PID
(`pgrep -f '^./target/release/...'`, anchored) and loop on `kill -0 $pid`. (`pgrep -f '^./target/release/...'`, anchored) and loop on `kill -0 $pid`.
- **The server is shared and gets overloaded** (load up to 92 on 12 cores,
2026-11-11): a starved collector falls behind and gets closed with 1013.
Check `cat /proc/loadavg` before a live run whose numbers are compared, and
alternate the builds in short rounds rather than running them back to back.
- **Live data beats the docs**: Polymarket's market channel has rules its - **Live data beats the docs**: Polymarket's market channel has rules its
docs don't state (`docs/notes.md`); check a protocol assumption on a docs don't state (`docs/notes.md`); check a protocol assumption on a
recording of the real feed (`tests/fixtures/record.py`) before relying on it. recording of the real feed (`tests/fixtures/record.py`) before relying on it.
+2 -1
View File
@@ -10,7 +10,8 @@ default-run = "engineering-challenge"
anyhow = "1" anyhow = "1"
chrono = "0.4" chrono = "0.4"
crossbeam = "0.8.4" crossbeam = "0.8.4"
ctrlc = "3" # `termination`: SIGTERM and SIGHUP stop the collector like SIGINT.
ctrlc = { version = "3", features = ["termination"] }
# Not used directly: tungstenite leaves rustls's crypto provider to the app, and # Not used directly: tungstenite leaves rustls's crypto provider to the app, and
# rustls panics at runtime unless exactly one (here ring) is enabled. # rustls panics at runtime unless exactly one (here ring) is enabled.
rustls = { version = "0.23", default-features = false, features = ["ring"] } rustls = { version = "0.23", default-features = false, features = ["ring"] }
+40 -15
View File
@@ -32,28 +32,53 @@ The collector's in-memory order book for one symbol: its bid and ask price
levels, each with its absolute quantity. levels, each with its absolute quantity.
_Avoid_: Order book state, depth _Avoid_: Order book state, depth
**Lane**: **Connection**:
A set of symbols whose books one book thread owns, fed by its own One websocket to the exchange. Every live symbol is subscribed on two
connections: one path from the wire to the writer. Every live symbol is in connections, chosen independently of each other.
exactly one lane. _Avoid_: Socket (for the concept; fine for the code), stream, lane (the
_Avoid_: Shard (a technique, not this thing), group, partition retired pairing of one connection with one book thread)
**Load**: **Load**:
How much a market weighs on its connection: the bytes per second of its How much a market weighs on a connection: the bytes per second of its frames
frames. A lane's load is the sum of its markets'. there. A connection's load is the sum of its markets'.
_Avoid_: Activity, volume (the exchange's traded amount, unrelated) _Avoid_: Activity, volume (the exchange's traded amount, unrelated)
**Connection**: **First one wins**:
One websocket to the exchange, carrying the symbols of one lane. The rule for a symbol's two connections: a change is taken from whichever
_Avoid_: Socket (for the concept; fine for the code), stream delivers it first, with that receive time, and the other's copy is dropped.
_Avoid_: Source, primary, standby, failover (no connection has a rank)
**Duplicate**:
A change another connection already delivered.
**Verified**:
Said of a symbol whose book gives the exchange's book hash after every
change. It stops being verified when a change fits on no connection in time.
**Held**:
Said of a change kept unwritten because it doesn't fit the book yet: a change
before it is still to come, from this connection or the other.
_Avoid_: Buffered, pending, queued
**Rescued**:
Said of a change one connection missed and the other delivered: the output
has it, where a single connection would have had a loss.
**Follow**:
To apply one connection's changes unchecked, while a symbol can't be
verified and until a fresh book says where it stands.
**Split**:
Moving half the markets off a connection the exchange keeps closing.
**Receive time**: **Receive time**:
When a message came off the socket, by the local clock. When a message came off the socket, by the local clock.
_Avoid_: Arrival time, local timestamp _Avoid_: Arrival time, local timestamp
**Loss**: **Loss**:
Any message for a symbol that the collector did not apply, which leaves its A change for a symbol that no connection delivered in time, which leaves its
book silently wrong. book silently wrong. A change only one connection missed is rescued, not
lost.
_Avoid_: Gap (implies sequence numbers) _Avoid_: Gap (implies sequence numbers)
**Resync**: **Resync**:
@@ -63,9 +88,9 @@ one differs: a loss, or a subscription gap.
_Avoid_: Re-snapshot, rebuild, refresh _Avoid_: Re-snapshot, rebuild, refresh
**Subscription gap**: **Subscription gap**:
The changes our own subscribe keeps from us, whether it opens, resyncs or The changes our own subscribe keeps from us, whether it opens or resyncs a
moves a symbol: the exchange holds them in the book it sends but never sends symbol: the exchange holds them in the book it sends but never sends them as
them as changes. changes. A connection that stays subscribed meanwhile has no such gap.
_Avoid_: Resync gap (the older, narrower name); calling it a loss (the state _Avoid_: Resync gap (the older, narrower name); calling it a loss (the state
stays right; only those updates are missing) stays right; only those updates are missing)
+4
View File
@@ -29,6 +29,10 @@ races by construction, and keeps the runtime pluggable: only the socket layer is
runtime-specific, so a later async swap is localized to that boundary rather than runtime-specific, so a later async swap is localized to that boundary rather than
a pipeline rewrite. Deferring async is therefore safe through the base challenge. a pipeline rewrite. Deferring async is therefore safe through the base challenge.
The M4 checkpoint was held in sprint 04 (2026-11-11): threads stay. The new
logic of redundancy sits in the book threads, not in socket handling, and
about 200 threads (175 connections, 16 book threads) are comfortable.
Trade-off: thread-per-socket does not scale to very large socket counts, and the Trade-off: thread-per-socket does not scale to very large socket counts, and the
extension-era socket-management logic (select / timeout / reconnect) is where extension-era socket-management logic (select / timeout / reconnect) is where
threads and async diverge most — hence the explicit M4 re-evaluation point rather threads and async diverge most — hence the explicit M4 re-evaluation point rather
+4
View File
@@ -1,5 +1,9 @@
# 0002. Lanes and an orchestrator # 0002. Lanes and an orchestrator
*Superseded in part by [0003](0003-first-one-wins.md) (sprint 04): the lanes are
gone, connections and book threads being separate sets; the orchestrator and
the fetchers stand.*
## Context ## Context
Every live Polymarket symbol is ~476k tokens (2026-11-04), an estimated Every live Polymarket symbol is ~476k tokens (2026-11-04), an estimated
+47
View File
@@ -0,0 +1,47 @@
# 0003. Two connections per symbol, first one wins
## Context
Extension 1 asks for redundant websockets: connections drop, go quiet, lag,
and get closed by the server (code 1013) when they carry too much. Sprint 03
gave each lane a connection and a book thread of its own
([0002](0002-lanes-and-orchestrator.md)). The obvious next step was a second,
identical connection per lane, one of them the trusted source, with a
Snapshot at each switch, as the brief sketches. Measured on 2026-11-11
([sprint 04](../../roadmap/04.%20redundant%20connections.md)): two connections
on the same symbols deliver the same changes in the same order, 0.5 ms apart
at the median and 100 ms or more for 1% of changes; and two connections
carrying the same symbols are closed together.
## Decision
Every symbol is subscribed on **two connections chosen independently and at
random**, and its one book takes each change from **whichever connection
delivers it first**. Polymarket's book hash, sent with every change, says
where a change belongs: already applied (a duplicate), next (it fits and is
written), or early (held until the change before it arrives). Only when a
change fits on no connection in time does the book thread fall back to
following one connection unchecked and resyncing on the other.
Connections therefore no longer belong to a book thread. A connection thread
stamps, parses and routes each event to the book thread of its market, one of
16, derived from the market's id. The orchestrator of 0002 stays; its lanes
are superseded.
## Why
First one wins gives each Delta the earliest receive time, turns a change one
connection missed into no loss at all, and needs no second book and no
Snapshot per switch. A source and a standby would have kept the source's
receive time and lost the change the source missed. Independent connections
were measured against twins on the same 48,038 symbols for 20 minutes: 272
symbol-seconds with both subscriptions down, against 3,787 for twins dealt at
random and 33,345 for twins in creation order. A move becomes a subscribe and
an unsubscribe with the book left where it is, so it opens no subscription
gap.
Trade-off: the routing step 0002 avoided is back, parsing moves into the
connection thread, and the book thread's rules get subtler (held changes, a
ceiling, a chain declared broken). The design leans on an exchange that lets
the same change be recognised across connections: Polymarket's hash does;
Kalshi's `clock` looks as if it would (to confirm in M6).
+2 -2
View File
@@ -46,8 +46,8 @@ Prefer these; they can't be gamed by strategy luck:
separately our *internal* added latency (edge-stamp → output byte). separately our *internal* added latency (edge-stamp → output byte).
- **Completeness:** reconciliation error rate (snapshot↔delta divergence); count - **Completeness:** reconciliation error rate (snapshot↔delta divergence); count
of missed deletes / stale levels under induced loss. of missed deletes / stale levels under induced loss.
- **Redundancy value (once ext1 exists):** messages the backup socket caught that - **Redundancy value:** changes one connection missed and the other delivered
the primary missed — literal "data loss avoided." (rescued) — literal "data loss avoided."
## Recommendation ## Recommendation
+26
View File
@@ -147,6 +147,32 @@ From a 60 s run of the M1 spike on one market, both tokens subscribed, and the
403 (curl and `ureq` get 200), most likely Cloudflare filtering by user agent. 403 (curl and `ureq` get 200), most likely Cloudflare filtering by user agent.
Fetch with curl in throwaway scripts. Fetch with curl in throwaway scripts.
## Two connections on the same symbols (observed 2026-11-11)
- **Same changes, same order**: two connections subscribed to the same 8,000
tokens for 335 s delivered the same 616,708 changes, none on only one, in
the same order per token. So the book hash orders changes across
connections too.
- **The second delivery's lag**: 0.5 ms at the median, over 10 ms for 6.5% of
changes, over 100 ms for 1%, 706 ms at worst; up to 82 changes of its token
behind. Neither connection leads (one was first 34% of the time).
- **Batches**: 95% of updates are one change. The others come one change per
frame, all with the hash of the whole batch, within 1 ms (40 ms at worst).
- **The server repeats some changes** on one connection (104 in that
recording, the same on both): harmless, quantities being absolute.
- **Connections carrying the same tokens are closed together**: two ramped on
the same tokens got their 1013 close 3 s apart. Over 20 minutes on 48,038
tokens with two subscriptions each, twins in creation order had both
subscriptions down for 33,345 token-seconds, twins dealt at random 3,787,
two independent random deals 272.
- **300 connections** opened at once from one address all stayed up.
- **A connection whose every subscribed market has resolved is closed** by the
server, code 1000, reason `all subscribed assets resolved` (seen on a
connection left with one market, 2026-11-11).
- **A change's `timestamp` never goes back within a token** on a connection
(0 of 660,084 changes on each of the two):
one older than the last applied is a late copy.
## Gamma, the market listing (observed 2026-11-04) ## Gamma, the market listing (observed 2026-11-04)
- **Size:** 238,308 markets `active` and not `closed`; 238,201 of them with - **Size:** 238,308 markets `active` and not `closed`; 238,201 of them with
+41 -45
View File
@@ -38,22 +38,22 @@ saves the data.
The interacting decisions it is made of: The interacting decisions it is made of:
- **Threading / runtime:** settled in sprint 01 — OS threads + channels, - **Threading / runtime:** settled in sprint 01 — OS threads + channels,
shared-nothing, async re-evaluated at M4 shared-nothing; async was re-evaluated at M4 and threads stay
([ADR 0001](adr/0001-concurrency-model.md)). ([ADR 0001](adr/0001-concurrency-model.md)).
- **Stamping point:** in the socket-read path, immediately, so as little as - **Stamping point:** in the socket-read path, immediately, so as little as
possible sits between the wire and the receive timestamp. possible sits between the wire and the receive timestamp.
- **Book ownership:** settled in sprint 03 — symbols are spread over lanes, - **Book ownership:** each book is owned by exactly one of 16 book threads,
each a connection and a book thread of its own, so each book is owned by chosen from its market's id; an orchestrator decides which connections
exactly one thread; an orchestrator assigns them carry each market ([ADR 0002](adr/0002-lanes-and-orchestrator.md), its
([ADR 0002](adr/0002-lanes-and-orchestrator.md)). lanes superseded by [ADR 0003](adr/0003-first-one-wins.md) in sprint 04).
- **Backpressure:** bounded vs unbounded channels — the core correctness/perf - **Backpressure:** bounded vs unbounded channels — the core correctness/perf
tension. Drop-on-full loses data (violates completeness); block adds latency tension. Drop-on-full loses data (violates completeness); block adds latency
and fills socket buffers. This decision is the bet in miniature. and fills socket buffers. This decision is the bet in miniature.
- **Output serialization:** one writer owns the single sink and consumes from - **Output serialization:** one writer owns the single sink and consumes from
all workers. all workers.
- **Redundancy fit:** multiple sockets feeding one symbol's worker, which - **Redundancy:** settled in sprint 04 — every symbol on two connections
dedups / selects by source — the topology must leave room for this from the chosen independently, its book taking each change from whichever delivers
start. it first ([ADR 0003](adr/0003-first-one-wins.md)).
## Requirements the pipeline must satisfy ## Requirements the pipeline must satisfy
@@ -118,7 +118,7 @@ Their operational form, for writing code, is in the project
Chunks must be 64 MiB: `AlignedSbeReader::open` reads nothing else. Chunks must be 64 MiB: `AlignedSbeReader::open` reads nothing else.
- **Crash ceiling** (accepted in sprint 02): the writer holds the current - **Crash ceiling** (accepted in sprint 02): the writer holds the current
64 MiB chunk in memory until it is full, so a hard crash loses it; flushing 64 MiB chunk in memory until it is full, so a hard crash loses it; flushing
more often pads each flush to 64 MiB. Taken up with extension 1 (M4). more often pads each flush to 64 MiB. Kept as a ceiling in extension 1 (M4).
## Workflow differences ## Workflow differences
@@ -135,34 +135,38 @@ and drop from the global workflow for a job-challenge repo.
## Pipeline ## Pipeline
The primary thing we debate sprint by sprint (this project's equivalent of the The primary thing we debate sprint by sprint (this project's equivalent of the
schema debate). As settled in sprint 03: schema debate). As settled in sprint 04:
``` ```
rescan fetcher ──findings──▶ ┌──────────────┐ ◀──findings── recent fetcher rescan fetcher ──findings──▶ ┌──────────────┐ ◀──findings── recent fetcher
(full Gamma scan) ◀─request─│ orchestrator │──request──▶ (new + closed since T) │ orchestrator │ ◀── 1013 closes ── connections
└──┬────────▲──┘ └──────┬───▲───┘
Add / MoveIn / Remove, to │ │ load reports, each minute Add / Move / Remove (a market and │ │ load reports, each minute
the market's lane │ │ (bytes/s per market) its connections), to its thread │ │ (bytes/s per market)
┌──────────────────────┼────────┴─────────────────┐ ┌──────▼───┴───┐
▼ ▼ ▼ connection 0..N ──events───▶ │ book threads │ ──Snapshot | Delta──▶ writer
lane 0: book thread ◀─frames─ connection lane 1: ... lane N: ... (read, stamp, parse, route) │ 0..15 │
│ └─subscribe / unsubscribe / resync─▶┘ ▲ └──────┬───────┘
└──────────── Snapshot | Delta ─────────────▶ writer thread (one) └── subscribe / unsubscribe / resync ┘
``` ```
- The **orchestrator** owns what is tracked (market → lane, the lanes, the - The **orchestrator** owns what is tracked (each market's two connections,
timers) and decides; it never blocks on HTTP, so later coordination the timers) and decides: placing, moving, splitting. It never blocks on
(switching sources in M4, recycling in M5) lives there too. HTTP.
- The two **fetchers** only page Gamma on request and stream what they read. - The two **fetchers** only page Gamma on request and stream what they read.
- Each **lane** is a connection (stamps receive time, sends subscriptions and - A **connection** reads, stamps the receive time, parses, and routes each
resyncs) and a book thread (fold, check, resync, Snapshots); M4 gives a lane event to the book thread of its market. Every symbol is on two connections,
several connections and has its book thread choose the trusted source. chosen independently.
- A **book thread** owns its markets' books and applies first one wins: a
change is a duplicate, the next one, or held until it fits (fold, check,
resync, Snapshots as before).
- One **writer** owns the output. - One **writer** owns the output.
Settled: OS threads, shared-nothing, no mutexes Settled: OS threads, shared-nothing, no mutexes
([ADR 0001](adr/0001-concurrency-model.md)); lanes and the orchestrator ([ADR 0001](adr/0001-concurrency-model.md)); the orchestrator
([ADR 0002](adr/0002-lanes-and-orchestrator.md)). Details: ([ADR 0002](adr/0002-lanes-and-orchestrator.md)); two connections per symbol,
[sprint 03](../roadmap/03.%20full%20coverage.md). first one wins ([ADR 0003](adr/0003-first-one-wins.md)). Details:
[sprint 04](../roadmap/04.%20redundant%20connections.md).
## Data model, first sketch ## Data model, first sketch
@@ -179,28 +183,20 @@ Not a DB — an in-memory domain plus the fixed SBE output schema. Main entities
whatever the exchange offers for loss detection (a sequence number, a book whatever the exchange offers for loss detection (a sequence number, a book
hash), and payload (side/price/quantity, or the full book). hash), and payload (side/price/quantity, or the full book).
- **Book** (one per symbol): bids and asks as ordered price→quantity maps, plus - **Book** (one per symbol): bids and asks as ordered price→quantity maps, plus
the last loss-detection marker seen, last exchange timestamp, and the id of the the last loss-detection marker seen and last exchange timestamp. It is fed
source currently trusted (for redundancy). by the symbol's two connections, first one wins.
- **Connection** (one websocket): the set of symbols it carries, those of its - **Connection** (one websocket): the set of symbols it carries. Live markets
lane. are placed on connections by `conditionId`, both tokens together;
- **Lane**: a set of symbols, the book thread that owns their books, and the Polymarket's lifecycle signal is Gamma (settled in sprint 03: a market is
connections that feed it. Live markets are assigned to lanes by live while active, not closed, with an order book and accepting orders).
`conditionId`; Polymarket's lifecycle signal is Gamma (settled in sprint 03:
a market is live while active, not closed, with an order book and accepting
orders).
- **Source / redundancy:** several connections may feed the same symbol; one is
the trusted source at a time.
Output schema is fixed by the SBE schema: `Snapshot` and `PriceLevelUpdate`. Output schema is fixed by the SBE schema: `Snapshot` and `PriceLevelUpdate`.
The debate lives in two places: the **NormalizedEvent** shape (both exchanges The debate lives in the **NormalizedEvent** shape (both exchanges must map
must map cleanly onto it) and **redundancy reconciliation** (how a symbol's cleanly onto it). Redundancy was settled in sprint 04: no source is trusted,
trusted source is chosen and switched, and emitting a fresh snapshot on switch). the first copy of each change wins ([ADR 0003](adr/0003-first-one-wins.md)).
## Open questions ## Open questions
- **Loss detection on Kalshi.** Its sequence semantics: checked in M6. - **Loss detection on Kalshi.** Its sequence semantics: checked in M6.
(Polymarket: settled in M2, by its book hash.) (Polymarket: settled in M2, by its book hash.)
- **Redundancy source-selection.** How to pick/switch the trusted source
(freshness, message rate, which source is ahead). Designed in the redundancy
sprint.
+26 -29
View File
@@ -40,31 +40,25 @@ abstraction is *extracted* once a second exchange exists, not designed up front.
*Ends:* every live Polymarket symbol tracked — base challenge satisfied for *Ends:* every live Polymarket symbol tracked — base challenge satisfied for
one exchange. one exchange.
4. **Extension 1 — redundant websockets** — *not started* 4. **Extension 1 — redundant websockets** — *done (2026-11-11)*
Multiple connections per symbol (several connections in a lane, its book Every symbol on two connections chosen independently, its book taking each
thread choosing among them, the orchestrator coordinating); choose a change from whichever delivers it first (first one wins); connections
trusted source and switch on loss/quiet/delay; emit a fresh snapshot on separated from book threads; a connection the server keeps closing with
every source switch. Recovery from errors, with the lane budget scaling 1013 is split, so a budget set too high corrects itself. The async
itself down when connections still get 1013 closes (no rate is safe: checkpoint was held: threads stay
[notes.md](notes.md), so a budget set too high must correct itself). ([ADR 0001](adr/0001-concurrency-model.md)).
**Async re-evaluation checkpoint:** before building this, reassess whether the Sprint: [../roadmap/04. redundant connections.md](../roadmap/04.%20redundant%20connections.md).
socket-management complexity (reconnect/timeout/select, redundancy) warrants Left as ceilings: the writer's in-memory chunk on a hard crash, and the
swapping the thread-only runtime for tokio (see ADR 0001). subscription gap when a symbol opens.
The lane that holds markets busy together (the 5- and 15-minute crypto *Ends:* redundancy demonstrably reduces data loss: fewer losses and
markets: closed 16 times in 75 minutes, in bursts a minute's load subscription gaps per million changes with two subscriptions than with one.
average doesn't show) is the case to beat. Result: losses a quarter lower in each pair of alternating runs; losses and
Also: surviving a hard crash without losing the writer's in-memory chunk, gaps together lower in one pair, even in the other (sprint 04's Outcome).
a move's Remove reaching the old lane before the new lane's first
Snapshot, the memory growth seen in sprint 03 (~90 MB every 10 minutes
over 75 minutes), and closing the subscription gap (the few ms of a
symbol's changes the server doesn't send between our unsubscribe and
subscribe); the crash chunk and the gap were accepted as ceilings in M2.
*Ends:* redundancy demonstrably reduces data loss; a Snapshot on every source
switch.
5. **Extension 2 — subscription monitoring** — *not started* 5. **Extension 2 — subscription monitoring** — *not started*
Per-symbol activity monitoring (data points per second/minute); unsubscribe + Per-symbol activity monitoring (data points per second/minute); unsubscribe +
resubscribe symbols that go too quiet; periodic full-socket recycle resubscribe symbols that go too quiet (with two connections, first one wins
hides a subscription gone quiet on one of them: it has to be looked for); periodic full-socket recycle
(disconnect → reconnect → resubscribe) to shed cumulative exchange-side bugs. (disconnect → reconnect → resubscribe) to shed cumulative exchange-side bugs.
*Ends:* monitoring + recycling running. *Ends:* monitoring + recycling running.
@@ -108,16 +102,19 @@ abstraction is *extracted* once a second exchange exists, not designed up front.
- Observability: metrics, structured logging, dashboards. - Observability: metrics, structured logging, dashboards.
- Config-driven symbol filtering and per-exchange tuning. - Config-driven symbol filtering and per-exchange tuning.
- Containerized packaging. - Containerized packaging.
- Counting a reconnect's differing books as losses rather than subscription
gaps when the symbol was resyncing or not yet verified since its opening
book (a reconnect marker through the frames channel; they are labelled
gaps).
- A `reconcile` that reads the collector's log, to tell subscription gaps from
losses in its own report.
- Polymarket's `new_market` websocket event as a faster discovery signal (it - Polymarket's `new_market` websocket event as a faster discovery signal (it
is broadcast, but lacks `orderMinSize` / `negRisk`, so Gamma is still is broadcast, but lacks `orderMinSize` / `negRisk`, so Gamma is still
needed; `market_resolved` was never seen). needed; `market_resolved` was never seen).
- Closing lanes that drain. - Closing connections that drain.
- Verifying a symbol again from the trade event's price after a trade, with
no resync (sprint 04's prototype did it for 15 of 100 broken chains).
- Telling a real loss from a resync book merely ahead of our stream, so the
loss count measures data missed (most of sprint 04's "losses" follow
trades, as often with one connection as with two).
- Closing the opening subscription gap with the second connection's opening
book, when the first one's chain breaks before any change verified.
- An exact rescued count: a change remembered past the window of 128 until
its copies are counted (sprint 04 reports an upper bound).
- Fewer Snapshots for unchanged books (the brief asks for every symbol every - Fewer Snapshots for unchanged books (the brief asks for every symbol every
minute: ~130 MB a minute at full coverage). minute: ~130 MB a minute at full coverage).
- Re-reading a market's `orderMinSize` / `negRisk` if they can change. - Re-reading a market's `orderMinSize` / `negRisk` if they can change.
+443
View File
@@ -0,0 +1,443 @@
# 04. Redundant connections: first one wins
**Status: done (2026-11-11)**
Every symbol is subscribed on two connections, and the first one to deliver a
change wins: it is written with that receive time, and the other's copy is
dropped. A change lost on one connection is then no loss, a slow connection
costs no time accuracy, and a connection that drops is covered by the other.
Connections stop belonging to book threads, so a symbol's two subscriptions
have different neighbours, and a connection the server keeps closing is split.
This is Extension 1 of the brief. Design:
[../docs/project.md](../docs/project.md) (pipeline),
[ADR 0003](../docs/adr/0003-first-one-wins.md) (this design; supersedes the
lanes of ADR 0002), [../docs/notes.md](../docs/notes.md) (protocol facts),
[../GLOSSARY.md](../GLOSSARY.md) (first one wins, duplicate, verified, held,
follow, rescued, split; loss, narrowed).
## Measured before planning (2026-11-11)
- **Two connections on the same 8,000 symbols, 335 s**: 616,708 changes, none
on only one connection, the same order per symbol on both. The second
delivery came 0.5 ms later at the median, more than 10 ms later for 6.5% of
changes, more than 100 ms for 1%, 706 ms at worst, and up to 82 changes of
its symbol behind. Neither connection led: one was first 34% of the time.
95% of updates were one change; the others came one change per frame, all
with the hash of the whole batch, within 1 ms (40 ms at worst).
- **Twins are closed together**: two connections ramped on the same symbols
got their 1013 close 3 s apart, at the same load.
- **Three layouts of the same 48,038 newest symbols, 20 minutes, two
subscriptions per symbol** (connections of 5,000 symbols, reconnecting after
1 s):
| Layout | Closes | Symbol-seconds with both down | Symbols ever with both down |
|---|---|---|---|
| Twins, symbols in creation order | 49 | 33,345 | 5,000 |
| Twins, symbols dealt randomly | 15 | 3,787 | 5,000 |
| Two independent random deals | 20 | 272 | 290 |
In creation order every close hit the pair holding the day's 5- and
15-minute crypto markets (~2.5 MB/s). The test connections carried 450 to
900 KB/s, above our budget: the ratios count, not the totals.
- **300 connections** opened at once from the server all stayed up. Its link
carried 50 MB/s on one download; two subscriptions per symbol are an
estimated 25 MB/s.
- **A prototype of the rules below on the two-connection recording** (Python,
real hash checks, 7,729 symbols): 340,424 changes written verified, 338,609
duplicates dropped, ~36,000 changes held then fit (batches), 100 chains
broken, all with both connections holding the same change (trades). With
1% of one connection's changes removed, and with one connection delayed by
2 s, the output kept the single connection's order per symbol. The delay
case first failed (the late changes were held, the chain declared broken,
the slow connection followed): the stale rule and "the book moved on" rule
below come from it.
- **Sprint 03's 75-minute run**: 151,897 resyncs for 169 M verified changes;
11,605 of them opened a subscription gap. 7 of its 31 closes came within
0.2 s after a clock minute ending in 4, 9, 0 or 5: bursts a minute's load
average doesn't show.
## Pipeline
```
rescan fetcher ──findings──▶ ┌──────────────┐ ◀──findings── recent fetcher
│ orchestrator │ ◀── 1013 closes ── connections
└──────┬───▲───┘
Add / Move / Remove (a market and │ │ load reports, each minute
its connections), to its thread │ │ (bytes/s per market)
┌──────▼───┴───┐
connection 0..N ──events───▶ │ book threads │ ──Snapshot | Delta──▶ writer
(read, stamp, parse, route) │ 0..15 │
▲ └──────┬───────┘
└── subscribe / unsubscribe / resync ┘
```
- **Connection** (one thread each, ~175 at today's size): reads a frame,
stamps its receive time, parses it, and sends each event to the book thread
of its market, with its own number and the frame's bytes shared among its
events. It tells every book thread when it comes up or goes down, and the
orchestrator when the server closes it with 1013. It keeps its token set and
subscribes it, in operations of 500, on every connect. Limits: 5,000 tokens
and 300 KB/s, as sprint 03's lanes had.
- **Book thread** (16): owns the books of its markets, a market's thread being
derived from its `conditionId` (16 hex digits of it, modulo 16), so every
thread computes it without a shared table. It applies first one wins
(below), writes Deltas and Snapshots, and is the one that asks connections
to subscribe, unsubscribe and resync: it knows a symbol before its book can
arrive. Each wall-clock minute it writes the minute Snapshots and reports
its markets' loads.
- **Orchestrator**: holds each market's two connections, places and moves
subscriptions, splits a connection, and runs the fetchers as in sprint 03.
- **Writer**: unchanged.
## First one wins
A symbol whose hash chain holds is **verified**. A change arriving from any of
its connections is then, in this order:
1. a **duplicate**, if it is one of the symbol's last 128 applied changes
(same hash, side, price and size): dropped;
2. **stale**, if its exchange timestamp is older than the last applied
change's: dropped (while the chain holds, everything up to the book's state
was applied);
3. otherwise **held** on its connection. The shortest tail of that
connection's held changes which, applied to the book, gives the hash of its
last change **fits**: it is applied and written; held changes before the
tail are stale and dropped; a change that leaves the book as it was is not
written. One rule covers a batch completing, a change the other connection
delivers after this one ran ahead, and late changes after a fresh book.
A held change is written with its own receive time, never earlier than the
symbol's previous Delta, so replaying the output by time keeps the order.
**The ceiling.** A change may be held for 1 s; for 50 ms once every connection
of the symbol that is up holds the same changes (nobody is left to wait for
but the rest of a batch); at once when a connection holds more than 512
changes of the symbol. Only tails that start a batch are tried against the
book: the whole held list and the last four. At the ceiling, if the book moved since the change
arrived, its connection is the one behind and its held changes are dropped.
If the book did not move, the chain is **broken**.
**A broken chain.** The book thread **follows** the connection whose held
changes reached the ceiling: its held changes and its later ones are applied
and written unchecked, the other connection's are dropped, and a resync goes
to the other connection, so the followed stream is not interrupted (to the
followed one when it is the only one up). The fresh book is compared as in
sprint 02: equal to ours or a state we passed, ours stands; otherwise it
replaces ours, a Snapshot is written and the difference is logged as a loss.
Either way the symbol is verified again. A symbol whose complete book fails
its own hash stays unverifiable, following one connection, as before.
**Full books.** Only books we asked for count: a subscribe's opening book
while the symbol awaits one (the first complete book from any of its
connections wins and is the symbol's first Snapshot), and a resync's answer.
Any other book is ignored: the second connection's opening book, a
reconnect's replay for verified symbols, the books the server sends after a
trade.
**Connections dropping.** A verified symbol needs nothing when one of its
connections drops: the other carries on, and the dropped one's held changes
are forgotten. When the followed connection drops, the other is followed and
asked for a fresh book, and a Snapshot is always written on it, as the brief
asks for a change of source. When every connection of a symbol is down, the
symbol awaits a book again: the first reconnect's opening book is taken as
fresh, and one that differs from ours is written as a Snapshot and logged as
a loss.
## Placing, moving, splitting
- **Startup**: the first rescan completes (1 to 2.5 minutes on the server,
Gamma's speed varying), then the markets are shuffled and dealt, each to two different
connections, at about 5,000 markets a second so the opening books fit the
link. Connections are opened for a fill of 4,500 tokens each.
- **A new market** goes on two different random connections with room in both
limits; a connection opens when fewer than two have room.
- **Load**: a market's load is the bytes per second of its frames on one
connection: its book thread counts every event's share from all of its
connections and divides by their number. A connection's load is the sum of
its markets'.
- **Rebalancing**, each minute: a connection over its budget has the busiest
markets' subscriptions moved, each to a random connection with room that
does not already carry the market: at most 50 moves a minute, a moved
subscription staying 10 minutes, none moved for a market with no load.
- **A move** is `Move(market, from, to)` to the market's book thread, which
subscribes on `to`, and unsubscribes on `from` once `to` has delivered a
book of the market: until then both carry it, so no change falls between
them. The book stays where it is: no Snapshot, no subscription gap, and the
new connection's opening book is ignored like any unasked one. While a
symbol is verified, a change is taken from any connection, the one it left
included: its copies in flight are the exchange's changes like any other.
- **A split**: a connection closed with 1013 twice within 10 minutes has every
second market, in load order, moved off at once, outside the two caps, and
is then left alone for 10 minutes.
- **Removal**: unchanged; the book thread unsubscribes both connections.
## Data model
- **Orchestrator**: `conditionId` → (tokens, its two connections, load, when
it last moved: both its subscriptions then stay); per connection, its token count, its load,
its last 1013 closes and its last split. The market stays the unit: both
tokens share the two connections.
- **Book thread**, per symbol: the book and the hash inputs (sprint 02); its
last 128 changes, each with its hash as 20 bytes (64 changes with the hash
as text today: ~2.4 GB at worst instead of ~5 GB); the held changes of each
of its connections; its state (awaiting a book, verified, following a
connection, unverifiable and following one); its connections; its last
Delta's receive time. Per market, the bytes since its last report. Per
connection, whether it is up.
- **Event** (connection → book thread): receive time, connection number,
bytes, and a parsed book, change, tick size change or trade; or a
connection up / down marker.
- **Cause lines**: every Snapshot written because a book differed, and every
Snapshot written for a change of followed connection, logs one line with
its symbol, its exact receive time and its cause (loss, subscription gap,
switch). One module, `src/cause.rs`, holds their format and their parser.
- **A tick size change** breaks the chain on the connection that delivered
it: the hashes change with the tick size, and the fresh book carries the
new one.
## Reconcile and the measures
- `reconcile OUTPUT [LOG]`: with the log, each mismatch is paired with the
cause line of its symbol and receive time; it prints the counts by cause
and fails on an unexplained mismatch or an orphan Delta. Without, every
mismatch is unexplained.
- Each book thread's minute line: changes written (and verified), duplicates
dropped, changes **rescued** (verified changes of a symbol on several
connections that only one delivered, counted once 2 s old, when every copy
has come), held changes dropped, chains broken, resyncs, books behind,
subscription gaps, losses, switches, the second copies' lag (median and
99th percentile), the most events waiting.
- The orchestrator's: markets, connections, added, removed, moved, splits, the
busiest connection's load.
## Decisions
- **First one wins, on one book per symbol** (ADR 0003). The book hash orders
changes across connections as a sequence number would. *Rejected:* a source
and a standby with a book each and a Snapshot per switch (my first
proposal): it loses the change the source missed, keeps the source's
receive time, and doubles the books.
- **A symbol's two connections are chosen independently** (ADR 0003;
supersedes ADR 0002's lanes, a connection and a book thread of its own,
because twins were closed together: 14 times more time with both down than
independent deals). *Rejected:* twin connections per lane.
- **Markets are placed at random** (supersedes "the least-loaded lane", and
"subscribe page by page at startup", because both build the clusters a load
average cannot see: random twins had a third of the closes of creation
order). Startup scans first, then deals. *Rejected:* least-loaded placement
(a batch of 1,000 markets created together lands on one connection);
dealing while scanning (needs the connection count before the scan ends).
- **The connection thread parses and routes.** The stamp still comes first; a
frame of several markets' books must be taken apart anyway; and the
connection becomes the exchange-specific decoder Extension 3 wants.
*Rejected:* forwarding the text after peeking at the market id.
- **16 book threads, a market's thread from its id.** One handled ~30,000
frames a second in sprint 02. *Rejected:* a thread per connection (the
lanes); a routing table (shared state).
- **Subscriptions go through the market's book thread**, which records the
market before asking. *Rejected:* the orchestrator telling connections
directly (a book can arrive before its thread knows the symbol).
- **Ceilings of 1 s and 50 ms; at the ceiling, held changes are dropped if the
book moved, else the chain is broken; a change older than the book is
dropped at once.** The lag reached 0.7 s, batches 40 ms. The last two come
from the prototype: a connection 2 s late was otherwise taken for a broken
chain and followed. *Rejected:* breaking at once (a change one connection
missed would cost a resync instead of nothing); no ceiling (a change could
wait unwritten for minutes).
- **A connection holds at most 512 changes of a symbol, and only the tails
starting a batch are tried, five at most** (found by the alternating runs:
at a 5-minute mark a crypto market's burst, behind one missing change, made
every arrival try every tail of thousands, each a hash; one book thread
then needed 9.5 minutes to drain after SIGINT). *Rejected:* trying every
tail (quadratic); no cap (a burst piles up for its whole ceiling).
- **Only asked-for books count.** A reconnect replays up to 5,000 books and
the server sends books after trades; reacting to them is what made
resyncs. *Rejected:* comparing every book.
- **The resync goes to the connection not followed**, so the followed stream
has no gap of its own. *Rejected:* resyncing the followed connection
(sprint 03: 11,605 gaps).
- **A split after two 1013 closes within 10 minutes.** One close can be the
server's moment (several connections within a second); a connection closed
again is overloaded. *Rejected:* splitting on the first close; lowering
every connection's budget.
- **Threads, not tokio** (ADR 0001's checkpoint, held here): the new logic is
in the book thread, not in socket handling, and ~200 threads are
comfortable. *Rejected:* an async socket layer now.
- **Two subscriptions per market, a constant**; the comparison run is the same
code at 1. *Rejected:* a runtime option.
- **Loss means a change no connection delivered in time**; one a single
connection missed is rescued. "Lane" is retired: connections and book
threads are separate things now.
- **`reconcile` reads the log, through one module shared with the
collector.** *Rejected:* a second, machine-readable output.
- **Rescued counts copies, not held changes** (supersedes "held, then fit
after another connection moved the book", because a change one connection
missed usually arrives on the other before the first's next change, and is
then never held: the tests' rescues counted 0). *Rejected:* counting held
changes that fit (most rescues go unseen).
- **A move leaves the old connection once the new one delivers a book**
(supersedes "subscribes on `to` and then unsubscribes on `from`", because
unsubscribing at once loses the changes between `from`'s unsubscribe and
`to`'s first book). *Rejected:* a fixed delay before unsubscribing.
- **SIGTERM (and SIGHUP) stop the collector like SIGINT** (`ctrlc`'s
`termination` feature): `timeout`, `docker stop` and a service manager send
SIGTERM, and only a clean stop flushes the last chunk.
- **Built in two steps, with a live run between**: first the connections
separated from the book threads with one subscription per market, which
must behave like sprint 03's collector; then the second subscription, first
one wins and the split.
## Tests
No network, except the local websocket server of sprint 03's reconnect test.
The fixture is re-recorded on two connections (`record.py`, extended) for the
same 40 tokens; replays feed both in receive order.
- **Same output**: two connections give the Deltas, in order per symbol, that
one alone gives; the other's copies are counted as duplicates.
- **A loss on one connection**: a frame removed from one changes nothing in
the output, and its changes count as rescued.
- **A loss on both**: the same frame removed from both leaves the next change
held; at the ceiling the chain breaks, the resync goes to the connection not
followed, and a differing fresh book is written as a Snapshot, logged a loss.
- **A slow connection**: one delayed by 2 s gives the same output, each Delta
with the earlier receive time, and no chain broken by the delay.
- **Stamps**: a held change is not stamped earlier than the symbol's previous
Delta.
- **A batch** is written only once it fits; a fresh book's late changes are
dropped as stale.
- **Books**: an unasked book changes nothing; the first of two opening books
is the Snapshot.
- **Connections dropping**: one down, the output goes on; the followed one
down, a Snapshot on the fresh book; both down, every symbol awaits a book.
- **Routing**: a frame of several markets' books reaches each market's book
thread; a market's thread is the same from the orchestrator and from a
connection.
- **Placement**: a market's two connections differ; a move never targets the
market's other connection; a connection opens when fewer than two have
room; the startup deal is even.
- **Move**: subscribe on the new connection, then unsubscribe on the old; no
Snapshot.
- **Split**: two closes within 10 minutes move every second market; one close
moves none; a second split waits 10 minutes.
- **Load**: a market on two connections reports the load of one.
- **Reconcile with a log**: explained mismatches pass; an unexplained one
fails.
- Sprint 03's tests of Gamma, the minute check, rescan removal, rebalancing's
caps and the reconnect stay, adapted to connections.
## Not in this sprint
- Acting on delay: the second delivery's lag is measured and logged; choosing
connections by it is M7's.
- The crash chunk: a hard crash still loses up to 64 MiB, ~10 s of output.
- Balancing on a market's peak second rather than its minute average.
- Reading the last trade price from the trade event to verify again without
a resync (the prototype did it for 15 of 100 broken chains): a Possible
feature.
- Detecting a subscription gone quiet on one connection and resubscribing it,
and recycling connections: M5. First one wins hides a quiet subscription;
M5 has to find it.
- A market's `min_order_size` or `neg_risk` changing: still read once.
## Done when
- `scripts/ci.sh` is green.
- **Step 1** (one subscription per market): a 10-minute live run has every
`reconcile` mismatch explained by the log.
- **The comparison**, started when the user says so: two 20-minute live runs
back to back, one subscription per market, then two. In the second:
- every `reconcile` mismatch is explained by the log;
- losses and subscription gaps per million changes are below the first
run's;
- rescued changes, duplicates, chains broken and the second delivery's lag
are reported;
- a split was seen, or its absence explained;
- peak memory and thread count are reported.
- Free disk is checked before each run, and the outputs deleted after.
- ADR 0003 is written, ADR 0001 records the checkpoint, ADR 0002 says what
was superseded; `GLOSSARY.md`, `docs/project.md`'s pipeline, `docs/notes.md`
(the measures above) and the roadmap match.
## Outcome
Every live market (~198k, ~396k symbols) is on two connections chosen at
random (~178 connections), and 16 book threads take each change from
whichever connection delivers it first. Built in the two planned steps;
step 1 (connections separated, one subscription) reconciled fully in a
10-minute live run, no connection lost.
**The comparison** was run as four alternating 10-minute rounds, one
subscription per market then two, twice (asked by the user once the server
proved overloaded during a first back-to-back pair). Per million changes
written:
| Round | Subscriptions | Losses | Subscription gaps | Losses + gaps | Chains broken |
|---|---|---|---|---|---|
| 1 | 1 | 69.1 | 163.7 | 232.8 | 525 |
| 2 | 2 | 49.7 | 160.2 | 209.9 | 566 |
| 3 | 1 | 42.1 | 142.0 | 184.1 | 530 |
| 4 | 2 | 32.3 | 152.1 | 184.5 | 503 |
Every round reconciled with each mismatch explained by its log line. Losses
fell by a quarter in each pair; losses and gaps together fell in the first
pair and tied in the second, the rates drifting down through the hour. The
user accepted this as the sprint's result. Subscription gaps, mostly from
opening subscribes that both connections share, are untouched by a second
connection. Most "losses" are likely not data missed: chains break as often
with one connection as with two (about the rate of trades), and a resync's
book that is a few changes ahead of our stream is logged a loss too.
Duplicates: one per change written. The second copy came 1 to 2 ms after
the first at the median, 45 to 120 ms at p99. Peak memory 1.78 GB with two
subscriptions, 1.67 GB with one; 200 threads. No split: no connection was
closed twice within 10 minutes (0 and 2 connections lost in the
two-subscription rounds). The first, overloaded pair (load up to 92 on 12
cores) still reconciled fully: a stress test passed.
**Rescued** (~110,000 per round) is an upper bound, accepted as such: a
change that leaves the last 128 remembered before its second copy comes, on
a busy symbol, is counted as delivered once; so is a batch whose changes the
two connections delivered interleaved (the held copy dropped uncounted).
**Fixed at the close**, from the code review: a move ending while a fresh
book was asked from the connection left kept asking it in vain (now asked
from another); a change held across a minute could be stamped before that
minute's Snapshot (a Snapshot now advances the symbol's last stamp); markets
closed during the first scan were dealt anyway. The first two have a test
that fails without the fix; the third sits in the orchestrator's loop, as
untested as the rest of it.
**Diverged from the plan**, each in Decisions: markets on independent
connections, not twins (the user's design, after the three-layout test);
rescued counted from copies; a move leaves the old connection once the new
one delivers a book; the held-change cap and batch-start tails (found by the
runs: a crypto burst behind one missing change made a book thread fall
455,000 events behind and take 9.5 minutes to stop); a tick size change
breaks the chain; cause lines carry the exact receive time.
**Left out**, and where it went: telling a real loss from a resync book
ahead of our stream, and the opening subscription gaps (Possible features);
an exact rescued count (Possible features); acting on delay (M7); the crash
chunk (M4's ceiling, kept); quiet subscriptions hidden by first one wins
(M5).
### Retrospective
- **Went well:** experiments settled the design before code: two recordings
showed the hash orders changes across connections, the three-layout test
turned a twin design into the user's independent one, a Python prototype
found the slow-connection flaw before any Rust. `reconcile` reading the log
made every live run a correctness check, and it held even under load.
- **Went wrong:** a quadratic loop marked "fine at a few dozen" met bursts of
thousands; it surfaced only as a book thread falling behind, then as a
shutdown that took 9.5 minutes. The first comparison pair ran on an
overloaded server and wasn't comparable. Wait loops on `pgrep -f` failed
twice more, once killing the wrong process.
- **Tested enough?** The rules are, on a two-connection recording with
injected losses, delays and drops, and a burst test that hung before the
fix. Not covered: the orchestrator's loop (pacing the deal, routing
commands, splits sent), the socket's up/down markers, a move whose new
connection never delivers, and the minute line's lag figures.
- **Next time:** load-test the book thread with a synthetic burst before a
live run, and check the server's load before a comparison; keep the
ceiling of every corner cut in a test, not just a comment.
+17 -9
View File
@@ -1,11 +1,14 @@
//! `reconcile OUTPUT`: runs the reconciliation check on a collector's output //! `reconcile OUTPUT [LOG]`: runs the reconciliation check on a collector's
//! and fails if any Snapshot disagrees with the Deltas before it. //! output and fails on an orphan Delta, or on a Snapshot that disagrees with
//! the Deltas before it, unless the collector's log explains it.
fn main() -> anyhow::Result<()> { fn main() -> anyhow::Result<()> {
let path = std::env::args() let mut args = std::env::args().skip(1);
.nth(1) let path = args
.ok_or_else(|| anyhow::anyhow!("usage: reconcile OUTPUT"))?; .next()
let report = engineering_challenge::reconcile::reconcile(path)?; .ok_or_else(|| anyhow::anyhow!("usage: reconcile OUTPUT [LOG]"))?;
let log = args.next().map(std::path::PathBuf::from);
let report = engineering_challenge::reconcile::reconcile(path, log.as_deref())?;
println!( println!(
"{} snapshots, {} deltas, {} orphan deltas, {} mismatches", "{} snapshots, {} deltas, {} orphan deltas, {} mismatches",
report.snapshots, report.snapshots,
@@ -13,11 +16,16 @@ fn main() -> anyhow::Result<()> {
report.orphan_deltas, report.orphan_deltas,
report.mismatches.len() report.mismatches.len()
); );
for (symbol, received_at) in &report.mismatches { let mut explained: Vec<_> = report.explained.iter().collect();
println!("mismatch: {symbol} at {received_at}"); explained.sort_by_key(|(cause, _)| cause.to_string());
for (cause, count) in explained {
println!("explained by {cause}: {count}");
}
for (symbol, received_at) in &report.unexplained {
println!("unexplained: {symbol} at {received_at}");
} }
anyhow::ensure!( anyhow::ensure!(
report.mismatches.is_empty() && report.orphan_deltas == 0, report.unexplained.is_empty() && report.orphan_deltas == 0,
"the output doesn't reconcile" "the output doesn't reconcile"
); );
Ok(()) Ok(())
+80
View File
@@ -0,0 +1,80 @@
//! Why a Snapshot outside the minute was written: one log line per Snapshot,
//! written by the collector and read back by `reconcile`, which pairs it with
//! a mismatch by symbol and receive time.
use std::fmt;
use chrono::{DateTime, Utc};
/// What a Snapshot that may differ from the book before it stands for.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub enum Cause {
/// A fresh book differed from ours: changes no connection delivered.
Loss,
/// A fresh book differed from ours before any change of the symbol was
/// verified: changes our own subscribe kept from us.
SubscriptionGap,
/// The connection followed changed: the new one's book, which may hold
/// changes the old one never delivered.
Switch,
}
impl Cause {
const ALL: [Cause; 3] = [Cause::Loss, Cause::SubscriptionGap, Cause::Switch];
fn name(self) -> &'static str {
match self {
Cause::Loss => "loss",
Cause::SubscriptionGap => "subscription gap",
Cause::Switch => "switch",
}
}
}
impl fmt::Display for Cause {
fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
f.write_str(self.name())
}
}
const MARK: &str = "Snapshot of ";
/// The line's text: `Snapshot of <symbol> at <receive time, µs>: <cause>`,
/// then any detail for a reader.
pub fn line(symbol: &str, received_at: DateTime<Utc>, cause: Cause, detail: &str) -> String {
let at = received_at.timestamp_micros();
format!("{MARK}{symbol} at {at}: {cause}{detail}")
}
/// The symbol, receive time and cause of a log line written from [`line`],
/// wherever it sits in the line.
pub fn parse(log_line: &str) -> Option<(String, DateTime<Utc>, Cause)> {
let rest = &log_line[log_line.find(MARK)? + MARK.len()..];
let (symbol, rest) = rest.split_once(" at ")?;
let (at, rest) = rest.split_once(": ")?;
let received_at = DateTime::from_timestamp_micros(at.parse().ok()?)?;
let cause = Cause::ALL
.into_iter()
.find(|cause| rest.starts_with(cause.name()))?;
Some((symbol.to_string(), received_at, cause))
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn a_line_reads_back() {
let at = DateTime::from_timestamp_micros(1_794_390_726_365_123).unwrap();
for cause in Cause::ALL {
let text = line("123", at, cause, " (its time 12:00:00.000)");
let logged = format!("12:00:00.001 polymarket: {text}");
assert_eq!(parse(&logged), Some(("123".to_string(), at, cause)));
}
assert_eq!(
parse("12:00:00.001 polymarket: connection 3: connected"),
None
);
assert_eq!(parse("Snapshot of 1 at soon: loss"), None);
}
}
+1
View File
@@ -1,4 +1,5 @@
pub mod book; pub mod book;
pub mod cause;
pub mod reconcile; pub mod reconcile;
pub mod util; pub mod util;
+10 -8
View File
@@ -1,7 +1,7 @@
//! The collector (roadmap/03): an orchestrator keeps the tracked markets equal //! The collector (roadmap/04): an orchestrator keeps the tracked markets equal
//! to Gamma's live ones and spreads them over lanes, each a connection (read, //! to Gamma's live ones and places them on connections (read, stamp, decode,
//! stamp) and a book thread (parse, fold, check, resync), all writing through //! route), which feed the book threads (fold, check, resync), all writing
//! one writer (SBE). //! through one writer (SBE).
//! Usage: `engineering-challenge [OUTPUT]`, OUTPUT defaulting to stdout; logs //! Usage: `engineering-challenge [OUTPUT]`, OUTPUT defaulting to stdout; logs
//! go to stderr. //! go to stderr.
@@ -43,9 +43,10 @@ fn main() -> anyhow::Result<()> {
writer_stop.store(true, Ordering::Relaxed); writer_stop.store(true, Ordering::Relaxed);
written written
}); });
// SIGINT stops the orchestrator and the connections; each book thread // SIGINT (and SIGTERM, SIGHUP) stops the orchestrator and the
// stops when its connection does, and the writer flushes once they all // connections; the book threads stop once every connection has, and the
// have, even after a panic upstream (the run still fails then). // writer flushes once they all have, even after a panic upstream (the run
// still fails then).
let failed = polymarket::orchestrator::run(outputs_tx, &stop); let failed = polymarket::orchestrator::run(outputs_tx, &stop);
let written = writer let written = writer
.join() .join()
@@ -60,7 +61,8 @@ fn main() -> anyhow::Result<()> {
/// thread is gone, then flushes the last chunk. /// thread is gone, then flushes the last chunk.
// ponytail: the chunk being filled lives only in memory until it is full, so // ponytail: the chunk being filled lives only in memory until it is full, so
// a hard crash loses it; flushing more often pads every flush to 64 MiB. // a hard crash loses it; flushing more often pads every flush to 64 MiB.
// Taken up in M4. // Kept as a ceiling in M4 (roadmap/04): writing to a file of our own and
// copying it into chunks would lift it.
fn write(path: &str, outputs: Receiver<Output>) -> anyhow::Result<()> { fn write(path: &str, outputs: Receiver<Output>) -> anyhow::Result<()> {
let mut writer = AlignedSbeWriter::create(path, CHUNK)?; let mut writer = AlignedSbeWriter::create(path, CHUNK)?;
for output in outputs { for output in outputs {
+1967 -673
View File
File diff suppressed because it is too large. Load diff
+37 -2
View File
@@ -1,10 +1,25 @@
//! Polymarket: which markets are live (Gamma), the orchestrator that spreads //! Polymarket: which markets are live (Gamma), the orchestrator that places
//! them over lanes, and each lane's connection and feed (roadmap/03). //! them on connections, the connections, and the book threads they feed
//! (docs/adr/0003-first-one-wins.md).
pub mod feed; pub mod feed;
pub mod gamma; pub mod gamma;
pub mod orchestrator; pub mod orchestrator;
pub mod socket; pub mod socket;
pub mod wire;
/// How many book threads own the books. One handled ~30,000 frames a second
/// in sprint 02.
pub const BOOK_THREADS: usize = 16;
/// The book thread that owns a market's books, from its `conditionId` (`0x`
/// and 64 hex digits, a hash: any 16 of them spread evenly). Computed, so the
/// orchestrator and every connection agree without a shared table.
pub fn book_thread(condition_id: &str) -> usize {
let digits = condition_id.get(2..18).unwrap_or_default();
// An id of another shape still gets a thread, always the same one.
u64::from_str_radix(digits, 16).unwrap_or(0) as usize % BOOK_THREADS
}
/// A live market: its symbols and the two inputs of their book hash that the /// A live market: its symbols and the two inputs of their book hash that the
/// websocket doesn't carry (docs/notes.md). /// websocket doesn't carry (docs/notes.md).
@@ -23,3 +38,23 @@ pub struct Market {
pub(crate) fn fixture_markets() -> Vec<Market> { pub(crate) fn fixture_markets() -> Vec<Market> {
gamma::parse_markets(include_str!("../../tests/fixtures/gamma-markets.json")) gamma::parse_markets(include_str!("../../tests/fixtures/gamma-markets.json"))
} }
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn a_market_always_gets_the_same_book_thread_and_markets_spread() {
let markets = fixture_markets();
let threads: std::collections::BTreeSet<usize> = markets
.iter()
.map(|market| book_thread(&market.condition_id))
.collect();
assert!(threads.iter().all(|&thread| thread < BOOK_THREADS));
// 20 markets over 16 threads: several threads, not one.
assert!(threads.len() > 5, "{threads:?}");
let id = &markets[0].condition_id;
assert_eq!(book_thread(id), book_thread(&id.clone()));
assert_eq!(book_thread("not an id"), 0);
}
}
File diff suppressed because it is too large. Load diff
+152 -26
View File
@@ -1,6 +1,6 @@
//! A lane's connection: reads the market channel and stamps each frame's //! A connection: reads the market channel, stamps each frame's receive time,
//! receive time, and sends what its book thread asks for (subscriptions, //! decodes it and sends each event to the book thread of its market; and
//! resyncs). Parsing is the book thread's. //! sends the exchange what the book threads ask for (subscriptions, resyncs).
use std::collections::BTreeSet; use std::collections::BTreeSet;
use std::io::ErrorKind; use std::io::ErrorKind;
@@ -13,8 +13,12 @@ use chrono::{DateTime, Utc};
use crossbeam::channel::{Receiver, Sender}; use crossbeam::channel::{Receiver, Sender};
use serde_json::json; use serde_json::json;
use tungstenite::http::Uri; use tungstenite::http::Uri;
use tungstenite::protocol::frame::coding::CloseCode;
use tungstenite::stream::MaybeTlsStream; use tungstenite::stream::MaybeTlsStream;
use tungstenite::{Message, Utf8Bytes, WebSocket}; use tungstenite::{Message, WebSocket};
use super::book_thread;
use super::wire::{self, Arrival, Received};
pub const MARKET_CHANNEL_URL: &str = "wss://ws-subscriptions-clob.polymarket.com/ws/market"; pub const MARKET_CHANNEL_URL: &str = "wss://ws-subscriptions-clob.polymarket.com/ws/market";
/// For the TCP connect and the TLS + websocket handshake, and for every send. /// For the TCP connect and the TLS + websocket handshake, and for every send.
@@ -29,10 +33,7 @@ const CLOSE_TIMEOUT: Duration = Duration::from_secs(2);
/// 500 gets no opening books (docs/notes.md). /// 500 gets no opening books (docs/notes.md).
const CHUNK: usize = 500; const CHUNK: usize = 500;
/// A frame's text with its receive time. /// What a book thread asks a connection for.
pub type Frame = (DateTime<Utc>, Utf8Bytes);
/// What the book thread asks its connection for.
#[derive(Debug, PartialEq, Eq)] #[derive(Debug, PartialEq, Eq)]
pub enum Command { pub enum Command {
Subscribe(Vec<String>), Subscribe(Vec<String>),
@@ -43,15 +44,23 @@ pub enum Command {
type Socket = WebSocket<MaybeTlsStream<TcpStream>>; type Socket = WebSocket<MaybeTlsStream<TcpStream>>;
/// Runs until `stop` is set or the book thread is gone, reconnecting (and /// Runs until `stop` is set or a book thread is gone, reconnecting (and
/// subscribing its tokens again) whenever the connection drops. /// subscribing its tokens again) whenever the connection drops, and telling
/// every book thread when it comes up and goes down. `book_threads` holds
/// every book thread's channel, by number; `closes` hears of each 1013 close.
pub fn run( pub fn run(
url: &str, url: &str,
lane: usize, number: usize,
frames: &Sender<Frame>, book_threads: &[Sender<Received>],
commands: &Receiver<Command>, commands: &Receiver<Command>,
closes: &Sender<usize>,
stop: &AtomicBool, stop: &AtomicBool,
) { ) {
let tell = |received: fn(usize) -> Received| {
for book_thread in book_threads {
_ = book_thread.send(received(number));
}
};
let mut tokens = BTreeSet::new(); let mut tokens = BTreeSet::new();
while !stop.load(Ordering::Relaxed) { while !stop.load(Ordering::Relaxed) {
// Asked for while disconnected: the subscribe on connect covers them, // Asked for while disconnected: the subscribe on connect covers them,
@@ -62,16 +71,36 @@ pub fn run(
match connect(url, &tokens) { match connect(url, &tokens) {
Ok(mut socket) => { Ok(mut socket) => {
log!( log!(
"polymarket: lane {lane}: connected, {} symbols", "polymarket: connection {number}: connected, {} symbols",
tokens.len() tokens.len()
); );
match read(&mut socket, &mut tokens, frames, commands, stop) { tell(Received::Up);
Ok(Ended::Stopped) => return close(&mut socket, lane, frames), let ended = read(
Ok(Ended::BookThreadGone) => return, &mut socket,
Err(error) => log!("polymarket: lane {lane}: connection lost: {error}"), number,
&mut tokens,
book_threads,
commands,
closes,
stop,
);
if !matches!(ended, Ok(Ended::Stopped)) {
tell(Received::Down);
}
match ended {
Ok(Ended::Stopped) => return close(&mut socket, number, book_threads),
Ok(Ended::BookThreadGone) => {
// Only a panic ends a book thread before us: its
// markets would go unserved, so the run stops.
log!("polymarket: connection {number}: a book thread is gone, stopping");
return stop.store(true, Ordering::Relaxed);
}
Err(error) => {
log!("polymarket: connection {number}: connection lost: {error}")
} }
} }
Err(error) => log!("polymarket: lane {lane}: connect failed: {error}"), }
Err(error) => log!("polymarket: connection {number}: connect failed: {error}"),
} }
thread::sleep(RECONNECT_AFTER); thread::sleep(RECONNECT_AFTER);
} }
@@ -150,11 +179,53 @@ fn connect(url: &str, tokens: &BTreeSet<String>) -> anyhow::Result<Socket> {
Ok(socket) Ok(socket)
} }
/// Decodes a frame and sends each event to the book thread of its market;
/// `false` once a book thread is gone.
fn route(
received_at: DateTime<Utc>,
text: &str,
number: usize,
book_threads: &[Sender<Received>],
) -> bool {
let events = match wire::decode(text) {
Ok(events) => events,
Err(error) => {
log!("polymarket: unreadable frame ({error}): {}", truncate(text));
return true;
}
};
let bytes = (text.len() / events.len().max(1)) as u64;
for event in events {
// An event of no market (none we ask for) has no book thread.
let Some(thread) = event.market().map(book_thread) else {
continue;
};
let arrival = Arrival {
received_at,
connection: number,
bytes,
event,
};
if book_threads[thread].send(Received::Event(arrival)).is_err() {
return false;
}
}
true
}
fn truncate(text: &str) -> &str {
text.char_indices()
.nth(200)
.map_or(text, |(end, _)| &text[..end])
}
fn read( fn read(
socket: &mut Socket, socket: &mut Socket,
number: usize,
tokens: &mut BTreeSet<String>, tokens: &mut BTreeSet<String>,
frames: &Sender<Frame>, book_threads: &[Sender<Received>],
commands: &Receiver<Command>, commands: &Receiver<Command>,
closes: &Sender<usize>,
stop: &AtomicBool, stop: &AtomicBool,
) -> anyhow::Result<Ended> { ) -> anyhow::Result<Ended> {
let mut last_ping = Instant::now(); let mut last_ping = Instant::now();
@@ -164,11 +235,21 @@ fn read(
let received_at = Utc::now(); let received_at = Utc::now();
match message { match message {
Ok(Message::Text(text)) => { Ok(Message::Text(text)) => {
if frames.send((received_at, text)).is_err() { if !route(received_at, text.as_str(), number, book_threads) {
return Ok(Ended::BookThreadGone); return Ok(Ended::BookThreadGone);
} }
} }
Ok(Message::Close(frame)) => anyhow::bail!("closed by the server: {frame:?}"), Ok(Message::Close(frame)) => {
// 1013: the server couldn't send to us fast enough. The
// orchestrator splits a connection closed so twice.
if frame
.as_ref()
.is_some_and(|frame| frame.code == CloseCode::Again)
{
_ = closes.send(number);
}
anyhow::bail!("closed by the server: {frame:?}")
}
Ok(_) => {} Ok(_) => {}
Err(tungstenite::Error::Io(error)) if timed_out(&error) => {} Err(tungstenite::Error::Io(error)) if timed_out(&error) => {}
Err(error) => return Err(error.into()), Err(error) => return Err(error.into()),
@@ -191,16 +272,16 @@ fn read(
/// Closes the connection, forwarding what still arrives until the server /// Closes the connection, forwarding what still arrives until the server
/// confirms or [`CLOSE_TIMEOUT`] passes. /// confirms or [`CLOSE_TIMEOUT`] passes.
fn close(socket: &mut Socket, lane: usize, frames: &Sender<Frame>) { fn close(socket: &mut Socket, number: usize, book_threads: &[Sender<Received>]) {
if let Err(error) = socket.close(None) { if let Err(error) = socket.close(None) {
return log!("polymarket: lane {lane}: close failed: {error}"); return log!("polymarket: connection {number}: close failed: {error}");
} }
let deadline = Instant::now() + CLOSE_TIMEOUT; let deadline = Instant::now() + CLOSE_TIMEOUT;
while Instant::now() < deadline { while Instant::now() < deadline {
let message = socket.read(); let message = socket.read();
let received_at = Utc::now(); let received_at = Utc::now();
match message { match message {
Ok(Message::Text(text)) => _ = frames.send((received_at, text)), Ok(Message::Text(text)) => _ = route(received_at, text.as_str(), number, book_threads),
Ok(_) => {} Ok(_) => {}
Err(tungstenite::Error::Io(error)) if timed_out(&error) => {} Err(tungstenite::Error::Io(error)) if timed_out(&error) => {}
// ConnectionClosed: the close handshake is done. // ConnectionClosed: the close handshake is done.
@@ -263,14 +344,16 @@ mod tests {
connections connections
}); });
let (frames_tx, _frames_rx) = unbounded(); let (events_tx, _events_rx) = unbounded();
let (commands_tx, commands_rx) = unbounded(); let (commands_tx, commands_rx) = unbounded();
commands_tx commands_tx
.send(Command::Subscribe(tokens.clone())) .send(Command::Subscribe(tokens.clone()))
.unwrap(); .unwrap();
let stop = Arc::new(AtomicBool::new(false)); let stop = Arc::new(AtomicBool::new(false));
let client_stop = Arc::clone(&stop); let client_stop = Arc::clone(&stop);
let client = thread::spawn(move || run(&url, 0, &frames_tx, &commands_rx, &client_stop)); let (closes, _closes_rx) = unbounded();
let client =
thread::spawn(move || run(&url, 0, &[events_tx], &commands_rx, &closes, &client_stop));
let connections = server.join().unwrap(); let connections = server.join().unwrap();
stop.store(true, Ordering::Relaxed); stop.store(true, Ordering::Relaxed);
@@ -285,6 +368,49 @@ mod tests {
} }
} }
/// The recording's first frame holds the opening books of 40 tokens, 20
/// markets: each book goes to its market's book thread, with its share of
/// the frame's bytes.
#[test]
fn a_frame_of_several_markets_reaches_each_markets_book_thread() {
#[derive(serde::Deserialize)]
struct Line {
text: String,
}
let path = concat!(
env!("CARGO_MANIFEST_DIR"),
"/tests/fixtures/polymarket-frames.jsonl"
);
let recording = std::fs::read_to_string(path).unwrap();
let line: Line = serde_json::from_str(recording.lines().next().unwrap()).unwrap();
let (senders, receivers): (Vec<_>, Vec<_>) = (0..crate::polymarket::BOOK_THREADS)
.map(|_| unbounded())
.unzip();
assert!(route(Utc::now(), &line.text, 3, &senders));
let mut books = 0;
for (thread, receiver) in receivers.iter().enumerate() {
for received in receiver.try_iter() {
let Received::Event(arrival) = received else {
panic!("not an event: {received:?}");
};
let market = arrival.event.market().unwrap().to_string();
assert_eq!(book_thread(&market), thread);
assert_eq!(arrival.connection, 3);
assert_eq!(arrival.bytes, line.text.len() as u64 / 40);
books += 1;
}
}
assert_eq!(books, 40);
// Neither a keepalive's answer nor an unreadable frame reaches one.
assert!(route(Utc::now(), "PONG", 3, &senders));
assert!(route(Utc::now(), "not json", 3, &senders));
assert!(receivers.iter().all(|receiver| receiver.is_empty()));
// A book thread gone stops the connection.
drop(receivers);
assert!(!route(Utc::now(), &line.text, 3, &senders));
}
#[test] #[test]
fn commands_update_the_token_set_and_say_what_to_send() { fn commands_update_the_token_set_and_say_what_to_send() {
let mut tokens = BTreeSet::new(); let mut tokens = BTreeSet::new();
+120
View File
@@ -0,0 +1,120 @@
//! The market channel's messages, as the exchange writes them, and how a
//! frame's text becomes events (docs/notes.md).
use chrono::{DateTime, Utc};
use serde::Deserialize;
/// What a connection sends a book thread.
// Nearly every message is an event: boxing it to shrink the two rare markers
// would cost an allocation per event for nothing.
#[allow(clippy::large_enum_variant)]
#[derive(Debug)]
pub enum Received {
Event(Arrival),
/// The connection came up: its subscriptions are being sent.
Up(usize),
/// The connection went down: nothing more comes from it until it is up.
Down(usize),
}
/// One event of a frame on its way to its market's book thread.
#[derive(Debug)]
pub struct Arrival {
/// The frame's receive time, stamped right after the socket read.
pub received_at: DateTime<Utc>,
/// The connection that delivered it.
pub connection: usize,
/// Its share of the frame's bytes: a frame is one market's change, or
/// books of several, each taking its share.
pub bytes: u64,
pub event: Event,
}
/// The events of a frame's text; none for the keepalive's `PONG` and for
/// `NO NEW ASSETS`.
pub fn decode(text: &str) -> Result<Vec<Event>, serde_json::Error> {
match text.trim() {
"PONG" | "NO NEW ASSETS" => Ok(Vec::new()),
text => Ok(match serde_json::from_str(text)? {
Events::Many(events) => events,
Events::One(event) => vec![event],
}),
}
}
impl Event {
/// The market it belongs to, which decides its book thread.
pub fn market(&self) -> Option<&str> {
match self {
Event::Book(book) => Some(&book.market),
Event::PriceChange(change) => Some(&change.market),
Event::TickSizeChange { market, .. } | Event::LastTradePrice { market } => {
market.as_deref()
}
Event::Other => None,
}
}
}
#[derive(Debug, Deserialize)]
#[serde(untagged)]
enum Events {
Many(Vec<Event>),
One(Event),
}
#[derive(Debug, Deserialize)]
#[serde(tag = "event_type", rename_all = "snake_case")]
pub enum Event {
Book(BookMessage),
PriceChange(PriceChangeMessage),
/// Only the symbol and its market are read: the hashes change with the
/// tick size, and a fresh book carries the new one. The server sends each
/// one twice (docs/notes.md). `market` was on every one seen; one without
/// it can't be routed and is dropped.
TickSizeChange {
market: Option<String>,
asset_id: String,
},
LastTradePrice {
market: Option<String>,
},
/// Dropped: the others come only with `custom_feature_enabled`, which we
/// don't ask for.
#[serde(other)]
Other,
}
#[derive(Debug, Deserialize)]
pub struct BookMessage {
pub market: String,
pub asset_id: String,
pub timestamp: String,
pub hash: String,
pub bids: Vec<LevelMessage>,
pub asks: Vec<LevelMessage>,
pub tick_size: Option<String>,
pub last_trade_price: Option<String>,
}
#[derive(Debug, Deserialize)]
pub struct LevelMessage {
pub price: String,
pub size: String,
}
#[derive(Debug, Deserialize)]
pub struct PriceChangeMessage {
pub market: String,
pub timestamp: String,
pub price_changes: Vec<ChangeMessage>,
}
#[derive(Debug, Deserialize)]
pub struct ChangeMessage {
pub asset_id: String,
pub price: String,
pub size: String,
pub side: String,
pub hash: String,
}
+93 -5
View File
@@ -1,11 +1,15 @@
//! Checks an output file against itself: per symbol, the Deltas written since //! Checks an output file against itself: per symbol, the Deltas written since
//! a Snapshot, applied to it, must give the next Snapshot. A mismatch is a loss //! a Snapshot, applied to it, must give the next Snapshot. A Snapshot that
//! or a subscription gap (the fresh book differs), each logged by the collector. //! doesn't is a mismatch, and the collector logs each with its cause
//! ([`crate::cause`]): with the log, every mismatch must have its line.
use std::collections::HashMap; use std::collections::{HashMap, HashSet};
use std::path::Path; use std::path::Path;
use chrono::{DateTime, Utc};
use crate::book::Book; use crate::book::Book;
use crate::cause::{self, Cause};
use crate::util::dtos::challenge_orderbook::{AnyMessage, PriceLevelUpdateDomain, SnapshotDomain}; use crate::util::dtos::challenge_orderbook::{AnyMessage, PriceLevelUpdateDomain, SnapshotDomain};
use crate::util::{AlignedSbeReader, OrderbookSchema}; use crate::util::{AlignedSbeReader, OrderbookSchema};
@@ -16,10 +20,15 @@ pub struct Report {
/// Deltas for a symbol with no Snapshot yet: the collector never writes one. /// Deltas for a symbol with no Snapshot yet: the collector never writes one.
pub orphan_deltas: usize, pub orphan_deltas: usize,
/// Each Snapshot that didn't match its symbol's book: (symbol, its `received_at`). /// Each Snapshot that didn't match its symbol's book: (symbol, its `received_at`).
pub mismatches: Vec<(String, chrono::DateTime<chrono::Utc>)>, pub mismatches: Vec<(String, DateTime<Utc>)>,
/// The mismatches each cause explains, with the log.
pub explained: HashMap<Cause, usize>,
/// The mismatches no line of the log explains; all of them without a log.
pub unexplained: Vec<(String, DateTime<Utc>)>,
} }
pub fn reconcile(path: impl AsRef<Path>) -> anyhow::Result<Report> { /// Reads the output at `path` and, if given, the collector's log.
pub fn reconcile(path: impl AsRef<Path>, log: Option<&Path>) -> anyhow::Result<Report> {
let mut reader = AlignedSbeReader::<OrderbookSchema>::open(path)?; let mut reader = AlignedSbeReader::<OrderbookSchema>::open(path)?;
let mut books: HashMap<String, Book> = HashMap::new(); let mut books: HashMap<String, Book> = HashMap::new();
let mut report = Report::default(); let mut report = Report::default();
@@ -61,5 +70,84 @@ pub fn reconcile(path: impl AsRef<Path>) -> anyhow::Result<Report> {
_ => anyhow::bail!("unknown message in the output"), _ => anyhow::bail!("unknown message in the output"),
} }
} }
let causes: HashMap<(String, DateTime<Utc>), Cause> = match log {
Some(log) => std::fs::read_to_string(log)?
.lines()
.filter_map(cause::parse)
.map(|(symbol, received_at, cause)| ((symbol, received_at), cause))
.collect(),
None => HashMap::new(),
};
let mut seen = HashSet::new();
for mismatch in &report.mismatches {
match causes.get(mismatch) {
// A line explains one Snapshot: the same one twice would be a bug.
Some(cause) if seen.insert(mismatch.clone()) => {
*report.explained.entry(*cause).or_default() += 1
}
_ => report.unexplained.push(mismatch.clone()),
}
}
Ok(report) Ok(report)
} }
#[cfg(test)]
mod tests {
use super::*;
use crate::Side;
use crate::util::AlignedSbeWriter;
use crate::util::dtos::challenge_orderbook::{PriceLevelUpdateDomain, SnapshotBidsEntryDomain};
/// An output with a symbol's Snapshot, a Delta, then a Snapshot that
/// misses it; and a log line for that second Snapshot when `explained`.
fn reconcile_with(explained: bool) -> Report {
let dir = std::env::temp_dir();
let id = format!("{}-{explained}", std::process::id());
let (output, log) = (
dir.join(format!("reconcile-{id}.sbe")),
dir.join(format!("reconcile-{id}.log")),
);
let at = |seconds: i64| DateTime::from_timestamp(1_794_390_000 + seconds, 0).unwrap();
let snapshot = |received_at, quantity| SnapshotDomain {
received_at,
exchange_timestamp: received_at,
bids: vec![SnapshotBidsEntryDomain {
price_level: 500_000,
quantity,
}],
asks: Vec::new(),
symbol: "1".to_string(),
};
let mut writer = AlignedSbeWriter::create(output.to_str().unwrap(), 64 << 20).unwrap();
writer.write_msg(&snapshot(at(0), 1_000_000)).unwrap();
writer
.write_msg(&PriceLevelUpdateDomain {
backtest_timestamp: at(1),
exchange_timestamp: at(1),
price_level: 500_000,
quantity: 2_000_000,
side: Side::Buy,
symbol: "1".to_string(),
})
.unwrap();
writer.write_msg(&snapshot(at(2), 3_000_000)).unwrap();
writer.flush().unwrap();
let line = cause::line("1", at(if explained { 2 } else { 3 }), Cause::Loss, "");
std::fs::write(&log, format!("12:00:00.000 polymarket: {line}\n")).unwrap();
let report = reconcile(&output, Some(&log)).unwrap();
std::fs::remove_file(output).unwrap();
std::fs::remove_file(log).unwrap();
report
}
#[test]
fn a_mismatch_with_its_log_line_is_explained_and_one_without_is_not() {
let report = reconcile_with(true);
assert_eq!(report.mismatches.len(), 1);
assert_eq!(report.explained, HashMap::from([(Cause::Loss, 1)]));
assert!(report.unexplained.is_empty());
let report = reconcile_with(false);
assert_eq!(report.unexplained.len(), 1);
assert!(report.explained.is_empty());
}
}
+9 -5
View File
@@ -1,8 +1,8 @@
# Test fixtures # Test fixtures
A recording of Polymarket's market channel, replayed by the tests in Two recordings of Polymarket's market channel, replayed by the tests in
`src/polymarket/feed.rs`, and Gamma pages read by `src/polymarket/gamma.rs` `src/polymarket/feed.rs` (the first also by `socket.rs`'s routing test), and
(no network in tests). Gamma pages read by `src/polymarket/gamma.rs` (no network in tests).
- `gamma-markets.json`: Gamma's top 20 markets by 24 h volume on 2026-11-04, - `gamma-markets.json`: Gamma's top 20 markets by 24 h volume on 2026-11-04,
trimmed to the fields the collector reads. `conditionId`, `active`, `closed` trimmed to the fields the collector reads. `conditionId`, `active`, `closed`
@@ -15,5 +15,9 @@ A recording of Polymarket's market channel, replayed by the tests in
minute boundaries) of the market channel for those markets' 40 tokens, one minute boundaries) of the market channel for those markets' 40 tokens, one
line per frame: `{"received_at_us": <receive time, µs>, "text": <frame>}`. line per frame: `{"received_at_us": <receive time, µs>, "text": <frame>}`.
Opening books, 3,850 changes, 28 mid-stream books, 14 trades, 8 `PONG`s. Opening books, 3,850 changes, 28 mid-stream books, 14 trades, 8 `PONG`s.
- `record.py`: the recorder, if the fixtures need redoing: - `polymarket-frames-two.jsonl`: 95 s (2026-11-11, 12:19:58 to 12:21:33 UTC)
`uvx --with websockets python -I record.py gamma-markets.json frames.jsonl 90`. of the same tokens on two connections at once, in receive order, each line
with its `connection` (0 or 1). 20 of the 40 tokens still had an order book.
- `record.py`: the recorder, if the fixtures need redoing, on CONNECTIONS
connections at once (default 1):
`uvx --with websockets python -I record.py gamma-markets.json frames.jsonl 90 [CONNECTIONS]`.
File diff suppressed because it is too large. Load diff
+9 -4
View File
@@ -1,10 +1,13 @@
# Records the market channel for the markets in gamma-markets.json: one JSON # Records the market channel for the markets in gamma-markets.json: one JSON
# line per frame, {"received_at_us": <micros>, "text": <frame text>}. # line per frame, in receive order,
# {"received_at_us": <micros>, "connection": <number>, "text": <frame text>}.
# CONNECTIONS (default 1) are opened at once, all on the same tokens.
# uvx --with websockets python -I record.py gamma-markets.json frames.jsonl 90 [CONNECTIONS]
import asyncio, json, sys, time import asyncio, json, sys, time
import websockets import websockets
markets = json.load(open(sys.argv[1])) markets = json.load(open(sys.argv[1]))
tokens = [t for m in markets if m["enableOrderBook"] for t in json.loads(m["clobTokenIds"])] tokens = [t for m in markets if m["enableOrderBook"] for t in json.loads(m["clobTokenIds"])]
async def main(out, seconds): async def record(connection, out, seconds):
async with websockets.connect("wss://ws-subscriptions-clob.polymarket.com/ws/market", max_size=None) as ws: async with websockets.connect("wss://ws-subscriptions-clob.polymarket.com/ws/market", max_size=None) as ws:
await ws.send(json.dumps({"type": "market", "assets_ids": tokens})) await ws.send(json.dumps({"type": "market", "assets_ids": tokens}))
loop = asyncio.get_event_loop(); end = loop.time() + seconds; last_ping = loop.time() loop = asyncio.get_event_loop(); end = loop.time() + seconds; last_ping = loop.time()
@@ -14,6 +17,8 @@ async def main(out, seconds):
try: text = await asyncio.wait_for(ws.recv(), 1) try: text = await asyncio.wait_for(ws.recv(), 1)
except asyncio.TimeoutError: continue except asyncio.TimeoutError: continue
received_at_us = time.time_ns() // 1000 received_at_us = time.time_ns() // 1000
out.write(json.dumps({"received_at_us": received_at_us, "text": text}) + "\n") out.write(json.dumps({"received_at_us": received_at_us, "connection": connection, "text": text}) + "\n")
async def main(out, seconds, connections):
await asyncio.gather(*(record(connection, out, seconds) for connection in range(connections)))
with open(sys.argv[2], "w") as out: with open(sys.argv[2], "w") as out:
asyncio.run(main(out, float(sys.argv[3]))) asyncio.run(main(out, float(sys.argv[3]), int(sys.argv[4]) if len(sys.argv) > 4 else 1))