Archived
feat(polymarket)!: two connections per market, first one wins
Every live market is subscribed on two connections chosen at random, and its book takes each change from whichever connection delivers it first: the book hash says whether a change is a duplicate, the next one, or one to hold until the change before it arrives. A change one connection misses costs nothing, each Delta carries the earliest receive time, and a connection that drops is covered by the other. When a change fits on no connection in time, the book follows one connection unchecked and asks the other for a fresh book. Connections now decode and route events to 16 book threads, independent of them; a move leaves its old connection only once the new one delivers, so it costs no change; a connection closed with 1013 twice within ten minutes is split. reconcile reads the collector's log and fails only on a mismatch no cause line explains. SIGTERM and SIGHUP stop the collector cleanly. Breaking: lanes are gone (connections and book threads are separate), and reconcile takes the log as a second argument.
This commit is contained in:
24 files changed
+8937
-1167
No files matched your search
@@ -12,17 +12,22 @@ outside this repo.
|
||||
- `docs/roadmap.md` and `roadmap/NN. name.md`: milestones and each sprint's
|
||||
agreed plan. Read the current sprint file before writing code.
|
||||
- `docs/adr/`: hard-to-reverse decisions (0001: threads, shared-nothing;
|
||||
0002: lanes and the orchestrator).
|
||||
0002: the orchestrator, its lanes superseded; 0003: two connections per
|
||||
symbol, first one wins).
|
||||
- `docs/notes.md`: domain rules and exchange protocol facts (delta semantics,
|
||||
loss detection, message shapes). Read before touching exchange parsing.
|
||||
- `GLOSSARY.md`: the terms (symbol, resync, loss, subscription gap, lane,
|
||||
load, Snapshot, Delta).
|
||||
- `src/polymarket/orchestrator.rs`: what is tracked, the lanes and their
|
||||
balancing; `src/polymarket/gamma.rs`: the fetchers (discovery);
|
||||
`src/polymarket/feed.rs`: a lane's book thread and its loss rules (the
|
||||
subtle part); `src/polymarket/socket.rs`: a lane's connection;
|
||||
`src/book.rs`: the book and fixed point; `src/reconcile.rs` + `src/bin/reconcile.rs`: the output
|
||||
check. `tests/fixtures/`: a 90 s recording of the live feed the tests replay.
|
||||
- `GLOSSARY.md`: the terms (symbol, connection, load, first one wins,
|
||||
duplicate, held, rescued, follow, resync, loss, subscription gap, split,
|
||||
Snapshot, Delta).
|
||||
- `src/polymarket/orchestrator.rs`: what is tracked, which connections carry
|
||||
each market, and their balancing; `src/polymarket/gamma.rs`: the fetchers
|
||||
(discovery); `src/polymarket/socket.rs`: a connection (read, stamp, decode,
|
||||
route); `src/polymarket/wire.rs`: the exchange's messages;
|
||||
`src/polymarket/feed.rs`: a book thread and its loss rules (the subtle
|
||||
part); `src/book.rs`: the book and fixed point; `src/reconcile.rs` +
|
||||
`src/bin/reconcile.rs`: the output check, reading the log's cause lines
|
||||
(`src/cause.rs`). `tests/fixtures/`: recordings of the live feed, on one and two
|
||||
connections, that the tests replay.
|
||||
- `src/util/` and `sbe_schemas/`: the template's SBE machinery, used as given so
|
||||
output stays readable by the template's `AlignedSbeReader`. Our code is the
|
||||
rest of `src/`.
|
||||
@@ -57,17 +62,21 @@ and let the user settle it. Claude owns the data model.
|
||||
A 403 is not DNS (see `docs/notes.md`, REST from scripts).
|
||||
- **Write 64 MiB chunks**: `AlignedSbeReader::open` hardcodes 64 MiB
|
||||
(`open_with_chunk_size` is private), so any other chunk size is unreadable.
|
||||
- **The collector runs until SIGINT, and only SIGINT flushes its output.**
|
||||
Any other signal (SIGTERM, SIGALRM) leaves the last 64 MiB chunk unwritten.
|
||||
Bound a run with GNU `timeout -s INT`, then check the output:
|
||||
- **Only a clean stop flushes the output**: SIGINT, SIGTERM or SIGHUP. SIGKILL
|
||||
or SIGALRM leaves the last 64 MiB chunk unwritten. Bound a run with GNU
|
||||
`timeout -s INT`, then check the output against the log:
|
||||
`timeout -s INT 600 ./target/release/engineering-challenge out.sbe 2> run.log;
|
||||
./target/release/reconcile out.sbe`. `reconcile` counts subscription gaps
|
||||
as mismatches too: compare its list with the log's "loss", "subscription
|
||||
gap" and "first book after a move" lines.
|
||||
./target/release/reconcile out.sbe run.log`. With the log, `reconcile`
|
||||
pairs each mismatch with its cause line and fails only on one left
|
||||
unexplained.
|
||||
- **Wait on a PID, never on a `pgrep -f` pattern**: `until ! pgrep -f
|
||||
"capacity scan"` matches the waiting shell's own command line and never
|
||||
ends, and `pkill -f PATTERN` can kill the shell running it. Take the PID
|
||||
(`pgrep -f '^./target/release/...'`, anchored) and loop on `kill -0 $pid`.
|
||||
- **The server is shared and gets overloaded** (load up to 92 on 12 cores,
|
||||
2026-11-11): a starved collector falls behind and gets closed with 1013.
|
||||
Check `cat /proc/loadavg` before a live run whose numbers are compared, and
|
||||
alternate the builds in short rounds rather than running them back to back.
|
||||
- **Live data beats the docs**: Polymarket's market channel has rules its
|
||||
docs don't state (`docs/notes.md`); check a protocol assumption on a
|
||||
recording of the real feed (`tests/fixtures/record.py`) before relying on it.
|
||||
|
||||
+2
-1
@@ -10,7 +10,8 @@ default-run = "engineering-challenge"
|
||||
anyhow = "1"
|
||||
chrono = "0.4"
|
||||
crossbeam = "0.8.4"
|
||||
ctrlc = "3"
|
||||
# `termination`: SIGTERM and SIGHUP stop the collector like SIGINT.
|
||||
ctrlc = { version = "3", features = ["termination"] }
|
||||
# Not used directly: tungstenite leaves rustls's crypto provider to the app, and
|
||||
# rustls panics at runtime unless exactly one (here ring) is enabled.
|
||||
rustls = { version = "0.23", default-features = false, features = ["ring"] }
|
||||
|
||||
+40
-15
@@ -32,28 +32,53 @@ The collector's in-memory order book for one symbol: its bid and ask price
|
||||
levels, each with its absolute quantity.
|
||||
_Avoid_: Order book state, depth
|
||||
|
||||
**Lane**:
|
||||
A set of symbols whose books one book thread owns, fed by its own
|
||||
connections: one path from the wire to the writer. Every live symbol is in
|
||||
exactly one lane.
|
||||
_Avoid_: Shard (a technique, not this thing), group, partition
|
||||
**Connection**:
|
||||
One websocket to the exchange. Every live symbol is subscribed on two
|
||||
connections, chosen independently of each other.
|
||||
_Avoid_: Socket (for the concept; fine for the code), stream, lane (the
|
||||
retired pairing of one connection with one book thread)
|
||||
|
||||
**Load**:
|
||||
How much a market weighs on its connection: the bytes per second of its
|
||||
frames. A lane's load is the sum of its markets'.
|
||||
How much a market weighs on a connection: the bytes per second of its frames
|
||||
there. A connection's load is the sum of its markets'.
|
||||
_Avoid_: Activity, volume (the exchange's traded amount, unrelated)
|
||||
|
||||
**Connection**:
|
||||
One websocket to the exchange, carrying the symbols of one lane.
|
||||
_Avoid_: Socket (for the concept; fine for the code), stream
|
||||
**First one wins**:
|
||||
The rule for a symbol's two connections: a change is taken from whichever
|
||||
delivers it first, with that receive time, and the other's copy is dropped.
|
||||
_Avoid_: Source, primary, standby, failover (no connection has a rank)
|
||||
|
||||
**Duplicate**:
|
||||
A change another connection already delivered.
|
||||
|
||||
**Verified**:
|
||||
Said of a symbol whose book gives the exchange's book hash after every
|
||||
change. It stops being verified when a change fits on no connection in time.
|
||||
|
||||
**Held**:
|
||||
Said of a change kept unwritten because it doesn't fit the book yet: a change
|
||||
before it is still to come, from this connection or the other.
|
||||
_Avoid_: Buffered, pending, queued
|
||||
|
||||
**Rescued**:
|
||||
Said of a change one connection missed and the other delivered: the output
|
||||
has it, where a single connection would have had a loss.
|
||||
|
||||
**Follow**:
|
||||
To apply one connection's changes unchecked, while a symbol can't be
|
||||
verified and until a fresh book says where it stands.
|
||||
|
||||
**Split**:
|
||||
Moving half the markets off a connection the exchange keeps closing.
|
||||
|
||||
**Receive time**:
|
||||
When a message came off the socket, by the local clock.
|
||||
_Avoid_: Arrival time, local timestamp
|
||||
|
||||
**Loss**:
|
||||
Any message for a symbol that the collector did not apply, which leaves its
|
||||
book silently wrong.
|
||||
A change for a symbol that no connection delivered in time, which leaves its
|
||||
book silently wrong. A change only one connection missed is rescued, not
|
||||
lost.
|
||||
_Avoid_: Gap (implies sequence numbers)
|
||||
|
||||
**Resync**:
|
||||
@@ -63,9 +88,9 @@ one differs: a loss, or a subscription gap.
|
||||
_Avoid_: Re-snapshot, rebuild, refresh
|
||||
|
||||
**Subscription gap**:
|
||||
The changes our own subscribe keeps from us, whether it opens, resyncs or
|
||||
moves a symbol: the exchange holds them in the book it sends but never sends
|
||||
them as changes.
|
||||
The changes our own subscribe keeps from us, whether it opens or resyncs a
|
||||
symbol: the exchange holds them in the book it sends but never sends them as
|
||||
changes. A connection that stays subscribed meanwhile has no such gap.
|
||||
_Avoid_: Resync gap (the older, narrower name); calling it a loss (the state
|
||||
stays right; only those updates are missing)
|
||||
|
||||
|
||||
@@ -29,6 +29,10 @@ races by construction, and keeps the runtime pluggable: only the socket layer is
|
||||
runtime-specific, so a later async swap is localized to that boundary rather than
|
||||
a pipeline rewrite. Deferring async is therefore safe through the base challenge.
|
||||
|
||||
The M4 checkpoint was held in sprint 04 (2026-11-11): threads stay. The new
|
||||
logic of redundancy sits in the book threads, not in socket handling, and
|
||||
about 200 threads (175 connections, 16 book threads) are comfortable.
|
||||
|
||||
Trade-off: thread-per-socket does not scale to very large socket counts, and the
|
||||
extension-era socket-management logic (select / timeout / reconnect) is where
|
||||
threads and async diverge most — hence the explicit M4 re-evaluation point rather
|
||||
|
||||
@@ -1,5 +1,9 @@
|
||||
# 0002. Lanes and an orchestrator
|
||||
|
||||
*Superseded in part by [0003](0003-first-one-wins.md) (sprint 04): the lanes are
|
||||
gone, connections and book threads being separate sets; the orchestrator and
|
||||
the fetchers stand.*
|
||||
|
||||
## Context
|
||||
|
||||
Every live Polymarket symbol is ~476k tokens (2026-11-04), an estimated
|
||||
|
||||
@@ -0,0 +1,47 @@
|
||||
# 0003. Two connections per symbol, first one wins
|
||||
|
||||
## Context
|
||||
|
||||
Extension 1 asks for redundant websockets: connections drop, go quiet, lag,
|
||||
and get closed by the server (code 1013) when they carry too much. Sprint 03
|
||||
gave each lane a connection and a book thread of its own
|
||||
([0002](0002-lanes-and-orchestrator.md)). The obvious next step was a second,
|
||||
identical connection per lane, one of them the trusted source, with a
|
||||
Snapshot at each switch, as the brief sketches. Measured on 2026-11-11
|
||||
([sprint 04](../../roadmap/04.%20redundant%20connections.md)): two connections
|
||||
on the same symbols deliver the same changes in the same order, 0.5 ms apart
|
||||
at the median and 100 ms or more for 1% of changes; and two connections
|
||||
carrying the same symbols are closed together.
|
||||
|
||||
## Decision
|
||||
|
||||
Every symbol is subscribed on **two connections chosen independently and at
|
||||
random**, and its one book takes each change from **whichever connection
|
||||
delivers it first**. Polymarket's book hash, sent with every change, says
|
||||
where a change belongs: already applied (a duplicate), next (it fits and is
|
||||
written), or early (held until the change before it arrives). Only when a
|
||||
change fits on no connection in time does the book thread fall back to
|
||||
following one connection unchecked and resyncing on the other.
|
||||
|
||||
Connections therefore no longer belong to a book thread. A connection thread
|
||||
stamps, parses and routes each event to the book thread of its market, one of
|
||||
16, derived from the market's id. The orchestrator of 0002 stays; its lanes
|
||||
are superseded.
|
||||
|
||||
## Why
|
||||
|
||||
First one wins gives each Delta the earliest receive time, turns a change one
|
||||
connection missed into no loss at all, and needs no second book and no
|
||||
Snapshot per switch. A source and a standby would have kept the source's
|
||||
receive time and lost the change the source missed. Independent connections
|
||||
were measured against twins on the same 48,038 symbols for 20 minutes: 272
|
||||
symbol-seconds with both subscriptions down, against 3,787 for twins dealt at
|
||||
random and 33,345 for twins in creation order. A move becomes a subscribe and
|
||||
an unsubscribe with the book left where it is, so it opens no subscription
|
||||
gap.
|
||||
|
||||
Trade-off: the routing step 0002 avoided is back, parsing moves into the
|
||||
connection thread, and the book thread's rules get subtler (held changes, a
|
||||
ceiling, a chain declared broken). The design leans on an exchange that lets
|
||||
the same change be recognised across connections: Polymarket's hash does;
|
||||
Kalshi's `clock` looks as if it would (to confirm in M6).
|
||||
+2
-2
@@ -46,8 +46,8 @@ Prefer these; they can't be gamed by strategy luck:
|
||||
separately our *internal* added latency (edge-stamp → output byte).
|
||||
- **Completeness:** reconciliation error rate (snapshot↔delta divergence); count
|
||||
of missed deletes / stale levels under induced loss.
|
||||
- **Redundancy value (once ext1 exists):** messages the backup socket caught that
|
||||
the primary missed — literal "data loss avoided."
|
||||
- **Redundancy value:** changes one connection missed and the other delivered
|
||||
(rescued) — literal "data loss avoided."
|
||||
|
||||
## Recommendation
|
||||
|
||||
|
||||
@@ -147,6 +147,32 @@ From a 60 s run of the M1 spike on one market, both tokens subscribed, and the
|
||||
403 (curl and `ureq` get 200), most likely Cloudflare filtering by user agent.
|
||||
Fetch with curl in throwaway scripts.
|
||||
|
||||
## Two connections on the same symbols (observed 2026-11-11)
|
||||
|
||||
- **Same changes, same order**: two connections subscribed to the same 8,000
|
||||
tokens for 335 s delivered the same 616,708 changes, none on only one, in
|
||||
the same order per token. So the book hash orders changes across
|
||||
connections too.
|
||||
- **The second delivery's lag**: 0.5 ms at the median, over 10 ms for 6.5% of
|
||||
changes, over 100 ms for 1%, 706 ms at worst; up to 82 changes of its token
|
||||
behind. Neither connection leads (one was first 34% of the time).
|
||||
- **Batches**: 95% of updates are one change. The others come one change per
|
||||
frame, all with the hash of the whole batch, within 1 ms (40 ms at worst).
|
||||
- **The server repeats some changes** on one connection (104 in that
|
||||
recording, the same on both): harmless, quantities being absolute.
|
||||
- **Connections carrying the same tokens are closed together**: two ramped on
|
||||
the same tokens got their 1013 close 3 s apart. Over 20 minutes on 48,038
|
||||
tokens with two subscriptions each, twins in creation order had both
|
||||
subscriptions down for 33,345 token-seconds, twins dealt at random 3,787,
|
||||
two independent random deals 272.
|
||||
- **300 connections** opened at once from one address all stayed up.
|
||||
- **A connection whose every subscribed market has resolved is closed** by the
|
||||
server, code 1000, reason `all subscribed assets resolved` (seen on a
|
||||
connection left with one market, 2026-11-11).
|
||||
- **A change's `timestamp` never goes back within a token** on a connection
|
||||
(0 of 660,084 changes on each of the two):
|
||||
one older than the last applied is a late copy.
|
||||
|
||||
## Gamma, the market listing (observed 2026-11-04)
|
||||
|
||||
- **Size:** 238,308 markets `active` and not `closed`; 238,201 of them with
|
||||
|
||||
+41
-45
@@ -38,22 +38,22 @@ saves the data.
|
||||
The interacting decisions it is made of:
|
||||
|
||||
- **Threading / runtime:** settled in sprint 01 — OS threads + channels,
|
||||
shared-nothing, async re-evaluated at M4
|
||||
shared-nothing; async was re-evaluated at M4 and threads stay
|
||||
([ADR 0001](adr/0001-concurrency-model.md)).
|
||||
- **Stamping point:** in the socket-read path, immediately, so as little as
|
||||
possible sits between the wire and the receive timestamp.
|
||||
- **Book ownership:** settled in sprint 03 — symbols are spread over lanes,
|
||||
each a connection and a book thread of its own, so each book is owned by
|
||||
exactly one thread; an orchestrator assigns them
|
||||
([ADR 0002](adr/0002-lanes-and-orchestrator.md)).
|
||||
- **Book ownership:** each book is owned by exactly one of 16 book threads,
|
||||
chosen from its market's id; an orchestrator decides which connections
|
||||
carry each market ([ADR 0002](adr/0002-lanes-and-orchestrator.md), its
|
||||
lanes superseded by [ADR 0003](adr/0003-first-one-wins.md) in sprint 04).
|
||||
- **Backpressure:** bounded vs unbounded channels — the core correctness/perf
|
||||
tension. Drop-on-full loses data (violates completeness); block adds latency
|
||||
and fills socket buffers. This decision is the bet in miniature.
|
||||
- **Output serialization:** one writer owns the single sink and consumes from
|
||||
all workers.
|
||||
- **Redundancy fit:** multiple sockets feeding one symbol's worker, which
|
||||
dedups / selects by source — the topology must leave room for this from the
|
||||
start.
|
||||
- **Redundancy:** settled in sprint 04 — every symbol on two connections
|
||||
chosen independently, its book taking each change from whichever delivers
|
||||
it first ([ADR 0003](adr/0003-first-one-wins.md)).
|
||||
|
||||
## Requirements the pipeline must satisfy
|
||||
|
||||
@@ -118,7 +118,7 @@ Their operational form, for writing code, is in the project
|
||||
Chunks must be 64 MiB: `AlignedSbeReader::open` reads nothing else.
|
||||
- **Crash ceiling** (accepted in sprint 02): the writer holds the current
|
||||
64 MiB chunk in memory until it is full, so a hard crash loses it; flushing
|
||||
more often pads each flush to 64 MiB. Taken up with extension 1 (M4).
|
||||
more often pads each flush to 64 MiB. Kept as a ceiling in extension 1 (M4).
|
||||
|
||||
## Workflow differences
|
||||
|
||||
@@ -135,34 +135,38 @@ and drop from the global workflow for a job-challenge repo.
|
||||
## Pipeline
|
||||
|
||||
The primary thing we debate sprint by sprint (this project's equivalent of the
|
||||
schema debate). As settled in sprint 03:
|
||||
schema debate). As settled in sprint 04:
|
||||
|
||||
```
|
||||
rescan fetcher ──findings──▶ ┌──────────────┐ ◀──findings── recent fetcher
|
||||
(full Gamma scan) ◀─request─│ orchestrator │──request──▶ (new + closed since T)
|
||||
└──┬────────▲──┘
|
||||
Add / MoveIn / Remove, to │ │ load reports, each minute
|
||||
the market's lane │ │ (bytes/s per market)
|
||||
┌──────────────────────┼────────┴─────────────────┐
|
||||
▼ ▼ ▼
|
||||
lane 0: book thread ◀─frames─ connection lane 1: ... lane N: ...
|
||||
│ └─subscribe / unsubscribe / resync─▶┘
|
||||
└──────────── Snapshot | Delta ─────────────▶ writer thread (one)
|
||||
│ orchestrator │ ◀── 1013 closes ── connections
|
||||
└──────┬───▲───┘
|
||||
Add / Move / Remove (a market and │ │ load reports, each minute
|
||||
its connections), to its thread │ │ (bytes/s per market)
|
||||
┌──────▼───┴───┐
|
||||
connection 0..N ──events───▶ │ book threads │ ──Snapshot | Delta──▶ writer
|
||||
(read, stamp, parse, route) │ 0..15 │
|
||||
▲ └──────┬───────┘
|
||||
└── subscribe / unsubscribe / resync ┘
|
||||
```
|
||||
|
||||
- The **orchestrator** owns what is tracked (market → lane, the lanes, the
|
||||
timers) and decides; it never blocks on HTTP, so later coordination
|
||||
(switching sources in M4, recycling in M5) lives there too.
|
||||
- The **orchestrator** owns what is tracked (each market's two connections,
|
||||
the timers) and decides: placing, moving, splitting. It never blocks on
|
||||
HTTP.
|
||||
- The two **fetchers** only page Gamma on request and stream what they read.
|
||||
- Each **lane** is a connection (stamps receive time, sends subscriptions and
|
||||
resyncs) and a book thread (fold, check, resync, Snapshots); M4 gives a lane
|
||||
several connections and has its book thread choose the trusted source.
|
||||
- A **connection** reads, stamps the receive time, parses, and routes each
|
||||
event to the book thread of its market. Every symbol is on two connections,
|
||||
chosen independently.
|
||||
- A **book thread** owns its markets' books and applies first one wins: a
|
||||
change is a duplicate, the next one, or held until it fits (fold, check,
|
||||
resync, Snapshots as before).
|
||||
- One **writer** owns the output.
|
||||
|
||||
Settled: OS threads, shared-nothing, no mutexes
|
||||
([ADR 0001](adr/0001-concurrency-model.md)); lanes and the orchestrator
|
||||
([ADR 0002](adr/0002-lanes-and-orchestrator.md)). Details:
|
||||
[sprint 03](../roadmap/03.%20full%20coverage.md).
|
||||
([ADR 0001](adr/0001-concurrency-model.md)); the orchestrator
|
||||
([ADR 0002](adr/0002-lanes-and-orchestrator.md)); two connections per symbol,
|
||||
first one wins ([ADR 0003](adr/0003-first-one-wins.md)). Details:
|
||||
[sprint 04](../roadmap/04.%20redundant%20connections.md).
|
||||
|
||||
## Data model, first sketch
|
||||
|
||||
@@ -179,28 +183,20 @@ Not a DB — an in-memory domain plus the fixed SBE output schema. Main entities
|
||||
whatever the exchange offers for loss detection (a sequence number, a book
|
||||
hash), and payload (side/price/quantity, or the full book).
|
||||
- **Book** (one per symbol): bids and asks as ordered price→quantity maps, plus
|
||||
the last loss-detection marker seen, last exchange timestamp, and the id of the
|
||||
source currently trusted (for redundancy).
|
||||
- **Connection** (one websocket): the set of symbols it carries, those of its
|
||||
lane.
|
||||
- **Lane**: a set of symbols, the book thread that owns their books, and the
|
||||
connections that feed it. Live markets are assigned to lanes by
|
||||
`conditionId`; Polymarket's lifecycle signal is Gamma (settled in sprint 03:
|
||||
a market is live while active, not closed, with an order book and accepting
|
||||
orders).
|
||||
- **Source / redundancy:** several connections may feed the same symbol; one is
|
||||
the trusted source at a time.
|
||||
the last loss-detection marker seen and last exchange timestamp. It is fed
|
||||
by the symbol's two connections, first one wins.
|
||||
- **Connection** (one websocket): the set of symbols it carries. Live markets
|
||||
are placed on connections by `conditionId`, both tokens together;
|
||||
Polymarket's lifecycle signal is Gamma (settled in sprint 03: a market is
|
||||
live while active, not closed, with an order book and accepting orders).
|
||||
|
||||
Output schema is fixed by the SBE schema: `Snapshot` and `PriceLevelUpdate`.
|
||||
|
||||
The debate lives in two places: the **NormalizedEvent** shape (both exchanges
|
||||
must map cleanly onto it) and **redundancy reconciliation** (how a symbol's
|
||||
trusted source is chosen and switched, and emitting a fresh snapshot on switch).
|
||||
The debate lives in the **NormalizedEvent** shape (both exchanges must map
|
||||
cleanly onto it). Redundancy was settled in sprint 04: no source is trusted,
|
||||
the first copy of each change wins ([ADR 0003](adr/0003-first-one-wins.md)).
|
||||
|
||||
## Open questions
|
||||
|
||||
- **Loss detection on Kalshi.** Its sequence semantics: checked in M6.
|
||||
(Polymarket: settled in M2, by its book hash.)
|
||||
- **Redundancy source-selection.** How to pick/switch the trusted source
|
||||
(freshness, message rate, which source is ahead). Designed in the redundancy
|
||||
sprint.
|
||||
+26
-29
@@ -40,31 +40,25 @@ abstraction is *extracted* once a second exchange exists, not designed up front.
|
||||
*Ends:* every live Polymarket symbol tracked — base challenge satisfied for
|
||||
one exchange.
|
||||
|
||||
4. **Extension 1 — redundant websockets** — *not started*
|
||||
Multiple connections per symbol (several connections in a lane, its book
|
||||
thread choosing among them, the orchestrator coordinating); choose a
|
||||
trusted source and switch on loss/quiet/delay; emit a fresh snapshot on
|
||||
every source switch. Recovery from errors, with the lane budget scaling
|
||||
itself down when connections still get 1013 closes (no rate is safe:
|
||||
[notes.md](notes.md), so a budget set too high must correct itself).
|
||||
**Async re-evaluation checkpoint:** before building this, reassess whether the
|
||||
socket-management complexity (reconnect/timeout/select, redundancy) warrants
|
||||
swapping the thread-only runtime for tokio (see ADR 0001).
|
||||
The lane that holds markets busy together (the 5- and 15-minute crypto
|
||||
markets: closed 16 times in 75 minutes, in bursts a minute's load
|
||||
average doesn't show) is the case to beat.
|
||||
Also: surviving a hard crash without losing the writer's in-memory chunk,
|
||||
a move's Remove reaching the old lane before the new lane's first
|
||||
Snapshot, the memory growth seen in sprint 03 (~90 MB every 10 minutes
|
||||
over 75 minutes), and closing the subscription gap (the few ms of a
|
||||
symbol's changes the server doesn't send between our unsubscribe and
|
||||
subscribe); the crash chunk and the gap were accepted as ceilings in M2.
|
||||
*Ends:* redundancy demonstrably reduces data loss; a Snapshot on every source
|
||||
switch.
|
||||
4. **Extension 1 — redundant websockets** — *done (2026-11-11)*
|
||||
Every symbol on two connections chosen independently, its book taking each
|
||||
change from whichever delivers it first (first one wins); connections
|
||||
separated from book threads; a connection the server keeps closing with
|
||||
1013 is split, so a budget set too high corrects itself. The async
|
||||
checkpoint was held: threads stay
|
||||
([ADR 0001](adr/0001-concurrency-model.md)).
|
||||
Sprint: [../roadmap/04. redundant connections.md](../roadmap/04.%20redundant%20connections.md).
|
||||
Left as ceilings: the writer's in-memory chunk on a hard crash, and the
|
||||
subscription gap when a symbol opens.
|
||||
*Ends:* redundancy demonstrably reduces data loss: fewer losses and
|
||||
subscription gaps per million changes with two subscriptions than with one.
|
||||
Result: losses a quarter lower in each pair of alternating runs; losses and
|
||||
gaps together lower in one pair, even in the other (sprint 04's Outcome).
|
||||
|
||||
5. **Extension 2 — subscription monitoring** — *not started*
|
||||
Per-symbol activity monitoring (data points per second/minute); unsubscribe +
|
||||
resubscribe symbols that go too quiet; periodic full-socket recycle
|
||||
resubscribe symbols that go too quiet (with two connections, first one wins
|
||||
hides a subscription gone quiet on one of them: it has to be looked for); periodic full-socket recycle
|
||||
(disconnect → reconnect → resubscribe) to shed cumulative exchange-side bugs.
|
||||
*Ends:* monitoring + recycling running.
|
||||
|
||||
@@ -108,16 +102,19 @@ abstraction is *extracted* once a second exchange exists, not designed up front.
|
||||
- Observability: metrics, structured logging, dashboards.
|
||||
- Config-driven symbol filtering and per-exchange tuning.
|
||||
- Containerized packaging.
|
||||
- Counting a reconnect's differing books as losses rather than subscription
|
||||
gaps when the symbol was resyncing or not yet verified since its opening
|
||||
book (a reconnect marker through the frames channel; they are labelled
|
||||
gaps).
|
||||
- A `reconcile` that reads the collector's log, to tell subscription gaps from
|
||||
losses in its own report.
|
||||
- Polymarket's `new_market` websocket event as a faster discovery signal (it
|
||||
is broadcast, but lacks `orderMinSize` / `negRisk`, so Gamma is still
|
||||
needed; `market_resolved` was never seen).
|
||||
- Closing lanes that drain.
|
||||
- Closing connections that drain.
|
||||
- Verifying a symbol again from the trade event's price after a trade, with
|
||||
no resync (sprint 04's prototype did it for 15 of 100 broken chains).
|
||||
- Telling a real loss from a resync book merely ahead of our stream, so the
|
||||
loss count measures data missed (most of sprint 04's "losses" follow
|
||||
trades, as often with one connection as with two).
|
||||
- Closing the opening subscription gap with the second connection's opening
|
||||
book, when the first one's chain breaks before any change verified.
|
||||
- An exact rescued count: a change remembered past the window of 128 until
|
||||
its copies are counted (sprint 04 reports an upper bound).
|
||||
- Fewer Snapshots for unchanged books (the brief asks for every symbol every
|
||||
minute: ~130 MB a minute at full coverage).
|
||||
- Re-reading a market's `orderMinSize` / `negRisk` if they can change.
|
||||
@@ -0,0 +1,443 @@
|
||||
# 04. Redundant connections: first one wins
|
||||
|
||||
**Status: done (2026-11-11)**
|
||||
|
||||
Every symbol is subscribed on two connections, and the first one to deliver a
|
||||
change wins: it is written with that receive time, and the other's copy is
|
||||
dropped. A change lost on one connection is then no loss, a slow connection
|
||||
costs no time accuracy, and a connection that drops is covered by the other.
|
||||
Connections stop belonging to book threads, so a symbol's two subscriptions
|
||||
have different neighbours, and a connection the server keeps closing is split.
|
||||
This is Extension 1 of the brief. Design:
|
||||
[../docs/project.md](../docs/project.md) (pipeline),
|
||||
[ADR 0003](../docs/adr/0003-first-one-wins.md) (this design; supersedes the
|
||||
lanes of ADR 0002), [../docs/notes.md](../docs/notes.md) (protocol facts),
|
||||
[../GLOSSARY.md](../GLOSSARY.md) (first one wins, duplicate, verified, held,
|
||||
follow, rescued, split; loss, narrowed).
|
||||
|
||||
## Measured before planning (2026-11-11)
|
||||
|
||||
- **Two connections on the same 8,000 symbols, 335 s**: 616,708 changes, none
|
||||
on only one connection, the same order per symbol on both. The second
|
||||
delivery came 0.5 ms later at the median, more than 10 ms later for 6.5% of
|
||||
changes, more than 100 ms for 1%, 706 ms at worst, and up to 82 changes of
|
||||
its symbol behind. Neither connection led: one was first 34% of the time.
|
||||
95% of updates were one change; the others came one change per frame, all
|
||||
with the hash of the whole batch, within 1 ms (40 ms at worst).
|
||||
- **Twins are closed together**: two connections ramped on the same symbols
|
||||
got their 1013 close 3 s apart, at the same load.
|
||||
- **Three layouts of the same 48,038 newest symbols, 20 minutes, two
|
||||
subscriptions per symbol** (connections of 5,000 symbols, reconnecting after
|
||||
1 s):
|
||||
|
||||
| Layout | Closes | Symbol-seconds with both down | Symbols ever with both down |
|
||||
|---|---|---|---|
|
||||
| Twins, symbols in creation order | 49 | 33,345 | 5,000 |
|
||||
| Twins, symbols dealt randomly | 15 | 3,787 | 5,000 |
|
||||
| Two independent random deals | 20 | 272 | 290 |
|
||||
|
||||
In creation order every close hit the pair holding the day's 5- and
|
||||
15-minute crypto markets (~2.5 MB/s). The test connections carried 450 to
|
||||
900 KB/s, above our budget: the ratios count, not the totals.
|
||||
- **300 connections** opened at once from the server all stayed up. Its link
|
||||
carried 50 MB/s on one download; two subscriptions per symbol are an
|
||||
estimated 25 MB/s.
|
||||
- **A prototype of the rules below on the two-connection recording** (Python,
|
||||
real hash checks, 7,729 symbols): 340,424 changes written verified, 338,609
|
||||
duplicates dropped, ~36,000 changes held then fit (batches), 100 chains
|
||||
broken, all with both connections holding the same change (trades). With
|
||||
1% of one connection's changes removed, and with one connection delayed by
|
||||
2 s, the output kept the single connection's order per symbol. The delay
|
||||
case first failed (the late changes were held, the chain declared broken,
|
||||
the slow connection followed): the stale rule and "the book moved on" rule
|
||||
below come from it.
|
||||
- **Sprint 03's 75-minute run**: 151,897 resyncs for 169 M verified changes;
|
||||
11,605 of them opened a subscription gap. 7 of its 31 closes came within
|
||||
0.2 s after a clock minute ending in 4, 9, 0 or 5: bursts a minute's load
|
||||
average doesn't show.
|
||||
|
||||
## Pipeline
|
||||
|
||||
```
|
||||
rescan fetcher ──findings──▶ ┌──────────────┐ ◀──findings── recent fetcher
|
||||
│ orchestrator │ ◀── 1013 closes ── connections
|
||||
└──────┬───▲───┘
|
||||
Add / Move / Remove (a market and │ │ load reports, each minute
|
||||
its connections), to its thread │ │ (bytes/s per market)
|
||||
┌──────▼───┴───┐
|
||||
connection 0..N ──events───▶ │ book threads │ ──Snapshot | Delta──▶ writer
|
||||
(read, stamp, parse, route) │ 0..15 │
|
||||
▲ └──────┬───────┘
|
||||
└── subscribe / unsubscribe / resync ┘
|
||||
```
|
||||
|
||||
- **Connection** (one thread each, ~175 at today's size): reads a frame,
|
||||
stamps its receive time, parses it, and sends each event to the book thread
|
||||
of its market, with its own number and the frame's bytes shared among its
|
||||
events. It tells every book thread when it comes up or goes down, and the
|
||||
orchestrator when the server closes it with 1013. It keeps its token set and
|
||||
subscribes it, in operations of 500, on every connect. Limits: 5,000 tokens
|
||||
and 300 KB/s, as sprint 03's lanes had.
|
||||
- **Book thread** (16): owns the books of its markets, a market's thread being
|
||||
derived from its `conditionId` (16 hex digits of it, modulo 16), so every
|
||||
thread computes it without a shared table. It applies first one wins
|
||||
(below), writes Deltas and Snapshots, and is the one that asks connections
|
||||
to subscribe, unsubscribe and resync: it knows a symbol before its book can
|
||||
arrive. Each wall-clock minute it writes the minute Snapshots and reports
|
||||
its markets' loads.
|
||||
- **Orchestrator**: holds each market's two connections, places and moves
|
||||
subscriptions, splits a connection, and runs the fetchers as in sprint 03.
|
||||
- **Writer**: unchanged.
|
||||
|
||||
## First one wins
|
||||
|
||||
A symbol whose hash chain holds is **verified**. A change arriving from any of
|
||||
its connections is then, in this order:
|
||||
|
||||
1. a **duplicate**, if it is one of the symbol's last 128 applied changes
|
||||
(same hash, side, price and size): dropped;
|
||||
2. **stale**, if its exchange timestamp is older than the last applied
|
||||
change's: dropped (while the chain holds, everything up to the book's state
|
||||
was applied);
|
||||
3. otherwise **held** on its connection. The shortest tail of that
|
||||
connection's held changes which, applied to the book, gives the hash of its
|
||||
last change **fits**: it is applied and written; held changes before the
|
||||
tail are stale and dropped; a change that leaves the book as it was is not
|
||||
written. One rule covers a batch completing, a change the other connection
|
||||
delivers after this one ran ahead, and late changes after a fresh book.
|
||||
|
||||
A held change is written with its own receive time, never earlier than the
|
||||
symbol's previous Delta, so replaying the output by time keeps the order.
|
||||
|
||||
**The ceiling.** A change may be held for 1 s; for 50 ms once every connection
|
||||
of the symbol that is up holds the same changes (nobody is left to wait for
|
||||
but the rest of a batch); at once when a connection holds more than 512
|
||||
changes of the symbol. Only tails that start a batch are tried against the
|
||||
book: the whole held list and the last four. At the ceiling, if the book moved since the change
|
||||
arrived, its connection is the one behind and its held changes are dropped.
|
||||
If the book did not move, the chain is **broken**.
|
||||
|
||||
**A broken chain.** The book thread **follows** the connection whose held
|
||||
changes reached the ceiling: its held changes and its later ones are applied
|
||||
and written unchecked, the other connection's are dropped, and a resync goes
|
||||
to the other connection, so the followed stream is not interrupted (to the
|
||||
followed one when it is the only one up). The fresh book is compared as in
|
||||
sprint 02: equal to ours or a state we passed, ours stands; otherwise it
|
||||
replaces ours, a Snapshot is written and the difference is logged as a loss.
|
||||
Either way the symbol is verified again. A symbol whose complete book fails
|
||||
its own hash stays unverifiable, following one connection, as before.
|
||||
|
||||
**Full books.** Only books we asked for count: a subscribe's opening book
|
||||
while the symbol awaits one (the first complete book from any of its
|
||||
connections wins and is the symbol's first Snapshot), and a resync's answer.
|
||||
Any other book is ignored: the second connection's opening book, a
|
||||
reconnect's replay for verified symbols, the books the server sends after a
|
||||
trade.
|
||||
|
||||
**Connections dropping.** A verified symbol needs nothing when one of its
|
||||
connections drops: the other carries on, and the dropped one's held changes
|
||||
are forgotten. When the followed connection drops, the other is followed and
|
||||
asked for a fresh book, and a Snapshot is always written on it, as the brief
|
||||
asks for a change of source. When every connection of a symbol is down, the
|
||||
symbol awaits a book again: the first reconnect's opening book is taken as
|
||||
fresh, and one that differs from ours is written as a Snapshot and logged as
|
||||
a loss.
|
||||
|
||||
## Placing, moving, splitting
|
||||
|
||||
- **Startup**: the first rescan completes (1 to 2.5 minutes on the server,
|
||||
Gamma's speed varying), then the markets are shuffled and dealt, each to two different
|
||||
connections, at about 5,000 markets a second so the opening books fit the
|
||||
link. Connections are opened for a fill of 4,500 tokens each.
|
||||
- **A new market** goes on two different random connections with room in both
|
||||
limits; a connection opens when fewer than two have room.
|
||||
- **Load**: a market's load is the bytes per second of its frames on one
|
||||
connection: its book thread counts every event's share from all of its
|
||||
connections and divides by their number. A connection's load is the sum of
|
||||
its markets'.
|
||||
- **Rebalancing**, each minute: a connection over its budget has the busiest
|
||||
markets' subscriptions moved, each to a random connection with room that
|
||||
does not already carry the market: at most 50 moves a minute, a moved
|
||||
subscription staying 10 minutes, none moved for a market with no load.
|
||||
- **A move** is `Move(market, from, to)` to the market's book thread, which
|
||||
subscribes on `to`, and unsubscribes on `from` once `to` has delivered a
|
||||
book of the market: until then both carry it, so no change falls between
|
||||
them. The book stays where it is: no Snapshot, no subscription gap, and the
|
||||
new connection's opening book is ignored like any unasked one. While a
|
||||
symbol is verified, a change is taken from any connection, the one it left
|
||||
included: its copies in flight are the exchange's changes like any other.
|
||||
- **A split**: a connection closed with 1013 twice within 10 minutes has every
|
||||
second market, in load order, moved off at once, outside the two caps, and
|
||||
is then left alone for 10 minutes.
|
||||
- **Removal**: unchanged; the book thread unsubscribes both connections.
|
||||
|
||||
## Data model
|
||||
|
||||
- **Orchestrator**: `conditionId` → (tokens, its two connections, load, when
|
||||
it last moved: both its subscriptions then stay); per connection, its token count, its load,
|
||||
its last 1013 closes and its last split. The market stays the unit: both
|
||||
tokens share the two connections.
|
||||
- **Book thread**, per symbol: the book and the hash inputs (sprint 02); its
|
||||
last 128 changes, each with its hash as 20 bytes (64 changes with the hash
|
||||
as text today: ~2.4 GB at worst instead of ~5 GB); the held changes of each
|
||||
of its connections; its state (awaiting a book, verified, following a
|
||||
connection, unverifiable and following one); its connections; its last
|
||||
Delta's receive time. Per market, the bytes since its last report. Per
|
||||
connection, whether it is up.
|
||||
- **Event** (connection → book thread): receive time, connection number,
|
||||
bytes, and a parsed book, change, tick size change or trade; or a
|
||||
connection up / down marker.
|
||||
- **Cause lines**: every Snapshot written because a book differed, and every
|
||||
Snapshot written for a change of followed connection, logs one line with
|
||||
its symbol, its exact receive time and its cause (loss, subscription gap,
|
||||
switch). One module, `src/cause.rs`, holds their format and their parser.
|
||||
- **A tick size change** breaks the chain on the connection that delivered
|
||||
it: the hashes change with the tick size, and the fresh book carries the
|
||||
new one.
|
||||
|
||||
## Reconcile and the measures
|
||||
|
||||
- `reconcile OUTPUT [LOG]`: with the log, each mismatch is paired with the
|
||||
cause line of its symbol and receive time; it prints the counts by cause
|
||||
and fails on an unexplained mismatch or an orphan Delta. Without, every
|
||||
mismatch is unexplained.
|
||||
- Each book thread's minute line: changes written (and verified), duplicates
|
||||
dropped, changes **rescued** (verified changes of a symbol on several
|
||||
connections that only one delivered, counted once 2 s old, when every copy
|
||||
has come), held changes dropped, chains broken, resyncs, books behind,
|
||||
subscription gaps, losses, switches, the second copies' lag (median and
|
||||
99th percentile), the most events waiting.
|
||||
- The orchestrator's: markets, connections, added, removed, moved, splits, the
|
||||
busiest connection's load.
|
||||
|
||||
## Decisions
|
||||
|
||||
- **First one wins, on one book per symbol** (ADR 0003). The book hash orders
|
||||
changes across connections as a sequence number would. *Rejected:* a source
|
||||
and a standby with a book each and a Snapshot per switch (my first
|
||||
proposal): it loses the change the source missed, keeps the source's
|
||||
receive time, and doubles the books.
|
||||
- **A symbol's two connections are chosen independently** (ADR 0003;
|
||||
supersedes ADR 0002's lanes, a connection and a book thread of its own,
|
||||
because twins were closed together: 14 times more time with both down than
|
||||
independent deals). *Rejected:* twin connections per lane.
|
||||
- **Markets are placed at random** (supersedes "the least-loaded lane", and
|
||||
"subscribe page by page at startup", because both build the clusters a load
|
||||
average cannot see: random twins had a third of the closes of creation
|
||||
order). Startup scans first, then deals. *Rejected:* least-loaded placement
|
||||
(a batch of 1,000 markets created together lands on one connection);
|
||||
dealing while scanning (needs the connection count before the scan ends).
|
||||
- **The connection thread parses and routes.** The stamp still comes first; a
|
||||
frame of several markets' books must be taken apart anyway; and the
|
||||
connection becomes the exchange-specific decoder Extension 3 wants.
|
||||
*Rejected:* forwarding the text after peeking at the market id.
|
||||
- **16 book threads, a market's thread from its id.** One handled ~30,000
|
||||
frames a second in sprint 02. *Rejected:* a thread per connection (the
|
||||
lanes); a routing table (shared state).
|
||||
- **Subscriptions go through the market's book thread**, which records the
|
||||
market before asking. *Rejected:* the orchestrator telling connections
|
||||
directly (a book can arrive before its thread knows the symbol).
|
||||
- **Ceilings of 1 s and 50 ms; at the ceiling, held changes are dropped if the
|
||||
book moved, else the chain is broken; a change older than the book is
|
||||
dropped at once.** The lag reached 0.7 s, batches 40 ms. The last two come
|
||||
from the prototype: a connection 2 s late was otherwise taken for a broken
|
||||
chain and followed. *Rejected:* breaking at once (a change one connection
|
||||
missed would cost a resync instead of nothing); no ceiling (a change could
|
||||
wait unwritten for minutes).
|
||||
- **A connection holds at most 512 changes of a symbol, and only the tails
|
||||
starting a batch are tried, five at most** (found by the alternating runs:
|
||||
at a 5-minute mark a crypto market's burst, behind one missing change, made
|
||||
every arrival try every tail of thousands, each a hash; one book thread
|
||||
then needed 9.5 minutes to drain after SIGINT). *Rejected:* trying every
|
||||
tail (quadratic); no cap (a burst piles up for its whole ceiling).
|
||||
- **Only asked-for books count.** A reconnect replays up to 5,000 books and
|
||||
the server sends books after trades; reacting to them is what made
|
||||
resyncs. *Rejected:* comparing every book.
|
||||
- **The resync goes to the connection not followed**, so the followed stream
|
||||
has no gap of its own. *Rejected:* resyncing the followed connection
|
||||
(sprint 03: 11,605 gaps).
|
||||
- **A split after two 1013 closes within 10 minutes.** One close can be the
|
||||
server's moment (several connections within a second); a connection closed
|
||||
again is overloaded. *Rejected:* splitting on the first close; lowering
|
||||
every connection's budget.
|
||||
- **Threads, not tokio** (ADR 0001's checkpoint, held here): the new logic is
|
||||
in the book thread, not in socket handling, and ~200 threads are
|
||||
comfortable. *Rejected:* an async socket layer now.
|
||||
- **Two subscriptions per market, a constant**; the comparison run is the same
|
||||
code at 1. *Rejected:* a runtime option.
|
||||
- **Loss means a change no connection delivered in time**; one a single
|
||||
connection missed is rescued. "Lane" is retired: connections and book
|
||||
threads are separate things now.
|
||||
- **`reconcile` reads the log, through one module shared with the
|
||||
collector.** *Rejected:* a second, machine-readable output.
|
||||
- **Rescued counts copies, not held changes** (supersedes "held, then fit
|
||||
after another connection moved the book", because a change one connection
|
||||
missed usually arrives on the other before the first's next change, and is
|
||||
then never held: the tests' rescues counted 0). *Rejected:* counting held
|
||||
changes that fit (most rescues go unseen).
|
||||
- **A move leaves the old connection once the new one delivers a book**
|
||||
(supersedes "subscribes on `to` and then unsubscribes on `from`", because
|
||||
unsubscribing at once loses the changes between `from`'s unsubscribe and
|
||||
`to`'s first book). *Rejected:* a fixed delay before unsubscribing.
|
||||
- **SIGTERM (and SIGHUP) stop the collector like SIGINT** (`ctrlc`'s
|
||||
`termination` feature): `timeout`, `docker stop` and a service manager send
|
||||
SIGTERM, and only a clean stop flushes the last chunk.
|
||||
- **Built in two steps, with a live run between**: first the connections
|
||||
separated from the book threads with one subscription per market, which
|
||||
must behave like sprint 03's collector; then the second subscription, first
|
||||
one wins and the split.
|
||||
|
||||
## Tests
|
||||
|
||||
No network, except the local websocket server of sprint 03's reconnect test.
|
||||
The fixture is re-recorded on two connections (`record.py`, extended) for the
|
||||
same 40 tokens; replays feed both in receive order.
|
||||
|
||||
- **Same output**: two connections give the Deltas, in order per symbol, that
|
||||
one alone gives; the other's copies are counted as duplicates.
|
||||
- **A loss on one connection**: a frame removed from one changes nothing in
|
||||
the output, and its changes count as rescued.
|
||||
- **A loss on both**: the same frame removed from both leaves the next change
|
||||
held; at the ceiling the chain breaks, the resync goes to the connection not
|
||||
followed, and a differing fresh book is written as a Snapshot, logged a loss.
|
||||
- **A slow connection**: one delayed by 2 s gives the same output, each Delta
|
||||
with the earlier receive time, and no chain broken by the delay.
|
||||
- **Stamps**: a held change is not stamped earlier than the symbol's previous
|
||||
Delta.
|
||||
- **A batch** is written only once it fits; a fresh book's late changes are
|
||||
dropped as stale.
|
||||
- **Books**: an unasked book changes nothing; the first of two opening books
|
||||
is the Snapshot.
|
||||
- **Connections dropping**: one down, the output goes on; the followed one
|
||||
down, a Snapshot on the fresh book; both down, every symbol awaits a book.
|
||||
- **Routing**: a frame of several markets' books reaches each market's book
|
||||
thread; a market's thread is the same from the orchestrator and from a
|
||||
connection.
|
||||
- **Placement**: a market's two connections differ; a move never targets the
|
||||
market's other connection; a connection opens when fewer than two have
|
||||
room; the startup deal is even.
|
||||
- **Move**: subscribe on the new connection, then unsubscribe on the old; no
|
||||
Snapshot.
|
||||
- **Split**: two closes within 10 minutes move every second market; one close
|
||||
moves none; a second split waits 10 minutes.
|
||||
- **Load**: a market on two connections reports the load of one.
|
||||
- **Reconcile with a log**: explained mismatches pass; an unexplained one
|
||||
fails.
|
||||
- Sprint 03's tests of Gamma, the minute check, rescan removal, rebalancing's
|
||||
caps and the reconnect stay, adapted to connections.
|
||||
|
||||
## Not in this sprint
|
||||
|
||||
- Acting on delay: the second delivery's lag is measured and logged; choosing
|
||||
connections by it is M7's.
|
||||
- The crash chunk: a hard crash still loses up to 64 MiB, ~10 s of output.
|
||||
- Balancing on a market's peak second rather than its minute average.
|
||||
- Reading the last trade price from the trade event to verify again without
|
||||
a resync (the prototype did it for 15 of 100 broken chains): a Possible
|
||||
feature.
|
||||
- Detecting a subscription gone quiet on one connection and resubscribing it,
|
||||
and recycling connections: M5. First one wins hides a quiet subscription;
|
||||
M5 has to find it.
|
||||
- A market's `min_order_size` or `neg_risk` changing: still read once.
|
||||
|
||||
## Done when
|
||||
|
||||
- `scripts/ci.sh` is green.
|
||||
- **Step 1** (one subscription per market): a 10-minute live run has every
|
||||
`reconcile` mismatch explained by the log.
|
||||
- **The comparison**, started when the user says so: two 20-minute live runs
|
||||
back to back, one subscription per market, then two. In the second:
|
||||
- every `reconcile` mismatch is explained by the log;
|
||||
- losses and subscription gaps per million changes are below the first
|
||||
run's;
|
||||
- rescued changes, duplicates, chains broken and the second delivery's lag
|
||||
are reported;
|
||||
- a split was seen, or its absence explained;
|
||||
- peak memory and thread count are reported.
|
||||
- Free disk is checked before each run, and the outputs deleted after.
|
||||
- ADR 0003 is written, ADR 0001 records the checkpoint, ADR 0002 says what
|
||||
was superseded; `GLOSSARY.md`, `docs/project.md`'s pipeline, `docs/notes.md`
|
||||
(the measures above) and the roadmap match.
|
||||
|
||||
## Outcome
|
||||
|
||||
Every live market (~198k, ~396k symbols) is on two connections chosen at
|
||||
random (~178 connections), and 16 book threads take each change from
|
||||
whichever connection delivers it first. Built in the two planned steps;
|
||||
step 1 (connections separated, one subscription) reconciled fully in a
|
||||
10-minute live run, no connection lost.
|
||||
|
||||
**The comparison** was run as four alternating 10-minute rounds, one
|
||||
subscription per market then two, twice (asked by the user once the server
|
||||
proved overloaded during a first back-to-back pair). Per million changes
|
||||
written:
|
||||
|
||||
| Round | Subscriptions | Losses | Subscription gaps | Losses + gaps | Chains broken |
|
||||
|---|---|---|---|---|---|
|
||||
| 1 | 1 | 69.1 | 163.7 | 232.8 | 525 |
|
||||
| 2 | 2 | 49.7 | 160.2 | 209.9 | 566 |
|
||||
| 3 | 1 | 42.1 | 142.0 | 184.1 | 530 |
|
||||
| 4 | 2 | 32.3 | 152.1 | 184.5 | 503 |
|
||||
|
||||
Every round reconciled with each mismatch explained by its log line. Losses
|
||||
fell by a quarter in each pair; losses and gaps together fell in the first
|
||||
pair and tied in the second, the rates drifting down through the hour. The
|
||||
user accepted this as the sprint's result. Subscription gaps, mostly from
|
||||
opening subscribes that both connections share, are untouched by a second
|
||||
connection. Most "losses" are likely not data missed: chains break as often
|
||||
with one connection as with two (about the rate of trades), and a resync's
|
||||
book that is a few changes ahead of our stream is logged a loss too.
|
||||
Duplicates: one per change written. The second copy came 1 to 2 ms after
|
||||
the first at the median, 45 to 120 ms at p99. Peak memory 1.78 GB with two
|
||||
subscriptions, 1.67 GB with one; 200 threads. No split: no connection was
|
||||
closed twice within 10 minutes (0 and 2 connections lost in the
|
||||
two-subscription rounds). The first, overloaded pair (load up to 92 on 12
|
||||
cores) still reconciled fully: a stress test passed.
|
||||
|
||||
**Rescued** (~110,000 per round) is an upper bound, accepted as such: a
|
||||
change that leaves the last 128 remembered before its second copy comes, on
|
||||
a busy symbol, is counted as delivered once; so is a batch whose changes the
|
||||
two connections delivered interleaved (the held copy dropped uncounted).
|
||||
|
||||
**Fixed at the close**, from the code review: a move ending while a fresh
|
||||
book was asked from the connection left kept asking it in vain (now asked
|
||||
from another); a change held across a minute could be stamped before that
|
||||
minute's Snapshot (a Snapshot now advances the symbol's last stamp); markets
|
||||
closed during the first scan were dealt anyway. The first two have a test
|
||||
that fails without the fix; the third sits in the orchestrator's loop, as
|
||||
untested as the rest of it.
|
||||
|
||||
**Diverged from the plan**, each in Decisions: markets on independent
|
||||
connections, not twins (the user's design, after the three-layout test);
|
||||
rescued counted from copies; a move leaves the old connection once the new
|
||||
one delivers a book; the held-change cap and batch-start tails (found by the
|
||||
runs: a crypto burst behind one missing change made a book thread fall
|
||||
455,000 events behind and take 9.5 minutes to stop); a tick size change
|
||||
breaks the chain; cause lines carry the exact receive time.
|
||||
|
||||
**Left out**, and where it went: telling a real loss from a resync book
|
||||
ahead of our stream, and the opening subscription gaps (Possible features);
|
||||
an exact rescued count (Possible features); acting on delay (M7); the crash
|
||||
chunk (M4's ceiling, kept); quiet subscriptions hidden by first one wins
|
||||
(M5).
|
||||
|
||||
### Retrospective
|
||||
|
||||
- **Went well:** experiments settled the design before code: two recordings
|
||||
showed the hash orders changes across connections, the three-layout test
|
||||
turned a twin design into the user's independent one, a Python prototype
|
||||
found the slow-connection flaw before any Rust. `reconcile` reading the log
|
||||
made every live run a correctness check, and it held even under load.
|
||||
- **Went wrong:** a quadratic loop marked "fine at a few dozen" met bursts of
|
||||
thousands; it surfaced only as a book thread falling behind, then as a
|
||||
shutdown that took 9.5 minutes. The first comparison pair ran on an
|
||||
overloaded server and wasn't comparable. Wait loops on `pgrep -f` failed
|
||||
twice more, once killing the wrong process.
|
||||
- **Tested enough?** The rules are, on a two-connection recording with
|
||||
injected losses, delays and drops, and a burst test that hung before the
|
||||
fix. Not covered: the orchestrator's loop (pacing the deal, routing
|
||||
commands, splits sent), the socket's up/down markers, a move whose new
|
||||
connection never delivers, and the minute line's lag figures.
|
||||
- **Next time:** load-test the book thread with a synthetic burst before a
|
||||
live run, and check the server's load before a comparison; keep the
|
||||
ceiling of every corner cut in a test, not just a comment.
|
||||
+17
-9
@@ -1,11 +1,14 @@
|
||||
//! `reconcile OUTPUT`: runs the reconciliation check on a collector's output
|
||||
//! and fails if any Snapshot disagrees with the Deltas before it.
|
||||
//! `reconcile OUTPUT [LOG]`: runs the reconciliation check on a collector's
|
||||
//! output and fails on an orphan Delta, or on a Snapshot that disagrees with
|
||||
//! the Deltas before it, unless the collector's log explains it.
|
||||
|
||||
fn main() -> anyhow::Result<()> {
|
||||
let path = std::env::args()
|
||||
.nth(1)
|
||||
.ok_or_else(|| anyhow::anyhow!("usage: reconcile OUTPUT"))?;
|
||||
let report = engineering_challenge::reconcile::reconcile(path)?;
|
||||
let mut args = std::env::args().skip(1);
|
||||
let path = args
|
||||
.next()
|
||||
.ok_or_else(|| anyhow::anyhow!("usage: reconcile OUTPUT [LOG]"))?;
|
||||
let log = args.next().map(std::path::PathBuf::from);
|
||||
let report = engineering_challenge::reconcile::reconcile(path, log.as_deref())?;
|
||||
println!(
|
||||
"{} snapshots, {} deltas, {} orphan deltas, {} mismatches",
|
||||
report.snapshots,
|
||||
@@ -13,11 +16,16 @@ fn main() -> anyhow::Result<()> {
|
||||
report.orphan_deltas,
|
||||
report.mismatches.len()
|
||||
);
|
||||
for (symbol, received_at) in &report.mismatches {
|
||||
println!("mismatch: {symbol} at {received_at}");
|
||||
let mut explained: Vec<_> = report.explained.iter().collect();
|
||||
explained.sort_by_key(|(cause, _)| cause.to_string());
|
||||
for (cause, count) in explained {
|
||||
println!("explained by {cause}: {count}");
|
||||
}
|
||||
for (symbol, received_at) in &report.unexplained {
|
||||
println!("unexplained: {symbol} at {received_at}");
|
||||
}
|
||||
anyhow::ensure!(
|
||||
report.mismatches.is_empty() && report.orphan_deltas == 0,
|
||||
report.unexplained.is_empty() && report.orphan_deltas == 0,
|
||||
"the output doesn't reconcile"
|
||||
);
|
||||
Ok(())
|
||||
|
||||
@@ -0,0 +1,80 @@
|
||||
//! Why a Snapshot outside the minute was written: one log line per Snapshot,
|
||||
//! written by the collector and read back by `reconcile`, which pairs it with
|
||||
//! a mismatch by symbol and receive time.
|
||||
|
||||
use std::fmt;
|
||||
|
||||
use chrono::{DateTime, Utc};
|
||||
|
||||
/// What a Snapshot that may differ from the book before it stands for.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
|
||||
pub enum Cause {
|
||||
/// A fresh book differed from ours: changes no connection delivered.
|
||||
Loss,
|
||||
/// A fresh book differed from ours before any change of the symbol was
|
||||
/// verified: changes our own subscribe kept from us.
|
||||
SubscriptionGap,
|
||||
/// The connection followed changed: the new one's book, which may hold
|
||||
/// changes the old one never delivered.
|
||||
Switch,
|
||||
}
|
||||
|
||||
impl Cause {
|
||||
const ALL: [Cause; 3] = [Cause::Loss, Cause::SubscriptionGap, Cause::Switch];
|
||||
|
||||
fn name(self) -> &'static str {
|
||||
match self {
|
||||
Cause::Loss => "loss",
|
||||
Cause::SubscriptionGap => "subscription gap",
|
||||
Cause::Switch => "switch",
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl fmt::Display for Cause {
|
||||
fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
|
||||
f.write_str(self.name())
|
||||
}
|
||||
}
|
||||
|
||||
const MARK: &str = "Snapshot of ";
|
||||
|
||||
/// The line's text: `Snapshot of <symbol> at <receive time, µs>: <cause>`,
|
||||
/// then any detail for a reader.
|
||||
pub fn line(symbol: &str, received_at: DateTime<Utc>, cause: Cause, detail: &str) -> String {
|
||||
let at = received_at.timestamp_micros();
|
||||
format!("{MARK}{symbol} at {at}: {cause}{detail}")
|
||||
}
|
||||
|
||||
/// The symbol, receive time and cause of a log line written from [`line`],
|
||||
/// wherever it sits in the line.
|
||||
pub fn parse(log_line: &str) -> Option<(String, DateTime<Utc>, Cause)> {
|
||||
let rest = &log_line[log_line.find(MARK)? + MARK.len()..];
|
||||
let (symbol, rest) = rest.split_once(" at ")?;
|
||||
let (at, rest) = rest.split_once(": ")?;
|
||||
let received_at = DateTime::from_timestamp_micros(at.parse().ok()?)?;
|
||||
let cause = Cause::ALL
|
||||
.into_iter()
|
||||
.find(|cause| rest.starts_with(cause.name()))?;
|
||||
Some((symbol.to_string(), received_at, cause))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn a_line_reads_back() {
|
||||
let at = DateTime::from_timestamp_micros(1_794_390_726_365_123).unwrap();
|
||||
for cause in Cause::ALL {
|
||||
let text = line("123", at, cause, " (its time 12:00:00.000)");
|
||||
let logged = format!("12:00:00.001 polymarket: {text}");
|
||||
assert_eq!(parse(&logged), Some(("123".to_string(), at, cause)));
|
||||
}
|
||||
assert_eq!(
|
||||
parse("12:00:00.001 polymarket: connection 3: connected"),
|
||||
None
|
||||
);
|
||||
assert_eq!(parse("Snapshot of 1 at soon: loss"), None);
|
||||
}
|
||||
}
|
||||
@@ -1,4 +1,5 @@
|
||||
pub mod book;
|
||||
pub mod cause;
|
||||
pub mod reconcile;
|
||||
pub mod util;
|
||||
|
||||
|
||||
+10
-8
@@ -1,7 +1,7 @@
|
||||
//! The collector (roadmap/03): an orchestrator keeps the tracked markets equal
|
||||
//! to Gamma's live ones and spreads them over lanes, each a connection (read,
|
||||
//! stamp) and a book thread (parse, fold, check, resync), all writing through
|
||||
//! one writer (SBE).
|
||||
//! The collector (roadmap/04): an orchestrator keeps the tracked markets equal
|
||||
//! to Gamma's live ones and places them on connections (read, stamp, decode,
|
||||
//! route), which feed the book threads (fold, check, resync), all writing
|
||||
//! through one writer (SBE).
|
||||
//! Usage: `engineering-challenge [OUTPUT]`, OUTPUT defaulting to stdout; logs
|
||||
//! go to stderr.
|
||||
|
||||
@@ -43,9 +43,10 @@ fn main() -> anyhow::Result<()> {
|
||||
writer_stop.store(true, Ordering::Relaxed);
|
||||
written
|
||||
});
|
||||
// SIGINT stops the orchestrator and the connections; each book thread
|
||||
// stops when its connection does, and the writer flushes once they all
|
||||
// have, even after a panic upstream (the run still fails then).
|
||||
// SIGINT (and SIGTERM, SIGHUP) stops the orchestrator and the
|
||||
// connections; the book threads stop once every connection has, and the
|
||||
// writer flushes once they all have, even after a panic upstream (the run
|
||||
// still fails then).
|
||||
let failed = polymarket::orchestrator::run(outputs_tx, &stop);
|
||||
let written = writer
|
||||
.join()
|
||||
@@ -60,7 +61,8 @@ fn main() -> anyhow::Result<()> {
|
||||
/// thread is gone, then flushes the last chunk.
|
||||
// ponytail: the chunk being filled lives only in memory until it is full, so
|
||||
// a hard crash loses it; flushing more often pads every flush to 64 MiB.
|
||||
// Taken up in M4.
|
||||
// Kept as a ceiling in M4 (roadmap/04): writing to a file of our own and
|
||||
// copying it into chunks would lift it.
|
||||
fn write(path: &str, outputs: Receiver<Output>) -> anyhow::Result<()> {
|
||||
let mut writer = AlignedSbeWriter::create(path, CHUNK)?;
|
||||
for output in outputs {
|
||||
|
||||
+1965
-671
File diff suppressed because it is too large.
Load diff
+37
-2
@@ -1,10 +1,25 @@
|
||||
//! Polymarket: which markets are live (Gamma), the orchestrator that spreads
|
||||
//! them over lanes, and each lane's connection and feed (roadmap/03).
|
||||
//! Polymarket: which markets are live (Gamma), the orchestrator that places
|
||||
//! them on connections, the connections, and the book threads they feed
|
||||
//! (docs/adr/0003-first-one-wins.md).
|
||||
|
||||
pub mod feed;
|
||||
pub mod gamma;
|
||||
pub mod orchestrator;
|
||||
pub mod socket;
|
||||
pub mod wire;
|
||||
|
||||
/// How many book threads own the books. One handled ~30,000 frames a second
|
||||
/// in sprint 02.
|
||||
pub const BOOK_THREADS: usize = 16;
|
||||
|
||||
/// The book thread that owns a market's books, from its `conditionId` (`0x`
|
||||
/// and 64 hex digits, a hash: any 16 of them spread evenly). Computed, so the
|
||||
/// orchestrator and every connection agree without a shared table.
|
||||
pub fn book_thread(condition_id: &str) -> usize {
|
||||
let digits = condition_id.get(2..18).unwrap_or_default();
|
||||
// An id of another shape still gets a thread, always the same one.
|
||||
u64::from_str_radix(digits, 16).unwrap_or(0) as usize % BOOK_THREADS
|
||||
}
|
||||
|
||||
/// A live market: its symbols and the two inputs of their book hash that the
|
||||
/// websocket doesn't carry (docs/notes.md).
|
||||
@@ -23,3 +38,23 @@ pub struct Market {
|
||||
pub(crate) fn fixture_markets() -> Vec<Market> {
|
||||
gamma::parse_markets(include_str!("../../tests/fixtures/gamma-markets.json"))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn a_market_always_gets_the_same_book_thread_and_markets_spread() {
|
||||
let markets = fixture_markets();
|
||||
let threads: std::collections::BTreeSet<usize> = markets
|
||||
.iter()
|
||||
.map(|market| book_thread(&market.condition_id))
|
||||
.collect();
|
||||
assert!(threads.iter().all(|&thread| thread < BOOK_THREADS));
|
||||
// 20 markets over 16 threads: several threads, not one.
|
||||
assert!(threads.len() > 5, "{threads:?}");
|
||||
let id = &markets[0].condition_id;
|
||||
assert_eq!(book_thread(id), book_thread(&id.clone()));
|
||||
assert_eq!(book_thread("not an id"), 0);
|
||||
}
|
||||
}
|
||||
+669
-330
File diff suppressed because it is too large.
Load diff
+152
-26
@@ -1,6 +1,6 @@
|
||||
//! A lane's connection: reads the market channel and stamps each frame's
|
||||
//! receive time, and sends what its book thread asks for (subscriptions,
|
||||
//! resyncs). Parsing is the book thread's.
|
||||
//! A connection: reads the market channel, stamps each frame's receive time,
|
||||
//! decodes it and sends each event to the book thread of its market; and
|
||||
//! sends the exchange what the book threads ask for (subscriptions, resyncs).
|
||||
|
||||
use std::collections::BTreeSet;
|
||||
use std::io::ErrorKind;
|
||||
@@ -13,8 +13,12 @@ use chrono::{DateTime, Utc};
|
||||
use crossbeam::channel::{Receiver, Sender};
|
||||
use serde_json::json;
|
||||
use tungstenite::http::Uri;
|
||||
use tungstenite::protocol::frame::coding::CloseCode;
|
||||
use tungstenite::stream::MaybeTlsStream;
|
||||
use tungstenite::{Message, Utf8Bytes, WebSocket};
|
||||
use tungstenite::{Message, WebSocket};
|
||||
|
||||
use super::book_thread;
|
||||
use super::wire::{self, Arrival, Received};
|
||||
|
||||
pub const MARKET_CHANNEL_URL: &str = "wss://ws-subscriptions-clob.polymarket.com/ws/market";
|
||||
/// For the TCP connect and the TLS + websocket handshake, and for every send.
|
||||
@@ -29,10 +33,7 @@ const CLOSE_TIMEOUT: Duration = Duration::from_secs(2);
|
||||
/// 500 gets no opening books (docs/notes.md).
|
||||
const CHUNK: usize = 500;
|
||||
|
||||
/// A frame's text with its receive time.
|
||||
pub type Frame = (DateTime<Utc>, Utf8Bytes);
|
||||
|
||||
/// What the book thread asks its connection for.
|
||||
/// What a book thread asks a connection for.
|
||||
#[derive(Debug, PartialEq, Eq)]
|
||||
pub enum Command {
|
||||
Subscribe(Vec<String>),
|
||||
@@ -43,15 +44,23 @@ pub enum Command {
|
||||
|
||||
type Socket = WebSocket<MaybeTlsStream<TcpStream>>;
|
||||
|
||||
/// Runs until `stop` is set or the book thread is gone, reconnecting (and
|
||||
/// subscribing its tokens again) whenever the connection drops.
|
||||
/// Runs until `stop` is set or a book thread is gone, reconnecting (and
|
||||
/// subscribing its tokens again) whenever the connection drops, and telling
|
||||
/// every book thread when it comes up and goes down. `book_threads` holds
|
||||
/// every book thread's channel, by number; `closes` hears of each 1013 close.
|
||||
pub fn run(
|
||||
url: &str,
|
||||
lane: usize,
|
||||
frames: &Sender<Frame>,
|
||||
number: usize,
|
||||
book_threads: &[Sender<Received>],
|
||||
commands: &Receiver<Command>,
|
||||
closes: &Sender<usize>,
|
||||
stop: &AtomicBool,
|
||||
) {
|
||||
let tell = |received: fn(usize) -> Received| {
|
||||
for book_thread in book_threads {
|
||||
_ = book_thread.send(received(number));
|
||||
}
|
||||
};
|
||||
let mut tokens = BTreeSet::new();
|
||||
while !stop.load(Ordering::Relaxed) {
|
||||
// Asked for while disconnected: the subscribe on connect covers them,
|
||||
@@ -62,16 +71,36 @@ pub fn run(
|
||||
match connect(url, &tokens) {
|
||||
Ok(mut socket) => {
|
||||
log!(
|
||||
"polymarket: lane {lane}: connected, {} symbols",
|
||||
"polymarket: connection {number}: connected, {} symbols",
|
||||
tokens.len()
|
||||
);
|
||||
match read(&mut socket, &mut tokens, frames, commands, stop) {
|
||||
Ok(Ended::Stopped) => return close(&mut socket, lane, frames),
|
||||
Ok(Ended::BookThreadGone) => return,
|
||||
Err(error) => log!("polymarket: lane {lane}: connection lost: {error}"),
|
||||
tell(Received::Up);
|
||||
let ended = read(
|
||||
&mut socket,
|
||||
number,
|
||||
&mut tokens,
|
||||
book_threads,
|
||||
commands,
|
||||
closes,
|
||||
stop,
|
||||
);
|
||||
if !matches!(ended, Ok(Ended::Stopped)) {
|
||||
tell(Received::Down);
|
||||
}
|
||||
match ended {
|
||||
Ok(Ended::Stopped) => return close(&mut socket, number, book_threads),
|
||||
Ok(Ended::BookThreadGone) => {
|
||||
// Only a panic ends a book thread before us: its
|
||||
// markets would go unserved, so the run stops.
|
||||
log!("polymarket: connection {number}: a book thread is gone, stopping");
|
||||
return stop.store(true, Ordering::Relaxed);
|
||||
}
|
||||
Err(error) => {
|
||||
log!("polymarket: connection {number}: connection lost: {error}")
|
||||
}
|
||||
}
|
||||
Err(error) => log!("polymarket: lane {lane}: connect failed: {error}"),
|
||||
}
|
||||
Err(error) => log!("polymarket: connection {number}: connect failed: {error}"),
|
||||
}
|
||||
thread::sleep(RECONNECT_AFTER);
|
||||
}
|
||||
@@ -150,11 +179,53 @@ fn connect(url: &str, tokens: &BTreeSet<String>) -> anyhow::Result<Socket> {
|
||||
Ok(socket)
|
||||
}
|
||||
|
||||
/// Decodes a frame and sends each event to the book thread of its market;
|
||||
/// `false` once a book thread is gone.
|
||||
fn route(
|
||||
received_at: DateTime<Utc>,
|
||||
text: &str,
|
||||
number: usize,
|
||||
book_threads: &[Sender<Received>],
|
||||
) -> bool {
|
||||
let events = match wire::decode(text) {
|
||||
Ok(events) => events,
|
||||
Err(error) => {
|
||||
log!("polymarket: unreadable frame ({error}): {}", truncate(text));
|
||||
return true;
|
||||
}
|
||||
};
|
||||
let bytes = (text.len() / events.len().max(1)) as u64;
|
||||
for event in events {
|
||||
// An event of no market (none we ask for) has no book thread.
|
||||
let Some(thread) = event.market().map(book_thread) else {
|
||||
continue;
|
||||
};
|
||||
let arrival = Arrival {
|
||||
received_at,
|
||||
connection: number,
|
||||
bytes,
|
||||
event,
|
||||
};
|
||||
if book_threads[thread].send(Received::Event(arrival)).is_err() {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
true
|
||||
}
|
||||
|
||||
fn truncate(text: &str) -> &str {
|
||||
text.char_indices()
|
||||
.nth(200)
|
||||
.map_or(text, |(end, _)| &text[..end])
|
||||
}
|
||||
|
||||
fn read(
|
||||
socket: &mut Socket,
|
||||
number: usize,
|
||||
tokens: &mut BTreeSet<String>,
|
||||
frames: &Sender<Frame>,
|
||||
book_threads: &[Sender<Received>],
|
||||
commands: &Receiver<Command>,
|
||||
closes: &Sender<usize>,
|
||||
stop: &AtomicBool,
|
||||
) -> anyhow::Result<Ended> {
|
||||
let mut last_ping = Instant::now();
|
||||
@@ -164,11 +235,21 @@ fn read(
|
||||
let received_at = Utc::now();
|
||||
match message {
|
||||
Ok(Message::Text(text)) => {
|
||||
if frames.send((received_at, text)).is_err() {
|
||||
if !route(received_at, text.as_str(), number, book_threads) {
|
||||
return Ok(Ended::BookThreadGone);
|
||||
}
|
||||
}
|
||||
Ok(Message::Close(frame)) => anyhow::bail!("closed by the server: {frame:?}"),
|
||||
Ok(Message::Close(frame)) => {
|
||||
// 1013: the server couldn't send to us fast enough. The
|
||||
// orchestrator splits a connection closed so twice.
|
||||
if frame
|
||||
.as_ref()
|
||||
.is_some_and(|frame| frame.code == CloseCode::Again)
|
||||
{
|
||||
_ = closes.send(number);
|
||||
}
|
||||
anyhow::bail!("closed by the server: {frame:?}")
|
||||
}
|
||||
Ok(_) => {}
|
||||
Err(tungstenite::Error::Io(error)) if timed_out(&error) => {}
|
||||
Err(error) => return Err(error.into()),
|
||||
@@ -191,16 +272,16 @@ fn read(
|
||||
|
||||
/// Closes the connection, forwarding what still arrives until the server
|
||||
/// confirms or [`CLOSE_TIMEOUT`] passes.
|
||||
fn close(socket: &mut Socket, lane: usize, frames: &Sender<Frame>) {
|
||||
fn close(socket: &mut Socket, number: usize, book_threads: &[Sender<Received>]) {
|
||||
if let Err(error) = socket.close(None) {
|
||||
return log!("polymarket: lane {lane}: close failed: {error}");
|
||||
return log!("polymarket: connection {number}: close failed: {error}");
|
||||
}
|
||||
let deadline = Instant::now() + CLOSE_TIMEOUT;
|
||||
while Instant::now() < deadline {
|
||||
let message = socket.read();
|
||||
let received_at = Utc::now();
|
||||
match message {
|
||||
Ok(Message::Text(text)) => _ = frames.send((received_at, text)),
|
||||
Ok(Message::Text(text)) => _ = route(received_at, text.as_str(), number, book_threads),
|
||||
Ok(_) => {}
|
||||
Err(tungstenite::Error::Io(error)) if timed_out(&error) => {}
|
||||
// ConnectionClosed: the close handshake is done.
|
||||
@@ -263,14 +344,16 @@ mod tests {
|
||||
connections
|
||||
});
|
||||
|
||||
let (frames_tx, _frames_rx) = unbounded();
|
||||
let (events_tx, _events_rx) = unbounded();
|
||||
let (commands_tx, commands_rx) = unbounded();
|
||||
commands_tx
|
||||
.send(Command::Subscribe(tokens.clone()))
|
||||
.unwrap();
|
||||
let stop = Arc::new(AtomicBool::new(false));
|
||||
let client_stop = Arc::clone(&stop);
|
||||
let client = thread::spawn(move || run(&url, 0, &frames_tx, &commands_rx, &client_stop));
|
||||
let (closes, _closes_rx) = unbounded();
|
||||
let client =
|
||||
thread::spawn(move || run(&url, 0, &[events_tx], &commands_rx, &closes, &client_stop));
|
||||
|
||||
let connections = server.join().unwrap();
|
||||
stop.store(true, Ordering::Relaxed);
|
||||
@@ -285,6 +368,49 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
/// The recording's first frame holds the opening books of 40 tokens, 20
|
||||
/// markets: each book goes to its market's book thread, with its share of
|
||||
/// the frame's bytes.
|
||||
#[test]
|
||||
fn a_frame_of_several_markets_reaches_each_markets_book_thread() {
|
||||
#[derive(serde::Deserialize)]
|
||||
struct Line {
|
||||
text: String,
|
||||
}
|
||||
let path = concat!(
|
||||
env!("CARGO_MANIFEST_DIR"),
|
||||
"/tests/fixtures/polymarket-frames.jsonl"
|
||||
);
|
||||
let recording = std::fs::read_to_string(path).unwrap();
|
||||
let line: Line = serde_json::from_str(recording.lines().next().unwrap()).unwrap();
|
||||
let (senders, receivers): (Vec<_>, Vec<_>) = (0..crate::polymarket::BOOK_THREADS)
|
||||
.map(|_| unbounded())
|
||||
.unzip();
|
||||
|
||||
assert!(route(Utc::now(), &line.text, 3, &senders));
|
||||
let mut books = 0;
|
||||
for (thread, receiver) in receivers.iter().enumerate() {
|
||||
for received in receiver.try_iter() {
|
||||
let Received::Event(arrival) = received else {
|
||||
panic!("not an event: {received:?}");
|
||||
};
|
||||
let market = arrival.event.market().unwrap().to_string();
|
||||
assert_eq!(book_thread(&market), thread);
|
||||
assert_eq!(arrival.connection, 3);
|
||||
assert_eq!(arrival.bytes, line.text.len() as u64 / 40);
|
||||
books += 1;
|
||||
}
|
||||
}
|
||||
assert_eq!(books, 40);
|
||||
// Neither a keepalive's answer nor an unreadable frame reaches one.
|
||||
assert!(route(Utc::now(), "PONG", 3, &senders));
|
||||
assert!(route(Utc::now(), "not json", 3, &senders));
|
||||
assert!(receivers.iter().all(|receiver| receiver.is_empty()));
|
||||
// A book thread gone stops the connection.
|
||||
drop(receivers);
|
||||
assert!(!route(Utc::now(), &line.text, 3, &senders));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn commands_update_the_token_set_and_say_what_to_send() {
|
||||
let mut tokens = BTreeSet::new();
|
||||
|
||||
@@ -0,0 +1,120 @@
|
||||
//! The market channel's messages, as the exchange writes them, and how a
|
||||
//! frame's text becomes events (docs/notes.md).
|
||||
|
||||
use chrono::{DateTime, Utc};
|
||||
use serde::Deserialize;
|
||||
|
||||
/// What a connection sends a book thread.
|
||||
// Nearly every message is an event: boxing it to shrink the two rare markers
|
||||
// would cost an allocation per event for nothing.
|
||||
#[allow(clippy::large_enum_variant)]
|
||||
#[derive(Debug)]
|
||||
pub enum Received {
|
||||
Event(Arrival),
|
||||
/// The connection came up: its subscriptions are being sent.
|
||||
Up(usize),
|
||||
/// The connection went down: nothing more comes from it until it is up.
|
||||
Down(usize),
|
||||
}
|
||||
|
||||
/// One event of a frame on its way to its market's book thread.
|
||||
#[derive(Debug)]
|
||||
pub struct Arrival {
|
||||
/// The frame's receive time, stamped right after the socket read.
|
||||
pub received_at: DateTime<Utc>,
|
||||
/// The connection that delivered it.
|
||||
pub connection: usize,
|
||||
/// Its share of the frame's bytes: a frame is one market's change, or
|
||||
/// books of several, each taking its share.
|
||||
pub bytes: u64,
|
||||
pub event: Event,
|
||||
}
|
||||
|
||||
/// The events of a frame's text; none for the keepalive's `PONG` and for
|
||||
/// `NO NEW ASSETS`.
|
||||
pub fn decode(text: &str) -> Result<Vec<Event>, serde_json::Error> {
|
||||
match text.trim() {
|
||||
"PONG" | "NO NEW ASSETS" => Ok(Vec::new()),
|
||||
text => Ok(match serde_json::from_str(text)? {
|
||||
Events::Many(events) => events,
|
||||
Events::One(event) => vec![event],
|
||||
}),
|
||||
}
|
||||
}
|
||||
|
||||
impl Event {
|
||||
/// The market it belongs to, which decides its book thread.
|
||||
pub fn market(&self) -> Option<&str> {
|
||||
match self {
|
||||
Event::Book(book) => Some(&book.market),
|
||||
Event::PriceChange(change) => Some(&change.market),
|
||||
Event::TickSizeChange { market, .. } | Event::LastTradePrice { market } => {
|
||||
market.as_deref()
|
||||
}
|
||||
Event::Other => None,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Deserialize)]
|
||||
#[serde(untagged)]
|
||||
enum Events {
|
||||
Many(Vec<Event>),
|
||||
One(Event),
|
||||
}
|
||||
|
||||
#[derive(Debug, Deserialize)]
|
||||
#[serde(tag = "event_type", rename_all = "snake_case")]
|
||||
pub enum Event {
|
||||
Book(BookMessage),
|
||||
PriceChange(PriceChangeMessage),
|
||||
/// Only the symbol and its market are read: the hashes change with the
|
||||
/// tick size, and a fresh book carries the new one. The server sends each
|
||||
/// one twice (docs/notes.md). `market` was on every one seen; one without
|
||||
/// it can't be routed and is dropped.
|
||||
TickSizeChange {
|
||||
market: Option<String>,
|
||||
asset_id: String,
|
||||
},
|
||||
LastTradePrice {
|
||||
market: Option<String>,
|
||||
},
|
||||
/// Dropped: the others come only with `custom_feature_enabled`, which we
|
||||
/// don't ask for.
|
||||
#[serde(other)]
|
||||
Other,
|
||||
}
|
||||
|
||||
#[derive(Debug, Deserialize)]
|
||||
pub struct BookMessage {
|
||||
pub market: String,
|
||||
pub asset_id: String,
|
||||
pub timestamp: String,
|
||||
pub hash: String,
|
||||
pub bids: Vec<LevelMessage>,
|
||||
pub asks: Vec<LevelMessage>,
|
||||
pub tick_size: Option<String>,
|
||||
pub last_trade_price: Option<String>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Deserialize)]
|
||||
pub struct LevelMessage {
|
||||
pub price: String,
|
||||
pub size: String,
|
||||
}
|
||||
|
||||
#[derive(Debug, Deserialize)]
|
||||
pub struct PriceChangeMessage {
|
||||
pub market: String,
|
||||
pub timestamp: String,
|
||||
pub price_changes: Vec<ChangeMessage>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Deserialize)]
|
||||
pub struct ChangeMessage {
|
||||
pub asset_id: String,
|
||||
pub price: String,
|
||||
pub size: String,
|
||||
pub side: String,
|
||||
pub hash: String,
|
||||
}
|
||||
+93
-5
@@ -1,11 +1,15 @@
|
||||
//! Checks an output file against itself: per symbol, the Deltas written since
|
||||
//! a Snapshot, applied to it, must give the next Snapshot. A mismatch is a loss
|
||||
//! or a subscription gap (the fresh book differs), each logged by the collector.
|
||||
//! a Snapshot, applied to it, must give the next Snapshot. A Snapshot that
|
||||
//! doesn't is a mismatch, and the collector logs each with its cause
|
||||
//! ([`crate::cause`]): with the log, every mismatch must have its line.
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::collections::{HashMap, HashSet};
|
||||
use std::path::Path;
|
||||
|
||||
use chrono::{DateTime, Utc};
|
||||
|
||||
use crate::book::Book;
|
||||
use crate::cause::{self, Cause};
|
||||
use crate::util::dtos::challenge_orderbook::{AnyMessage, PriceLevelUpdateDomain, SnapshotDomain};
|
||||
use crate::util::{AlignedSbeReader, OrderbookSchema};
|
||||
|
||||
@@ -16,10 +20,15 @@ pub struct Report {
|
||||
/// Deltas for a symbol with no Snapshot yet: the collector never writes one.
|
||||
pub orphan_deltas: usize,
|
||||
/// Each Snapshot that didn't match its symbol's book: (symbol, its `received_at`).
|
||||
pub mismatches: Vec<(String, chrono::DateTime<chrono::Utc>)>,
|
||||
pub mismatches: Vec<(String, DateTime<Utc>)>,
|
||||
/// The mismatches each cause explains, with the log.
|
||||
pub explained: HashMap<Cause, usize>,
|
||||
/// The mismatches no line of the log explains; all of them without a log.
|
||||
pub unexplained: Vec<(String, DateTime<Utc>)>,
|
||||
}
|
||||
|
||||
pub fn reconcile(path: impl AsRef<Path>) -> anyhow::Result<Report> {
|
||||
/// Reads the output at `path` and, if given, the collector's log.
|
||||
pub fn reconcile(path: impl AsRef<Path>, log: Option<&Path>) -> anyhow::Result<Report> {
|
||||
let mut reader = AlignedSbeReader::<OrderbookSchema>::open(path)?;
|
||||
let mut books: HashMap<String, Book> = HashMap::new();
|
||||
let mut report = Report::default();
|
||||
@@ -61,5 +70,84 @@ pub fn reconcile(path: impl AsRef<Path>) -> anyhow::Result<Report> {
|
||||
_ => anyhow::bail!("unknown message in the output"),
|
||||
}
|
||||
}
|
||||
let causes: HashMap<(String, DateTime<Utc>), Cause> = match log {
|
||||
Some(log) => std::fs::read_to_string(log)?
|
||||
.lines()
|
||||
.filter_map(cause::parse)
|
||||
.map(|(symbol, received_at, cause)| ((symbol, received_at), cause))
|
||||
.collect(),
|
||||
None => HashMap::new(),
|
||||
};
|
||||
let mut seen = HashSet::new();
|
||||
for mismatch in &report.mismatches {
|
||||
match causes.get(mismatch) {
|
||||
// A line explains one Snapshot: the same one twice would be a bug.
|
||||
Some(cause) if seen.insert(mismatch.clone()) => {
|
||||
*report.explained.entry(*cause).or_default() += 1
|
||||
}
|
||||
_ => report.unexplained.push(mismatch.clone()),
|
||||
}
|
||||
}
|
||||
Ok(report)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::Side;
|
||||
use crate::util::AlignedSbeWriter;
|
||||
use crate::util::dtos::challenge_orderbook::{PriceLevelUpdateDomain, SnapshotBidsEntryDomain};
|
||||
|
||||
/// An output with a symbol's Snapshot, a Delta, then a Snapshot that
|
||||
/// misses it; and a log line for that second Snapshot when `explained`.
|
||||
fn reconcile_with(explained: bool) -> Report {
|
||||
let dir = std::env::temp_dir();
|
||||
let id = format!("{}-{explained}", std::process::id());
|
||||
let (output, log) = (
|
||||
dir.join(format!("reconcile-{id}.sbe")),
|
||||
dir.join(format!("reconcile-{id}.log")),
|
||||
);
|
||||
let at = |seconds: i64| DateTime::from_timestamp(1_794_390_000 + seconds, 0).unwrap();
|
||||
let snapshot = |received_at, quantity| SnapshotDomain {
|
||||
received_at,
|
||||
exchange_timestamp: received_at,
|
||||
bids: vec![SnapshotBidsEntryDomain {
|
||||
price_level: 500_000,
|
||||
quantity,
|
||||
}],
|
||||
asks: Vec::new(),
|
||||
symbol: "1".to_string(),
|
||||
};
|
||||
let mut writer = AlignedSbeWriter::create(output.to_str().unwrap(), 64 << 20).unwrap();
|
||||
writer.write_msg(&snapshot(at(0), 1_000_000)).unwrap();
|
||||
writer
|
||||
.write_msg(&PriceLevelUpdateDomain {
|
||||
backtest_timestamp: at(1),
|
||||
exchange_timestamp: at(1),
|
||||
price_level: 500_000,
|
||||
quantity: 2_000_000,
|
||||
side: Side::Buy,
|
||||
symbol: "1".to_string(),
|
||||
})
|
||||
.unwrap();
|
||||
writer.write_msg(&snapshot(at(2), 3_000_000)).unwrap();
|
||||
writer.flush().unwrap();
|
||||
let line = cause::line("1", at(if explained { 2 } else { 3 }), Cause::Loss, "");
|
||||
std::fs::write(&log, format!("12:00:00.000 polymarket: {line}\n")).unwrap();
|
||||
let report = reconcile(&output, Some(&log)).unwrap();
|
||||
std::fs::remove_file(output).unwrap();
|
||||
std::fs::remove_file(log).unwrap();
|
||||
report
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_mismatch_with_its_log_line_is_explained_and_one_without_is_not() {
|
||||
let report = reconcile_with(true);
|
||||
assert_eq!(report.mismatches.len(), 1);
|
||||
assert_eq!(report.explained, HashMap::from([(Cause::Loss, 1)]));
|
||||
assert!(report.unexplained.is_empty());
|
||||
let report = reconcile_with(false);
|
||||
assert_eq!(report.unexplained.len(), 1);
|
||||
assert!(report.explained.is_empty());
|
||||
}
|
||||
}
|
||||
Vendored
+9
-5
@@ -1,8 +1,8 @@
|
||||
# Test fixtures
|
||||
|
||||
A recording of Polymarket's market channel, replayed by the tests in
|
||||
`src/polymarket/feed.rs`, and Gamma pages read by `src/polymarket/gamma.rs`
|
||||
(no network in tests).
|
||||
Two recordings of Polymarket's market channel, replayed by the tests in
|
||||
`src/polymarket/feed.rs` (the first also by `socket.rs`'s routing test), and
|
||||
Gamma pages read by `src/polymarket/gamma.rs` (no network in tests).
|
||||
|
||||
- `gamma-markets.json`: Gamma's top 20 markets by 24 h volume on 2026-11-04,
|
||||
trimmed to the fields the collector reads. `conditionId`, `active`, `closed`
|
||||
@@ -15,5 +15,9 @@ A recording of Polymarket's market channel, replayed by the tests in
|
||||
minute boundaries) of the market channel for those markets' 40 tokens, one
|
||||
line per frame: `{"received_at_us": <receive time, µs>, "text": <frame>}`.
|
||||
Opening books, 3,850 changes, 28 mid-stream books, 14 trades, 8 `PONG`s.
|
||||
- `record.py`: the recorder, if the fixtures need redoing:
|
||||
`uvx --with websockets python -I record.py gamma-markets.json frames.jsonl 90`.
|
||||
- `polymarket-frames-two.jsonl`: 95 s (2026-11-11, 12:19:58 to 12:21:33 UTC)
|
||||
of the same tokens on two connections at once, in receive order, each line
|
||||
with its `connection` (0 or 1). 20 of the 40 tokens still had an order book.
|
||||
- `record.py`: the recorder, if the fixtures need redoing, on CONNECTIONS
|
||||
connections at once (default 1):
|
||||
`uvx --with websockets python -I record.py gamma-markets.json frames.jsonl 90 [CONNECTIONS]`.
|
||||
+5116
File diff suppressed because it is too large.
Load diff
Vendored
+9
-4
@@ -1,10 +1,13 @@
|
||||
# Records the market channel for the markets in gamma-markets.json: one JSON
|
||||
# line per frame, {"received_at_us": <micros>, "text": <frame text>}.
|
||||
# line per frame, in receive order,
|
||||
# {"received_at_us": <micros>, "connection": <number>, "text": <frame text>}.
|
||||
# CONNECTIONS (default 1) are opened at once, all on the same tokens.
|
||||
# uvx --with websockets python -I record.py gamma-markets.json frames.jsonl 90 [CONNECTIONS]
|
||||
import asyncio, json, sys, time
|
||||
import websockets
|
||||
markets = json.load(open(sys.argv[1]))
|
||||
tokens = [t for m in markets if m["enableOrderBook"] for t in json.loads(m["clobTokenIds"])]
|
||||
async def main(out, seconds):
|
||||
async def record(connection, out, seconds):
|
||||
async with websockets.connect("wss://ws-subscriptions-clob.polymarket.com/ws/market", max_size=None) as ws:
|
||||
await ws.send(json.dumps({"type": "market", "assets_ids": tokens}))
|
||||
loop = asyncio.get_event_loop(); end = loop.time() + seconds; last_ping = loop.time()
|
||||
@@ -14,6 +17,8 @@ async def main(out, seconds):
|
||||
try: text = await asyncio.wait_for(ws.recv(), 1)
|
||||
except asyncio.TimeoutError: continue
|
||||
received_at_us = time.time_ns() // 1000
|
||||
out.write(json.dumps({"received_at_us": received_at_us, "text": text}) + "\n")
|
||||
out.write(json.dumps({"received_at_us": received_at_us, "connection": connection, "text": text}) + "\n")
|
||||
async def main(out, seconds, connections):
|
||||
await asyncio.gather(*(record(connection, out, seconds) for connection in range(connections)))
|
||||
with open(sys.argv[2], "w") as out:
|
||||
asyncio.run(main(out, float(sys.argv[3])))
|
||||
asyncio.run(main(out, float(sys.argv[3]), int(sys.argv[4]) if len(sys.argv) > 4 else 1))
|
||||
Reference in new issue
Block a user