feat(polymarket)!: two connections per market, first one wins

Every live market is subscribed on two connections chosen at random, and its
book takes each change from whichever connection delivers it first: the book
hash says whether a change is a duplicate, the next one, or one to hold until
the change before it arrives. A change one connection misses costs nothing,
each Delta carries the earliest receive time, and a connection that drops is
covered by the other. When a change fits on no connection in time, the book
follows one connection unchecked and asks the other for a fresh book.
Connections now decode and route events to 16 book threads, independent of
them; a move leaves its old connection only once the new one delivers, so it
costs no change; a connection closed with 1013 twice within ten minutes is
split. reconcile reads the collector's log and fails only on a mismatch no
cause line explains. SIGTERM and SIGHUP stop the collector cleanly.

Breaking: lanes are gone (connections and book threads are separate), and
reconcile takes the log as a second argument.
This commit is contained in:
pascal committed 2026-11-11 15:18:48 +00:00
1 parent 8114a51dc2
commit 02747b882f
24 files changed
+8984 -1214

No files matched your search

+443
View File
@@ -0,0 +1,443 @@
# 04. Redundant connections: first one wins
**Status: done (2026-11-11)**
Every symbol is subscribed on two connections, and the first one to deliver a
change wins: it is written with that receive time, and the other's copy is
dropped. A change lost on one connection is then no loss, a slow connection
costs no time accuracy, and a connection that drops is covered by the other.
Connections stop belonging to book threads, so a symbol's two subscriptions
have different neighbours, and a connection the server keeps closing is split.
This is Extension 1 of the brief. Design:
[../docs/project.md](../docs/project.md) (pipeline),
[ADR 0003](../docs/adr/0003-first-one-wins.md) (this design; supersedes the
lanes of ADR 0002), [../docs/notes.md](../docs/notes.md) (protocol facts),
[../GLOSSARY.md](../GLOSSARY.md) (first one wins, duplicate, verified, held,
follow, rescued, split; loss, narrowed).
## Measured before planning (2026-11-11)
- **Two connections on the same 8,000 symbols, 335 s**: 616,708 changes, none
on only one connection, the same order per symbol on both. The second
delivery came 0.5 ms later at the median, more than 10 ms later for 6.5% of
changes, more than 100 ms for 1%, 706 ms at worst, and up to 82 changes of
its symbol behind. Neither connection led: one was first 34% of the time.
95% of updates were one change; the others came one change per frame, all
with the hash of the whole batch, within 1 ms (40 ms at worst).
- **Twins are closed together**: two connections ramped on the same symbols
got their 1013 close 3 s apart, at the same load.
- **Three layouts of the same 48,038 newest symbols, 20 minutes, two
subscriptions per symbol** (connections of 5,000 symbols, reconnecting after
1 s):
| Layout | Closes | Symbol-seconds with both down | Symbols ever with both down |
|---|---|---|---|
| Twins, symbols in creation order | 49 | 33,345 | 5,000 |
| Twins, symbols dealt randomly | 15 | 3,787 | 5,000 |
| Two independent random deals | 20 | 272 | 290 |
In creation order every close hit the pair holding the day's 5- and
15-minute crypto markets (~2.5 MB/s). The test connections carried 450 to
900 KB/s, above our budget: the ratios count, not the totals.
- **300 connections** opened at once from the server all stayed up. Its link
carried 50 MB/s on one download; two subscriptions per symbol are an
estimated 25 MB/s.
- **A prototype of the rules below on the two-connection recording** (Python,
real hash checks, 7,729 symbols): 340,424 changes written verified, 338,609
duplicates dropped, ~36,000 changes held then fit (batches), 100 chains
broken, all with both connections holding the same change (trades). With
1% of one connection's changes removed, and with one connection delayed by
2 s, the output kept the single connection's order per symbol. The delay
case first failed (the late changes were held, the chain declared broken,
the slow connection followed): the stale rule and "the book moved on" rule
below come from it.
- **Sprint 03's 75-minute run**: 151,897 resyncs for 169 M verified changes;
11,605 of them opened a subscription gap. 7 of its 31 closes came within
0.2 s after a clock minute ending in 4, 9, 0 or 5: bursts a minute's load
average doesn't show.
## Pipeline
```
rescan fetcher ──findings──▶ ┌──────────────┐ ◀──findings── recent fetcher
│ orchestrator │ ◀── 1013 closes ── connections
└──────┬───▲───┘
Add / Move / Remove (a market and │ │ load reports, each minute
its connections), to its thread │ │ (bytes/s per market)
┌──────▼───┴───┐
connection 0..N ──events───▶ │ book threads │ ──Snapshot | Delta──▶ writer
(read, stamp, parse, route) │ 0..15 │
▲ └──────┬───────┘
└── subscribe / unsubscribe / resync ┘
```
- **Connection** (one thread each, ~175 at today's size): reads a frame,
stamps its receive time, parses it, and sends each event to the book thread
of its market, with its own number and the frame's bytes shared among its
events. It tells every book thread when it comes up or goes down, and the
orchestrator when the server closes it with 1013. It keeps its token set and
subscribes it, in operations of 500, on every connect. Limits: 5,000 tokens
and 300 KB/s, as sprint 03's lanes had.
- **Book thread** (16): owns the books of its markets, a market's thread being
derived from its `conditionId` (16 hex digits of it, modulo 16), so every
thread computes it without a shared table. It applies first one wins
(below), writes Deltas and Snapshots, and is the one that asks connections
to subscribe, unsubscribe and resync: it knows a symbol before its book can
arrive. Each wall-clock minute it writes the minute Snapshots and reports
its markets' loads.
- **Orchestrator**: holds each market's two connections, places and moves
subscriptions, splits a connection, and runs the fetchers as in sprint 03.
- **Writer**: unchanged.
## First one wins
A symbol whose hash chain holds is **verified**. A change arriving from any of
its connections is then, in this order:
1. a **duplicate**, if it is one of the symbol's last 128 applied changes
(same hash, side, price and size): dropped;
2. **stale**, if its exchange timestamp is older than the last applied
change's: dropped (while the chain holds, everything up to the book's state
was applied);
3. otherwise **held** on its connection. The shortest tail of that
connection's held changes which, applied to the book, gives the hash of its
last change **fits**: it is applied and written; held changes before the
tail are stale and dropped; a change that leaves the book as it was is not
written. One rule covers a batch completing, a change the other connection
delivers after this one ran ahead, and late changes after a fresh book.
A held change is written with its own receive time, never earlier than the
symbol's previous Delta, so replaying the output by time keeps the order.
**The ceiling.** A change may be held for 1 s; for 50 ms once every connection
of the symbol that is up holds the same changes (nobody is left to wait for
but the rest of a batch); at once when a connection holds more than 512
changes of the symbol. Only tails that start a batch are tried against the
book: the whole held list and the last four. At the ceiling, if the book moved since the change
arrived, its connection is the one behind and its held changes are dropped.
If the book did not move, the chain is **broken**.
**A broken chain.** The book thread **follows** the connection whose held
changes reached the ceiling: its held changes and its later ones are applied
and written unchecked, the other connection's are dropped, and a resync goes
to the other connection, so the followed stream is not interrupted (to the
followed one when it is the only one up). The fresh book is compared as in
sprint 02: equal to ours or a state we passed, ours stands; otherwise it
replaces ours, a Snapshot is written and the difference is logged as a loss.
Either way the symbol is verified again. A symbol whose complete book fails
its own hash stays unverifiable, following one connection, as before.
**Full books.** Only books we asked for count: a subscribe's opening book
while the symbol awaits one (the first complete book from any of its
connections wins and is the symbol's first Snapshot), and a resync's answer.
Any other book is ignored: the second connection's opening book, a
reconnect's replay for verified symbols, the books the server sends after a
trade.
**Connections dropping.** A verified symbol needs nothing when one of its
connections drops: the other carries on, and the dropped one's held changes
are forgotten. When the followed connection drops, the other is followed and
asked for a fresh book, and a Snapshot is always written on it, as the brief
asks for a change of source. When every connection of a symbol is down, the
symbol awaits a book again: the first reconnect's opening book is taken as
fresh, and one that differs from ours is written as a Snapshot and logged as
a loss.
## Placing, moving, splitting
- **Startup**: the first rescan completes (1 to 2.5 minutes on the server,
Gamma's speed varying), then the markets are shuffled and dealt, each to two different
connections, at about 5,000 markets a second so the opening books fit the
link. Connections are opened for a fill of 4,500 tokens each.
- **A new market** goes on two different random connections with room in both
limits; a connection opens when fewer than two have room.
- **Load**: a market's load is the bytes per second of its frames on one
connection: its book thread counts every event's share from all of its
connections and divides by their number. A connection's load is the sum of
its markets'.
- **Rebalancing**, each minute: a connection over its budget has the busiest
markets' subscriptions moved, each to a random connection with room that
does not already carry the market: at most 50 moves a minute, a moved
subscription staying 10 minutes, none moved for a market with no load.
- **A move** is `Move(market, from, to)` to the market's book thread, which
subscribes on `to`, and unsubscribes on `from` once `to` has delivered a
book of the market: until then both carry it, so no change falls between
them. The book stays where it is: no Snapshot, no subscription gap, and the
new connection's opening book is ignored like any unasked one. While a
symbol is verified, a change is taken from any connection, the one it left
included: its copies in flight are the exchange's changes like any other.
- **A split**: a connection closed with 1013 twice within 10 minutes has every
second market, in load order, moved off at once, outside the two caps, and
is then left alone for 10 minutes.
- **Removal**: unchanged; the book thread unsubscribes both connections.
## Data model
- **Orchestrator**: `conditionId` → (tokens, its two connections, load, when
it last moved: both its subscriptions then stay); per connection, its token count, its load,
its last 1013 closes and its last split. The market stays the unit: both
tokens share the two connections.
- **Book thread**, per symbol: the book and the hash inputs (sprint 02); its
last 128 changes, each with its hash as 20 bytes (64 changes with the hash
as text today: ~2.4 GB at worst instead of ~5 GB); the held changes of each
of its connections; its state (awaiting a book, verified, following a
connection, unverifiable and following one); its connections; its last
Delta's receive time. Per market, the bytes since its last report. Per
connection, whether it is up.
- **Event** (connection → book thread): receive time, connection number,
bytes, and a parsed book, change, tick size change or trade; or a
connection up / down marker.
- **Cause lines**: every Snapshot written because a book differed, and every
Snapshot written for a change of followed connection, logs one line with
its symbol, its exact receive time and its cause (loss, subscription gap,
switch). One module, `src/cause.rs`, holds their format and their parser.
- **A tick size change** breaks the chain on the connection that delivered
it: the hashes change with the tick size, and the fresh book carries the
new one.
## Reconcile and the measures
- `reconcile OUTPUT [LOG]`: with the log, each mismatch is paired with the
cause line of its symbol and receive time; it prints the counts by cause
and fails on an unexplained mismatch or an orphan Delta. Without, every
mismatch is unexplained.
- Each book thread's minute line: changes written (and verified), duplicates
dropped, changes **rescued** (verified changes of a symbol on several
connections that only one delivered, counted once 2 s old, when every copy
has come), held changes dropped, chains broken, resyncs, books behind,
subscription gaps, losses, switches, the second copies' lag (median and
99th percentile), the most events waiting.
- The orchestrator's: markets, connections, added, removed, moved, splits, the
busiest connection's load.
## Decisions
- **First one wins, on one book per symbol** (ADR 0003). The book hash orders
changes across connections as a sequence number would. *Rejected:* a source
and a standby with a book each and a Snapshot per switch (my first
proposal): it loses the change the source missed, keeps the source's
receive time, and doubles the books.
- **A symbol's two connections are chosen independently** (ADR 0003;
supersedes ADR 0002's lanes, a connection and a book thread of its own,
because twins were closed together: 14 times more time with both down than
independent deals). *Rejected:* twin connections per lane.
- **Markets are placed at random** (supersedes "the least-loaded lane", and
"subscribe page by page at startup", because both build the clusters a load
average cannot see: random twins had a third of the closes of creation
order). Startup scans first, then deals. *Rejected:* least-loaded placement
(a batch of 1,000 markets created together lands on one connection);
dealing while scanning (needs the connection count before the scan ends).
- **The connection thread parses and routes.** The stamp still comes first; a
frame of several markets' books must be taken apart anyway; and the
connection becomes the exchange-specific decoder Extension 3 wants.
*Rejected:* forwarding the text after peeking at the market id.
- **16 book threads, a market's thread from its id.** One handled ~30,000
frames a second in sprint 02. *Rejected:* a thread per connection (the
lanes); a routing table (shared state).
- **Subscriptions go through the market's book thread**, which records the
market before asking. *Rejected:* the orchestrator telling connections
directly (a book can arrive before its thread knows the symbol).
- **Ceilings of 1 s and 50 ms; at the ceiling, held changes are dropped if the
book moved, else the chain is broken; a change older than the book is
dropped at once.** The lag reached 0.7 s, batches 40 ms. The last two come
from the prototype: a connection 2 s late was otherwise taken for a broken
chain and followed. *Rejected:* breaking at once (a change one connection
missed would cost a resync instead of nothing); no ceiling (a change could
wait unwritten for minutes).
- **A connection holds at most 512 changes of a symbol, and only the tails
starting a batch are tried, five at most** (found by the alternating runs:
at a 5-minute mark a crypto market's burst, behind one missing change, made
every arrival try every tail of thousands, each a hash; one book thread
then needed 9.5 minutes to drain after SIGINT). *Rejected:* trying every
tail (quadratic); no cap (a burst piles up for its whole ceiling).
- **Only asked-for books count.** A reconnect replays up to 5,000 books and
the server sends books after trades; reacting to them is what made
resyncs. *Rejected:* comparing every book.
- **The resync goes to the connection not followed**, so the followed stream
has no gap of its own. *Rejected:* resyncing the followed connection
(sprint 03: 11,605 gaps).
- **A split after two 1013 closes within 10 minutes.** One close can be the
server's moment (several connections within a second); a connection closed
again is overloaded. *Rejected:* splitting on the first close; lowering
every connection's budget.
- **Threads, not tokio** (ADR 0001's checkpoint, held here): the new logic is
in the book thread, not in socket handling, and ~200 threads are
comfortable. *Rejected:* an async socket layer now.
- **Two subscriptions per market, a constant**; the comparison run is the same
code at 1. *Rejected:* a runtime option.
- **Loss means a change no connection delivered in time**; one a single
connection missed is rescued. "Lane" is retired: connections and book
threads are separate things now.
- **`reconcile` reads the log, through one module shared with the
collector.** *Rejected:* a second, machine-readable output.
- **Rescued counts copies, not held changes** (supersedes "held, then fit
after another connection moved the book", because a change one connection
missed usually arrives on the other before the first's next change, and is
then never held: the tests' rescues counted 0). *Rejected:* counting held
changes that fit (most rescues go unseen).
- **A move leaves the old connection once the new one delivers a book**
(supersedes "subscribes on `to` and then unsubscribes on `from`", because
unsubscribing at once loses the changes between `from`'s unsubscribe and
`to`'s first book). *Rejected:* a fixed delay before unsubscribing.
- **SIGTERM (and SIGHUP) stop the collector like SIGINT** (`ctrlc`'s
`termination` feature): `timeout`, `docker stop` and a service manager send
SIGTERM, and only a clean stop flushes the last chunk.
- **Built in two steps, with a live run between**: first the connections
separated from the book threads with one subscription per market, which
must behave like sprint 03's collector; then the second subscription, first
one wins and the split.
## Tests
No network, except the local websocket server of sprint 03's reconnect test.
The fixture is re-recorded on two connections (`record.py`, extended) for the
same 40 tokens; replays feed both in receive order.
- **Same output**: two connections give the Deltas, in order per symbol, that
one alone gives; the other's copies are counted as duplicates.
- **A loss on one connection**: a frame removed from one changes nothing in
the output, and its changes count as rescued.
- **A loss on both**: the same frame removed from both leaves the next change
held; at the ceiling the chain breaks, the resync goes to the connection not
followed, and a differing fresh book is written as a Snapshot, logged a loss.
- **A slow connection**: one delayed by 2 s gives the same output, each Delta
with the earlier receive time, and no chain broken by the delay.
- **Stamps**: a held change is not stamped earlier than the symbol's previous
Delta.
- **A batch** is written only once it fits; a fresh book's late changes are
dropped as stale.
- **Books**: an unasked book changes nothing; the first of two opening books
is the Snapshot.
- **Connections dropping**: one down, the output goes on; the followed one
down, a Snapshot on the fresh book; both down, every symbol awaits a book.
- **Routing**: a frame of several markets' books reaches each market's book
thread; a market's thread is the same from the orchestrator and from a
connection.
- **Placement**: a market's two connections differ; a move never targets the
market's other connection; a connection opens when fewer than two have
room; the startup deal is even.
- **Move**: subscribe on the new connection, then unsubscribe on the old; no
Snapshot.
- **Split**: two closes within 10 minutes move every second market; one close
moves none; a second split waits 10 minutes.
- **Load**: a market on two connections reports the load of one.
- **Reconcile with a log**: explained mismatches pass; an unexplained one
fails.
- Sprint 03's tests of Gamma, the minute check, rescan removal, rebalancing's
caps and the reconnect stay, adapted to connections.
## Not in this sprint
- Acting on delay: the second delivery's lag is measured and logged; choosing
connections by it is M7's.
- The crash chunk: a hard crash still loses up to 64 MiB, ~10 s of output.
- Balancing on a market's peak second rather than its minute average.
- Reading the last trade price from the trade event to verify again without
a resync (the prototype did it for 15 of 100 broken chains): a Possible
feature.
- Detecting a subscription gone quiet on one connection and resubscribing it,
and recycling connections: M5. First one wins hides a quiet subscription;
M5 has to find it.
- A market's `min_order_size` or `neg_risk` changing: still read once.
## Done when
- `scripts/ci.sh` is green.
- **Step 1** (one subscription per market): a 10-minute live run has every
`reconcile` mismatch explained by the log.
- **The comparison**, started when the user says so: two 20-minute live runs
back to back, one subscription per market, then two. In the second:
- every `reconcile` mismatch is explained by the log;
- losses and subscription gaps per million changes are below the first
run's;
- rescued changes, duplicates, chains broken and the second delivery's lag
are reported;
- a split was seen, or its absence explained;
- peak memory and thread count are reported.
- Free disk is checked before each run, and the outputs deleted after.
- ADR 0003 is written, ADR 0001 records the checkpoint, ADR 0002 says what
was superseded; `GLOSSARY.md`, `docs/project.md`'s pipeline, `docs/notes.md`
(the measures above) and the roadmap match.
## Outcome
Every live market (~198k, ~396k symbols) is on two connections chosen at
random (~178 connections), and 16 book threads take each change from
whichever connection delivers it first. Built in the two planned steps;
step 1 (connections separated, one subscription) reconciled fully in a
10-minute live run, no connection lost.
**The comparison** was run as four alternating 10-minute rounds, one
subscription per market then two, twice (asked by the user once the server
proved overloaded during a first back-to-back pair). Per million changes
written:
| Round | Subscriptions | Losses | Subscription gaps | Losses + gaps | Chains broken |
|---|---|---|---|---|---|
| 1 | 1 | 69.1 | 163.7 | 232.8 | 525 |
| 2 | 2 | 49.7 | 160.2 | 209.9 | 566 |
| 3 | 1 | 42.1 | 142.0 | 184.1 | 530 |
| 4 | 2 | 32.3 | 152.1 | 184.5 | 503 |
Every round reconciled with each mismatch explained by its log line. Losses
fell by a quarter in each pair; losses and gaps together fell in the first
pair and tied in the second, the rates drifting down through the hour. The
user accepted this as the sprint's result. Subscription gaps, mostly from
opening subscribes that both connections share, are untouched by a second
connection. Most "losses" are likely not data missed: chains break as often
with one connection as with two (about the rate of trades), and a resync's
book that is a few changes ahead of our stream is logged a loss too.
Duplicates: one per change written. The second copy came 1 to 2 ms after
the first at the median, 45 to 120 ms at p99. Peak memory 1.78 GB with two
subscriptions, 1.67 GB with one; 200 threads. No split: no connection was
closed twice within 10 minutes (0 and 2 connections lost in the
two-subscription rounds). The first, overloaded pair (load up to 92 on 12
cores) still reconciled fully: a stress test passed.
**Rescued** (~110,000 per round) is an upper bound, accepted as such: a
change that leaves the last 128 remembered before its second copy comes, on
a busy symbol, is counted as delivered once; so is a batch whose changes the
two connections delivered interleaved (the held copy dropped uncounted).
**Fixed at the close**, from the code review: a move ending while a fresh
book was asked from the connection left kept asking it in vain (now asked
from another); a change held across a minute could be stamped before that
minute's Snapshot (a Snapshot now advances the symbol's last stamp); markets
closed during the first scan were dealt anyway. The first two have a test
that fails without the fix; the third sits in the orchestrator's loop, as
untested as the rest of it.
**Diverged from the plan**, each in Decisions: markets on independent
connections, not twins (the user's design, after the three-layout test);
rescued counted from copies; a move leaves the old connection once the new
one delivers a book; the held-change cap and batch-start tails (found by the
runs: a crypto burst behind one missing change made a book thread fall
455,000 events behind and take 9.5 minutes to stop); a tick size change
breaks the chain; cause lines carry the exact receive time.
**Left out**, and where it went: telling a real loss from a resync book
ahead of our stream, and the opening subscription gaps (Possible features);
an exact rescued count (Possible features); acting on delay (M7); the crash
chunk (M4's ceiling, kept); quiet subscriptions hidden by first one wins
(M5).
### Retrospective
- **Went well:** experiments settled the design before code: two recordings
showed the hash orders changes across connections, the three-layout test
turned a twin design into the user's independent one, a Python prototype
found the slow-connection flaw before any Rust. `reconcile` reading the log
made every live run a correctness check, and it held even under load.
- **Went wrong:** a quadratic loop marked "fine at a few dozen" met bursts of
thousands; it surfaced only as a book thread falling behind, then as a
shutdown that took 9.5 minutes. The first comparison pair ran on an
overloaded server and wasn't comparable. Wait loops on `pgrep -f` failed
twice more, once killing the wrong process.
- **Tested enough?** The rules are, on a two-connection recording with
injected losses, delays and drops, and a burst test that hung before the
fix. Not covered: the orchestrator's loop (pacing the deal, routing
commands, splits sent), the socket's up/down markers, a move whose new
connection never delivers, and the minute line's lag figures.
- **Next time:** load-test the book thread with a synthetic burst before a
live run, and check the server's load before a comparison; keep the
ceiling of every corner cut in a test, not just a comment.