Every live market is subscribed on two connections chosen at random, and its book takes each change from whichever connection delivers it first: the book hash says whether a change is a duplicate, the next one, or one to hold until the change before it arrives. A change one connection misses costs nothing, each Delta carries the earliest receive time, and a connection that drops is covered by the other. When a change fits on no connection in time, the book follows one connection unchecked and asks the other for a fresh book. Connections now decode and route events to 16 book threads, independent of them; a move leaves its old connection only once the new one delivers, so it costs no change; a connection closed with 1013 twice within ten minutes is split. reconcile reads the collector's log and fails only on a mismatch no cause line explains. SIGTERM and SIGHUP stop the collector cleanly. Breaking: lanes are gone (connections and book threads are separate), and reconcile takes the log as a second argument.
25 KiB
04. Redundant connections: first one wins
Status: done (2026-11-11)
Every symbol is subscribed on two connections, and the first one to deliver a change wins: it is written with that receive time, and the other's copy is dropped. A change lost on one connection is then no loss, a slow connection costs no time accuracy, and a connection that drops is covered by the other. Connections stop belonging to book threads, so a symbol's two subscriptions have different neighbours, and a connection the server keeps closing is split. This is Extension 1 of the brief. Design: ../docs/project.md (pipeline), ADR 0003 (this design; supersedes the lanes of ADR 0002), ../docs/notes.md (protocol facts), ../GLOSSARY.md (first one wins, duplicate, verified, held, follow, rescued, split; loss, narrowed).
Measured before planning (2026-11-11)
-
Two connections on the same 8,000 symbols, 335 s: 616,708 changes, none on only one connection, the same order per symbol on both. The second delivery came 0.5 ms later at the median, more than 10 ms later for 6.5% of changes, more than 100 ms for 1%, 706 ms at worst, and up to 82 changes of its symbol behind. Neither connection led: one was first 34% of the time. 95% of updates were one change; the others came one change per frame, all with the hash of the whole batch, within 1 ms (40 ms at worst).
-
Twins are closed together: two connections ramped on the same symbols got their 1013 close 3 s apart, at the same load.
-
Three layouts of the same 48,038 newest symbols, 20 minutes, two subscriptions per symbol (connections of 5,000 symbols, reconnecting after 1 s):
Layout Closes Symbol-seconds with both down Symbols ever with both down Twins, symbols in creation order 49 33,345 5,000 Twins, symbols dealt randomly 15 3,787 5,000 Two independent random deals 20 272 290 In creation order every close hit the pair holding the day's 5- and 15-minute crypto markets (~2.5 MB/s). The test connections carried 450 to 900 KB/s, above our budget: the ratios count, not the totals.
-
300 connections opened at once from the server all stayed up. Its link carried 50 MB/s on one download; two subscriptions per symbol are an estimated 25 MB/s.
-
A prototype of the rules below on the two-connection recording (Python, real hash checks, 7,729 symbols): 340,424 changes written verified, 338,609 duplicates dropped, ~36,000 changes held then fit (batches), 100 chains broken, all with both connections holding the same change (trades). With 1% of one connection's changes removed, and with one connection delayed by 2 s, the output kept the single connection's order per symbol. The delay case first failed (the late changes were held, the chain declared broken, the slow connection followed): the stale rule and "the book moved on" rule below come from it.
-
Sprint 03's 75-minute run: 151,897 resyncs for 169 M verified changes; 11,605 of them opened a subscription gap. 7 of its 31 closes came within 0.2 s after a clock minute ending in 4, 9, 0 or 5: bursts a minute's load average doesn't show.
Pipeline
rescan fetcher ──findings──▶ ┌──────────────┐ ◀──findings── recent fetcher
│ orchestrator │ ◀── 1013 closes ── connections
└──────┬───▲───┘
Add / Move / Remove (a market and │ │ load reports, each minute
its connections), to its thread │ │ (bytes/s per market)
┌──────▼───┴───┐
connection 0..N ──events───▶ │ book threads │ ──Snapshot | Delta──▶ writer
(read, stamp, parse, route) │ 0..15 │
▲ └──────┬───────┘
└── subscribe / unsubscribe / resync ┘
- Connection (one thread each, ~175 at today's size): reads a frame, stamps its receive time, parses it, and sends each event to the book thread of its market, with its own number and the frame's bytes shared among its events. It tells every book thread when it comes up or goes down, and the orchestrator when the server closes it with 1013. It keeps its token set and subscribes it, in operations of 500, on every connect. Limits: 5,000 tokens and 300 KB/s, as sprint 03's lanes had.
- Book thread (16): owns the books of its markets, a market's thread being
derived from its
conditionId(16 hex digits of it, modulo 16), so every thread computes it without a shared table. It applies first one wins (below), writes Deltas and Snapshots, and is the one that asks connections to subscribe, unsubscribe and resync: it knows a symbol before its book can arrive. Each wall-clock minute it writes the minute Snapshots and reports its markets' loads. - Orchestrator: holds each market's two connections, places and moves subscriptions, splits a connection, and runs the fetchers as in sprint 03.
- Writer: unchanged.
First one wins
A symbol whose hash chain holds is verified. A change arriving from any of its connections is then, in this order:
- a duplicate, if it is one of the symbol's last 128 applied changes (same hash, side, price and size): dropped;
- stale, if its exchange timestamp is older than the last applied change's: dropped (while the chain holds, everything up to the book's state was applied);
- otherwise held on its connection. The shortest tail of that connection's held changes which, applied to the book, gives the hash of its last change fits: it is applied and written; held changes before the tail are stale and dropped; a change that leaves the book as it was is not written. One rule covers a batch completing, a change the other connection delivers after this one ran ahead, and late changes after a fresh book.
A held change is written with its own receive time, never earlier than the symbol's previous Delta, so replaying the output by time keeps the order.
The ceiling. A change may be held for 1 s; for 50 ms once every connection of the symbol that is up holds the same changes (nobody is left to wait for but the rest of a batch); at once when a connection holds more than 512 changes of the symbol. Only tails that start a batch are tried against the book: the whole held list and the last four. At the ceiling, if the book moved since the change arrived, its connection is the one behind and its held changes are dropped. If the book did not move, the chain is broken.
A broken chain. The book thread follows the connection whose held changes reached the ceiling: its held changes and its later ones are applied and written unchecked, the other connection's are dropped, and a resync goes to the other connection, so the followed stream is not interrupted (to the followed one when it is the only one up). The fresh book is compared as in sprint 02: equal to ours or a state we passed, ours stands; otherwise it replaces ours, a Snapshot is written and the difference is logged as a loss. Either way the symbol is verified again. A symbol whose complete book fails its own hash stays unverifiable, following one connection, as before.
Full books. Only books we asked for count: a subscribe's opening book while the symbol awaits one (the first complete book from any of its connections wins and is the symbol's first Snapshot), and a resync's answer. Any other book is ignored: the second connection's opening book, a reconnect's replay for verified symbols, the books the server sends after a trade.
Connections dropping. A verified symbol needs nothing when one of its connections drops: the other carries on, and the dropped one's held changes are forgotten. When the followed connection drops, the other is followed and asked for a fresh book, and a Snapshot is always written on it, as the brief asks for a change of source. When every connection of a symbol is down, the symbol awaits a book again: the first reconnect's opening book is taken as fresh, and one that differs from ours is written as a Snapshot and logged as a loss.
Placing, moving, splitting
- Startup: the first rescan completes (1 to 2.5 minutes on the server, Gamma's speed varying), then the markets are shuffled and dealt, each to two different connections, at about 5,000 markets a second so the opening books fit the link. Connections are opened for a fill of 4,500 tokens each.
- A new market goes on two different random connections with room in both limits; a connection opens when fewer than two have room.
- Load: a market's load is the bytes per second of its frames on one connection: its book thread counts every event's share from all of its connections and divides by their number. A connection's load is the sum of its markets'.
- Rebalancing, each minute: a connection over its budget has the busiest markets' subscriptions moved, each to a random connection with room that does not already carry the market: at most 50 moves a minute, a moved subscription staying 10 minutes, none moved for a market with no load.
- A move is
Move(market, from, to)to the market's book thread, which subscribes onto, and unsubscribes onfromoncetohas delivered a book of the market: until then both carry it, so no change falls between them. The book stays where it is: no Snapshot, no subscription gap, and the new connection's opening book is ignored like any unasked one. While a symbol is verified, a change is taken from any connection, the one it left included: its copies in flight are the exchange's changes like any other. - A split: a connection closed with 1013 twice within 10 minutes has every second market, in load order, moved off at once, outside the two caps, and is then left alone for 10 minutes.
- Removal: unchanged; the book thread unsubscribes both connections.
Data model
- Orchestrator:
conditionId→ (tokens, its two connections, load, when it last moved: both its subscriptions then stay); per connection, its token count, its load, its last 1013 closes and its last split. The market stays the unit: both tokens share the two connections. - Book thread, per symbol: the book and the hash inputs (sprint 02); its last 128 changes, each with its hash as 20 bytes (64 changes with the hash as text today: ~2.4 GB at worst instead of ~5 GB); the held changes of each of its connections; its state (awaiting a book, verified, following a connection, unverifiable and following one); its connections; its last Delta's receive time. Per market, the bytes since its last report. Per connection, whether it is up.
- Event (connection → book thread): receive time, connection number, bytes, and a parsed book, change, tick size change or trade; or a connection up / down marker.
- Cause lines: every Snapshot written because a book differed, and every
Snapshot written for a change of followed connection, logs one line with
its symbol, its exact receive time and its cause (loss, subscription gap,
switch). One module,
src/cause.rs, holds their format and their parser. - A tick size change breaks the chain on the connection that delivered it: the hashes change with the tick size, and the fresh book carries the new one.
Reconcile and the measures
reconcile OUTPUT [LOG]: with the log, each mismatch is paired with the cause line of its symbol and receive time; it prints the counts by cause and fails on an unexplained mismatch or an orphan Delta. Without, every mismatch is unexplained.- Each book thread's minute line: changes written (and verified), duplicates dropped, changes rescued (verified changes of a symbol on several connections that only one delivered, counted once 2 s old, when every copy has come), held changes dropped, chains broken, resyncs, books behind, subscription gaps, losses, switches, the second copies' lag (median and 99th percentile), the most events waiting.
- The orchestrator's: markets, connections, added, removed, moved, splits, the busiest connection's load.
Decisions
- First one wins, on one book per symbol (ADR 0003). The book hash orders changes across connections as a sequence number would. Rejected: a source and a standby with a book each and a Snapshot per switch (my first proposal): it loses the change the source missed, keeps the source's receive time, and doubles the books.
- A symbol's two connections are chosen independently (ADR 0003; supersedes ADR 0002's lanes, a connection and a book thread of its own, because twins were closed together: 14 times more time with both down than independent deals). Rejected: twin connections per lane.
- Markets are placed at random (supersedes "the least-loaded lane", and "subscribe page by page at startup", because both build the clusters a load average cannot see: random twins had a third of the closes of creation order). Startup scans first, then deals. Rejected: least-loaded placement (a batch of 1,000 markets created together lands on one connection); dealing while scanning (needs the connection count before the scan ends).
- The connection thread parses and routes. The stamp still comes first; a frame of several markets' books must be taken apart anyway; and the connection becomes the exchange-specific decoder Extension 3 wants. Rejected: forwarding the text after peeking at the market id.
- 16 book threads, a market's thread from its id. One handled ~30,000 frames a second in sprint 02. Rejected: a thread per connection (the lanes); a routing table (shared state).
- Subscriptions go through the market's book thread, which records the market before asking. Rejected: the orchestrator telling connections directly (a book can arrive before its thread knows the symbol).
- Ceilings of 1 s and 50 ms; at the ceiling, held changes are dropped if the book moved, else the chain is broken; a change older than the book is dropped at once. The lag reached 0.7 s, batches 40 ms. The last two come from the prototype: a connection 2 s late was otherwise taken for a broken chain and followed. Rejected: breaking at once (a change one connection missed would cost a resync instead of nothing); no ceiling (a change could wait unwritten for minutes).
- A connection holds at most 512 changes of a symbol, and only the tails starting a batch are tried, five at most (found by the alternating runs: at a 5-minute mark a crypto market's burst, behind one missing change, made every arrival try every tail of thousands, each a hash; one book thread then needed 9.5 minutes to drain after SIGINT). Rejected: trying every tail (quadratic); no cap (a burst piles up for its whole ceiling).
- Only asked-for books count. A reconnect replays up to 5,000 books and the server sends books after trades; reacting to them is what made resyncs. Rejected: comparing every book.
- The resync goes to the connection not followed, so the followed stream has no gap of its own. Rejected: resyncing the followed connection (sprint 03: 11,605 gaps).
- A split after two 1013 closes within 10 minutes. One close can be the server's moment (several connections within a second); a connection closed again is overloaded. Rejected: splitting on the first close; lowering every connection's budget.
- Threads, not tokio (ADR 0001's checkpoint, held here): the new logic is in the book thread, not in socket handling, and ~200 threads are comfortable. Rejected: an async socket layer now.
- Two subscriptions per market, a constant; the comparison run is the same code at 1. Rejected: a runtime option.
- Loss means a change no connection delivered in time; one a single connection missed is rescued. "Lane" is retired: connections and book threads are separate things now.
reconcilereads the log, through one module shared with the collector. Rejected: a second, machine-readable output.- Rescued counts copies, not held changes (supersedes "held, then fit after another connection moved the book", because a change one connection missed usually arrives on the other before the first's next change, and is then never held: the tests' rescues counted 0). Rejected: counting held changes that fit (most rescues go unseen).
- A move leaves the old connection once the new one delivers a book
(supersedes "subscribes on
toand then unsubscribes onfrom", because unsubscribing at once loses the changes betweenfrom's unsubscribe andto's first book). Rejected: a fixed delay before unsubscribing. - SIGTERM (and SIGHUP) stop the collector like SIGINT (
ctrlc'sterminationfeature):timeout,docker stopand a service manager send SIGTERM, and only a clean stop flushes the last chunk. - Built in two steps, with a live run between: first the connections separated from the book threads with one subscription per market, which must behave like sprint 03's collector; then the second subscription, first one wins and the split.
Tests
No network, except the local websocket server of sprint 03's reconnect test.
The fixture is re-recorded on two connections (record.py, extended) for the
same 40 tokens; replays feed both in receive order.
- Same output: two connections give the Deltas, in order per symbol, that one alone gives; the other's copies are counted as duplicates.
- A loss on one connection: a frame removed from one changes nothing in the output, and its changes count as rescued.
- A loss on both: the same frame removed from both leaves the next change held; at the ceiling the chain breaks, the resync goes to the connection not followed, and a differing fresh book is written as a Snapshot, logged a loss.
- A slow connection: one delayed by 2 s gives the same output, each Delta with the earlier receive time, and no chain broken by the delay.
- Stamps: a held change is not stamped earlier than the symbol's previous Delta.
- A batch is written only once it fits; a fresh book's late changes are dropped as stale.
- Books: an unasked book changes nothing; the first of two opening books is the Snapshot.
- Connections dropping: one down, the output goes on; the followed one down, a Snapshot on the fresh book; both down, every symbol awaits a book.
- Routing: a frame of several markets' books reaches each market's book thread; a market's thread is the same from the orchestrator and from a connection.
- Placement: a market's two connections differ; a move never targets the market's other connection; a connection opens when fewer than two have room; the startup deal is even.
- Move: subscribe on the new connection, then unsubscribe on the old; no Snapshot.
- Split: two closes within 10 minutes move every second market; one close moves none; a second split waits 10 minutes.
- Load: a market on two connections reports the load of one.
- Reconcile with a log: explained mismatches pass; an unexplained one fails.
- Sprint 03's tests of Gamma, the minute check, rescan removal, rebalancing's caps and the reconnect stay, adapted to connections.
Not in this sprint
- Acting on delay: the second delivery's lag is measured and logged; choosing connections by it is M7's.
- The crash chunk: a hard crash still loses up to 64 MiB, ~10 s of output.
- Balancing on a market's peak second rather than its minute average.
- Reading the last trade price from the trade event to verify again without a resync (the prototype did it for 15 of 100 broken chains): a Possible feature.
- Detecting a subscription gone quiet on one connection and resubscribing it, and recycling connections: M5. First one wins hides a quiet subscription; M5 has to find it.
- A market's
min_order_sizeorneg_riskchanging: still read once.
Done when
scripts/ci.shis green.- Step 1 (one subscription per market): a 10-minute live run has every
reconcilemismatch explained by the log. - The comparison, started when the user says so: two 20-minute live runs
back to back, one subscription per market, then two. In the second:
- every
reconcilemismatch is explained by the log; - losses and subscription gaps per million changes are below the first run's;
- rescued changes, duplicates, chains broken and the second delivery's lag are reported;
- a split was seen, or its absence explained;
- peak memory and thread count are reported.
- every
- Free disk is checked before each run, and the outputs deleted after.
- ADR 0003 is written, ADR 0001 records the checkpoint, ADR 0002 says what
was superseded;
GLOSSARY.md,docs/project.md's pipeline,docs/notes.md(the measures above) and the roadmap match.
Outcome
Every live market (~198k, ~396k symbols) is on two connections chosen at random (~178 connections), and 16 book threads take each change from whichever connection delivers it first. Built in the two planned steps; step 1 (connections separated, one subscription) reconciled fully in a 10-minute live run, no connection lost.
The comparison was run as four alternating 10-minute rounds, one subscription per market then two, twice (asked by the user once the server proved overloaded during a first back-to-back pair). Per million changes written:
| Round | Subscriptions | Losses | Subscription gaps | Losses + gaps | Chains broken |
|---|---|---|---|---|---|
| 1 | 1 | 69.1 | 163.7 | 232.8 | 525 |
| 2 | 2 | 49.7 | 160.2 | 209.9 | 566 |
| 3 | 1 | 42.1 | 142.0 | 184.1 | 530 |
| 4 | 2 | 32.3 | 152.1 | 184.5 | 503 |
Every round reconciled with each mismatch explained by its log line. Losses fell by a quarter in each pair; losses and gaps together fell in the first pair and tied in the second, the rates drifting down through the hour. The user accepted this as the sprint's result. Subscription gaps, mostly from opening subscribes that both connections share, are untouched by a second connection. Most "losses" are likely not data missed: chains break as often with one connection as with two (about the rate of trades), and a resync's book that is a few changes ahead of our stream is logged a loss too. Duplicates: one per change written. The second copy came 1 to 2 ms after the first at the median, 45 to 120 ms at p99. Peak memory 1.78 GB with two subscriptions, 1.67 GB with one; 200 threads. No split: no connection was closed twice within 10 minutes (0 and 2 connections lost in the two-subscription rounds). The first, overloaded pair (load up to 92 on 12 cores) still reconciled fully: a stress test passed.
Rescued (~110,000 per round) is an upper bound, accepted as such: a change that leaves the last 128 remembered before its second copy comes, on a busy symbol, is counted as delivered once; so is a batch whose changes the two connections delivered interleaved (the held copy dropped uncounted).
Fixed at the close, from the code review: a move ending while a fresh book was asked from the connection left kept asking it in vain (now asked from another); a change held across a minute could be stamped before that minute's Snapshot (a Snapshot now advances the symbol's last stamp); markets closed during the first scan were dealt anyway. The first two have a test that fails without the fix; the third sits in the orchestrator's loop, as untested as the rest of it.
Diverged from the plan, each in Decisions: markets on independent connections, not twins (the user's design, after the three-layout test); rescued counted from copies; a move leaves the old connection once the new one delivers a book; the held-change cap and batch-start tails (found by the runs: a crypto burst behind one missing change made a book thread fall 455,000 events behind and take 9.5 minutes to stop); a tick size change breaks the chain; cause lines carry the exact receive time.
Left out, and where it went: telling a real loss from a resync book ahead of our stream, and the opening subscription gaps (Possible features); an exact rescued count (Possible features); acting on delay (M7); the crash chunk (M4's ceiling, kept); quiet subscriptions hidden by first one wins (M5).
Retrospective
- Went well: experiments settled the design before code: two recordings
showed the hash orders changes across connections, the three-layout test
turned a twin design into the user's independent one, a Python prototype
found the slow-connection flaw before any Rust.
reconcilereading the log made every live run a correctness check, and it held even under load. - Went wrong: a quadratic loop marked "fine at a few dozen" met bursts of
thousands; it surfaced only as a book thread falling behind, then as a
shutdown that took 9.5 minutes. The first comparison pair ran on an
overloaded server and wasn't comparable. Wait loops on
pgrep -ffailed twice more, once killing the wrong process. - Tested enough? The rules are, on a two-connection recording with injected losses, delays and drops, and a burst test that hung before the fix. Not covered: the orchestrator's loop (pacing the deal, routing commands, splits sent), the socket's up/down markers, a move whose new connection never delivers, and the minute line's lag figures.
- Next time: load-test the book thread with a synthetic burst before a live run, and check the server's load before a comparison; keep the ceiling of every corner cut in a test, not just a comment.