Archived
Every live market is subscribed on two connections chosen at random, and its book takes each change from whichever connection delivers it first: the book hash says whether a change is a duplicate, the next one, or one to hold until the change before it arrives. A change one connection misses costs nothing, each Delta carries the earliest receive time, and a connection that drops is covered by the other. When a change fits on no connection in time, the book follows one connection unchecked and asks the other for a fresh book. Connections now decode and route events to 16 book threads, independent of them; a move leaves its old connection only once the new one delivers, so it costs no change; a connection closed with 1013 twice within ten minutes is split. reconcile reads the collector's log and fails only on a mismatch no cause line explains. SIGTERM and SIGHUP stop the collector cleanly. Breaking: lanes are gone (connections and book threads are separate), and reconcile takes the log as a second argument.
444 lines
25 KiB
Markdown
444 lines
25 KiB
Markdown
# 04. Redundant connections: first one wins
|
|
|
|
**Status: done (2026-11-11)**
|
|
|
|
Every symbol is subscribed on two connections, and the first one to deliver a
|
|
change wins: it is written with that receive time, and the other's copy is
|
|
dropped. A change lost on one connection is then no loss, a slow connection
|
|
costs no time accuracy, and a connection that drops is covered by the other.
|
|
Connections stop belonging to book threads, so a symbol's two subscriptions
|
|
have different neighbours, and a connection the server keeps closing is split.
|
|
This is Extension 1 of the brief. Design:
|
|
[../docs/project.md](../docs/project.md) (pipeline),
|
|
[ADR 0003](../docs/adr/0003-first-one-wins.md) (this design; supersedes the
|
|
lanes of ADR 0002), [../docs/notes.md](../docs/notes.md) (protocol facts),
|
|
[../GLOSSARY.md](../GLOSSARY.md) (first one wins, duplicate, verified, held,
|
|
follow, rescued, split; loss, narrowed).
|
|
|
|
## Measured before planning (2026-11-11)
|
|
|
|
- **Two connections on the same 8,000 symbols, 335 s**: 616,708 changes, none
|
|
on only one connection, the same order per symbol on both. The second
|
|
delivery came 0.5 ms later at the median, more than 10 ms later for 6.5% of
|
|
changes, more than 100 ms for 1%, 706 ms at worst, and up to 82 changes of
|
|
its symbol behind. Neither connection led: one was first 34% of the time.
|
|
95% of updates were one change; the others came one change per frame, all
|
|
with the hash of the whole batch, within 1 ms (40 ms at worst).
|
|
- **Twins are closed together**: two connections ramped on the same symbols
|
|
got their 1013 close 3 s apart, at the same load.
|
|
- **Three layouts of the same 48,038 newest symbols, 20 minutes, two
|
|
subscriptions per symbol** (connections of 5,000 symbols, reconnecting after
|
|
1 s):
|
|
|
|
| Layout | Closes | Symbol-seconds with both down | Symbols ever with both down |
|
|
|---|---|---|---|
|
|
| Twins, symbols in creation order | 49 | 33,345 | 5,000 |
|
|
| Twins, symbols dealt randomly | 15 | 3,787 | 5,000 |
|
|
| Two independent random deals | 20 | 272 | 290 |
|
|
|
|
In creation order every close hit the pair holding the day's 5- and
|
|
15-minute crypto markets (~2.5 MB/s). The test connections carried 450 to
|
|
900 KB/s, above our budget: the ratios count, not the totals.
|
|
- **300 connections** opened at once from the server all stayed up. Its link
|
|
carried 50 MB/s on one download; two subscriptions per symbol are an
|
|
estimated 25 MB/s.
|
|
- **A prototype of the rules below on the two-connection recording** (Python,
|
|
real hash checks, 7,729 symbols): 340,424 changes written verified, 338,609
|
|
duplicates dropped, ~36,000 changes held then fit (batches), 100 chains
|
|
broken, all with both connections holding the same change (trades). With
|
|
1% of one connection's changes removed, and with one connection delayed by
|
|
2 s, the output kept the single connection's order per symbol. The delay
|
|
case first failed (the late changes were held, the chain declared broken,
|
|
the slow connection followed): the stale rule and "the book moved on" rule
|
|
below come from it.
|
|
- **Sprint 03's 75-minute run**: 151,897 resyncs for 169 M verified changes;
|
|
11,605 of them opened a subscription gap. 7 of its 31 closes came within
|
|
0.2 s after a clock minute ending in 4, 9, 0 or 5: bursts a minute's load
|
|
average doesn't show.
|
|
|
|
## Pipeline
|
|
|
|
```
|
|
rescan fetcher ──findings──▶ ┌──────────────┐ ◀──findings── recent fetcher
|
|
│ orchestrator │ ◀── 1013 closes ── connections
|
|
└──────┬───▲───┘
|
|
Add / Move / Remove (a market and │ │ load reports, each minute
|
|
its connections), to its thread │ │ (bytes/s per market)
|
|
┌──────▼───┴───┐
|
|
connection 0..N ──events───▶ │ book threads │ ──Snapshot | Delta──▶ writer
|
|
(read, stamp, parse, route) │ 0..15 │
|
|
▲ └──────┬───────┘
|
|
└── subscribe / unsubscribe / resync ┘
|
|
```
|
|
|
|
- **Connection** (one thread each, ~175 at today's size): reads a frame,
|
|
stamps its receive time, parses it, and sends each event to the book thread
|
|
of its market, with its own number and the frame's bytes shared among its
|
|
events. It tells every book thread when it comes up or goes down, and the
|
|
orchestrator when the server closes it with 1013. It keeps its token set and
|
|
subscribes it, in operations of 500, on every connect. Limits: 5,000 tokens
|
|
and 300 KB/s, as sprint 03's lanes had.
|
|
- **Book thread** (16): owns the books of its markets, a market's thread being
|
|
derived from its `conditionId` (16 hex digits of it, modulo 16), so every
|
|
thread computes it without a shared table. It applies first one wins
|
|
(below), writes Deltas and Snapshots, and is the one that asks connections
|
|
to subscribe, unsubscribe and resync: it knows a symbol before its book can
|
|
arrive. Each wall-clock minute it writes the minute Snapshots and reports
|
|
its markets' loads.
|
|
- **Orchestrator**: holds each market's two connections, places and moves
|
|
subscriptions, splits a connection, and runs the fetchers as in sprint 03.
|
|
- **Writer**: unchanged.
|
|
|
|
## First one wins
|
|
|
|
A symbol whose hash chain holds is **verified**. A change arriving from any of
|
|
its connections is then, in this order:
|
|
|
|
1. a **duplicate**, if it is one of the symbol's last 128 applied changes
|
|
(same hash, side, price and size): dropped;
|
|
2. **stale**, if its exchange timestamp is older than the last applied
|
|
change's: dropped (while the chain holds, everything up to the book's state
|
|
was applied);
|
|
3. otherwise **held** on its connection. The shortest tail of that
|
|
connection's held changes which, applied to the book, gives the hash of its
|
|
last change **fits**: it is applied and written; held changes before the
|
|
tail are stale and dropped; a change that leaves the book as it was is not
|
|
written. One rule covers a batch completing, a change the other connection
|
|
delivers after this one ran ahead, and late changes after a fresh book.
|
|
|
|
A held change is written with its own receive time, never earlier than the
|
|
symbol's previous Delta, so replaying the output by time keeps the order.
|
|
|
|
**The ceiling.** A change may be held for 1 s; for 50 ms once every connection
|
|
of the symbol that is up holds the same changes (nobody is left to wait for
|
|
but the rest of a batch); at once when a connection holds more than 512
|
|
changes of the symbol. Only tails that start a batch are tried against the
|
|
book: the whole held list and the last four. At the ceiling, if the book moved since the change
|
|
arrived, its connection is the one behind and its held changes are dropped.
|
|
If the book did not move, the chain is **broken**.
|
|
|
|
**A broken chain.** The book thread **follows** the connection whose held
|
|
changes reached the ceiling: its held changes and its later ones are applied
|
|
and written unchecked, the other connection's are dropped, and a resync goes
|
|
to the other connection, so the followed stream is not interrupted (to the
|
|
followed one when it is the only one up). The fresh book is compared as in
|
|
sprint 02: equal to ours or a state we passed, ours stands; otherwise it
|
|
replaces ours, a Snapshot is written and the difference is logged as a loss.
|
|
Either way the symbol is verified again. A symbol whose complete book fails
|
|
its own hash stays unverifiable, following one connection, as before.
|
|
|
|
**Full books.** Only books we asked for count: a subscribe's opening book
|
|
while the symbol awaits one (the first complete book from any of its
|
|
connections wins and is the symbol's first Snapshot), and a resync's answer.
|
|
Any other book is ignored: the second connection's opening book, a
|
|
reconnect's replay for verified symbols, the books the server sends after a
|
|
trade.
|
|
|
|
**Connections dropping.** A verified symbol needs nothing when one of its
|
|
connections drops: the other carries on, and the dropped one's held changes
|
|
are forgotten. When the followed connection drops, the other is followed and
|
|
asked for a fresh book, and a Snapshot is always written on it, as the brief
|
|
asks for a change of source. When every connection of a symbol is down, the
|
|
symbol awaits a book again: the first reconnect's opening book is taken as
|
|
fresh, and one that differs from ours is written as a Snapshot and logged as
|
|
a loss.
|
|
|
|
## Placing, moving, splitting
|
|
|
|
- **Startup**: the first rescan completes (1 to 2.5 minutes on the server,
|
|
Gamma's speed varying), then the markets are shuffled and dealt, each to two different
|
|
connections, at about 5,000 markets a second so the opening books fit the
|
|
link. Connections are opened for a fill of 4,500 tokens each.
|
|
- **A new market** goes on two different random connections with room in both
|
|
limits; a connection opens when fewer than two have room.
|
|
- **Load**: a market's load is the bytes per second of its frames on one
|
|
connection: its book thread counts every event's share from all of its
|
|
connections and divides by their number. A connection's load is the sum of
|
|
its markets'.
|
|
- **Rebalancing**, each minute: a connection over its budget has the busiest
|
|
markets' subscriptions moved, each to a random connection with room that
|
|
does not already carry the market: at most 50 moves a minute, a moved
|
|
subscription staying 10 minutes, none moved for a market with no load.
|
|
- **A move** is `Move(market, from, to)` to the market's book thread, which
|
|
subscribes on `to`, and unsubscribes on `from` once `to` has delivered a
|
|
book of the market: until then both carry it, so no change falls between
|
|
them. The book stays where it is: no Snapshot, no subscription gap, and the
|
|
new connection's opening book is ignored like any unasked one. While a
|
|
symbol is verified, a change is taken from any connection, the one it left
|
|
included: its copies in flight are the exchange's changes like any other.
|
|
- **A split**: a connection closed with 1013 twice within 10 minutes has every
|
|
second market, in load order, moved off at once, outside the two caps, and
|
|
is then left alone for 10 minutes.
|
|
- **Removal**: unchanged; the book thread unsubscribes both connections.
|
|
|
|
## Data model
|
|
|
|
- **Orchestrator**: `conditionId` → (tokens, its two connections, load, when
|
|
it last moved: both its subscriptions then stay); per connection, its token count, its load,
|
|
its last 1013 closes and its last split. The market stays the unit: both
|
|
tokens share the two connections.
|
|
- **Book thread**, per symbol: the book and the hash inputs (sprint 02); its
|
|
last 128 changes, each with its hash as 20 bytes (64 changes with the hash
|
|
as text today: ~2.4 GB at worst instead of ~5 GB); the held changes of each
|
|
of its connections; its state (awaiting a book, verified, following a
|
|
connection, unverifiable and following one); its connections; its last
|
|
Delta's receive time. Per market, the bytes since its last report. Per
|
|
connection, whether it is up.
|
|
- **Event** (connection → book thread): receive time, connection number,
|
|
bytes, and a parsed book, change, tick size change or trade; or a
|
|
connection up / down marker.
|
|
- **Cause lines**: every Snapshot written because a book differed, and every
|
|
Snapshot written for a change of followed connection, logs one line with
|
|
its symbol, its exact receive time and its cause (loss, subscription gap,
|
|
switch). One module, `src/cause.rs`, holds their format and their parser.
|
|
- **A tick size change** breaks the chain on the connection that delivered
|
|
it: the hashes change with the tick size, and the fresh book carries the
|
|
new one.
|
|
|
|
## Reconcile and the measures
|
|
|
|
- `reconcile OUTPUT [LOG]`: with the log, each mismatch is paired with the
|
|
cause line of its symbol and receive time; it prints the counts by cause
|
|
and fails on an unexplained mismatch or an orphan Delta. Without, every
|
|
mismatch is unexplained.
|
|
- Each book thread's minute line: changes written (and verified), duplicates
|
|
dropped, changes **rescued** (verified changes of a symbol on several
|
|
connections that only one delivered, counted once 2 s old, when every copy
|
|
has come), held changes dropped, chains broken, resyncs, books behind,
|
|
subscription gaps, losses, switches, the second copies' lag (median and
|
|
99th percentile), the most events waiting.
|
|
- The orchestrator's: markets, connections, added, removed, moved, splits, the
|
|
busiest connection's load.
|
|
|
|
## Decisions
|
|
|
|
- **First one wins, on one book per symbol** (ADR 0003). The book hash orders
|
|
changes across connections as a sequence number would. *Rejected:* a source
|
|
and a standby with a book each and a Snapshot per switch (my first
|
|
proposal): it loses the change the source missed, keeps the source's
|
|
receive time, and doubles the books.
|
|
- **A symbol's two connections are chosen independently** (ADR 0003;
|
|
supersedes ADR 0002's lanes, a connection and a book thread of its own,
|
|
because twins were closed together: 14 times more time with both down than
|
|
independent deals). *Rejected:* twin connections per lane.
|
|
- **Markets are placed at random** (supersedes "the least-loaded lane", and
|
|
"subscribe page by page at startup", because both build the clusters a load
|
|
average cannot see: random twins had a third of the closes of creation
|
|
order). Startup scans first, then deals. *Rejected:* least-loaded placement
|
|
(a batch of 1,000 markets created together lands on one connection);
|
|
dealing while scanning (needs the connection count before the scan ends).
|
|
- **The connection thread parses and routes.** The stamp still comes first; a
|
|
frame of several markets' books must be taken apart anyway; and the
|
|
connection becomes the exchange-specific decoder Extension 3 wants.
|
|
*Rejected:* forwarding the text after peeking at the market id.
|
|
- **16 book threads, a market's thread from its id.** One handled ~30,000
|
|
frames a second in sprint 02. *Rejected:* a thread per connection (the
|
|
lanes); a routing table (shared state).
|
|
- **Subscriptions go through the market's book thread**, which records the
|
|
market before asking. *Rejected:* the orchestrator telling connections
|
|
directly (a book can arrive before its thread knows the symbol).
|
|
- **Ceilings of 1 s and 50 ms; at the ceiling, held changes are dropped if the
|
|
book moved, else the chain is broken; a change older than the book is
|
|
dropped at once.** The lag reached 0.7 s, batches 40 ms. The last two come
|
|
from the prototype: a connection 2 s late was otherwise taken for a broken
|
|
chain and followed. *Rejected:* breaking at once (a change one connection
|
|
missed would cost a resync instead of nothing); no ceiling (a change could
|
|
wait unwritten for minutes).
|
|
- **A connection holds at most 512 changes of a symbol, and only the tails
|
|
starting a batch are tried, five at most** (found by the alternating runs:
|
|
at a 5-minute mark a crypto market's burst, behind one missing change, made
|
|
every arrival try every tail of thousands, each a hash; one book thread
|
|
then needed 9.5 minutes to drain after SIGINT). *Rejected:* trying every
|
|
tail (quadratic); no cap (a burst piles up for its whole ceiling).
|
|
- **Only asked-for books count.** A reconnect replays up to 5,000 books and
|
|
the server sends books after trades; reacting to them is what made
|
|
resyncs. *Rejected:* comparing every book.
|
|
- **The resync goes to the connection not followed**, so the followed stream
|
|
has no gap of its own. *Rejected:* resyncing the followed connection
|
|
(sprint 03: 11,605 gaps).
|
|
- **A split after two 1013 closes within 10 minutes.** One close can be the
|
|
server's moment (several connections within a second); a connection closed
|
|
again is overloaded. *Rejected:* splitting on the first close; lowering
|
|
every connection's budget.
|
|
- **Threads, not tokio** (ADR 0001's checkpoint, held here): the new logic is
|
|
in the book thread, not in socket handling, and ~200 threads are
|
|
comfortable. *Rejected:* an async socket layer now.
|
|
- **Two subscriptions per market, a constant**; the comparison run is the same
|
|
code at 1. *Rejected:* a runtime option.
|
|
- **Loss means a change no connection delivered in time**; one a single
|
|
connection missed is rescued. "Lane" is retired: connections and book
|
|
threads are separate things now.
|
|
- **`reconcile` reads the log, through one module shared with the
|
|
collector.** *Rejected:* a second, machine-readable output.
|
|
- **Rescued counts copies, not held changes** (supersedes "held, then fit
|
|
after another connection moved the book", because a change one connection
|
|
missed usually arrives on the other before the first's next change, and is
|
|
then never held: the tests' rescues counted 0). *Rejected:* counting held
|
|
changes that fit (most rescues go unseen).
|
|
- **A move leaves the old connection once the new one delivers a book**
|
|
(supersedes "subscribes on `to` and then unsubscribes on `from`", because
|
|
unsubscribing at once loses the changes between `from`'s unsubscribe and
|
|
`to`'s first book). *Rejected:* a fixed delay before unsubscribing.
|
|
- **SIGTERM (and SIGHUP) stop the collector like SIGINT** (`ctrlc`'s
|
|
`termination` feature): `timeout`, `docker stop` and a service manager send
|
|
SIGTERM, and only a clean stop flushes the last chunk.
|
|
- **Built in two steps, with a live run between**: first the connections
|
|
separated from the book threads with one subscription per market, which
|
|
must behave like sprint 03's collector; then the second subscription, first
|
|
one wins and the split.
|
|
|
|
## Tests
|
|
|
|
No network, except the local websocket server of sprint 03's reconnect test.
|
|
The fixture is re-recorded on two connections (`record.py`, extended) for the
|
|
same 40 tokens; replays feed both in receive order.
|
|
|
|
- **Same output**: two connections give the Deltas, in order per symbol, that
|
|
one alone gives; the other's copies are counted as duplicates.
|
|
- **A loss on one connection**: a frame removed from one changes nothing in
|
|
the output, and its changes count as rescued.
|
|
- **A loss on both**: the same frame removed from both leaves the next change
|
|
held; at the ceiling the chain breaks, the resync goes to the connection not
|
|
followed, and a differing fresh book is written as a Snapshot, logged a loss.
|
|
- **A slow connection**: one delayed by 2 s gives the same output, each Delta
|
|
with the earlier receive time, and no chain broken by the delay.
|
|
- **Stamps**: a held change is not stamped earlier than the symbol's previous
|
|
Delta.
|
|
- **A batch** is written only once it fits; a fresh book's late changes are
|
|
dropped as stale.
|
|
- **Books**: an unasked book changes nothing; the first of two opening books
|
|
is the Snapshot.
|
|
- **Connections dropping**: one down, the output goes on; the followed one
|
|
down, a Snapshot on the fresh book; both down, every symbol awaits a book.
|
|
- **Routing**: a frame of several markets' books reaches each market's book
|
|
thread; a market's thread is the same from the orchestrator and from a
|
|
connection.
|
|
- **Placement**: a market's two connections differ; a move never targets the
|
|
market's other connection; a connection opens when fewer than two have
|
|
room; the startup deal is even.
|
|
- **Move**: subscribe on the new connection, then unsubscribe on the old; no
|
|
Snapshot.
|
|
- **Split**: two closes within 10 minutes move every second market; one close
|
|
moves none; a second split waits 10 minutes.
|
|
- **Load**: a market on two connections reports the load of one.
|
|
- **Reconcile with a log**: explained mismatches pass; an unexplained one
|
|
fails.
|
|
- Sprint 03's tests of Gamma, the minute check, rescan removal, rebalancing's
|
|
caps and the reconnect stay, adapted to connections.
|
|
|
|
## Not in this sprint
|
|
|
|
- Acting on delay: the second delivery's lag is measured and logged; choosing
|
|
connections by it is M7's.
|
|
- The crash chunk: a hard crash still loses up to 64 MiB, ~10 s of output.
|
|
- Balancing on a market's peak second rather than its minute average.
|
|
- Reading the last trade price from the trade event to verify again without
|
|
a resync (the prototype did it for 15 of 100 broken chains): a Possible
|
|
feature.
|
|
- Detecting a subscription gone quiet on one connection and resubscribing it,
|
|
and recycling connections: M5. First one wins hides a quiet subscription;
|
|
M5 has to find it.
|
|
- A market's `min_order_size` or `neg_risk` changing: still read once.
|
|
|
|
## Done when
|
|
|
|
- `scripts/ci.sh` is green.
|
|
- **Step 1** (one subscription per market): a 10-minute live run has every
|
|
`reconcile` mismatch explained by the log.
|
|
- **The comparison**, started when the user says so: two 20-minute live runs
|
|
back to back, one subscription per market, then two. In the second:
|
|
- every `reconcile` mismatch is explained by the log;
|
|
- losses and subscription gaps per million changes are below the first
|
|
run's;
|
|
- rescued changes, duplicates, chains broken and the second delivery's lag
|
|
are reported;
|
|
- a split was seen, or its absence explained;
|
|
- peak memory and thread count are reported.
|
|
- Free disk is checked before each run, and the outputs deleted after.
|
|
- ADR 0003 is written, ADR 0001 records the checkpoint, ADR 0002 says what
|
|
was superseded; `GLOSSARY.md`, `docs/project.md`'s pipeline, `docs/notes.md`
|
|
(the measures above) and the roadmap match.
|
|
|
|
## Outcome
|
|
|
|
Every live market (~198k, ~396k symbols) is on two connections chosen at
|
|
random (~178 connections), and 16 book threads take each change from
|
|
whichever connection delivers it first. Built in the two planned steps;
|
|
step 1 (connections separated, one subscription) reconciled fully in a
|
|
10-minute live run, no connection lost.
|
|
|
|
**The comparison** was run as four alternating 10-minute rounds, one
|
|
subscription per market then two, twice (asked by the user once the server
|
|
proved overloaded during a first back-to-back pair). Per million changes
|
|
written:
|
|
|
|
| Round | Subscriptions | Losses | Subscription gaps | Losses + gaps | Chains broken |
|
|
|---|---|---|---|---|---|
|
|
| 1 | 1 | 69.1 | 163.7 | 232.8 | 525 |
|
|
| 2 | 2 | 49.7 | 160.2 | 209.9 | 566 |
|
|
| 3 | 1 | 42.1 | 142.0 | 184.1 | 530 |
|
|
| 4 | 2 | 32.3 | 152.1 | 184.5 | 503 |
|
|
|
|
Every round reconciled with each mismatch explained by its log line. Losses
|
|
fell by a quarter in each pair; losses and gaps together fell in the first
|
|
pair and tied in the second, the rates drifting down through the hour. The
|
|
user accepted this as the sprint's result. Subscription gaps, mostly from
|
|
opening subscribes that both connections share, are untouched by a second
|
|
connection. Most "losses" are likely not data missed: chains break as often
|
|
with one connection as with two (about the rate of trades), and a resync's
|
|
book that is a few changes ahead of our stream is logged a loss too.
|
|
Duplicates: one per change written. The second copy came 1 to 2 ms after
|
|
the first at the median, 45 to 120 ms at p99. Peak memory 1.78 GB with two
|
|
subscriptions, 1.67 GB with one; 200 threads. No split: no connection was
|
|
closed twice within 10 minutes (0 and 2 connections lost in the
|
|
two-subscription rounds). The first, overloaded pair (load up to 92 on 12
|
|
cores) still reconciled fully: a stress test passed.
|
|
|
|
**Rescued** (~110,000 per round) is an upper bound, accepted as such: a
|
|
change that leaves the last 128 remembered before its second copy comes, on
|
|
a busy symbol, is counted as delivered once; so is a batch whose changes the
|
|
two connections delivered interleaved (the held copy dropped uncounted).
|
|
|
|
**Fixed at the close**, from the code review: a move ending while a fresh
|
|
book was asked from the connection left kept asking it in vain (now asked
|
|
from another); a change held across a minute could be stamped before that
|
|
minute's Snapshot (a Snapshot now advances the symbol's last stamp); markets
|
|
closed during the first scan were dealt anyway. The first two have a test
|
|
that fails without the fix; the third sits in the orchestrator's loop, as
|
|
untested as the rest of it.
|
|
|
|
**Diverged from the plan**, each in Decisions: markets on independent
|
|
connections, not twins (the user's design, after the three-layout test);
|
|
rescued counted from copies; a move leaves the old connection once the new
|
|
one delivers a book; the held-change cap and batch-start tails (found by the
|
|
runs: a crypto burst behind one missing change made a book thread fall
|
|
455,000 events behind and take 9.5 minutes to stop); a tick size change
|
|
breaks the chain; cause lines carry the exact receive time.
|
|
|
|
**Left out**, and where it went: telling a real loss from a resync book
|
|
ahead of our stream, and the opening subscription gaps (Possible features);
|
|
an exact rescued count (Possible features); acting on delay (M7); the crash
|
|
chunk (M4's ceiling, kept); quiet subscriptions hidden by first one wins
|
|
(M5).
|
|
|
|
### Retrospective
|
|
|
|
- **Went well:** experiments settled the design before code: two recordings
|
|
showed the hash orders changes across connections, the three-layout test
|
|
turned a twin design into the user's independent one, a Python prototype
|
|
found the slow-connection flaw before any Rust. `reconcile` reading the log
|
|
made every live run a correctness check, and it held even under load.
|
|
- **Went wrong:** a quadratic loop marked "fine at a few dozen" met bursts of
|
|
thousands; it surfaced only as a book thread falling behind, then as a
|
|
shutdown that took 9.5 minutes. The first comparison pair ran on an
|
|
overloaded server and wasn't comparable. Wait loops on `pgrep -f` failed
|
|
twice more, once killing the wrong process.
|
|
- **Tested enough?** The rules are, on a two-connection recording with
|
|
injected losses, delays and drops, and a burst test that hung before the
|
|
fix. Not covered: the orchestrator's loop (pacing the deal, routing
|
|
commands, splits sent), the socket's up/down markers, a move whose new
|
|
connection never delivers, and the minute line's lag figures.
|
|
- **Next time:** load-test the book thread with a synthetic burst before a
|
|
live run, and check the server's load before a comparison; keep the
|
|
ceiling of every corner cut in a test, not just a comment.
|