The exchange can fail without telling us, and first one wins hides a subscription gone quiet on one of a symbol's two connections. - A connection that receives nothing for 30 s, not even the keepalive's answer, reconnects. - Each book thread counts a symbol's changes per connection every minute: a subscription with none against three or more on the other is resubscribed, and moved to another connection if it stays quiet. - The orchestrator recycles one connection at a time, each about hourly, and never while another connection is down or just back. - A reconnect's opening book of a symbol unchanged for 10 s is judged like a resync's answer: one newer than ours that differs shows a symbol quiet on both connections, written as a Snapshot and logged as a loss.
16 KiB
05. Subscription monitoring
Status: done (2026-11-11)
The exchange can fail without telling us: a connection stays open and sends nothing, a symbol's subscription goes quiet on a connection that still works, and a connection accumulates the exchange's own bugs. First one wins hides a quiet subscription as long as the symbol's other connection delivers, so it has to be looked for. This sprint finds a silent connection and reconnects it, finds a subscription quiet on one connection and resubscribes it (then moves it), and recycles every connection about once an hour; the recycle's opening books also show the symbols quiet on both connections. This is Extension 2 of the brief. Design: ../docs/project.md (pipeline), ADR 0003, ../docs/notes.md, ../GLOSSARY.md (quiet, silent, recycle).
Measured before planning (2026-11-11)
A throwaway build, 25 minutes live (~198k markets, ~180 connections, the server loaded: 14 to 45 on 12 cores), with a forced recycle every 60 s:
- Quiet subscriptions are rare. Of ~392,000 symbols judged each minute, ~75,000 changed. In 21 minutes, one market once had 3 or more changes on one connection and none on the other (not the next minute); none had 10. One change against none came 0 to 16 times a minute: a copy landing in the next minute. Both tokens of a market go one-sided together.
- A reconnect's opening book can lag by minutes. For symbols unchanged for 10 s or more, 514 of 19,379 reconnect books differed from ours; 512 of them equal one of our earlier states (1 to 4 levels, some over 10 minutes old), never with a newer timestamp. 2 (one market) were neither. All 42,096 books of symbols never changed since their first book matched. Under 10 s, some books ran ahead of our stream (2 between 1 and 10 s).
- A recycle is cheap: 1.2 s from close to up; all opening books 0.3 s later on most connections, 20 s on one, and 2 of 4,490 never on another. No 1013 close in the run; chains broken and losses per minute the same during recycles as before.
A silent connection
A connection that has received nothing, data or PONG, for 30 s (three
keepalives) closes itself and reconnects like a dropped one, telling the book
threads it went down. Today nothing notices: the read loop
(socket.rs:222) times out every 200 ms and carries on, and PONG decodes
to no event (wire.rs:37).
A quiet subscription
Each book thread counts, per symbol and per connection, the changes received each wall-clock minute: every one, whether it is then applied, a duplicate, held or stale. At the minute, a verified symbol on two connections, both up the whole minute and subscribed on both since before it began, whose one connection delivered at least 3 changes and the other none, is quiet on the other:
- the first time, that connection gets a
Resubscribe(unsubscribe, subscribe: the command a resync already used, renamed fromResyncsince here no book is asked for). The symbol stays verified on the other connection, so the book that comes back is ignored like any unasked one: no Snapshot, no subscription gap; - quiet again the next minute on the same connection, the book thread reports
the market to the orchestrator, which moves that subscription with the
existing
Move, outside the cap of 50 moves a minute, and logs it; a market moved in the last 10 minutes stays, as for rebalancing.
A symbol with no change on either connection is not judged: the market may just be still. That case is the recycle's.
Recycling
The orchestrator recycles, once the startup deal is complete: every 3600/N seconds (N connections), the connection whose last connect is oldest closes and reconnects through the path a dropped connection takes (1 s, then its symbols subscribed in operations of 500). The next recycle waits until that connection has reported up again, none starts while any connection is down, nor for one interval after any connection came up (up means its subscribes are sent; its opening books take 0.3 to 20 s), so a recycle never takes down a market's two connections together. A connection that dropped on its own counts as recycled from its reconnect.
For this the orchestrator keeps each connection's command sender (today
Threads::open only hands it to the book threads), a Recycle command
closes the socket, and the channel that carries 1013 closes today
(Sender<usize>) carries a connection's notices: up, down, closed with
1013, closed for silence.
Reconnect books
Sprint 04 decided that only asked-for books count, because a moving symbol's books arrive ahead of or behind our stream. A reconnect's opening book for a symbol unchanged for at least 10 s is an exception: nothing moved, so it is judged like a resync's answer. Equal to ours or to one of our earlier states (the server's opening books lag, by minutes: above), ours stands and it counts as behind; so does any book whose timestamp is no newer than our last change, since our chain holds up to that change and we only remember 128 earlier states. A book that is unreadable or fails its own hash is ignored. Otherwise both subscriptions were quiet: it replaces ours, is written as a Snapshot with a cause line (a loss, its detail saying a reconnect found it), and the symbol's other connection is resubscribed. This holds for any reconnect, recycled or dropped.
Data model
- Book thread, per symbol: the changes received this minute on each of its connections, and since when it is subscribed on each; whether it was quiet last minute, and on which connection. Per connection, when it last came up.
- Book thread → orchestrator: the load report, or a market quiet twice on a connection.
- Connection → orchestrator: up, down, a 1013 close, a close for silence.
- Orchestrator, per connection: when it last connected; the recycle under way.
- Cause lines: a reconnect book that differs is a
loss, with a detail naming the reconnect, soreconcilepairs it and the log tells them apart.
The minute lines
Each book thread's adds: the changes received this minute (all connections together: one figure per connection would put ~180 in each line), quiet subscriptions resubscribed and reported, reconnect books judged and found differing (those behind count with the resyncs' "full books behind ours"). The orchestrator's adds: connections recycled, quiet moves, and silent connections closed. Totals only: a per-symbol table would be ~400k lines a minute.
Decisions
- Changes received per symbol and connection, per minute are the activity measure. Rejected: bytes (the load is per market, averaged over its connections); changes applied (the second connection applies almost none, by design).
- Quiet means one connection delivered and the other nothing; a dip against the symbol's own past rate is not used. Prediction markets are bursty, so a dip is a weak signal, while the other connection, or a fresh book, is a real test. Rejected: the brief's literal per-symbol rate threshold.
- Resubscribe once, then move. Rejected: resubscribing every minute forever (never leaves a connection broken for that symbol).
- The book thread reports a market quiet twice; the orchestrator moves it. Only the orchestrator knows which connections have room. Rejected: the book thread choosing a target.
- The orchestrator recycles one connection at a time, oldest first, about hourly, starting after the startup deal. Rejected: each connection recycling itself at an age of an hour plus jitter (~600 markets an hour would have both connections down at once); waiting an hour before the first recycle (a 30-minute run would show none).
- A recycle is a plain close and reconnect. M4's second connection covers the seconds it takes. Rejected: opening the replacement before closing (a third connection per symbol and a handover).
- No recycle while any connection is down. Covers every market with no per-market lookup. Rejected: checking each market's other connection.
- A reconnect's opening book is compared for a long-unchanged symbol, on every reconnect (amends sprint 04's "only asked-for books count", whose reason, books ahead of or behind a moving stream, doesn't apply to a symbol that hasn't moved). Rejected: comparing only after a recycle (a dropped connection's reconnect is the same check).
- Quiet at 3 changes against none. Crossed once in 21 live minutes; a needless resubscribe costs no Snapshot or gap. Rejected: 1 (up to 16 needless resubscribes a minute, from copies landing in the next minute); 10 (a slowly trading market's quiet subscription goes unseen).
- A reconnect book is judged by the resync's rule, from 10 s unchanged (supersedes "it must equal ours", because 2.7% of them equal an earlier state of ours: the server's books lag by minutes). Rejected: exact equality (~500 false losses per 15 recycles); under 10 s (books run ahead of a moving stream).
- The orchestrator hears down and silent too (supersedes "a 1013 close, or up", because "no recycle while any connection is down" needs down, and the minute line counts the silent closes).
- A reconnect book no newer than our last change is behind (added at the code review: a book lagging further back than the 128 changes we remember was taken for a loss and replaced a correct book). Rejected: remembering more states.
- No recycle for one interval after any connection came up (added at the code review: an overdue recycle fired the moment a connection reported up, before its opening books).
- A differing reconnect book is a loss, its cause line saying how it was found. Rejected: a new cause (the changes are ones no connection delivered: that is a loss).
- A silent connection is closed after 30 s with nothing received, one constant.
Tests
No network, except the local websocket server of sprint 03's reconnect test.
- Quiet: the two-connection recording, one symbol's changes removed from one connection for a minute: that connection is resubscribed and no Snapshot is written; removed again the next minute: the market is reported. A subscription younger than the minute is not judged; a symbol with no change on either connection is not quiet.
- Reconnect books: after an up marker, an opening book equal to ours for a long-unchanged symbol changes nothing; one that differs is written as a Snapshot with its cause line and resubscribes the other connection; a book for a recently changed symbol is ignored as before.
- Recycle (on the
Tracker): the oldest connection first, the next only after the previous is up, none while a connection is down, paced at 3600/N. - A quiet market reported is moved, outside the cap.
- Silent: on the local websocket server, a connection that hears nothing for the timeout (shortened in the test) reconnects with its symbols.
- Reconcile: a reconnect book's cause line pairs with its Snapshot.
Not in this sprint
- A dip against a symbol's own past rate (see Decisions).
- Opening a replacement connection before closing the recycled one.
- Per-second rates: a minute decides a resubscribe.
- Acting on delay: M7.
- Injecting a fault into a live run: the tests carry the rules.
Done when
scripts/ci.shis green.- A 30-minute live run with recycling on (about 90 connections recycled at
~178 connections):
- every
reconcilemismatch is explained by the log; - the minute lines show recycles, quiet subscriptions and reconnect books, and their counts are reported (none quiet in the run: the tests carry it, and the Outcome says so);
- losses and subscription gaps per million changes are no higher than sprint 04's two-subscription rounds (32 to 50 losses, 150 to 160 gaps);
- peak memory and thread count are reported.
- every
- Free disk and the server's load are checked before the run, the output deleted after.
GLOSSARY.md(quiet, silent, recycle),docs/project.md's pipeline,docs/notes.md(the measures above) and the roadmap match.
Outcome
The collector now looks for the exchange failing silently. A connection that hears nothing for 30 s reconnects. Each book thread counts every symbol's changes per connection and, at the minute, resubscribes a subscription with none against three or more on the other; quiet again, its market is reported and the orchestrator moves it. The orchestrator recycles one connection at a time, the one up the longest, each about hourly, and a reconnect's opening book of a symbol unchanged for 10 s is judged like a resync's answer.
Two live runs, each reconciled with every mismatch explained by its log line, no orphan Delta:
| Run | Recycles | Losses / M changes | Gaps / M | Quiet resubscribed | Reconnect books judged | Peak memory, threads |
|---|---|---|---|---|---|---|
| 30 min, before the review's fixes | 107 | 26.5 | 56 | 26 (none quiet again) | 335,488 (5 losses) | 2.46 GB, 204 |
| 10 min, after them | 19 | 28.4 | 190 | 0 | 69,340 (0 losses) | 1.94 GB, 202 |
Sprint 04's two-subscription rounds (10 minutes each) had 32 to 50 losses and 150 to 160 gaps per million. Losses are no higher with recycling. The gaps are the startup subscribes' (3,111 of them by minute 10, 3,275 by minute 30), so their rate says how long the run was, not what recycling costs: about 160 more in 20 minutes of recycling. No connection fell silent and no market was reported quiet twice in either run: the tests carry both, and the silent close's count. Memory grows with the run's length (the last 128 changes of each symbol fill up); at 10 minutes it is 0.16 GB above sprint 04's, not looked into further.
The 5 reconnect losses of the first run are doubtful: 4 came at 17:00:04 with book times 0.3 to 2 s old. The review then found that a book lagging further back than the changes we remember was taken for a loss, and the rule "no newer than our last change is behind" was added; the second run judged 69,340 books and found none.
Fixed at the close, from the code review, each with a test that fails without it: a reconnect book older than our last change, unreadable, or failing its own hash now leaves a verified symbol alone (the first replaced a correct book, the last made the symbol unverifiable for good); no recycle for one interval after a connection came up (an overdue one fired before the returning connection's books); a market moved in the last 10 minutes is not moved again for being quiet (it ended the move under way). The tests were strengthened after the reviewer removed rules and they stayed green: twelve rules were then removed one by one, each caught by a named test.
Diverged from the plan, each in Decisions or its section: the
orchestrator hears down and silent as well as up; Resync the command
became Resubscribe; the minute line gives one total of changes received;
the three rules above.
Left out, and where it went: telling a change still in flight from a
symbol quiet on both connections when a reconnect book is newer than ours
(Possible features); the dip against a symbol's own past rate, the
replacement connection opened first, per-second rates (not planned, see
Not in this sprint); acting on delay (M7). Not covered by a test: the
orchestrator's loop (sending the Recycle, counting the notices), a quiet
judging skipped for a symbol being resynced, and reconcile pairing a
reconnect's loss end to end (its cause line's parsing is tested, and the
first live run paired five).
Retrospective
- Went well: the measurement before planning overturned a rule on paper (reconnect books lag by minutes) instead of in a live run. The reviewer's mutation check, removing a rule and running the tests, showed which tests proved nothing; redone here after the fixes, it took minutes.
- Went wrong: the first tests were written to pass, not to fail without
their rule: six rules were uncovered. The reconnect rule leaned on a
memory of 128 changes without asking what happens past it. The first live
run was launched with a tool the server doesn't have (
/usr/bin/time), and I announced its end time wrongly. - Tested enough? The rules are, by mutation. The orchestrator's loop is still untested, as in sprint 04, and neither silent connections nor quiet moves happened live.
- Next time: for each rule, write the test, then remove the rule and see it fail, before the review does. Ask of every "we remember N" what happens at N + 1.