Archived
The exchange can fail without telling us, and first one wins hides a subscription gone quiet on one of a symbol's two connections. - A connection that receives nothing for 30 s, not even the keepalive's answer, reconnects. - Each book thread counts a symbol's changes per connection every minute: a subscription with none against three or more on the other is resubscribed, and moved to another connection if it stays quiet. - The orchestrator recycles one connection at a time, each about hourly, and never while another connection is down or just back. - A reconnect's opening book of a symbol unchanged for 10 s is judged like a resync's answer: one newer than ours that differs shows a symbol quiet on both connections, written as a Snapshot and logged as a loss.
302 lines
16 KiB
Markdown
302 lines
16 KiB
Markdown
# 05. Subscription monitoring
|
|
|
|
**Status: done (2026-11-11)**
|
|
|
|
The exchange can fail without telling us: a connection stays open and sends
|
|
nothing, a symbol's subscription goes quiet on a connection that still works,
|
|
and a connection accumulates the exchange's own bugs. First one wins hides a
|
|
quiet subscription as long as the symbol's other connection delivers, so it
|
|
has to be looked for. This sprint finds a silent connection and reconnects
|
|
it, finds a subscription quiet on one connection and resubscribes it (then
|
|
moves it), and recycles every connection about once an hour; the recycle's
|
|
opening books also show the symbols quiet on both connections. This is
|
|
Extension 2 of the brief. Design: [../docs/project.md](../docs/project.md)
|
|
(pipeline), [ADR 0003](../docs/adr/0003-first-one-wins.md),
|
|
[../docs/notes.md](../docs/notes.md), [../GLOSSARY.md](../GLOSSARY.md)
|
|
(quiet, silent, recycle).
|
|
|
|
## Measured before planning (2026-11-11)
|
|
|
|
A throwaway build, 25 minutes live (~198k markets, ~180 connections, the
|
|
server loaded: 14 to 45 on 12 cores), with a forced recycle every 60 s:
|
|
|
|
- **Quiet subscriptions are rare.** Of ~392,000 symbols judged each minute,
|
|
~75,000 changed. In 21 minutes, one market once had 3 or more changes on
|
|
one connection and none on the other (not the next minute); none had 10.
|
|
One change against none came 0 to 16 times a minute: a copy landing in
|
|
the next minute. Both tokens of a market go one-sided together.
|
|
- **A reconnect's opening book can lag by minutes.** For symbols unchanged
|
|
for 10 s or more, 514 of 19,379 reconnect books differed from ours; 512 of
|
|
them equal one of our earlier states (1 to 4 levels, some over 10 minutes
|
|
old), never with a newer timestamp. 2 (one market) were neither. All
|
|
42,096 books of symbols never changed since their first book matched.
|
|
Under 10 s, some books ran ahead of our stream (2 between 1 and 10 s).
|
|
- **A recycle is cheap**: 1.2 s from close to up; all opening books 0.3 s
|
|
later on most connections, 20 s on one, and 2 of 4,490 never on another.
|
|
No 1013 close in the run; chains broken and losses per minute the same
|
|
during recycles as before.
|
|
|
|
## A silent connection
|
|
|
|
A connection that has received nothing, data or `PONG`, for 30 s (three
|
|
keepalives) closes itself and reconnects like a dropped one, telling the book
|
|
threads it went down. Today nothing notices: the read loop
|
|
(`socket.rs:222`) times out every 200 ms and carries on, and `PONG` decodes
|
|
to no event (`wire.rs:37`).
|
|
|
|
## A quiet subscription
|
|
|
|
Each book thread counts, per symbol and per connection, the changes received
|
|
each wall-clock minute: every one, whether it is then applied, a duplicate,
|
|
held or stale. At the minute, a verified symbol on two connections, both up
|
|
the whole minute and subscribed on both since before it began, whose one
|
|
connection delivered at least 3 changes and the other none, is
|
|
**quiet** on the other:
|
|
|
|
- the first time, that connection gets a `Resubscribe` (unsubscribe,
|
|
subscribe: the command a resync already used, renamed from `Resync` since
|
|
here no book is asked for). The symbol stays verified on the other
|
|
connection, so the book that comes back is ignored like any unasked one:
|
|
no Snapshot, no subscription gap;
|
|
- quiet again the next minute on the same connection, the book thread reports
|
|
the market to the orchestrator, which moves that subscription with the
|
|
existing `Move`, outside the cap of 50 moves a minute, and logs it; a
|
|
market moved in the last 10 minutes stays, as for rebalancing.
|
|
|
|
A symbol with no change on either connection is not judged: the market may
|
|
just be still. That case is the recycle's.
|
|
|
|
## Recycling
|
|
|
|
The orchestrator recycles, once the startup deal is complete: every 3600/N
|
|
seconds (N connections), the connection whose last connect is oldest closes
|
|
and reconnects through the path a dropped connection takes (1 s, then its
|
|
symbols subscribed in operations of 500). The next recycle waits until that
|
|
connection has reported up again, none starts while any connection is down,
|
|
nor for one interval after any connection came up (up means its subscribes
|
|
are sent; its opening books take 0.3 to 20 s), so a recycle never takes down
|
|
a market's two connections together. A connection that dropped on its own
|
|
counts as recycled from its reconnect.
|
|
|
|
For this the orchestrator keeps each connection's command sender (today
|
|
`Threads::open` only hands it to the book threads), a `Recycle` command
|
|
closes the socket, and the channel that carries 1013 closes today
|
|
(`Sender<usize>`) carries a connection's notices: up, down, closed with
|
|
1013, closed for silence.
|
|
|
|
## Reconnect books
|
|
|
|
Sprint 04 decided that only asked-for books count, because a moving symbol's
|
|
books arrive ahead of or behind our stream. A reconnect's opening book for a
|
|
symbol unchanged for at least 10 s is an exception: nothing moved, so it is
|
|
judged like a resync's answer. Equal to ours or to one of our earlier states
|
|
(the server's opening books lag, by minutes: above), ours stands and it
|
|
counts as behind; so does any book whose timestamp is no newer than our last
|
|
change, since our chain holds up to that change and we only remember 128
|
|
earlier states. A book that is unreadable or fails its own hash is ignored.
|
|
Otherwise both subscriptions were quiet: it replaces ours,
|
|
is written as a Snapshot with a cause line (a loss, its detail saying a
|
|
reconnect found it), and the symbol's other connection is resubscribed.
|
|
This holds for any reconnect, recycled or dropped.
|
|
|
|
## Data model
|
|
|
|
- **Book thread**, per symbol: the changes received this minute on each of
|
|
its connections, and since when it is subscribed on each; whether it was
|
|
quiet last minute, and on which connection. Per connection, when it last
|
|
came up.
|
|
- **Book thread → orchestrator**: the load report, or a market quiet twice on
|
|
a connection.
|
|
- **Connection → orchestrator**: up, down, a 1013 close, a close for silence.
|
|
- **Orchestrator**, per connection: when it last connected; the recycle under
|
|
way.
|
|
- **Cause lines**: a reconnect book that differs is a `loss`, with a detail
|
|
naming the reconnect, so `reconcile` pairs it and the log tells them apart.
|
|
|
|
## The minute lines
|
|
|
|
Each book thread's adds: the changes received this minute (all connections
|
|
together: one figure per connection would put ~180 in each line), quiet
|
|
subscriptions resubscribed and reported, reconnect books judged and found
|
|
differing (those behind count with the resyncs' "full books behind ours"). The orchestrator's adds: connections recycled, quiet moves, and
|
|
silent connections closed. Totals only: a per-symbol table would be ~400k
|
|
lines a minute.
|
|
|
|
## Decisions
|
|
|
|
- **Changes received per symbol and connection, per minute** are the
|
|
activity measure. *Rejected:* bytes (the load is per market, averaged over
|
|
its connections); changes applied (the second connection applies almost
|
|
none, by design).
|
|
- **Quiet means one connection delivered and the other nothing**; a dip
|
|
against the symbol's own past rate is not used. Prediction markets are
|
|
bursty, so a dip is a weak signal, while the other connection, or a fresh
|
|
book, is a real test. *Rejected:* the brief's literal per-symbol rate
|
|
threshold.
|
|
- **Resubscribe once, then move.** *Rejected:* resubscribing every minute
|
|
forever (never leaves a connection broken for that symbol).
|
|
- **The book thread reports a market quiet twice; the orchestrator moves
|
|
it.** Only the orchestrator knows which connections have room.
|
|
*Rejected:* the book thread choosing a target.
|
|
- **The orchestrator recycles one connection at a time, oldest first, about
|
|
hourly, starting after the startup deal.** *Rejected:* each connection
|
|
recycling itself at an age of an hour plus jitter (~600 markets an hour
|
|
would have both connections down at once); waiting an hour before the
|
|
first recycle (a 30-minute run would show none).
|
|
- **A recycle is a plain close and reconnect.** M4's second connection
|
|
covers the seconds it takes. *Rejected:* opening the replacement before
|
|
closing (a third connection per symbol and a handover).
|
|
- **No recycle while any connection is down.** Covers every market with no
|
|
per-market lookup. *Rejected:* checking each market's other connection.
|
|
- **A reconnect's opening book is compared for a long-unchanged symbol, on
|
|
every reconnect** (amends sprint 04's "only asked-for books count", whose
|
|
reason, books ahead of or behind a moving stream, doesn't apply to a
|
|
symbol that hasn't moved). *Rejected:* comparing only after a recycle (a
|
|
dropped connection's reconnect is the same check).
|
|
- **Quiet at 3 changes against none.** Crossed once in 21 live minutes; a
|
|
needless resubscribe costs no Snapshot or gap. *Rejected:* 1 (up to 16
|
|
needless resubscribes a minute, from copies landing in the next minute);
|
|
10 (a slowly trading market's quiet subscription goes unseen).
|
|
- **A reconnect book is judged by the resync's rule, from 10 s unchanged**
|
|
(supersedes "it must equal ours", because 2.7% of them equal an earlier
|
|
state of ours: the server's books lag by minutes). *Rejected:* exact
|
|
equality (~500 false losses per 15 recycles); under 10 s (books run ahead
|
|
of a moving stream).
|
|
- **The orchestrator hears down and silent too** (supersedes "a 1013 close,
|
|
or up", because "no recycle while any connection is down" needs down, and
|
|
the minute line counts the silent closes).
|
|
- **A reconnect book no newer than our last change is behind** (added at the
|
|
code review: a book lagging further back than the 128 changes we remember
|
|
was taken for a loss and replaced a correct book). *Rejected:* remembering
|
|
more states.
|
|
- **No recycle for one interval after any connection came up** (added at the
|
|
code review: an overdue recycle fired the moment a connection reported up,
|
|
before its opening books).
|
|
- **A differing reconnect book is a loss**, its cause line saying how it was
|
|
found. *Rejected:* a new cause (the changes are ones no connection
|
|
delivered: that is a loss).
|
|
- **A silent connection is closed after 30 s with nothing received**, one
|
|
constant.
|
|
|
|
## Tests
|
|
|
|
No network, except the local websocket server of sprint 03's reconnect test.
|
|
|
|
- **Quiet**: the two-connection recording, one symbol's changes removed from
|
|
one connection for a minute: that connection is resubscribed and no
|
|
Snapshot is written; removed again the next minute: the market is reported.
|
|
A subscription younger than the minute is not judged; a symbol with no
|
|
change on either connection is not quiet.
|
|
- **Reconnect books**: after an up marker, an opening book equal to ours for
|
|
a long-unchanged symbol changes nothing; one that differs is written as a
|
|
Snapshot with its cause line and resubscribes the other connection; a book
|
|
for a recently changed symbol is ignored as before.
|
|
- **Recycle** (on the `Tracker`): the oldest connection first, the next only
|
|
after the previous is up, none while a connection is down, paced at 3600/N.
|
|
- **A quiet market reported** is moved, outside the cap.
|
|
- **Silent**: on the local websocket server, a connection that hears nothing
|
|
for the timeout (shortened in the test) reconnects with its symbols.
|
|
- **Reconcile**: a reconnect book's cause line pairs with its Snapshot.
|
|
|
|
## Not in this sprint
|
|
|
|
- A dip against a symbol's own past rate (see Decisions).
|
|
- Opening a replacement connection before closing the recycled one.
|
|
- Per-second rates: a minute decides a resubscribe.
|
|
- Acting on delay: M7.
|
|
- Injecting a fault into a live run: the tests carry the rules.
|
|
|
|
## Done when
|
|
|
|
- `scripts/ci.sh` is green.
|
|
- A 30-minute live run with recycling on (about 90 connections recycled at
|
|
~178 connections):
|
|
- every `reconcile` mismatch is explained by the log;
|
|
- the minute lines show recycles, quiet subscriptions and reconnect books,
|
|
and their counts are reported (none quiet in the run: the tests carry it,
|
|
and the Outcome says so);
|
|
- losses and subscription gaps per million changes are no higher than
|
|
sprint 04's two-subscription rounds (32 to 50 losses, 150 to 160 gaps);
|
|
- peak memory and thread count are reported.
|
|
- Free disk and the server's load are checked before the run, the output
|
|
deleted after.
|
|
- `GLOSSARY.md` (quiet, silent, recycle), `docs/project.md`'s pipeline,
|
|
`docs/notes.md` (the measures above) and the roadmap match.
|
|
|
|
## Outcome
|
|
|
|
The collector now looks for the exchange failing silently. A connection that
|
|
hears nothing for 30 s reconnects. Each book thread counts every symbol's
|
|
changes per connection and, at the minute, resubscribes a subscription with
|
|
none against three or more on the other; quiet again, its market is reported
|
|
and the orchestrator moves it. The orchestrator recycles one connection at a
|
|
time, the one up the longest, each about hourly, and a reconnect's opening
|
|
book of a symbol unchanged for 10 s is judged like a resync's answer.
|
|
|
|
**Two live runs**, each reconciled with every mismatch explained by its log
|
|
line, no orphan Delta:
|
|
|
|
| Run | Recycles | Losses / M changes | Gaps / M | Quiet resubscribed | Reconnect books judged | Peak memory, threads |
|
|
|---|---|---|---|---|---|---|
|
|
| 30 min, before the review's fixes | 107 | 26.5 | 56 | 26 (none quiet again) | 335,488 (5 losses) | 2.46 GB, 204 |
|
|
| 10 min, after them | 19 | 28.4 | 190 | 0 | 69,340 (0 losses) | 1.94 GB, 202 |
|
|
|
|
Sprint 04's two-subscription rounds (10 minutes each) had 32 to 50 losses
|
|
and 150 to 160 gaps per million. Losses are no higher with recycling. The
|
|
gaps are the startup subscribes' (3,111 of them by minute 10, 3,275 by
|
|
minute 30), so their rate says how long the run was, not what recycling
|
|
costs: about 160 more in 20 minutes of recycling. No connection fell silent
|
|
and no market was reported quiet twice in either run: the tests carry both,
|
|
and the silent close's count. Memory grows with the run's length (the last
|
|
128 changes of each symbol fill up); at 10 minutes it is 0.16 GB above
|
|
sprint 04's, not looked into further.
|
|
|
|
**The 5 reconnect losses of the first run** are doubtful: 4 came at
|
|
17:00:04 with book times 0.3 to 2 s old. The review then found that a book
|
|
lagging further back than the changes we remember was taken for a loss, and
|
|
the rule "no newer than our last change is behind" was added; the second run
|
|
judged 69,340 books and found none.
|
|
|
|
**Fixed at the close**, from the code review, each with a test that fails
|
|
without it: a reconnect book older than our last change, unreadable, or
|
|
failing its own hash now leaves a verified symbol alone (the first replaced
|
|
a correct book, the last made the symbol unverifiable for good); no recycle
|
|
for one interval after a connection came up (an overdue one fired before
|
|
the returning connection's books); a market moved in the last 10 minutes is
|
|
not moved again for being quiet (it ended the move under way). The tests
|
|
were strengthened after the reviewer removed rules and they stayed green:
|
|
twelve rules were then removed one by one, each caught by a named test.
|
|
|
|
**Diverged from the plan**, each in Decisions or its section: the
|
|
orchestrator hears down and silent as well as up; `Resync` the command
|
|
became `Resubscribe`; the minute line gives one total of changes received;
|
|
the three rules above.
|
|
|
|
**Left out**, and where it went: telling a change still in flight from a
|
|
symbol quiet on both connections when a reconnect book is newer than ours
|
|
(Possible features); the dip against a symbol's own past rate, the
|
|
replacement connection opened first, per-second rates (not planned, see
|
|
Not in this sprint); acting on delay (M7). Not covered by a test: the
|
|
orchestrator's loop (sending the Recycle, counting the notices), a quiet
|
|
judging skipped for a symbol being resynced, and `reconcile` pairing a
|
|
reconnect's loss end to end (its cause line's parsing is tested, and the
|
|
first live run paired five).
|
|
|
|
### Retrospective
|
|
|
|
- **Went well:** the measurement before planning overturned a rule on paper
|
|
(reconnect books lag by minutes) instead of in a live run. The reviewer's
|
|
mutation check, removing a rule and running the tests, showed which tests
|
|
proved nothing; redone here after the fixes, it took minutes.
|
|
- **Went wrong:** the first tests were written to pass, not to fail without
|
|
their rule: six rules were uncovered. The reconnect rule leaned on a
|
|
memory of 128 changes without asking what happens past it. The first live
|
|
run was launched with a tool the server doesn't have (`/usr/bin/time`),
|
|
and I announced its end time wrongly.
|
|
- **Tested enough?** The rules are, by mutation. The orchestrator's loop is
|
|
still untested, as in sprint 04, and neither silent connections nor quiet
|
|
moves happened live.
|
|
- **Next time:** for each rule, write the test, then remove the rule and see
|
|
it fail, before the review does. Ask of every "we remember N" what happens
|
|
at N + 1.
|