feat(polymarket): subscription monitoring and connection recycling

The exchange can fail without telling us, and first one wins hides a
subscription gone quiet on one of a symbol's two connections.

- A connection that receives nothing for 30 s, not even the keepalive's
  answer, reconnects.
- Each book thread counts a symbol's changes per connection every minute:
  a subscription with none against three or more on the other is
  resubscribed, and moved to another connection if it stays quiet.
- The orchestrator recycles one connection at a time, each about hourly,
  and never while another connection is down or just back.
- A reconnect's opening book of a symbol unchanged for 10 s is judged like
  a resync's answer: one newer than ours that differs shows a symbol quiet
  on both connections, written as a Snapshot and logged as a loss.
This commit is contained in:
pascal committed 2026-11-11 17:42:27 +00:00
1 parent 02747b882f
commit 480ee3ad2d
10 files changed
+1346 -102

No files matched your search

+301
View File
@@ -0,0 +1,301 @@
# 05. Subscription monitoring
**Status: done (2026-11-11)**
The exchange can fail without telling us: a connection stays open and sends
nothing, a symbol's subscription goes quiet on a connection that still works,
and a connection accumulates the exchange's own bugs. First one wins hides a
quiet subscription as long as the symbol's other connection delivers, so it
has to be looked for. This sprint finds a silent connection and reconnects
it, finds a subscription quiet on one connection and resubscribes it (then
moves it), and recycles every connection about once an hour; the recycle's
opening books also show the symbols quiet on both connections. This is
Extension 2 of the brief. Design: [../docs/project.md](../docs/project.md)
(pipeline), [ADR 0003](../docs/adr/0003-first-one-wins.md),
[../docs/notes.md](../docs/notes.md), [../GLOSSARY.md](../GLOSSARY.md)
(quiet, silent, recycle).
## Measured before planning (2026-11-11)
A throwaway build, 25 minutes live (~198k markets, ~180 connections, the
server loaded: 14 to 45 on 12 cores), with a forced recycle every 60 s:
- **Quiet subscriptions are rare.** Of ~392,000 symbols judged each minute,
~75,000 changed. In 21 minutes, one market once had 3 or more changes on
one connection and none on the other (not the next minute); none had 10.
One change against none came 0 to 16 times a minute: a copy landing in
the next minute. Both tokens of a market go one-sided together.
- **A reconnect's opening book can lag by minutes.** For symbols unchanged
for 10 s or more, 514 of 19,379 reconnect books differed from ours; 512 of
them equal one of our earlier states (1 to 4 levels, some over 10 minutes
old), never with a newer timestamp. 2 (one market) were neither. All
42,096 books of symbols never changed since their first book matched.
Under 10 s, some books ran ahead of our stream (2 between 1 and 10 s).
- **A recycle is cheap**: 1.2 s from close to up; all opening books 0.3 s
later on most connections, 20 s on one, and 2 of 4,490 never on another.
No 1013 close in the run; chains broken and losses per minute the same
during recycles as before.
## A silent connection
A connection that has received nothing, data or `PONG`, for 30 s (three
keepalives) closes itself and reconnects like a dropped one, telling the book
threads it went down. Today nothing notices: the read loop
(`socket.rs:222`) times out every 200 ms and carries on, and `PONG` decodes
to no event (`wire.rs:37`).
## A quiet subscription
Each book thread counts, per symbol and per connection, the changes received
each wall-clock minute: every one, whether it is then applied, a duplicate,
held or stale. At the minute, a verified symbol on two connections, both up
the whole minute and subscribed on both since before it began, whose one
connection delivered at least 3 changes and the other none, is
**quiet** on the other:
- the first time, that connection gets a `Resubscribe` (unsubscribe,
subscribe: the command a resync already used, renamed from `Resync` since
here no book is asked for). The symbol stays verified on the other
connection, so the book that comes back is ignored like any unasked one:
no Snapshot, no subscription gap;
- quiet again the next minute on the same connection, the book thread reports
the market to the orchestrator, which moves that subscription with the
existing `Move`, outside the cap of 50 moves a minute, and logs it; a
market moved in the last 10 minutes stays, as for rebalancing.
A symbol with no change on either connection is not judged: the market may
just be still. That case is the recycle's.
## Recycling
The orchestrator recycles, once the startup deal is complete: every 3600/N
seconds (N connections), the connection whose last connect is oldest closes
and reconnects through the path a dropped connection takes (1 s, then its
symbols subscribed in operations of 500). The next recycle waits until that
connection has reported up again, none starts while any connection is down,
nor for one interval after any connection came up (up means its subscribes
are sent; its opening books take 0.3 to 20 s), so a recycle never takes down
a market's two connections together. A connection that dropped on its own
counts as recycled from its reconnect.
For this the orchestrator keeps each connection's command sender (today
`Threads::open` only hands it to the book threads), a `Recycle` command
closes the socket, and the channel that carries 1013 closes today
(`Sender<usize>`) carries a connection's notices: up, down, closed with
1013, closed for silence.
## Reconnect books
Sprint 04 decided that only asked-for books count, because a moving symbol's
books arrive ahead of or behind our stream. A reconnect's opening book for a
symbol unchanged for at least 10 s is an exception: nothing moved, so it is
judged like a resync's answer. Equal to ours or to one of our earlier states
(the server's opening books lag, by minutes: above), ours stands and it
counts as behind; so does any book whose timestamp is no newer than our last
change, since our chain holds up to that change and we only remember 128
earlier states. A book that is unreadable or fails its own hash is ignored.
Otherwise both subscriptions were quiet: it replaces ours,
is written as a Snapshot with a cause line (a loss, its detail saying a
reconnect found it), and the symbol's other connection is resubscribed.
This holds for any reconnect, recycled or dropped.
## Data model
- **Book thread**, per symbol: the changes received this minute on each of
its connections, and since when it is subscribed on each; whether it was
quiet last minute, and on which connection. Per connection, when it last
came up.
- **Book thread → orchestrator**: the load report, or a market quiet twice on
a connection.
- **Connection → orchestrator**: up, down, a 1013 close, a close for silence.
- **Orchestrator**, per connection: when it last connected; the recycle under
way.
- **Cause lines**: a reconnect book that differs is a `loss`, with a detail
naming the reconnect, so `reconcile` pairs it and the log tells them apart.
## The minute lines
Each book thread's adds: the changes received this minute (all connections
together: one figure per connection would put ~180 in each line), quiet
subscriptions resubscribed and reported, reconnect books judged and found
differing (those behind count with the resyncs' "full books behind ours"). The orchestrator's adds: connections recycled, quiet moves, and
silent connections closed. Totals only: a per-symbol table would be ~400k
lines a minute.
## Decisions
- **Changes received per symbol and connection, per minute** are the
activity measure. *Rejected:* bytes (the load is per market, averaged over
its connections); changes applied (the second connection applies almost
none, by design).
- **Quiet means one connection delivered and the other nothing**; a dip
against the symbol's own past rate is not used. Prediction markets are
bursty, so a dip is a weak signal, while the other connection, or a fresh
book, is a real test. *Rejected:* the brief's literal per-symbol rate
threshold.
- **Resubscribe once, then move.** *Rejected:* resubscribing every minute
forever (never leaves a connection broken for that symbol).
- **The book thread reports a market quiet twice; the orchestrator moves
it.** Only the orchestrator knows which connections have room.
*Rejected:* the book thread choosing a target.
- **The orchestrator recycles one connection at a time, oldest first, about
hourly, starting after the startup deal.** *Rejected:* each connection
recycling itself at an age of an hour plus jitter (~600 markets an hour
would have both connections down at once); waiting an hour before the
first recycle (a 30-minute run would show none).
- **A recycle is a plain close and reconnect.** M4's second connection
covers the seconds it takes. *Rejected:* opening the replacement before
closing (a third connection per symbol and a handover).
- **No recycle while any connection is down.** Covers every market with no
per-market lookup. *Rejected:* checking each market's other connection.
- **A reconnect's opening book is compared for a long-unchanged symbol, on
every reconnect** (amends sprint 04's "only asked-for books count", whose
reason, books ahead of or behind a moving stream, doesn't apply to a
symbol that hasn't moved). *Rejected:* comparing only after a recycle (a
dropped connection's reconnect is the same check).
- **Quiet at 3 changes against none.** Crossed once in 21 live minutes; a
needless resubscribe costs no Snapshot or gap. *Rejected:* 1 (up to 16
needless resubscribes a minute, from copies landing in the next minute);
10 (a slowly trading market's quiet subscription goes unseen).
- **A reconnect book is judged by the resync's rule, from 10 s unchanged**
(supersedes "it must equal ours", because 2.7% of them equal an earlier
state of ours: the server's books lag by minutes). *Rejected:* exact
equality (~500 false losses per 15 recycles); under 10 s (books run ahead
of a moving stream).
- **The orchestrator hears down and silent too** (supersedes "a 1013 close,
or up", because "no recycle while any connection is down" needs down, and
the minute line counts the silent closes).
- **A reconnect book no newer than our last change is behind** (added at the
code review: a book lagging further back than the 128 changes we remember
was taken for a loss and replaced a correct book). *Rejected:* remembering
more states.
- **No recycle for one interval after any connection came up** (added at the
code review: an overdue recycle fired the moment a connection reported up,
before its opening books).
- **A differing reconnect book is a loss**, its cause line saying how it was
found. *Rejected:* a new cause (the changes are ones no connection
delivered: that is a loss).
- **A silent connection is closed after 30 s with nothing received**, one
constant.
## Tests
No network, except the local websocket server of sprint 03's reconnect test.
- **Quiet**: the two-connection recording, one symbol's changes removed from
one connection for a minute: that connection is resubscribed and no
Snapshot is written; removed again the next minute: the market is reported.
A subscription younger than the minute is not judged; a symbol with no
change on either connection is not quiet.
- **Reconnect books**: after an up marker, an opening book equal to ours for
a long-unchanged symbol changes nothing; one that differs is written as a
Snapshot with its cause line and resubscribes the other connection; a book
for a recently changed symbol is ignored as before.
- **Recycle** (on the `Tracker`): the oldest connection first, the next only
after the previous is up, none while a connection is down, paced at 3600/N.
- **A quiet market reported** is moved, outside the cap.
- **Silent**: on the local websocket server, a connection that hears nothing
for the timeout (shortened in the test) reconnects with its symbols.
- **Reconcile**: a reconnect book's cause line pairs with its Snapshot.
## Not in this sprint
- A dip against a symbol's own past rate (see Decisions).
- Opening a replacement connection before closing the recycled one.
- Per-second rates: a minute decides a resubscribe.
- Acting on delay: M7.
- Injecting a fault into a live run: the tests carry the rules.
## Done when
- `scripts/ci.sh` is green.
- A 30-minute live run with recycling on (about 90 connections recycled at
~178 connections):
- every `reconcile` mismatch is explained by the log;
- the minute lines show recycles, quiet subscriptions and reconnect books,
and their counts are reported (none quiet in the run: the tests carry it,
and the Outcome says so);
- losses and subscription gaps per million changes are no higher than
sprint 04's two-subscription rounds (32 to 50 losses, 150 to 160 gaps);
- peak memory and thread count are reported.
- Free disk and the server's load are checked before the run, the output
deleted after.
- `GLOSSARY.md` (quiet, silent, recycle), `docs/project.md`'s pipeline,
`docs/notes.md` (the measures above) and the roadmap match.
## Outcome
The collector now looks for the exchange failing silently. A connection that
hears nothing for 30 s reconnects. Each book thread counts every symbol's
changes per connection and, at the minute, resubscribes a subscription with
none against three or more on the other; quiet again, its market is reported
and the orchestrator moves it. The orchestrator recycles one connection at a
time, the one up the longest, each about hourly, and a reconnect's opening
book of a symbol unchanged for 10 s is judged like a resync's answer.
**Two live runs**, each reconciled with every mismatch explained by its log
line, no orphan Delta:
| Run | Recycles | Losses / M changes | Gaps / M | Quiet resubscribed | Reconnect books judged | Peak memory, threads |
|---|---|---|---|---|---|---|
| 30 min, before the review's fixes | 107 | 26.5 | 56 | 26 (none quiet again) | 335,488 (5 losses) | 2.46 GB, 204 |
| 10 min, after them | 19 | 28.4 | 190 | 0 | 69,340 (0 losses) | 1.94 GB, 202 |
Sprint 04's two-subscription rounds (10 minutes each) had 32 to 50 losses
and 150 to 160 gaps per million. Losses are no higher with recycling. The
gaps are the startup subscribes' (3,111 of them by minute 10, 3,275 by
minute 30), so their rate says how long the run was, not what recycling
costs: about 160 more in 20 minutes of recycling. No connection fell silent
and no market was reported quiet twice in either run: the tests carry both,
and the silent close's count. Memory grows with the run's length (the last
128 changes of each symbol fill up); at 10 minutes it is 0.16 GB above
sprint 04's, not looked into further.
**The 5 reconnect losses of the first run** are doubtful: 4 came at
17:00:04 with book times 0.3 to 2 s old. The review then found that a book
lagging further back than the changes we remember was taken for a loss, and
the rule "no newer than our last change is behind" was added; the second run
judged 69,340 books and found none.
**Fixed at the close**, from the code review, each with a test that fails
without it: a reconnect book older than our last change, unreadable, or
failing its own hash now leaves a verified symbol alone (the first replaced
a correct book, the last made the symbol unverifiable for good); no recycle
for one interval after a connection came up (an overdue one fired before
the returning connection's books); a market moved in the last 10 minutes is
not moved again for being quiet (it ended the move under way). The tests
were strengthened after the reviewer removed rules and they stayed green:
twelve rules were then removed one by one, each caught by a named test.
**Diverged from the plan**, each in Decisions or its section: the
orchestrator hears down and silent as well as up; `Resync` the command
became `Resubscribe`; the minute line gives one total of changes received;
the three rules above.
**Left out**, and where it went: telling a change still in flight from a
symbol quiet on both connections when a reconnect book is newer than ours
(Possible features); the dip against a symbol's own past rate, the
replacement connection opened first, per-second rates (not planned, see
Not in this sprint); acting on delay (M7). Not covered by a test: the
orchestrator's loop (sending the Recycle, counting the notices), a quiet
judging skipped for a symbol being resynced, and `reconcile` pairing a
reconnect's loss end to end (its cause line's parsing is tested, and the
first live run paired five).
### Retrospective
- **Went well:** the measurement before planning overturned a rule on paper
(reconnect books lag by minutes) instead of in a live run. The reviewer's
mutation check, removing a rule and running the tests, showed which tests
proved nothing; redone here after the fixes, it took minutes.
- **Went wrong:** the first tests were written to pass, not to fail without
their rule: six rules were uncovered. The reconnect rule leaned on a
memory of 128 changes without asking what happens past it. The first live
run was launched with a tool the server doesn't have (`/usr/bin/time`),
and I announced its end time wrongly.
- **Tested enough?** The rules are, by mutation. The orchestrator's loop is
still untested, as in sprint 04, and neither silent connections nor quiet
moves happened live.
- **Next time:** for each rule, write the test, then remove the rule and see
it fail, before the review does. Ask of every "we remember N" what happens
at N + 1.