feat(polymarket): subscription monitoring and connection recycling

The exchange can fail without telling us, and first one wins hides a
subscription gone quiet on one of a symbol's two connections.

- A connection that receives nothing for 30 s, not even the keepalive's
  answer, reconnects.
- Each book thread counts a symbol's changes per connection every minute:
  a subscription with none against three or more on the other is
  resubscribed, and moved to another connection if it stays quiet.
- The orchestrator recycles one connection at a time, each about hourly,
  and never while another connection is down or just back.
- A reconnect's opening book of a symbol unchanged for 10 s is judged like
  a resync's answer: one newer than ours that differs shows a symbol quiet
  on both connections, written as a Snapshot and logged as a loss.
This commit is contained in:
pascal committed 2026-11-11 17:42:27 +00:00
1 parent 02747b882f
commit 480ee3ad2d
10 files changed
+1346 -102

No files matched your search

+9 -5
View File
@@ -18,13 +18,14 @@ outside this repo.
loss detection, message shapes). Read before touching exchange parsing.
- `GLOSSARY.md`: the terms (symbol, connection, load, first one wins,
duplicate, held, rescued, follow, resync, loss, subscription gap, split,
Snapshot, Delta).
quiet, resubscribe, silent, recycle, Snapshot, Delta).
- `src/polymarket/orchestrator.rs`: what is tracked, which connections carry
each market, and their balancing; `src/polymarket/gamma.rs`: the fetchers
each market, their balancing and their recycling; `src/polymarket/gamma.rs`: the fetchers
(discovery); `src/polymarket/socket.rs`: a connection (read, stamp, decode,
route); `src/polymarket/wire.rs`: the exchange's messages;
`src/polymarket/feed.rs`: a book thread and its loss rules (the subtle
part); `src/book.rs`: the book and fixed point; `src/reconcile.rs` +
route; reconnects when dropped, silent or recycled);
`src/polymarket/wire.rs`: the exchange's messages;
`src/polymarket/feed.rs`: a book thread, its loss rules (the subtle
part) and its watch for quiet subscriptions; `src/book.rs`: the book and fixed point; `src/reconcile.rs` +
`src/bin/reconcile.rs`: the output check, reading the log's cause lines
(`src/cause.rs`). `tests/fixtures/`: recordings of the live feed, on one and two
connections, that the tests replay.
@@ -73,6 +74,9 @@ and let the user settle it. Claude owns the data model.
"capacity scan"` matches the waiting shell's own command line and never
ends, and `pkill -f PATTERN` can kill the shell running it. Take the PID
(`pgrep -f '^./target/release/...'`, anchored) and loop on `kill -0 $pid`.
- **No `/usr/bin/time` on the server**: for a run's peak memory and thread
count, read `VmHWM` and `Threads` from `/proc/<pid>/status` while it runs
(sprint 05's run script sampled them every 30 s).
- **The server is shared and gets overloaded** (load up to 92 on 12 cores,
2026-11-11): a starved collector falls behind and gets closed with 1013.
Check `cat /proc/loadavg` before a live run whose numbers are compared, and
+21
View File
@@ -71,6 +71,27 @@ verified and until a fresh book says where it stands.
**Split**:
Moving half the markets off a connection the exchange keeps closing.
**Quiet**:
Said of a symbol's subscription on a connection that is up, while the
exchange has stopped sending that symbol's changes there. Nothing tells us:
it has to be looked for.
_Avoid_: Stale, dead (a connection can be fine while one of its
subscriptions is quiet)
**Resubscribe**:
To unsubscribe a symbol on one connection and subscribe it again there, which
makes the exchange start its subscription afresh. A resync is a resubscribe
whose book we asked for.
**Silent**:
Said of a connection that is open but delivers nothing at all, not even the
answer to its keepalive.
**Recycle**:
To close a working connection on purpose and reconnect it with the same
symbols, to shed what has gone wrong on the exchange's side of it.
_Avoid_: Restart, rotate
**Receive time**:
When a message came off the socket, by the local clock.
_Avoid_: Arrival time, local timestamp
+21
View File
@@ -173,6 +173,27 @@ From a 60 s run of the M1 spike on one market, both tokens subscribed, and the
(0 of 660,084 changes on each of the two):
one older than the last applied is a late copy.
## Quiet subscriptions and reconnects (observed 2026-11-11)
From a 25-minute live run on every live market, two connections each, one
connection recycled every 60 s (sprint 05).
- **Quiet subscriptions are rare**: in 21 minutes, one market once had 3 or
more changes on one connection and none on the other; none had 10. Both
tokens of a market go one-sided together.
- **A reconnect's opening book can lag by minutes**: for symbols unchanged
for 10 s or more, 514 of 19,379 differed from ours, 512 of them equal to
one of our earlier states (1 to 4 levels behind, some over 10 minutes),
none with a newer timestamp. Books of symbols never changed all matched
(42,096). Under 10 s, some ran ahead of our stream.
- **With the monitoring on** (30 minutes, 107 recycles): 26 subscriptions
resubscribed for being quiet, none quiet again the next minute; of 335,488
reconnect books of unchanged symbols, 7,240 fresh books in all were behind
ours; no connection silent for 30 s.
- **A reconnect takes ~1.2 s** from close to up (1 s of it our wait); the
opening books of ~4,500 tokens come 0.3 s later on most connections, 20 s
on one; 2 of 4,490 never came on another.
## Gamma, the market listing (observed 2026-11-04)
- **Size:** 238,308 markets `active` and not `closed`; 238,201 of them with
+10 -8
View File
@@ -135,14 +135,15 @@ and drop from the global workflow for a job-challenge repo.
## Pipeline
The primary thing we debate sprint by sprint (this project's equivalent of the
schema debate). As settled in sprint 04:
schema debate). As settled in sprints 04 and 05:
```
rescan fetcher ──findings──▶ ┌──────────────┐ ◀──findings── recent fetcher
│ orchestrator │ ◀── 1013 closes ── connections
└──────┬───▲───┘
│ orchestrator │ ◀── up, down, 1013 ─── connections
└──────┬───▲───┘ ──── recycle ────────▶ connections
Add / Move / Remove (a market and │ │ load reports, each minute
its connections), to its thread │ │ (bytes/s per market)
its connections), to its thread │ │ (bytes/s per market); markets
│ │ quiet twice on a connection
┌──────▼───┴───┐
connection 0..N ──events───▶ │ book threads │ ──Snapshot | Delta──▶ writer
(read, stamp, parse, route) │ 0..15 │
@@ -151,15 +152,16 @@ schema debate). As settled in sprint 04:
```
- The **orchestrator** owns what is tracked (each market's two connections,
the timers) and decides: placing, moving, splitting. It never blocks on
HTTP.
the timers) and decides: placing, moving, splitting, and recycling one
connection at a time, about hourly. It never blocks on HTTP.
- The two **fetchers** only page Gamma on request and stream what they read.
- A **connection** reads, stamps the receive time, parses, and routes each
event to the book thread of its market. Every symbol is on two connections,
chosen independently.
chosen independently. One silent for 30 s reconnects.
- A **book thread** owns its markets' books and applies first one wins: a
change is a duplicate, the next one, or held until it fits (fold, check,
resync, Snapshots as before).
resync, Snapshots as before). It resubscribes a subscription gone quiet on
one connection, and judges a reconnect's books of unchanged symbols.
- One **writer** owns the output.
Settled: OS threads, shared-nothing, no mutexes
+8 -1
View File
@@ -55,12 +55,16 @@ abstraction is *extracted* once a second exchange exists, not designed up front.
Result: losses a quarter lower in each pair of alternating runs; losses and
gaps together lower in one pair, even in the other (sprint 04's Outcome).
5. **Extension 2 — subscription monitoring** — *not started*
5. **Extension 2 — subscription monitoring** — *done (2026-11-11)*
Per-symbol activity monitoring (data points per second/minute); unsubscribe +
resubscribe symbols that go too quiet (with two connections, first one wins
hides a subscription gone quiet on one of them: it has to be looked for); periodic full-socket recycle
(disconnect → reconnect → resubscribe) to shed cumulative exchange-side bugs.
Sprint: [../roadmap/05. subscription monitoring.md](../roadmap/05.%20subscription%20monitoring.md).
*Ends:* monitoring + recycling running.
Result: a 30-minute run recycled 107 connections with losses no higher
than sprint 04's; quiet subscriptions are rare (26 resubscribed, none
quiet again) and no connection fell silent (sprint 05's Outcome).
6. **Extension 3 — generic core + Kalshi adapter** — *not started*
Extract the exchange-agnostic core (the `NormalizedEvent` seam) from the
@@ -115,6 +119,9 @@ abstraction is *extracted* once a second exchange exists, not designed up front.
book, when the first one's chain breaks before any change verified.
- An exact rescued count: a change remembered past the window of 128 until
its copies are counted (sprint 04 reports an upper bound).
- Telling a change still in flight from a symbol quiet on both connections,
when a reconnect's book is newer than ours (`maybe`: sprint 05 logs both
as a loss; the Snapshot is right either way).
- Fewer Snapshots for unchanged books (the brief asks for every symbol every
minute: ~130 MB a minute at full coverage).
- Re-reading a market's `orderMinSize` / `negRisk` if they can change.
+301
View File
@@ -0,0 +1,301 @@
# 05. Subscription monitoring
**Status: done (2026-11-11)**
The exchange can fail without telling us: a connection stays open and sends
nothing, a symbol's subscription goes quiet on a connection that still works,
and a connection accumulates the exchange's own bugs. First one wins hides a
quiet subscription as long as the symbol's other connection delivers, so it
has to be looked for. This sprint finds a silent connection and reconnects
it, finds a subscription quiet on one connection and resubscribes it (then
moves it), and recycles every connection about once an hour; the recycle's
opening books also show the symbols quiet on both connections. This is
Extension 2 of the brief. Design: [../docs/project.md](../docs/project.md)
(pipeline), [ADR 0003](../docs/adr/0003-first-one-wins.md),
[../docs/notes.md](../docs/notes.md), [../GLOSSARY.md](../GLOSSARY.md)
(quiet, silent, recycle).
## Measured before planning (2026-11-11)
A throwaway build, 25 minutes live (~198k markets, ~180 connections, the
server loaded: 14 to 45 on 12 cores), with a forced recycle every 60 s:
- **Quiet subscriptions are rare.** Of ~392,000 symbols judged each minute,
~75,000 changed. In 21 minutes, one market once had 3 or more changes on
one connection and none on the other (not the next minute); none had 10.
One change against none came 0 to 16 times a minute: a copy landing in
the next minute. Both tokens of a market go one-sided together.
- **A reconnect's opening book can lag by minutes.** For symbols unchanged
for 10 s or more, 514 of 19,379 reconnect books differed from ours; 512 of
them equal one of our earlier states (1 to 4 levels, some over 10 minutes
old), never with a newer timestamp. 2 (one market) were neither. All
42,096 books of symbols never changed since their first book matched.
Under 10 s, some books ran ahead of our stream (2 between 1 and 10 s).
- **A recycle is cheap**: 1.2 s from close to up; all opening books 0.3 s
later on most connections, 20 s on one, and 2 of 4,490 never on another.
No 1013 close in the run; chains broken and losses per minute the same
during recycles as before.
## A silent connection
A connection that has received nothing, data or `PONG`, for 30 s (three
keepalives) closes itself and reconnects like a dropped one, telling the book
threads it went down. Today nothing notices: the read loop
(`socket.rs:222`) times out every 200 ms and carries on, and `PONG` decodes
to no event (`wire.rs:37`).
## A quiet subscription
Each book thread counts, per symbol and per connection, the changes received
each wall-clock minute: every one, whether it is then applied, a duplicate,
held or stale. At the minute, a verified symbol on two connections, both up
the whole minute and subscribed on both since before it began, whose one
connection delivered at least 3 changes and the other none, is
**quiet** on the other:
- the first time, that connection gets a `Resubscribe` (unsubscribe,
subscribe: the command a resync already used, renamed from `Resync` since
here no book is asked for). The symbol stays verified on the other
connection, so the book that comes back is ignored like any unasked one:
no Snapshot, no subscription gap;
- quiet again the next minute on the same connection, the book thread reports
the market to the orchestrator, which moves that subscription with the
existing `Move`, outside the cap of 50 moves a minute, and logs it; a
market moved in the last 10 minutes stays, as for rebalancing.
A symbol with no change on either connection is not judged: the market may
just be still. That case is the recycle's.
## Recycling
The orchestrator recycles, once the startup deal is complete: every 3600/N
seconds (N connections), the connection whose last connect is oldest closes
and reconnects through the path a dropped connection takes (1 s, then its
symbols subscribed in operations of 500). The next recycle waits until that
connection has reported up again, none starts while any connection is down,
nor for one interval after any connection came up (up means its subscribes
are sent; its opening books take 0.3 to 20 s), so a recycle never takes down
a market's two connections together. A connection that dropped on its own
counts as recycled from its reconnect.
For this the orchestrator keeps each connection's command sender (today
`Threads::open` only hands it to the book threads), a `Recycle` command
closes the socket, and the channel that carries 1013 closes today
(`Sender<usize>`) carries a connection's notices: up, down, closed with
1013, closed for silence.
## Reconnect books
Sprint 04 decided that only asked-for books count, because a moving symbol's
books arrive ahead of or behind our stream. A reconnect's opening book for a
symbol unchanged for at least 10 s is an exception: nothing moved, so it is
judged like a resync's answer. Equal to ours or to one of our earlier states
(the server's opening books lag, by minutes: above), ours stands and it
counts as behind; so does any book whose timestamp is no newer than our last
change, since our chain holds up to that change and we only remember 128
earlier states. A book that is unreadable or fails its own hash is ignored.
Otherwise both subscriptions were quiet: it replaces ours,
is written as a Snapshot with a cause line (a loss, its detail saying a
reconnect found it), and the symbol's other connection is resubscribed.
This holds for any reconnect, recycled or dropped.
## Data model
- **Book thread**, per symbol: the changes received this minute on each of
its connections, and since when it is subscribed on each; whether it was
quiet last minute, and on which connection. Per connection, when it last
came up.
- **Book thread → orchestrator**: the load report, or a market quiet twice on
a connection.
- **Connection → orchestrator**: up, down, a 1013 close, a close for silence.
- **Orchestrator**, per connection: when it last connected; the recycle under
way.
- **Cause lines**: a reconnect book that differs is a `loss`, with a detail
naming the reconnect, so `reconcile` pairs it and the log tells them apart.
## The minute lines
Each book thread's adds: the changes received this minute (all connections
together: one figure per connection would put ~180 in each line), quiet
subscriptions resubscribed and reported, reconnect books judged and found
differing (those behind count with the resyncs' "full books behind ours"). The orchestrator's adds: connections recycled, quiet moves, and
silent connections closed. Totals only: a per-symbol table would be ~400k
lines a minute.
## Decisions
- **Changes received per symbol and connection, per minute** are the
activity measure. *Rejected:* bytes (the load is per market, averaged over
its connections); changes applied (the second connection applies almost
none, by design).
- **Quiet means one connection delivered and the other nothing**; a dip
against the symbol's own past rate is not used. Prediction markets are
bursty, so a dip is a weak signal, while the other connection, or a fresh
book, is a real test. *Rejected:* the brief's literal per-symbol rate
threshold.
- **Resubscribe once, then move.** *Rejected:* resubscribing every minute
forever (never leaves a connection broken for that symbol).
- **The book thread reports a market quiet twice; the orchestrator moves
it.** Only the orchestrator knows which connections have room.
*Rejected:* the book thread choosing a target.
- **The orchestrator recycles one connection at a time, oldest first, about
hourly, starting after the startup deal.** *Rejected:* each connection
recycling itself at an age of an hour plus jitter (~600 markets an hour
would have both connections down at once); waiting an hour before the
first recycle (a 30-minute run would show none).
- **A recycle is a plain close and reconnect.** M4's second connection
covers the seconds it takes. *Rejected:* opening the replacement before
closing (a third connection per symbol and a handover).
- **No recycle while any connection is down.** Covers every market with no
per-market lookup. *Rejected:* checking each market's other connection.
- **A reconnect's opening book is compared for a long-unchanged symbol, on
every reconnect** (amends sprint 04's "only asked-for books count", whose
reason, books ahead of or behind a moving stream, doesn't apply to a
symbol that hasn't moved). *Rejected:* comparing only after a recycle (a
dropped connection's reconnect is the same check).
- **Quiet at 3 changes against none.** Crossed once in 21 live minutes; a
needless resubscribe costs no Snapshot or gap. *Rejected:* 1 (up to 16
needless resubscribes a minute, from copies landing in the next minute);
10 (a slowly trading market's quiet subscription goes unseen).
- **A reconnect book is judged by the resync's rule, from 10 s unchanged**
(supersedes "it must equal ours", because 2.7% of them equal an earlier
state of ours: the server's books lag by minutes). *Rejected:* exact
equality (~500 false losses per 15 recycles); under 10 s (books run ahead
of a moving stream).
- **The orchestrator hears down and silent too** (supersedes "a 1013 close,
or up", because "no recycle while any connection is down" needs down, and
the minute line counts the silent closes).
- **A reconnect book no newer than our last change is behind** (added at the
code review: a book lagging further back than the 128 changes we remember
was taken for a loss and replaced a correct book). *Rejected:* remembering
more states.
- **No recycle for one interval after any connection came up** (added at the
code review: an overdue recycle fired the moment a connection reported up,
before its opening books).
- **A differing reconnect book is a loss**, its cause line saying how it was
found. *Rejected:* a new cause (the changes are ones no connection
delivered: that is a loss).
- **A silent connection is closed after 30 s with nothing received**, one
constant.
## Tests
No network, except the local websocket server of sprint 03's reconnect test.
- **Quiet**: the two-connection recording, one symbol's changes removed from
one connection for a minute: that connection is resubscribed and no
Snapshot is written; removed again the next minute: the market is reported.
A subscription younger than the minute is not judged; a symbol with no
change on either connection is not quiet.
- **Reconnect books**: after an up marker, an opening book equal to ours for
a long-unchanged symbol changes nothing; one that differs is written as a
Snapshot with its cause line and resubscribes the other connection; a book
for a recently changed symbol is ignored as before.
- **Recycle** (on the `Tracker`): the oldest connection first, the next only
after the previous is up, none while a connection is down, paced at 3600/N.
- **A quiet market reported** is moved, outside the cap.
- **Silent**: on the local websocket server, a connection that hears nothing
for the timeout (shortened in the test) reconnects with its symbols.
- **Reconcile**: a reconnect book's cause line pairs with its Snapshot.
## Not in this sprint
- A dip against a symbol's own past rate (see Decisions).
- Opening a replacement connection before closing the recycled one.
- Per-second rates: a minute decides a resubscribe.
- Acting on delay: M7.
- Injecting a fault into a live run: the tests carry the rules.
## Done when
- `scripts/ci.sh` is green.
- A 30-minute live run with recycling on (about 90 connections recycled at
~178 connections):
- every `reconcile` mismatch is explained by the log;
- the minute lines show recycles, quiet subscriptions and reconnect books,
and their counts are reported (none quiet in the run: the tests carry it,
and the Outcome says so);
- losses and subscription gaps per million changes are no higher than
sprint 04's two-subscription rounds (32 to 50 losses, 150 to 160 gaps);
- peak memory and thread count are reported.
- Free disk and the server's load are checked before the run, the output
deleted after.
- `GLOSSARY.md` (quiet, silent, recycle), `docs/project.md`'s pipeline,
`docs/notes.md` (the measures above) and the roadmap match.
## Outcome
The collector now looks for the exchange failing silently. A connection that
hears nothing for 30 s reconnects. Each book thread counts every symbol's
changes per connection and, at the minute, resubscribes a subscription with
none against three or more on the other; quiet again, its market is reported
and the orchestrator moves it. The orchestrator recycles one connection at a
time, the one up the longest, each about hourly, and a reconnect's opening
book of a symbol unchanged for 10 s is judged like a resync's answer.
**Two live runs**, each reconciled with every mismatch explained by its log
line, no orphan Delta:
| Run | Recycles | Losses / M changes | Gaps / M | Quiet resubscribed | Reconnect books judged | Peak memory, threads |
|---|---|---|---|---|---|---|
| 30 min, before the review's fixes | 107 | 26.5 | 56 | 26 (none quiet again) | 335,488 (5 losses) | 2.46 GB, 204 |
| 10 min, after them | 19 | 28.4 | 190 | 0 | 69,340 (0 losses) | 1.94 GB, 202 |
Sprint 04's two-subscription rounds (10 minutes each) had 32 to 50 losses
and 150 to 160 gaps per million. Losses are no higher with recycling. The
gaps are the startup subscribes' (3,111 of them by minute 10, 3,275 by
minute 30), so their rate says how long the run was, not what recycling
costs: about 160 more in 20 minutes of recycling. No connection fell silent
and no market was reported quiet twice in either run: the tests carry both,
and the silent close's count. Memory grows with the run's length (the last
128 changes of each symbol fill up); at 10 minutes it is 0.16 GB above
sprint 04's, not looked into further.
**The 5 reconnect losses of the first run** are doubtful: 4 came at
17:00:04 with book times 0.3 to 2 s old. The review then found that a book
lagging further back than the changes we remember was taken for a loss, and
the rule "no newer than our last change is behind" was added; the second run
judged 69,340 books and found none.
**Fixed at the close**, from the code review, each with a test that fails
without it: a reconnect book older than our last change, unreadable, or
failing its own hash now leaves a verified symbol alone (the first replaced
a correct book, the last made the symbol unverifiable for good); no recycle
for one interval after a connection came up (an overdue one fired before
the returning connection's books); a market moved in the last 10 minutes is
not moved again for being quiet (it ended the move under way). The tests
were strengthened after the reviewer removed rules and they stayed green:
twelve rules were then removed one by one, each caught by a named test.
**Diverged from the plan**, each in Decisions or its section: the
orchestrator hears down and silent as well as up; `Resync` the command
became `Resubscribe`; the minute line gives one total of changes received;
the three rules above.
**Left out**, and where it went: telling a change still in flight from a
symbol quiet on both connections when a reconnect book is newer than ours
(Possible features); the dip against a symbol's own past rate, the
replacement connection opened first, per-second rates (not planned, see
Not in this sprint); acting on delay (M7). Not covered by a test: the
orchestrator's loop (sending the Recycle, counting the notices), a quiet
judging skipped for a symbol being resynced, and `reconcile` pairing a
reconnect's loss end to end (its cause line's parsing is tested, and the
first live run paired five).
### Retrospective
- **Went well:** the measurement before planning overturned a rule on paper
(reconnect books lag by minutes) instead of in a live run. The reviewer's
mutation check, removing a rule and running the tests, showed which tests
proved nothing; redone here after the fixes, it took minutes.
- **Went wrong:** the first tests were written to pass, not to fail without
their rule: six rules were uncovered. The reconnect rule leaned on a
memory of 128 changes without asking what happens past it. The first live
run was launched with a tool the server doesn't have (`/usr/bin/time`),
and I announced its end time wrongly.
- **Tested enough?** The rules are, by mutation. The orchestrator's loop is
still untested, as in sprint 04, and neither silent connections nor quiet
moves happened live.
- **Next time:** for each rule, write the test, then remove the rule and see
it fail, before the review does. Ask of every "we remember N" what happens
at N + 1.
+8
View File
@@ -71,6 +71,14 @@ mod tests {
let logged = format!("12:00:00.001 polymarket: {text}");
assert_eq!(parse(&logged), Some(("123".to_string(), at, cause)));
}
// A reconnect's loss says how it was found, after its cause.
let text = line(
"123",
at,
Cause::Loss,
", found by a reconnect (its time 12:00:00.000)",
);
assert_eq!(parse(&text), Some(("123".to_string(), at, Cause::Loss)));
assert_eq!(
parse("12:00:00.001 polymarket: connection 3: connected"),
None
+591 -41
View File
@@ -2,8 +2,11 @@
//! per symbol, first one wins: a change is taken from whichever connection
//! delivers it first, Polymarket's book hash saying where it belongs
//! (ADR 0003). It resyncs when a change fits on no connection in time, and
//! emits Deltas and Snapshots. The rules are in
//! `roadmap/04. redundant connections.md`, the protocol facts and the hash
//! emits Deltas and Snapshots. It also looks for what first one wins hides:
//! a subscription gone quiet on one connection, each minute, and symbols
//! quiet on both, in a reconnect's opening books. The rules are in
//! `roadmap/04. redundant connections.md` and
//! `roadmap/05. subscription monitoring.md`, the protocol facts and the hash
//! formula in `docs/notes.md`, the terms in `GLOSSARY.md`.
use std::collections::{BTreeMap, BTreeSet, HashMap, VecDeque};
@@ -47,15 +50,34 @@ const MAX_HELD: usize = 512;
const TAILS_TRIED: usize = 4;
/// When a change's copies are counted: no second copy came later than 0.7 s.
const COPIES_BY: TimeDelta = TimeDelta::seconds(2);
/// A connection that delivered none of a symbol's changes in a minute while
/// another delivered at least this many is quiet for it (sprint 05: crossed
/// once in 21 live minutes; at 1, copies landing in the next minute would
/// count).
const QUIET_AGAINST: u32 = 3;
/// A reconnect's opening book is judged only for a symbol unchanged this
/// long: a moving symbol's books run ahead of or behind our stream.
const STILL_FOR: TimeDelta = TimeDelta::seconds(10);
/// What the book thread does with an event or a command: write output, ask a
/// connection (by number) for something, or report its markets' loads to the
/// orchestrator.
/// connection (by number) for something, or report to the orchestrator.
#[derive(Debug)]
pub enum Effect {
Write(Output),
Send(usize, Command),
Report(Loads),
Report(Report),
}
/// What a book thread tells the orchestrator.
#[derive(Debug, PartialEq, Eq)]
pub enum Report {
Loads(Loads),
/// The market stayed quiet on the connection after being resubscribed
/// there: its subscription should move.
Quiet {
market: String,
connection: usize,
},
}
/// Each market's load (`conditionId` → bytes per second on one connection),
@@ -130,6 +152,33 @@ impl RecentChange {
}
}
/// A symbol's subscription on one of its connections, as watched for going
/// quiet.
#[derive(Debug, Clone, Copy)]
struct Watch {
connection: usize,
/// Its changes received from this connection this minute: every one,
/// whatever first one wins then made of it.
received: u32,
/// When it was last subscribed there: a minute it didn't cover whole is
/// not judged.
since: DateTime<Utc>,
/// When the connection last delivered a complete book of it: its first
/// since the connection came up is the reconnect's.
book_at: DateTime<Utc>,
}
impl Watch {
fn new(connection: usize, since: DateTime<Utc>) -> Self {
Self {
connection,
received: 0,
since,
book_at: DateTime::UNIX_EPOCH,
}
}
}
/// A change as applied, with the exchange's hash of the book after it.
struct RecentChange {
side: Side,
@@ -172,6 +221,10 @@ struct Symbol {
had_book: bool,
/// The connections it is subscribed on (three while a move overlaps).
connections: Vec<usize>,
/// One per connection in `connections`.
watches: Vec<Watch>,
/// The connection it was quiet on last minute, and resubscribed.
quiet_on: Option<usize>,
/// The receive time of its last Delta: a held change is stamped no earlier,
/// so replaying the output by time keeps its order.
last_written: DateTime<Utc>,
@@ -184,7 +237,7 @@ struct Symbol {
}
impl Symbol {
fn new(market: &Market, connections: &[usize]) -> Self {
fn new(market: &Market, connections: &[usize], now: DateTime<Utc>) -> Self {
Self {
state: State::AwaitingBook,
book: Book::default(),
@@ -196,6 +249,11 @@ impl Symbol {
verified_since_book: false,
had_book: false,
connections: connections.to_vec(),
watches: connections
.iter()
.map(|&connection| Watch::new(connection, now))
.collect(),
quiet_on: None,
last_written: DateTime::UNIX_EPOCH,
market: market.condition_id.clone(),
tick_size: String::new(),
@@ -216,7 +274,39 @@ impl Symbol {
// A switch already asked stays one until its book comes.
let switch = switch || self.asked.is_some_and(|(_, switch)| switch);
self.asked = Some((connection, switch));
effects.push(Effect::Send(connection, Command::Resync(token.to_string())));
effects.push(Effect::Send(
connection,
Command::Resubscribe(token.to_string()),
));
}
fn watch_mut(&mut self, connection: usize) -> Option<&mut Watch> {
self.watches
.iter_mut()
.find(|watch| watch.connection == connection)
}
/// The connection quiet for it over the minute begun at `started`: one
/// that delivered none of its changes while another delivered at least
/// [`QUIET_AGAINST`], both up and subscribed for the whole minute.
fn quiet(&self, started: DateTime<Utc>, up_since: &[Option<DateTime<Utc>>]) -> Option<usize> {
let judged: Vec<&Watch> = self
.watches
.iter()
.filter(|watch| {
watch.since <= started
&& up_since
.get(watch.connection)
.copied()
.flatten()
.is_some_and(|up| up <= started)
})
.collect();
let busiest = judged.iter().map(|watch| watch.received).max()?;
(busiest >= QUIET_AGAINST)
.then(|| judged.iter().find(|watch| watch.received == 0))
.flatten()
.map(|watch| watch.connection)
}
/// One of its connections that is up, other than `except`.
@@ -404,6 +494,15 @@ struct Counts {
subscription_gaps: u64,
losses: u64,
switches: u64,
/// Subscriptions found quiet, each resubscribed.
quiet: u64,
/// Markets quiet again after their resubscribe: reported for a move.
quiet_reported: u64,
/// A reconnect's opening books judged, those of still symbols.
reconnect_books: u64,
/// Of them, books that differed from every state of ours: both
/// subscriptions were quiet.
reconnect_losses: u64,
}
pub struct Feed {
@@ -414,6 +513,13 @@ pub struct Feed {
markets: HashMap<String, Tracked>,
/// By connection number: whether it is up.
up: Vec<bool>,
/// By connection number: since when it is up, if it is.
up_since: Vec<Option<DateTime<Utc>>>,
/// When the minute being counted began: its quiet subscriptions are
/// judged at its end. None before the first.
minute_started: Option<DateTime<Utc>>,
/// Changes received this minute, from all connections.
received_this_minute: u64,
/// The symbols with held changes: their ceilings are checked.
waiting: BTreeSet<String>,
/// The receive time of the last event applied: the time a Snapshot's book
@@ -436,6 +542,9 @@ impl Feed {
symbols: BTreeMap::new(),
markets: HashMap::new(),
up: Vec::new(),
up_since: Vec::new(),
minute_started: None,
received_this_minute: 0,
waiting: BTreeSet::new(),
last_received_at: None,
next_minute_at: None,
@@ -456,7 +565,7 @@ impl Feed {
) -> Vec<Effect> {
for token in &market.tokens {
self.symbols
.insert(token.clone(), Symbol::new(market, connections));
.insert(token.clone(), Symbol::new(market, connections, now));
}
let tracked = Tracked {
tokens: market.tokens.clone(),
@@ -476,7 +585,13 @@ impl Feed {
/// Moves a market's subscription from one connection to another: it
/// subscribes on `to` now, and leaves `from` once `to` delivers a book of
/// the market. The book stays as it is: no Snapshot, no subscription gap.
pub fn move_market(&mut self, market: &Market, from: usize, to: usize) -> Vec<Effect> {
pub fn move_market(
&mut self,
market: &Market,
from: usize,
to: usize,
now: DateTime<Utc>,
) -> Vec<Effect> {
let Some(tracked) = self.markets.get_mut(&market.condition_id) else {
return Vec::new();
};
@@ -493,6 +608,7 @@ impl Feed {
&& !symbol.connections.contains(&to)
{
symbol.connections.push(to);
symbol.watches.push(Watch::new(to, now));
}
}
effects.push(Effect::Send(to, Command::Subscribe(market.tokens.clone())));
@@ -513,6 +629,9 @@ impl Feed {
for token in &tokens {
if let Some(symbol) = self.symbols.get_mut(token) {
symbol.connections.retain(|&on| on != connection);
symbol
.watches
.retain(|watch| watch.connection != connection);
symbol.held.remove(&connection);
Self::lose(
token,
@@ -558,16 +677,25 @@ impl Feed {
match received {
Received::Event(arrival) => self.on_event(arrival),
Received::Up(connection) => {
self.set_up(connection, true);
// Its first events come after this marker; any other
// connection's before it were received earlier.
let at = self.last_received_at.unwrap_or(DateTime::UNIX_EPOCH);
self.set_up(connection, true, at);
Vec::new()
}
Received::Down(connection) => self.on_down(connection),
}
}
fn set_up(&mut self, connection: usize, up: bool) {
fn set_up(&mut self, connection: usize, up: bool, at: DateTime<Utc>) {
if self.up.len() <= connection {
self.up.resize(connection + 1, false);
self.up_since.resize(connection + 1, None);
}
if up && !self.up[connection] {
self.up_since[connection] = Some(at);
} else if !up {
self.up_since[connection] = None;
}
self.up[connection] = up;
}
@@ -576,7 +704,7 @@ impl Feed {
let (received_at, connection) = (arrival.received_at, arrival.connection);
// A connection that delivers is up, its marker possibly still behind.
if !is_up(&self.up, connection) {
self.set_up(connection, true);
self.set_up(connection, true, received_at);
}
// Minute Snapshots due before this event hold the books as they were.
let mut effects = self.tick(received_at);
@@ -631,7 +759,7 @@ impl Feed {
}
fn on_down(&mut self, connection: usize) -> Vec<Effect> {
self.set_up(connection, false);
self.set_up(connection, false, DateTime::UNIX_EPOCH);
let mut effects = Vec::new();
for (token, symbol) in &mut self.symbols {
if !symbol.connections.contains(&connection) {
@@ -692,14 +820,16 @@ impl Feed {
}
/// Once `now` reaches the next wall-clock minute: reports the loads,
/// emits the minute Snapshots, and asks again for the fresh books still
/// not come.
/// emits the minute Snapshots, asks again for the fresh books still not
/// come, and resubscribes (or reports) the subscriptions quiet over the
/// minute that ends.
pub fn tick(&mut self, now: DateTime<Utc>) -> Vec<Effect> {
let next = *self.next_minute_at.get_or_insert_with(|| next_minute(now));
if now < next {
return Vec::new();
}
self.next_minute_at = Some(next_minute(now));
let started = self.minute_started.replace(now);
let mut loads = Loads::new();
for (market, tracked) in &mut self.markets {
let connections = tracked
@@ -711,13 +841,48 @@ impl Feed {
loads.insert(market.clone(), load);
}
}
let mut effects = vec![Effect::Report(loads)];
let mut effects = vec![Effect::Report(Report::Loads(loads))];
let Some(received_at) = self.last_received_at else {
return effects;
};
self.log_minute();
let mut reported = BTreeSet::new();
for (token, symbol) in &mut self.symbols {
symbol.count_old_copies(now, &mut self.counts);
let quiet = started
.filter(|_| symbol.state == State::Verified && symbol.asked.is_none())
.and_then(|started| symbol.quiet(started, &self.up_since));
for watch in &mut symbol.watches {
watch.received = 0;
}
match quiet {
// Resubscribed already, and still quiet: it should move.
Some(connection) if symbol.quiet_on == Some(connection) => {
symbol.quiet_on = None;
if reported.insert((symbol.market.clone(), connection)) {
self.counts.quiet_reported += 1;
effects.push(Effect::Report(Report::Quiet {
market: symbol.market.clone(),
connection,
}));
}
}
// The other connection carries it meanwhile: the book that
// comes back is ignored, no Snapshot, no subscription gap.
Some(connection) => {
log!("polymarket: {token}: quiet on connection {connection}, resubscribed");
self.counts.quiet += 1;
symbol.quiet_on = Some(connection);
if let Some(watch) = symbol.watch_mut(connection) {
watch.since = now;
}
effects.push(Effect::Send(
connection,
Command::Resubscribe(token.clone()),
));
}
None => symbol.quiet_on = None,
}
if symbol.state != State::AwaitingBook {
effects.push(Effect::Write(symbol.book.snapshot(
token,
@@ -759,8 +924,10 @@ impl Feed {
log!(
"polymarket: book thread {}: {} symbols; so far {} changes written ({} verified), {} duplicates, \
{} rescued, {} held dropped, {} chains broken, {} resyncs, {} full books behind ours, \
{} subscription gaps, {} losses, {} switches; second copies {:.1} ms late at the median, \
{:.1} ms at p99; at most {} events waited this minute",
{} subscription gaps, {} losses, {} switches, {} quiet subscriptions resubscribed, \
{} markets reported quiet again, {} reconnect books judged ({} losses); second copies \
{:.1} ms late at the median, {:.1} ms at p99; this minute {} changes received, at most \
{} events waited",
self.number,
self.symbols.len(),
counts.written,
@@ -774,12 +941,18 @@ impl Feed {
counts.subscription_gaps,
counts.losses,
counts.switches,
counts.quiet,
counts.quiet_reported,
counts.reconnect_books,
counts.reconnect_losses,
lag(0.5),
lag(0.99),
self.received_this_minute,
self.peak_waiting
);
self.lags.clear();
self.peak_waiting = 0;
self.received_this_minute = 0;
}
/// Held changes past their ceiling: dropped if the book moved on without
@@ -862,11 +1035,14 @@ impl Feed {
counts.resyncs += 1;
}
/// A full book counts only when asked for: a symbol's first (from any of
/// its connections), or the answer to a resync. It replaces ours, brought
/// forward first if it is behind our stream, and is written as a Snapshot
/// when it is the symbol's first, when it differs from ours, or when the
/// connection followed changed. Then it is checked against its own hash.
/// A full book counts when asked for (a symbol's first, from any of its
/// connections, or the answer to a resync), or when it is a reconnect's
/// opening book of a symbol unchanged for [`STILL_FOR`]. It replaces
/// ours, brought forward first if it is behind our stream, and is written
/// as a Snapshot when it is the symbol's first, when it differs from
/// ours, or when the connection followed changed. Then it is checked
/// against its own hash. A reconnect's book that is ours, or behind
/// ours, changes nothing at all.
fn on_book(
&mut self,
received_at: DateTime<Utc>,
@@ -895,7 +1071,24 @@ impl Feed {
let complete = message.last_trade_price.is_some();
let asked = symbol.asked.filter(|&(asked, _)| asked == connection);
let first = symbol.state == State::AwaitingBook && symbol.connections.contains(&connection);
if !complete || (asked.is_none() && !first) {
// The connection's first complete book since it came up, for a
// symbol subscribed there before: the reconnect's opening book.
let up_since = self.up_since.get(connection).copied().flatten();
let reconnect = complete
&& symbol.watch_mut(connection).is_some_and(|watch| {
// A book at the very instant it came up came before it.
let opening = up_since.is_some_and(|up| watch.since < up && watch.book_at <= up);
watch.book_at = received_at;
opening
});
// Judged only for a still, verified symbol, nothing held or asked.
let reconnect = reconnect
&& asked.is_none()
&& symbol.asked.is_none()
&& symbol.state == State::Verified
&& symbol.held.is_empty()
&& received_at - symbol.advanced_at >= STILL_FOR;
if !complete || (asked.is_none() && !first && !reconnect) {
return;
}
let (Some(book), Some(exchange_timestamp), Some(book_hash)) = (
@@ -903,6 +1096,10 @@ impl Feed {
parse_millis(&message.timestamp),
parse_hash(&message.hash),
) else {
// Nobody asked for a reconnect's book: unreadable, it is ignored.
if reconnect {
return;
}
// A protocol error: ask again. A book that stays unreadable is asked
// for once per answer, and its symbol stays unchecked meanwhile.
log!("polymarket: {token}: unreadable book, asking for a fresh one");
@@ -910,6 +1107,28 @@ impl Feed {
self.counts.resyncs += 1;
return;
};
if reconnect {
self.counts.reconnect_books += 1;
// The exchange's opening books lag by minutes (docs/notes.md): one
// that is ours, or a state we passed, leaves ours, and the hash
// inputs, as they are.
let brought = symbol
.recent_changes
.iter()
.rposition(|change| change.hash == book_hash)
.map(|position| symbol.brought_forward(&book, position + 1));
if book == symbol.book || brought.as_ref() == Some(&symbol.book) {
return;
}
// Our chain holds up to our last change, so a book no newer than
// it holds nothing we missed, however far back it lags: further
// than the changes we remember, too.
if exchange_timestamp <= symbol.exchange_timestamp || symbol.was_ours(&book) {
self.counts.books_behind += 1;
return;
}
}
let inputs = reconnect.then(|| (symbol.tick_size.clone(), symbol.last_trade_price.clone()));
if let Some(tick_size) = message.tick_size {
symbol.tick_size = tick_size;
}
@@ -918,6 +1137,25 @@ impl Feed {
}
// Checked as received, before being brought forward below.
let verified = hash(&token, symbol, &book, &message.timestamp) == book_hash;
if let Some(inputs) = inputs {
// A book failing its own hash proves nothing against a verified
// one: ours stands, with its hash inputs.
if !verified {
(symbol.tick_size, symbol.last_trade_price) = inputs;
return;
}
// Newer than ours and no state of ours: both subscriptions were
// quiet. The other one is renewed too; its book is not waited for.
self.counts.reconnect_losses += 1;
for other in symbol.connections.clone() {
if other != connection {
if let Some(watch) = symbol.watch_mut(other) {
watch.since = received_at;
}
effects.push(Effect::Send(other, Command::Resubscribe(token.clone())));
}
}
}
let switch = asked.is_some_and(|(_, switch)| switch);
let compared = !first || symbol.had_book;
@@ -947,7 +1185,7 @@ impl Feed {
let differs = compared && book != symbol.book;
let cause = if switch {
Some(Cause::Switch)
} else if differs && !symbol.verified_since_book && !symbol.had_book {
} else if differs && !reconnect && !symbol.verified_since_book && !symbol.had_book {
// A subscribe hides a few ms of changes (docs/notes.md): until a
// change is verified, a differing book is that gap.
Some(Cause::SubscriptionGap)
@@ -978,8 +1216,13 @@ impl Feed {
effects.push(Effect::Write(snapshot));
symbol.last_written = symbol.last_written.max(received_at);
if let Some(cause) = cause {
let found = if reconnect {
", found by a reconnect"
} else {
""
};
let detail = format!(
" (its time {}, ours {})",
"{found} (its time {}, ours {})",
book_time.format("%H:%M:%S%.3f"),
exchange_timestamp.format("%H:%M:%S%.3f")
);
@@ -1015,6 +1258,10 @@ impl Feed {
let Some(symbol) = self.symbols.get_mut(&token) else {
continue;
};
if let Some(watch) = symbol.watch_mut(connection) {
watch.received += 1;
}
self.received_this_minute += 1;
let side = match change.side.as_str() {
"BUY" => Some(Side::Buy),
"SELL" => Some(Side::Sell),
@@ -1146,7 +1393,7 @@ pub fn run(
events: Receiver<Received>,
commands: Receiver<BookCommand>,
outputs: Sender<Output>,
reports: Sender<Loads>,
reports: Sender<Report>,
) {
serve(feed, Vec::new(), events, commands, outputs, reports)
}
@@ -1159,7 +1406,7 @@ fn serve(
events: Receiver<Received>,
commands: Receiver<BookCommand>,
outputs: Sender<Output>,
reports: Sender<Loads>,
reports: Sender<Report>,
) {
let no_commands = never();
let mut orchestrator_gone = false;
@@ -1190,7 +1437,9 @@ fn serve(
Vec::new()
}
Ok(BookCommand::Add(market, on)) => feed.add_market(&market, &on, Utc::now()),
Ok(BookCommand::Move { market, from, to }) => feed.move_market(&market, from, to),
Ok(BookCommand::Move { market, from, to }) => {
feed.move_market(&market, from, to, Utc::now())
}
Ok(BookCommand::Remove(market)) => feed.remove_market(&market),
// It is stopping: the connections' end still decides ours.
Err(_) => {
@@ -1210,7 +1459,7 @@ fn serve(
// The connection may be gone already, while stopping, and the
// orchestrator too.
Effect::Send(connection, command) => _ = connections[connection].send(command),
Effect::Report(loads) => _ = reports.send(loads),
Effect::Report(report) => _ = reports.send(report),
}
}
}
@@ -1417,7 +1666,7 @@ mod tests {
effects
.iter()
.filter_map(|effect| match effect {
Effect::Send(connection, Command::Resync(token)) => {
Effect::Send(connection, Command::Resubscribe(token)) => {
Some((*connection, token.as_str()))
}
_ => None,
@@ -1549,9 +1798,13 @@ mod tests {
/// A complete book frame for `book`, as the exchange would send it in
/// answer to a resync, 5 ms after the symbol's last change.
fn fresh_book(token: &str, symbol: &Symbol, book: &Book) -> String {
let timestamp = (symbol.exchange_timestamp + TimeDelta::milliseconds(5))
.timestamp_millis()
.to_string();
let at = symbol.exchange_timestamp + TimeDelta::milliseconds(5);
book_frame(token, symbol, book, at)
}
/// A complete book frame for `book`, last changed at `at` (exchange time).
fn book_frame(token: &str, symbol: &Symbol, book: &Book, at: DateTime<Utc>) -> String {
let timestamp = at.timestamp_millis().to_string();
let levels = |levels: &BTreeMap<i64, i64>| {
levels
.iter()
@@ -2379,6 +2632,302 @@ mod tests {
assert_eq!(feed.counts.losses, losses + 1);
}
/// A frame setting `token`'s bid at `price` to `quantity`, `millis` its
/// exchange time, from the book as it is now.
fn change(feed: &Feed, token: &str, millis: i64, price: i64, quantity: i64) -> String {
let symbol = &feed.symbols[token];
let changes = [(Side::Buy, price, quantity)];
changes_frame(token, symbol, &symbol.book, &changes, millis).0
}
fn resyncs_of<'a>(effects: &'a [Effect], token: &str) -> Vec<(usize, &'a str)> {
resyncs(effects)
.into_iter()
.filter(|&(_, theirs)| theirs == token)
.collect()
}
/// From the next wall-clock minute after `at`, which the feed has begun:
/// the following minute is counted whole.
fn begin_minute(feed: &mut Feed, at: DateTime<Utc>) -> DateTime<Utc> {
let minute = next_minute(at);
feed.idle(minute);
minute
}
/// `count` changes of `token`, from 10 s after `start`, on the
/// connections `on` only.
fn changes(
feed: &mut Feed,
token: &str,
on: &[usize],
start: DateTime<Utc>,
price: i64,
count: i64,
) {
let millis = feed.symbols[token].exchange_timestamp.timestamp_millis() + 1_000;
for step in 0..count {
let at = start + TimeDelta::seconds(10 + step);
let frame = change(feed, token, millis + step, price, (step + 1) * 1_000_000);
for &connection in on {
feed.on_frame_from(connection, at, &frame);
}
}
}
fn quiet_reports(effects: &[Effect]) -> Vec<(&str, usize)> {
effects
.iter()
.filter_map(|effect| match effect {
Effect::Report(Report::Quiet { market, connection }) => {
Some((market.as_str(), *connection))
}
_ => None,
})
.collect()
}
#[test]
fn a_subscription_quiet_on_one_connection_is_resubscribed_then_reported() {
let (mut feed, token, _, free, _, at) = two_connection_symbol();
let market = feed.symbols[&token].market.clone();
let minute = |n: i64| next_minute(at) + TimeDelta::minutes(n);
begin_minute(&mut feed, at);
let (losses, gaps) = (feed.counts.losses, feed.counts.subscription_gaps);
// Two changes against none: too few to say.
changes(&mut feed, &token, &[0], minute(0), free, 2);
let effects = feed.idle(minute(1));
assert!(resyncs_of(&effects, &token).is_empty());
assert_eq!(feed.counts.quiet, 0);
// Three: quiet on connection 1, resubscribed there.
changes(&mut feed, &token, &[0], minute(1), free, 3);
let effects = feed.idle(minute(2));
assert_eq!(resyncs_of(&effects, &token), [(1, token.as_str())]);
assert!(quiet_reports(&effects).is_empty());
assert_eq!(feed.counts.quiet, 1);
assert_eq!(feed.symbols[&token].state, State::Verified);
// The book the resubscribe brings back is not asked for: ignored.
let book = feed.symbols[&token].book.clone();
let back = fresh_book(&token, &feed.symbols[&token], &book);
let effects = feed.on_frame_from(1, minute(2) + TimeDelta::seconds(1), &back);
assert!(snapshot_symbols(&effects).is_empty());
// Still quiet the minute after: reported for a move, not resubscribed.
changes(&mut feed, &token, &[0], minute(2), free, 3);
let effects = feed.idle(minute(3));
assert!(resyncs_of(&effects, &token).is_empty());
assert_eq!(quiet_reports(&effects), [(market.as_str(), 1)]);
assert_eq!(
(feed.counts.losses, feed.counts.subscription_gaps),
(losses, gaps)
);
}
/// Both tokens of a market go quiet together: one report for the market.
#[test]
fn a_market_quiet_on_both_its_tokens_is_reported_once() {
let (mut feed, token, _, free, _, at) = two_connection_symbol();
let market = feed.symbols[&token].market.clone();
let tokens = feed.markets[&market].tokens.clone();
assert!(
tokens
.iter()
.all(|token| feed.symbols[token].state == State::Verified)
);
let minute = |n: i64| next_minute(at) + TimeDelta::minutes(n);
begin_minute(&mut feed, at);
for n in 0..2 {
for token in &tokens {
changes(&mut feed, token, &[0], minute(n), free, 3);
}
let effects = feed.idle(minute(n + 1));
if n == 0 {
assert_eq!(feed.counts.quiet, 2);
} else {
assert_eq!(quiet_reports(&effects), [(market.as_str(), 1)]);
}
}
}
#[test]
fn a_subscription_delivering_or_not_there_the_whole_minute_is_not_quiet() {
let (mut feed, token, _, free, _, at) = two_connection_symbol();
let markets = fixture_markets();
let market = markets
.iter()
.find(|market| market.tokens.contains(&token))
.unwrap();
// Connection 5 is up from before the minutes below.
feed.on_received(Received::Up(5));
let minute = |n: i64| next_minute(at) + TimeDelta::minutes(n);
begin_minute(&mut feed, at);
// Both deliver: neither is quiet; the symbols that didn't change
// aren't either.
changes(&mut feed, &token, &[0, 1], minute(0), free, 3);
let effects = feed.idle(minute(1));
assert!(resyncs(&effects).iter().all(|&(_, theirs)| theirs != token));
// Connection 1 dropped and came back within the minute: not judged
// for the changes it missed.
changes(&mut feed, &token, &[0], minute(1), free, 3);
feed.on_received(Received::Down(1));
feed.on_received(Received::Up(1));
let effects = feed.idle(minute(2));
assert!(resyncs_of(&effects, &token).is_empty());
// Connection 5, subscribed in the middle of the minute and delivering
// nothing yet: not judged either.
feed.move_market(market, 1, 5, minute(2) + TimeDelta::seconds(5));
changes(&mut feed, &token, &[0, 1], minute(2), free, 3);
let effects = feed.idle(minute(3));
assert!(resyncs_of(&effects, &token).is_empty());
assert_eq!(feed.counts.quiet, 0);
}
/// A verified symbol on connections 0 and 1, changed once and then still
/// for 30 s: the feed, its token, its book before the change and now, a
/// free bid price, and the time 30 s after.
fn still_symbol() -> (Feed, String, Book, Book, i64, DateTime<Utc>) {
let (mut feed, token, before, free, millis, at) = two_connection_symbol();
let x = change(&feed, &token, millis, free, 1_000_000);
feed.on_frame_from(0, at, &x);
feed.on_frame_from(1, at, &x);
let ours = feed.symbols[&token].book.clone();
let later = at + TimeDelta::seconds(30);
// The minute's Snapshots, out of the way.
feed.idle(later);
(feed, token, before, ours, free, later)
}
/// Connection 1 drops and reconnects; its opening book of `token` holds
/// `book`.
fn reconnect_with(feed: &mut Feed, token: &str, book: &Book, at: DateTime<Utc>) -> Vec<Effect> {
feed.on_received(Received::Down(1));
feed.on_received(Received::Up(1));
let opening = fresh_book(token, &feed.symbols[token], book);
feed.on_frame_from(1, at, &opening)
}
#[test]
fn a_reconnect_book_of_a_still_symbol_equal_to_ours_or_behind_changes_nothing() {
let (mut feed, token, before, ours, _, later) = still_symbol();
let losses = feed.counts.losses;
let effects = reconnect_with(&mut feed, &token, &ours, later);
assert!(snapshot_symbols(&effects).is_empty());
assert_eq!(feed.counts.reconnect_books, 1);
// The server's opening books lag: a state we passed is no loss.
let behind = feed.counts.books_behind;
let effects = reconnect_with(&mut feed, &token, &before, later + TimeDelta::seconds(1));
assert!(snapshot_symbols(&effects).is_empty());
assert_eq!(feed.counts.reconnect_books, 2);
assert_eq!(feed.counts.books_behind, behind + 1);
assert_eq!(feed.symbols[&token].book, ours);
// It lags further back than the changes we remember: no newer than
// our last change, it still holds nothing we missed.
let symbol = feed.symbols.get_mut(&token).unwrap();
symbol.recent_changes.clear();
let old = symbol.exchange_timestamp - TimeDelta::minutes(10);
feed.on_received(Received::Down(1));
feed.on_received(Received::Up(1));
let lagging = book_frame(&token, &feed.symbols[&token], &before, old);
let effects = feed.on_frame_from(1, later + TimeDelta::seconds(2), &lagging);
assert!(snapshot_symbols(&effects).is_empty());
assert_eq!(feed.counts.books_behind, behind + 2);
assert_eq!(feed.symbols[&token].book, ours);
assert_eq!(feed.counts.losses, losses);
assert_eq!(feed.symbols[&token].state, State::Verified);
}
/// A reconnect's book that fails its own hash proves nothing: the symbol
/// stays verified, with its book and its hash inputs.
#[test]
fn a_reconnect_book_failing_its_own_hash_changes_nothing() {
let (mut feed, token, _, ours, free, later) = still_symbol();
let mut other = ours.clone();
other.set(Side::Buy, free, 7_000_000);
let frame = fresh_book(&token, &feed.symbols[&token], &other);
let mut frame: serde_json::Value = serde_json::from_str(&frame).unwrap();
frame["hash"] = "0".repeat(40).into();
frame["last_trade_price"] = "0.123".into();
let last_trade_price = feed.symbols[&token].last_trade_price.clone();
feed.on_received(Received::Down(1));
feed.on_received(Received::Up(1));
let effects = feed.on_frame_from(1, later, &frame.to_string());
assert!(effects.is_empty());
let symbol = &feed.symbols[&token];
assert_eq!(symbol.state, State::Verified);
assert_eq!(symbol.book, ours);
assert_eq!(symbol.last_trade_price, last_trade_price);
}
/// Only a reconnect's book of a symbol with nothing asked is judged: not
/// the opening book of a subscription made since the connection came up
/// (a move's), nor any while a fresh book is asked.
#[test]
fn a_book_that_is_no_reconnects_or_comes_while_one_is_asked_is_not_judged() {
let (mut feed, token, _, ours, free, later) = still_symbol();
let markets = fixture_markets();
let market = markets
.iter()
.find(|market| market.tokens.contains(&token))
.unwrap();
let mut other = ours.clone();
other.set(Side::Buy, free, 7_000_000);
feed.on_received(Received::Up(5));
feed.move_market(market, 0, 5, later);
let opening = fresh_book(&token, &feed.symbols[&token], &other);
let effects = feed.on_frame_from(5, later + TimeDelta::seconds(1), &opening);
assert!(snapshot_symbols(&effects).is_empty());
assert_eq!(feed.counts.reconnect_books, 0);
feed.symbols.get_mut(&token).unwrap().asked = Some((5, false));
let effects = reconnect_with(&mut feed, &token, &other, later + TimeDelta::seconds(2));
assert!(snapshot_symbols(&effects).is_empty());
assert_eq!(feed.counts.reconnect_books, 0);
assert_eq!(feed.symbols[&token].book, ours);
}
#[test]
fn a_reconnect_book_of_a_still_symbol_that_differs_is_a_loss_and_resubscribes_the_other() {
let (mut feed, token, _, ours, free, later) = still_symbol();
let (losses, gaps) = (feed.counts.losses, feed.counts.subscription_gaps);
// Even before any change of it was verified: a loss, not the gap of
// a subscribe, which is long past.
feed.symbols.get_mut(&token).unwrap().verified_since_book = false;
let mut other = ours.clone();
other.set(Side::Buy, free, 7_000_000);
let effects = reconnect_with(&mut feed, &token, &other, later);
assert_eq!(snapshot_symbols(&effects), [token.as_str()]);
assert_eq!(resyncs_of(&effects, &token), [(0, token.as_str())]);
assert_eq!(feed.counts.losses, losses + 1);
assert_eq!(feed.counts.subscription_gaps, gaps);
assert_eq!(feed.counts.reconnect_losses, 1);
assert_eq!(feed.symbols[&token].book, other);
assert_eq!(feed.symbols[&token].state, State::Verified);
}
#[test]
fn a_reconnect_book_of_a_symbol_changed_lately_is_ignored() {
let (mut feed, token, _, ours, free, later) = still_symbol();
let millis = feed.symbols[&token].exchange_timestamp.timestamp_millis() + 1_000;
let x = change(&feed, &token, millis, free, 2_000_000);
feed.on_frame_from(0, later, &x);
let mut other = ours.clone();
other.set(Side::Buy, free, 7_000_000);
let effects = reconnect_with(&mut feed, &token, &other, later + TimeDelta::seconds(1));
assert!(snapshot_symbols(&effects).is_empty());
assert_eq!(feed.counts.reconnect_books, 0);
// And the next book of it from that connection is no reconnect's.
feed.idle(later + TimeDelta::seconds(30));
let opening = fresh_book(&token, &feed.symbols[&token], &other);
let effects = feed.on_frame_from(1, later + TimeDelta::seconds(31), &opening);
assert!(snapshot_symbols(&effects).is_empty());
assert_eq!(feed.counts.reconnect_books, 0);
}
#[test]
fn the_followed_connection_down_switches_with_a_snapshot() {
let (mut feed, token, book, _, _, at) = two_connection_symbol();
@@ -2402,7 +2951,7 @@ mod tests {
.iter()
.find(|market| market.tokens.contains(&token))
.unwrap();
let effects = feed.move_market(market, 0, 5);
let effects = feed.move_market(market, 0, 5, at);
assert!(matches!(
effects.as_slice(),
[Effect::Send(5, Command::Subscribe(tokens))] if *tokens == market.tokens
@@ -2442,7 +2991,7 @@ mod tests {
let symbol = feed.symbols.get_mut(&token).unwrap();
symbol.state = State::Following(1);
symbol.asked = Some((0, false));
feed.move_market(market, 0, 5);
feed.move_market(market, 0, 5, at);
let opening = fresh_book(&token, &feed.symbols[&token], &book);
let effects = feed.on_frame_from(5, at, &opening);
assert!(
@@ -2515,7 +3064,7 @@ mod tests {
feed.on_frame_from(0, minute + TimeDelta::seconds(10), &change);
feed.on_frame_from(1, minute + TimeDelta::seconds(10), &change);
let effects = feed.tick(minute + TimeDelta::seconds(60));
let Some(Effect::Report(loads)) = effects.first() else {
let Some(Effect::Report(Report::Loads(loads))) = effects.first() else {
panic!("no report");
};
assert_eq!(loads["0xm"], change.len() as u64 / 60);
@@ -2583,9 +3132,10 @@ mod tests {
let mentions = |effect: &Effect| match effect {
Effect::Write(Output::Snapshot(snapshot)) => removed.contains(&snapshot.symbol),
Effect::Write(Output::Delta(delta)) => removed.contains(&delta.symbol),
Effect::Send(_, Command::Resync(token)) => removed.contains(token),
Effect::Send(_, Command::Resubscribe(token)) => removed.contains(token),
Effect::Send(..) => false,
Effect::Report(loads) => loads.contains_key(&markets[0].condition_id),
Effect::Report(Report::Loads(loads)) => loads.contains_key(&markets[0].condition_id),
Effect::Report(Report::Quiet { market, .. }) => *market == markets[0].condition_id,
};
assert!(!effects.iter().any(mentions));
// The minute Snapshots of the 38 others, twice.
@@ -2627,7 +3177,7 @@ mod tests {
let mut reports = Vec::new();
let mut on_frame = |feed: &mut Feed, seconds, text: &str| {
for effect in feed.on_frame(at(seconds), text) {
if let Effect::Report(loads) = effect {
if let Effect::Report(Report::Loads(loads)) = effect {
reports.push(loads);
}
}
@@ -2671,7 +3221,7 @@ mod tests {
let reports: Vec<Loads> = replay(&mut feed, &frames)
.into_iter()
.filter_map(|effect| match effect {
Effect::Report(loads) => Some(loads),
Effect::Report(Report::Loads(loads)) => Some(loads),
_ => None,
})
.collect();
+206 -29
View File
@@ -1,7 +1,7 @@
//! The orchestrator: the one owner of what is tracked. It keeps the tracked
//! markets equal to Gamma's live ones, decides which connections carry each
//! market and owns every timer; the two fetchers only fetch (ADR 0002, ADR
//! 0003, roadmap/04).
//! market and owns every timer, recycling connections among them; the two
//! fetchers only fetch (ADR 0002, ADR 0003, roadmap/04, roadmap/05).
use std::cmp::Reverse;
use std::collections::{HashMap, HashSet, VecDeque};
@@ -14,8 +14,9 @@ use chrono::{DateTime, TimeDelta, Utc};
use crossbeam::channel::{Sender, select, unbounded};
use engineering_challenge::book::Output;
use super::feed::{self, BookCommand, Feed, Loads};
use super::feed::{self, BookCommand, Feed, Loads, Report};
use super::gamma::{self, Finding, Request};
use super::socket::{Command, Notice};
use super::wire::Received;
use super::{BOOK_THREADS, Market, book_thread, socket};
@@ -50,6 +51,9 @@ const RESCAN_EVERY: Duration = Duration::from_secs(60 * 60);
/// `createdAt` (docs/notes.md); known markets are skipped, so the margin costs
/// only the pages it reads again.
const RECENT_MARGIN: TimeDelta = TimeDelta::minutes(15);
/// Each connection is recycled about this often, one at a time, to shed what
/// accumulates on the exchange's side of it.
const RECYCLE_EVERY: Duration = Duration::from_secs(60 * 60);
/// So SIGINT is seen while no timer is due.
const WAKE_EVERY: Duration = Duration::from_secs(1);
@@ -70,10 +74,10 @@ pub fn run(outputs: Sender<Output>, stop: &Arc<AtomicBool>) -> usize {
})
.collect();
drop(findings_tx);
let (reports_tx, reports) = unbounded::<Loads>();
let (closes_tx, closes) = unbounded::<usize>();
let mut threads = Threads::start(&outputs, &reports_tx, &closes_tx, stop);
drop(closes_tx);
let (reports_tx, reports) = unbounded::<Report>();
let (notices_tx, notices) = unbounded::<Notice>();
let mut threads = Threads::start(&outputs, &reports_tx, &notices_tx, stop);
drop(notices_tx);
// The book threads hold the only senders the writer waits for.
drop(outputs);
drop(reports_tx);
@@ -91,6 +95,7 @@ pub fn run(outputs: Sender<Output>, stop: &Arc<AtomicBool>) -> usize {
let mut queue: VecDeque<BookCommand> = VecDeque::new();
let mut next_deal = Instant::now();
let (mut added, mut removed, mut moved, mut splits) = (0, 0, 0, 0);
let (mut recycled, mut quiet_moves, mut silent) = (0, 0, 0);
// The minute checks cover the first scan from its start.
let mut recent_since = Utc::now() - RECENT_MARGIN;
let mut recent_asked_at: Option<DateTime<Utc>> = None;
@@ -134,12 +139,22 @@ pub fn run(outputs: Sender<Output>, stop: &Arc<AtomicBool>) -> usize {
let busiest = tracker.connections.iter().map(|one| one.load).max();
log!(
"polymarket: orchestrator: {} markets on {} connections; {added} added, {removed} \
removed, {moved} moved, {splits} splits this minute; busiest connection at {} bytes/s",
removed, {moved} moved ({quiet_moves} quiet), {splits} splits, {recycled} recycled, \
{silent} silent this minute; busiest connection at {} bytes/s",
tracker.markets.len(),
tracker.connections.len(),
busiest.unwrap_or(0)
);
(added, removed, moved, splits) = (0, 0, 0, 0);
(recycled, quiet_moves, silent) = (0, 0, 0);
}
// Recycling starts once the startup deal is sent.
if first_scan.is_none() && queue.is_empty() {
tracker.start_recycling(now);
}
if let Some(connection) = tracker.recycle(now) {
threads.recycle(connection);
recycled += 1;
}
if !queue.is_empty() && now >= next_deal {
for command in queue.drain(..DEAL_AT_ONCE.min(queue.len())) {
@@ -217,13 +232,26 @@ pub fn run(outputs: Sender<Output>, stop: &Arc<AtomicBool>) -> usize {
// Both fetchers are gone: the check above stops the run.
Err(_) => {}
},
recv(reports) -> loads => {
if let Ok(loads) = loads {
tracker.report(&loads);
recv(reports) -> report => match report {
Ok(Report::Loads(loads)) => tracker.report(&loads),
Ok(Report::Quiet { market, connection }) => {
if let Some(command) = tracker.quiet(&market, connection, Instant::now()) {
log!(
"polymarket: orchestrator: {market} still quiet on connection \
{connection} after a resubscribe, moved"
);
quiet_moves += 1;
moved += 1;
commands.push(command);
}
}
Err(_) => {}
},
recv(closes) -> closed => {
if let Ok(connection) = closed {
recv(notices) -> notice => match notice {
Ok(Notice::Up(connection)) => tracker.up(connection, Instant::now()),
Ok(Notice::Down(connection)) => tracker.down(connection),
Ok(Notice::Silent(_)) => silent += 1,
Ok(Notice::TooSlow(connection)) => {
let moves = tracker.closed(connection, Instant::now());
if !moves.is_empty() {
log!(
@@ -236,6 +264,7 @@ pub fn run(outputs: Sender<Output>, stop: &Arc<AtomicBool>) -> usize {
}
commands.extend(moves);
}
Err(_) => {}
},
default(timeout) => {}
}
@@ -261,7 +290,9 @@ struct Threads {
/// Every book thread's events channel, by number: each connection gets a
/// clone of them all.
events: Vec<Sender<Received>>,
closes: Sender<usize>,
notices: Sender<Notice>,
/// Each connection's commands, by number, for its recycles.
connection_commands: Vec<Sender<Command>>,
book_threads: Vec<JoinHandle<()>>,
connections: Vec<JoinHandle<()>>,
stop: Arc<AtomicBool>,
@@ -270,14 +301,15 @@ struct Threads {
impl Threads {
fn start(
outputs: &Sender<Output>,
reports: &Sender<Loads>,
closes: &Sender<usize>,
reports: &Sender<Report>,
notices: &Sender<Notice>,
stop: &Arc<AtomicBool>,
) -> Self {
let mut threads = Self {
book_commands: Vec::new(),
events: Vec::new(),
closes: closes.clone(),
notices: notices.clone(),
connection_commands: Vec::new(),
book_threads: Vec::new(),
connections: Vec::new(),
stop: Arc::clone(stop),
@@ -302,23 +334,29 @@ impl Threads {
let number = self.connections.len();
let (commands, commands_rx) = unbounded();
let (events, stop) = (self.events.clone(), Arc::clone(&self.stop));
let closes = self.closes.clone();
let notices = self.notices.clone();
self.connections.push(thread::spawn(move || {
socket::run(
socket::MARKET_CHANNEL_URL,
number,
&events,
&commands_rx,
&closes,
&notices,
socket::SILENT_AFTER,
&stop,
)
}));
self.connection_commands.push(commands.clone());
for book_thread in &self.book_commands {
_ = book_thread.send(BookCommand::Connected(number, commands.clone()));
}
}
}
fn recycle(&self, connection: usize) {
_ = self.connection_commands[connection].send(Command::Recycle);
}
/// To the market's book thread. One that died keeps its markets unserved,
/// and its connections stop the run.
fn send(&self, command: BookCommand) {
@@ -360,16 +398,19 @@ struct Tracked {
moved_at: Option<Instant>,
}
/// A connection's tokens and load, against its two limits, and its 1013
/// closes.
/// A connection's tokens and load, against its two limits, its 1013 closes,
/// and whether and since when it is up.
#[derive(Debug, Default, Clone, PartialEq, Eq)]
struct ConnectionLoad {
struct ConnectionState {
tokens: usize,
/// The sum of its markets' loads, bytes per second.
load: u64,
/// When it was last closed with 1013.
closed_at: Option<Instant>,
split_at: Option<Instant>,
up: bool,
/// When it last came up: the one longest ago is recycled next.
connected_at: Option<Instant>,
}
/// Which connections carry each market and what it weighs, and which markets
@@ -379,10 +420,14 @@ struct Tracker {
/// By `conditionId`.
markets: HashMap<String, Tracked>,
/// By connection number.
connections: Vec<ConnectionLoad>,
connections: Vec<ConnectionState>,
/// While a rescan runs: the markets known at its start, not seen since.
unseen: Option<HashSet<String>>,
random: Random,
/// When the next recycle is due; None until recycling starts.
next_recycle: Option<Instant>,
/// The connection recycled and not yet up again.
recycling: Option<usize>,
}
impl Tracker {
@@ -393,6 +438,8 @@ impl Tracker {
connections: Vec::new(),
unseen: None,
random: Random(seed | 1),
next_recycle: None,
recycling: None,
}
}
@@ -410,7 +457,7 @@ impl Tracker {
let count = (self.subscriptions * tokens)
.div_ceil(STARTUP_FILL)
.max(self.subscriptions);
self.connections = vec![ConnectionLoad::default(); count];
self.connections = vec![ConnectionState::default(); count];
let mut placed = vec![Vec::new(); markets.len()];
for _ in 0..self.subscriptions {
let mut order: Vec<usize> = (0..markets.len()).collect();
@@ -474,7 +521,7 @@ impl Tracker {
})
.collect();
if with_room.is_empty() {
self.connections.push(ConnectionLoad::default());
self.connections.push(ConnectionState::default());
return self.connections.len() - 1;
}
with_room[self.random.below(with_room.len())]
@@ -604,6 +651,73 @@ impl Tracker {
.collect()
}
/// A market the book thread found quiet on `connection` again after a
/// resubscribe: its subscription there moves, beyond rebalancing's
/// limits. None if it no longer is there.
fn quiet(&mut self, id: &str, connection: usize, now: Instant) -> Option<BookCommand> {
let tracked = self.markets.get(id)?;
// A market moved lately stays, as for rebalancing: its last move may
// still be under way, and a second one would end it early.
let stays = tracked
.moved_at
.is_some_and(|at| now.duration_since(at) < MOVED_STAYS);
(tracked.connections.contains(&connection) && !stays)
.then(|| self.move_off(id, connection, now))
}
/// A connection came up: its subscribes are sent, its opening books
/// still to come (0.3 to 20 s, docs/notes.md), so no recycle starts for
/// one interval.
fn up(&mut self, connection: usize, now: Instant) {
if let Some(up) = self.connections.get_mut(connection) {
up.up = true;
up.connected_at = Some(now);
}
if self.recycling == Some(connection) {
self.recycling = None;
}
let settled = now + self.recycle_interval();
if let Some(next) = &mut self.next_recycle {
*next = (*next).max(settled);
}
}
fn down(&mut self, connection: usize) {
if let Some(down) = self.connections.get_mut(connection) {
down.up = false;
}
}
fn start_recycling(&mut self, now: Instant) {
if self.next_recycle.is_none() {
self.next_recycle = Some(now + self.recycle_interval());
}
}
/// Each connection about once per [`RECYCLE_EVERY`], one at a time.
fn recycle_interval(&self) -> Duration {
RECYCLE_EVERY / self.connections.len().max(1) as u32
}
/// The connection to recycle now, if one is due: the one up the longest,
/// once the last recycled is up again, and only while every connection is
/// up and has been for an interval, so no market loses both of its
/// connections at once.
fn recycle(&mut self, now: Instant) -> Option<usize> {
let due = self.next_recycle.is_some_and(|next| now >= next);
if !due
|| self.recycling.is_some()
|| self.connections.iter().any(|connection| !connection.up)
{
return None;
}
let oldest = (0..self.connections.len())
.min_by_key(|&number| self.connections[number].connected_at)?;
self.recycling = Some(oldest);
self.next_recycle = Some(now + self.recycle_interval());
Some(oldest)
}
fn scan_started(&mut self) {
self.unseen = Some(self.markets.keys().cloned().collect());
}
@@ -795,7 +909,7 @@ mod tests {
for id in 0..10 {
tracker.add(market(id));
}
tracker.connections.push(ConnectionLoad::default());
tracker.connections.push(ConnectionState::default());
let budget = CONNECTION_BYTES_PER_SECOND;
// Both connections at 1.5 budgets, market 0 weighing half a budget.
let mut loads = vec![(0, budget / 2)];
@@ -831,7 +945,7 @@ mod tests {
tracker.add(market(id));
}
// Room elsewhere, on several connections.
tracker.connections.resize(6, ConnectionLoad::default());
tracker.connections.resize(6, ConnectionState::default());
report(
&mut tracker,
&[(0, CONNECTION_BYTES_PER_SECOND), (1, 1_000)],
@@ -850,8 +964,8 @@ mod tests {
fn the_busiest_connection_over_budget_is_rebalanced_first() {
let mut tracker = Tracker::new(1, 7);
tracker.deal(vec![market(0), market(1)]);
tracker.connections.push(ConnectionLoad::default());
tracker.connections.push(ConnectionLoad::default());
tracker.connections.push(ConnectionState::default());
tracker.connections.push(ConnectionState::default());
// One connection so far: put markets 2 and 3 on another.
for id in [2, 3] {
tracker.connections[0].tokens = CONNECTION_TOKENS;
@@ -931,6 +1045,69 @@ mod tests {
assert_eq!(moves(&commands), [("0x2", 1, 2)]);
}
#[test]
fn recycling_takes_the_oldest_connection_one_at_a_time_while_every_one_is_up() {
let mut tracker = tracker();
tracker.connections = vec![ConnectionState::default(); 3];
let start = Instant::now();
let at = |seconds: u64| start + Duration::from_secs(seconds);
// Up in the order 2, 0, 1.
for (seconds, connection) in [(0, 2), (1, 0), (2, 1)] {
tracker.up(connection, at(seconds));
}
assert_eq!(tracker.recycle(at(5_000)), None, "not started");
tracker.start_recycling(at(10));
// One every hour / 3 connections.
assert_eq!(tracker.recycle(at(1_209)), None);
assert_eq!(tracker.recycle(at(1_210)), Some(2));
// The next waits for the recycled one, whose Down may not even have
// come yet, to be up again...
assert_eq!(tracker.recycle(at(2_410)), None);
tracker.down(2);
assert_eq!(tracker.recycle(at(2_410)), None);
tracker.up(2, at(2_411));
// ...and for its opening books: an interval more.
assert_eq!(tracker.recycle(at(3_610)), None);
assert_eq!(tracker.recycle(at(3_611)), Some(0));
tracker.down(0);
tracker.up(0, at(3_612));
// A connection down for its own reasons holds every recycle, and its
// return delays the next like any other.
tracker.down(2);
assert_eq!(tracker.recycle(at(6_000)), None);
tracker.up(2, at(6_001));
assert_eq!(tracker.recycle(at(7_200)), None);
assert_eq!(tracker.recycle(at(7_201)), Some(1));
}
#[test]
fn a_market_quiet_again_moves_off_that_connection_unless_it_moved_lately() {
let mut tracker = tracker();
tracker.deal((0..50).map(market).collect());
let now = Instant::now();
let on = connections(&tracker, 0);
let first = [tracker.quiet("0x0", on[0], now).expect("a move")];
let [(id, from, to)] = moves(&first)[..] else {
panic!("one move");
};
assert_eq!((id, from), ("0x0", on[0]));
assert!(!on.contains(&to));
// Moved a minute ago, its move perhaps still under way: it stays,
// as for rebalancing.
let on = connections(&tracker, 0);
let soon = now + Duration::from_secs(60);
assert!(tracker.quiet("0x0", on[1], soon).is_none());
assert!(tracker.quiet("0x0", on[1], now + MOVED_STAYS).is_some());
let now = now + MOVED_STAYS + MOVED_STAYS;
// Reported for a connection it isn't on (any more): nothing.
let on = connections(&tracker, 0);
let elsewhere = (0..tracker.connections.len())
.find(|connection| !on.contains(connection))
.unwrap();
assert!(tracker.quiet("0x0", elsewhere, now).is_none());
assert!(tracker.quiet("0xunknown", 0, now).is_none());
}
#[test]
fn a_connection_closed_twice_within_ten_minutes_is_split() {
let mut tracker = Tracker::new(1, 7);
+171 -18
View File
@@ -1,6 +1,7 @@
//! A connection: reads the market channel, stamps each frame's receive time,
//! decodes it and sends each event to the book thread of its market; and
//! sends the exchange what the book threads ask for (subscriptions, resyncs).
//! It reconnects when it drops, falls silent, or is recycled.
use std::collections::BTreeSet;
use std::io::ErrorKind;
@@ -29,6 +30,9 @@ const READ_TIMEOUT: Duration = Duration::from_millis(200);
const PING_EVERY: Duration = Duration::from_secs(10);
const RECONNECT_AFTER: Duration = Duration::from_secs(1);
const CLOSE_TIMEOUT: Duration = Duration::from_secs(2);
/// Nothing received this long, not even the answer to three keepalives: the
/// connection is silent, and reconnects.
pub const SILENT_AFTER: Duration = Duration::from_secs(30);
/// Tokens per subscribe or unsubscribe message: a first message of more than
/// 500 gets no opening books (docs/notes.md).
const CHUNK: usize = 500;
@@ -39,27 +43,46 @@ pub enum Command {
Subscribe(Vec<String>),
Unsubscribe(Vec<String>),
/// Unsubscribe and subscribe again: the server then sends a fresh book.
Resync(String),
/// A resync asks for that book; a quiet subscription only for the new
/// subscription.
Resubscribe(String),
/// Close and reconnect, subscribing every token again.
Recycle,
}
/// What a connection tells the orchestrator.
#[derive(Debug, PartialEq, Eq)]
pub enum Notice {
/// Connected, its tokens subscribed.
Up(usize),
Down(usize),
/// Closed by the server with 1013: it couldn't send to us fast enough.
TooSlow(usize),
/// Closed by us: nothing came for [`SILENT_AFTER`].
Silent(usize),
}
type Socket = WebSocket<MaybeTlsStream<TcpStream>>;
/// Runs until `stop` is set or a book thread is gone, reconnecting (and
/// subscribing its tokens again) whenever the connection drops, and telling
/// every book thread when it comes up and goes down. `book_threads` holds
/// every book thread's channel, by number; `closes` hears of each 1013 close.
/// subscribing its tokens again) whenever the connection drops, falls silent
/// for `silent_after` or is recycled, and telling every book thread and the
/// orchestrator (`notices`) when it comes up and goes down. `book_threads`
/// holds every book thread's channel, by number.
pub fn run(
url: &str,
number: usize,
book_threads: &[Sender<Received>],
commands: &Receiver<Command>,
closes: &Sender<usize>,
notices: &Sender<Notice>,
silent_after: Duration,
stop: &AtomicBool,
) {
let tell = |received: fn(usize) -> Received| {
let tell = |received: fn(usize) -> Received, notice: fn(usize) -> Notice| {
for book_thread in book_threads {
_ = book_thread.send(received(number));
}
_ = notices.send(notice(number));
};
let mut tokens = BTreeSet::new();
while !stop.load(Ordering::Relaxed) {
@@ -74,21 +97,23 @@ pub fn run(
"polymarket: connection {number}: connected, {} symbols",
tokens.len()
);
tell(Received::Up);
tell(Received::Up, Notice::Up);
let ended = read(
&mut socket,
number,
&mut tokens,
book_threads,
commands,
closes,
notices,
silent_after,
stop,
);
if !matches!(ended, Ok(Ended::Stopped)) {
tell(Received::Down);
tell(Received::Down, Notice::Down);
}
match ended {
Ok(Ended::Stopped) => return close(&mut socket, number, book_threads),
Ok(Ended::Recycled) => log!("polymarket: connection {number}: recycled"),
Ok(Ended::BookThreadGone) => {
// Only a panic ends a book thread before us: its
// markets would go unserved, so the run stops.
@@ -109,6 +134,7 @@ pub fn run(
enum Ended {
Stopped,
BookThreadGone,
Recycled,
}
/// Updates the token set, and returns the messages that carry `command` out
@@ -127,12 +153,14 @@ fn apply(tokens: &mut BTreeSet<String>, command: &Command) -> Vec<String> {
// Subscribing a token already subscribed sends nothing back but
// `NO NEW ASSETS`: unsubscribing first makes the server send a fresh
// book.
Command::Resync(token) if tokens.contains(token) => [
Command::Resubscribe(token) if tokens.contains(token) => [
operations("unsubscribe", [token]),
operations("subscribe", [token]),
]
.concat(),
Command::Resync(_) => Vec::new(),
Command::Resubscribe(_) => Vec::new(),
// The read loop acts on it; disconnected, the reconnect does.
Command::Recycle => Vec::new(),
}
}
@@ -219,20 +247,27 @@ fn truncate(text: &str) -> &str {
.map_or(text, |(end, _)| &text[..end])
}
// The connection's whole context, each a different part of it.
#[allow(clippy::too_many_arguments)]
fn read(
socket: &mut Socket,
number: usize,
tokens: &mut BTreeSet<String>,
book_threads: &[Sender<Received>],
commands: &Receiver<Command>,
closes: &Sender<usize>,
notices: &Sender<Notice>,
silent_after: Duration,
stop: &AtomicBool,
) -> anyhow::Result<Ended> {
let mut last_ping = Instant::now();
let mut last_heard = Instant::now();
loop {
let message = socket.read();
// Stamp as soon as the frame is off the wire, before any processing.
let received_at = Utc::now();
if message.is_ok() {
last_heard = Instant::now();
}
match message {
Ok(Message::Text(text)) => {
if !route(received_at, text.as_str(), number, book_threads) {
@@ -246,7 +281,7 @@ fn read(
.as_ref()
.is_some_and(|frame| frame.code == CloseCode::Again)
{
_ = closes.send(number);
_ = notices.send(Notice::TooSlow(number));
}
anyhow::bail!("closed by the server: {frame:?}")
}
@@ -258,11 +293,23 @@ fn read(
if stop.load(Ordering::Relaxed) {
return Ok(Ended::Stopped);
}
if last_heard.elapsed() >= silent_after {
_ = notices.send(Notice::Silent(number));
anyhow::bail!("silent for {} s", silent_after.as_secs_f64());
}
if last_ping.elapsed() >= PING_EVERY {
socket.send(Message::text("PING"))?;
last_ping = Instant::now();
}
for command in commands.try_iter() {
if command == Command::Recycle {
// Not waited on: whatever still comes would arrive from a
// connection already down for the book threads. The commands
// behind it are applied before the reconnect.
_ = socket.close(None);
_ = socket.flush();
return Ok(Ended::Recycled);
}
for message in apply(tokens, &command) {
socket.send(Message::text(message))?;
}
@@ -351,9 +398,18 @@ mod tests {
.unwrap();
let stop = Arc::new(AtomicBool::new(false));
let client_stop = Arc::clone(&stop);
let (closes, _closes_rx) = unbounded();
let client =
thread::spawn(move || run(&url, 0, &[events_tx], &commands_rx, &closes, &client_stop));
let (notices, _notices_rx) = unbounded();
let client = thread::spawn(move || {
run(
&url,
0,
&[events_tx],
&commands_rx,
&notices,
SILENT_AFTER,
&client_stop,
)
});
let connections = server.join().unwrap();
stop.store(true, Ordering::Relaxed);
@@ -368,6 +424,103 @@ mod tests {
}
}
/// A local server that accepts two connections of `total` tokens, each
/// handled by `serve` once subscribed, and answers what the client
/// subscribed on each.
fn serving_twice(
total: usize,
serve: fn(&mut WebSocket<TcpStream>),
) -> (String, thread::JoinHandle<Vec<Vec<Vec<String>>>>) {
let listener = TcpListener::bind("127.0.0.1:0").unwrap();
let url = format!("ws://{}/ws/market", listener.local_addr().unwrap());
let server = thread::spawn(move || {
(0..2)
.map(|_| {
let (stream, _) = listener.accept().unwrap();
stream
.set_read_timeout(Some(Duration::from_secs(5)))
.unwrap();
let mut socket = tungstenite::accept(stream).unwrap();
let chunks = subscribed(&mut socket, total);
serve(&mut socket);
chunks
})
.collect()
});
(url, server)
}
/// Runs a connection of `tokens` against `url` until `until` returns,
/// then stops it: the notices it sent.
fn client(
url: String,
tokens: Vec<String>,
silent_after: Duration,
until: impl FnOnce(&Sender<Command>),
) -> Vec<Notice> {
let (events_tx, _events_rx) = unbounded();
let (commands_tx, commands_rx) = unbounded();
commands_tx.send(Command::Subscribe(tokens)).unwrap();
let stop = Arc::new(AtomicBool::new(false));
let client_stop = Arc::clone(&stop);
let (notices, notices_rx) = unbounded();
let client = thread::spawn(move || {
run(
&url,
0,
&[events_tx],
&commands_rx,
&notices,
silent_after,
&client_stop,
)
});
until(&commands_tx);
stop.store(true, Ordering::Relaxed);
client.join().unwrap();
notices_rx.try_iter().collect()
}
#[test]
fn a_silent_connection_reconnects_with_its_tokens() {
let tokens: Vec<String> = (0..10).map(|token| token.to_string()).collect();
// Silent: reads what comes (the keepalives) and answers nothing,
// until the client closes.
let (url, server) = serving_twice(tokens.len(), |socket| while socket.read().is_ok() {});
let notices = client(url, tokens.clone(), Duration::from_millis(500), |_| {
thread::sleep(Duration::from_millis(2_500));
});
let connections = server.join().unwrap();
assert_eq!(connections.len(), 2);
assert!(connections.iter().all(|chunks| chunks.concat() == tokens));
assert_eq!(
notices[..3],
[Notice::Up(0), Notice::Silent(0), Notice::Down(0)]
);
assert_eq!(notices[3], Notice::Up(0));
}
#[test]
fn a_recycle_closes_and_subscribes_every_token_again() {
let tokens: Vec<String> = (0..10).map(|token| token.to_string()).collect();
// Answers the keepalives, as the exchange does, until closed.
let (url, server) = serving_twice(tokens.len(), |socket| {
while let Ok(message) = socket.read() {
if message.is_text() {
_ = socket.send(Message::text("PONG"));
}
}
});
let notices = client(url, tokens.clone(), SILENT_AFTER, |commands| {
thread::sleep(Duration::from_millis(300));
commands.send(Command::Recycle).unwrap();
thread::sleep(Duration::from_millis(1_800));
});
let connections = server.join().unwrap();
assert!(connections.iter().all(|chunks| chunks.concat() == tokens));
assert_eq!(notices, [Notice::Up(0), Notice::Down(0), Notice::Up(0)]);
}
/// The recording's first frame holds the opening books of 40 tokens, 20
/// markets: each book goes to its market's book thread, with its share of
/// the frame's bytes.
@@ -433,7 +586,7 @@ mod tests {
[op("subscribe", &["a", "b"])]
);
assert_eq!(
sent(apply(&mut tokens, &Command::Resync("a".into()))),
sent(apply(&mut tokens, &Command::Resubscribe("a".into()))),
[op("unsubscribe", &["a"]), op("subscribe", &["a"])]
);
let unsubscribe = Command::Unsubscribe(vec!["a".into()]);
@@ -441,7 +594,7 @@ mod tests {
sent(apply(&mut tokens, &unsubscribe)),
[op("unsubscribe", &["a"])]
);
assert!(apply(&mut tokens, &Command::Resync("a".into())).is_empty());
assert!(apply(&mut tokens, &Command::Resubscribe("a".into())).is_empty());
assert_eq!(tokens.into_iter().collect::<Vec<_>>(), ["b"]);
}
}