This repository has been archived on 2026-11-16. You can view files and clone it. You cannot open issues or pull requests or push a commit.
Files
pascal 8114a51dc2 feat(polymarket): track every live market over load-balanced lanes
An orchestrator keeps the tracked set equal to Gamma's live markets (a scan at
startup and every hour, new and closed markets every minute, reaching back 15
minutes for Gamma's indexing delay) and spreads them over lanes, each a
connection and a book thread, all writing through the one writer. Lanes report
each market's load in bytes per second; the orchestrator puts new markets on
the least-loaded lane and moves the busiest markets off lanes over a 300 KB/s
budget, busiest lane first. Differing books our own subscribe explains are
subscription gaps, not losses, and every first book after a move is logged,
so reconcile's mismatches are all accounted for in the log. A 75-minute live
run tracked ~195k markets (~390k symbols) in 89 lanes.
2026-11-11 06:28:25 +00:00

432 lines
26 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# 03. Full coverage: discovery and lanes
**Status: done (2026-11-11)**
Track every live Polymarket symbol, which completes the base challenge for one
exchange. An orchestrator keeps the tracked set equal to the live set (Gamma,
scanned at startup and every hour, checked every minute for new and closed
markets) and spreads the symbols over lanes, each a connection and a book
thread of its own, all writing through the one writer; it balances the lanes
by the load they report. Design:
[../docs/project.md](../docs/project.md) (pipeline),
[ADR 0002](../docs/adr/0002-lanes-and-orchestrator.md) (lanes, orchestrator),
[../docs/notes.md](../docs/notes.md) (protocol facts),
[../GLOSSARY.md](../GLOSSARY.md) (live, lane, connection, load, subscription
gap).
## The target, measured (2026-11-04)
Measured before planning, as sprint 02's retrospective asked; the throwaway
scripts read Gamma and the market channel live.
- **Size.** Gamma lists 238,308 markets active and not closed; 238,201 of
them have an order book and accept orders, so ~476k symbols. 20,212 traded
in the last 24 h (the day after the US midterms). 32,092 accept orders past
their end date, waiting to be resolved. ~20k markets are created a day,
mostly crypto "Up or Down" markets created a day ahead.
- **Load.** A random sample of 2,500 markets (5,000 tokens) on one connection
for 90 s: 4,998 books within 0.45 s; levels per book median 5, mean 10.9,
p90 29, max 136, 15% empty; 133 frames/s, 266 changes/s. The top 50,000
tokens by volume: ~10,000 frames/s. Estimated for all symbols: 10–25k
frames/s, near the ~30k frames/s one book thread did in sprint 02.
- **Connections.** No cap on tokens per connection: one served 50,000 (books
for 48,798, changes for 27,882 in 40 s). But a first subscribe message of
more than 500 tokens gets no opening books (1,000 to 10,000 tried), while
subscribe operations of 500 or 1,000 do. A connection opened with no token
stays up and serves a later subscribe. The server closes a connection it
can't send to fast enough: code 1013, "slow consumer: send buffer full" (3
of 5 connections of the Python test reader, which parsed every frame).
- **Gamma.** 100 markets a page whatever `limit` says: a full scan of
`/markets/keyset` takes 6 min 41 s. Sorting by `createdAt` (live markets) or
`closedTime` (`closed=true`) works, and keyset pages keep that order across
`after_cursor`; the `*_min` date filters are ignored, and `/markets` refuses
offsets past 2,000. 2–20 markets are created and 10–18 closed a minute;
more than 420 a minute change `updatedAt`.
- **Websocket lifecycle events.** With `custom_feature_enabled`, `new_market`
is broadcast for every market, subscribed or not (69 in 5 minutes), with its
token ids but not `orderMinSize` or `negRisk`, which the hash needs. No
`market_resolved` came in those 5 minutes.
- **Estimated output.** Minute Snapshots ~130 MB a minute (476k books of ~275
bytes); Deltas ~190 MB a minute (~25k changes/s of ~125 bytes): ~320 MB
a minute in all. Books in memory:
~1 GB. The machine: 16 GB, 10 cores.
## The smoke run (2026-11-04)
An 8-minute live run of the first build, on every live symbol. It changed the
plan (Decisions mark what it superseded):
- **Startup**: the scan took ~2 minutes, not 7 (the Python estimate paid a
process per page): 239,752 markets in 96 lanes. ~1.3 GB of memory, 198
threads; 3.15 GB of output (~390 MB a minute).
- **One lane was closed by the server six times** (1013, slow consumer): lane
85, which held NFL quarter-spread markets created together, so filled
together. It carried ~5,000 verified changes a second; the next busiest
lane ~1,300. Their 24 h volume was small: volume says nothing of load.
- **2,758 "losses" and 1,296 resync gaps.** Logged per loss: most were
complete books answering a resync, for a symbol whose first check after its
opening book had failed; the levels that differed were ones our changes
never touched. Our subscribe keeps a few ms of changes from us, as a resync
does: a **subscription gap**. The rest were books after lane 85's
reconnects, changes really missed while it was down.
- **62 `Unverifiable` symbols**: the 31 markets with a null `orderMinSize`
(fixed, see Data model).
- **Left for later in this sprint**: `reconcile` found 3,315 mismatches for
4,054 logged losses and gaps, which should be equal; `tick_size_change`
came live for the first time, logged twice per symbol (checked 2026-11-10:
the server sends each twice, and the second asks for nothing).
## Pipeline
```
rescan fetcher ──findings──▶ ┌──────────────┐ ◀──findings── recent fetcher
(full Gamma scan) ◀─request─│ orchestrator │──request──▶ (new + closed since T)
└──┬────────▲──┘
Add / MoveIn / Remove, to │ │ load reports, each minute
the market's lane │ │ (bytes/s per market)
┌──────────────────────┼────────┴─────────────────┐
▼ ▼ ▼
lane 0: book thread ◀─frames─ connection lane 1: ... lane N: ...
│ └─subscribe / unsubscribe / resync─▶┘
└──────────── Snapshot | Delta ─────────────▶ writer thread (one)
```
- **Orchestrator** (one thread): the single owner of what is tracked. It holds
the market map (market, by `conditionId` → its lane and tokens), the lanes it
opened and how many tokens each holds, and every timer. Each minute it asks
the recent fetcher for what changed since its last successful check; every
hour it asks the rescan fetcher for a full scan. A request while the
previous one of its kind is still running is skipped. It applies the
findings: a live market it doesn't know is added to the lane with the
least load that still has token room, opening a lane when none has; a
closed market it knows is removed. It reads the lanes' load reports and
rebalances (below). It waits on the findings and reports with a timeout
until its next timer, at most 1 s, so it sees SIGINT. Lane health beyond
load (up, quiet, reconnecting) is M4's and M5's.
- **Fetchers** (two threads): each answers a request by paging Gamma and
streaming what it reads, and holds no state or timer. Two, so a 6 min 41 s
scan never holds up the minute check.
- **Rescan**: `Started`, every page of live markets, then `Done`, or
`Failed` after 5 failed tries at one page, 5 s apart.
- **Recent**: the live markets created since T, then the markets closed
since T (each in its keyset order, down to T), then `Done` or `Failed`.
T is the start of the orchestrator's last successful check, minus 15
minutes for Gamma's indexing delay; known markets are skipped, so the
margin costs only the pages it reads again.
- **Lanes**: a lane is opened by the orchestrator with its first market and
never closed. Its book thread waits on its frames, on the orchestrator's
commands and on the next wall-clock minute (`crossbeam`'s `select!`). It
learns a market (its tokens, `min_order_size`, `neg_risk`) or forgets it,
then asks its connection to subscribe or unsubscribe the tokens: the book
thread knows a symbol before its book can arrive. At each wall-clock minute
it sends the orchestrator its load report. Everything else is sprint 02's
book thread.
- **Connection** (one per lane): sprint 02's socket thread, now keeping its own
token set from the book thread's commands. It opens with an empty
subscribe message and subscribes its tokens in operations of at most 500,
on every connect and reconnect.
- **Writer**: unchanged, fed by every lane's book thread through one channel.
**Load balancing.** A market's **load** is the bytes per second of its frames,
every event type, measured by its lane's book thread over each minute, or
since the market joined the lane if that is later; a market measured for
under 30 s waits for the next report, since its opening books alone would read
as a burst (a `price_change` frame is one market's; a book frame is shared by
the books it holds). A lane has two
limits: a byte budget, 300 KB/s (under the lowest rate at which a connection
was seen to get a 1013 close), and 5,000 tokens, since a reconnect replays
every book at once. A new market joins the lane with the least load that still has token
room (ties: fewest tokens); at startup nothing is measured yet, so the scan
fills lanes in order, and rebalancing undoes the clusters. Each minute, on the
reports, every lane over its budget, the busiest lane first, has its busiest
markets moved, one by one, to the least-loaded lane with room for them in both
limits (a new lane if none has), until it is under budget: at most 50 moves a
minute, and a moved market stays put for 10 minutes. A market alone on its
lane is never moved: over budget on its own, it would only carry its load from
lane to lane; nor is one with no load, which would lighten nothing. A move is
a Remove on the old lane, then a MoveIn on the new one: an Add whose lane
writes the market's first book as a Snapshot, the changes in between being a
subscription gap. The new lane can't tell whether that book differs from the
old lane's, so it logs every first book after a move.
**Subscription gaps.** A fresh book whose levels differ from ours is a
subscription gap, not a loss, when it answers our resync (sprint 02's rule:
complete, at or after our last change, not behind), or when no change of the
symbol has been verified since its opening book; it is logged with its cause
(opening, resync). A book after a reconnect that differs stays a loss: those
changes were missed while the connection was down.
**Startup** is the first rescan, retried by the orchestrator until it succeeds:
markets are assigned and subscribed page by page, so the first books are live
within seconds and lanes open as they fill. The minute checks run from the
start, the first covering everything since startup.
**Removal.** A market is gone when it no longer matches the live predicate:
the minute check removes the markets newly `closed`; a completed rescan
removes the markets that were known at its `Started` and not seen in it (one
that stopped accepting orders, or vanished from the listing). Markets added
during a rescan are not at risk, and a failed rescan removes nothing. A
removed symbol is unsubscribed and dropped: nothing is written for it, and its
frames still in flight are ignored, like any unknown token's.
**Shutdown.** SIGINT reaches the orchestrator and the connections through the
stop flag. Each connection closes, so its book thread drains and ends; the
orchestrator drops its sender to the writer, which flushes once every sender
is gone. Every Gamma request has a 10 s timeout, so a request in flight delays
nothing past that; the fetchers aren't waited for. A fetcher that dies (only a
panic ends one) stops the run, which then fails: nothing would keep the
tracked set current.
## Data model
- **Live** (Gamma): `active`, not `closed`, `enableOrderBook` and
`acceptingOrders`. The end date doesn't count. A live market missing a hash
input takes what the hash uses (230 had a null `negRisk`: `false`, all
verified live; 31 a null `orderMinSize`: `"0"`, as REST's `/book` says; the
first guess, `"5"`, made all 62 tokens `Unverifiable` in the smoke run). A
wrong value shows as `Unverifiable`, its Deltas still written. A market with no token ids is not tracked.
- **Orchestrator**: `conditionId` → (lane, tokens, last reported load, when
it last moved); per lane, its token count, its load (the sum of its
markets') and its command sender. The market is the unit of assignment,
removal and moves: its two tokens always share a lane and leave together.
- **Lane limits**: 5,000 tokens, and a byte budget of 300 KB/s (a constant,
from the capacity experiment).
- **Load report** (book thread → orchestrator, each minute): its lane number
and, per market (`conditionId`), its load since the last report.
- **Book thread**: per symbol, sprint 02's state, plus whether a change has
been verified since its opening book (the subscription gap rule); per
market, its bytes since the last report and since when. Memory is measured on the final run; if it
passes 4 GB, the recent hashes become `[u8; 20]` instead of `String`.
## Decisions
- **Live means Gamma's active, not closed, with an order book and accepting
orders: every such symbol, ~476k.** The brief asks for all live symbols, and
the measured size is within reach. *Rejected:* only markets with recent
trades or liquidity, which misses a market the moment it starts trading.
- **Lanes: each a connection and a book thread of its own, one writer for
all** (ADR 0002). The symbols are sharded by construction, with no routing
stage, and M4 adds connections to a lane. *Rejected:* a router to book
shards by symbol (an extra stage, sizes we don't need to decouple yet);
several connections into one book thread (at its measured ceiling).
- **An orchestrator decides, two fetchers only fetch.** One owner of the
market map, the lanes and the timers, which never blocks on HTTP; the room
for M4's source switching and M5's recycling. *Rejected:* a discovery thread
that both fetches and decides (blocks on Gamma); fetchers with their own
timers (two places deciding when to rescan); one fetcher (the scan holds up
the minute check); interleaving scan pages with minute checks on one thread
(more loop state for the same result).
- **The market is keyed by `conditionId`.** The websocket calls it `market`.
*Rejected:* token keys (two entries that must never split); Gamma's `id`
(the websocket never shows it).
- **Lanes are balanced by load, measured as bytes per second per market by
the lanes** (supersedes "the least-full lane by token count" and "5,000
tokens, halved to 2,500 on a 1013 close", because the smoke run's one
closed lane held markets created together, with little volume and ~4 times
the next lane's changes: halving would only split the cluster in two).
Bytes, because the 1013 close is the server's send buffer filling.
*Rejected:* token counts alone (what failed); changes or frames per second
(their size varies); Gamma volume (shown wrong); the first lane with room
(hot spots); a fixed lane count (doesn't follow growth).
- **The byte budget is 300 KB/s, under the lowest rate seen to close one
connection (0.46 MB/s); 5,000 tokens stay a second limit** (supersedes
"half the rate measured to close one connection", because connections on
the server were closed anywhere from 0.46 to 5.1 MB/s: there is no single
rate). A 1013 can still come below it; recovering from one is M4's.
*Rejected:* a guessed budget; tokens alone; 500 or 350 KB/s (too near the
lowest close); 230 KB/s, half of it (more lanes for a rule no rate obeys).
- **A new market joins the least-loaded lane with room; rebalancing moves the
busiest markets off a lane over budget, busiest lane first, at most 50 a
minute, each staying 10 minutes.** Startup clusters can't be seen before
they are measured, and the limits stop markets bouncing. Busiest lane first,
since the 75-minute run's hottest lane (closed 4 times in 3 minutes) lost 10
symbols in 2 minutes while lanes numbered before it took the 50 moves. *Rejected:* round-robin at startup (needs the
lane count before the scan ends); weighting by volume; never moving.
- **A move is break-before-make: Remove, then MoveIn (an Add).** The new lane's first book
is a Snapshot, so the output stays right, and the few ms between are a
subscription gap. *Rejected:* make-before-break (two lanes writing one
symbol: M4's source switching).
- **A market's load is measured from when it joins its lane, and needs 30 s
before it is reported** (found by the reviews: over a partial minute it was
under-reported, and over the last seconds of one its opening books read as
a burst; the 15-minute run's first rebalance made its full 50 moves on
them). *Rejected:* a full minute for everyone (a new market weighs nothing
for up to two minutes); a 1 s floor (opening books still read as load).
- **Lanes send the orchestrator a load report each minute** (supersedes "no
lane health reports in M3": load is the first). *Rejected:* shared atomic
counters (shared state).
- **Subscription gap replaces resync gap: the changes our own subscribe keeps
from us, whether it opens, resyncs or moves the symbol** (supersedes sprint
02's "resync gap", because the smoke run showed opening subscribes hiding
changes the same way). A differing book counts as one when it answers a
resync, or when nothing has been verified since the opening book; books
after a reconnect stay losses. *Rejected:* a second term for the opening
case (two names for one mechanism); leaving them as losses (every loss
explained could never pass).
- **Minute check by `createdAt` and `closedTime`, both on keyset pages, down
to the last check minus 15 minutes; a full rescan every hour** (supersedes
"plain pages, a full rescan past offset 2,000", because keyset pages keep
the sort and have no offset limit; and "minus 5 minutes", because the
75-minute run's rescan added 347 markets the minute checks had missed, and
Gamma listed new markets 3.7 to 5 minutes after their `createdAt`). *Rejected:* polling by `updatedAt`
(more than 420 a minute); `new_market` events (they lack the hash's
metadata, and `market_resolved` never came: Possible features).
- **Only a completed rescan removes, and only what it could have seen.** A
failed scan would make half the markets look gone, and markets the minute
check adds during a scan may sit on pages already passed. *Rejected:*
removing whatever a scan didn't list.
- **Every first book after a move is logged** (found by the 75-minute run:
506 of its 25,621 mismatches had no log line, 468 of them within 3 s of a
minute's moves). Only the old lane had the book to compare it with, so the
check counts a mismatch at a logged move as its subscription gap.
*Rejected:* sending the old lane's book with the move (shared state between
lanes, for a log line); leaving the mismatches unexplained.
- **A removed symbol writes nothing.** An empty Snapshot would state a book
the exchange never sent. *Rejected:* an empty Snapshot.
- **Connections open with an empty subscribe and subscribe in operations of
at most 500** (supersedes "the first message holds the first 500", because
an empty first message was tried and works: one rule, no special first
chunk). *Rejected:* larger operations (1,000 worked once; 500 costs
nothing).
- **Subscribe page by page at startup.** The first books come within
seconds. *Rejected:* scan, then subscribe (7 minutes of nothing, then one
burst).
- **Each lane logs its own minute line**, prefixed with its number, and the
orchestrator its own (markets tracked and lanes; added, removed and moved
this minute; the busiest lane's load). *Rejected:*
one aggregated line (a stats channel and an owner for the totals).
- **10 s timeout on every Gamma request.** SIGINT then waits at most that
long. *Rejected:* no timeout (a stalled request hangs the shutdown).
- **Lanes that drain stay open.** The least-loaded rule refills them first,
and a connection with nothing subscribed stays up. *Rejected:* closing
empty lanes.
## Tests
No network, except the local websocket server below. The 40-symbol replay
tests of sprint 02 stay.
- **Gamma parsing**: a saved keyset page of live markets and one of closed
markets parse.
- **Minute check**: it stops at T, and skips markets already known.
- **Rescan**: a completed one removes only the markets known at `Started`
and not seen; one added during it stays; a failed one removes nothing.
- **Assignment**: a market joins the least-loaded lane with token room
(ties: fewest tokens); a lane opens when none has room; both tokens share a
lane.
- **Rebalancing**: a lane over budget has its busiest markets moved to the
least-loaded lanes with room until it is under, the busiest lane first; no
more than 50 moves a minute; a market moved less than 10 minutes ago stays,
and one with no load stays; a move is a Remove to the old lane, then a
MoveIn to the new one.
- **Load report**: a book thread's report holds each market's bytes per
second, every event type counted, over the minute or since it joined; one
measured under 30 s waits for the next report.
- **Subscription gap**: a symbol whose first check after its opening book
fails, then gets a differing fresh book, counts a subscription gap, not a
loss; one verified since counts a loss.
- **Move**: a market moved in logs its symbols' first books, once.
- **Feed**: an Add mid-run asks for a subscribe, and the symbol's first book
is written as a Snapshot; after a Remove, its frames are ignored and it is
out of the minute Snapshots.
- **Reconnect**: the connection loop against a local `tungstenite` server
(tried: a client reconnects to one inside `cargo test`) that closes the
socket once; on reconnect every token is subscribed again, in operations of
at most 500.
## Not in this sprint
- Closing lanes.
- `new_market` / `market_resolved` as a discovery signal.
- Shrinking the Snapshot volume (the brief asks for every symbol every minute).
- Lane health beyond load, redundant connections, closing subscription gaps,
the crash chunk (M4); activity monitoring, recycling (M5).
- A market's `min_order_size` or `neg_risk` changing after it was added: read
once, a ceiling (a change would make its books fail their own hash).
## Done when
- `scripts/ci.sh` is green.
- A 75-minute live run on every live symbol, started only once the user says
so and ended by SIGINT (startup scan, ~70 minute checks, one hourly rescan),
writes a file that reads back with `AlignedSbeReader`, and:
- the markets tracked at the end match Gamma's live count, within a
minute's churn;
- `reconcile`'s mismatches are exactly the logged losses and subscription
gaps (and, since the 75-minute run, the logged first books after a move);
- every `Unverifiable` symbol is counted and its cause looked into: fixed if
the formula is wrong, else explained here;
- no connection was closed with 1013, or each is explained;
- peak memory (sampled with `ps`) and each lane's peak of frames waiting
are reported.
- Free disk is checked before the run (~24 GB of output estimated, 39 GB
free today), and the output file is deleted after the check.
- `docs/notes.md` holds the measured facts above; `docs/project.md`'s
pipeline sketch and the roadmap match the lanes and the orchestrator.
## Outcome
Every live Polymarket symbol is tracked: ~195k markets (~390k symbols) over
85–89 lanes, kept equal to Gamma's live set by the minute checks and the
hourly rescan, and balanced by the load the lanes report. The base challenge
is met for one exchange.
**The 75-minute run** (2026-11-10, 16:27–17:42, on the server): startup scan
under a minute (189,474 markets in 76 lanes); 194,935 markets tracked at the
end against 195,130 live in Gamma ten minutes later; 30.2 GB written (~400 MB a
minute), read back whole by `reconcile`: 29.1M Snapshots, 175.6M Deltas, no
orphan Delta. 25,115 of its 25,621 mismatches were logged losses and
subscription gaps; the other 506 were moved markets, whose first book on the
new lane nobody logged (fixed: every first book after a move is logged; a
15-minute run then explained all 6,137 of its mismatches). No `Unverifiable`
symbol. Peak memory 1.81 GB, 183 threads (sampled from minute 14 on; it grew
~90 MB every 10 minutes). Peak frames waiting: 1,295, on lane 68.
**1013 closes**: 31 in 75 minutes, and 2 resets. Lane 68 took 16 of them,
then lane 69: the crypto "Up or Down" markets of 5 and 15 minutes, created
together a day ahead and busy together when their windows open and close, so
many closes fall on 5-minute marks. The rest came several lanes within a
second, which only the server explains. Rebalancing doesn't cure that lane:
its load sits in a few bursty markets that stay put 10 minutes once moved, and
a minute average hides the bursts that get it closed. Recovery is M4's.
**Diverged from the plan**, each in Decisions: the byte budget is 300 KB/s,
under the lowest close (no closing rate exists to halve); loads are bytes per
second over every event type, from when the market joined and after 30 s of
measuring; rebalancing takes the
busiest lane first and leaves markets with no load; first books after a move
are logged; the minute check reaches back 15 minutes, not 5 (Gamma lists new
markets up to 5 minutes late; the hourly rescan found 347 markets missed).
`tick_size_change` was recorded live: sent twice per token, harmless.
**Left out**, and where it went: counting a reconnect's differing books as
losses (Possible features); delivering a move's Remove before its Add's first
Snapshot across lanes (M4, with source switching); reacting to 1013 closes and
the burst-driven lane (M4); the memory growth (M4's list).
### Retrospective
- **Went well:** measuring before deciding. The capacity ramps overturned the
budget rule, a live capture settled `tick_size_change`, a ten-minute poll
explained the missed markets. Pairing `reconcile`'s mismatches with the log
by symbol and time found the unlogged moves, and the code review found the
rebalancing order that starved the hottest lane.
- **Went wrong:** the 75-minute run used a binary built before the review's
fixes, so a second run was needed to see them live. A first capacity
reading was taken for a ceiling from one sample, and the budget was stated
as "half the lowest close" before the last closes came in. Wait loops
written as `until ! pgrep -f PATTERN` matched their own shell and ran for
12 hours; a `pkill -f` stopped its own shell, and the memory sampler watched
`timeout` for its first 14 minutes.
- **Tested enough?** The rules are, by replay and unit tests. Not covered:
the minute check's T and the 5 s retries (inside the orchestrator's loop);
the orchestrator loop itself (sending each command, stopping on a dead
fetcher), shown only by live runs; a reconnect followed by a loss. The last
review's fixes (the 30 s measuring floor, a dead fetcher failing the run)
came after the last live run and haven't run live.
- **Next time:** review the code before the long acceptance run; wait on a
PID, never on a `pgrep -f` pattern (now in CLAUDE.md); the run check
(`reconcile` against the log) is needed by every acceptance run: it is the
Possible feature "a `reconcile` that reads the collector's log", worth
doing early in M4.