Archived
feat(polymarket): track every live market over load-balanced lanes
An orchestrator keeps the tracked set equal to Gamma's live markets (a scan at startup and every hour, new and closed markets every minute, reaching back 15 minutes for Gamma's indexing delay) and spreads them over lanes, each a connection and a book thread, all writing through the one writer. Lanes report each market's load in bytes per second; the orchestrator puts new markets on the least-loaded lane and moves the busiest markets off lanes over a 300 KB/s budget, busiest lane first. Differing books our own subscribe explains are subscription gaps, not losses, and every first book after a move is logged, so reconcile's mismatches are all accounted for in the log. A 75-minute live run tracked ~195k markets (~390k symbols) in 89 lanes.
This commit is contained in:
18 files changed
+2826
-320
No files matched your search
@@ -0,0 +1,431 @@
|
||||
# 03. Full coverage: discovery and lanes
|
||||
|
||||
**Status: done (2026-11-11)**
|
||||
|
||||
Track every live Polymarket symbol, which completes the base challenge for one
|
||||
exchange. An orchestrator keeps the tracked set equal to the live set (Gamma,
|
||||
scanned at startup and every hour, checked every minute for new and closed
|
||||
markets) and spreads the symbols over lanes, each a connection and a book
|
||||
thread of its own, all writing through the one writer; it balances the lanes
|
||||
by the load they report. Design:
|
||||
[../docs/project.md](../docs/project.md) (pipeline),
|
||||
[ADR 0002](../docs/adr/0002-lanes-and-orchestrator.md) (lanes, orchestrator),
|
||||
[../docs/notes.md](../docs/notes.md) (protocol facts),
|
||||
[../GLOSSARY.md](../GLOSSARY.md) (live, lane, connection, load, subscription
|
||||
gap).
|
||||
|
||||
## The target, measured (2026-11-04)
|
||||
|
||||
Measured before planning, as sprint 02's retrospective asked; the throwaway
|
||||
scripts read Gamma and the market channel live.
|
||||
|
||||
- **Size.** Gamma lists 238,308 markets active and not closed; 238,201 of
|
||||
them have an order book and accept orders, so ~476k symbols. 20,212 traded
|
||||
in the last 24 h (the day after the US midterms). 32,092 accept orders past
|
||||
their end date, waiting to be resolved. ~20k markets are created a day,
|
||||
mostly crypto "Up or Down" markets created a day ahead.
|
||||
- **Load.** A random sample of 2,500 markets (5,000 tokens) on one connection
|
||||
for 90 s: 4,998 books within 0.45 s; levels per book median 5, mean 10.9,
|
||||
p90 29, max 136, 15% empty; 133 frames/s, 266 changes/s. The top 50,000
|
||||
tokens by volume: ~10,000 frames/s. Estimated for all symbols: 10–25k
|
||||
frames/s, near the ~30k frames/s one book thread did in sprint 02.
|
||||
- **Connections.** No cap on tokens per connection: one served 50,000 (books
|
||||
for 48,798, changes for 27,882 in 40 s). But a first subscribe message of
|
||||
more than 500 tokens gets no opening books (1,000 to 10,000 tried), while
|
||||
subscribe operations of 500 or 1,000 do. A connection opened with no token
|
||||
stays up and serves a later subscribe. The server closes a connection it
|
||||
can't send to fast enough: code 1013, "slow consumer: send buffer full" (3
|
||||
of 5 connections of the Python test reader, which parsed every frame).
|
||||
- **Gamma.** 100 markets a page whatever `limit` says: a full scan of
|
||||
`/markets/keyset` takes 6 min 41 s. Sorting by `createdAt` (live markets) or
|
||||
`closedTime` (`closed=true`) works, and keyset pages keep that order across
|
||||
`after_cursor`; the `*_min` date filters are ignored, and `/markets` refuses
|
||||
offsets past 2,000. 2–20 markets are created and 10–18 closed a minute;
|
||||
more than 420 a minute change `updatedAt`.
|
||||
- **Websocket lifecycle events.** With `custom_feature_enabled`, `new_market`
|
||||
is broadcast for every market, subscribed or not (69 in 5 minutes), with its
|
||||
token ids but not `orderMinSize` or `negRisk`, which the hash needs. No
|
||||
`market_resolved` came in those 5 minutes.
|
||||
- **Estimated output.** Minute Snapshots ~130 MB a minute (476k books of ~275
|
||||
bytes); Deltas ~190 MB a minute (~25k changes/s of ~125 bytes): ~320 MB
|
||||
a minute in all. Books in memory:
|
||||
~1 GB. The machine: 16 GB, 10 cores.
|
||||
|
||||
## The smoke run (2026-11-04)
|
||||
|
||||
An 8-minute live run of the first build, on every live symbol. It changed the
|
||||
plan (Decisions mark what it superseded):
|
||||
|
||||
- **Startup**: the scan took ~2 minutes, not 7 (the Python estimate paid a
|
||||
process per page): 239,752 markets in 96 lanes. ~1.3 GB of memory, 198
|
||||
threads; 3.15 GB of output (~390 MB a minute).
|
||||
- **One lane was closed by the server six times** (1013, slow consumer): lane
|
||||
85, which held NFL quarter-spread markets created together, so filled
|
||||
together. It carried ~5,000 verified changes a second; the next busiest
|
||||
lane ~1,300. Their 24 h volume was small: volume says nothing of load.
|
||||
- **2,758 "losses" and 1,296 resync gaps.** Logged per loss: most were
|
||||
complete books answering a resync, for a symbol whose first check after its
|
||||
opening book had failed; the levels that differed were ones our changes
|
||||
never touched. Our subscribe keeps a few ms of changes from us, as a resync
|
||||
does: a **subscription gap**. The rest were books after lane 85's
|
||||
reconnects, changes really missed while it was down.
|
||||
- **62 `Unverifiable` symbols**: the 31 markets with a null `orderMinSize`
|
||||
(fixed, see Data model).
|
||||
- **Left for later in this sprint**: `reconcile` found 3,315 mismatches for
|
||||
4,054 logged losses and gaps, which should be equal; `tick_size_change`
|
||||
came live for the first time, logged twice per symbol (checked 2026-11-10:
|
||||
the server sends each twice, and the second asks for nothing).
|
||||
|
||||
## Pipeline
|
||||
|
||||
```
|
||||
rescan fetcher ──findings──▶ ┌──────────────┐ ◀──findings── recent fetcher
|
||||
(full Gamma scan) ◀─request─│ orchestrator │──request──▶ (new + closed since T)
|
||||
└──┬────────▲──┘
|
||||
Add / MoveIn / Remove, to │ │ load reports, each minute
|
||||
the market's lane │ │ (bytes/s per market)
|
||||
┌──────────────────────┼────────┴─────────────────┐
|
||||
▼ ▼ ▼
|
||||
lane 0: book thread ◀─frames─ connection lane 1: ... lane N: ...
|
||||
│ └─subscribe / unsubscribe / resync─▶┘
|
||||
└──────────── Snapshot | Delta ─────────────▶ writer thread (one)
|
||||
```
|
||||
|
||||
- **Orchestrator** (one thread): the single owner of what is tracked. It holds
|
||||
the market map (market, by `conditionId` → its lane and tokens), the lanes it
|
||||
opened and how many tokens each holds, and every timer. Each minute it asks
|
||||
the recent fetcher for what changed since its last successful check; every
|
||||
hour it asks the rescan fetcher for a full scan. A request while the
|
||||
previous one of its kind is still running is skipped. It applies the
|
||||
findings: a live market it doesn't know is added to the lane with the
|
||||
least load that still has token room, opening a lane when none has; a
|
||||
closed market it knows is removed. It reads the lanes' load reports and
|
||||
rebalances (below). It waits on the findings and reports with a timeout
|
||||
until its next timer, at most 1 s, so it sees SIGINT. Lane health beyond
|
||||
load (up, quiet, reconnecting) is M4's and M5's.
|
||||
- **Fetchers** (two threads): each answers a request by paging Gamma and
|
||||
streaming what it reads, and holds no state or timer. Two, so a 6 min 41 s
|
||||
scan never holds up the minute check.
|
||||
- **Rescan**: `Started`, every page of live markets, then `Done`, or
|
||||
`Failed` after 5 failed tries at one page, 5 s apart.
|
||||
- **Recent**: the live markets created since T, then the markets closed
|
||||
since T (each in its keyset order, down to T), then `Done` or `Failed`.
|
||||
T is the start of the orchestrator's last successful check, minus 15
|
||||
minutes for Gamma's indexing delay; known markets are skipped, so the
|
||||
margin costs only the pages it reads again.
|
||||
- **Lanes**: a lane is opened by the orchestrator with its first market and
|
||||
never closed. Its book thread waits on its frames, on the orchestrator's
|
||||
commands and on the next wall-clock minute (`crossbeam`'s `select!`). It
|
||||
learns a market (its tokens, `min_order_size`, `neg_risk`) or forgets it,
|
||||
then asks its connection to subscribe or unsubscribe the tokens: the book
|
||||
thread knows a symbol before its book can arrive. At each wall-clock minute
|
||||
it sends the orchestrator its load report. Everything else is sprint 02's
|
||||
book thread.
|
||||
- **Connection** (one per lane): sprint 02's socket thread, now keeping its own
|
||||
token set from the book thread's commands. It opens with an empty
|
||||
subscribe message and subscribes its tokens in operations of at most 500,
|
||||
on every connect and reconnect.
|
||||
- **Writer**: unchanged, fed by every lane's book thread through one channel.
|
||||
|
||||
**Load balancing.** A market's **load** is the bytes per second of its frames,
|
||||
every event type, measured by its lane's book thread over each minute, or
|
||||
since the market joined the lane if that is later; a market measured for
|
||||
under 30 s waits for the next report, since its opening books alone would read
|
||||
as a burst (a `price_change` frame is one market's; a book frame is shared by
|
||||
the books it holds). A lane has two
|
||||
limits: a byte budget, 300 KB/s (under the lowest rate at which a connection
|
||||
was seen to get a 1013 close), and 5,000 tokens, since a reconnect replays
|
||||
every book at once. A new market joins the lane with the least load that still has token
|
||||
room (ties: fewest tokens); at startup nothing is measured yet, so the scan
|
||||
fills lanes in order, and rebalancing undoes the clusters. Each minute, on the
|
||||
reports, every lane over its budget, the busiest lane first, has its busiest
|
||||
markets moved, one by one, to the least-loaded lane with room for them in both
|
||||
limits (a new lane if none has), until it is under budget: at most 50 moves a
|
||||
minute, and a moved market stays put for 10 minutes. A market alone on its
|
||||
lane is never moved: over budget on its own, it would only carry its load from
|
||||
lane to lane; nor is one with no load, which would lighten nothing. A move is
|
||||
a Remove on the old lane, then a MoveIn on the new one: an Add whose lane
|
||||
writes the market's first book as a Snapshot, the changes in between being a
|
||||
subscription gap. The new lane can't tell whether that book differs from the
|
||||
old lane's, so it logs every first book after a move.
|
||||
|
||||
**Subscription gaps.** A fresh book whose levels differ from ours is a
|
||||
subscription gap, not a loss, when it answers our resync (sprint 02's rule:
|
||||
complete, at or after our last change, not behind), or when no change of the
|
||||
symbol has been verified since its opening book; it is logged with its cause
|
||||
(opening, resync). A book after a reconnect that differs stays a loss: those
|
||||
changes were missed while the connection was down.
|
||||
|
||||
**Startup** is the first rescan, retried by the orchestrator until it succeeds:
|
||||
markets are assigned and subscribed page by page, so the first books are live
|
||||
within seconds and lanes open as they fill. The minute checks run from the
|
||||
start, the first covering everything since startup.
|
||||
|
||||
**Removal.** A market is gone when it no longer matches the live predicate:
|
||||
the minute check removes the markets newly `closed`; a completed rescan
|
||||
removes the markets that were known at its `Started` and not seen in it (one
|
||||
that stopped accepting orders, or vanished from the listing). Markets added
|
||||
during a rescan are not at risk, and a failed rescan removes nothing. A
|
||||
removed symbol is unsubscribed and dropped: nothing is written for it, and its
|
||||
frames still in flight are ignored, like any unknown token's.
|
||||
|
||||
**Shutdown.** SIGINT reaches the orchestrator and the connections through the
|
||||
stop flag. Each connection closes, so its book thread drains and ends; the
|
||||
orchestrator drops its sender to the writer, which flushes once every sender
|
||||
is gone. Every Gamma request has a 10 s timeout, so a request in flight delays
|
||||
nothing past that; the fetchers aren't waited for. A fetcher that dies (only a
|
||||
panic ends one) stops the run, which then fails: nothing would keep the
|
||||
tracked set current.
|
||||
|
||||
## Data model
|
||||
|
||||
- **Live** (Gamma): `active`, not `closed`, `enableOrderBook` and
|
||||
`acceptingOrders`. The end date doesn't count. A live market missing a hash
|
||||
input takes what the hash uses (230 had a null `negRisk`: `false`, all
|
||||
verified live; 31 a null `orderMinSize`: `"0"`, as REST's `/book` says; the
|
||||
first guess, `"5"`, made all 62 tokens `Unverifiable` in the smoke run). A
|
||||
wrong value shows as `Unverifiable`, its Deltas still written. A market with no token ids is not tracked.
|
||||
- **Orchestrator**: `conditionId` → (lane, tokens, last reported load, when
|
||||
it last moved); per lane, its token count, its load (the sum of its
|
||||
markets') and its command sender. The market is the unit of assignment,
|
||||
removal and moves: its two tokens always share a lane and leave together.
|
||||
- **Lane limits**: 5,000 tokens, and a byte budget of 300 KB/s (a constant,
|
||||
from the capacity experiment).
|
||||
- **Load report** (book thread → orchestrator, each minute): its lane number
|
||||
and, per market (`conditionId`), its load since the last report.
|
||||
- **Book thread**: per symbol, sprint 02's state, plus whether a change has
|
||||
been verified since its opening book (the subscription gap rule); per
|
||||
market, its bytes since the last report and since when. Memory is measured on the final run; if it
|
||||
passes 4 GB, the recent hashes become `[u8; 20]` instead of `String`.
|
||||
|
||||
## Decisions
|
||||
|
||||
- **Live means Gamma's active, not closed, with an order book and accepting
|
||||
orders: every such symbol, ~476k.** The brief asks for all live symbols, and
|
||||
the measured size is within reach. *Rejected:* only markets with recent
|
||||
trades or liquidity, which misses a market the moment it starts trading.
|
||||
- **Lanes: each a connection and a book thread of its own, one writer for
|
||||
all** (ADR 0002). The symbols are sharded by construction, with no routing
|
||||
stage, and M4 adds connections to a lane. *Rejected:* a router to book
|
||||
shards by symbol (an extra stage, sizes we don't need to decouple yet);
|
||||
several connections into one book thread (at its measured ceiling).
|
||||
- **An orchestrator decides, two fetchers only fetch.** One owner of the
|
||||
market map, the lanes and the timers, which never blocks on HTTP; the room
|
||||
for M4's source switching and M5's recycling. *Rejected:* a discovery thread
|
||||
that both fetches and decides (blocks on Gamma); fetchers with their own
|
||||
timers (two places deciding when to rescan); one fetcher (the scan holds up
|
||||
the minute check); interleaving scan pages with minute checks on one thread
|
||||
(more loop state for the same result).
|
||||
- **The market is keyed by `conditionId`.** The websocket calls it `market`.
|
||||
*Rejected:* token keys (two entries that must never split); Gamma's `id`
|
||||
(the websocket never shows it).
|
||||
- **Lanes are balanced by load, measured as bytes per second per market by
|
||||
the lanes** (supersedes "the least-full lane by token count" and "5,000
|
||||
tokens, halved to 2,500 on a 1013 close", because the smoke run's one
|
||||
closed lane held markets created together, with little volume and ~4 times
|
||||
the next lane's changes: halving would only split the cluster in two).
|
||||
Bytes, because the 1013 close is the server's send buffer filling.
|
||||
*Rejected:* token counts alone (what failed); changes or frames per second
|
||||
(their size varies); Gamma volume (shown wrong); the first lane with room
|
||||
(hot spots); a fixed lane count (doesn't follow growth).
|
||||
- **The byte budget is 300 KB/s, under the lowest rate seen to close one
|
||||
connection (0.46 MB/s); 5,000 tokens stay a second limit** (supersedes
|
||||
"half the rate measured to close one connection", because connections on
|
||||
the server were closed anywhere from 0.46 to 5.1 MB/s: there is no single
|
||||
rate). A 1013 can still come below it; recovering from one is M4's.
|
||||
*Rejected:* a guessed budget; tokens alone; 500 or 350 KB/s (too near the
|
||||
lowest close); 230 KB/s, half of it (more lanes for a rule no rate obeys).
|
||||
- **A new market joins the least-loaded lane with room; rebalancing moves the
|
||||
busiest markets off a lane over budget, busiest lane first, at most 50 a
|
||||
minute, each staying 10 minutes.** Startup clusters can't be seen before
|
||||
they are measured, and the limits stop markets bouncing. Busiest lane first,
|
||||
since the 75-minute run's hottest lane (closed 4 times in 3 minutes) lost 10
|
||||
symbols in 2 minutes while lanes numbered before it took the 50 moves. *Rejected:* round-robin at startup (needs the
|
||||
lane count before the scan ends); weighting by volume; never moving.
|
||||
- **A move is break-before-make: Remove, then MoveIn (an Add).** The new lane's first book
|
||||
is a Snapshot, so the output stays right, and the few ms between are a
|
||||
subscription gap. *Rejected:* make-before-break (two lanes writing one
|
||||
symbol: M4's source switching).
|
||||
- **A market's load is measured from when it joins its lane, and needs 30 s
|
||||
before it is reported** (found by the reviews: over a partial minute it was
|
||||
under-reported, and over the last seconds of one its opening books read as
|
||||
a burst; the 15-minute run's first rebalance made its full 50 moves on
|
||||
them). *Rejected:* a full minute for everyone (a new market weighs nothing
|
||||
for up to two minutes); a 1 s floor (opening books still read as load).
|
||||
- **Lanes send the orchestrator a load report each minute** (supersedes "no
|
||||
lane health reports in M3": load is the first). *Rejected:* shared atomic
|
||||
counters (shared state).
|
||||
- **Subscription gap replaces resync gap: the changes our own subscribe keeps
|
||||
from us, whether it opens, resyncs or moves the symbol** (supersedes sprint
|
||||
02's "resync gap", because the smoke run showed opening subscribes hiding
|
||||
changes the same way). A differing book counts as one when it answers a
|
||||
resync, or when nothing has been verified since the opening book; books
|
||||
after a reconnect stay losses. *Rejected:* a second term for the opening
|
||||
case (two names for one mechanism); leaving them as losses (every loss
|
||||
explained could never pass).
|
||||
- **Minute check by `createdAt` and `closedTime`, both on keyset pages, down
|
||||
to the last check minus 15 minutes; a full rescan every hour** (supersedes
|
||||
"plain pages, a full rescan past offset 2,000", because keyset pages keep
|
||||
the sort and have no offset limit; and "minus 5 minutes", because the
|
||||
75-minute run's rescan added 347 markets the minute checks had missed, and
|
||||
Gamma listed new markets 3.7 to 5 minutes after their `createdAt`). *Rejected:* polling by `updatedAt`
|
||||
(more than 420 a minute); `new_market` events (they lack the hash's
|
||||
metadata, and `market_resolved` never came: Possible features).
|
||||
- **Only a completed rescan removes, and only what it could have seen.** A
|
||||
failed scan would make half the markets look gone, and markets the minute
|
||||
check adds during a scan may sit on pages already passed. *Rejected:*
|
||||
removing whatever a scan didn't list.
|
||||
- **Every first book after a move is logged** (found by the 75-minute run:
|
||||
506 of its 25,621 mismatches had no log line, 468 of them within 3 s of a
|
||||
minute's moves). Only the old lane had the book to compare it with, so the
|
||||
check counts a mismatch at a logged move as its subscription gap.
|
||||
*Rejected:* sending the old lane's book with the move (shared state between
|
||||
lanes, for a log line); leaving the mismatches unexplained.
|
||||
- **A removed symbol writes nothing.** An empty Snapshot would state a book
|
||||
the exchange never sent. *Rejected:* an empty Snapshot.
|
||||
- **Connections open with an empty subscribe and subscribe in operations of
|
||||
at most 500** (supersedes "the first message holds the first 500", because
|
||||
an empty first message was tried and works: one rule, no special first
|
||||
chunk). *Rejected:* larger operations (1,000 worked once; 500 costs
|
||||
nothing).
|
||||
- **Subscribe page by page at startup.** The first books come within
|
||||
seconds. *Rejected:* scan, then subscribe (7 minutes of nothing, then one
|
||||
burst).
|
||||
- **Each lane logs its own minute line**, prefixed with its number, and the
|
||||
orchestrator its own (markets tracked and lanes; added, removed and moved
|
||||
this minute; the busiest lane's load). *Rejected:*
|
||||
one aggregated line (a stats channel and an owner for the totals).
|
||||
- **10 s timeout on every Gamma request.** SIGINT then waits at most that
|
||||
long. *Rejected:* no timeout (a stalled request hangs the shutdown).
|
||||
- **Lanes that drain stay open.** The least-loaded rule refills them first,
|
||||
and a connection with nothing subscribed stays up. *Rejected:* closing
|
||||
empty lanes.
|
||||
|
||||
## Tests
|
||||
|
||||
No network, except the local websocket server below. The 40-symbol replay
|
||||
tests of sprint 02 stay.
|
||||
|
||||
- **Gamma parsing**: a saved keyset page of live markets and one of closed
|
||||
markets parse.
|
||||
- **Minute check**: it stops at T, and skips markets already known.
|
||||
- **Rescan**: a completed one removes only the markets known at `Started`
|
||||
and not seen; one added during it stays; a failed one removes nothing.
|
||||
- **Assignment**: a market joins the least-loaded lane with token room
|
||||
(ties: fewest tokens); a lane opens when none has room; both tokens share a
|
||||
lane.
|
||||
- **Rebalancing**: a lane over budget has its busiest markets moved to the
|
||||
least-loaded lanes with room until it is under, the busiest lane first; no
|
||||
more than 50 moves a minute; a market moved less than 10 minutes ago stays,
|
||||
and one with no load stays; a move is a Remove to the old lane, then a
|
||||
MoveIn to the new one.
|
||||
- **Load report**: a book thread's report holds each market's bytes per
|
||||
second, every event type counted, over the minute or since it joined; one
|
||||
measured under 30 s waits for the next report.
|
||||
- **Subscription gap**: a symbol whose first check after its opening book
|
||||
fails, then gets a differing fresh book, counts a subscription gap, not a
|
||||
loss; one verified since counts a loss.
|
||||
- **Move**: a market moved in logs its symbols' first books, once.
|
||||
- **Feed**: an Add mid-run asks for a subscribe, and the symbol's first book
|
||||
is written as a Snapshot; after a Remove, its frames are ignored and it is
|
||||
out of the minute Snapshots.
|
||||
- **Reconnect**: the connection loop against a local `tungstenite` server
|
||||
(tried: a client reconnects to one inside `cargo test`) that closes the
|
||||
socket once; on reconnect every token is subscribed again, in operations of
|
||||
at most 500.
|
||||
|
||||
## Not in this sprint
|
||||
|
||||
- Closing lanes.
|
||||
- `new_market` / `market_resolved` as a discovery signal.
|
||||
- Shrinking the Snapshot volume (the brief asks for every symbol every minute).
|
||||
- Lane health beyond load, redundant connections, closing subscription gaps,
|
||||
the crash chunk (M4); activity monitoring, recycling (M5).
|
||||
- A market's `min_order_size` or `neg_risk` changing after it was added: read
|
||||
once, a ceiling (a change would make its books fail their own hash).
|
||||
|
||||
## Done when
|
||||
|
||||
- `scripts/ci.sh` is green.
|
||||
- A 75-minute live run on every live symbol, started only once the user says
|
||||
so and ended by SIGINT (startup scan, ~70 minute checks, one hourly rescan),
|
||||
writes a file that reads back with `AlignedSbeReader`, and:
|
||||
- the markets tracked at the end match Gamma's live count, within a
|
||||
minute's churn;
|
||||
- `reconcile`'s mismatches are exactly the logged losses and subscription
|
||||
gaps (and, since the 75-minute run, the logged first books after a move);
|
||||
- every `Unverifiable` symbol is counted and its cause looked into: fixed if
|
||||
the formula is wrong, else explained here;
|
||||
- no connection was closed with 1013, or each is explained;
|
||||
- peak memory (sampled with `ps`) and each lane's peak of frames waiting
|
||||
are reported.
|
||||
- Free disk is checked before the run (~24 GB of output estimated, 39 GB
|
||||
free today), and the output file is deleted after the check.
|
||||
- `docs/notes.md` holds the measured facts above; `docs/project.md`'s
|
||||
pipeline sketch and the roadmap match the lanes and the orchestrator.
|
||||
|
||||
## Outcome
|
||||
|
||||
Every live Polymarket symbol is tracked: ~195k markets (~390k symbols) over
|
||||
85–89 lanes, kept equal to Gamma's live set by the minute checks and the
|
||||
hourly rescan, and balanced by the load the lanes report. The base challenge
|
||||
is met for one exchange.
|
||||
|
||||
**The 75-minute run** (2026-11-10, 16:27–17:42, on the server): startup scan
|
||||
under a minute (189,474 markets in 76 lanes); 194,935 markets tracked at the
|
||||
end against 195,130 live in Gamma ten minutes later; 30.2 GB written (~400 MB a
|
||||
minute), read back whole by `reconcile`: 29.1M Snapshots, 175.6M Deltas, no
|
||||
orphan Delta. 25,115 of its 25,621 mismatches were logged losses and
|
||||
subscription gaps; the other 506 were moved markets, whose first book on the
|
||||
new lane nobody logged (fixed: every first book after a move is logged; a
|
||||
15-minute run then explained all 6,137 of its mismatches). No `Unverifiable`
|
||||
symbol. Peak memory 1.81 GB, 183 threads (sampled from minute 14 on; it grew
|
||||
~90 MB every 10 minutes). Peak frames waiting: 1,295, on lane 68.
|
||||
|
||||
**1013 closes**: 31 in 75 minutes, and 2 resets. Lane 68 took 16 of them,
|
||||
then lane 69: the crypto "Up or Down" markets of 5 and 15 minutes, created
|
||||
together a day ahead and busy together when their windows open and close, so
|
||||
many closes fall on 5-minute marks. The rest came several lanes within a
|
||||
second, which only the server explains. Rebalancing doesn't cure that lane:
|
||||
its load sits in a few bursty markets that stay put 10 minutes once moved, and
|
||||
a minute average hides the bursts that get it closed. Recovery is M4's.
|
||||
|
||||
**Diverged from the plan**, each in Decisions: the byte budget is 300 KB/s,
|
||||
under the lowest close (no closing rate exists to halve); loads are bytes per
|
||||
second over every event type, from when the market joined and after 30 s of
|
||||
measuring; rebalancing takes the
|
||||
busiest lane first and leaves markets with no load; first books after a move
|
||||
are logged; the minute check reaches back 15 minutes, not 5 (Gamma lists new
|
||||
markets up to 5 minutes late; the hourly rescan found 347 markets missed).
|
||||
`tick_size_change` was recorded live: sent twice per token, harmless.
|
||||
|
||||
**Left out**, and where it went: counting a reconnect's differing books as
|
||||
losses (Possible features); delivering a move's Remove before its Add's first
|
||||
Snapshot across lanes (M4, with source switching); reacting to 1013 closes and
|
||||
the burst-driven lane (M4); the memory growth (M4's list).
|
||||
|
||||
### Retrospective
|
||||
|
||||
- **Went well:** measuring before deciding. The capacity ramps overturned the
|
||||
budget rule, a live capture settled `tick_size_change`, a ten-minute poll
|
||||
explained the missed markets. Pairing `reconcile`'s mismatches with the log
|
||||
by symbol and time found the unlogged moves, and the code review found the
|
||||
rebalancing order that starved the hottest lane.
|
||||
- **Went wrong:** the 75-minute run used a binary built before the review's
|
||||
fixes, so a second run was needed to see them live. A first capacity
|
||||
reading was taken for a ceiling from one sample, and the budget was stated
|
||||
as "half the lowest close" before the last closes came in. Wait loops
|
||||
written as `until ! pgrep -f PATTERN` matched their own shell and ran for
|
||||
12 hours; a `pkill -f` stopped its own shell, and the memory sampler watched
|
||||
`timeout` for its first 14 minutes.
|
||||
- **Tested enough?** The rules are, by replay and unit tests. Not covered:
|
||||
the minute check's T and the 5 s retries (inside the orchestrator's loop);
|
||||
the orchestrator loop itself (sending each command, stopping on a dead
|
||||
fetcher), shown only by live runs; a reconnect followed by a loss. The last
|
||||
review's fixes (the 30 s measuring floor, a dead fetcher failing the run)
|
||||
came after the last live run and haven't run live.
|
||||
- **Next time:** review the code before the long acceptance run; wait on a
|
||||
PID, never on a `pgrep -f` pattern (now in CLAUDE.md); the run check
|
||||
(`reconcile` against the log) is needed by every acceptance run: it is the
|
||||
Possible feature "a `reconcile` that reads the collector's log", worth
|
||||
doing early in M4.
|
||||
Reference in new issue
Block a user