feat(polymarket): track every live market over load-balanced lanes

An orchestrator keeps the tracked set equal to Gamma's live markets (a scan at
startup and every hour, new and closed markets every minute, reaching back 15
minutes for Gamma's indexing delay) and spreads them over lanes, each a
connection and a book thread, all writing through the one writer. Lanes report
each market's load in bytes per second; the orchestrator puts new markets on
the least-loaded lane and moves the busiest markets off lanes over a 300 KB/s
budget, busiest lane first. Differing books our own subscribe explains are
subscription gaps, not losses, and every first book after a move is logged,
so reconcile's mismatches are all accounted for in the log. A 75-minute live
run tracked ~195k markets (~390k symbols) in 89 lanes.
This commit is contained in:
pascal committed 2026-11-11 06:28:25 +00:00
1 parent adb7fee8d8
commit 8114a51dc2
18 files changed
+2821 -315

No files matched your search

+25 -18
View File
@@ -11,13 +11,17 @@ outside this repo.
reasons behind the design principles. Read before any design discussion.
- `docs/roadmap.md` and `roadmap/NN. name.md`: milestones and each sprint's
agreed plan. Read the current sprint file before writing code.
- `docs/adr/`: hard-to-reverse decisions (0001: threads, shared-nothing).
- `docs/adr/`: hard-to-reverse decisions (0001: threads, shared-nothing;
0002: lanes and the orchestrator).
- `docs/notes.md`: domain rules and exchange protocol facts (delta semantics,
loss detection, message shapes). Read before touching exchange parsing.
- `GLOSSARY.md`: the terms (symbol, resync, loss, resync gap, Snapshot, Delta).
- `src/polymarket/feed.rs`: the book thread and its loss rules (the subtle
part); `src/polymarket/socket.rs`: the socket thread; `src/book.rs`: the
book and fixed point; `src/reconcile.rs` + `src/bin/reconcile.rs`: the output
- `GLOSSARY.md`: the terms (symbol, resync, loss, subscription gap, lane,
load, Snapshot, Delta).
- `src/polymarket/orchestrator.rs`: what is tracked, the lanes and their
balancing; `src/polymarket/gamma.rs`: the fetchers (discovery);
`src/polymarket/feed.rs`: a lane's book thread and its loss rules (the
subtle part); `src/polymarket/socket.rs`: a lane's connection;
`src/book.rs`: the book and fixed point; `src/reconcile.rs` + `src/bin/reconcile.rs`: the output
check. `tests/fixtures/`: a 90 s recording of the live feed the tests replay.
- `src/util/` and `sbe_schemas/`: the template's SBE machinery, used as given so
output stays readable by the template's `AlignedSbeReader`. Our code is the
@@ -45,22 +49,25 @@ and let the user settle it. Claude owns the data model.
## Environment traps
- **Polymarket and Kalshi are DNS-blocked on the dev network**: the local
resolver answers `195.186.4.210` / `2001:4d98:4:b10c::1` (a Swisscom block
page). The user pins the hosts with `sudo ../host.sh on` (workspace root,
outside this repo; `off` removes them). A timeout or the block address again
means the CDN addresses rotated: ask the user to rerun it. A host not yet in
`host.sh`'s `HOSTS` list fails the same way: add it there and ask the user to
rerun it. A 403 is not DNS (see `docs/notes.md`, REST from scripts).
- **DNS: the exchanges resolve normally on the server** (a remote-claude
workspace, since 2026-11-09). The Mac's dev network blocks them (its resolver
answers `195.186.4.210` / `2001:4d98:4:b10c::1`, a Swisscom block page);
there the user pins the hosts with `sudo ../host.sh on` (workspace root,
outside this repo; `off` removes them), rerun when the CDN addresses rotate.
A 403 is not DNS (see `docs/notes.md`, REST from scripts).
- **Write 64 MiB chunks**: `AlignedSbeReader::open` hardcodes 64 MiB
(`open_with_chunk_size` is private), so any other chunk size is unreadable.
- **The collector runs until SIGINT, and only SIGINT flushes its output.**
macOS has no `timeout`, and a `perl alarm` kill (SIGALRM) leaves the last
64 MiB chunk unwritten. Bound a run with SIGINT, then check the output:
`./target/release/engineering-challenge out.sbe 2> run.log & pid=$!;
sleep 600; kill -INT $pid; wait $pid; ./target/release/reconcile out.sbe`.
`reconcile` counts resync gaps as mismatches too: compare its list with the
log's "loss" and "resync gap" lines.
Any other signal (SIGTERM, SIGALRM) leaves the last 64 MiB chunk unwritten.
Bound a run with GNU `timeout -s INT`, then check the output:
`timeout -s INT 600 ./target/release/engineering-challenge out.sbe 2> run.log;
./target/release/reconcile out.sbe`. `reconcile` counts subscription gaps
as mismatches too: compare its list with the log's "loss", "subscription
gap" and "first book after a move" lines.
- **Wait on a PID, never on a `pgrep -f` pattern**: `until ! pgrep -f
"capacity scan"` matches the waiting shell's own command line and never
ends, and `pkill -f PATTERN` can kill the shell running it. Take the PID
(`pgrep -f '^./target/release/...'`, anchored) and loop on `kill -0 $pid`.
- **Live data beats the docs**: Polymarket's market channel has rules its
docs don't state (`docs/notes.md`); check a protocol assumption on a
recording of the real feed (`tests/fixtures/record.py`) before relying on it.
+29 -6
View File
@@ -15,6 +15,12 @@ The unit that has one order book: on Polymarket a token (one outcome of a
market), named by its token id.
_Avoid_: Asset, instrument, ticker (for Polymarket)
**Live**:
Said of a market (and its symbols) the exchange lists as open for trading on
its order book. A market stops being live when it closes or stops accepting
orders, not at its end date.
_Avoid_: Active (Gamma's flag, only one input of being live)
**Book hash**:
The exchange's fingerprint of a symbol's full book state, sent with every
full book and every change.
@@ -26,6 +32,21 @@ The collector's in-memory order book for one symbol: its bid and ask price
levels, each with its absolute quantity.
_Avoid_: Order book state, depth
**Lane**:
A set of symbols whose books one book thread owns, fed by its own
connections: one path from the wire to the writer. Every live symbol is in
exactly one lane.
_Avoid_: Shard (a technique, not this thing), group, partition
**Load**:
How much a market weighs on its connection: the bytes per second of its
frames. A lane's load is the sum of its markets'.
_Avoid_: Activity, volume (the exchange's traded amount, unrelated)
**Connection**:
One websocket to the exchange, carrying the symbols of one lane.
_Avoid_: Socket (for the concept; fine for the code), stream
**Receive time**:
When a message came off the socket, by the local clock.
_Avoid_: Arrival time, local timestamp
@@ -37,14 +58,16 @@ _Avoid_: Gap (implies sequence numbers)
**Resync**:
Asking the exchange for a fresh full book of a symbol and replacing ours with
it, whenever our book can't be verified. Our book was wrong (a loss) only if
the fresh one differs.
it, whenever our book can't be verified. Our book was wrong only if the fresh
one differs: a loss, or a subscription gap.
_Avoid_: Re-snapshot, rebuild, refresh
**Resync gap**:
The changes a symbol's resync itself keeps from us: the exchange holds them in
the fresh book but never sends them as changes.
_Avoid_: calling it a loss (the state stays right; only those updates are missing)
**Subscription gap**:
The changes our own subscribe keeps from us, whether it opens, resyncs or
moves a symbol: the exchange holds them in the book it sends but never sends
them as changes.
_Avoid_: Resync gap (the older, narrower name); calling it a loss (the state
stays right; only those updates are missing)
## Output
+39
View File
@@ -0,0 +1,39 @@
# 0002. Lanes and an orchestrator
## Context
Every live Polymarket symbol is ~476k tokens (2026-11-04), an estimated
10–25k frames/s, near the ~30k frames/s one book thread did in sprint 02. One
connection can carry 50,000 tokens, but the server closes a connection it
can't send to fast enough (1013, "slow consumer"), so connections must be
many and read by threads that do nothing else. The set of live symbols
changes by ~20k markets a day, and Gamma, the only complete source, takes
minutes to scan (~2 from the collector).
## Decision
Symbols are spread over **lanes**: each lane is a connection and a book
thread of its own, within two limits (a byte budget for its load, and 5,000
tokens), and every lane's book thread writes to the one writer. An **orchestrator** thread is the single
owner of what is tracked: the market → lane map, the lanes and the timers. It
adds a new live market to the least-loaded lane, opening one when all are
full, removes markets that are gone, and moves the busiest markets off a lane
over its budget, from the load each lane reports every minute. Two fetcher threads only fetch Gamma (the
full scan; the new and closed markets since a time) and stream what they read
to it.
## Why
A lane shards the book state by construction: no routing stage between
connections and book threads, nothing shared, and redundancy (M4) adds
connections to a lane rather than reshaping the pipeline. A router to book
shards would decouple connection and book thread counts, which nothing needs
yet. The orchestrator keeps every decision in one thread that never blocks on
HTTP, which is where M4's source switching and M5's recycling will live; the
fetchers stay stateless so a scan never holds up the minute check.
Trade-off: a move unsubscribes a market on one lane before subscribing it on
the other, so it costs a few ms of its changes (a subscription gap); moving it
without that gap means two lanes writing one symbol, which is M4's source
switching. Moves are rate-limited so balancing doesn't churn. Balancing by
token count alone was tried first: the first live run had one lane closed by
the server six times, its markets created together and busy together.
+71 -5
View File
@@ -100,15 +100,81 @@ From a 60 s run of the M1 spike on one market, both tokens subscribed, and the
answers `PONG`. The spike survived 60 s without it, but it is required.
- **Server restarts:** the server can close the socket with code `Away`,
reason `server shutting down` (seen once in about 8 minutes of connections).
- **`last_trade_price`:** `asset_id`, `price`, `size`, `side`, `fee_rate_bps`,
`timestamp`, `transaction_hash` (one per trade, for one token).
- **Not observed:** `tick_size_change` and the custom-feature events. The docs
now show these only in an SDK's wrapped form (`topic` / `payload`,
camelCase), not the raw wire shape above.
- **`last_trade_price`:** `market`, `asset_id`, `price`, `size`, `side`,
`fee_rate_bps`, `timestamp`, `transaction_hash` (one per trade, for one
token).
- **Not observed:** `market_resolved`, in 5 minutes with
`custom_feature_enabled` on 10 tokens. The docs now show it only in an
SDK's wrapped form (`topic` / `payload`, camelCase), not the raw wire
shape above.
- **`new_market` is broadcast** (with `custom_feature_enabled`, 2026-11-04):
every market created, subscribed or not (69 in 5 minutes), with
`condition_id`, `assets_ids` / `clob_token_ids`, `question`, `active`
(`false` for markets created ahead), but no `order_min_size` or `neg_risk`,
which the book hash needs.
- **Tokens per connection (2026-11-04):** no cap seen: one connection served
50,000 tokens. But a first `{"type":"market"}` message with more than 500
tokens gets no opening books (1,000 to 10,000 tried; changes still flow),
while `subscribe` operations of 500 or 1,000 get theirs. A connection opened
with `"assets_ids": []` stays up and serves a later `subscribe`.
- **Slow consumers are closed:** a connection the server can't send to fast
enough is closed with code 1013, reason `slow consumer: send buffer full`
(3 of 5 test connections, with a Python reader parsing every frame at up to
~10,000 frames/s). The collector's reader, which only stamps, got it too:
six times in 8 minutes on the one connection carrying ~5,000 changes a
second (NFL quarter spreads, little traded); none on the others, the next
at ~1,300. No rate is safe (2026-11-10, server, 5 ms from the edge): four
connections ramped on random live tokens were closed at 0.7, 2.1, 2.3 and
5.1 MB/s (minute averages of frame bytes), with subscribes of 5,000 or 500
tokens alike. 5,000 random tokens now carry ~380 frames/s, ~200 KB/s.
Closes come together: 5 of 20 connections (~1 MB/s each) were closed
within 45 s of each other, one at 0.46 MB/s.
- **A subscribe hides a few ms of changes** (smoke run, 2026-11-04): after
subscribing a market on a busy connection, the first check of its opening
book often failed, and the fresh book differed from ours at levels none of
our changes had touched, though the opening book matched its own hash. The
server never sent those changes: like the resync's unsubscribe/subscribe,
any subscribe opens this gap (glossary: subscription gap).
- **`tick_size_change`** (recorded 2026-11-10 on every live token: 62 markets
in 10 minutes, all `0.01` to `0.001`): one object per token, both tokens of
the market, each sent twice on the same connection, identical:
`{"market", "asset_id", "old_tick_size", "new_tick_size", "timestamp",
"event_type": "tick_size_change"}`.
- **Load (2026-11-04):** a random 5,000 of the live tokens gave 133 frames/s
and 266 changes/s; their books had a median of 5 levels (mean 10.9, p90 29,
max 136), 15% empty. The top 50,000 by volume gave ~10,000 frames/s.
- **REST from scripts:** `clob.polymarket.com` answers Python's `urllib` with
403 (curl and `ureq` get 200), most likely Cloudflare filtering by user agent.
Fetch with curl in throwaway scripts.
## Gamma, the market listing (observed 2026-11-04)
- **Size:** 238,308 markets `active` and not `closed`; 238,201 of them with
`enableOrderBook` and `acceptingOrders` (~476k tokens). 32,092 still accept
orders past their `endDate`, waiting to be resolved: the end date doesn't
mean closed. ~20k markets are created a day, mostly crypto "Up or Down"
markets created a day ahead and accepting orders at once.
- **Pages of 100**, whatever `limit` asks. `/markets` refuses offsets past
2,000 ("use /markets/keyset"); `/markets/keyset` pages with `after_cursor`
(the `next_cursor` of the last page) and has no such limit. A full keyset
scan of the live markets took 6 min 41 s.
- **Sorting works, date filters don't:** `order=createdAt&ascending=false`
(newest first) and `closed=true&order=closedTime&ascending=false` sort both
endpoints, and keyset pages keep the order across cursors. `created_at_min`
and `closed_time_min` are ignored. 2–20 markets were created and 10–18
closed a minute; more than 420 a minute changed `updatedAt`.
- **`conditionId`** is the websocket's `market`; `clobTokenIds` (a JSON array
inside a string) are its symbols.
- **Null hash inputs:** a null `orderMinSize` (31 live markets) hashes as
`"0"`, what REST's `/book` reports; a null `negRisk` (230) as `false`.
- **Scan speed:** the full keyset scan took 6 min 41 s from a Python script
spawning curl per page, ~2 minutes from the collector (`ureq`, one agent),
under a minute from the server (2026-11-10).
- **Indexing delay (2026-11-10):** new markets appear in the newest-first
listing 219 to 296 s after their `createdAt` (8 markets, polled every
20 s): a check reaching back 5 minutes misses some. 195,130 markets were
live that evening.
## Clock accuracy bounds time accuracy
`received_at` comes from the local clock, so it carries the clock's error. On
2026-11-04 the dev machine ran ~30 ms slow (`sntp`: +30.6 ms ± 13 ms), which
+36 -19
View File
@@ -42,8 +42,10 @@ The interacting decisions it is made of:
([ADR 0001](adr/0001-concurrency-model.md)).
- **Stamping point:** in the socket-read path, immediately, so as little as
possible sits between the wire and the receive timestamp.
- **Book ownership:** shard symbols so each book is owned by exactly one worker
(lock-free) rather than a shared, locked map.
- **Book ownership:** settled in sprint 03 — symbols are spread over lanes,
each a connection and a book thread of its own, so each book is owned by
exactly one thread; an orchestrator assigns them
([ADR 0002](adr/0002-lanes-and-orchestrator.md)).
- **Backpressure:** bounded vs unbounded channels — the core correctness/perf
tension. Drop-on-full loses data (violates completeness); block adds latency
and fills socket buffers. This decision is the bet in miniature.
@@ -130,25 +132,37 @@ and drop from the global workflow for a job-challenge repo.
- Any exchange beyond Polymarket and Kalshi.
- Persistence/replay infrastructure beyond what the SBE output already gives.
## Pipeline, first sketch
## Pipeline
The primary thing we debate sprint by sprint (this project's equivalent of the
schema debate). Rough shape:
schema debate). As settled in sprint 03:
```
discovery ──▶ subscription manager ──▶ N socket readers (stamp receive time here)
│
route by symbol
▼
per-symbol book workers (fold, loss-detect, dedup sources)
│
▼
single output writer (SBE sink)
rescan fetcher ──findings──▶ ┌──────────────┐ ◀──findings── recent fetcher
(full Gamma scan) ◀─request─│ orchestrator │──request──▶ (new + closed since T)
└──┬────────▲──┘
Add / MoveIn / Remove, to │ │ load reports, each minute
the market's lane │ │ (bytes/s per market)
┌──────────────────────┼────────┴─────────────────┐
▼ ▼ ▼
lane 0: book thread ◀─frames─ connection lane 1: ... lane N: ...
│ └─subscribe / unsubscribe / resync─▶┘
└──────────── Snapshot | Delta ─────────────▶ writer thread (one)
```
Each stage and the channels between them are open until settled in a sprint.
Settled so far: OS threads, shared-nothing, no mutexes
([ADR 0001](adr/0001-concurrency-model.md)).
- The **orchestrator** owns what is tracked (market → lane, the lanes, the
timers) and decides; it never blocks on HTTP, so later coordination
(switching sources in M4, recycling in M5) lives there too.
- The two **fetchers** only page Gamma on request and stream what they read.
- Each **lane** is a connection (stamps receive time, sends subscriptions and
resyncs) and a book thread (fold, check, resync, Snapshots); M4 gives a lane
several connections and has its book thread choose the trusted source.
- One **writer** owns the output.
Settled: OS threads, shared-nothing, no mutexes
([ADR 0001](adr/0001-concurrency-model.md)); lanes and the orchestrator
([ADR 0002](adr/0002-lanes-and-orchestrator.md)). Details:
[sprint 03](../roadmap/03.%20full%20coverage.md).
## Data model, first sketch
@@ -167,7 +181,13 @@ Not a DB — an in-memory domain plus the fixed SBE output schema. Main entities
- **Book** (one per symbol): bids and asks as ordered price→quantity maps, plus
the last loss-detection marker seen, last exchange timestamp, and the id of the
source currently trusted (for redundancy).
- **Connection** (one websocket): the set of symbols it carries.
- **Connection** (one websocket): the set of symbols it carries, those of its
lane.
- **Lane**: a set of symbols, the book thread that owns their books, and the
connections that feed it. Live markets are assigned to lanes by
`conditionId`; Polymarket's lifecycle signal is Gamma (settled in sprint 03:
a market is live while active, not closed, with an order book and accepting
orders).
- **Source / redundancy:** several connections may feed the same symbol; one is
the trusted source at a time.
@@ -181,9 +201,6 @@ trusted source is chosen and switched, and emitting a fresh snapshot on switch).
- **Loss detection on Kalshi.** Its sequence semantics: checked in M6.
(Polymarket: settled in M2, by its book hash.)
- **Symbol lifecycle signal.** What marks a market "no longer trading"?
Polymarket's `new_market` / `market_resolved` events (with
`custom_feature_enabled`) are candidates; settle in M3.
- **Redundancy source-selection.** How to pick/switch the trusted source
(freshness, message rate, which source is ahead). Designed in the redundancy
sprint.
+33 -22
View File
@@ -30,33 +30,35 @@ abstraction is *extracted* once a second exchange exists, not designed up front.
Sprint: [../roadmap/02. book pipeline.md](../roadmap/02.%20book%20pipeline.md).
*Ends:* correct SBE output for a symbol subset; tests green.
3. **Full coverage: discovery + sharding (base challenge complete)** — *not started*
Discovery with diffing (subscribe new, unsubscribe gone) — REST polling, or
Polymarket's `new_market` / `market_resolved` events — using subscribe /
unsubscribe on open connections; load-balance all live regular-market symbols
across N sockets.
**Opening questions (from M2):** what "live" means, and so how many symbols:
Gamma lists at least 100,000 markets `active` and not `closed` (2026-11-04,
count capped), at least 13,000 of them traded in the last 24 h, against the
40 symbols M2 runs. At that size the minute Snapshots alone (~2 KB each)
would be ~0.5 TB a day for 200,000 symbols, mostly unchanged books; and
memory (~10 KB a book), sockets per connection, resyncs per minute and the
book thread's throughput (M2: ~30,000 frames/s on one thread, 43 frames/s
of load) all need measuring. Gamma pages past 10,000 markets only through
`/markets/keyset` with `after_cursor`.
3. **Full coverage: discovery + lanes (base challenge complete)** — *done (2026-11-11)*
Every live symbol (~476k tokens, 2026-11-04) spread over lanes, each a
connection and a book thread of its own; an orchestrator keeps the tracked
set equal to Gamma's live set (a scan at startup and every hour, new and
closed markets every minute) through subscribe / unsubscribe on open
connections.
Sprint: [../roadmap/03. full coverage.md](../roadmap/03.%20full%20coverage.md).
*Ends:* every live Polymarket symbol tracked — base challenge satisfied for
one exchange.
4. **Extension 1 — redundant websockets** — *not started*
Multiple connections per symbol; choose a trusted source and switch on
loss/quiet/delay; emit a fresh snapshot on every source switch.
Multiple connections per symbol (several connections in a lane, its book
thread choosing among them, the orchestrator coordinating); choose a
trusted source and switch on loss/quiet/delay; emit a fresh snapshot on
every source switch. Recovery from errors, with the lane budget scaling
itself down when connections still get 1013 closes (no rate is safe:
[notes.md](notes.md), so a budget set too high must correct itself).
**Async re-evaluation checkpoint:** before building this, reassess whether the
socket-management complexity (reconnect/timeout/select, redundancy) warrants
swapping the thread-only runtime for tokio (see ADR 0001).
The lane that holds markets busy together (the 5- and 15-minute crypto
markets: closed 16 times in 75 minutes, in bursts a minute's load
average doesn't show) is the case to beat.
Also: surviving a hard crash without losing the writer's in-memory chunk,
and closing the resync gap (the few ms of a symbol's changes the server
doesn't send between our unsubscribe and subscribe), both accepted as
ceilings in M2.
a move's Remove reaching the old lane before the new lane's first
Snapshot, the memory growth seen in sprint 03 (~90 MB every 10 minutes
over 75 minutes), and closing the subscription gap (the few ms of a
symbol's changes the server doesn't send between our unsubscribe and
subscribe); the crash chunk and the gap were accepted as ceilings in M2.
*Ends:* redundancy demonstrably reduces data loss; a Snapshot on every source
switch.
@@ -106,7 +108,16 @@ abstraction is *extracted* once a second exchange exists, not designed up front.
- Observability: metrics, structured logging, dashboards.
- Config-driven symbol filtering and per-exchange tuning.
- Containerized packaging.
- Counting a reconnect's differing books as losses rather than resync gaps
(a reconnect marker through the frames channel; M2 labels them gaps).
- A `reconcile` that reads the collector's log, to tell resync gaps from
- Counting a reconnect's differing books as losses rather than subscription
gaps when the symbol was resyncing or not yet verified since its opening
book (a reconnect marker through the frames channel; they are labelled
gaps).
- A `reconcile` that reads the collector's log, to tell subscription gaps from
losses in its own report.
- Polymarket's `new_market` websocket event as a faster discovery signal (it
is broadcast, but lacks `orderMinSize` / `negRisk`, so Gamma is still
needed; `market_resolved` was never seen).
- Closing lanes that drain.
- Fewer Snapshots for unchanged books (the brief asks for every symbol every
minute: ~130 MB a minute at full coverage).
- Re-reading a market's `orderMinSize` / `negRisk` if they can change.
+431
View File
@@ -0,0 +1,431 @@
# 03. Full coverage: discovery and lanes
**Status: done (2026-11-11)**
Track every live Polymarket symbol, which completes the base challenge for one
exchange. An orchestrator keeps the tracked set equal to the live set (Gamma,
scanned at startup and every hour, checked every minute for new and closed
markets) and spreads the symbols over lanes, each a connection and a book
thread of its own, all writing through the one writer; it balances the lanes
by the load they report. Design:
[../docs/project.md](../docs/project.md) (pipeline),
[ADR 0002](../docs/adr/0002-lanes-and-orchestrator.md) (lanes, orchestrator),
[../docs/notes.md](../docs/notes.md) (protocol facts),
[../GLOSSARY.md](../GLOSSARY.md) (live, lane, connection, load, subscription
gap).
## The target, measured (2026-11-04)
Measured before planning, as sprint 02's retrospective asked; the throwaway
scripts read Gamma and the market channel live.
- **Size.** Gamma lists 238,308 markets active and not closed; 238,201 of
them have an order book and accept orders, so ~476k symbols. 20,212 traded
in the last 24 h (the day after the US midterms). 32,092 accept orders past
their end date, waiting to be resolved. ~20k markets are created a day,
mostly crypto "Up or Down" markets created a day ahead.
- **Load.** A random sample of 2,500 markets (5,000 tokens) on one connection
for 90 s: 4,998 books within 0.45 s; levels per book median 5, mean 10.9,
p90 29, max 136, 15% empty; 133 frames/s, 266 changes/s. The top 50,000
tokens by volume: ~10,000 frames/s. Estimated for all symbols: 10–25k
frames/s, near the ~30k frames/s one book thread did in sprint 02.
- **Connections.** No cap on tokens per connection: one served 50,000 (books
for 48,798, changes for 27,882 in 40 s). But a first subscribe message of
more than 500 tokens gets no opening books (1,000 to 10,000 tried), while
subscribe operations of 500 or 1,000 do. A connection opened with no token
stays up and serves a later subscribe. The server closes a connection it
can't send to fast enough: code 1013, "slow consumer: send buffer full" (3
of 5 connections of the Python test reader, which parsed every frame).
- **Gamma.** 100 markets a page whatever `limit` says: a full scan of
`/markets/keyset` takes 6 min 41 s. Sorting by `createdAt` (live markets) or
`closedTime` (`closed=true`) works, and keyset pages keep that order across
`after_cursor`; the `*_min` date filters are ignored, and `/markets` refuses
offsets past 2,000. 2–20 markets are created and 10–18 closed a minute;
more than 420 a minute change `updatedAt`.
- **Websocket lifecycle events.** With `custom_feature_enabled`, `new_market`
is broadcast for every market, subscribed or not (69 in 5 minutes), with its
token ids but not `orderMinSize` or `negRisk`, which the hash needs. No
`market_resolved` came in those 5 minutes.
- **Estimated output.** Minute Snapshots ~130 MB a minute (476k books of ~275
bytes); Deltas ~190 MB a minute (~25k changes/s of ~125 bytes): ~320 MB
a minute in all. Books in memory:
~1 GB. The machine: 16 GB, 10 cores.
## The smoke run (2026-11-04)
An 8-minute live run of the first build, on every live symbol. It changed the
plan (Decisions mark what it superseded):
- **Startup**: the scan took ~2 minutes, not 7 (the Python estimate paid a
process per page): 239,752 markets in 96 lanes. ~1.3 GB of memory, 198
threads; 3.15 GB of output (~390 MB a minute).
- **One lane was closed by the server six times** (1013, slow consumer): lane
85, which held NFL quarter-spread markets created together, so filled
together. It carried ~5,000 verified changes a second; the next busiest
lane ~1,300. Their 24 h volume was small: volume says nothing of load.
- **2,758 "losses" and 1,296 resync gaps.** Logged per loss: most were
complete books answering a resync, for a symbol whose first check after its
opening book had failed; the levels that differed were ones our changes
never touched. Our subscribe keeps a few ms of changes from us, as a resync
does: a **subscription gap**. The rest were books after lane 85's
reconnects, changes really missed while it was down.
- **62 `Unverifiable` symbols**: the 31 markets with a null `orderMinSize`
(fixed, see Data model).
- **Left for later in this sprint**: `reconcile` found 3,315 mismatches for
4,054 logged losses and gaps, which should be equal; `tick_size_change`
came live for the first time, logged twice per symbol (checked 2026-11-10:
the server sends each twice, and the second asks for nothing).
## Pipeline
```
rescan fetcher ──findings──▶ ┌──────────────┐ ◀──findings── recent fetcher
(full Gamma scan) ◀─request─│ orchestrator │──request──▶ (new + closed since T)
└──┬────────▲──┘
Add / MoveIn / Remove, to │ │ load reports, each minute
the market's lane │ │ (bytes/s per market)
┌──────────────────────┼────────┴─────────────────┐
▼ ▼ ▼
lane 0: book thread ◀─frames─ connection lane 1: ... lane N: ...
│ └─subscribe / unsubscribe / resync─▶┘
└──────────── Snapshot | Delta ─────────────▶ writer thread (one)
```
- **Orchestrator** (one thread): the single owner of what is tracked. It holds
the market map (market, by `conditionId` → its lane and tokens), the lanes it
opened and how many tokens each holds, and every timer. Each minute it asks
the recent fetcher for what changed since its last successful check; every
hour it asks the rescan fetcher for a full scan. A request while the
previous one of its kind is still running is skipped. It applies the
findings: a live market it doesn't know is added to the lane with the
least load that still has token room, opening a lane when none has; a
closed market it knows is removed. It reads the lanes' load reports and
rebalances (below). It waits on the findings and reports with a timeout
until its next timer, at most 1 s, so it sees SIGINT. Lane health beyond
load (up, quiet, reconnecting) is M4's and M5's.
- **Fetchers** (two threads): each answers a request by paging Gamma and
streaming what it reads, and holds no state or timer. Two, so a 6 min 41 s
scan never holds up the minute check.
- **Rescan**: `Started`, every page of live markets, then `Done`, or
`Failed` after 5 failed tries at one page, 5 s apart.
- **Recent**: the live markets created since T, then the markets closed
since T (each in its keyset order, down to T), then `Done` or `Failed`.
T is the start of the orchestrator's last successful check, minus 15
minutes for Gamma's indexing delay; known markets are skipped, so the
margin costs only the pages it reads again.
- **Lanes**: a lane is opened by the orchestrator with its first market and
never closed. Its book thread waits on its frames, on the orchestrator's
commands and on the next wall-clock minute (`crossbeam`'s `select!`). It
learns a market (its tokens, `min_order_size`, `neg_risk`) or forgets it,
then asks its connection to subscribe or unsubscribe the tokens: the book
thread knows a symbol before its book can arrive. At each wall-clock minute
it sends the orchestrator its load report. Everything else is sprint 02's
book thread.
- **Connection** (one per lane): sprint 02's socket thread, now keeping its own
token set from the book thread's commands. It opens with an empty
subscribe message and subscribes its tokens in operations of at most 500,
on every connect and reconnect.
- **Writer**: unchanged, fed by every lane's book thread through one channel.
**Load balancing.** A market's **load** is the bytes per second of its frames,
every event type, measured by its lane's book thread over each minute, or
since the market joined the lane if that is later; a market measured for
under 30 s waits for the next report, since its opening books alone would read
as a burst (a `price_change` frame is one market's; a book frame is shared by
the books it holds). A lane has two
limits: a byte budget, 300 KB/s (under the lowest rate at which a connection
was seen to get a 1013 close), and 5,000 tokens, since a reconnect replays
every book at once. A new market joins the lane with the least load that still has token
room (ties: fewest tokens); at startup nothing is measured yet, so the scan
fills lanes in order, and rebalancing undoes the clusters. Each minute, on the
reports, every lane over its budget, the busiest lane first, has its busiest
markets moved, one by one, to the least-loaded lane with room for them in both
limits (a new lane if none has), until it is under budget: at most 50 moves a
minute, and a moved market stays put for 10 minutes. A market alone on its
lane is never moved: over budget on its own, it would only carry its load from
lane to lane; nor is one with no load, which would lighten nothing. A move is
a Remove on the old lane, then a MoveIn on the new one: an Add whose lane
writes the market's first book as a Snapshot, the changes in between being a
subscription gap. The new lane can't tell whether that book differs from the
old lane's, so it logs every first book after a move.
**Subscription gaps.** A fresh book whose levels differ from ours is a
subscription gap, not a loss, when it answers our resync (sprint 02's rule:
complete, at or after our last change, not behind), or when no change of the
symbol has been verified since its opening book; it is logged with its cause
(opening, resync). A book after a reconnect that differs stays a loss: those
changes were missed while the connection was down.
**Startup** is the first rescan, retried by the orchestrator until it succeeds:
markets are assigned and subscribed page by page, so the first books are live
within seconds and lanes open as they fill. The minute checks run from the
start, the first covering everything since startup.
**Removal.** A market is gone when it no longer matches the live predicate:
the minute check removes the markets newly `closed`; a completed rescan
removes the markets that were known at its `Started` and not seen in it (one
that stopped accepting orders, or vanished from the listing). Markets added
during a rescan are not at risk, and a failed rescan removes nothing. A
removed symbol is unsubscribed and dropped: nothing is written for it, and its
frames still in flight are ignored, like any unknown token's.
**Shutdown.** SIGINT reaches the orchestrator and the connections through the
stop flag. Each connection closes, so its book thread drains and ends; the
orchestrator drops its sender to the writer, which flushes once every sender
is gone. Every Gamma request has a 10 s timeout, so a request in flight delays
nothing past that; the fetchers aren't waited for. A fetcher that dies (only a
panic ends one) stops the run, which then fails: nothing would keep the
tracked set current.
## Data model
- **Live** (Gamma): `active`, not `closed`, `enableOrderBook` and
`acceptingOrders`. The end date doesn't count. A live market missing a hash
input takes what the hash uses (230 had a null `negRisk`: `false`, all
verified live; 31 a null `orderMinSize`: `"0"`, as REST's `/book` says; the
first guess, `"5"`, made all 62 tokens `Unverifiable` in the smoke run). A
wrong value shows as `Unverifiable`, its Deltas still written. A market with no token ids is not tracked.
- **Orchestrator**: `conditionId` → (lane, tokens, last reported load, when
it last moved); per lane, its token count, its load (the sum of its
markets') and its command sender. The market is the unit of assignment,
removal and moves: its two tokens always share a lane and leave together.
- **Lane limits**: 5,000 tokens, and a byte budget of 300 KB/s (a constant,
from the capacity experiment).
- **Load report** (book thread → orchestrator, each minute): its lane number
and, per market (`conditionId`), its load since the last report.
- **Book thread**: per symbol, sprint 02's state, plus whether a change has
been verified since its opening book (the subscription gap rule); per
market, its bytes since the last report and since when. Memory is measured on the final run; if it
passes 4 GB, the recent hashes become `[u8; 20]` instead of `String`.
## Decisions
- **Live means Gamma's active, not closed, with an order book and accepting
orders: every such symbol, ~476k.** The brief asks for all live symbols, and
the measured size is within reach. *Rejected:* only markets with recent
trades or liquidity, which misses a market the moment it starts trading.
- **Lanes: each a connection and a book thread of its own, one writer for
all** (ADR 0002). The symbols are sharded by construction, with no routing
stage, and M4 adds connections to a lane. *Rejected:* a router to book
shards by symbol (an extra stage, sizes we don't need to decouple yet);
several connections into one book thread (at its measured ceiling).
- **An orchestrator decides, two fetchers only fetch.** One owner of the
market map, the lanes and the timers, which never blocks on HTTP; the room
for M4's source switching and M5's recycling. *Rejected:* a discovery thread
that both fetches and decides (blocks on Gamma); fetchers with their own
timers (two places deciding when to rescan); one fetcher (the scan holds up
the minute check); interleaving scan pages with minute checks on one thread
(more loop state for the same result).
- **The market is keyed by `conditionId`.** The websocket calls it `market`.
*Rejected:* token keys (two entries that must never split); Gamma's `id`
(the websocket never shows it).
- **Lanes are balanced by load, measured as bytes per second per market by
the lanes** (supersedes "the least-full lane by token count" and "5,000
tokens, halved to 2,500 on a 1013 close", because the smoke run's one
closed lane held markets created together, with little volume and ~4 times
the next lane's changes: halving would only split the cluster in two).
Bytes, because the 1013 close is the server's send buffer filling.
*Rejected:* token counts alone (what failed); changes or frames per second
(their size varies); Gamma volume (shown wrong); the first lane with room
(hot spots); a fixed lane count (doesn't follow growth).
- **The byte budget is 300 KB/s, under the lowest rate seen to close one
connection (0.46 MB/s); 5,000 tokens stay a second limit** (supersedes
"half the rate measured to close one connection", because connections on
the server were closed anywhere from 0.46 to 5.1 MB/s: there is no single
rate). A 1013 can still come below it; recovering from one is M4's.
*Rejected:* a guessed budget; tokens alone; 500 or 350 KB/s (too near the
lowest close); 230 KB/s, half of it (more lanes for a rule no rate obeys).
- **A new market joins the least-loaded lane with room; rebalancing moves the
busiest markets off a lane over budget, busiest lane first, at most 50 a
minute, each staying 10 minutes.** Startup clusters can't be seen before
they are measured, and the limits stop markets bouncing. Busiest lane first,
since the 75-minute run's hottest lane (closed 4 times in 3 minutes) lost 10
symbols in 2 minutes while lanes numbered before it took the 50 moves. *Rejected:* round-robin at startup (needs the
lane count before the scan ends); weighting by volume; never moving.
- **A move is break-before-make: Remove, then MoveIn (an Add).** The new lane's first book
is a Snapshot, so the output stays right, and the few ms between are a
subscription gap. *Rejected:* make-before-break (two lanes writing one
symbol: M4's source switching).
- **A market's load is measured from when it joins its lane, and needs 30 s
before it is reported** (found by the reviews: over a partial minute it was
under-reported, and over the last seconds of one its opening books read as
a burst; the 15-minute run's first rebalance made its full 50 moves on
them). *Rejected:* a full minute for everyone (a new market weighs nothing
for up to two minutes); a 1 s floor (opening books still read as load).
- **Lanes send the orchestrator a load report each minute** (supersedes "no
lane health reports in M3": load is the first). *Rejected:* shared atomic
counters (shared state).
- **Subscription gap replaces resync gap: the changes our own subscribe keeps
from us, whether it opens, resyncs or moves the symbol** (supersedes sprint
02's "resync gap", because the smoke run showed opening subscribes hiding
changes the same way). A differing book counts as one when it answers a
resync, or when nothing has been verified since the opening book; books
after a reconnect stay losses. *Rejected:* a second term for the opening
case (two names for one mechanism); leaving them as losses (every loss
explained could never pass).
- **Minute check by `createdAt` and `closedTime`, both on keyset pages, down
to the last check minus 15 minutes; a full rescan every hour** (supersedes
"plain pages, a full rescan past offset 2,000", because keyset pages keep
the sort and have no offset limit; and "minus 5 minutes", because the
75-minute run's rescan added 347 markets the minute checks had missed, and
Gamma listed new markets 3.7 to 5 minutes after their `createdAt`). *Rejected:* polling by `updatedAt`
(more than 420 a minute); `new_market` events (they lack the hash's
metadata, and `market_resolved` never came: Possible features).
- **Only a completed rescan removes, and only what it could have seen.** A
failed scan would make half the markets look gone, and markets the minute
check adds during a scan may sit on pages already passed. *Rejected:*
removing whatever a scan didn't list.
- **Every first book after a move is logged** (found by the 75-minute run:
506 of its 25,621 mismatches had no log line, 468 of them within 3 s of a
minute's moves). Only the old lane had the book to compare it with, so the
check counts a mismatch at a logged move as its subscription gap.
*Rejected:* sending the old lane's book with the move (shared state between
lanes, for a log line); leaving the mismatches unexplained.
- **A removed symbol writes nothing.** An empty Snapshot would state a book
the exchange never sent. *Rejected:* an empty Snapshot.
- **Connections open with an empty subscribe and subscribe in operations of
at most 500** (supersedes "the first message holds the first 500", because
an empty first message was tried and works: one rule, no special first
chunk). *Rejected:* larger operations (1,000 worked once; 500 costs
nothing).
- **Subscribe page by page at startup.** The first books come within
seconds. *Rejected:* scan, then subscribe (7 minutes of nothing, then one
burst).
- **Each lane logs its own minute line**, prefixed with its number, and the
orchestrator its own (markets tracked and lanes; added, removed and moved
this minute; the busiest lane's load). *Rejected:*
one aggregated line (a stats channel and an owner for the totals).
- **10 s timeout on every Gamma request.** SIGINT then waits at most that
long. *Rejected:* no timeout (a stalled request hangs the shutdown).
- **Lanes that drain stay open.** The least-loaded rule refills them first,
and a connection with nothing subscribed stays up. *Rejected:* closing
empty lanes.
## Tests
No network, except the local websocket server below. The 40-symbol replay
tests of sprint 02 stay.
- **Gamma parsing**: a saved keyset page of live markets and one of closed
markets parse.
- **Minute check**: it stops at T, and skips markets already known.
- **Rescan**: a completed one removes only the markets known at `Started`
and not seen; one added during it stays; a failed one removes nothing.
- **Assignment**: a market joins the least-loaded lane with token room
(ties: fewest tokens); a lane opens when none has room; both tokens share a
lane.
- **Rebalancing**: a lane over budget has its busiest markets moved to the
least-loaded lanes with room until it is under, the busiest lane first; no
more than 50 moves a minute; a market moved less than 10 minutes ago stays,
and one with no load stays; a move is a Remove to the old lane, then a
MoveIn to the new one.
- **Load report**: a book thread's report holds each market's bytes per
second, every event type counted, over the minute or since it joined; one
measured under 30 s waits for the next report.
- **Subscription gap**: a symbol whose first check after its opening book
fails, then gets a differing fresh book, counts a subscription gap, not a
loss; one verified since counts a loss.
- **Move**: a market moved in logs its symbols' first books, once.
- **Feed**: an Add mid-run asks for a subscribe, and the symbol's first book
is written as a Snapshot; after a Remove, its frames are ignored and it is
out of the minute Snapshots.
- **Reconnect**: the connection loop against a local `tungstenite` server
(tried: a client reconnects to one inside `cargo test`) that closes the
socket once; on reconnect every token is subscribed again, in operations of
at most 500.
## Not in this sprint
- Closing lanes.
- `new_market` / `market_resolved` as a discovery signal.
- Shrinking the Snapshot volume (the brief asks for every symbol every minute).
- Lane health beyond load, redundant connections, closing subscription gaps,
the crash chunk (M4); activity monitoring, recycling (M5).
- A market's `min_order_size` or `neg_risk` changing after it was added: read
once, a ceiling (a change would make its books fail their own hash).
## Done when
- `scripts/ci.sh` is green.
- A 75-minute live run on every live symbol, started only once the user says
so and ended by SIGINT (startup scan, ~70 minute checks, one hourly rescan),
writes a file that reads back with `AlignedSbeReader`, and:
- the markets tracked at the end match Gamma's live count, within a
minute's churn;
- `reconcile`'s mismatches are exactly the logged losses and subscription
gaps (and, since the 75-minute run, the logged first books after a move);
- every `Unverifiable` symbol is counted and its cause looked into: fixed if
the formula is wrong, else explained here;
- no connection was closed with 1013, or each is explained;
- peak memory (sampled with `ps`) and each lane's peak of frames waiting
are reported.
- Free disk is checked before the run (~24 GB of output estimated, 39 GB
free today), and the output file is deleted after the check.
- `docs/notes.md` holds the measured facts above; `docs/project.md`'s
pipeline sketch and the roadmap match the lanes and the orchestrator.
## Outcome
Every live Polymarket symbol is tracked: ~195k markets (~390k symbols) over
85–89 lanes, kept equal to Gamma's live set by the minute checks and the
hourly rescan, and balanced by the load the lanes report. The base challenge
is met for one exchange.
**The 75-minute run** (2026-11-10, 16:27–17:42, on the server): startup scan
under a minute (189,474 markets in 76 lanes); 194,935 markets tracked at the
end against 195,130 live in Gamma ten minutes later; 30.2 GB written (~400 MB a
minute), read back whole by `reconcile`: 29.1M Snapshots, 175.6M Deltas, no
orphan Delta. 25,115 of its 25,621 mismatches were logged losses and
subscription gaps; the other 506 were moved markets, whose first book on the
new lane nobody logged (fixed: every first book after a move is logged; a
15-minute run then explained all 6,137 of its mismatches). No `Unverifiable`
symbol. Peak memory 1.81 GB, 183 threads (sampled from minute 14 on; it grew
~90 MB every 10 minutes). Peak frames waiting: 1,295, on lane 68.
**1013 closes**: 31 in 75 minutes, and 2 resets. Lane 68 took 16 of them,
then lane 69: the crypto "Up or Down" markets of 5 and 15 minutes, created
together a day ahead and busy together when their windows open and close, so
many closes fall on 5-minute marks. The rest came several lanes within a
second, which only the server explains. Rebalancing doesn't cure that lane:
its load sits in a few bursty markets that stay put 10 minutes once moved, and
a minute average hides the bursts that get it closed. Recovery is M4's.
**Diverged from the plan**, each in Decisions: the byte budget is 300 KB/s,
under the lowest close (no closing rate exists to halve); loads are bytes per
second over every event type, from when the market joined and after 30 s of
measuring; rebalancing takes the
busiest lane first and leaves markets with no load; first books after a move
are logged; the minute check reaches back 15 minutes, not 5 (Gamma lists new
markets up to 5 minutes late; the hourly rescan found 347 markets missed).
`tick_size_change` was recorded live: sent twice per token, harmless.
**Left out**, and where it went: counting a reconnect's differing books as
losses (Possible features); delivering a move's Remove before its Add's first
Snapshot across lanes (M4, with source switching); reacting to 1013 closes and
the burst-driven lane (M4); the memory growth (M4's list).
### Retrospective
- **Went well:** measuring before deciding. The capacity ramps overturned the
budget rule, a live capture settled `tick_size_change`, a ten-minute poll
explained the missed markets. Pairing `reconcile`'s mismatches with the log
by symbol and time found the unlogged moves, and the code review found the
rebalancing order that starved the hottest lane.
- **Went wrong:** the 75-minute run used a binary built before the review's
fixes, so a second run was needed to see them live. A first capacity
reading was taken for a ceiling from one sample, and the budget was stated
as "half the lowest close" before the last closes came in. Wait loops
written as `until ! pgrep -f PATTERN` matched their own shell and ran for
12 hours; a `pkill -f` stopped its own shell, and the memory sampler watched
`timeout` for its first 14 minutes.
- **Tested enough?** The rules are, by replay and unit tests. Not covered:
the minute check's T and the 5 s retries (inside the orchestrator's loop);
the orchestrator loop itself (sending each command, stopping on a dead
fetcher), shown only by live runs; a reconnect followed by a loss. The last
review's fixes (the 30 s measuring floor, a dead fetcher failing the run)
came after the last live run and haven't run live.
- **Next time:** review the code before the long acceptance run; wait on a
PID, never on a `pgrep -f` pattern (now in CLAUDE.md); the run check
(`reconcile` against the log) is needed by every acceptance run: it is the
Possible feature "a `reconcile` that reads the collector's log", worth
doing early in M4.
+20 -27
View File
@@ -1,5 +1,7 @@
//! The collector: three threads joined by channels (roadmap/02):
//! socket (read, stamp) → book (parse, fold, check, resync) → writer (SBE).
//! The collector (roadmap/03): an orchestrator keeps the tracked markets equal
//! to Gamma's live ones and spreads them over lanes, each a connection (read,
//! stamp) and a book thread (parse, fold, check, resync), all writing through
//! one writer (SBE).
//! Usage: `engineering-challenge [OUTPUT]`, OUTPUT defaulting to stdout; logs
//! go to stderr.
@@ -20,54 +22,45 @@ macro_rules! log {
mod polymarket;
use engineering_challenge::book::Output;
use polymarket::feed::Feed;
/// `AlignedSbeReader::open` reads nothing else.
const CHUNK: usize = 64 << 20;
const MARKETS: u32 = 20;
fn main() -> anyhow::Result<()> {
let output = std::env::args()
.nth(1)
.unwrap_or_else(|| "/dev/stdout".to_string());
let markets = polymarket::top_markets(MARKETS)?;
let feed = Feed::new(&markets);
let tokens = feed.tokens();
let stop = Arc::new(AtomicBool::new(false));
let on_sigint = Arc::clone(&stop);
ctrlc::set_handler(move || on_sigint.store(true, Ordering::Relaxed))?;
let (frames_tx, frames_rx) = unbounded();
let (resyncs_tx, resyncs_rx) = unbounded();
let (outputs_tx, outputs_rx) = unbounded();
// Each stage stops when the one before it does, so SIGINT only reaches the
// socket thread; the writer flushes whatever came before.
let socket =
thread::spawn(move || polymarket::socket::run(&tokens, &frames_tx, &resyncs_rx, &stop));
let book =
thread::spawn(move || polymarket::feed::run(feed, frames_rx, outputs_tx, resyncs_tx));
let writer = thread::spawn(move || write(&output, outputs_rx));
// Every thread is joined, so the writer flushes even after a panic
// upstream; the run still fails then.
let socket_panicked = socket.join().is_err();
let book_panicked = book.join().is_err();
let writer_stop = Arc::clone(&stop);
let writer = thread::spawn(move || {
let written = write(&output, outputs_rx);
// A writer that failed stops the rest: nothing it reads is kept.
writer_stop.store(true, Ordering::Relaxed);
written
});
// SIGINT stops the orchestrator and the connections; each book thread
// stops when its connection does, and the writer flushes once they all
// have, even after a panic upstream (the run still fails then).
let failed = polymarket::orchestrator::run(outputs_tx, &stop);
let written = writer
.join()
.map_err(|_| anyhow::anyhow!("the writer thread panicked"))?;
log!("output closed");
written?;
anyhow::ensure!(!socket_panicked, "the socket thread panicked");
anyhow::ensure!(!book_panicked, "the book thread panicked");
anyhow::ensure!(failed == 0, "{failed} threads failed");
Ok(())
}
/// The writer thread: writes outputs in arrival order until the book thread
/// is gone, then flushes the last chunk.
/// The writer thread: writes outputs in arrival order until every book
/// thread is gone, then flushes the last chunk.
// ponytail: the chunk being filled lives only in memory until it is full, so
// a hard crash loses it (hours of data); flushing more often pads every flush
// to 64 MiB. Taken up in M4.
// a hard crash loses it; flushing more often pads every flush to 64 MiB.
// Taken up in M4.
fn write(path: &str, outputs: Receiver<Output>) -> anyhow::Result<()> {
let mut writer = AlignedSbeWriter::create(path, CHUNK)?;
for output in outputs {
+538 -101
View File
@@ -1,32 +1,58 @@
//! The book thread: parses market-channel frames, folds them into one book per
//! symbol, checks them against Polymarket's book hash, resyncs when a check
//! fails, and emits Deltas and Snapshots. The rules are in
//! `roadmap/02. book pipeline.md`; the hash formula in `docs/notes.md`.
//! fails, and emits Deltas and Snapshots. The protocol facts and the hash
//! formula are in `docs/notes.md`, the terms (loss, resync, subscription gap,
//! load) in `GLOSSARY.md`.
use std::collections::{BTreeMap, VecDeque};
use std::collections::{BTreeMap, HashMap, VecDeque};
use std::time::Duration;
use chrono::{DateTime, DurationRound, TimeDelta, Utc};
use crossbeam::channel::{Receiver, RecvTimeoutError, Sender};
use crossbeam::channel::{Receiver, Sender, never, select};
use engineering_challenge::Side;
use engineering_challenge::util::dtos::challenge_orderbook::PriceLevelUpdateDomain;
use serde::{Deserialize, Serialize};
use sha1::{Digest, Sha1};
use super::Market;
use super::socket::Frame;
use super::socket::{Command, Frame};
use engineering_challenge::book::{Book, Output, format_fixed, parse_fixed};
/// A market measured for less than this waits for the next load report: over
/// a few seconds, its opening books alone would read as a burst.
const MIN_MEASURED: TimeDelta = TimeDelta::seconds(30);
/// How far back a full book is recognised as behind our stream: the lag seen
/// live is a few ms, a handful of changes.
const RECENT_CHANGES: usize = 64;
/// What the book thread does with a frame: write output, or ask the socket
/// thread for a resync.
/// What the book thread does with a frame or a command: write output, ask its
/// connection for something, or report its load to the orchestrator.
#[derive(Debug)]
pub enum Effect {
Write(Output),
Resync(String),
Send(Command),
Report(Loads),
}
/// Each market's load (`conditionId` → bytes per second), every market of the
/// lane measured for at least [`MIN_MEASURED`].
pub type Loads = HashMap<String, u64>;
/// A lane's loads since its last report, for the orchestrator.
#[derive(Debug)]
pub struct LoadReport {
pub lane: usize,
pub loads: Loads,
}
/// What the orchestrator tells a lane's book thread.
#[derive(Debug)]
pub enum LaneCommand {
Add(Market),
/// An Add for a market moved here from another lane.
MoveIn(Market),
Remove(Market),
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
@@ -34,7 +60,7 @@ enum State {
/// No book yet since subscribing: changes are dropped until it comes.
AwaitingBook,
/// Its last full book matched its hash: every change is checked.
Live,
Verified,
/// A check failed and a fresh book is asked for: changes are applied and
/// written unchecked until it comes. After a trade, it usually equals ours.
Resyncing,
@@ -53,6 +79,12 @@ struct Symbol {
/// Its last [`RECENT_CHANGES`] changes: a full book carrying the hash of
/// one of them is the exchange's state after it.
recent_changes: VecDeque<RecentChange>,
/// Whether a change has been verified since its opening book: until one
/// is, a differing fresh book is the gap our subscribe opened.
verified_since_book: bool,
/// Moved in from another lane, no book yet: its first book holds the
/// move's subscription gap, which only the old lane's book would show.
moved_in: bool,
// The other inputs of the book hash, as the exchange writes them.
market: String,
tick_size: String,
@@ -62,6 +94,30 @@ struct Symbol {
}
impl Symbol {
fn new(market: &Market) -> Self {
Self {
state: State::AwaitingBook,
book: Book::default(),
exchange_timestamp: DateTime::UNIX_EPOCH,
open_batch: None,
recent_changes: VecDeque::new(),
verified_since_book: false,
moved_in: false,
market: String::new(),
tick_size: String::new(),
last_trade_price: String::new(),
min_order_size: market.min_order_size.clone(),
neg_risk: market.neg_risk,
}
}
/// A change matched the exchange's hash: counted in `verified_changes`,
/// and the book was right since its opening book.
fn verified(&mut self, verified_changes: &mut u64) {
*verified_changes += 1;
self.verified_since_book = true;
}
/// Asks for a fresh book, unless one is already coming.
fn resync(&mut self, token: &str, effects: &mut Vec<Effect>) {
if matches!(self.state, State::AwaitingBook | State::Resyncing) {
@@ -77,7 +133,7 @@ impl Symbol {
self.state = State::Resyncing;
}
self.open_batch = None;
effects.push(Effect::Resync(token.to_string()));
effects.push(Effect::Send(Command::Resync(token.to_string())));
}
/// Whether `book` is ours as it was after one of our recent changes: ours
@@ -102,6 +158,29 @@ impl Symbol {
}
}
/// A market's frame bytes since `since`.
struct Measure {
bytes: u64,
since: DateTime<Utc>,
}
impl Measure {
/// Bytes per second since `since`, then starts again from `now`; none
/// before [`MIN_MEASURED`], the bytes kept for the next report.
fn take(&mut self, now: DateTime<Utc>) -> Option<u64> {
let measured = now - self.since;
if measured < MIN_MEASURED {
return None;
}
let load = self.bytes * 1_000 / measured.num_milliseconds() as u64;
*self = Measure {
bytes: 0,
since: now,
};
Some(load)
}
}
/// A change as applied, with the exchange's hash of the book after it.
struct RecentChange {
side: Side,
@@ -113,17 +192,25 @@ struct RecentChange {
}
pub struct Feed {
/// Its lane's number, for the logs.
lane: usize,
symbols: BTreeMap<String, Symbol>,
/// Bytes of frames per market since its last load report.
measures: HashMap<String, Measure>,
/// The receive time of the last frame applied: the time a Snapshot's book
/// is exact as of.
last_received_at: Option<DateTime<Utc>>,
next_snapshots: Option<DateTime<Utc>>,
/// The next wall-clock minute: its Snapshots, load report and log line.
next_minute_at: Option<DateTime<Utc>>,
verified_changes: u64,
resyncs: u64,
/// Full books that were a state we had already passed (see `on_book`).
books_behind: u64,
/// Fresh books ahead of our stream by changes our resync kept from us.
resync_gaps: u64,
/// Fresh books holding changes our own subscribe kept from us.
subscription_gaps: u64,
/// First books of markets moved in, each logged: whether one holds a
/// subscription gap only the old lane's book would tell.
books_after_moves: u64,
/// Full books whose levels differed from ours otherwise.
losses: u64,
/// The most frames waiting for the book thread since the last minute log:
@@ -132,42 +219,56 @@ pub struct Feed {
}
impl Feed {
pub fn new(markets: &[Market]) -> Self {
let symbols = markets
.iter()
.flat_map(|market| {
market.tokens.iter().map(|token| {
let symbol = Symbol {
state: State::AwaitingBook,
book: Book::default(),
exchange_timestamp: DateTime::UNIX_EPOCH,
open_batch: None,
recent_changes: VecDeque::new(),
market: String::new(),
tick_size: String::new(),
last_trade_price: String::new(),
min_order_size: market.min_order_size.clone(),
neg_risk: market.neg_risk,
};
(token.clone(), symbol)
})
})
.collect();
pub fn new(lane: usize) -> Self {
Self {
symbols,
lane,
symbols: BTreeMap::new(),
measures: HashMap::new(),
last_received_at: None,
next_snapshots: None,
next_minute_at: None,
verified_changes: 0,
resyncs: 0,
books_behind: 0,
resync_gaps: 0,
subscription_gaps: 0,
books_after_moves: 0,
losses: 0,
peak_waiting: 0,
}
}
pub fn tokens(&self) -> Vec<String> {
self.symbols.keys().cloned().collect()
/// Tracks a market's symbols, then asks the connection to subscribe them:
/// a book can't arrive for a symbol not yet known. Its load is measured
/// from `now`.
pub fn add_market(&mut self, market: &Market, now: DateTime<Utc>) -> Vec<Effect> {
for token in &market.tokens {
self.symbols.insert(token.clone(), Symbol::new(market));
}
let measure = Measure {
bytes: 0,
since: now,
};
self.measures.insert(market.condition_id.clone(), measure);
vec![Effect::Send(Command::Subscribe(market.tokens.clone()))]
}
/// [`Feed::add_market`] for a market moved in from another lane.
pub fn move_in(&mut self, market: &Market, now: DateTime<Utc>) -> Vec<Effect> {
let effects = self.add_market(market, now);
for token in &market.tokens {
self.symbols.get_mut(token).expect("just added").moved_in = true;
}
effects
}
/// Forgets a market's symbols and unsubscribes them; nothing is written
/// for them. Their frames still on the way are ignored, as any unknown
/// token's.
pub fn remove_market(&mut self, market: &Market) -> Vec<Effect> {
for token in &market.tokens {
self.symbols.remove(token);
}
self.measures.remove(&market.condition_id);
vec![Effect::Send(Command::Unsubscribe(market.tokens.clone()))]
}
pub fn on_frame(&mut self, received_at: DateTime<Utc>, text: &str) -> Vec<Effect> {
@@ -186,51 +287,74 @@ impl Feed {
}
},
};
// A frame is one market's change, or books of several: each book takes
// its share.
let share = (text.len() / events.len().max(1)) as u64;
for event in events {
let market = match &event {
Event::Book(book) => Some(&book.market),
Event::PriceChange(change) => Some(&change.market),
Event::TickSizeChange { market, .. } | Event::LastTradePrice { market } => {
market.as_ref()
}
Event::Other => None,
};
if let Some(measure) = market.and_then(|market| self.measures.get_mut(market)) {
measure.bytes += share;
}
match event {
Event::Book(book) => self.on_book(received_at, book, &mut effects),
Event::PriceChange(change) => {
self.on_price_change(received_at, change, &mut effects)
}
Event::TickSizeChange { asset_id } => {
Event::TickSizeChange { asset_id, .. } => {
if let Some(symbol) = self.symbols.get_mut(&asset_id) {
log!("polymarket: {asset_id}: tick size changed, resync");
symbol.resync(&asset_id, &mut effects);
}
}
// Among them `last_trade_price`: the hash takes a new last
// trade price before this event, and not always its value, so
// a fresh book brings it instead (docs/notes.md).
Event::Other => {}
// The hash takes a new last trade price before this event,
// and not always its value, so a fresh book brings it instead
// (docs/notes.md).
Event::LastTradePrice { .. } | Event::Other => {}
}
}
self.resyncs += count_resyncs(&effects[ticked..]);
effects
}
/// Once `now` reaches the next wall-clock minute: emits the minute
/// Snapshots, and asks again for the fresh books still not come.
/// Once `now` reaches the next wall-clock minute: reports the minute's
/// load, emits the minute Snapshots, and asks again for the fresh books
/// still not come.
pub fn tick(&mut self, now: DateTime<Utc>) -> Vec<Effect> {
let next = *self.next_snapshots.get_or_insert_with(|| next_minute(now));
let next = *self.next_minute_at.get_or_insert_with(|| next_minute(now));
if now < next {
return Vec::new();
}
self.next_snapshots = Some(next_minute(now));
self.next_minute_at = Some(next_minute(now));
let loads = self
.measures
.iter_mut()
.filter_map(|(market, measure)| Some((market.clone(), measure.take(now)?)))
.collect();
let mut effects = vec![Effect::Report(loads)];
let Some(received_at) = self.last_received_at else {
return Vec::new();
return effects;
};
log!(
"polymarket: so far {} changes verified, {} resyncs, {} full books behind ours, \
{} resync gaps, {} losses; at most {} frames waited this minute",
"polymarket: lane {}: {} symbols; so far {} changes verified, {} resyncs, {} full books behind ours, \
{} subscription gaps, {} books after moves, {} losses; at most {} frames waited this minute",
self.lane,
self.symbols.len(),
self.verified_changes,
self.resyncs,
self.books_behind,
self.resync_gaps,
self.subscription_gaps,
self.books_after_moves,
self.losses,
self.peak_waiting
);
self.peak_waiting = 0;
let mut effects = Vec::new();
for (token, symbol) in &mut self.symbols {
if symbol.state != State::AwaitingBook {
effects.push(Effect::Write(symbol.book.snapshot(
@@ -249,8 +373,8 @@ impl Feed {
/// A full book replaces ours, brought forward first if it is behind our
/// stream, and is written as a Snapshot when it is the symbol's first
/// since subscribing, or its levels differ from ours (a resync gap or a
/// loss). Then it is checked against its own hash, as received.
/// since subscribing, or its levels differ from ours (a subscription gap
/// or a loss). Then it is checked against its own hash, as received.
fn on_book(
&mut self,
received_at: DateTime<Utc>,
@@ -285,6 +409,13 @@ impl Feed {
let verified = hash(&token, symbol, &book, &message.timestamp) == message.hash;
let first = symbol.state == State::AwaitingBook;
if first && symbol.moved_in {
// The check matches it with the Snapshot below if that differs
// from the old lane's last.
symbol.moved_in = false;
self.books_after_moves += 1;
log!("polymarket: {token}: first book after a move, a subscription gap if it differs");
}
let mut book = book;
let book_time = exchange_timestamp;
let mut exchange_timestamp = exchange_timestamp;
@@ -312,17 +443,24 @@ impl Feed {
brought_forward = true;
}
let differs = !first && book != symbol.book;
// A fresh book answering our resync can also be ahead: between our
// unsubscribe and subscribe the server sent us none of its changes.
let gap = differs
&& !brought_forward
// A subscribe hides a few ms of changes (docs/notes.md): a fresh book
// answering our resync can be ahead of our stream, since between our
// unsubscribe and subscribe the server sent none of its changes; and
// so can any book before a change was verified since the opening one.
let opening = !symbol.verified_since_book;
let resync = !brought_forward
&& complete
&& symbol.state == State::Resyncing
&& book_time >= symbol.exchange_timestamp;
let kind = if gap { "resync gap" } else { "loss" };
let gap = differs && (opening || resync);
let kind = match (gap, opening) {
(false, _) => "loss",
(true, true) => "subscription gap (opening)",
(true, false) => "subscription gap (resync)",
};
if differs {
if gap {
self.resync_gaps += 1;
self.subscription_gaps += 1;
} else {
self.losses += 1;
}
@@ -347,7 +485,7 @@ impl Feed {
}
if verified {
symbol.state = State::Live;
symbol.state = State::Verified;
} else if complete {
if symbol.state != State::Unverifiable {
log!(
@@ -355,10 +493,10 @@ impl Feed {
);
}
symbol.state = State::Unverifiable;
} else if matches!(symbol.state, State::AwaitingBook | State::Live) {
// Its last trade price may be stale: a fresh book carries it. Live
// first, since it has a book now, which resync() requires.
symbol.state = State::Live;
} else if matches!(symbol.state, State::AwaitingBook | State::Verified) {
// Its last trade price may be stale: a fresh book carries it.
// Verified first, since it has a book now, which resync() requires.
symbol.state = State::Verified;
symbol.resync(&token, effects);
}
}
@@ -384,7 +522,7 @@ impl Feed {
.take_if(|(_, batch_hash)| *batch_hash != change.hash)
{
if hash(&token, symbol, &symbol.book, &timestamp) == batch_hash {
self.verified_changes += 1;
symbol.verified(&mut self.verified_changes);
} else {
symbol.resync(&token, effects);
}
@@ -416,9 +554,9 @@ impl Feed {
hash: change.hash.clone(),
});
symbol.exchange_timestamp = exchange_timestamp;
if symbol.state == State::Live {
if symbol.state == State::Verified {
if hash(&token, symbol, &symbol.book, &message.timestamp) == change.hash {
self.verified_changes += 1;
symbol.verified(&mut self.verified_changes);
} else {
symbol.open_batch = Some((message.timestamp.clone(), change.hash));
}
@@ -435,8 +573,8 @@ impl Feed {
}
}
/// The book thread: runs until the socket thread is gone, then drops its
/// senders so the writer thread finishes too.
/// A lane's book thread: runs until its connection is gone, then drops its
/// senders so the writer thread finishes once every lane has.
// ponytail: unbounded channels never block the socket read nor drop a frame;
// the ceiling is memory if a consumer stays behind. The minute log's peak of
// frames waiting makes that visible; bounded channels are a constructor change
@@ -444,20 +582,41 @@ impl Feed {
pub fn run(
mut feed: Feed,
frames: Receiver<Frame>,
commands: Receiver<LaneCommand>,
outputs: Sender<Output>,
resyncs: Sender<String>,
connection: Sender<Command>,
reports: Sender<LoadReport>,
) {
let no_commands = never();
let mut orchestrator_gone = false;
loop {
let until_next_minute = (next_minute(Utc::now()) - Utc::now())
.to_std()
.unwrap_or(Duration::ZERO);
let effects = match frames.recv_timeout(until_next_minute) {
let commands = if orchestrator_gone {
&no_commands
} else {
&commands
};
let effects = select! {
recv(frames) -> frame => match frame {
Ok((received_at, text)) => {
feed.peak_waiting = feed.peak_waiting.max(frames.len());
feed.on_frame(received_at, text.as_str())
}
Err(RecvTimeoutError::Timeout) => feed.tick(Utc::now()),
Err(RecvTimeoutError::Disconnected) => return,
Err(_) => return,
},
recv(commands) -> command => match command {
Ok(LaneCommand::Add(market)) => feed.add_market(&market, Utc::now()),
Ok(LaneCommand::MoveIn(market)) => feed.move_in(&market, Utc::now()),
Ok(LaneCommand::Remove(market)) => feed.remove_market(&market),
// It is stopping: the connection's end still decides ours.
Err(_) => {
orchestrator_gone = true;
Vec::new()
}
},
default(until_next_minute) => feed.tick(Utc::now()),
};
for effect in effects {
match effect {
@@ -466,8 +625,15 @@ pub fn run(
return log!("polymarket: the writer thread is gone, stopping");
}
}
// The socket thread may be gone already, while stopping.
Effect::Resync(token) => _ = resyncs.send(token),
// The connection may be gone already, while stopping, and the
// orchestrator too.
Effect::Send(command) => _ = connection.send(command),
Effect::Report(loads) => {
_ = reports.send(LoadReport {
lane: feed.lane,
loads,
})
}
}
}
}
@@ -531,11 +697,19 @@ enum Events {
enum Event {
Book(BookMessage),
PriceChange(PriceChangeMessage),
/// Its wire shape was never observed: only the symbol is read, and it
/// resyncs, since the fresh book carries the new tick size.
/// Only the symbol and its market are read, and it resyncs, since the fresh book carries
/// the new tick size. The server sends each one twice: the second finds
/// the symbol resyncing and asks for nothing (docs/notes.md).
/// `market` was on every one seen; without it, only the load misses it.
TickSizeChange {
market: Option<String>,
asset_id: String,
},
LastTradePrice {
market: Option<String>,
},
/// Not counted in any load: the others come only with
/// `custom_feature_enabled`, which we don't ask for.
#[serde(other)]
Other,
}
@@ -560,6 +734,7 @@ struct LevelMessage {
#[derive(Deserialize)]
struct PriceChangeMessage {
market: String,
timestamp: String,
price_changes: Vec<ChangeMessage>,
}
@@ -600,7 +775,7 @@ fn next_minute(now: DateTime<Utc>) -> DateTime<Utc> {
fn count_resyncs(effects: &[Effect]) -> u64 {
effects
.iter()
.filter(|effect| matches!(effect, Effect::Resync(_)))
.filter(|effect| matches!(effect, Effect::Send(Command::Resync(_))))
.count() as u64
}
@@ -615,6 +790,17 @@ mod tests {
use super::*;
use crate::polymarket::fixture_markets;
/// A lane's feed holding `markets`, as the orchestrator's Adds leave it
/// at the start of the recording.
fn feed_of(markets: &[Market]) -> Feed {
let mut feed = Feed::new(0);
let start = fixture_frames()[0].0;
for market in markets {
feed.add_market(market, start);
}
feed
}
/// The recorded live feed for the fixture's markets (tests/fixtures/README.md).
fn fixture_frames() -> Vec<(DateTime<Utc>, String)> {
#[derive(Deserialize)]
@@ -653,8 +839,8 @@ mod tests {
effects
.iter()
.filter_map(|effect| match effect {
Effect::Resync(token) => Some(token.as_str()),
Effect::Write(_) => None,
Effect::Send(Command::Resync(token)) => Some(token.as_str()),
_ => None,
})
.collect()
}
@@ -685,7 +871,7 @@ mod tests {
/// the recording must replay without a single loss.
#[test]
fn replay_has_no_loss() {
let mut feed = Feed::new(&fixture_markets());
let mut feed = feed_of(&fixture_markets());
replay(&mut feed, &fixture_frames());
assert_eq!(feed.losses, 0);
assert!(
@@ -712,7 +898,7 @@ mod tests {
let markets = fixture_markets();
// The full replay, noting the change frames whose symbols are all
// checked when they come.
let mut control = Feed::new(&markets);
let mut control = feed_of(&markets);
let mut candidates = Vec::new();
let control_effects: Vec<Vec<Effect>> = frames
.iter()
@@ -723,7 +909,7 @@ mod tests {
&& !tokens.is_empty()
&& tokens
.iter()
.all(|token| control.symbols[token].state == State::Live)
.all(|token| control.symbols[token].state == State::Verified)
{
candidates.push(index);
}
@@ -733,7 +919,7 @@ mod tests {
let lost = candidates[0];
let lost_tokens = changed_tokens(&frames[lost].1);
let mut feed = Feed::new(&markets);
let mut feed = feed_of(&markets);
replay(&mut feed, &frames[..lost]);
let (caught, token) = frames
.iter()
@@ -766,7 +952,7 @@ mod tests {
let effects = feed.on_frame(frames[caught].0, &book.to_string());
assert_eq!(snapshot_symbols(&effects), [token.as_str()]);
assert_eq!(feed.losses, losses + 1);
assert_eq!(feed.symbols[&token].state, State::Live);
assert_eq!(feed.symbols[&token].state, State::Verified);
}
/// A feed replayed past a mid-stream book and its symbol's next change, and
@@ -788,7 +974,7 @@ mod tests {
let moved_on = (stale + 1..frames.len())
.find(|&index| changed_tokens(&frames[index].1).contains(&token))
.unwrap();
let mut feed = Feed::new(&fixture_markets());
let mut feed = feed_of(&fixture_markets());
replay(&mut feed, &frames[..=moved_on]);
(feed, token, frames[moved_on].0, frames[stale].1.clone())
}
@@ -839,25 +1025,26 @@ mod tests {
assert!(snapshot_symbols(&effects).is_empty());
assert_eq!(
(feed.books_behind, feed.resync_gaps, feed.losses),
(feed.books_behind, feed.subscription_gaps, feed.losses),
(1, 0, 0)
);
assert_eq!(feed.symbols[&token].book, ours);
assert_eq!(feed.symbols[&token].state, State::Live);
assert_eq!(feed.symbols[&token].state, State::Verified);
}
/// Between our unsubscribe and subscribe the server sends no changes, so
/// the fresh book can hold some we never got: a resync gap, not a loss.
/// the fresh book can hold some we never got: a subscription gap, not a
/// loss.
#[test]
fn a_fresh_book_ahead_of_our_stream_is_a_resync_gap() {
fn a_fresh_book_ahead_of_our_stream_is_a_subscription_gap() {
let frames = fixture_frames();
let now = frames.last().unwrap().0;
let mut feed = Feed::new(&fixture_markets());
let mut feed = feed_of(&fixture_markets());
replay(&mut feed, &frames);
let (token, symbol) = feed
.symbols
.iter_mut()
.find(|(_, symbol)| symbol.state == State::Live)
.find(|(_, symbol)| symbol.state == State::Verified)
.unwrap();
let token = token.clone();
symbol.resync(&token, &mut Vec::new());
@@ -870,9 +1057,9 @@ mod tests {
let effects = feed.on_frame(now, &book);
assert_eq!(snapshot_symbols(&effects), [token.as_str()]);
assert_eq!((feed.resync_gaps, feed.losses), (1, 0));
assert_eq!((feed.subscription_gaps, feed.losses), (1, 0));
assert_eq!(feed.symbols[&token].book, ahead);
assert_eq!(feed.symbols[&token].state, State::Live);
assert_eq!(feed.symbols[&token].state, State::Verified);
}
/// The level test must not mistake our wrong book for a newer one: we
@@ -941,7 +1128,7 @@ mod tests {
for market in &mut markets {
market.min_order_size = "6".to_string();
}
let mut feed = Feed::new(&markets);
let mut feed = feed_of(&markets);
let frames = fixture_frames();
let opening = feed.on_frame(frames[0].0, &frames[0].1);
assert_eq!(snapshot_symbols(&opening).len(), 40);
@@ -963,7 +1150,7 @@ mod tests {
#[test]
fn frames_that_are_not_events_are_skipped() {
let mut feed = Feed::new(&fixture_markets());
let mut feed = feed_of(&fixture_markets());
let at = fixture_frames()[0].0;
for text in [
"PONG",
@@ -976,18 +1163,268 @@ mod tests {
}
}
/// A market's opening books, as a subscribe brings them.
fn opening_books(frames: &[(DateTime<Utc>, String)], market: &Market) -> String {
let opening: Vec<serde_json::Value> = serde_json::from_str(&frames[0].1).unwrap();
let books: Vec<_> = opening
.iter()
.filter(|book| {
market
.tokens
.iter()
.any(|token| book["asset_id"] == **token)
})
.collect();
serde_json::to_string(&books).unwrap()
}
#[test]
fn a_moved_market_logs_its_first_books_once() {
let frames = fixture_frames();
let markets = fixture_markets();
let mut feed = feed_of(&markets[1..]);
replay(&mut feed, &frames[1..100]);
let effects = feed.move_in(&markets[0], frames[100].0);
assert!(matches!(
effects.as_slice(),
[Effect::Send(Command::Subscribe(tokens))] if *tokens == markets[0].tokens
));
let books = opening_books(&frames, &markets[0]);
let effects = feed.on_frame(frames[100].0, &books);
assert_eq!(snapshot_symbols(&effects).len(), 2);
assert_eq!(feed.books_after_moves, 2);
// Later books are the lane's own business.
feed.on_frame(frames[101].0, &books);
assert_eq!(feed.books_after_moves, 2);
assert_eq!((feed.subscription_gaps, feed.losses), (0, 0));
}
#[test]
fn a_market_added_mid_run_is_subscribed_and_its_first_book_written() {
let frames = fixture_frames();
let markets = fixture_markets();
let mut feed = feed_of(&markets[1..]);
replay(&mut feed, &frames[1..100]);
let effects = feed.add_market(&markets[0], frames[100].0);
assert!(matches!(
effects.as_slice(),
[Effect::Send(Command::Subscribe(tokens))] if *tokens == markets[0].tokens
));
let effects = feed.on_frame(frames[100].0, &opening_books(&frames, &markets[0]));
let mut written = snapshot_symbols(&effects);
written.sort();
let mut tokens: Vec<&str> = markets[0].tokens.iter().map(String::as_str).collect();
tokens.sort();
assert_eq!(written, tokens);
assert!(
markets[0]
.tokens
.iter()
.all(|token| feed.symbols[token].state == State::Verified)
);
}
#[test]
fn a_removed_market_is_unsubscribed_and_written_no_more() {
let frames = fixture_frames();
let markets = fixture_markets();
let mut feed = feed_of(&markets);
replay(&mut feed, &frames[..1]);
let removed = &markets[0].tokens;
let effects = feed.remove_market(&markets[0]);
assert!(matches!(
effects.as_slice(),
[Effect::Send(Command::Unsubscribe(tokens))] if tokens == removed
));
// The rest of the recording crosses two minutes: no Delta, Snapshot or
// resync names a removed token.
let effects = replay(&mut feed, &frames[1..]);
let mentions = |effect: &Effect| match effect {
Effect::Write(Output::Snapshot(snapshot)) => removed.contains(&snapshot.symbol),
Effect::Write(Output::Delta(delta)) => removed.contains(&delta.symbol),
Effect::Send(Command::Resync(token)) => removed.contains(token),
Effect::Send(_) => false,
Effect::Report(loads) => loads.contains_key(&markets[0].condition_id),
};
assert!(!effects.iter().any(mentions));
// The minute Snapshots of the 38 others, twice.
assert_eq!(snapshot_symbols(&effects).len(), 38 * 2);
}
/// Seen live: our subscribe hides changes made just before the opening
/// book reaches us, so the first fresh book differs at a level none of our
/// changes touched, before any change was verified.
#[test]
fn a_differing_book_before_any_verified_change_is_a_subscription_gap() {
let frames = fixture_frames();
let mut feed = feed_of(&fixture_markets());
replay(&mut feed, &frames[..1]);
let (token, symbol) = feed
.symbols
.iter()
.find(|(_, symbol)| symbol.state == State::Verified)
.unwrap();
let token = token.clone();
assert!(!symbol.verified_since_book);
let mut exchange = symbol.book.clone();
exchange.set(Side::Buy, 1_000, 123_000_000);
let book = fresh_book(&token, symbol, &exchange);
let effects = feed.on_frame(frames[1].0, &book);
assert_eq!(snapshot_symbols(&effects), [token.as_str()]);
assert_eq!((feed.subscription_gaps, feed.losses), (1, 0));
assert_eq!(feed.symbols[&token].book, exchange);
}
/// Once a change has been verified since the opening book, our book was
/// right then: a fresh book that differs is a loss.
#[test]
fn a_differing_book_after_a_verified_change_is_a_loss() {
let frames = fixture_frames();
let mut feed = feed_of(&fixture_markets());
replay(&mut feed, &frames[..frames.len() / 2]);
let (token, symbol) = feed
.symbols
.iter()
.find(|(_, symbol)| symbol.state == State::Verified && symbol.verified_since_book)
.unwrap();
let token = token.clone();
let mut exchange = symbol.book.clone();
exchange.set(Side::Buy, 1_000, 123_000_000);
let book = fresh_book(&token, symbol, &exchange);
let (gaps, losses) = (feed.subscription_gaps, feed.losses);
let effects = feed.on_frame(frames[frames.len() / 2].0, &book);
assert_eq!(snapshot_symbols(&effects), [token.as_str()]);
assert_eq!((feed.subscription_gaps, feed.losses), (gaps, losses + 1));
assert_eq!(feed.symbols[&token].book, exchange);
}
#[test]
fn each_minute_reports_the_bytes_per_second_of_every_market() {
let minute = "2026-11-10T16:00:00Z".parse::<DateTime<Utc>>().unwrap();
let at = |seconds| minute + TimeDelta::seconds(seconds);
let market = |id: &str| Market {
condition_id: id.to_string(),
tokens: vec![format!("{id}a"), format!("{id}b")],
min_order_size: "5".to_string(),
neg_risk: false,
};
let (busy, quiet, late) = (market("0xbusy"), market("0xquiet"), market("0xlate"));
let mut feed = Feed::new(0);
feed.add_market(&busy, at(0));
// Added halfway through the minute: measured over its 30 s.
feed.add_market(&quiet, at(30));
// Added 15 s before it: kept for the next report.
feed.add_market(&late, at(45));
let change = |id: &str| {
serde_json::json!({"event_type": "price_change", "market": id, "timestamp": "1",
"price_changes": []})
.to_string()
};
let trade = serde_json::json!({"event_type": "last_trade_price", "market": "0xbusy",
"asset_id": "0xbusya", "price": "0.5", "size": "1", "side": "BUY",
"timestamp": "1"})
.to_string();
let tick_size = serde_json::json!({"event_type": "tick_size_change", "market": "0xquiet",
"asset_id": "0xquieta", "old_tick_size": "0.01", "new_tick_size": "0.001",
"timestamp": "1"})
.to_string();
// One frame, two markets: each takes half.
let shared = format!("[{},{}]", change("0xbusy"), change("0xquiet"));
let mut reports = Vec::new();
let mut on_frame = |feed: &mut Feed, seconds, text: &str| {
for effect in feed.on_frame(at(seconds), text) {
if let Effect::Report(loads) = effect {
reports.push(loads);
}
}
};
on_frame(&mut feed, 10, &change("0xbusy"));
on_frame(&mut feed, 20, &trade);
on_frame(&mut feed, 40, &tick_size);
on_frame(&mut feed, 50, &change("0xlate"));
on_frame(&mut feed, 55, &change("0xunknown"));
// The next minute: the first report, then nothing more until the one
// after, which starts again from zero.
on_frame(&mut feed, 60, &shared);
on_frame(&mut feed, 120, &change("0xunknown"));
let bytes = |text: &String| text.len() as u64;
let half = bytes(&shared) / 2;
assert_eq!(
reports,
[
Loads::from([
(
"0xbusy".to_string(),
(bytes(&change("0xbusy")) + bytes(&trade)) / 60
),
("0xquiet".to_string(), bytes(&tick_size) / 30),
]),
Loads::from([
("0xbusy".to_string(), half / 60),
("0xquiet".to_string(), half / 60),
("0xlate".to_string(), bytes(&change("0xlate")) / 75),
]),
]
);
}
#[test]
fn the_recording_reports_every_market_once_measured_long_enough() {
let frames = fixture_frames();
let markets = fixture_markets();
let mut feed = feed_of(&markets);
let reports: Vec<Loads> = replay(&mut feed, &frames)
.into_iter()
.filter_map(|effect| match effect {
Effect::Report(loads) => Some(loads),
_ => None,
})
.collect();
// The recording crosses two minutes, the first too soon after its
// start for any market to be measured.
let start = frames[0].0;
assert!(next_minute(start) - start < MIN_MEASURED);
assert_eq!(reports.len(), 2);
assert!(reports[0].is_empty());
let loads = &reports[1];
assert_eq!(loads.len(), markets.len());
assert!(
markets
.iter()
.all(|market| loads.contains_key(&market.condition_id))
);
assert!(loads.values().sum::<u64>() > 0);
}
#[test]
fn the_output_reconciles() {
// The book and writer threads as main runs them, fed the recording.
let (frames_tx, frames_rx) = crossbeam::channel::unbounded();
let (outputs_tx, outputs_rx) = crossbeam::channel::unbounded();
let (resyncs_tx, _resyncs_rx) = crossbeam::channel::unbounded();
let (connection_tx, _connection_rx) = crossbeam::channel::unbounded();
let (_commands_tx, commands_rx) = crossbeam::channel::unbounded();
let (reports_tx, _reports_rx) = crossbeam::channel::unbounded();
for (received_at, text) in fixture_frames() {
frames_tx.send((received_at, text.into())).unwrap();
}
drop(frames_tx);
let feed = Feed::new(&fixture_markets());
let book = std::thread::spawn(move || run(feed, frames_rx, outputs_tx, resyncs_tx));
let feed = feed_of(&fixture_markets());
let book = std::thread::spawn(move || {
run(
feed,
frames_rx,
commands_rx,
outputs_tx,
connection_tx,
reports_tx,
)
});
let path = std::env::temp_dir().join(format!("reconcile-{}.sbe", std::process::id()));
let written = crate::write(path.to_str().unwrap(), outputs_rx);
book.join().unwrap();
+341
View File
@@ -0,0 +1,341 @@
//! Gamma, Polymarket's market listing: which markets are live. Two fetcher
//! threads page it on the orchestrator's request and stream what they read;
//! they hold no state and no timer (roadmap/03).
use std::thread;
use std::time::Duration;
use chrono::{DateTime, Utc};
use crossbeam::channel::{Receiver, Sender};
use serde::Deserialize;
use super::Market;
const KEYSET_URL: &str = "https://gamma-api.polymarket.com/markets/keyset";
/// Gamma's page size, whatever a larger `limit` asks for.
const PAGE_SIZE: &str = "100";
/// For every request, so SIGINT never waits on Gamma longer than this.
const TIMEOUT: Duration = Duration::from_secs(10);
/// A scan's tries at one page, and the wait between them.
const SCAN_TRIES: u32 = 5;
const SCAN_RETRY_AFTER: Duration = Duration::from_secs(5);
/// What REST's `/book` (and so the hash) says when Gamma's `orderMinSize` is
/// null (checked on 4 such tokens, 2026-11-04).
const MISSING_MIN_ORDER_SIZE: &str = "0";
pub enum Request {
/// Every live market.
Rescan,
/// The live markets created and the markets closed since then.
Recent { since: DateTime<Utc> },
}
#[derive(Debug)]
pub enum Finding {
/// A rescan began: once it is done, a market known now and not seen in it
/// is gone.
ScanStarted,
ScanPage(Vec<Market>),
ScanDone,
ScanFailed,
/// Live markets created since the request's time.
Created(Vec<Market>),
/// The `conditionId`s of markets closed since the request's time.
Closed(Vec<String>),
RecentDone,
RecentFailed,
}
/// A fetcher thread: answers each request until the orchestrator is gone.
pub fn run(requests: Receiver<Request>, findings: Sender<Finding>) {
let agent = ureq::Agent::new_with_config(
ureq::Agent::config_builder()
.timeout_global(Some(TIMEOUT))
.build(),
);
for request in requests {
let sent = match request {
Request::Rescan => rescan(&agent, &findings),
Request::Recent { since } => recent(&agent, since, &findings),
};
if sent.is_err() {
return;
}
}
}
/// `Err` once the orchestrator is gone.
type Sent = Result<(), crossbeam::channel::SendError<Finding>>;
fn rescan(agent: &ureq::Agent, findings: &Sender<Finding>) -> Sent {
findings.send(Finding::ScanStarted)?;
let query = [("active", "true"), ("closed", "false")];
let mut cursor = None;
loop {
let page = match with_retries(|| page(agent, &query, cursor.as_deref())) {
Ok(page) => page,
Err(error) => {
log!("polymarket: rescan failed: {error}");
return findings.send(Finding::ScanFailed);
}
};
let last = page.markets.is_empty();
findings.send(Finding::ScanPage(live(page.markets)))?;
cursor = page.next_cursor;
if last || cursor.is_none() {
return findings.send(Finding::ScanDone);
}
}
}
fn with_retries(mut fetch: impl FnMut() -> anyhow::Result<Page>) -> anyhow::Result<Page> {
let mut tries = 1;
loop {
match fetch() {
Err(_) if tries < SCAN_TRIES => {
tries += 1;
thread::sleep(SCAN_RETRY_AFTER);
}
result => return result,
}
}
}
fn recent(agent: &ureq::Agent, since: DateTime<Utc>, findings: &Sender<Finding>) -> Sent {
let created = [
("active", "true"),
("closed", "false"),
("order", "createdAt"),
("ascending", "false"),
];
let closed = [
("closed", "true"),
("order", "closedTime"),
("ascending", "false"),
];
let result = since_then(agent, &created, since, created_at).and_then(|markets| {
findings.send(Finding::Created(live(markets)))?;
since_then(agent, &closed, since, closed_time)
});
match result {
Ok(markets) => {
let ids = markets.into_iter().map(|market| market.condition_id);
findings.send(Finding::Closed(ids.collect()))?;
findings.send(Finding::RecentDone)
}
Err(error) => {
log!("polymarket: recent check failed: {error}");
findings.send(Finding::RecentFailed)
}
}
}
fn created_at(market: &GammaMarket) -> &Option<String> {
&market.created_at
}
fn closed_time(market: &GammaMarket) -> &Option<String> {
&market.closed_time
}
/// The markets of `query`'s pages, sorted newest first by `time`, down to
/// `since`.
fn since_then(
agent: &ureq::Agent,
query: &[(&str, &str)],
since: DateTime<Utc>,
time: impl Fn(&GammaMarket) -> &Option<String>,
) -> anyhow::Result<Vec<GammaMarket>> {
let mut markets = Vec::new();
let mut cursor = None;
loop {
let page = page(agent, query, cursor.as_deref())?;
let (newer, reached) = newer_than(page.markets, since, &time);
markets.extend(newer);
cursor = page.next_cursor;
if reached || cursor.is_none() {
return Ok(markets);
}
}
}
/// The markets of a newest-first page at or after `since`, and whether the
/// page reached past it. A market with no readable time is kept.
fn newer_than(
markets: Vec<GammaMarket>,
since: DateTime<Utc>,
time: impl Fn(&GammaMarket) -> &Option<String>,
) -> (Vec<GammaMarket>, bool) {
let reached = markets.is_empty()
|| markets
.iter()
.any(|market| parse_time(time(market)).is_some_and(|time| time < since));
let newer = markets
.into_iter()
.filter(|market| parse_time(time(market)).is_none_or(|time| time >= since))
.collect();
(newer, reached)
}
/// `createdAt` is RFC 3339 (`2026-11-04T16:57:29.989119Z`), `closedTime` not
/// (`2026-11-04 16:57:50.384485+00`).
fn parse_time(text: &Option<String>) -> Option<DateTime<Utc>> {
let text = text.as_deref()?;
DateTime::parse_from_rfc3339(text)
.or_else(|_| DateTime::parse_from_str(text, "%Y-%m-%d %H:%M:%S%.f%#z"))
.ok()
.map(|time| time.to_utc())
}
#[derive(Deserialize)]
struct Page {
markets: Vec<GammaMarket>,
next_cursor: Option<String>,
}
/// Every field can be missing or null on some market (2026-11-04: 230 live
/// markets had no `negRisk`, 31 no `orderMinSize`).
#[derive(Deserialize)]
#[serde(rename_all = "camelCase")]
struct GammaMarket {
#[serde(default)]
condition_id: String,
/// A JSON array inside a string.
clob_token_ids: Option<String>,
order_min_size: Option<serde_json::Number>,
neg_risk: Option<bool>,
active: Option<bool>,
closed: Option<bool>,
enable_order_book: Option<bool>,
accepting_orders: Option<bool>,
created_at: Option<String>,
closed_time: Option<String>,
}
fn page(agent: &ureq::Agent, query: &[(&str, &str)], cursor: Option<&str>) -> anyhow::Result<Page> {
let mut request = agent
.get(KEYSET_URL)
.query("limit", PAGE_SIZE)
.query_pairs(query.iter().copied());
if let Some(cursor) = cursor {
request = request.query("after_cursor", cursor);
}
let body = request.call()?.body_mut().read_to_string()?;
Ok(serde_json::from_str(&body)?)
}
/// The live markets among `markets`: active, not closed, with an order book
/// that accepts orders. The end date doesn't count (docs/notes.md).
fn live(markets: Vec<GammaMarket>) -> Vec<Market> {
markets
.into_iter()
.filter(|market| {
market.active == Some(true)
&& market.closed != Some(true)
&& market.enable_order_book == Some(true)
&& market.accepting_orders == Some(true)
&& !market.condition_id.is_empty()
})
.filter_map(|market| {
let tokens: Vec<String> =
serde_json::from_str(market.clob_token_ids.as_deref()?).ok()?;
if tokens.is_empty() {
return None;
}
Some(Market {
condition_id: market.condition_id,
tokens,
min_order_size: market
.order_min_size
.map_or(MISSING_MIN_ORDER_SIZE.to_string(), |size| size.to_string()),
neg_risk: market.neg_risk.unwrap_or(false),
})
})
.collect()
}
/// The live markets of a saved `/markets` response (a plain array).
#[cfg(test)]
pub(crate) fn parse_markets(body: &str) -> Vec<Market> {
live(serde_json::from_str(body).unwrap())
}
#[cfg(test)]
mod tests {
use super::*;
fn fixture(name: &str) -> Page {
let path = format!("{}/tests/fixtures/{name}", env!("CARGO_MANIFEST_DIR"));
serde_json::from_str(&std::fs::read_to_string(path).unwrap()).unwrap()
}
fn time(text: &str) -> DateTime<Utc> {
parse_time(&Some(text.to_string())).unwrap()
}
#[test]
fn parses_gamma_markets() {
let markets = parse_markets(include_str!("../../tests/fixtures/gamma-markets.json"));
assert_eq!(markets.len(), 20);
assert!(markets.iter().all(|market| market.tokens.len() == 2));
assert!(markets[0].condition_id.starts_with("0x"));
assert_eq!(markets[0].min_order_size, "5");
}
#[test]
fn keyset_pages_parse_and_closed_markets_are_not_live() {
let created = fixture("gamma-created.json");
assert_eq!(created.markets.len(), 10);
assert!(created.next_cursor.is_some());
assert_eq!(live(created.markets).len(), 10);
let closed = fixture("gamma-closed.json");
assert_eq!(closed.markets.len(), 10);
assert!(live(closed.markets).is_empty());
}
#[test]
fn both_time_formats_parse() {
assert_eq!(
time("2026-11-04T16:57:29.989119Z"),
time("2026-11-04 16:57:29.989119+00")
);
}
#[test]
fn a_page_is_kept_down_to_since_and_reports_reaching_it() {
// The fixture's createdAt: 16:57:29.989, 16:54:27.166, .084, .011, then
// six from 16:54:26.943 down to 16:54:26.595.
let page = fixture("gamma-created.json");
let (newer, reached) = newer_than(page.markets, time("2026-11-04T16:54:27Z"), created_at);
assert_eq!(newer.len(), 4);
assert!(reached);
let page = fixture("gamma-created.json");
let (newer, reached) = newer_than(page.markets, time("2026-11-04T16:00:00Z"), created_at);
assert_eq!(newer.len(), 10);
assert!(!reached);
let (newer, reached) = newer_than(Vec::new(), time("2026-11-04T16:00:00Z"), created_at);
assert!(newer.is_empty() && reached);
}
#[test]
fn a_live_market_takes_what_the_hash_uses_and_needs_token_ids() {
let markets = live(
serde_json::from_str(
r#"[{"conditionId":"0xa","clobTokenIds":"[\"1\",\"2\"]","negRisk":null,
"active":true,"closed":false,"enableOrderBook":true,"acceptingOrders":true},
{"conditionId":"0xb","clobTokenIds":null,
"active":true,"closed":false,"enableOrderBook":true,"acceptingOrders":true},
{"conditionId":"0xd","clobTokenIds":"[]",
"active":true,"closed":false,"enableOrderBook":true,"acceptingOrders":true},
{"conditionId":"0xc","clobTokenIds":"[\"3\",\"4\"]",
"active":true,"closed":false,"enableOrderBook":true,"acceptingOrders":null}]"#,
)
.unwrap(),
);
assert_eq!(markets.len(), 1);
assert_eq!(markets[0].min_order_size, "0");
assert!(!markets[0].neg_risk);
}
}
+8 -59
View File
@@ -1,17 +1,17 @@
//! Polymarket: which markets to track (Gamma), the market-channel socket, and
//! the feed that turns its frames into books and output.
//! Polymarket: which markets are live (Gamma), the orchestrator that spreads
//! them over lanes, and each lane's connection and feed (roadmap/03).
pub mod feed;
pub mod gamma;
pub mod orchestrator;
pub mod socket;
use serde::Deserialize;
const GAMMA_MARKETS_URL: &str = "https://gamma-api.polymarket.com/markets";
/// A market's symbols and the two inputs of their book hash that the
/// A live market: its symbols and the two inputs of their book hash that the
/// websocket doesn't carry (docs/notes.md).
#[derive(Debug, Clone)]
pub struct Market {
/// Gamma's `conditionId`, the websocket's `market`.
pub condition_id: String,
/// Its token ids, one per outcome: each is a symbol.
pub tokens: Vec<String>,
/// Exactly as the hash expects it, e.g. `"5"`.
@@ -19,58 +19,7 @@ pub struct Market {
pub neg_risk: bool,
}
#[derive(Deserialize)]
#[serde(rename_all = "camelCase")]
struct GammaMarket {
/// A JSON array inside a string.
clob_token_ids: String,
order_min_size: serde_json::Number,
neg_risk: bool,
enable_order_book: bool,
}
/// The `limit` live markets with the most 24 h volume, keeping those with an
/// order book. Fixed for the run: discovery is M3.
pub fn top_markets(limit: u32) -> anyhow::Result<Vec<Market>> {
let body = ureq::get(GAMMA_MARKETS_URL)
.query("limit", limit.to_string())
.query("active", "true")
.query("closed", "false")
.query("order", "volume24hr")
.query("ascending", "false")
.call()?
.body_mut()
.read_to_string()?;
parse_markets(&body)
}
fn parse_markets(body: &str) -> anyhow::Result<Vec<Market>> {
let markets: Vec<GammaMarket> = serde_json::from_str(body)?;
markets
.into_iter()
.filter(|market| market.enable_order_book)
.map(|market| {
Ok(Market {
tokens: serde_json::from_str(&market.clob_token_ids)?,
min_order_size: market.order_min_size.to_string(),
neg_risk: market.neg_risk,
})
})
.collect()
}
#[cfg(test)]
pub(crate) fn fixture_markets() -> Vec<Market> {
parse_markets(include_str!("../../tests/fixtures/gamma-markets.json")).unwrap()
}
#[cfg(test)]
mod tests {
#[test]
fn parses_gamma_markets() {
let markets = super::fixture_markets();
assert_eq!(markets.len(), 20);
assert!(markets.iter().all(|market| market.tokens.len() == 2));
assert_eq!(markets[0].min_order_size, "5");
}
gamma::parse_markets(include_str!("../../tests/fixtures/gamma-markets.json"))
}
+678
View File
@@ -0,0 +1,678 @@
//! The orchestrator: the one owner of what is tracked. It keeps the tracked
//! markets equal to Gamma's live ones, spreads them over lanes and owns every
//! timer; the two fetchers only fetch (roadmap/03, ADR 0002).
use std::cmp::Reverse;
use std::collections::{HashMap, HashSet};
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
use std::thread::{self, JoinHandle};
use std::time::{Duration, Instant};
use chrono::{DateTime, TimeDelta, Utc};
use crossbeam::channel::{Sender, select, unbounded};
use engineering_challenge::book::Output;
use super::feed::{self, Feed, LaneCommand, LoadReport, Loads};
use super::gamma::{self, Finding, Request};
use super::{Market, socket};
/// The most tokens a lane holds: a reconnect replays every one's book at once.
const LANE_TOKENS: usize = 5_000;
/// A lane's byte budget, under the lowest rate at which a connection was seen
/// to get a 1013 close (0.46 MB/s; docs/notes.md). Closes came at up to
/// 5.1 MB/s too: no rate is safe, only less likely to be closed.
const LANE_BYTES_PER_SECOND: u64 = 300_000;
/// Moves are limited so balancing doesn't churn. Rebalancing runs with each
/// minute check, every [`RECENT_EVERY`].
const MOVES_PER_MINUTE: usize = 50;
const MOVED_STAYS: Duration = Duration::from_secs(10 * 60);
const RECENT_EVERY: Duration = Duration::from_secs(60);
const RESCAN_EVERY: Duration = Duration::from_secs(60 * 60);
/// Gamma's indexing delay: each recent check reaches this far before the
/// last successful one. New markets were listed up to 5 minutes after their
/// `createdAt` (docs/notes.md); known markets are skipped, so the margin costs
/// only the pages it reads again.
const RECENT_MARGIN: TimeDelta = TimeDelta::minutes(15);
/// So SIGINT is seen while no timer is due.
const WAKE_EVERY: Duration = Duration::from_secs(1);
/// Runs until `stop` is set, then waits for every lane. Returns how many
/// threads failed: the lane threads that panicked, and one if a fetcher died.
pub fn run(outputs: Sender<Output>, stop: &Arc<AtomicBool>) -> usize {
let (findings_tx, findings) = unbounded();
let (rescan, rescan_requests) = unbounded();
let (recent, recent_requests) = unbounded();
// Two, so a full scan never holds up the minute check. Not joined: they
// hold no output, and each ends at its next send.
let fetchers: Vec<JoinHandle<()>> = [rescan_requests, recent_requests]
.into_iter()
.map(|requests| {
let findings = findings_tx.clone();
thread::spawn(move || gamma::run(requests, findings))
})
.collect();
drop(findings_tx);
let (reports_tx, reports) = unbounded::<LoadReport>();
let mut tracker = Tracker::default();
let mut lanes: Vec<Lane> = Vec::new();
let (mut added, mut removed, mut moved) = (0, 0, 0);
// Startup is the first rescan; the minute checks cover it from its start.
let mut recent_since = Utc::now() - RECENT_MARGIN;
let mut recent_asked_at: Option<DateTime<Utc>> = None;
let mut scanning = false;
let mut scanned = false;
let mut fetcher_died = false;
let mut next_rescan = Instant::now();
let mut next_recent = Instant::now() + RECENT_EVERY;
// Opens the lane on its first command.
let send = |lanes: &mut Vec<Lane>, lane: usize, command: LaneCommand| {
if lane == lanes.len() {
lanes.push(open_lane(lane, &outputs, &reports_tx, stop));
}
// A lane whose book thread died keeps its markets unserved: lane
// health is M5's.
_ = lanes[lane].commands.send(command);
};
while !stop.load(Ordering::Relaxed) {
// Only a panic ends a fetcher while this runs, even one in the middle
// of a request, whose answer would never come: nothing would keep the
// set current.
if fetchers.iter().any(JoinHandle::is_finished) {
log!("polymarket: orchestrator: a fetcher died, stopping");
stop.store(true, Ordering::Relaxed);
fetcher_died = true;
break;
}
let now = Instant::now();
if now >= next_rescan {
if !scanning {
_ = rescan.send(Request::Rescan);
scanning = true;
}
next_rescan = now + RESCAN_EVERY;
}
if now >= next_recent {
if recent_asked_at.is_none() {
_ = recent.send(Request::Recent {
since: recent_since,
});
recent_asked_at = Some(Utc::now());
}
next_recent = now + RECENT_EVERY;
// On the loads the lanes reported at the wall-clock minute.
let moves = tracker.rebalance(now);
moved += moves.len();
for (lane, command) in moves.into_iter().flat_map(Move::commands) {
send(&mut lanes, lane, command);
}
let busiest = tracker.lanes.iter().map(|lane| lane.load).max();
log!(
"polymarket: orchestrator: {} markets in {} lanes; {added} added, {removed} \
removed, {moved} moved this minute; busiest lane at {} bytes/s",
tracker.markets.len(),
lanes.len(),
busiest.unwrap_or(0)
);
(added, removed, moved) = (0, 0, 0);
}
let timeout = next_rescan
.min(next_recent)
.saturating_duration_since(Instant::now())
.min(WAKE_EVERY);
let finding = select! {
recv(findings) -> finding => match finding {
Ok(finding) => finding,
// Both fetchers are gone: the check above stops the run.
Err(_) => continue,
},
recv(reports) -> report => {
if let Ok(report) = report {
tracker.report(report.lane, &report.loads);
}
continue;
},
default(timeout) => continue,
};
let commands = match finding {
Finding::ScanStarted => {
tracker.scan_started();
Vec::new()
}
Finding::ScanPage(markets) | Finding::Created(markets) => markets
.into_iter()
.filter_map(|market| tracker.add(market))
.collect(),
Finding::ScanDone => {
scanning = false;
scanned = true;
tracker.scan_done()
}
Finding::ScanFailed => {
scanning = false;
tracker.scan_failed();
// Nothing can be tracked before a first scan: retry at once.
if !scanned {
next_rescan = Instant::now();
}
Vec::new()
}
Finding::Closed(ids) => ids.iter().filter_map(|id| tracker.remove(id)).collect(),
Finding::RecentDone => {
if let Some(asked_at) = recent_asked_at.take() {
recent_since = asked_at - RECENT_MARGIN;
}
Vec::new()
}
Finding::RecentFailed => {
recent_asked_at = None;
Vec::new()
}
};
for (lane, command) in commands {
match command {
LaneCommand::Add(_) => added += 1,
LaneCommand::Remove(_) => removed += 1,
// Only rebalancing moves, counted there.
LaneCommand::MoveIn(_) => {}
}
send(&mut lanes, lane, command);
}
}
drop(outputs);
let panicked = lanes
.into_iter()
.flat_map(|lane| lane.threads)
.map(JoinHandle::join)
.filter(Result::is_err)
.count();
panicked + usize::from(fetcher_died)
}
struct Lane {
commands: Sender<LaneCommand>,
threads: [JoinHandle<()>; 2],
}
/// Lane `number`'s connection and book thread, joined by [`run`].
fn open_lane(
number: usize,
outputs: &Sender<Output>,
reports: &Sender<LoadReport>,
stop: &Arc<AtomicBool>,
) -> Lane {
let (commands, commands_rx) = unbounded();
let (frames, frames_rx) = unbounded();
let (connection, connection_rx) = unbounded();
let stop = Arc::clone(stop);
let outputs = outputs.clone();
let reports = reports.clone();
let connection_thread = thread::spawn(move || {
socket::run(
socket::MARKET_CHANNEL_URL,
number,
&frames,
&connection_rx,
&stop,
)
});
let book_thread = thread::spawn(move || {
feed::run(
Feed::new(number),
frames_rx,
commands_rx,
outputs,
connection,
reports,
)
});
Lane {
commands,
threads: [connection_thread, book_thread],
}
}
/// A market leaving lane `from` for lane `to`.
#[derive(Debug)]
struct Move {
market: Market,
from: usize,
to: usize,
}
impl Move {
/// A Remove for the old lane, then a MoveIn for the new one, in the order
/// they are sent.
fn commands(self) -> [(usize, LaneCommand); 2] {
[
(self.from, LaneCommand::Remove(self.market.clone())),
(self.to, LaneCommand::MoveIn(self.market)),
]
}
}
/// A tracked market, where it is and what it weighs.
struct Tracked {
market: Market,
lane: usize,
/// Bytes per second over its lane's last report; 0 until one comes.
load: u64,
moved_at: Option<Instant>,
}
/// A lane's tokens and load, against its two limits.
#[derive(Debug, Default, Clone, Copy, PartialEq, Eq)]
struct LaneLoad {
tokens: usize,
/// The sum of its markets' loads, bytes per second.
load: u64,
}
/// Which lane each market is in and what it weighs, and which markets a
/// rescan in progress hasn't seen yet.
#[derive(Default)]
struct Tracker {
/// By `conditionId`.
markets: HashMap<String, Tracked>,
/// By lane number.
lanes: Vec<LaneLoad>,
/// While a rescan runs: the markets known at its start, not seen since.
unseen: Option<HashSet<String>>,
}
impl Tracker {
/// A market it doesn't know goes to the least-loaded lane with token room,
/// a new lane if none has: the lane and its command.
fn add(&mut self, market: Market) -> Option<(usize, LaneCommand)> {
if let Some(unseen) = &mut self.unseen {
unseen.remove(&market.condition_id);
}
if self.markets.contains_key(&market.condition_id) {
return None;
}
let lane = self.lane_for(market.tokens.len(), None, None);
self.lanes[lane].tokens += market.tokens.len();
let tracked = Tracked {
market: market.clone(),
lane,
load: 0,
moved_at: None,
};
self.markets.insert(market.condition_id.clone(), tracked);
Some((lane, LaneCommand::Add(market)))
}
/// The least-loaded lane (then the one with fewest tokens), other than
/// `except`, with room for `tokens` more and, if given, `load` more
/// bytes/s; a new lane if none has.
fn lane_for(&mut self, tokens: usize, load: Option<u64>, except: Option<usize>) -> usize {
self.lanes
.iter()
.enumerate()
.filter(|&(lane, lane_load)| {
Some(lane) != except
&& lane_load.tokens + tokens <= LANE_TOKENS
&& load.is_none_or(|load| lane_load.load + load <= LANE_BYTES_PER_SECOND)
})
.min_by_key(|&(_, lane_load)| (lane_load.load, lane_load.tokens))
.map(|(lane, _)| lane)
.unwrap_or_else(|| {
self.lanes.push(LaneLoad::default());
self.lanes.len() - 1
})
}
fn remove(&mut self, condition_id: &str) -> Option<(usize, LaneCommand)> {
let tracked = self.markets.remove(condition_id)?;
let lane = &mut self.lanes[tracked.lane];
lane.tokens -= tracked.market.tokens.len();
lane.load -= tracked.load;
Some((tracked.lane, LaneCommand::Remove(tracked.market)))
}
/// A lane's loads per market. A market reported by a lane it has since
/// moved from keeps the load it has.
fn report(&mut self, lane: usize, loads: &Loads) {
for (condition_id, &load) in loads {
let Some(tracked) = self.markets.get_mut(condition_id) else {
continue;
};
if tracked.lane != lane {
continue;
}
let lane_load = &mut self.lanes[lane];
lane_load.load = lane_load.load - tracked.load + load;
tracked.load = load;
}
}
/// Moves the busiest markets off every lane over its byte budget, busiest
/// lane first, to the least-loaded lanes with room for them, until it is
/// under: at most [`MOVES_PER_MINUTE`], and none moved in the last
/// [`MOVED_STAYS`].
fn rebalance(&mut self, now: Instant) -> Vec<Move> {
let mut moves = Vec::new();
let mut over: Vec<usize> = (0..self.lanes.len())
.filter(|&lane| self.lanes[lane].load > LANE_BYTES_PER_SECOND)
.collect();
over.sort_by_key(|&lane| Reverse(self.lanes[lane].load));
for from in over {
// ponytail: a scan of every market per lane over budget, fine at a
// few hundred thousand once a minute; per-lane market sets if not.
let mut busiest: Vec<(String, u64)> = self
.markets
.iter()
.filter(|(_, tracked)| {
tracked.lane == from
&& tracked
.moved_at
.is_none_or(|at| now.duration_since(at) >= MOVED_STAYS)
})
.map(|(id, tracked)| (id.clone(), tracked.load))
.collect();
busiest.sort_by_key(|&(_, load)| Reverse(load));
for (id, load) in busiest {
let size = self.markets[&id].market.tokens.len();
// Alone on its lane, a market would only take its load along;
// one with no load would lighten nothing.
if self.lanes[from].load <= LANE_BYTES_PER_SECOND
|| moves.len() == MOVES_PER_MINUTE
|| self.lanes[from].tokens == size
|| load == 0
{
break;
}
let to = self.lane_for(size, Some(load), Some(from));
self.lanes[from].tokens -= size;
self.lanes[from].load -= load;
self.lanes[to].tokens += size;
self.lanes[to].load += load;
let tracked = self.markets.get_mut(&id).expect("listed above");
tracked.lane = to;
tracked.moved_at = Some(now);
moves.push(Move {
market: tracked.market.clone(),
from,
to,
});
}
}
moves
}
fn scan_started(&mut self) {
self.unseen = Some(self.markets.keys().cloned().collect());
}
/// Removes the markets known at the scan's start and not seen in it:
/// those added since were never at risk.
fn scan_done(&mut self) -> Vec<(usize, LaneCommand)> {
let unseen = self.unseen.take().unwrap_or_default();
unseen.iter().filter_map(|id| self.remove(id)).collect()
}
/// Half a listing says nothing of the other half: nothing is removed.
fn scan_failed(&mut self) {
self.unseen = None;
}
}
#[cfg(test)]
mod tests {
use super::*;
fn market(id: usize) -> Market {
Market {
condition_id: format!("0x{id}"),
tokens: vec![format!("{id}a"), format!("{id}b")],
min_order_size: "5".to_string(),
neg_risk: false,
}
}
fn lanes(commands: &[(usize, LaneCommand)]) -> Vec<usize> {
commands.iter().map(|(lane, _)| *lane).collect()
}
fn tokens(tracker: &Tracker) -> Vec<usize> {
tracker.lanes.iter().map(|lane| lane.tokens).collect()
}
/// Reports `bytes_per_second` for each market, as its lane would.
fn report(tracker: &mut Tracker, loads: &[(usize, u64)]) {
for &(id, bytes_per_second) in loads {
let lane = tracker.markets[&format!("0x{id}")].lane;
let loads = Loads::from([(format!("0x{id}"), bytes_per_second)]);
tracker.report(lane, &loads);
}
}
#[test]
fn a_market_joins_the_least_loaded_lane_with_room_and_lanes_open_when_full() {
let mut tracker = Tracker::default();
let per_lane = LANE_TOKENS / 2;
let first: Vec<_> = (0..per_lane + 1)
.filter_map(|id| tracker.add(market(id)))
.collect();
// Nothing measured yet: the first lane fills, then a second opens.
assert!(lanes(&first[..per_lane]).iter().all(|&lane| lane == 0));
assert_eq!(first[per_lane].0, 1);
// A hole in lane 0 draws the next market once lane 1 weighs more.
tracker.remove("0x0").unwrap();
report(&mut tracker, &[(per_lane, 1_000)]);
assert_eq!(tracker.add(market(per_lane + 1)).unwrap().0, 0);
// Both tokens of a market share its lane.
assert_eq!(tracker.markets["0x1"].lane, 0);
assert_eq!(tokens(&tracker), [LANE_TOKENS, 2]);
}
#[test]
fn between_lanes_of_equal_load_a_market_joins_the_one_with_fewest_tokens() {
let mut tracker = Tracker {
lanes: vec![
LaneLoad { tokens: 4, load: 0 },
LaneLoad { tokens: 2, load: 0 },
],
..Tracker::default()
};
assert_eq!(lanes(&[tracker.add(market(0)).unwrap()]), [1]);
assert_eq!(tokens(&tracker), [4, 4]);
}
#[test]
fn a_known_market_is_skipped() {
let mut tracker = Tracker::default();
assert!(tracker.add(market(1)).is_some());
assert!(tracker.add(market(1)).is_none());
assert_eq!(tokens(&tracker), [2]);
}
#[test]
fn a_report_sets_its_lanes_markets_and_no_others() {
let mut tracker = Tracker::default();
tracker.add(market(1));
tracker.add(market(2));
report(&mut tracker, &[(1, 300), (2, 100)]);
assert_eq!(tracker.lanes[0].load, 400);
// A lane the market has left reports it late: ignored.
tracker.lanes.push(LaneLoad::default());
let stale = Loads::from([("0x1".to_string(), 900)]);
tracker.report(1, &stale);
assert_eq!(tracker.markets["0x1"].load, 300);
assert_eq!(tracker.lanes[1].load, 0);
// A market gone takes its load along.
tracker.remove("0x1");
assert_eq!(tracker.lanes[0].load, 100);
}
#[test]
fn a_lane_over_budget_moves_its_busiest_markets_until_under() {
let mut tracker = Tracker::default();
for id in 0..10 {
tracker.add(market(id));
}
tracker.lanes.push(LaneLoad::default());
let budget = LANE_BYTES_PER_SECOND;
// Lane 0 at 1.5 budgets: its busiest market (half a budget) moving
// is enough.
let mut loads = vec![(0, budget / 2)];
loads.extend((1..10).map(|id| (id, budget / 9)));
report(&mut tracker, &loads);
let now = Instant::now();
let moves = tracker.rebalance(now);
assert!(matches!(
moves.as_slice(),
[Move { market, from: 0, to: 1 }] if market.condition_id == "0x0"
));
assert_eq!(tracker.markets["0x0"].lane, 1);
let [
(0, LaneCommand::Remove(removed)),
(1, LaneCommand::MoveIn(added)),
] = moves.into_iter().next().unwrap().commands()
else {
panic!("a move is a Remove for the old lane, then a MoveIn for the new");
};
assert_eq!(
(removed.condition_id, added.condition_id),
("0x0".into(), "0x0".into())
);
assert!(tracker.lanes[0].load <= budget);
assert_eq!(tokens(&tracker), [18, 2]);
assert_eq!(tracker.lanes[1].load, budget / 2);
// Under budget now: nothing more moves.
assert!(tracker.rebalance(now).is_empty());
}
#[test]
fn the_busiest_lane_over_budget_is_rebalanced_first() {
let mut tracker = Tracker::default();
tracker.add(market(0));
tracker.add(market(1));
tracker.lanes.push(LaneLoad::default());
// The empty lane is the least loaded: both join it.
tracker.add(market(2));
tracker.add(market(3));
assert_eq!(tokens(&tracker), [4, 4]);
let budget = LANE_BYTES_PER_SECOND;
report(&mut tracker, &[(0, budget), (1, budget / 10)]);
report(&mut tracker, &[(2, budget / 2), (3, budget)]);
let moves = tracker.rebalance(Instant::now());
assert!(matches!(
moves.first(),
Some(Move { market, from: 1, .. }) if market.condition_id == "0x3"
));
}
/// The markets that may move have no load: moving them would empty the
/// lane and lighten nothing.
#[test]
fn markets_with_no_load_are_not_moved() {
let mut tracker = Tracker::default();
for id in 0..4 {
tracker.add(market(id));
}
report(&mut tracker, &[(0, 2 * LANE_BYTES_PER_SECOND)]);
let now = Instant::now();
tracker.markets.get_mut("0x0").unwrap().moved_at = Some(now);
assert!(tracker.rebalance(now).is_empty());
}
#[test]
fn moves_are_limited_and_a_moved_market_stays() {
let mut tracker = Tracker::default();
let count = 2 * MOVES_PER_MINUTE;
for id in 0..count {
tracker.add(market(id));
}
// Every market alone near a budget: each needs a lane of its own.
let loads: Vec<_> = (0..count)
.map(|id| (id, LANE_BYTES_PER_SECOND * 9 / 10))
.collect();
report(&mut tracker, &loads);
let now = Instant::now();
let first = tracker.rebalance(now);
assert_eq!(first.len(), MOVES_PER_MINUTE);
// The rest move a minute later, but the last, alone and under budget.
let second = tracker.rebalance(now + Duration::from_secs(60));
assert_eq!(second.len(), MOVES_PER_MINUTE - 1);
assert!(second.iter().all(|moved| moved.from == 0));
// Ten minutes on, lane 0 holds one market: it is left alone.
assert!(tracker.rebalance(now + MOVED_STAYS).is_empty());
assert_eq!(tracker.lanes[0].tokens, 2);
}
/// Over budget on its own, a market would only carry its load from lane
/// to lane every ten minutes.
#[test]
fn a_market_alone_on_its_lane_is_not_moved() {
let mut tracker = Tracker::default();
tracker.add(market(0));
report(&mut tracker, &[(0, 2 * LANE_BYTES_PER_SECOND)]);
assert!(tracker.rebalance(Instant::now()).is_empty());
assert_eq!(tracker.lanes.len(), 1);
}
#[test]
fn a_market_moved_less_than_ten_minutes_ago_stays() {
let mut tracker = Tracker::default();
let budget = LANE_BYTES_PER_SECOND;
tracker.add(market(0));
tracker.add(market(1));
report(&mut tracker, &[(0, budget * 7 / 10), (1, budget / 2)]);
tracker.lanes.push(LaneLoad::default());
let now = Instant::now();
tracker.add(market(2));
assert_eq!(tracker.markets["0x2"].lane, 1);
report(&mut tracker, &[(2, budget / 10)]);
// Market 0 moves to lane 1, which then turns busy.
assert!(matches!(
tracker.rebalance(now).as_slice(),
[Move { from: 0, to: 1, .. }]
));
report(&mut tracker, &[(0, budget * 7 / 10), (2, budget * 6 / 10)]);
let moves = tracker.rebalance(now + Duration::from_secs(60));
assert!(matches!(
moves.as_slice(),
[Move { market, from: 1, to: 2 }] if market.condition_id == "0x2"
));
}
#[test]
fn a_completed_rescan_removes_only_the_known_markets_it_did_not_see() {
let mut tracker = Tracker::default();
for id in 0..3 {
tracker.add(market(id));
}
tracker.scan_started();
tracker.add(market(0));
// Added by a minute check during the scan, on a page already passed.
tracker.add(market(9));
// Closed during the scan.
tracker.remove("0x1");
let removed = tracker.scan_done();
assert!(matches!(
removed.as_slice(),
[(0, LaneCommand::Remove(market))] if market.condition_id == "0x2"
));
let mut tracked: Vec<_> = tracker.markets.keys().cloned().collect();
tracked.sort();
assert_eq!(tracked, ["0x0", "0x9"]);
assert_eq!(tokens(&tracker), [4]);
}
#[test]
fn a_failed_rescan_removes_nothing() {
let mut tracker = Tracker::default();
tracker.add(market(0));
tracker.scan_started();
tracker.scan_failed();
assert!(tracker.scan_done().is_empty());
assert_eq!(tracker.markets.len(), 1);
}
}
+203 -35
View File
@@ -1,6 +1,8 @@
//! The socket thread: reads the market channel and stamps each frame's receive
//! time, and nothing else. Parsing is the book thread's.
//! A lane's connection: reads the market channel and stamps each frame's
//! receive time, and sends what its book thread asks for (subscriptions,
//! resyncs). Parsing is the book thread's.
use std::collections::BTreeSet;
use std::io::ErrorKind;
use std::net::{TcpStream, ToSocketAddrs};
use std::sync::atomic::{AtomicBool, Ordering};
@@ -10,42 +12,66 @@ use std::time::{Duration, Instant};
use chrono::{DateTime, Utc};
use crossbeam::channel::{Receiver, Sender};
use serde_json::json;
use tungstenite::http::Uri;
use tungstenite::stream::MaybeTlsStream;
use tungstenite::{Message, Utf8Bytes, WebSocket};
const MARKET_CHANNEL_HOST: &str = "ws-subscriptions-clob.polymarket.com";
const MARKET_CHANNEL_URL: &str = "wss://ws-subscriptions-clob.polymarket.com/ws/market";
pub const MARKET_CHANNEL_URL: &str = "wss://ws-subscriptions-clob.polymarket.com/ws/market";
/// For the TCP connect and the TLS + websocket handshake, and for every send.
const CONNECT_TIMEOUT: Duration = Duration::from_secs(10);
/// How often the loop wakes when no frame comes: tungstenite has no `split`,
/// so the reads and the writes below (PING, resyncs, close) share this thread.
/// so the reads and the writes below (PING, commands, close) share this thread.
const READ_TIMEOUT: Duration = Duration::from_millis(200);
const PING_EVERY: Duration = Duration::from_secs(10);
const RECONNECT_AFTER: Duration = Duration::from_secs(1);
const CLOSE_TIMEOUT: Duration = Duration::from_secs(2);
/// Tokens per subscribe or unsubscribe message: a first message of more than
/// 500 gets no opening books (docs/notes.md).
const CHUNK: usize = 500;
/// A frame's text with its receive time.
pub type Frame = (DateTime<Utc>, Utf8Bytes);
/// What the book thread asks its connection for.
#[derive(Debug, PartialEq, Eq)]
pub enum Command {
Subscribe(Vec<String>),
Unsubscribe(Vec<String>),
/// Unsubscribe and subscribe again: the server then sends a fresh book.
Resync(String),
}
type Socket = WebSocket<MaybeTlsStream<TcpStream>>;
/// Runs until `stop` is set or the book thread is gone, reconnecting (and
/// subscribing every token again) whenever the connection drops. `resyncs`
/// carries the tokens to unsubscribe and subscribe again.
/// subscribing its tokens again) whenever the connection drops.
pub fn run(
tokens: &[String],
url: &str,
lane: usize,
frames: &Sender<Frame>,
resyncs: &Receiver<String>,
commands: &Receiver<Command>,
stop: &AtomicBool,
) {
let mut tokens = BTreeSet::new();
while !stop.load(Ordering::Relaxed) {
match connect(tokens) {
Ok(mut socket) => match read(&mut socket, frames, resyncs, stop) {
Ok(Ended::Stopped) => return close(&mut socket, frames),
// Asked for while disconnected: the subscribe on connect covers them,
// and its books are fresh, so resyncs have nothing left to do.
for command in commands.try_iter() {
_ = apply(&mut tokens, &command);
}
match connect(url, &tokens) {
Ok(mut socket) => {
log!(
"polymarket: lane {lane}: connected, {} symbols",
tokens.len()
);
match read(&mut socket, &mut tokens, frames, commands, stop) {
Ok(Ended::Stopped) => return close(&mut socket, lane, frames),
Ok(Ended::BookThreadGone) => return,
Err(error) => log!("polymarket: connection lost: {error}"),
},
Err(error) => log!("polymarket: connect failed: {error}"),
Err(error) => log!("polymarket: lane {lane}: connection lost: {error}"),
}
}
Err(error) => log!("polymarket: lane {lane}: connect failed: {error}"),
}
thread::sleep(RECONNECT_AFTER);
}
@@ -56,17 +82,57 @@ enum Ended {
BookThreadGone,
}
fn connect(tokens: &[String]) -> anyhow::Result<Socket> {
/// Updates the token set, and returns the messages that carry `command` out
/// on an open connection: none for a resync of a token not in the set.
#[must_use]
fn apply(tokens: &mut BTreeSet<String>, command: &Command) -> Vec<String> {
match command {
Command::Subscribe(added) => {
tokens.extend(added.iter().cloned());
operations("subscribe", added)
}
Command::Unsubscribe(removed) => {
removed.iter().for_each(|token| _ = tokens.remove(token));
operations("unsubscribe", removed)
}
// Subscribing a token already subscribed sends nothing back but
// `NO NEW ASSETS`: unsubscribing first makes the server send a fresh
// book.
Command::Resync(token) if tokens.contains(token) => [
operations("unsubscribe", [token]),
operations("subscribe", [token]),
]
.concat(),
Command::Resync(_) => Vec::new(),
}
}
/// The messages that subscribe or unsubscribe `tokens` on an open connection.
fn operations<'a>(operation: &str, tokens: impl IntoIterator<Item = &'a String>) -> Vec<String> {
let tokens: Vec<_> = tokens.into_iter().collect();
tokens
.chunks(CHUNK)
.map(|chunk| json!({"assets_ids": chunk, "operation": operation}).to_string())
.collect()
}
fn connect(url: &str, tokens: &BTreeSet<String>) -> anyhow::Result<Socket> {
let uri: Uri = url.parse()?;
let host = uri
.host()
.ok_or_else(|| anyhow::anyhow!("{url} has no host"))?;
// The market channel is `wss://` on TLS's port; the tests name theirs.
let port = uri.port_u16().unwrap_or(443);
// Timed, unlike `tungstenite::connect`, so a stalled connect or handshake
// can't keep SIGINT waiting. The name lookup itself has no timeout.
let address = (MARKET_CHANNEL_HOST, 443)
let address = (host, port)
.to_socket_addrs()?
.next()
.ok_or_else(|| anyhow::anyhow!("{MARKET_CHANNEL_HOST} has no address"))?;
.ok_or_else(|| anyhow::anyhow!("{host} has no address"))?;
let stream = TcpStream::connect_timeout(&address, CONNECT_TIMEOUT)?;
stream.set_read_timeout(Some(CONNECT_TIMEOUT))?;
stream.set_write_timeout(Some(CONNECT_TIMEOUT))?;
let (mut socket, _response) = tungstenite::client_tls(MARKET_CHANNEL_URL, stream)
let (mut socket, _response) = tungstenite::client_tls(url, stream)
.map_err(|error| anyhow::anyhow!("handshake failed: {error}"))?;
let stream = match socket.get_ref() {
MaybeTlsStream::Plain(stream) => stream,
@@ -74,17 +140,21 @@ fn connect(tokens: &[String]) -> anyhow::Result<Socket> {
_ => anyhow::bail!("unexpected TLS stream"),
};
stream.set_read_timeout(Some(READ_TIMEOUT))?;
// Opened empty, then subscribed in chunks: one rule for every token.
socket.send(Message::text(
json!({"type": "market", "assets_ids": tokens}).to_string(),
json!({"type": "market", "assets_ids": []}).to_string(),
))?;
log!("polymarket: connected, subscribed {} symbols", tokens.len());
for message in operations("subscribe", tokens) {
socket.send(Message::text(message))?;
}
Ok(socket)
}
fn read(
socket: &mut Socket,
tokens: &mut BTreeSet<String>,
frames: &Sender<Frame>,
resyncs: &Receiver<String>,
commands: &Receiver<Command>,
stop: &AtomicBool,
) -> anyhow::Result<Ended> {
let mut last_ping = Instant::now();
@@ -111,25 +181,19 @@ fn read(
socket.send(Message::text("PING"))?;
last_ping = Instant::now();
}
for token in resyncs.try_iter() {
// Subscribing a token already subscribed sends nothing back but
// `NO NEW ASSETS`: unsubscribing first makes the server send a
// fresh book.
socket.send(Message::text(
json!({"assets_ids": [&token], "operation": "unsubscribe"}).to_string(),
))?;
socket.send(Message::text(
json!({"assets_ids": [&token], "operation": "subscribe"}).to_string(),
))?;
for command in commands.try_iter() {
for message in apply(tokens, &command) {
socket.send(Message::text(message))?;
}
}
}
}
/// Closes the connection, forwarding what still arrives until the server
/// confirms or [`CLOSE_TIMEOUT`] passes.
fn close(socket: &mut Socket, frames: &Sender<Frame>) {
fn close(socket: &mut Socket, lane: usize, frames: &Sender<Frame>) {
if let Err(error) = socket.close(None) {
return log!("polymarket: close failed: {error}");
return log!("polymarket: lane {lane}: close failed: {error}");
}
let deadline = Instant::now() + CLOSE_TIMEOUT;
while Instant::now() < deadline {
@@ -143,7 +207,6 @@ fn close(socket: &mut Socket, frames: &Sender<Frame>) {
Err(_) => break,
}
}
log!("polymarket: socket closed");
}
/// A read timeout shows up as `WouldBlock` on macOS and `TimedOut` elsewhere;
@@ -151,3 +214,108 @@ fn close(socket: &mut Socket, frames: &Sender<Frame>) {
fn timed_out(error: &std::io::Error) -> bool {
matches!(error.kind(), ErrorKind::WouldBlock | ErrorKind::TimedOut)
}
#[cfg(test)]
mod tests {
use std::net::TcpListener;
use std::sync::Arc;
use crossbeam::channel::unbounded;
use super::*;
/// The tokens each message of one connection subscribed, after the first.
fn subscribed(socket: &mut WebSocket<TcpStream>, tokens: usize) -> Vec<Vec<String>> {
let first = socket.read().unwrap().into_text().unwrap();
let first: serde_json::Value = serde_json::from_str(&first).unwrap();
assert_eq!(first, json!({"type": "market", "assets_ids": []}));
let mut chunks = Vec::new();
while chunks.iter().map(Vec::len).sum::<usize>() < tokens {
let text = socket.read().unwrap().into_text().unwrap();
let message: serde_json::Value = serde_json::from_str(&text).unwrap();
assert_eq!(message["operation"], "subscribe");
chunks.push(serde_json::from_value(message["assets_ids"].clone()).unwrap());
}
chunks
}
#[test]
fn a_reconnect_subscribes_every_token_again_in_chunks() {
let listener = TcpListener::bind("127.0.0.1:0").unwrap();
let url = format!("ws://{}/ws/market", listener.local_addr().unwrap());
let tokens: Vec<String> = (0..1_200).map(|token| token.to_string()).collect();
let total = tokens.len();
let server = thread::spawn(move || {
let mut connections = Vec::new();
for _ in 0..2 {
let (stream, _) = listener.accept().unwrap();
// A client that never sends fails the test rather than hangs it.
stream
.set_read_timeout(Some(Duration::from_secs(5)))
.unwrap();
let mut socket = tungstenite::accept(stream).unwrap();
connections.push(subscribed(&mut socket, total));
// The client reconnects after this close; after the second,
// it finds no server and stops.
socket.close(None).unwrap();
while socket.read().is_ok() {}
}
connections
});
let (frames_tx, _frames_rx) = unbounded();
let (commands_tx, commands_rx) = unbounded();
commands_tx
.send(Command::Subscribe(tokens.clone()))
.unwrap();
let stop = Arc::new(AtomicBool::new(false));
let client_stop = Arc::clone(&stop);
let client = thread::spawn(move || run(&url, 0, &frames_tx, &commands_rx, &client_stop));
let connections = server.join().unwrap();
stop.store(true, Ordering::Relaxed);
client.join().unwrap();
let mut tokens = tokens;
tokens.sort();
for chunks in connections {
assert!(chunks.iter().all(|chunk| chunk.len() <= CHUNK));
let mut subscribed = chunks.concat();
subscribed.sort();
assert_eq!(subscribed, tokens);
}
}
#[test]
fn commands_update_the_token_set_and_say_what_to_send() {
let mut tokens = BTreeSet::new();
let operation = |message: &String| -> (String, Vec<String>) {
let message: serde_json::Value = serde_json::from_str(message).unwrap();
let tokens = serde_json::from_value(message["assets_ids"].clone()).unwrap();
(message["operation"].as_str().unwrap().to_string(), tokens)
};
let sent = |messages: Vec<String>| messages.iter().map(operation).collect::<Vec<_>>();
let op = |name: &str, tokens: &[&str]| {
(
name.to_string(),
tokens.iter().map(|token| token.to_string()).collect(),
)
};
let subscribe = Command::Subscribe(vec!["a".into(), "b".into()]);
assert_eq!(
sent(apply(&mut tokens, &subscribe)),
[op("subscribe", &["a", "b"])]
);
assert_eq!(
sent(apply(&mut tokens, &Command::Resync("a".into()))),
[op("unsubscribe", &["a"]), op("subscribe", &["a"])]
);
let unsubscribe = Command::Unsubscribe(vec!["a".into()]);
assert_eq!(
sent(apply(&mut tokens, &unsubscribe)),
[op("unsubscribe", &["a"])]
);
assert!(apply(&mut tokens, &Command::Resync("a".into())).is_empty());
assert_eq!(tokens.into_iter().collect::<Vec<_>>(), ["b"]);
}
}
+1 -1
View File
@@ -1,6 +1,6 @@
//! Checks an output file against itself: per symbol, the Deltas written since
//! a Snapshot, applied to it, must give the next Snapshot. A mismatch is a loss
//! or a resync gap (the fresh book differs), each logged by the collector.
//! or a subscription gap (the fresh book differs), each logged by the collector.
use std::collections::HashMap;
use std::path::Path;
+8 -2
View File
@@ -1,10 +1,16 @@
# Test fixtures
A recording of Polymarket's market channel, replayed by the tests in
`src/polymarket/feed.rs` (no network in tests).
`src/polymarket/feed.rs`, and Gamma pages read by `src/polymarket/gamma.rs`
(no network in tests).
- `gamma-markets.json`: Gamma's top 20 markets by 24 h volume on 2026-11-04,
trimmed to the fields the collector reads.
trimmed to the fields the collector reads. `conditionId`, `active`, `closed`
and `acceptingOrders` were added in sprint 03 (the ids from the recorded
books' `market`).
- `gamma-created.json`, `gamma-closed.json`: one `/markets/keyset` page each
(`limit=10`) of the newest live markets by `createdAt` and the latest closed
by `closedTime`, 2026-11-04 ~17:00 UTC, trimmed the same way.
- `polymarket-frames.jsonl`: the first 90 s (14:24:44 to 14:26:14 UTC, so two
minute boundaries) of the market channel for those markets' 40 tokens, one
line per frame: `{"received_at_us": <receive time, µs>, "text": <frame>}`.
+135
View File
@@ -0,0 +1,135 @@
{
"markets": [
{
"question": "Populovskyi Eduard vs. Kovalenko Anatolii",
"conditionId": "0xefab23442b429fa445b8e52394c86caf3f092bf3f71300829b1c1eb0ce7f0b50",
"clobTokenIds": "[\"44826501889331181717150371523936754626679193531166396487334749539805478460209\", \"36302256322938166611393018282312068242391943224849867955841298097733298988891\"]",
"orderMinSize": 5,
"negRisk": false,
"active": true,
"closed": true,
"enableOrderBook": true,
"acceptingOrders": false,
"createdAt": "2026-11-03T19:53:35.292723Z",
"closedTime": "2026-11-04 16:57:50.384485+00"
},
{
"question": "Set Handicap: Niels Peter Van Noord (-1.5) vs Keshav Chopra (+1.5)",
"conditionId": "0x2e131dc2a29bf2d276d34d7e3c0cdfca4e3b4cd3512f4b04a1a006e626f4ae61",
"clobTokenIds": "[\"39895080082116076744071586278013201425211458703422956002772510408434190677884\", \"19051851145542724727656665058814185905321034954829010093680773644546996513535\"]",
"orderMinSize": 5,
"negRisk": false,
"active": true,
"closed": true,
"enableOrderBook": true,
"acceptingOrders": false,
"createdAt": "2026-11-04T03:44:14.574104Z",
"closedTime": "2026-11-04 16:57:50.358086+00"
},
{
"question": "Set Handicap: Kayden Colombo (-1.5) vs Johan Alexander Rodriguez (+1.5)",
"conditionId": "0x5353ac2ea7d3f4e21acb4d0861cb5b44ac779a096de9f0ffc546ff8140feb5fa",
"clobTokenIds": "[\"94958295102858405885818380458434883996828517060091040376117137374766737083914\", \"7671610100222080411744632394529476881255214291291880601575663863683487114654\"]",
"orderMinSize": 5,
"negRisk": false,
"active": true,
"closed": true,
"enableOrderBook": true,
"acceptingOrders": false,
"createdAt": "2026-11-04T03:44:13.474846Z",
"closedTime": "2026-11-04 16:57:22.344836+00"
},
{
"question": "Set Handicap: Maria Jose Sanchez Uribe (-1.5) vs Luciana Agudelo (+1.5)",
"conditionId": "0xe9a4d2cc5013a9ef930769a06067bf1793b44e277262cedec9382605d1f4d789",
"clobTokenIds": "[\"21745108597700168910860076616721481120538218170946858631858304920364048593626\", \"88297511931567368303913767416363723944767307572967321208894173417589979621188\"]",
"orderMinSize": 5,
"negRisk": false,
"active": true,
"closed": true,
"enableOrderBook": true,
"acceptingOrders": false,
"createdAt": "2026-11-03T21:44:03.719514Z",
"closedTime": "2026-11-04 16:57:22.316215+00"
},
{
"question": "Kuzyshyn Oleh vs. Blashkun Oleksandr",
"conditionId": "0xf9a84b5f47465e3256630198c3e593819ff00978d660842dc4f259f23d45b852",
"clobTokenIds": "[\"97554300278659417619290655189631481316508823986205552306096670194286646500988\", \"20252670317414328146520571690921965527846507756708657221097476588874672725450\"]",
"orderMinSize": 5,
"negRisk": false,
"active": true,
"closed": true,
"enableOrderBook": true,
"acceptingOrders": false,
"createdAt": "2026-11-03T19:52:18.066777Z",
"closedTime": "2026-11-04 16:57:22.288775+00"
},
{
"question": "Set Handicap: Weronika Ewald (-1.5) vs Ida Wobker (+1.5)",
"conditionId": "0x4367cfbecaee2995e81fb1514ccf3ab26502f41cd9b8d588136fdecc3160bdb7",
"clobTokenIds": "[\"13297249568103066689520348076229200996848266474396001529378944683098932755914\", \"89894550493893133858598989669017150578230792898149665317182501688736361564700\"]",
"orderMinSize": 5,
"negRisk": false,
"active": true,
"closed": true,
"enableOrderBook": true,
"acceptingOrders": false,
"createdAt": "2026-11-03T21:44:04.165837Z",
"closedTime": "2026-11-04 16:57:22.261927+00"
},
{
"question": "Will Tesla, Inc. (TSLA) hit (LOW) $390 Week of November 2 2026?",
"conditionId": "0xadd174f7eeb05feba859b8d4a4990b6fdd62de6382d6b0b34a5278aa0cec9e93",
"clobTokenIds": "[\"53111618025983699124066570152720566574297036520169668823377484893951216981918\", \"1738729912475637393699992934041272180198757392022311537253739693114313017539\"]",
"orderMinSize": 5,
"negRisk": false,
"active": true,
"closed": true,
"enableOrderBook": true,
"acceptingOrders": false,
"createdAt": "2026-10-30T22:00:27.078976Z",
"closedTime": "2026-11-04 16:57:22.234043+00"
},
{
"question": "W35 Liberec: Ida Wobker vs Weronika Ewald",
"conditionId": "0xd45f79466a8883b774390fc1abb30f63a791ba3e9a85b7d6d92c9015acb115c0",
"clobTokenIds": "[\"37553866556598063681604558627906332172694310397407486121862876197091898752599\", \"77201422525387544766857381326308690506358974601454997287230251810366877318676\"]",
"orderMinSize": 5,
"negRisk": false,
"active": true,
"closed": true,
"enableOrderBook": true,
"acceptingOrders": false,
"createdAt": "2026-11-03T21:14:12.671977Z",
"closedTime": "2026-11-04 16:57:22.204847+00"
},
{
"question": "W50 Heraklion: Carlota Martinez Cirez vs Daria Yesypchuk",
"conditionId": "0x76bce54b6da6ec1663e6c7a9fe3eb3e4ee57ccb6e4bd64aec2cb0c925ff5708d",
"clobTokenIds": "[\"67085659236189310000851293394749121730386584794981034590417392334188473111344\", \"64682819352472807351567187431129959173116373378073567759284036383866483667934\"]",
"orderMinSize": 5,
"negRisk": false,
"active": true,
"closed": true,
"enableOrderBook": true,
"acceptingOrders": false,
"createdAt": "2026-11-03T21:14:11.443782Z",
"closedTime": "2026-11-04 16:57:06.115352+00"
},
{
"question": "Djokovic vs. Hijikata: Match O/U 22.5",
"conditionId": "0x8c8b46f1b0bfaa40e45de277b1aa0cae3f50e5752dd11492b68a6bcd66af7d79",
"clobTokenIds": "[\"69853942453264725463023948212497122784872761667832747694549450815793109801231\", \"99642552909265075117113081841538849880069795592738032133677294961975308569760\"]",
"orderMinSize": 5,
"negRisk": false,
"active": true,
"closed": true,
"enableOrderBook": true,
"acceptingOrders": false,
"createdAt": "2026-11-03T18:44:14.155967Z",
"closedTime": "2026-11-04 16:57:06.086161+00"
}
],
"next_cursor": "YM0JZysiu5w7xwmDjxuLOiH9N_GyhP7sKFS4xl4RPWB7InYiOjEsImsiOiJtYXJrZXRzIiwib2giOiJjNmI2MzAwYWRmM2Q3NWFmYTZhYzAzNDUwNTM3MzZhMDk2ODc3Mzc2NDM5OTQ0MjIzYjA0MDRiMjU0MTVkMDVkIiwia2V5cyI6W3sidCI6InN0cmluZyIsInYiOiIyMDI2LTExLTA0IDE2OjU3OjA2LjA4NjE2MSswMCJ9LHsidCI6InN0cmluZyIsInYiOiI2NjQ3OTE4In1dfQ"
}
+125
View File
@@ -0,0 +1,125 @@
{
"markets": [
{
"question": "XRP Up or Down - November 5, 11:50AM-11:55AM ET",
"conditionId": "0x7af9827676c3790cce6662e5a335d6f017ae99ecd9bd22b1d6f02bd4cf686f9b",
"clobTokenIds": "[\"115574995017995341868238121220540154732508720721227808951068953932623414227263\", \"42196855944990128389362302048239777171495224595711316446705919987418523770734\"]",
"orderMinSize": 5,
"negRisk": false,
"active": true,
"closed": false,
"enableOrderBook": true,
"acceptingOrders": true,
"createdAt": "2026-11-04T16:57:29.989119Z"
},
{
"question": "Exact Score: Any Other Score?",
"conditionId": "0xa19ecbe2c5ee9809b9902ddadf27c002171e91a4dc1eff5f3df515072ac63580",
"clobTokenIds": "[\"89984761009969090606365049502308215774844729861018229163736541869010668521883\", \"109098292314077657939935494027719386110066867173218583078610092308306402652377\"]",
"orderMinSize": 5,
"negRisk": true,
"active": true,
"closed": false,
"enableOrderBook": true,
"acceptingOrders": true,
"createdAt": "2026-11-04T16:54:27.166917Z"
},
{
"question": "Exact Score: South Shields 5 - 5 Southport FC?",
"conditionId": "0x526a2e209ecdc996126f7b0ce8aca970e238d00a5e8e1462583b319f161a81e2",
"clobTokenIds": "[\"23383971224876511286751474417288589641928550057174153221853770194514531463145\", \"102514263000521269570516172503620470436693609695432994526595167508530804022833\"]",
"orderMinSize": 5,
"negRisk": true,
"active": true,
"closed": false,
"enableOrderBook": true,
"acceptingOrders": true,
"createdAt": "2026-11-04T16:54:27.084521Z"
},
{
"question": "Exact Score: South Shields 5 - 4 Southport FC?",
"conditionId": "0xef7541f4318da1290833e7d4228eec646c8b8c3dc56c1e54f19aab5999b79eac",
"clobTokenIds": "[\"9590662984010730528349011455067967517680765388249930403038178415563136879677\", \"27306691953532238445589487371954210070154006783856682982476322761457890551225\"]",
"orderMinSize": 5,
"negRisk": true,
"active": true,
"closed": false,
"enableOrderBook": true,
"acceptingOrders": true,
"createdAt": "2026-11-04T16:54:27.011508Z"
},
{
"question": "Exact Score: South Shields 4 - 5 Southport FC?",
"conditionId": "0x650c95e95ded1b214ffd5ce3807ebbce5b53a5652e8d9bc4e5fc81a9b88e7e91",
"clobTokenIds": "[\"20427555995993676408608258992371178601145040240934408396949232679359337375578\", \"103063473308836065441608442719693527177410664791261885269576628238500009671761\"]",
"orderMinSize": 5,
"negRisk": true,
"active": true,
"closed": false,
"enableOrderBook": true,
"acceptingOrders": true,
"createdAt": "2026-11-04T16:54:26.943636Z"
},
{
"question": "Exact Score: South Shields 5 - 3 Southport FC?",
"conditionId": "0x6b8873a1138c840f212b1232decf41022d3c3d9152fc3630ac208ec373d223e4",
"clobTokenIds": "[\"113885867652625558735658932195210301609406698665198534717816428669601426385071\", \"111530920921450186157632707389308324089042969410875400892559036758646303736272\"]",
"orderMinSize": 5,
"negRisk": true,
"active": true,
"closed": false,
"enableOrderBook": true,
"acceptingOrders": true,
"createdAt": "2026-11-04T16:54:26.857401Z"
},
{
"question": "Exact Score: South Shields 4 - 4 Southport FC?",
"conditionId": "0xead8378060d14693e9b389460df33f4744f00624abebcac49f065fe4df71a8e1",
"clobTokenIds": "[\"74671658663122677156313407534212107516599429188436460342260016431503072138377\", \"98530071447174348550020247148894521999579673609192238144726023516722974096996\"]",
"orderMinSize": 5,
"negRisk": true,
"active": true,
"closed": false,
"enableOrderBook": true,
"acceptingOrders": true,
"createdAt": "2026-11-04T16:54:26.793546Z"
},
{
"question": "Exact Score: South Shields 3 - 5 Southport FC?",
"conditionId": "0x2b1ce687525f18ad61a59e36f23450d21f054f63f1e45f8ad1280599bcec5a9c",
"clobTokenIds": "[\"57945964795574745726585942326101714792053389461524739487276980421631849920286\", \"49065546046569817187747663996196121254480922109385243099359350773652868328963\"]",
"orderMinSize": 5,
"negRisk": true,
"active": true,
"closed": false,
"enableOrderBook": true,
"acceptingOrders": true,
"createdAt": "2026-11-04T16:54:26.720058Z"
},
{
"question": "Exact Score: South Shields 5 - 2 Southport FC?",
"conditionId": "0x73e7aad3c25b27f6ea5690533fbadda5232275bc3b2093be105bf2e8f0451f86",
"clobTokenIds": "[\"7212691718059316105797204867601393556290090412559254640981266981182975515579\", \"61803842573819523268874917567084084298265802112323835114562959941367399972826\"]",
"orderMinSize": 5,
"negRisk": true,
"active": true,
"closed": false,
"enableOrderBook": true,
"acceptingOrders": true,
"createdAt": "2026-11-04T16:54:26.635393Z"
},
{
"question": "Exact Score: South Shields 4 - 3 Southport FC?",
"conditionId": "0x39774e1c4ea209f84a503fd99093e2ba0857b1e4ac9850e7fc7aea843f9ab68f",
"clobTokenIds": "[\"95978542179162501111386271847728617693164545763728410170416353787697899447782\", \"82831019763051352453486760727930593092008868235043819951383814785887774411178\"]",
"orderMinSize": 5,
"negRisk": true,
"active": true,
"closed": false,
"enableOrderBook": true,
"acceptingOrders": true,
"createdAt": "2026-11-04T16:54:26.595371Z"
}
],
"next_cursor": "HD40wpBf4vc1G7nlWBWBeQio70FL4a-CS_Q9LExFBo17InYiOjEsImsiOiJtYXJrZXRzIiwib2giOiI5ZGExMThlZTZhMDViZmVkYTVjM2E1MzFjY2RhZWNkZTY0ZDlkYjQyZWQyMzQxZmE0NDcwNDBjOGViYWI1YzAxIiwia2V5cyI6W3sidCI6InRpbWUiLCJ2IjoiMjAyNi0xMS0wNFQxNjo1NDoyNi41OTUzNzFaIn0seyJ0Ijoic3RyaW5nIiwidiI6IjY2ODMyMTkifV19"
}
+100 -20
View File
@@ -4,139 +4,219 @@
"clobTokenIds": "[\"34722410608062854697106861099776685947172185964394483545370684749662285977831\", \"24483563228660028121020430722896740519524131851526579619586176931221901842798\"]",
"orderMinSize": 5,
"negRisk": true,
"enableOrderBook": true
"enableOrderBook": true,
"conditionId": "0x16c63b7cc37f012b9f59ee164ec03877914c701d06d48291ae8d6fc08a088b0d",
"active": true,
"closed": false,
"acceptingOrders": true
},
{
"question": "2026 Balance of Power: R Senate, R House",
"clobTokenIds": "[\"103704141773947678931823410030956181918562062788486034785782641603149828893320\", \"25869296386911654049715489068824757745174681288397238521780118949553760488529\"]",
"orderMinSize": 5,
"negRisk": true,
"enableOrderBook": true
"enableOrderBook": true,
"conditionId": "0xc5eae79d1ffe716572353962eb926b1e3964c500a4880a7a94f58408218ee76b",
"active": true,
"closed": false,
"acceptingOrders": true
},
{
"question": "Will Republicans lose the Alaska, Iowa, Kansas, Maine, Michigan, Ohio, and Texas Senate seats?",
"clobTokenIds": "[\"13535941900723044344907002451427688051727855016747811259193368015899108383450\", \"80727480162707694817992062873857913056698208056368499870278974630056151503113\"]",
"orderMinSize": 5,
"negRisk": true,
"enableOrderBook": true
"enableOrderBook": true,
"conditionId": "0xa32317ba83c3c83dc43ef260c3744c22ed8000db53d30bc09372c45cff5fb07a",
"active": true,
"closed": false,
"acceptingOrders": true
},
{
"question": "2026 Balance of Power: R Senate, D House",
"clobTokenIds": "[\"6302359956133594764084277082169634158291609371270652093164054687145970756151\", \"558600218792430158020964384658762756367786275998757133304987611939841005404\"]",
"orderMinSize": 5,
"negRisk": true,
"enableOrderBook": true
"enableOrderBook": true,
"conditionId": "0x998bc71817b2d76921d1999ce0f3431cfd5945583667a371280ca2b430b0c06e",
"active": true,
"closed": false,
"acceptingOrders": true
},
{
"question": "Will the Republicans win the Mississippi Senate race in 2026?",
"clobTokenIds": "[\"36589074670521348842786325911391532620430247097981345860356484520500173888573\", \"4639340586725578716005500791739559303082924048072534329825226706547465941088\"]",
"orderMinSize": 5,
"negRisk": true,
"enableOrderBook": true
"enableOrderBook": true,
"conditionId": "0xc8c5760c26490b0cd0bedcf1c61aa86c5e09c8b2886b84ff4ea2db81d5f29388",
"active": true,
"closed": false,
"acceptingOrders": true
},
{
"question": "Will Mary Peltola win the Alaska Senate race in 2026?",
"clobTokenIds": "[\"72139607190315861034411702195742538105117960260100800877125381826122747504258\", \"38274379050640062523611849065721372699838119672720475300345842413156391540232\"]",
"orderMinSize": 5,
"negRisk": true,
"enableOrderBook": true
"enableOrderBook": true,
"conditionId": "0x19d2ff72e389d190c183e6b114817301d29eba651d67d75a38ee201a05188fd9",
"active": true,
"closed": false,
"acceptingOrders": true
},
{
"question": "Will the Democrats win the Mississippi Senate race in 2026?",
"clobTokenIds": "[\"77001639760095266374534198365421436593389036687975321536722226607654949914567\", \"109329484956764506378760583725617100193692863947320202695087573398990069238209\"]",
"orderMinSize": 5,
"negRisk": true,
"enableOrderBook": true
"enableOrderBook": true,
"conditionId": "0x8c9b8ad3a64088a06d45e0b296a9a836203bc7f3ef35d134c57abead446a8b72",
"active": true,
"closed": false,
"acceptingOrders": true
},
{
"question": "2026 Balance of Power: D Senate, R House",
"clobTokenIds": "[\"70997927349469817841862065582625658840347600365813612622959588796331622340305\", \"45720947308530468765087874228584729250108790729495800420104302263408503667446\"]",
"orderMinSize": 5,
"negRisk": true,
"enableOrderBook": true
"enableOrderBook": true,
"conditionId": "0x0808de4f0cfd47947f2d1be51f9a9c52ea0fec76f73a75cfbe79ddec98d8a908",
"active": true,
"closed": false,
"acceptingOrders": true
},
{
"question": "Will Republicans win Iowa, and lose Alaska, Kansas, Maine, Michigan, Ohio, and Texas?",
"clobTokenIds": "[\"106253100341058888227154298197221161777912145262343559280892425406157624656915\", \"38883886691124225989332035378913105158890603529288875957747834557386418386450\"]",
"orderMinSize": 5,
"negRisk": true,
"enableOrderBook": true
"enableOrderBook": true,
"conditionId": "0xd116777a710b6a7c336fe12cbf742fafb0d8ac508cf400446159973cec9aec29",
"active": true,
"closed": false,
"acceptingOrders": true
},
{
"question": "Will the US confirm that aliens exist before 2027?",
"clobTokenIds": "[\"107505882767731489358349912513945399560393482969656700824895970500493757150417\", \"7305630249804085635496399869905769372294302716159034447326228509068694952392\"]",
"orderMinSize": 5,
"negRisk": false,
"enableOrderBook": true
"enableOrderBook": true,
"conditionId": "0x747dc809fb79e1b05be09c42d6179459a58de2ef3e40f02484a4e1260f741f75",
"active": true,
"closed": false,
"acceptingOrders": true
},
{
"question": "Rolex Paris Masters: Alexander Zverev vs Lorenzo Musetti",
"clobTokenIds": "[\"97202122435166371596632600271207731098197012320781592658111953138552052790871\", \"64561849525628736112915971493239643546218846988377083760471198523625509677016\"]",
"orderMinSize": 5,
"negRisk": false,
"enableOrderBook": true
"enableOrderBook": true,
"conditionId": "0x4b5ff3a152f6535ac82fbefb2034a8e5cf1444f6458b6f0491efabe9a7c73289",
"active": true,
"closed": false,
"acceptingOrders": true
},
{
"question": "Will Republicans win Alaska, and lose Iowa, Kansas, Maine, Michigan, Ohio, and Texas?",
"clobTokenIds": "[\"83965128865532878903206942328238684643656639972076348062693215949819184139617\", \"93889370639599634853686597701962997508099936018794651953480468245159849951085\"]",
"orderMinSize": 5,
"negRisk": true,
"enableOrderBook": true
"enableOrderBook": true,
"conditionId": "0xe3b51058c3bebfb100c5e1d59ab89d395c8938150a788f1417157fe360bea48a",
"active": true,
"closed": false,
"acceptingOrders": true
},
{
"question": "US x Iran ceasefire continues through October 31?",
"clobTokenIds": "[\"67102598924726349989742049657609063097299442010642912942032681022413402597017\", \"114475889232765203212032217024645308497828327658421967586050553616813506019119\"]",
"orderMinSize": 5,
"negRisk": false,
"enableOrderBook": true
"enableOrderBook": true,
"conditionId": "0x3738ad9665480991affe3ba286cd035516e17d1f8b7703e05088ec19c70c21a4",
"active": true,
"closed": false,
"acceptingOrders": true
},
{
"question": "Will Real Madrid CF win on 2026-11-04?",
"clobTokenIds": "[\"39515423851331707664526519001996875757487006788622502797081364834883774387767\", \"3771084942539439798998256191956824579778203065651547449472385427482459623087\"]",
"orderMinSize": 5,
"negRisk": true,
"enableOrderBook": true
"enableOrderBook": true,
"conditionId": "0x8941f985cce9ac4a175ce7a0a3d75f38536a43de46517d75aa9658ee2cf55574",
"active": true,
"closed": false,
"acceptingOrders": true
},
{
"question": "Will Republicans win Iowa and Kansas, and lose Alaska, Maine, Michigan, Ohio, and Texas?",
"clobTokenIds": "[\"105070040449310495327131730101841566321530681945885747881686068698995174349681\", \"111381299800799485509083912497813429956512687399857260440199290536659860679263\"]",
"orderMinSize": 5,
"negRisk": true,
"enableOrderBook": true
"enableOrderBook": true,
"conditionId": "0x564959d4c61eeb7c568d6f8e4f5a0dcf3622434d4f264d418d9674102e5a7a88",
"active": true,
"closed": false,
"acceptingOrders": true
},
{
"question": "Will the U.S. invade Iran before 2027?",
"clobTokenIds": "[\"55115078421062885512539156303747803058407616201213034911037320915726138659123\", \"1910830010387565971650098373488592514702818137344973088263643820608151819241\"]",
"orderMinSize": 5,
"negRisk": false,
"enableOrderBook": true
"enableOrderBook": true,
"conditionId": "0x5db999fad322cea2914535aae5517060c3f80ad6d8c0231cde2124a434d16846",
"active": true,
"closed": false,
"acceptingOrders": true
},
{
"question": "Will Benjamin Netanyahu be the next Prime Minister of Israel?",
"clobTokenIds": "[\"20006732765674855733524007935991362439352594042415161039767275461950712817548\", \"54533569528927082430133047556849600511985191158927477533124630301934489498530\"]",
"orderMinSize": 5,
"negRisk": true,
"enableOrderBook": true
"enableOrderBook": true,
"conditionId": "0x7586a96520578acaaaa4ea84a2582f197f84255da1f3392a7aa300386c187b37",
"active": true,
"closed": false,
"acceptingOrders": true
},
{
"question": "Will Dan Sullivan win the Alaska Senate race in 2026?",
"clobTokenIds": "[\"2618138322856109565904469085316346602627882727483900432328802655317079880335\", \"64264976954058187238724333554507208255303776435657387002861226761784169752307\"]",
"orderMinSize": 5,
"negRisk": true,
"enableOrderBook": true
"enableOrderBook": true,
"conditionId": "0x741bb46008b72234857c5118c8365ab3c2f69dbf28b30b59de04f173b8a778de",
"active": true,
"closed": false,
"acceptingOrders": true
},
{
"question": "Will Nithya Raman win the 2026 Los Angeles mayoral election?",
"clobTokenIds": "[\"50292933818751585369672690878774599873941191925538325333138798248932182222864\", \"58507084608782512565480280053882807505189448738098588563704773804344223603355\"]",
"orderMinSize": 5,
"negRisk": true,
"enableOrderBook": true
"enableOrderBook": true,
"conditionId": "0x694595d0a008f10e4701285d071e4b08407495f6f8a6bd8cbdd7de522fa9b463",
"active": true,
"closed": false,
"acceptingOrders": true
},
{
"question": "Will John McEntee win the 2028 Republican presidential nomination?",
"clobTokenIds": "[\"50730508396692945396787154191308995513666087024795374376582570929738968294242\", \"7808428608295632540409928244677767077616163699109217296255297216219773833140\"]",
"orderMinSize": 5,
"negRisk": true,
"enableOrderBook": true
"enableOrderBook": true,
"conditionId": "0x90ba5eeb935da5a2e7d16ee635ae441f95b5b62fd0db9336eb9f8a59645bf714",
"active": true,
"closed": false,
"acceptingOrders": true
}
]