diff --git a/CLAUDE.md b/CLAUDE.md index c719a46..a8bf0dc 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -11,13 +11,17 @@ outside this repo. reasons behind the design principles. Read before any design discussion. - `docs/roadmap.md` and `roadmap/NN. name.md`: milestones and each sprint's agreed plan. Read the current sprint file before writing code. -- `docs/adr/`: hard-to-reverse decisions (0001: threads, shared-nothing). +- `docs/adr/`: hard-to-reverse decisions (0001: threads, shared-nothing; + 0002: lanes and the orchestrator). - `docs/notes.md`: domain rules and exchange protocol facts (delta semantics, loss detection, message shapes). Read before touching exchange parsing. -- `GLOSSARY.md`: the terms (symbol, resync, loss, resync gap, Snapshot, Delta). -- `src/polymarket/feed.rs`: the book thread and its loss rules (the subtle - part); `src/polymarket/socket.rs`: the socket thread; `src/book.rs`: the - book and fixed point; `src/reconcile.rs` + `src/bin/reconcile.rs`: the output +- `GLOSSARY.md`: the terms (symbol, resync, loss, subscription gap, lane, + load, Snapshot, Delta). +- `src/polymarket/orchestrator.rs`: what is tracked, the lanes and their + balancing; `src/polymarket/gamma.rs`: the fetchers (discovery); + `src/polymarket/feed.rs`: a lane's book thread and its loss rules (the + subtle part); `src/polymarket/socket.rs`: a lane's connection; + `src/book.rs`: the book and fixed point; `src/reconcile.rs` + `src/bin/reconcile.rs`: the output check. `tests/fixtures/`: a 90 s recording of the live feed the tests replay. - `src/util/` and `sbe_schemas/`: the template's SBE machinery, used as given so output stays readable by the template's `AlignedSbeReader`. Our code is the @@ -45,22 +49,25 @@ and let the user settle it. Claude owns the data model. ## Environment traps -- **Polymarket and Kalshi are DNS-blocked on the dev network**: the local - resolver answers `195.186.4.210` / `2001:4d98:4:b10c::1` (a Swisscom block - page). The user pins the hosts with `sudo ../host.sh on` (workspace root, - outside this repo; `off` removes them). A timeout or the block address again - means the CDN addresses rotated: ask the user to rerun it. A host not yet in - `host.sh`'s `HOSTS` list fails the same way: add it there and ask the user to - rerun it. A 403 is not DNS (see `docs/notes.md`, REST from scripts). +- **DNS: the exchanges resolve normally on the server** (a remote-claude + workspace, since 2026-11-09). The Mac's dev network blocks them (its resolver + answers `195.186.4.210` / `2001:4d98:4:b10c::1`, a Swisscom block page); + there the user pins the hosts with `sudo ../host.sh on` (workspace root, + outside this repo; `off` removes them), rerun when the CDN addresses rotate. + A 403 is not DNS (see `docs/notes.md`, REST from scripts). - **Write 64 MiB chunks**: `AlignedSbeReader::open` hardcodes 64 MiB (`open_with_chunk_size` is private), so any other chunk size is unreadable. - **The collector runs until SIGINT, and only SIGINT flushes its output.** - macOS has no `timeout`, and a `perl alarm` kill (SIGALRM) leaves the last - 64 MiB chunk unwritten. Bound a run with SIGINT, then check the output: - `./target/release/engineering-challenge out.sbe 2> run.log & pid=$!; - sleep 600; kill -INT $pid; wait $pid; ./target/release/reconcile out.sbe`. - `reconcile` counts resync gaps as mismatches too: compare its list with the - log's "loss" and "resync gap" lines. + Any other signal (SIGTERM, SIGALRM) leaves the last 64 MiB chunk unwritten. + Bound a run with GNU `timeout -s INT`, then check the output: + `timeout -s INT 600 ./target/release/engineering-challenge out.sbe 2> run.log; + ./target/release/reconcile out.sbe`. `reconcile` counts subscription gaps + as mismatches too: compare its list with the log's "loss", "subscription + gap" and "first book after a move" lines. +- **Wait on a PID, never on a `pgrep -f` pattern**: `until ! pgrep -f + "capacity scan"` matches the waiting shell's own command line and never + ends, and `pkill -f PATTERN` can kill the shell running it. Take the PID + (`pgrep -f '^./target/release/...'`, anchored) and loop on `kill -0 $pid`. - **Live data beats the docs**: Polymarket's market channel has rules its docs don't state (`docs/notes.md`); check a protocol assumption on a recording of the real feed (`tests/fixtures/record.py`) before relying on it. diff --git a/GLOSSARY.md b/GLOSSARY.md index 8173ce1..aac4177 100644 --- a/GLOSSARY.md +++ b/GLOSSARY.md @@ -15,6 +15,12 @@ The unit that has one order book: on Polymarket a token (one outcome of a market), named by its token id. _Avoid_: Asset, instrument, ticker (for Polymarket) +**Live**: +Said of a market (and its symbols) the exchange lists as open for trading on +its order book. A market stops being live when it closes or stops accepting +orders, not at its end date. +_Avoid_: Active (Gamma's flag, only one input of being live) + **Book hash**: The exchange's fingerprint of a symbol's full book state, sent with every full book and every change. @@ -26,6 +32,21 @@ The collector's in-memory order book for one symbol: its bid and ask price levels, each with its absolute quantity. _Avoid_: Order book state, depth +**Lane**: +A set of symbols whose books one book thread owns, fed by its own +connections: one path from the wire to the writer. Every live symbol is in +exactly one lane. +_Avoid_: Shard (a technique, not this thing), group, partition + +**Load**: +How much a market weighs on its connection: the bytes per second of its +frames. A lane's load is the sum of its markets'. +_Avoid_: Activity, volume (the exchange's traded amount, unrelated) + +**Connection**: +One websocket to the exchange, carrying the symbols of one lane. +_Avoid_: Socket (for the concept; fine for the code), stream + **Receive time**: When a message came off the socket, by the local clock. _Avoid_: Arrival time, local timestamp @@ -37,14 +58,16 @@ _Avoid_: Gap (implies sequence numbers) **Resync**: Asking the exchange for a fresh full book of a symbol and replacing ours with -it, whenever our book can't be verified. Our book was wrong (a loss) only if -the fresh one differs. +it, whenever our book can't be verified. Our book was wrong only if the fresh +one differs: a loss, or a subscription gap. _Avoid_: Re-snapshot, rebuild, refresh -**Resync gap**: -The changes a symbol's resync itself keeps from us: the exchange holds them in -the fresh book but never sends them as changes. -_Avoid_: calling it a loss (the state stays right; only those updates are missing) +**Subscription gap**: +The changes our own subscribe keeps from us, whether it opens, resyncs or +moves a symbol: the exchange holds them in the book it sends but never sends +them as changes. +_Avoid_: Resync gap (the older, narrower name); calling it a loss (the state +stays right; only those updates are missing) ## Output diff --git a/docs/adr/0002-lanes-and-orchestrator.md b/docs/adr/0002-lanes-and-orchestrator.md new file mode 100644 index 0000000..3791efa --- /dev/null +++ b/docs/adr/0002-lanes-and-orchestrator.md @@ -0,0 +1,39 @@ +# 0002. Lanes and an orchestrator + +## Context + +Every live Polymarket symbol is ~476k tokens (2026-11-04), an estimated +10–25k frames/s, near the ~30k frames/s one book thread did in sprint 02. One +connection can carry 50,000 tokens, but the server closes a connection it +can't send to fast enough (1013, "slow consumer"), so connections must be +many and read by threads that do nothing else. The set of live symbols +changes by ~20k markets a day, and Gamma, the only complete source, takes +minutes to scan (~2 from the collector). + +## Decision + +Symbols are spread over **lanes**: each lane is a connection and a book +thread of its own, within two limits (a byte budget for its load, and 5,000 +tokens), and every lane's book thread writes to the one writer. An **orchestrator** thread is the single +owner of what is tracked: the market → lane map, the lanes and the timers. It +adds a new live market to the least-loaded lane, opening one when all are +full, removes markets that are gone, and moves the busiest markets off a lane +over its budget, from the load each lane reports every minute. Two fetcher threads only fetch Gamma (the +full scan; the new and closed markets since a time) and stream what they read +to it. + +## Why + +A lane shards the book state by construction: no routing stage between +connections and book threads, nothing shared, and redundancy (M4) adds +connections to a lane rather than reshaping the pipeline. A router to book +shards would decouple connection and book thread counts, which nothing needs +yet. The orchestrator keeps every decision in one thread that never blocks on +HTTP, which is where M4's source switching and M5's recycling will live; the +fetchers stay stateless so a scan never holds up the minute check. +Trade-off: a move unsubscribes a market on one lane before subscribing it on +the other, so it costs a few ms of its changes (a subscription gap); moving it +without that gap means two lanes writing one symbol, which is M4's source +switching. Moves are rate-limited so balancing doesn't churn. Balancing by +token count alone was tried first: the first live run had one lane closed by +the server six times, its markets created together and busy together. diff --git a/docs/notes.md b/docs/notes.md index 3b47e38..f4134ff 100644 --- a/docs/notes.md +++ b/docs/notes.md @@ -100,15 +100,81 @@ From a 60 s run of the M1 spike on one market, both tokens subscribed, and the answers `PONG`. The spike survived 60 s without it, but it is required. - **Server restarts:** the server can close the socket with code `Away`, reason `server shutting down` (seen once in about 8 minutes of connections). -- **`last_trade_price`:** `asset_id`, `price`, `size`, `side`, `fee_rate_bps`, - `timestamp`, `transaction_hash` (one per trade, for one token). -- **Not observed:** `tick_size_change` and the custom-feature events. The docs - now show these only in an SDK's wrapped form (`topic` / `payload`, - camelCase), not the raw wire shape above. +- **`last_trade_price`:** `market`, `asset_id`, `price`, `size`, `side`, + `fee_rate_bps`, `timestamp`, `transaction_hash` (one per trade, for one + token). +- **Not observed:** `market_resolved`, in 5 minutes with + `custom_feature_enabled` on 10 tokens. The docs now show it only in an + SDK's wrapped form (`topic` / `payload`, camelCase), not the raw wire + shape above. +- **`new_market` is broadcast** (with `custom_feature_enabled`, 2026-11-04): + every market created, subscribed or not (69 in 5 minutes), with + `condition_id`, `assets_ids` / `clob_token_ids`, `question`, `active` + (`false` for markets created ahead), but no `order_min_size` or `neg_risk`, + which the book hash needs. +- **Tokens per connection (2026-11-04):** no cap seen: one connection served + 50,000 tokens. But a first `{"type":"market"}` message with more than 500 + tokens gets no opening books (1,000 to 10,000 tried; changes still flow), + while `subscribe` operations of 500 or 1,000 get theirs. A connection opened + with `"assets_ids": []` stays up and serves a later `subscribe`. +- **Slow consumers are closed:** a connection the server can't send to fast + enough is closed with code 1013, reason `slow consumer: send buffer full` + (3 of 5 test connections, with a Python reader parsing every frame at up to + ~10,000 frames/s). The collector's reader, which only stamps, got it too: + six times in 8 minutes on the one connection carrying ~5,000 changes a + second (NFL quarter spreads, little traded); none on the others, the next + at ~1,300. No rate is safe (2026-11-10, server, 5 ms from the edge): four + connections ramped on random live tokens were closed at 0.7, 2.1, 2.3 and + 5.1 MB/s (minute averages of frame bytes), with subscribes of 5,000 or 500 + tokens alike. 5,000 random tokens now carry ~380 frames/s, ~200 KB/s. + Closes come together: 5 of 20 connections (~1 MB/s each) were closed + within 45 s of each other, one at 0.46 MB/s. +- **A subscribe hides a few ms of changes** (smoke run, 2026-11-04): after + subscribing a market on a busy connection, the first check of its opening + book often failed, and the fresh book differed from ours at levels none of + our changes had touched, though the opening book matched its own hash. The + server never sent those changes: like the resync's unsubscribe/subscribe, + any subscribe opens this gap (glossary: subscription gap). +- **`tick_size_change`** (recorded 2026-11-10 on every live token: 62 markets + in 10 minutes, all `0.01` to `0.001`): one object per token, both tokens of + the market, each sent twice on the same connection, identical: + `{"market", "asset_id", "old_tick_size", "new_tick_size", "timestamp", + "event_type": "tick_size_change"}`. +- **Load (2026-11-04):** a random 5,000 of the live tokens gave 133 frames/s + and 266 changes/s; their books had a median of 5 levels (mean 10.9, p90 29, + max 136), 15% empty. The top 50,000 by volume gave ~10,000 frames/s. - **REST from scripts:** `clob.polymarket.com` answers Python's `urllib` with 403 (curl and `ureq` get 200), most likely Cloudflare filtering by user agent. Fetch with curl in throwaway scripts. +## Gamma, the market listing (observed 2026-11-04) + +- **Size:** 238,308 markets `active` and not `closed`; 238,201 of them with + `enableOrderBook` and `acceptingOrders` (~476k tokens). 32,092 still accept + orders past their `endDate`, waiting to be resolved: the end date doesn't + mean closed. ~20k markets are created a day, mostly crypto "Up or Down" + markets created a day ahead and accepting orders at once. +- **Pages of 100**, whatever `limit` asks. `/markets` refuses offsets past + 2,000 ("use /markets/keyset"); `/markets/keyset` pages with `after_cursor` + (the `next_cursor` of the last page) and has no such limit. A full keyset + scan of the live markets took 6 min 41 s. +- **Sorting works, date filters don't:** `order=createdAt&ascending=false` + (newest first) and `closed=true&order=closedTime&ascending=false` sort both + endpoints, and keyset pages keep the order across cursors. `created_at_min` + and `closed_time_min` are ignored. 2–20 markets were created and 10–18 + closed a minute; more than 420 a minute changed `updatedAt`. +- **`conditionId`** is the websocket's `market`; `clobTokenIds` (a JSON array + inside a string) are its symbols. +- **Null hash inputs:** a null `orderMinSize` (31 live markets) hashes as + `"0"`, what REST's `/book` reports; a null `negRisk` (230) as `false`. +- **Scan speed:** the full keyset scan took 6 min 41 s from a Python script + spawning curl per page, ~2 minutes from the collector (`ureq`, one agent), + under a minute from the server (2026-11-10). +- **Indexing delay (2026-11-10):** new markets appear in the newest-first + listing 219 to 296 s after their `createdAt` (8 markets, polled every + 20 s): a check reaching back 5 minutes misses some. 195,130 markets were + live that evening. + ## Clock accuracy bounds time accuracy `received_at` comes from the local clock, so it carries the clock's error. On 2026-11-04 the dev machine ran ~30 ms slow (`sntp`: +30.6 ms ± 13 ms), which diff --git a/docs/project.md b/docs/project.md index 0a13d58..afe5355 100644 --- a/docs/project.md +++ b/docs/project.md @@ -42,8 +42,10 @@ The interacting decisions it is made of: ([ADR 0001](adr/0001-concurrency-model.md)). - **Stamping point:** in the socket-read path, immediately, so as little as possible sits between the wire and the receive timestamp. -- **Book ownership:** shard symbols so each book is owned by exactly one worker - (lock-free) rather than a shared, locked map. +- **Book ownership:** settled in sprint 03 — symbols are spread over lanes, + each a connection and a book thread of its own, so each book is owned by + exactly one thread; an orchestrator assigns them + ([ADR 0002](adr/0002-lanes-and-orchestrator.md)). - **Backpressure:** bounded vs unbounded channels — the core correctness/perf tension. Drop-on-full loses data (violates completeness); block adds latency and fills socket buffers. This decision is the bet in miniature. @@ -130,25 +132,37 @@ and drop from the global workflow for a job-challenge repo. - Any exchange beyond Polymarket and Kalshi. - Persistence/replay infrastructure beyond what the SBE output already gives. -## Pipeline, first sketch +## Pipeline The primary thing we debate sprint by sprint (this project's equivalent of the -schema debate). Rough shape: +schema debate). As settled in sprint 03: ``` -discovery ──▶ subscription manager ──▶ N socket readers (stamp receive time here) - │ - route by symbol - ▼ - per-symbol book workers (fold, loss-detect, dedup sources) - │ - ▼ - single output writer (SBE sink) + rescan fetcher ──findings──▶ ┌──────────────┐ ◀──findings── recent fetcher + (full Gamma scan) ◀─request─│ orchestrator │──request──▶ (new + closed since T) + └──┬────────▲──┘ + Add / MoveIn / Remove, to │ │ load reports, each minute + the market's lane │ │ (bytes/s per market) + ┌──────────────────────┼────────┴─────────────────┐ + ▼ ▼ ▼ + lane 0: book thread ◀─frames─ connection lane 1: ... lane N: ... + │ └─subscribe / unsubscribe / resync─▶┘ + └──────────── Snapshot | Delta ─────────────▶ writer thread (one) ``` -Each stage and the channels between them are open until settled in a sprint. -Settled so far: OS threads, shared-nothing, no mutexes -([ADR 0001](adr/0001-concurrency-model.md)). +- The **orchestrator** owns what is tracked (market → lane, the lanes, the + timers) and decides; it never blocks on HTTP, so later coordination + (switching sources in M4, recycling in M5) lives there too. +- The two **fetchers** only page Gamma on request and stream what they read. +- Each **lane** is a connection (stamps receive time, sends subscriptions and + resyncs) and a book thread (fold, check, resync, Snapshots); M4 gives a lane + several connections and has its book thread choose the trusted source. +- One **writer** owns the output. + +Settled: OS threads, shared-nothing, no mutexes +([ADR 0001](adr/0001-concurrency-model.md)); lanes and the orchestrator +([ADR 0002](adr/0002-lanes-and-orchestrator.md)). Details: +[sprint 03](../roadmap/03.%20full%20coverage.md). ## Data model, first sketch @@ -167,7 +181,13 @@ Not a DB — an in-memory domain plus the fixed SBE output schema. Main entities - **Book** (one per symbol): bids and asks as ordered price→quantity maps, plus the last loss-detection marker seen, last exchange timestamp, and the id of the source currently trusted (for redundancy). -- **Connection** (one websocket): the set of symbols it carries. +- **Connection** (one websocket): the set of symbols it carries, those of its + lane. +- **Lane**: a set of symbols, the book thread that owns their books, and the + connections that feed it. Live markets are assigned to lanes by + `conditionId`; Polymarket's lifecycle signal is Gamma (settled in sprint 03: + a market is live while active, not closed, with an order book and accepting + orders). - **Source / redundancy:** several connections may feed the same symbol; one is the trusted source at a time. @@ -181,9 +201,6 @@ trusted source is chosen and switched, and emitting a fresh snapshot on switch). - **Loss detection on Kalshi.** Its sequence semantics: checked in M6. (Polymarket: settled in M2, by its book hash.) -- **Symbol lifecycle signal.** What marks a market "no longer trading"? - Polymarket's `new_market` / `market_resolved` events (with - `custom_feature_enabled`) are candidates; settle in M3. - **Redundancy source-selection.** How to pick/switch the trusted source (freshness, message rate, which source is ahead). Designed in the redundancy sprint. diff --git a/docs/roadmap.md b/docs/roadmap.md index aed493b..fbffb6c 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -30,33 +30,35 @@ abstraction is *extracted* once a second exchange exists, not designed up front. Sprint: [../roadmap/02. book pipeline.md](../roadmap/02.%20book%20pipeline.md). *Ends:* correct SBE output for a symbol subset; tests green. -3. **Full coverage: discovery + sharding (base challenge complete)** — *not started* - Discovery with diffing (subscribe new, unsubscribe gone) — REST polling, or - Polymarket's `new_market` / `market_resolved` events — using subscribe / - unsubscribe on open connections; load-balance all live regular-market symbols - across N sockets. - **Opening questions (from M2):** what "live" means, and so how many symbols: - Gamma lists at least 100,000 markets `active` and not `closed` (2026-11-04, - count capped), at least 13,000 of them traded in the last 24 h, against the - 40 symbols M2 runs. At that size the minute Snapshots alone (~2 KB each) - would be ~0.5 TB a day for 200,000 symbols, mostly unchanged books; and - memory (~10 KB a book), sockets per connection, resyncs per minute and the - book thread's throughput (M2: ~30,000 frames/s on one thread, 43 frames/s - of load) all need measuring. Gamma pages past 10,000 markets only through - `/markets/keyset` with `after_cursor`. +3. **Full coverage: discovery + lanes (base challenge complete)** — *done (2026-11-11)* + Every live symbol (~476k tokens, 2026-11-04) spread over lanes, each a + connection and a book thread of its own; an orchestrator keeps the tracked + set equal to Gamma's live set (a scan at startup and every hour, new and + closed markets every minute) through subscribe / unsubscribe on open + connections. + Sprint: [../roadmap/03. full coverage.md](../roadmap/03.%20full%20coverage.md). *Ends:* every live Polymarket symbol tracked — base challenge satisfied for one exchange. 4. **Extension 1 — redundant websockets** — *not started* - Multiple connections per symbol; choose a trusted source and switch on - loss/quiet/delay; emit a fresh snapshot on every source switch. + Multiple connections per symbol (several connections in a lane, its book + thread choosing among them, the orchestrator coordinating); choose a + trusted source and switch on loss/quiet/delay; emit a fresh snapshot on + every source switch. Recovery from errors, with the lane budget scaling + itself down when connections still get 1013 closes (no rate is safe: + [notes.md](notes.md), so a budget set too high must correct itself). **Async re-evaluation checkpoint:** before building this, reassess whether the socket-management complexity (reconnect/timeout/select, redundancy) warrants swapping the thread-only runtime for tokio (see ADR 0001). + The lane that holds markets busy together (the 5- and 15-minute crypto + markets: closed 16 times in 75 minutes, in bursts a minute's load + average doesn't show) is the case to beat. Also: surviving a hard crash without losing the writer's in-memory chunk, - and closing the resync gap (the few ms of a symbol's changes the server - doesn't send between our unsubscribe and subscribe), both accepted as - ceilings in M2. + a move's Remove reaching the old lane before the new lane's first + Snapshot, the memory growth seen in sprint 03 (~90 MB every 10 minutes + over 75 minutes), and closing the subscription gap (the few ms of a + symbol's changes the server doesn't send between our unsubscribe and + subscribe); the crash chunk and the gap were accepted as ceilings in M2. *Ends:* redundancy demonstrably reduces data loss; a Snapshot on every source switch. @@ -106,7 +108,16 @@ abstraction is *extracted* once a second exchange exists, not designed up front. - Observability: metrics, structured logging, dashboards. - Config-driven symbol filtering and per-exchange tuning. - Containerized packaging. -- Counting a reconnect's differing books as losses rather than resync gaps - (a reconnect marker through the frames channel; M2 labels them gaps). -- A `reconcile` that reads the collector's log, to tell resync gaps from +- Counting a reconnect's differing books as losses rather than subscription + gaps when the symbol was resyncing or not yet verified since its opening + book (a reconnect marker through the frames channel; they are labelled + gaps). +- A `reconcile` that reads the collector's log, to tell subscription gaps from losses in its own report. +- Polymarket's `new_market` websocket event as a faster discovery signal (it + is broadcast, but lacks `orderMinSize` / `negRisk`, so Gamma is still + needed; `market_resolved` was never seen). +- Closing lanes that drain. +- Fewer Snapshots for unchanged books (the brief asks for every symbol every + minute: ~130 MB a minute at full coverage). +- Re-reading a market's `orderMinSize` / `negRisk` if they can change. diff --git a/roadmap/03. full coverage.md b/roadmap/03. full coverage.md new file mode 100644 index 0000000..2e49b14 --- /dev/null +++ b/roadmap/03. full coverage.md @@ -0,0 +1,431 @@ +# 03. Full coverage: discovery and lanes + +**Status: done (2026-11-11)** + +Track every live Polymarket symbol, which completes the base challenge for one +exchange. An orchestrator keeps the tracked set equal to the live set (Gamma, +scanned at startup and every hour, checked every minute for new and closed +markets) and spreads the symbols over lanes, each a connection and a book +thread of its own, all writing through the one writer; it balances the lanes +by the load they report. Design: +[../docs/project.md](../docs/project.md) (pipeline), +[ADR 0002](../docs/adr/0002-lanes-and-orchestrator.md) (lanes, orchestrator), +[../docs/notes.md](../docs/notes.md) (protocol facts), +[../GLOSSARY.md](../GLOSSARY.md) (live, lane, connection, load, subscription +gap). + +## The target, measured (2026-11-04) + +Measured before planning, as sprint 02's retrospective asked; the throwaway +scripts read Gamma and the market channel live. + +- **Size.** Gamma lists 238,308 markets active and not closed; 238,201 of + them have an order book and accept orders, so ~476k symbols. 20,212 traded + in the last 24 h (the day after the US midterms). 32,092 accept orders past + their end date, waiting to be resolved. ~20k markets are created a day, + mostly crypto "Up or Down" markets created a day ahead. +- **Load.** A random sample of 2,500 markets (5,000 tokens) on one connection + for 90 s: 4,998 books within 0.45 s; levels per book median 5, mean 10.9, + p90 29, max 136, 15% empty; 133 frames/s, 266 changes/s. The top 50,000 + tokens by volume: ~10,000 frames/s. Estimated for all symbols: 10–25k + frames/s, near the ~30k frames/s one book thread did in sprint 02. +- **Connections.** No cap on tokens per connection: one served 50,000 (books + for 48,798, changes for 27,882 in 40 s). But a first subscribe message of + more than 500 tokens gets no opening books (1,000 to 10,000 tried), while + subscribe operations of 500 or 1,000 do. A connection opened with no token + stays up and serves a later subscribe. The server closes a connection it + can't send to fast enough: code 1013, "slow consumer: send buffer full" (3 + of 5 connections of the Python test reader, which parsed every frame). +- **Gamma.** 100 markets a page whatever `limit` says: a full scan of + `/markets/keyset` takes 6 min 41 s. Sorting by `createdAt` (live markets) or + `closedTime` (`closed=true`) works, and keyset pages keep that order across + `after_cursor`; the `*_min` date filters are ignored, and `/markets` refuses + offsets past 2,000. 2–20 markets are created and 10–18 closed a minute; + more than 420 a minute change `updatedAt`. +- **Websocket lifecycle events.** With `custom_feature_enabled`, `new_market` + is broadcast for every market, subscribed or not (69 in 5 minutes), with its + token ids but not `orderMinSize` or `negRisk`, which the hash needs. No + `market_resolved` came in those 5 minutes. +- **Estimated output.** Minute Snapshots ~130 MB a minute (476k books of ~275 + bytes); Deltas ~190 MB a minute (~25k changes/s of ~125 bytes): ~320 MB + a minute in all. Books in memory: + ~1 GB. The machine: 16 GB, 10 cores. + +## The smoke run (2026-11-04) + +An 8-minute live run of the first build, on every live symbol. It changed the +plan (Decisions mark what it superseded): + +- **Startup**: the scan took ~2 minutes, not 7 (the Python estimate paid a + process per page): 239,752 markets in 96 lanes. ~1.3 GB of memory, 198 + threads; 3.15 GB of output (~390 MB a minute). +- **One lane was closed by the server six times** (1013, slow consumer): lane + 85, which held NFL quarter-spread markets created together, so filled + together. It carried ~5,000 verified changes a second; the next busiest + lane ~1,300. Their 24 h volume was small: volume says nothing of load. +- **2,758 "losses" and 1,296 resync gaps.** Logged per loss: most were + complete books answering a resync, for a symbol whose first check after its + opening book had failed; the levels that differed were ones our changes + never touched. Our subscribe keeps a few ms of changes from us, as a resync + does: a **subscription gap**. The rest were books after lane 85's + reconnects, changes really missed while it was down. +- **62 `Unverifiable` symbols**: the 31 markets with a null `orderMinSize` + (fixed, see Data model). +- **Left for later in this sprint**: `reconcile` found 3,315 mismatches for + 4,054 logged losses and gaps, which should be equal; `tick_size_change` + came live for the first time, logged twice per symbol (checked 2026-11-10: + the server sends each twice, and the second asks for nothing). + +## Pipeline + +``` + rescan fetcher ──findings──▶ ┌──────────────┐ ◀──findings── recent fetcher + (full Gamma scan) ◀─request─│ orchestrator │──request──▶ (new + closed since T) + └──┬────────▲──┘ + Add / MoveIn / Remove, to │ │ load reports, each minute + the market's lane │ │ (bytes/s per market) + ┌──────────────────────┼────────┴─────────────────┐ + ▼ ▼ ▼ + lane 0: book thread ◀─frames─ connection lane 1: ... lane N: ... + │ └─subscribe / unsubscribe / resync─▶┘ + └──────────── Snapshot | Delta ─────────────▶ writer thread (one) +``` + +- **Orchestrator** (one thread): the single owner of what is tracked. It holds + the market map (market, by `conditionId` → its lane and tokens), the lanes it + opened and how many tokens each holds, and every timer. Each minute it asks + the recent fetcher for what changed since its last successful check; every + hour it asks the rescan fetcher for a full scan. A request while the + previous one of its kind is still running is skipped. It applies the + findings: a live market it doesn't know is added to the lane with the + least load that still has token room, opening a lane when none has; a + closed market it knows is removed. It reads the lanes' load reports and + rebalances (below). It waits on the findings and reports with a timeout + until its next timer, at most 1 s, so it sees SIGINT. Lane health beyond + load (up, quiet, reconnecting) is M4's and M5's. +- **Fetchers** (two threads): each answers a request by paging Gamma and + streaming what it reads, and holds no state or timer. Two, so a 6 min 41 s + scan never holds up the minute check. + - **Rescan**: `Started`, every page of live markets, then `Done`, or + `Failed` after 5 failed tries at one page, 5 s apart. + - **Recent**: the live markets created since T, then the markets closed + since T (each in its keyset order, down to T), then `Done` or `Failed`. + T is the start of the orchestrator's last successful check, minus 15 + minutes for Gamma's indexing delay; known markets are skipped, so the + margin costs only the pages it reads again. +- **Lanes**: a lane is opened by the orchestrator with its first market and + never closed. Its book thread waits on its frames, on the orchestrator's + commands and on the next wall-clock minute (`crossbeam`'s `select!`). It + learns a market (its tokens, `min_order_size`, `neg_risk`) or forgets it, + then asks its connection to subscribe or unsubscribe the tokens: the book + thread knows a symbol before its book can arrive. At each wall-clock minute + it sends the orchestrator its load report. Everything else is sprint 02's + book thread. +- **Connection** (one per lane): sprint 02's socket thread, now keeping its own + token set from the book thread's commands. It opens with an empty + subscribe message and subscribes its tokens in operations of at most 500, + on every connect and reconnect. +- **Writer**: unchanged, fed by every lane's book thread through one channel. + +**Load balancing.** A market's **load** is the bytes per second of its frames, +every event type, measured by its lane's book thread over each minute, or +since the market joined the lane if that is later; a market measured for +under 30 s waits for the next report, since its opening books alone would read +as a burst (a `price_change` frame is one market's; a book frame is shared by +the books it holds). A lane has two +limits: a byte budget, 300 KB/s (under the lowest rate at which a connection +was seen to get a 1013 close), and 5,000 tokens, since a reconnect replays +every book at once. A new market joins the lane with the least load that still has token +room (ties: fewest tokens); at startup nothing is measured yet, so the scan +fills lanes in order, and rebalancing undoes the clusters. Each minute, on the +reports, every lane over its budget, the busiest lane first, has its busiest +markets moved, one by one, to the least-loaded lane with room for them in both +limits (a new lane if none has), until it is under budget: at most 50 moves a +minute, and a moved market stays put for 10 minutes. A market alone on its +lane is never moved: over budget on its own, it would only carry its load from +lane to lane; nor is one with no load, which would lighten nothing. A move is +a Remove on the old lane, then a MoveIn on the new one: an Add whose lane +writes the market's first book as a Snapshot, the changes in between being a +subscription gap. The new lane can't tell whether that book differs from the +old lane's, so it logs every first book after a move. + +**Subscription gaps.** A fresh book whose levels differ from ours is a +subscription gap, not a loss, when it answers our resync (sprint 02's rule: +complete, at or after our last change, not behind), or when no change of the +symbol has been verified since its opening book; it is logged with its cause +(opening, resync). A book after a reconnect that differs stays a loss: those +changes were missed while the connection was down. + +**Startup** is the first rescan, retried by the orchestrator until it succeeds: +markets are assigned and subscribed page by page, so the first books are live +within seconds and lanes open as they fill. The minute checks run from the +start, the first covering everything since startup. + +**Removal.** A market is gone when it no longer matches the live predicate: +the minute check removes the markets newly `closed`; a completed rescan +removes the markets that were known at its `Started` and not seen in it (one +that stopped accepting orders, or vanished from the listing). Markets added +during a rescan are not at risk, and a failed rescan removes nothing. A +removed symbol is unsubscribed and dropped: nothing is written for it, and its +frames still in flight are ignored, like any unknown token's. + +**Shutdown.** SIGINT reaches the orchestrator and the connections through the +stop flag. Each connection closes, so its book thread drains and ends; the +orchestrator drops its sender to the writer, which flushes once every sender +is gone. Every Gamma request has a 10 s timeout, so a request in flight delays +nothing past that; the fetchers aren't waited for. A fetcher that dies (only a +panic ends one) stops the run, which then fails: nothing would keep the +tracked set current. + +## Data model + +- **Live** (Gamma): `active`, not `closed`, `enableOrderBook` and + `acceptingOrders`. The end date doesn't count. A live market missing a hash + input takes what the hash uses (230 had a null `negRisk`: `false`, all + verified live; 31 a null `orderMinSize`: `"0"`, as REST's `/book` says; the + first guess, `"5"`, made all 62 tokens `Unverifiable` in the smoke run). A + wrong value shows as `Unverifiable`, its Deltas still written. A market with no token ids is not tracked. +- **Orchestrator**: `conditionId` → (lane, tokens, last reported load, when + it last moved); per lane, its token count, its load (the sum of its + markets') and its command sender. The market is the unit of assignment, + removal and moves: its two tokens always share a lane and leave together. +- **Lane limits**: 5,000 tokens, and a byte budget of 300 KB/s (a constant, + from the capacity experiment). +- **Load report** (book thread → orchestrator, each minute): its lane number + and, per market (`conditionId`), its load since the last report. +- **Book thread**: per symbol, sprint 02's state, plus whether a change has + been verified since its opening book (the subscription gap rule); per + market, its bytes since the last report and since when. Memory is measured on the final run; if it + passes 4 GB, the recent hashes become `[u8; 20]` instead of `String`. + +## Decisions + +- **Live means Gamma's active, not closed, with an order book and accepting + orders: every such symbol, ~476k.** The brief asks for all live symbols, and + the measured size is within reach. *Rejected:* only markets with recent + trades or liquidity, which misses a market the moment it starts trading. +- **Lanes: each a connection and a book thread of its own, one writer for + all** (ADR 0002). The symbols are sharded by construction, with no routing + stage, and M4 adds connections to a lane. *Rejected:* a router to book + shards by symbol (an extra stage, sizes we don't need to decouple yet); + several connections into one book thread (at its measured ceiling). +- **An orchestrator decides, two fetchers only fetch.** One owner of the + market map, the lanes and the timers, which never blocks on HTTP; the room + for M4's source switching and M5's recycling. *Rejected:* a discovery thread + that both fetches and decides (blocks on Gamma); fetchers with their own + timers (two places deciding when to rescan); one fetcher (the scan holds up + the minute check); interleaving scan pages with minute checks on one thread + (more loop state for the same result). +- **The market is keyed by `conditionId`.** The websocket calls it `market`. + *Rejected:* token keys (two entries that must never split); Gamma's `id` + (the websocket never shows it). +- **Lanes are balanced by load, measured as bytes per second per market by + the lanes** (supersedes "the least-full lane by token count" and "5,000 + tokens, halved to 2,500 on a 1013 close", because the smoke run's one + closed lane held markets created together, with little volume and ~4 times + the next lane's changes: halving would only split the cluster in two). + Bytes, because the 1013 close is the server's send buffer filling. + *Rejected:* token counts alone (what failed); changes or frames per second + (their size varies); Gamma volume (shown wrong); the first lane with room + (hot spots); a fixed lane count (doesn't follow growth). +- **The byte budget is 300 KB/s, under the lowest rate seen to close one + connection (0.46 MB/s); 5,000 tokens stay a second limit** (supersedes + "half the rate measured to close one connection", because connections on + the server were closed anywhere from 0.46 to 5.1 MB/s: there is no single + rate). A 1013 can still come below it; recovering from one is M4's. + *Rejected:* a guessed budget; tokens alone; 500 or 350 KB/s (too near the + lowest close); 230 KB/s, half of it (more lanes for a rule no rate obeys). +- **A new market joins the least-loaded lane with room; rebalancing moves the + busiest markets off a lane over budget, busiest lane first, at most 50 a + minute, each staying 10 minutes.** Startup clusters can't be seen before + they are measured, and the limits stop markets bouncing. Busiest lane first, + since the 75-minute run's hottest lane (closed 4 times in 3 minutes) lost 10 + symbols in 2 minutes while lanes numbered before it took the 50 moves. *Rejected:* round-robin at startup (needs the + lane count before the scan ends); weighting by volume; never moving. +- **A move is break-before-make: Remove, then MoveIn (an Add).** The new lane's first book + is a Snapshot, so the output stays right, and the few ms between are a + subscription gap. *Rejected:* make-before-break (two lanes writing one + symbol: M4's source switching). +- **A market's load is measured from when it joins its lane, and needs 30 s + before it is reported** (found by the reviews: over a partial minute it was + under-reported, and over the last seconds of one its opening books read as + a burst; the 15-minute run's first rebalance made its full 50 moves on + them). *Rejected:* a full minute for everyone (a new market weighs nothing + for up to two minutes); a 1 s floor (opening books still read as load). +- **Lanes send the orchestrator a load report each minute** (supersedes "no + lane health reports in M3": load is the first). *Rejected:* shared atomic + counters (shared state). +- **Subscription gap replaces resync gap: the changes our own subscribe keeps + from us, whether it opens, resyncs or moves the symbol** (supersedes sprint + 02's "resync gap", because the smoke run showed opening subscribes hiding + changes the same way). A differing book counts as one when it answers a + resync, or when nothing has been verified since the opening book; books + after a reconnect stay losses. *Rejected:* a second term for the opening + case (two names for one mechanism); leaving them as losses (every loss + explained could never pass). +- **Minute check by `createdAt` and `closedTime`, both on keyset pages, down + to the last check minus 15 minutes; a full rescan every hour** (supersedes + "plain pages, a full rescan past offset 2,000", because keyset pages keep + the sort and have no offset limit; and "minus 5 minutes", because the + 75-minute run's rescan added 347 markets the minute checks had missed, and + Gamma listed new markets 3.7 to 5 minutes after their `createdAt`). *Rejected:* polling by `updatedAt` + (more than 420 a minute); `new_market` events (they lack the hash's + metadata, and `market_resolved` never came: Possible features). +- **Only a completed rescan removes, and only what it could have seen.** A + failed scan would make half the markets look gone, and markets the minute + check adds during a scan may sit on pages already passed. *Rejected:* + removing whatever a scan didn't list. +- **Every first book after a move is logged** (found by the 75-minute run: + 506 of its 25,621 mismatches had no log line, 468 of them within 3 s of a + minute's moves). Only the old lane had the book to compare it with, so the + check counts a mismatch at a logged move as its subscription gap. + *Rejected:* sending the old lane's book with the move (shared state between + lanes, for a log line); leaving the mismatches unexplained. +- **A removed symbol writes nothing.** An empty Snapshot would state a book + the exchange never sent. *Rejected:* an empty Snapshot. +- **Connections open with an empty subscribe and subscribe in operations of + at most 500** (supersedes "the first message holds the first 500", because + an empty first message was tried and works: one rule, no special first + chunk). *Rejected:* larger operations (1,000 worked once; 500 costs + nothing). +- **Subscribe page by page at startup.** The first books come within + seconds. *Rejected:* scan, then subscribe (7 minutes of nothing, then one + burst). +- **Each lane logs its own minute line**, prefixed with its number, and the + orchestrator its own (markets tracked and lanes; added, removed and moved + this minute; the busiest lane's load). *Rejected:* + one aggregated line (a stats channel and an owner for the totals). +- **10 s timeout on every Gamma request.** SIGINT then waits at most that + long. *Rejected:* no timeout (a stalled request hangs the shutdown). +- **Lanes that drain stay open.** The least-loaded rule refills them first, + and a connection with nothing subscribed stays up. *Rejected:* closing + empty lanes. + +## Tests + +No network, except the local websocket server below. The 40-symbol replay +tests of sprint 02 stay. + +- **Gamma parsing**: a saved keyset page of live markets and one of closed + markets parse. +- **Minute check**: it stops at T, and skips markets already known. +- **Rescan**: a completed one removes only the markets known at `Started` + and not seen; one added during it stays; a failed one removes nothing. +- **Assignment**: a market joins the least-loaded lane with token room + (ties: fewest tokens); a lane opens when none has room; both tokens share a + lane. +- **Rebalancing**: a lane over budget has its busiest markets moved to the + least-loaded lanes with room until it is under, the busiest lane first; no + more than 50 moves a minute; a market moved less than 10 minutes ago stays, + and one with no load stays; a move is a Remove to the old lane, then a + MoveIn to the new one. +- **Load report**: a book thread's report holds each market's bytes per + second, every event type counted, over the minute or since it joined; one + measured under 30 s waits for the next report. +- **Subscription gap**: a symbol whose first check after its opening book + fails, then gets a differing fresh book, counts a subscription gap, not a + loss; one verified since counts a loss. +- **Move**: a market moved in logs its symbols' first books, once. +- **Feed**: an Add mid-run asks for a subscribe, and the symbol's first book + is written as a Snapshot; after a Remove, its frames are ignored and it is + out of the minute Snapshots. +- **Reconnect**: the connection loop against a local `tungstenite` server + (tried: a client reconnects to one inside `cargo test`) that closes the + socket once; on reconnect every token is subscribed again, in operations of + at most 500. + +## Not in this sprint + +- Closing lanes. +- `new_market` / `market_resolved` as a discovery signal. +- Shrinking the Snapshot volume (the brief asks for every symbol every minute). +- Lane health beyond load, redundant connections, closing subscription gaps, + the crash chunk (M4); activity monitoring, recycling (M5). +- A market's `min_order_size` or `neg_risk` changing after it was added: read + once, a ceiling (a change would make its books fail their own hash). + +## Done when + +- `scripts/ci.sh` is green. +- A 75-minute live run on every live symbol, started only once the user says + so and ended by SIGINT (startup scan, ~70 minute checks, one hourly rescan), + writes a file that reads back with `AlignedSbeReader`, and: + - the markets tracked at the end match Gamma's live count, within a + minute's churn; + - `reconcile`'s mismatches are exactly the logged losses and subscription + gaps (and, since the 75-minute run, the logged first books after a move); + - every `Unverifiable` symbol is counted and its cause looked into: fixed if + the formula is wrong, else explained here; + - no connection was closed with 1013, or each is explained; + - peak memory (sampled with `ps`) and each lane's peak of frames waiting + are reported. +- Free disk is checked before the run (~24 GB of output estimated, 39 GB + free today), and the output file is deleted after the check. +- `docs/notes.md` holds the measured facts above; `docs/project.md`'s + pipeline sketch and the roadmap match the lanes and the orchestrator. + +## Outcome + +Every live Polymarket symbol is tracked: ~195k markets (~390k symbols) over +85–89 lanes, kept equal to Gamma's live set by the minute checks and the +hourly rescan, and balanced by the load the lanes report. The base challenge +is met for one exchange. + +**The 75-minute run** (2026-11-10, 16:27–17:42, on the server): startup scan +under a minute (189,474 markets in 76 lanes); 194,935 markets tracked at the +end against 195,130 live in Gamma ten minutes later; 30.2 GB written (~400 MB a +minute), read back whole by `reconcile`: 29.1M Snapshots, 175.6M Deltas, no +orphan Delta. 25,115 of its 25,621 mismatches were logged losses and +subscription gaps; the other 506 were moved markets, whose first book on the +new lane nobody logged (fixed: every first book after a move is logged; a +15-minute run then explained all 6,137 of its mismatches). No `Unverifiable` +symbol. Peak memory 1.81 GB, 183 threads (sampled from minute 14 on; it grew +~90 MB every 10 minutes). Peak frames waiting: 1,295, on lane 68. + +**1013 closes**: 31 in 75 minutes, and 2 resets. Lane 68 took 16 of them, +then lane 69: the crypto "Up or Down" markets of 5 and 15 minutes, created +together a day ahead and busy together when their windows open and close, so +many closes fall on 5-minute marks. The rest came several lanes within a +second, which only the server explains. Rebalancing doesn't cure that lane: +its load sits in a few bursty markets that stay put 10 minutes once moved, and +a minute average hides the bursts that get it closed. Recovery is M4's. + +**Diverged from the plan**, each in Decisions: the byte budget is 300 KB/s, +under the lowest close (no closing rate exists to halve); loads are bytes per +second over every event type, from when the market joined and after 30 s of +measuring; rebalancing takes the +busiest lane first and leaves markets with no load; first books after a move +are logged; the minute check reaches back 15 minutes, not 5 (Gamma lists new +markets up to 5 minutes late; the hourly rescan found 347 markets missed). +`tick_size_change` was recorded live: sent twice per token, harmless. + +**Left out**, and where it went: counting a reconnect's differing books as +losses (Possible features); delivering a move's Remove before its Add's first +Snapshot across lanes (M4, with source switching); reacting to 1013 closes and +the burst-driven lane (M4); the memory growth (M4's list). + +### Retrospective + +- **Went well:** measuring before deciding. The capacity ramps overturned the + budget rule, a live capture settled `tick_size_change`, a ten-minute poll + explained the missed markets. Pairing `reconcile`'s mismatches with the log + by symbol and time found the unlogged moves, and the code review found the + rebalancing order that starved the hottest lane. +- **Went wrong:** the 75-minute run used a binary built before the review's + fixes, so a second run was needed to see them live. A first capacity + reading was taken for a ceiling from one sample, and the budget was stated + as "half the lowest close" before the last closes came in. Wait loops + written as `until ! pgrep -f PATTERN` matched their own shell and ran for + 12 hours; a `pkill -f` stopped its own shell, and the memory sampler watched + `timeout` for its first 14 minutes. +- **Tested enough?** The rules are, by replay and unit tests. Not covered: + the minute check's T and the 5 s retries (inside the orchestrator's loop); + the orchestrator loop itself (sending each command, stopping on a dead + fetcher), shown only by live runs; a reconnect followed by a loss. The last + review's fixes (the 30 s measuring floor, a dead fetcher failing the run) + came after the last live run and haven't run live. +- **Next time:** review the code before the long acceptance run; wait on a + PID, never on a `pgrep -f` pattern (now in CLAUDE.md); the run check + (`reconcile` against the log) is needed by every acceptance run: it is the + Possible feature "a `reconcile` that reads the collector's log", worth + doing early in M4. diff --git a/src/main.rs b/src/main.rs index dd229df..2eebd08 100644 --- a/src/main.rs +++ b/src/main.rs @@ -1,5 +1,7 @@ -//! The collector: three threads joined by channels (roadmap/02): -//! socket (read, stamp) → book (parse, fold, check, resync) → writer (SBE). +//! The collector (roadmap/03): an orchestrator keeps the tracked markets equal +//! to Gamma's live ones and spreads them over lanes, each a connection (read, +//! stamp) and a book thread (parse, fold, check, resync), all writing through +//! one writer (SBE). //! Usage: `engineering-challenge [OUTPUT]`, OUTPUT defaulting to stdout; logs //! go to stderr. @@ -20,54 +22,45 @@ macro_rules! log { mod polymarket; use engineering_challenge::book::Output; -use polymarket::feed::Feed; /// `AlignedSbeReader::open` reads nothing else. const CHUNK: usize = 64 << 20; -const MARKETS: u32 = 20; fn main() -> anyhow::Result<()> { let output = std::env::args() .nth(1) .unwrap_or_else(|| "/dev/stdout".to_string()); - let markets = polymarket::top_markets(MARKETS)?; - let feed = Feed::new(&markets); - let tokens = feed.tokens(); let stop = Arc::new(AtomicBool::new(false)); let on_sigint = Arc::clone(&stop); ctrlc::set_handler(move || on_sigint.store(true, Ordering::Relaxed))?; - let (frames_tx, frames_rx) = unbounded(); - let (resyncs_tx, resyncs_rx) = unbounded(); let (outputs_tx, outputs_rx) = unbounded(); - // Each stage stops when the one before it does, so SIGINT only reaches the - // socket thread; the writer flushes whatever came before. - let socket = - thread::spawn(move || polymarket::socket::run(&tokens, &frames_tx, &resyncs_rx, &stop)); - let book = - thread::spawn(move || polymarket::feed::run(feed, frames_rx, outputs_tx, resyncs_tx)); - let writer = thread::spawn(move || write(&output, outputs_rx)); - - // Every thread is joined, so the writer flushes even after a panic - // upstream; the run still fails then. - let socket_panicked = socket.join().is_err(); - let book_panicked = book.join().is_err(); + let writer_stop = Arc::clone(&stop); + let writer = thread::spawn(move || { + let written = write(&output, outputs_rx); + // A writer that failed stops the rest: nothing it reads is kept. + writer_stop.store(true, Ordering::Relaxed); + written + }); + // SIGINT stops the orchestrator and the connections; each book thread + // stops when its connection does, and the writer flushes once they all + // have, even after a panic upstream (the run still fails then). + let failed = polymarket::orchestrator::run(outputs_tx, &stop); let written = writer .join() .map_err(|_| anyhow::anyhow!("the writer thread panicked"))?; log!("output closed"); written?; - anyhow::ensure!(!socket_panicked, "the socket thread panicked"); - anyhow::ensure!(!book_panicked, "the book thread panicked"); + anyhow::ensure!(failed == 0, "{failed} threads failed"); Ok(()) } -/// The writer thread: writes outputs in arrival order until the book thread -/// is gone, then flushes the last chunk. +/// The writer thread: writes outputs in arrival order until every book +/// thread is gone, then flushes the last chunk. // ponytail: the chunk being filled lives only in memory until it is full, so -// a hard crash loses it (hours of data); flushing more often pads every flush -// to 64 MiB. Taken up in M4. +// a hard crash loses it; flushing more often pads every flush to 64 MiB. +// Taken up in M4. fn write(path: &str, outputs: Receiver) -> anyhow::Result<()> { let mut writer = AlignedSbeWriter::create(path, CHUNK)?; for output in outputs { diff --git a/src/polymarket/feed.rs b/src/polymarket/feed.rs index 6f79f81..8a07236 100644 --- a/src/polymarket/feed.rs +++ b/src/polymarket/feed.rs @@ -1,32 +1,58 @@ //! The book thread: parses market-channel frames, folds them into one book per //! symbol, checks them against Polymarket's book hash, resyncs when a check -//! fails, and emits Deltas and Snapshots. The rules are in -//! `roadmap/02. book pipeline.md`; the hash formula in `docs/notes.md`. +//! fails, and emits Deltas and Snapshots. The protocol facts and the hash +//! formula are in `docs/notes.md`, the terms (loss, resync, subscription gap, +//! load) in `GLOSSARY.md`. -use std::collections::{BTreeMap, VecDeque}; +use std::collections::{BTreeMap, HashMap, VecDeque}; use std::time::Duration; use chrono::{DateTime, DurationRound, TimeDelta, Utc}; -use crossbeam::channel::{Receiver, RecvTimeoutError, Sender}; +use crossbeam::channel::{Receiver, Sender, never, select}; use engineering_challenge::Side; use engineering_challenge::util::dtos::challenge_orderbook::PriceLevelUpdateDomain; use serde::{Deserialize, Serialize}; use sha1::{Digest, Sha1}; use super::Market; -use super::socket::Frame; +use super::socket::{Command, Frame}; use engineering_challenge::book::{Book, Output, format_fixed, parse_fixed}; +/// A market measured for less than this waits for the next load report: over +/// a few seconds, its opening books alone would read as a burst. +const MIN_MEASURED: TimeDelta = TimeDelta::seconds(30); + /// How far back a full book is recognised as behind our stream: the lag seen /// live is a few ms, a handful of changes. const RECENT_CHANGES: usize = 64; -/// What the book thread does with a frame: write output, or ask the socket -/// thread for a resync. +/// What the book thread does with a frame or a command: write output, ask its +/// connection for something, or report its load to the orchestrator. #[derive(Debug)] pub enum Effect { Write(Output), - Resync(String), + Send(Command), + Report(Loads), +} + +/// Each market's load (`conditionId` → bytes per second), every market of the +/// lane measured for at least [`MIN_MEASURED`]. +pub type Loads = HashMap; + +/// A lane's loads since its last report, for the orchestrator. +#[derive(Debug)] +pub struct LoadReport { + pub lane: usize, + pub loads: Loads, +} + +/// What the orchestrator tells a lane's book thread. +#[derive(Debug)] +pub enum LaneCommand { + Add(Market), + /// An Add for a market moved here from another lane. + MoveIn(Market), + Remove(Market), } #[derive(Debug, Clone, Copy, PartialEq, Eq)] @@ -34,7 +60,7 @@ enum State { /// No book yet since subscribing: changes are dropped until it comes. AwaitingBook, /// Its last full book matched its hash: every change is checked. - Live, + Verified, /// A check failed and a fresh book is asked for: changes are applied and /// written unchecked until it comes. After a trade, it usually equals ours. Resyncing, @@ -53,6 +79,12 @@ struct Symbol { /// Its last [`RECENT_CHANGES`] changes: a full book carrying the hash of /// one of them is the exchange's state after it. recent_changes: VecDeque, + /// Whether a change has been verified since its opening book: until one + /// is, a differing fresh book is the gap our subscribe opened. + verified_since_book: bool, + /// Moved in from another lane, no book yet: its first book holds the + /// move's subscription gap, which only the old lane's book would show. + moved_in: bool, // The other inputs of the book hash, as the exchange writes them. market: String, tick_size: String, @@ -62,6 +94,30 @@ struct Symbol { } impl Symbol { + fn new(market: &Market) -> Self { + Self { + state: State::AwaitingBook, + book: Book::default(), + exchange_timestamp: DateTime::UNIX_EPOCH, + open_batch: None, + recent_changes: VecDeque::new(), + verified_since_book: false, + moved_in: false, + market: String::new(), + tick_size: String::new(), + last_trade_price: String::new(), + min_order_size: market.min_order_size.clone(), + neg_risk: market.neg_risk, + } + } + + /// A change matched the exchange's hash: counted in `verified_changes`, + /// and the book was right since its opening book. + fn verified(&mut self, verified_changes: &mut u64) { + *verified_changes += 1; + self.verified_since_book = true; + } + /// Asks for a fresh book, unless one is already coming. fn resync(&mut self, token: &str, effects: &mut Vec) { if matches!(self.state, State::AwaitingBook | State::Resyncing) { @@ -77,7 +133,7 @@ impl Symbol { self.state = State::Resyncing; } self.open_batch = None; - effects.push(Effect::Resync(token.to_string())); + effects.push(Effect::Send(Command::Resync(token.to_string()))); } /// Whether `book` is ours as it was after one of our recent changes: ours @@ -102,6 +158,29 @@ impl Symbol { } } +/// A market's frame bytes since `since`. +struct Measure { + bytes: u64, + since: DateTime, +} + +impl Measure { + /// Bytes per second since `since`, then starts again from `now`; none + /// before [`MIN_MEASURED`], the bytes kept for the next report. + fn take(&mut self, now: DateTime) -> Option { + let measured = now - self.since; + if measured < MIN_MEASURED { + return None; + } + let load = self.bytes * 1_000 / measured.num_milliseconds() as u64; + *self = Measure { + bytes: 0, + since: now, + }; + Some(load) + } +} + /// A change as applied, with the exchange's hash of the book after it. struct RecentChange { side: Side, @@ -113,17 +192,25 @@ struct RecentChange { } pub struct Feed { + /// Its lane's number, for the logs. + lane: usize, symbols: BTreeMap, + /// Bytes of frames per market since its last load report. + measures: HashMap, /// The receive time of the last frame applied: the time a Snapshot's book /// is exact as of. last_received_at: Option>, - next_snapshots: Option>, + /// The next wall-clock minute: its Snapshots, load report and log line. + next_minute_at: Option>, verified_changes: u64, resyncs: u64, /// Full books that were a state we had already passed (see `on_book`). books_behind: u64, - /// Fresh books ahead of our stream by changes our resync kept from us. - resync_gaps: u64, + /// Fresh books holding changes our own subscribe kept from us. + subscription_gaps: u64, + /// First books of markets moved in, each logged: whether one holds a + /// subscription gap only the old lane's book would tell. + books_after_moves: u64, /// Full books whose levels differed from ours otherwise. losses: u64, /// The most frames waiting for the book thread since the last minute log: @@ -132,42 +219,56 @@ pub struct Feed { } impl Feed { - pub fn new(markets: &[Market]) -> Self { - let symbols = markets - .iter() - .flat_map(|market| { - market.tokens.iter().map(|token| { - let symbol = Symbol { - state: State::AwaitingBook, - book: Book::default(), - exchange_timestamp: DateTime::UNIX_EPOCH, - open_batch: None, - recent_changes: VecDeque::new(), - market: String::new(), - tick_size: String::new(), - last_trade_price: String::new(), - min_order_size: market.min_order_size.clone(), - neg_risk: market.neg_risk, - }; - (token.clone(), symbol) - }) - }) - .collect(); + pub fn new(lane: usize) -> Self { Self { - symbols, + lane, + symbols: BTreeMap::new(), + measures: HashMap::new(), last_received_at: None, - next_snapshots: None, + next_minute_at: None, verified_changes: 0, resyncs: 0, books_behind: 0, - resync_gaps: 0, + subscription_gaps: 0, + books_after_moves: 0, losses: 0, peak_waiting: 0, } } - pub fn tokens(&self) -> Vec { - self.symbols.keys().cloned().collect() + /// Tracks a market's symbols, then asks the connection to subscribe them: + /// a book can't arrive for a symbol not yet known. Its load is measured + /// from `now`. + pub fn add_market(&mut self, market: &Market, now: DateTime) -> Vec { + for token in &market.tokens { + self.symbols.insert(token.clone(), Symbol::new(market)); + } + let measure = Measure { + bytes: 0, + since: now, + }; + self.measures.insert(market.condition_id.clone(), measure); + vec![Effect::Send(Command::Subscribe(market.tokens.clone()))] + } + + /// [`Feed::add_market`] for a market moved in from another lane. + pub fn move_in(&mut self, market: &Market, now: DateTime) -> Vec { + let effects = self.add_market(market, now); + for token in &market.tokens { + self.symbols.get_mut(token).expect("just added").moved_in = true; + } + effects + } + + /// Forgets a market's symbols and unsubscribes them; nothing is written + /// for them. Their frames still on the way are ignored, as any unknown + /// token's. + pub fn remove_market(&mut self, market: &Market) -> Vec { + for token in &market.tokens { + self.symbols.remove(token); + } + self.measures.remove(&market.condition_id); + vec![Effect::Send(Command::Unsubscribe(market.tokens.clone()))] } pub fn on_frame(&mut self, received_at: DateTime, text: &str) -> Vec { @@ -186,51 +287,74 @@ impl Feed { } }, }; + // A frame is one market's change, or books of several: each book takes + // its share. + let share = (text.len() / events.len().max(1)) as u64; for event in events { + let market = match &event { + Event::Book(book) => Some(&book.market), + Event::PriceChange(change) => Some(&change.market), + Event::TickSizeChange { market, .. } | Event::LastTradePrice { market } => { + market.as_ref() + } + Event::Other => None, + }; + if let Some(measure) = market.and_then(|market| self.measures.get_mut(market)) { + measure.bytes += share; + } match event { Event::Book(book) => self.on_book(received_at, book, &mut effects), Event::PriceChange(change) => { self.on_price_change(received_at, change, &mut effects) } - Event::TickSizeChange { asset_id } => { + Event::TickSizeChange { asset_id, .. } => { if let Some(symbol) = self.symbols.get_mut(&asset_id) { log!("polymarket: {asset_id}: tick size changed, resync"); symbol.resync(&asset_id, &mut effects); } } - // Among them `last_trade_price`: the hash takes a new last - // trade price before this event, and not always its value, so - // a fresh book brings it instead (docs/notes.md). - Event::Other => {} + // The hash takes a new last trade price before this event, + // and not always its value, so a fresh book brings it instead + // (docs/notes.md). + Event::LastTradePrice { .. } | Event::Other => {} } } self.resyncs += count_resyncs(&effects[ticked..]); effects } - /// Once `now` reaches the next wall-clock minute: emits the minute - /// Snapshots, and asks again for the fresh books still not come. + /// Once `now` reaches the next wall-clock minute: reports the minute's + /// load, emits the minute Snapshots, and asks again for the fresh books + /// still not come. pub fn tick(&mut self, now: DateTime) -> Vec { - let next = *self.next_snapshots.get_or_insert_with(|| next_minute(now)); + let next = *self.next_minute_at.get_or_insert_with(|| next_minute(now)); if now < next { return Vec::new(); } - self.next_snapshots = Some(next_minute(now)); + self.next_minute_at = Some(next_minute(now)); + let loads = self + .measures + .iter_mut() + .filter_map(|(market, measure)| Some((market.clone(), measure.take(now)?))) + .collect(); + let mut effects = vec![Effect::Report(loads)]; let Some(received_at) = self.last_received_at else { - return Vec::new(); + return effects; }; log!( - "polymarket: so far {} changes verified, {} resyncs, {} full books behind ours, \ - {} resync gaps, {} losses; at most {} frames waited this minute", + "polymarket: lane {}: {} symbols; so far {} changes verified, {} resyncs, {} full books behind ours, \ + {} subscription gaps, {} books after moves, {} losses; at most {} frames waited this minute", + self.lane, + self.symbols.len(), self.verified_changes, self.resyncs, self.books_behind, - self.resync_gaps, + self.subscription_gaps, + self.books_after_moves, self.losses, self.peak_waiting ); self.peak_waiting = 0; - let mut effects = Vec::new(); for (token, symbol) in &mut self.symbols { if symbol.state != State::AwaitingBook { effects.push(Effect::Write(symbol.book.snapshot( @@ -249,8 +373,8 @@ impl Feed { /// A full book replaces ours, brought forward first if it is behind our /// stream, and is written as a Snapshot when it is the symbol's first - /// since subscribing, or its levels differ from ours (a resync gap or a - /// loss). Then it is checked against its own hash, as received. + /// since subscribing, or its levels differ from ours (a subscription gap + /// or a loss). Then it is checked against its own hash, as received. fn on_book( &mut self, received_at: DateTime, @@ -285,6 +409,13 @@ impl Feed { let verified = hash(&token, symbol, &book, &message.timestamp) == message.hash; let first = symbol.state == State::AwaitingBook; + if first && symbol.moved_in { + // The check matches it with the Snapshot below if that differs + // from the old lane's last. + symbol.moved_in = false; + self.books_after_moves += 1; + log!("polymarket: {token}: first book after a move, a subscription gap if it differs"); + } let mut book = book; let book_time = exchange_timestamp; let mut exchange_timestamp = exchange_timestamp; @@ -312,17 +443,24 @@ impl Feed { brought_forward = true; } let differs = !first && book != symbol.book; - // A fresh book answering our resync can also be ahead: between our - // unsubscribe and subscribe the server sent us none of its changes. - let gap = differs - && !brought_forward + // A subscribe hides a few ms of changes (docs/notes.md): a fresh book + // answering our resync can be ahead of our stream, since between our + // unsubscribe and subscribe the server sent none of its changes; and + // so can any book before a change was verified since the opening one. + let opening = !symbol.verified_since_book; + let resync = !brought_forward && complete && symbol.state == State::Resyncing && book_time >= symbol.exchange_timestamp; - let kind = if gap { "resync gap" } else { "loss" }; + let gap = differs && (opening || resync); + let kind = match (gap, opening) { + (false, _) => "loss", + (true, true) => "subscription gap (opening)", + (true, false) => "subscription gap (resync)", + }; if differs { if gap { - self.resync_gaps += 1; + self.subscription_gaps += 1; } else { self.losses += 1; } @@ -347,7 +485,7 @@ impl Feed { } if verified { - symbol.state = State::Live; + symbol.state = State::Verified; } else if complete { if symbol.state != State::Unverifiable { log!( @@ -355,10 +493,10 @@ impl Feed { ); } symbol.state = State::Unverifiable; - } else if matches!(symbol.state, State::AwaitingBook | State::Live) { - // Its last trade price may be stale: a fresh book carries it. Live - // first, since it has a book now, which resync() requires. - symbol.state = State::Live; + } else if matches!(symbol.state, State::AwaitingBook | State::Verified) { + // Its last trade price may be stale: a fresh book carries it. + // Verified first, since it has a book now, which resync() requires. + symbol.state = State::Verified; symbol.resync(&token, effects); } } @@ -384,7 +522,7 @@ impl Feed { .take_if(|(_, batch_hash)| *batch_hash != change.hash) { if hash(&token, symbol, &symbol.book, ×tamp) == batch_hash { - self.verified_changes += 1; + symbol.verified(&mut self.verified_changes); } else { symbol.resync(&token, effects); } @@ -416,9 +554,9 @@ impl Feed { hash: change.hash.clone(), }); symbol.exchange_timestamp = exchange_timestamp; - if symbol.state == State::Live { + if symbol.state == State::Verified { if hash(&token, symbol, &symbol.book, &message.timestamp) == change.hash { - self.verified_changes += 1; + symbol.verified(&mut self.verified_changes); } else { symbol.open_batch = Some((message.timestamp.clone(), change.hash)); } @@ -435,8 +573,8 @@ impl Feed { } } -/// The book thread: runs until the socket thread is gone, then drops its -/// senders so the writer thread finishes too. +/// A lane's book thread: runs until its connection is gone, then drops its +/// senders so the writer thread finishes once every lane has. // ponytail: unbounded channels never block the socket read nor drop a frame; // the ceiling is memory if a consumer stays behind. The minute log's peak of // frames waiting makes that visible; bounded channels are a constructor change @@ -444,20 +582,41 @@ impl Feed { pub fn run( mut feed: Feed, frames: Receiver, + commands: Receiver, outputs: Sender, - resyncs: Sender, + connection: Sender, + reports: Sender, ) { + let no_commands = never(); + let mut orchestrator_gone = false; loop { let until_next_minute = (next_minute(Utc::now()) - Utc::now()) .to_std() .unwrap_or(Duration::ZERO); - let effects = match frames.recv_timeout(until_next_minute) { - Ok((received_at, text)) => { - feed.peak_waiting = feed.peak_waiting.max(frames.len()); - feed.on_frame(received_at, text.as_str()) - } - Err(RecvTimeoutError::Timeout) => feed.tick(Utc::now()), - Err(RecvTimeoutError::Disconnected) => return, + let commands = if orchestrator_gone { + &no_commands + } else { + &commands + }; + let effects = select! { + recv(frames) -> frame => match frame { + Ok((received_at, text)) => { + feed.peak_waiting = feed.peak_waiting.max(frames.len()); + feed.on_frame(received_at, text.as_str()) + } + Err(_) => return, + }, + recv(commands) -> command => match command { + Ok(LaneCommand::Add(market)) => feed.add_market(&market, Utc::now()), + Ok(LaneCommand::MoveIn(market)) => feed.move_in(&market, Utc::now()), + Ok(LaneCommand::Remove(market)) => feed.remove_market(&market), + // It is stopping: the connection's end still decides ours. + Err(_) => { + orchestrator_gone = true; + Vec::new() + } + }, + default(until_next_minute) => feed.tick(Utc::now()), }; for effect in effects { match effect { @@ -466,8 +625,15 @@ pub fn run( return log!("polymarket: the writer thread is gone, stopping"); } } - // The socket thread may be gone already, while stopping. - Effect::Resync(token) => _ = resyncs.send(token), + // The connection may be gone already, while stopping, and the + // orchestrator too. + Effect::Send(command) => _ = connection.send(command), + Effect::Report(loads) => { + _ = reports.send(LoadReport { + lane: feed.lane, + loads, + }) + } } } } @@ -531,11 +697,19 @@ enum Events { enum Event { Book(BookMessage), PriceChange(PriceChangeMessage), - /// Its wire shape was never observed: only the symbol is read, and it - /// resyncs, since the fresh book carries the new tick size. + /// Only the symbol and its market are read, and it resyncs, since the fresh book carries + /// the new tick size. The server sends each one twice: the second finds + /// the symbol resyncing and asks for nothing (docs/notes.md). + /// `market` was on every one seen; without it, only the load misses it. TickSizeChange { + market: Option, asset_id: String, }, + LastTradePrice { + market: Option, + }, + /// Not counted in any load: the others come only with + /// `custom_feature_enabled`, which we don't ask for. #[serde(other)] Other, } @@ -560,6 +734,7 @@ struct LevelMessage { #[derive(Deserialize)] struct PriceChangeMessage { + market: String, timestamp: String, price_changes: Vec, } @@ -600,7 +775,7 @@ fn next_minute(now: DateTime) -> DateTime { fn count_resyncs(effects: &[Effect]) -> u64 { effects .iter() - .filter(|effect| matches!(effect, Effect::Resync(_))) + .filter(|effect| matches!(effect, Effect::Send(Command::Resync(_)))) .count() as u64 } @@ -615,6 +790,17 @@ mod tests { use super::*; use crate::polymarket::fixture_markets; + /// A lane's feed holding `markets`, as the orchestrator's Adds leave it + /// at the start of the recording. + fn feed_of(markets: &[Market]) -> Feed { + let mut feed = Feed::new(0); + let start = fixture_frames()[0].0; + for market in markets { + feed.add_market(market, start); + } + feed + } + /// The recorded live feed for the fixture's markets (tests/fixtures/README.md). fn fixture_frames() -> Vec<(DateTime, String)> { #[derive(Deserialize)] @@ -653,8 +839,8 @@ mod tests { effects .iter() .filter_map(|effect| match effect { - Effect::Resync(token) => Some(token.as_str()), - Effect::Write(_) => None, + Effect::Send(Command::Resync(token)) => Some(token.as_str()), + _ => None, }) .collect() } @@ -685,7 +871,7 @@ mod tests { /// the recording must replay without a single loss. #[test] fn replay_has_no_loss() { - let mut feed = Feed::new(&fixture_markets()); + let mut feed = feed_of(&fixture_markets()); replay(&mut feed, &fixture_frames()); assert_eq!(feed.losses, 0); assert!( @@ -712,7 +898,7 @@ mod tests { let markets = fixture_markets(); // The full replay, noting the change frames whose symbols are all // checked when they come. - let mut control = Feed::new(&markets); + let mut control = feed_of(&markets); let mut candidates = Vec::new(); let control_effects: Vec> = frames .iter() @@ -723,7 +909,7 @@ mod tests { && !tokens.is_empty() && tokens .iter() - .all(|token| control.symbols[token].state == State::Live) + .all(|token| control.symbols[token].state == State::Verified) { candidates.push(index); } @@ -733,7 +919,7 @@ mod tests { let lost = candidates[0]; let lost_tokens = changed_tokens(&frames[lost].1); - let mut feed = Feed::new(&markets); + let mut feed = feed_of(&markets); replay(&mut feed, &frames[..lost]); let (caught, token) = frames .iter() @@ -766,7 +952,7 @@ mod tests { let effects = feed.on_frame(frames[caught].0, &book.to_string()); assert_eq!(snapshot_symbols(&effects), [token.as_str()]); assert_eq!(feed.losses, losses + 1); - assert_eq!(feed.symbols[&token].state, State::Live); + assert_eq!(feed.symbols[&token].state, State::Verified); } /// A feed replayed past a mid-stream book and its symbol's next change, and @@ -788,7 +974,7 @@ mod tests { let moved_on = (stale + 1..frames.len()) .find(|&index| changed_tokens(&frames[index].1).contains(&token)) .unwrap(); - let mut feed = Feed::new(&fixture_markets()); + let mut feed = feed_of(&fixture_markets()); replay(&mut feed, &frames[..=moved_on]); (feed, token, frames[moved_on].0, frames[stale].1.clone()) } @@ -839,25 +1025,26 @@ mod tests { assert!(snapshot_symbols(&effects).is_empty()); assert_eq!( - (feed.books_behind, feed.resync_gaps, feed.losses), + (feed.books_behind, feed.subscription_gaps, feed.losses), (1, 0, 0) ); assert_eq!(feed.symbols[&token].book, ours); - assert_eq!(feed.symbols[&token].state, State::Live); + assert_eq!(feed.symbols[&token].state, State::Verified); } /// Between our unsubscribe and subscribe the server sends no changes, so - /// the fresh book can hold some we never got: a resync gap, not a loss. + /// the fresh book can hold some we never got: a subscription gap, not a + /// loss. #[test] - fn a_fresh_book_ahead_of_our_stream_is_a_resync_gap() { + fn a_fresh_book_ahead_of_our_stream_is_a_subscription_gap() { let frames = fixture_frames(); let now = frames.last().unwrap().0; - let mut feed = Feed::new(&fixture_markets()); + let mut feed = feed_of(&fixture_markets()); replay(&mut feed, &frames); let (token, symbol) = feed .symbols .iter_mut() - .find(|(_, symbol)| symbol.state == State::Live) + .find(|(_, symbol)| symbol.state == State::Verified) .unwrap(); let token = token.clone(); symbol.resync(&token, &mut Vec::new()); @@ -870,9 +1057,9 @@ mod tests { let effects = feed.on_frame(now, &book); assert_eq!(snapshot_symbols(&effects), [token.as_str()]); - assert_eq!((feed.resync_gaps, feed.losses), (1, 0)); + assert_eq!((feed.subscription_gaps, feed.losses), (1, 0)); assert_eq!(feed.symbols[&token].book, ahead); - assert_eq!(feed.symbols[&token].state, State::Live); + assert_eq!(feed.symbols[&token].state, State::Verified); } /// The level test must not mistake our wrong book for a newer one: we @@ -941,7 +1128,7 @@ mod tests { for market in &mut markets { market.min_order_size = "6".to_string(); } - let mut feed = Feed::new(&markets); + let mut feed = feed_of(&markets); let frames = fixture_frames(); let opening = feed.on_frame(frames[0].0, &frames[0].1); assert_eq!(snapshot_symbols(&opening).len(), 40); @@ -963,7 +1150,7 @@ mod tests { #[test] fn frames_that_are_not_events_are_skipped() { - let mut feed = Feed::new(&fixture_markets()); + let mut feed = feed_of(&fixture_markets()); let at = fixture_frames()[0].0; for text in [ "PONG", @@ -976,18 +1163,268 @@ mod tests { } } + /// A market's opening books, as a subscribe brings them. + fn opening_books(frames: &[(DateTime, String)], market: &Market) -> String { + let opening: Vec = serde_json::from_str(&frames[0].1).unwrap(); + let books: Vec<_> = opening + .iter() + .filter(|book| { + market + .tokens + .iter() + .any(|token| book["asset_id"] == **token) + }) + .collect(); + serde_json::to_string(&books).unwrap() + } + + #[test] + fn a_moved_market_logs_its_first_books_once() { + let frames = fixture_frames(); + let markets = fixture_markets(); + let mut feed = feed_of(&markets[1..]); + replay(&mut feed, &frames[1..100]); + + let effects = feed.move_in(&markets[0], frames[100].0); + assert!(matches!( + effects.as_slice(), + [Effect::Send(Command::Subscribe(tokens))] if *tokens == markets[0].tokens + )); + let books = opening_books(&frames, &markets[0]); + let effects = feed.on_frame(frames[100].0, &books); + assert_eq!(snapshot_symbols(&effects).len(), 2); + assert_eq!(feed.books_after_moves, 2); + // Later books are the lane's own business. + feed.on_frame(frames[101].0, &books); + assert_eq!(feed.books_after_moves, 2); + assert_eq!((feed.subscription_gaps, feed.losses), (0, 0)); + } + + #[test] + fn a_market_added_mid_run_is_subscribed_and_its_first_book_written() { + let frames = fixture_frames(); + let markets = fixture_markets(); + let mut feed = feed_of(&markets[1..]); + replay(&mut feed, &frames[1..100]); + + let effects = feed.add_market(&markets[0], frames[100].0); + assert!(matches!( + effects.as_slice(), + [Effect::Send(Command::Subscribe(tokens))] if *tokens == markets[0].tokens + )); + let effects = feed.on_frame(frames[100].0, &opening_books(&frames, &markets[0])); + let mut written = snapshot_symbols(&effects); + written.sort(); + let mut tokens: Vec<&str> = markets[0].tokens.iter().map(String::as_str).collect(); + tokens.sort(); + assert_eq!(written, tokens); + assert!( + markets[0] + .tokens + .iter() + .all(|token| feed.symbols[token].state == State::Verified) + ); + } + + #[test] + fn a_removed_market_is_unsubscribed_and_written_no_more() { + let frames = fixture_frames(); + let markets = fixture_markets(); + let mut feed = feed_of(&markets); + replay(&mut feed, &frames[..1]); + + let removed = &markets[0].tokens; + let effects = feed.remove_market(&markets[0]); + assert!(matches!( + effects.as_slice(), + [Effect::Send(Command::Unsubscribe(tokens))] if tokens == removed + )); + // The rest of the recording crosses two minutes: no Delta, Snapshot or + // resync names a removed token. + let effects = replay(&mut feed, &frames[1..]); + let mentions = |effect: &Effect| match effect { + Effect::Write(Output::Snapshot(snapshot)) => removed.contains(&snapshot.symbol), + Effect::Write(Output::Delta(delta)) => removed.contains(&delta.symbol), + Effect::Send(Command::Resync(token)) => removed.contains(token), + Effect::Send(_) => false, + Effect::Report(loads) => loads.contains_key(&markets[0].condition_id), + }; + assert!(!effects.iter().any(mentions)); + // The minute Snapshots of the 38 others, twice. + assert_eq!(snapshot_symbols(&effects).len(), 38 * 2); + } + + /// Seen live: our subscribe hides changes made just before the opening + /// book reaches us, so the first fresh book differs at a level none of our + /// changes touched, before any change was verified. + #[test] + fn a_differing_book_before_any_verified_change_is_a_subscription_gap() { + let frames = fixture_frames(); + let mut feed = feed_of(&fixture_markets()); + replay(&mut feed, &frames[..1]); + let (token, symbol) = feed + .symbols + .iter() + .find(|(_, symbol)| symbol.state == State::Verified) + .unwrap(); + let token = token.clone(); + assert!(!symbol.verified_since_book); + let mut exchange = symbol.book.clone(); + exchange.set(Side::Buy, 1_000, 123_000_000); + let book = fresh_book(&token, symbol, &exchange); + let effects = feed.on_frame(frames[1].0, &book); + + assert_eq!(snapshot_symbols(&effects), [token.as_str()]); + assert_eq!((feed.subscription_gaps, feed.losses), (1, 0)); + assert_eq!(feed.symbols[&token].book, exchange); + } + + /// Once a change has been verified since the opening book, our book was + /// right then: a fresh book that differs is a loss. + #[test] + fn a_differing_book_after_a_verified_change_is_a_loss() { + let frames = fixture_frames(); + let mut feed = feed_of(&fixture_markets()); + replay(&mut feed, &frames[..frames.len() / 2]); + let (token, symbol) = feed + .symbols + .iter() + .find(|(_, symbol)| symbol.state == State::Verified && symbol.verified_since_book) + .unwrap(); + let token = token.clone(); + let mut exchange = symbol.book.clone(); + exchange.set(Side::Buy, 1_000, 123_000_000); + let book = fresh_book(&token, symbol, &exchange); + let (gaps, losses) = (feed.subscription_gaps, feed.losses); + let effects = feed.on_frame(frames[frames.len() / 2].0, &book); + + assert_eq!(snapshot_symbols(&effects), [token.as_str()]); + assert_eq!((feed.subscription_gaps, feed.losses), (gaps, losses + 1)); + assert_eq!(feed.symbols[&token].book, exchange); + } + + #[test] + fn each_minute_reports_the_bytes_per_second_of_every_market() { + let minute = "2026-11-10T16:00:00Z".parse::>().unwrap(); + let at = |seconds| minute + TimeDelta::seconds(seconds); + let market = |id: &str| Market { + condition_id: id.to_string(), + tokens: vec![format!("{id}a"), format!("{id}b")], + min_order_size: "5".to_string(), + neg_risk: false, + }; + let (busy, quiet, late) = (market("0xbusy"), market("0xquiet"), market("0xlate")); + let mut feed = Feed::new(0); + feed.add_market(&busy, at(0)); + // Added halfway through the minute: measured over its 30 s. + feed.add_market(&quiet, at(30)); + // Added 15 s before it: kept for the next report. + feed.add_market(&late, at(45)); + let change = |id: &str| { + serde_json::json!({"event_type": "price_change", "market": id, "timestamp": "1", + "price_changes": []}) + .to_string() + }; + let trade = serde_json::json!({"event_type": "last_trade_price", "market": "0xbusy", + "asset_id": "0xbusya", "price": "0.5", "size": "1", "side": "BUY", + "timestamp": "1"}) + .to_string(); + let tick_size = serde_json::json!({"event_type": "tick_size_change", "market": "0xquiet", + "asset_id": "0xquieta", "old_tick_size": "0.01", "new_tick_size": "0.001", + "timestamp": "1"}) + .to_string(); + // One frame, two markets: each takes half. + let shared = format!("[{},{}]", change("0xbusy"), change("0xquiet")); + let mut reports = Vec::new(); + let mut on_frame = |feed: &mut Feed, seconds, text: &str| { + for effect in feed.on_frame(at(seconds), text) { + if let Effect::Report(loads) = effect { + reports.push(loads); + } + } + }; + on_frame(&mut feed, 10, &change("0xbusy")); + on_frame(&mut feed, 20, &trade); + on_frame(&mut feed, 40, &tick_size); + on_frame(&mut feed, 50, &change("0xlate")); + on_frame(&mut feed, 55, &change("0xunknown")); + // The next minute: the first report, then nothing more until the one + // after, which starts again from zero. + on_frame(&mut feed, 60, &shared); + on_frame(&mut feed, 120, &change("0xunknown")); + + let bytes = |text: &String| text.len() as u64; + let half = bytes(&shared) / 2; + assert_eq!( + reports, + [ + Loads::from([ + ( + "0xbusy".to_string(), + (bytes(&change("0xbusy")) + bytes(&trade)) / 60 + ), + ("0xquiet".to_string(), bytes(&tick_size) / 30), + ]), + Loads::from([ + ("0xbusy".to_string(), half / 60), + ("0xquiet".to_string(), half / 60), + ("0xlate".to_string(), bytes(&change("0xlate")) / 75), + ]), + ] + ); + } + + #[test] + fn the_recording_reports_every_market_once_measured_long_enough() { + let frames = fixture_frames(); + let markets = fixture_markets(); + let mut feed = feed_of(&markets); + let reports: Vec = replay(&mut feed, &frames) + .into_iter() + .filter_map(|effect| match effect { + Effect::Report(loads) => Some(loads), + _ => None, + }) + .collect(); + // The recording crosses two minutes, the first too soon after its + // start for any market to be measured. + let start = frames[0].0; + assert!(next_minute(start) - start < MIN_MEASURED); + assert_eq!(reports.len(), 2); + assert!(reports[0].is_empty()); + let loads = &reports[1]; + assert_eq!(loads.len(), markets.len()); + assert!( + markets + .iter() + .all(|market| loads.contains_key(&market.condition_id)) + ); + assert!(loads.values().sum::() > 0); + } + #[test] fn the_output_reconciles() { // The book and writer threads as main runs them, fed the recording. let (frames_tx, frames_rx) = crossbeam::channel::unbounded(); let (outputs_tx, outputs_rx) = crossbeam::channel::unbounded(); - let (resyncs_tx, _resyncs_rx) = crossbeam::channel::unbounded(); + let (connection_tx, _connection_rx) = crossbeam::channel::unbounded(); + let (_commands_tx, commands_rx) = crossbeam::channel::unbounded(); + let (reports_tx, _reports_rx) = crossbeam::channel::unbounded(); for (received_at, text) in fixture_frames() { frames_tx.send((received_at, text.into())).unwrap(); } drop(frames_tx); - let feed = Feed::new(&fixture_markets()); - let book = std::thread::spawn(move || run(feed, frames_rx, outputs_tx, resyncs_tx)); + let feed = feed_of(&fixture_markets()); + let book = std::thread::spawn(move || { + run( + feed, + frames_rx, + commands_rx, + outputs_tx, + connection_tx, + reports_tx, + ) + }); let path = std::env::temp_dir().join(format!("reconcile-{}.sbe", std::process::id())); let written = crate::write(path.to_str().unwrap(), outputs_rx); book.join().unwrap(); diff --git a/src/polymarket/gamma.rs b/src/polymarket/gamma.rs new file mode 100644 index 0000000..122603a --- /dev/null +++ b/src/polymarket/gamma.rs @@ -0,0 +1,341 @@ +//! Gamma, Polymarket's market listing: which markets are live. Two fetcher +//! threads page it on the orchestrator's request and stream what they read; +//! they hold no state and no timer (roadmap/03). + +use std::thread; +use std::time::Duration; + +use chrono::{DateTime, Utc}; +use crossbeam::channel::{Receiver, Sender}; +use serde::Deserialize; + +use super::Market; + +const KEYSET_URL: &str = "https://gamma-api.polymarket.com/markets/keyset"; +/// Gamma's page size, whatever a larger `limit` asks for. +const PAGE_SIZE: &str = "100"; +/// For every request, so SIGINT never waits on Gamma longer than this. +const TIMEOUT: Duration = Duration::from_secs(10); +/// A scan's tries at one page, and the wait between them. +const SCAN_TRIES: u32 = 5; +const SCAN_RETRY_AFTER: Duration = Duration::from_secs(5); +/// What REST's `/book` (and so the hash) says when Gamma's `orderMinSize` is +/// null (checked on 4 such tokens, 2026-11-04). +const MISSING_MIN_ORDER_SIZE: &str = "0"; + +pub enum Request { + /// Every live market. + Rescan, + /// The live markets created and the markets closed since then. + Recent { since: DateTime }, +} + +#[derive(Debug)] +pub enum Finding { + /// A rescan began: once it is done, a market known now and not seen in it + /// is gone. + ScanStarted, + ScanPage(Vec), + ScanDone, + ScanFailed, + /// Live markets created since the request's time. + Created(Vec), + /// The `conditionId`s of markets closed since the request's time. + Closed(Vec), + RecentDone, + RecentFailed, +} + +/// A fetcher thread: answers each request until the orchestrator is gone. +pub fn run(requests: Receiver, findings: Sender) { + let agent = ureq::Agent::new_with_config( + ureq::Agent::config_builder() + .timeout_global(Some(TIMEOUT)) + .build(), + ); + for request in requests { + let sent = match request { + Request::Rescan => rescan(&agent, &findings), + Request::Recent { since } => recent(&agent, since, &findings), + }; + if sent.is_err() { + return; + } + } +} + +/// `Err` once the orchestrator is gone. +type Sent = Result<(), crossbeam::channel::SendError>; + +fn rescan(agent: &ureq::Agent, findings: &Sender) -> Sent { + findings.send(Finding::ScanStarted)?; + let query = [("active", "true"), ("closed", "false")]; + let mut cursor = None; + loop { + let page = match with_retries(|| page(agent, &query, cursor.as_deref())) { + Ok(page) => page, + Err(error) => { + log!("polymarket: rescan failed: {error}"); + return findings.send(Finding::ScanFailed); + } + }; + let last = page.markets.is_empty(); + findings.send(Finding::ScanPage(live(page.markets)))?; + cursor = page.next_cursor; + if last || cursor.is_none() { + return findings.send(Finding::ScanDone); + } + } +} + +fn with_retries(mut fetch: impl FnMut() -> anyhow::Result) -> anyhow::Result { + let mut tries = 1; + loop { + match fetch() { + Err(_) if tries < SCAN_TRIES => { + tries += 1; + thread::sleep(SCAN_RETRY_AFTER); + } + result => return result, + } + } +} + +fn recent(agent: &ureq::Agent, since: DateTime, findings: &Sender) -> Sent { + let created = [ + ("active", "true"), + ("closed", "false"), + ("order", "createdAt"), + ("ascending", "false"), + ]; + let closed = [ + ("closed", "true"), + ("order", "closedTime"), + ("ascending", "false"), + ]; + let result = since_then(agent, &created, since, created_at).and_then(|markets| { + findings.send(Finding::Created(live(markets)))?; + since_then(agent, &closed, since, closed_time) + }); + match result { + Ok(markets) => { + let ids = markets.into_iter().map(|market| market.condition_id); + findings.send(Finding::Closed(ids.collect()))?; + findings.send(Finding::RecentDone) + } + Err(error) => { + log!("polymarket: recent check failed: {error}"); + findings.send(Finding::RecentFailed) + } + } +} + +fn created_at(market: &GammaMarket) -> &Option { + &market.created_at +} + +fn closed_time(market: &GammaMarket) -> &Option { + &market.closed_time +} + +/// The markets of `query`'s pages, sorted newest first by `time`, down to +/// `since`. +fn since_then( + agent: &ureq::Agent, + query: &[(&str, &str)], + since: DateTime, + time: impl Fn(&GammaMarket) -> &Option, +) -> anyhow::Result> { + let mut markets = Vec::new(); + let mut cursor = None; + loop { + let page = page(agent, query, cursor.as_deref())?; + let (newer, reached) = newer_than(page.markets, since, &time); + markets.extend(newer); + cursor = page.next_cursor; + if reached || cursor.is_none() { + return Ok(markets); + } + } +} + +/// The markets of a newest-first page at or after `since`, and whether the +/// page reached past it. A market with no readable time is kept. +fn newer_than( + markets: Vec, + since: DateTime, + time: impl Fn(&GammaMarket) -> &Option, +) -> (Vec, bool) { + let reached = markets.is_empty() + || markets + .iter() + .any(|market| parse_time(time(market)).is_some_and(|time| time < since)); + let newer = markets + .into_iter() + .filter(|market| parse_time(time(market)).is_none_or(|time| time >= since)) + .collect(); + (newer, reached) +} + +/// `createdAt` is RFC 3339 (`2026-11-04T16:57:29.989119Z`), `closedTime` not +/// (`2026-11-04 16:57:50.384485+00`). +fn parse_time(text: &Option) -> Option> { + let text = text.as_deref()?; + DateTime::parse_from_rfc3339(text) + .or_else(|_| DateTime::parse_from_str(text, "%Y-%m-%d %H:%M:%S%.f%#z")) + .ok() + .map(|time| time.to_utc()) +} + +#[derive(Deserialize)] +struct Page { + markets: Vec, + next_cursor: Option, +} + +/// Every field can be missing or null on some market (2026-11-04: 230 live +/// markets had no `negRisk`, 31 no `orderMinSize`). +#[derive(Deserialize)] +#[serde(rename_all = "camelCase")] +struct GammaMarket { + #[serde(default)] + condition_id: String, + /// A JSON array inside a string. + clob_token_ids: Option, + order_min_size: Option, + neg_risk: Option, + active: Option, + closed: Option, + enable_order_book: Option, + accepting_orders: Option, + created_at: Option, + closed_time: Option, +} + +fn page(agent: &ureq::Agent, query: &[(&str, &str)], cursor: Option<&str>) -> anyhow::Result { + let mut request = agent + .get(KEYSET_URL) + .query("limit", PAGE_SIZE) + .query_pairs(query.iter().copied()); + if let Some(cursor) = cursor { + request = request.query("after_cursor", cursor); + } + let body = request.call()?.body_mut().read_to_string()?; + Ok(serde_json::from_str(&body)?) +} + +/// The live markets among `markets`: active, not closed, with an order book +/// that accepts orders. The end date doesn't count (docs/notes.md). +fn live(markets: Vec) -> Vec { + markets + .into_iter() + .filter(|market| { + market.active == Some(true) + && market.closed != Some(true) + && market.enable_order_book == Some(true) + && market.accepting_orders == Some(true) + && !market.condition_id.is_empty() + }) + .filter_map(|market| { + let tokens: Vec = + serde_json::from_str(market.clob_token_ids.as_deref()?).ok()?; + if tokens.is_empty() { + return None; + } + Some(Market { + condition_id: market.condition_id, + tokens, + min_order_size: market + .order_min_size + .map_or(MISSING_MIN_ORDER_SIZE.to_string(), |size| size.to_string()), + neg_risk: market.neg_risk.unwrap_or(false), + }) + }) + .collect() +} + +/// The live markets of a saved `/markets` response (a plain array). +#[cfg(test)] +pub(crate) fn parse_markets(body: &str) -> Vec { + live(serde_json::from_str(body).unwrap()) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn fixture(name: &str) -> Page { + let path = format!("{}/tests/fixtures/{name}", env!("CARGO_MANIFEST_DIR")); + serde_json::from_str(&std::fs::read_to_string(path).unwrap()).unwrap() + } + + fn time(text: &str) -> DateTime { + parse_time(&Some(text.to_string())).unwrap() + } + + #[test] + fn parses_gamma_markets() { + let markets = parse_markets(include_str!("../../tests/fixtures/gamma-markets.json")); + assert_eq!(markets.len(), 20); + assert!(markets.iter().all(|market| market.tokens.len() == 2)); + assert!(markets[0].condition_id.starts_with("0x")); + assert_eq!(markets[0].min_order_size, "5"); + } + + #[test] + fn keyset_pages_parse_and_closed_markets_are_not_live() { + let created = fixture("gamma-created.json"); + assert_eq!(created.markets.len(), 10); + assert!(created.next_cursor.is_some()); + assert_eq!(live(created.markets).len(), 10); + let closed = fixture("gamma-closed.json"); + assert_eq!(closed.markets.len(), 10); + assert!(live(closed.markets).is_empty()); + } + + #[test] + fn both_time_formats_parse() { + assert_eq!( + time("2026-11-04T16:57:29.989119Z"), + time("2026-11-04 16:57:29.989119+00") + ); + } + + #[test] + fn a_page_is_kept_down_to_since_and_reports_reaching_it() { + // The fixture's createdAt: 16:57:29.989, 16:54:27.166, .084, .011, then + // six from 16:54:26.943 down to 16:54:26.595. + let page = fixture("gamma-created.json"); + let (newer, reached) = newer_than(page.markets, time("2026-11-04T16:54:27Z"), created_at); + assert_eq!(newer.len(), 4); + assert!(reached); + + let page = fixture("gamma-created.json"); + let (newer, reached) = newer_than(page.markets, time("2026-11-04T16:00:00Z"), created_at); + assert_eq!(newer.len(), 10); + assert!(!reached); + + let (newer, reached) = newer_than(Vec::new(), time("2026-11-04T16:00:00Z"), created_at); + assert!(newer.is_empty() && reached); + } + + #[test] + fn a_live_market_takes_what_the_hash_uses_and_needs_token_ids() { + let markets = live( + serde_json::from_str( + r#"[{"conditionId":"0xa","clobTokenIds":"[\"1\",\"2\"]","negRisk":null, + "active":true,"closed":false,"enableOrderBook":true,"acceptingOrders":true}, + {"conditionId":"0xb","clobTokenIds":null, + "active":true,"closed":false,"enableOrderBook":true,"acceptingOrders":true}, + {"conditionId":"0xd","clobTokenIds":"[]", + "active":true,"closed":false,"enableOrderBook":true,"acceptingOrders":true}, + {"conditionId":"0xc","clobTokenIds":"[\"3\",\"4\"]", + "active":true,"closed":false,"enableOrderBook":true,"acceptingOrders":null}]"#, + ) + .unwrap(), + ); + assert_eq!(markets.len(), 1); + assert_eq!(markets[0].min_order_size, "0"); + assert!(!markets[0].neg_risk); + } +} diff --git a/src/polymarket/mod.rs b/src/polymarket/mod.rs index 11f92a5..68dc589 100644 --- a/src/polymarket/mod.rs +++ b/src/polymarket/mod.rs @@ -1,17 +1,17 @@ -//! Polymarket: which markets to track (Gamma), the market-channel socket, and -//! the feed that turns its frames into books and output. +//! Polymarket: which markets are live (Gamma), the orchestrator that spreads +//! them over lanes, and each lane's connection and feed (roadmap/03). pub mod feed; +pub mod gamma; +pub mod orchestrator; pub mod socket; -use serde::Deserialize; - -const GAMMA_MARKETS_URL: &str = "https://gamma-api.polymarket.com/markets"; - -/// A market's symbols and the two inputs of their book hash that the +/// A live market: its symbols and the two inputs of their book hash that the /// websocket doesn't carry (docs/notes.md). #[derive(Debug, Clone)] pub struct Market { + /// Gamma's `conditionId`, the websocket's `market`. + pub condition_id: String, /// Its token ids, one per outcome: each is a symbol. pub tokens: Vec, /// Exactly as the hash expects it, e.g. `"5"`. @@ -19,58 +19,7 @@ pub struct Market { pub neg_risk: bool, } -#[derive(Deserialize)] -#[serde(rename_all = "camelCase")] -struct GammaMarket { - /// A JSON array inside a string. - clob_token_ids: String, - order_min_size: serde_json::Number, - neg_risk: bool, - enable_order_book: bool, -} - -/// The `limit` live markets with the most 24 h volume, keeping those with an -/// order book. Fixed for the run: discovery is M3. -pub fn top_markets(limit: u32) -> anyhow::Result> { - let body = ureq::get(GAMMA_MARKETS_URL) - .query("limit", limit.to_string()) - .query("active", "true") - .query("closed", "false") - .query("order", "volume24hr") - .query("ascending", "false") - .call()? - .body_mut() - .read_to_string()?; - parse_markets(&body) -} - -fn parse_markets(body: &str) -> anyhow::Result> { - let markets: Vec = serde_json::from_str(body)?; - markets - .into_iter() - .filter(|market| market.enable_order_book) - .map(|market| { - Ok(Market { - tokens: serde_json::from_str(&market.clob_token_ids)?, - min_order_size: market.order_min_size.to_string(), - neg_risk: market.neg_risk, - }) - }) - .collect() -} - #[cfg(test)] pub(crate) fn fixture_markets() -> Vec { - parse_markets(include_str!("../../tests/fixtures/gamma-markets.json")).unwrap() -} - -#[cfg(test)] -mod tests { - #[test] - fn parses_gamma_markets() { - let markets = super::fixture_markets(); - assert_eq!(markets.len(), 20); - assert!(markets.iter().all(|market| market.tokens.len() == 2)); - assert_eq!(markets[0].min_order_size, "5"); - } + gamma::parse_markets(include_str!("../../tests/fixtures/gamma-markets.json")) } diff --git a/src/polymarket/orchestrator.rs b/src/polymarket/orchestrator.rs new file mode 100644 index 0000000..9cf2e14 --- /dev/null +++ b/src/polymarket/orchestrator.rs @@ -0,0 +1,678 @@ +//! The orchestrator: the one owner of what is tracked. It keeps the tracked +//! markets equal to Gamma's live ones, spreads them over lanes and owns every +//! timer; the two fetchers only fetch (roadmap/03, ADR 0002). + +use std::cmp::Reverse; +use std::collections::{HashMap, HashSet}; +use std::sync::Arc; +use std::sync::atomic::{AtomicBool, Ordering}; +use std::thread::{self, JoinHandle}; +use std::time::{Duration, Instant}; + +use chrono::{DateTime, TimeDelta, Utc}; +use crossbeam::channel::{Sender, select, unbounded}; +use engineering_challenge::book::Output; + +use super::feed::{self, Feed, LaneCommand, LoadReport, Loads}; +use super::gamma::{self, Finding, Request}; +use super::{Market, socket}; + +/// The most tokens a lane holds: a reconnect replays every one's book at once. +const LANE_TOKENS: usize = 5_000; +/// A lane's byte budget, under the lowest rate at which a connection was seen +/// to get a 1013 close (0.46 MB/s; docs/notes.md). Closes came at up to +/// 5.1 MB/s too: no rate is safe, only less likely to be closed. +const LANE_BYTES_PER_SECOND: u64 = 300_000; +/// Moves are limited so balancing doesn't churn. Rebalancing runs with each +/// minute check, every [`RECENT_EVERY`]. +const MOVES_PER_MINUTE: usize = 50; +const MOVED_STAYS: Duration = Duration::from_secs(10 * 60); +const RECENT_EVERY: Duration = Duration::from_secs(60); +const RESCAN_EVERY: Duration = Duration::from_secs(60 * 60); +/// Gamma's indexing delay: each recent check reaches this far before the +/// last successful one. New markets were listed up to 5 minutes after their +/// `createdAt` (docs/notes.md); known markets are skipped, so the margin costs +/// only the pages it reads again. +const RECENT_MARGIN: TimeDelta = TimeDelta::minutes(15); +/// So SIGINT is seen while no timer is due. +const WAKE_EVERY: Duration = Duration::from_secs(1); + +/// Runs until `stop` is set, then waits for every lane. Returns how many +/// threads failed: the lane threads that panicked, and one if a fetcher died. +pub fn run(outputs: Sender, stop: &Arc) -> usize { + let (findings_tx, findings) = unbounded(); + let (rescan, rescan_requests) = unbounded(); + let (recent, recent_requests) = unbounded(); + // Two, so a full scan never holds up the minute check. Not joined: they + // hold no output, and each ends at its next send. + let fetchers: Vec> = [rescan_requests, recent_requests] + .into_iter() + .map(|requests| { + let findings = findings_tx.clone(); + thread::spawn(move || gamma::run(requests, findings)) + }) + .collect(); + drop(findings_tx); + let (reports_tx, reports) = unbounded::(); + + let mut tracker = Tracker::default(); + let mut lanes: Vec = Vec::new(); + let (mut added, mut removed, mut moved) = (0, 0, 0); + // Startup is the first rescan; the minute checks cover it from its start. + let mut recent_since = Utc::now() - RECENT_MARGIN; + let mut recent_asked_at: Option> = None; + let mut scanning = false; + let mut scanned = false; + let mut fetcher_died = false; + let mut next_rescan = Instant::now(); + let mut next_recent = Instant::now() + RECENT_EVERY; + // Opens the lane on its first command. + let send = |lanes: &mut Vec, lane: usize, command: LaneCommand| { + if lane == lanes.len() { + lanes.push(open_lane(lane, &outputs, &reports_tx, stop)); + } + // A lane whose book thread died keeps its markets unserved: lane + // health is M5's. + _ = lanes[lane].commands.send(command); + }; + + while !stop.load(Ordering::Relaxed) { + // Only a panic ends a fetcher while this runs, even one in the middle + // of a request, whose answer would never come: nothing would keep the + // set current. + if fetchers.iter().any(JoinHandle::is_finished) { + log!("polymarket: orchestrator: a fetcher died, stopping"); + stop.store(true, Ordering::Relaxed); + fetcher_died = true; + break; + } + let now = Instant::now(); + if now >= next_rescan { + if !scanning { + _ = rescan.send(Request::Rescan); + scanning = true; + } + next_rescan = now + RESCAN_EVERY; + } + if now >= next_recent { + if recent_asked_at.is_none() { + _ = recent.send(Request::Recent { + since: recent_since, + }); + recent_asked_at = Some(Utc::now()); + } + next_recent = now + RECENT_EVERY; + // On the loads the lanes reported at the wall-clock minute. + let moves = tracker.rebalance(now); + moved += moves.len(); + for (lane, command) in moves.into_iter().flat_map(Move::commands) { + send(&mut lanes, lane, command); + } + let busiest = tracker.lanes.iter().map(|lane| lane.load).max(); + log!( + "polymarket: orchestrator: {} markets in {} lanes; {added} added, {removed} \ + removed, {moved} moved this minute; busiest lane at {} bytes/s", + tracker.markets.len(), + lanes.len(), + busiest.unwrap_or(0) + ); + (added, removed, moved) = (0, 0, 0); + } + + let timeout = next_rescan + .min(next_recent) + .saturating_duration_since(Instant::now()) + .min(WAKE_EVERY); + let finding = select! { + recv(findings) -> finding => match finding { + Ok(finding) => finding, + // Both fetchers are gone: the check above stops the run. + Err(_) => continue, + }, + recv(reports) -> report => { + if let Ok(report) = report { + tracker.report(report.lane, &report.loads); + } + continue; + }, + default(timeout) => continue, + }; + let commands = match finding { + Finding::ScanStarted => { + tracker.scan_started(); + Vec::new() + } + Finding::ScanPage(markets) | Finding::Created(markets) => markets + .into_iter() + .filter_map(|market| tracker.add(market)) + .collect(), + Finding::ScanDone => { + scanning = false; + scanned = true; + tracker.scan_done() + } + Finding::ScanFailed => { + scanning = false; + tracker.scan_failed(); + // Nothing can be tracked before a first scan: retry at once. + if !scanned { + next_rescan = Instant::now(); + } + Vec::new() + } + Finding::Closed(ids) => ids.iter().filter_map(|id| tracker.remove(id)).collect(), + Finding::RecentDone => { + if let Some(asked_at) = recent_asked_at.take() { + recent_since = asked_at - RECENT_MARGIN; + } + Vec::new() + } + Finding::RecentFailed => { + recent_asked_at = None; + Vec::new() + } + }; + for (lane, command) in commands { + match command { + LaneCommand::Add(_) => added += 1, + LaneCommand::Remove(_) => removed += 1, + // Only rebalancing moves, counted there. + LaneCommand::MoveIn(_) => {} + } + send(&mut lanes, lane, command); + } + } + + drop(outputs); + let panicked = lanes + .into_iter() + .flat_map(|lane| lane.threads) + .map(JoinHandle::join) + .filter(Result::is_err) + .count(); + panicked + usize::from(fetcher_died) +} + +struct Lane { + commands: Sender, + threads: [JoinHandle<()>; 2], +} + +/// Lane `number`'s connection and book thread, joined by [`run`]. +fn open_lane( + number: usize, + outputs: &Sender, + reports: &Sender, + stop: &Arc, +) -> Lane { + let (commands, commands_rx) = unbounded(); + let (frames, frames_rx) = unbounded(); + let (connection, connection_rx) = unbounded(); + let stop = Arc::clone(stop); + let outputs = outputs.clone(); + let reports = reports.clone(); + let connection_thread = thread::spawn(move || { + socket::run( + socket::MARKET_CHANNEL_URL, + number, + &frames, + &connection_rx, + &stop, + ) + }); + let book_thread = thread::spawn(move || { + feed::run( + Feed::new(number), + frames_rx, + commands_rx, + outputs, + connection, + reports, + ) + }); + Lane { + commands, + threads: [connection_thread, book_thread], + } +} + +/// A market leaving lane `from` for lane `to`. +#[derive(Debug)] +struct Move { + market: Market, + from: usize, + to: usize, +} + +impl Move { + /// A Remove for the old lane, then a MoveIn for the new one, in the order + /// they are sent. + fn commands(self) -> [(usize, LaneCommand); 2] { + [ + (self.from, LaneCommand::Remove(self.market.clone())), + (self.to, LaneCommand::MoveIn(self.market)), + ] + } +} + +/// A tracked market, where it is and what it weighs. +struct Tracked { + market: Market, + lane: usize, + /// Bytes per second over its lane's last report; 0 until one comes. + load: u64, + moved_at: Option, +} + +/// A lane's tokens and load, against its two limits. +#[derive(Debug, Default, Clone, Copy, PartialEq, Eq)] +struct LaneLoad { + tokens: usize, + /// The sum of its markets' loads, bytes per second. + load: u64, +} + +/// Which lane each market is in and what it weighs, and which markets a +/// rescan in progress hasn't seen yet. +#[derive(Default)] +struct Tracker { + /// By `conditionId`. + markets: HashMap, + /// By lane number. + lanes: Vec, + /// While a rescan runs: the markets known at its start, not seen since. + unseen: Option>, +} + +impl Tracker { + /// A market it doesn't know goes to the least-loaded lane with token room, + /// a new lane if none has: the lane and its command. + fn add(&mut self, market: Market) -> Option<(usize, LaneCommand)> { + if let Some(unseen) = &mut self.unseen { + unseen.remove(&market.condition_id); + } + if self.markets.contains_key(&market.condition_id) { + return None; + } + let lane = self.lane_for(market.tokens.len(), None, None); + self.lanes[lane].tokens += market.tokens.len(); + let tracked = Tracked { + market: market.clone(), + lane, + load: 0, + moved_at: None, + }; + self.markets.insert(market.condition_id.clone(), tracked); + Some((lane, LaneCommand::Add(market))) + } + + /// The least-loaded lane (then the one with fewest tokens), other than + /// `except`, with room for `tokens` more and, if given, `load` more + /// bytes/s; a new lane if none has. + fn lane_for(&mut self, tokens: usize, load: Option, except: Option) -> usize { + self.lanes + .iter() + .enumerate() + .filter(|&(lane, lane_load)| { + Some(lane) != except + && lane_load.tokens + tokens <= LANE_TOKENS + && load.is_none_or(|load| lane_load.load + load <= LANE_BYTES_PER_SECOND) + }) + .min_by_key(|&(_, lane_load)| (lane_load.load, lane_load.tokens)) + .map(|(lane, _)| lane) + .unwrap_or_else(|| { + self.lanes.push(LaneLoad::default()); + self.lanes.len() - 1 + }) + } + + fn remove(&mut self, condition_id: &str) -> Option<(usize, LaneCommand)> { + let tracked = self.markets.remove(condition_id)?; + let lane = &mut self.lanes[tracked.lane]; + lane.tokens -= tracked.market.tokens.len(); + lane.load -= tracked.load; + Some((tracked.lane, LaneCommand::Remove(tracked.market))) + } + + /// A lane's loads per market. A market reported by a lane it has since + /// moved from keeps the load it has. + fn report(&mut self, lane: usize, loads: &Loads) { + for (condition_id, &load) in loads { + let Some(tracked) = self.markets.get_mut(condition_id) else { + continue; + }; + if tracked.lane != lane { + continue; + } + let lane_load = &mut self.lanes[lane]; + lane_load.load = lane_load.load - tracked.load + load; + tracked.load = load; + } + } + + /// Moves the busiest markets off every lane over its byte budget, busiest + /// lane first, to the least-loaded lanes with room for them, until it is + /// under: at most [`MOVES_PER_MINUTE`], and none moved in the last + /// [`MOVED_STAYS`]. + fn rebalance(&mut self, now: Instant) -> Vec { + let mut moves = Vec::new(); + let mut over: Vec = (0..self.lanes.len()) + .filter(|&lane| self.lanes[lane].load > LANE_BYTES_PER_SECOND) + .collect(); + over.sort_by_key(|&lane| Reverse(self.lanes[lane].load)); + for from in over { + // ponytail: a scan of every market per lane over budget, fine at a + // few hundred thousand once a minute; per-lane market sets if not. + let mut busiest: Vec<(String, u64)> = self + .markets + .iter() + .filter(|(_, tracked)| { + tracked.lane == from + && tracked + .moved_at + .is_none_or(|at| now.duration_since(at) >= MOVED_STAYS) + }) + .map(|(id, tracked)| (id.clone(), tracked.load)) + .collect(); + busiest.sort_by_key(|&(_, load)| Reverse(load)); + for (id, load) in busiest { + let size = self.markets[&id].market.tokens.len(); + // Alone on its lane, a market would only take its load along; + // one with no load would lighten nothing. + if self.lanes[from].load <= LANE_BYTES_PER_SECOND + || moves.len() == MOVES_PER_MINUTE + || self.lanes[from].tokens == size + || load == 0 + { + break; + } + let to = self.lane_for(size, Some(load), Some(from)); + self.lanes[from].tokens -= size; + self.lanes[from].load -= load; + self.lanes[to].tokens += size; + self.lanes[to].load += load; + let tracked = self.markets.get_mut(&id).expect("listed above"); + tracked.lane = to; + tracked.moved_at = Some(now); + moves.push(Move { + market: tracked.market.clone(), + from, + to, + }); + } + } + moves + } + + fn scan_started(&mut self) { + self.unseen = Some(self.markets.keys().cloned().collect()); + } + + /// Removes the markets known at the scan's start and not seen in it: + /// those added since were never at risk. + fn scan_done(&mut self) -> Vec<(usize, LaneCommand)> { + let unseen = self.unseen.take().unwrap_or_default(); + unseen.iter().filter_map(|id| self.remove(id)).collect() + } + + /// Half a listing says nothing of the other half: nothing is removed. + fn scan_failed(&mut self) { + self.unseen = None; + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn market(id: usize) -> Market { + Market { + condition_id: format!("0x{id}"), + tokens: vec![format!("{id}a"), format!("{id}b")], + min_order_size: "5".to_string(), + neg_risk: false, + } + } + + fn lanes(commands: &[(usize, LaneCommand)]) -> Vec { + commands.iter().map(|(lane, _)| *lane).collect() + } + + fn tokens(tracker: &Tracker) -> Vec { + tracker.lanes.iter().map(|lane| lane.tokens).collect() + } + + /// Reports `bytes_per_second` for each market, as its lane would. + fn report(tracker: &mut Tracker, loads: &[(usize, u64)]) { + for &(id, bytes_per_second) in loads { + let lane = tracker.markets[&format!("0x{id}")].lane; + let loads = Loads::from([(format!("0x{id}"), bytes_per_second)]); + tracker.report(lane, &loads); + } + } + + #[test] + fn a_market_joins_the_least_loaded_lane_with_room_and_lanes_open_when_full() { + let mut tracker = Tracker::default(); + let per_lane = LANE_TOKENS / 2; + let first: Vec<_> = (0..per_lane + 1) + .filter_map(|id| tracker.add(market(id))) + .collect(); + // Nothing measured yet: the first lane fills, then a second opens. + assert!(lanes(&first[..per_lane]).iter().all(|&lane| lane == 0)); + assert_eq!(first[per_lane].0, 1); + // A hole in lane 0 draws the next market once lane 1 weighs more. + tracker.remove("0x0").unwrap(); + report(&mut tracker, &[(per_lane, 1_000)]); + assert_eq!(tracker.add(market(per_lane + 1)).unwrap().0, 0); + // Both tokens of a market share its lane. + assert_eq!(tracker.markets["0x1"].lane, 0); + assert_eq!(tokens(&tracker), [LANE_TOKENS, 2]); + } + + #[test] + fn between_lanes_of_equal_load_a_market_joins_the_one_with_fewest_tokens() { + let mut tracker = Tracker { + lanes: vec![ + LaneLoad { tokens: 4, load: 0 }, + LaneLoad { tokens: 2, load: 0 }, + ], + ..Tracker::default() + }; + assert_eq!(lanes(&[tracker.add(market(0)).unwrap()]), [1]); + assert_eq!(tokens(&tracker), [4, 4]); + } + + #[test] + fn a_known_market_is_skipped() { + let mut tracker = Tracker::default(); + assert!(tracker.add(market(1)).is_some()); + assert!(tracker.add(market(1)).is_none()); + assert_eq!(tokens(&tracker), [2]); + } + + #[test] + fn a_report_sets_its_lanes_markets_and_no_others() { + let mut tracker = Tracker::default(); + tracker.add(market(1)); + tracker.add(market(2)); + report(&mut tracker, &[(1, 300), (2, 100)]); + assert_eq!(tracker.lanes[0].load, 400); + // A lane the market has left reports it late: ignored. + tracker.lanes.push(LaneLoad::default()); + let stale = Loads::from([("0x1".to_string(), 900)]); + tracker.report(1, &stale); + assert_eq!(tracker.markets["0x1"].load, 300); + assert_eq!(tracker.lanes[1].load, 0); + // A market gone takes its load along. + tracker.remove("0x1"); + assert_eq!(tracker.lanes[0].load, 100); + } + + #[test] + fn a_lane_over_budget_moves_its_busiest_markets_until_under() { + let mut tracker = Tracker::default(); + for id in 0..10 { + tracker.add(market(id)); + } + tracker.lanes.push(LaneLoad::default()); + let budget = LANE_BYTES_PER_SECOND; + // Lane 0 at 1.5 budgets: its busiest market (half a budget) moving + // is enough. + let mut loads = vec![(0, budget / 2)]; + loads.extend((1..10).map(|id| (id, budget / 9))); + report(&mut tracker, &loads); + let now = Instant::now(); + let moves = tracker.rebalance(now); + + assert!(matches!( + moves.as_slice(), + [Move { market, from: 0, to: 1 }] if market.condition_id == "0x0" + )); + assert_eq!(tracker.markets["0x0"].lane, 1); + let [ + (0, LaneCommand::Remove(removed)), + (1, LaneCommand::MoveIn(added)), + ] = moves.into_iter().next().unwrap().commands() + else { + panic!("a move is a Remove for the old lane, then a MoveIn for the new"); + }; + assert_eq!( + (removed.condition_id, added.condition_id), + ("0x0".into(), "0x0".into()) + ); + assert!(tracker.lanes[0].load <= budget); + assert_eq!(tokens(&tracker), [18, 2]); + assert_eq!(tracker.lanes[1].load, budget / 2); + // Under budget now: nothing more moves. + assert!(tracker.rebalance(now).is_empty()); + } + + #[test] + fn the_busiest_lane_over_budget_is_rebalanced_first() { + let mut tracker = Tracker::default(); + tracker.add(market(0)); + tracker.add(market(1)); + tracker.lanes.push(LaneLoad::default()); + // The empty lane is the least loaded: both join it. + tracker.add(market(2)); + tracker.add(market(3)); + assert_eq!(tokens(&tracker), [4, 4]); + let budget = LANE_BYTES_PER_SECOND; + report(&mut tracker, &[(0, budget), (1, budget / 10)]); + report(&mut tracker, &[(2, budget / 2), (3, budget)]); + let moves = tracker.rebalance(Instant::now()); + assert!(matches!( + moves.first(), + Some(Move { market, from: 1, .. }) if market.condition_id == "0x3" + )); + } + + /// The markets that may move have no load: moving them would empty the + /// lane and lighten nothing. + #[test] + fn markets_with_no_load_are_not_moved() { + let mut tracker = Tracker::default(); + for id in 0..4 { + tracker.add(market(id)); + } + report(&mut tracker, &[(0, 2 * LANE_BYTES_PER_SECOND)]); + let now = Instant::now(); + tracker.markets.get_mut("0x0").unwrap().moved_at = Some(now); + assert!(tracker.rebalance(now).is_empty()); + } + + #[test] + fn moves_are_limited_and_a_moved_market_stays() { + let mut tracker = Tracker::default(); + let count = 2 * MOVES_PER_MINUTE; + for id in 0..count { + tracker.add(market(id)); + } + // Every market alone near a budget: each needs a lane of its own. + let loads: Vec<_> = (0..count) + .map(|id| (id, LANE_BYTES_PER_SECOND * 9 / 10)) + .collect(); + report(&mut tracker, &loads); + let now = Instant::now(); + let first = tracker.rebalance(now); + assert_eq!(first.len(), MOVES_PER_MINUTE); + // The rest move a minute later, but the last, alone and under budget. + let second = tracker.rebalance(now + Duration::from_secs(60)); + assert_eq!(second.len(), MOVES_PER_MINUTE - 1); + assert!(second.iter().all(|moved| moved.from == 0)); + // Ten minutes on, lane 0 holds one market: it is left alone. + assert!(tracker.rebalance(now + MOVED_STAYS).is_empty()); + assert_eq!(tracker.lanes[0].tokens, 2); + } + + /// Over budget on its own, a market would only carry its load from lane + /// to lane every ten minutes. + #[test] + fn a_market_alone_on_its_lane_is_not_moved() { + let mut tracker = Tracker::default(); + tracker.add(market(0)); + report(&mut tracker, &[(0, 2 * LANE_BYTES_PER_SECOND)]); + assert!(tracker.rebalance(Instant::now()).is_empty()); + assert_eq!(tracker.lanes.len(), 1); + } + + #[test] + fn a_market_moved_less_than_ten_minutes_ago_stays() { + let mut tracker = Tracker::default(); + let budget = LANE_BYTES_PER_SECOND; + tracker.add(market(0)); + tracker.add(market(1)); + report(&mut tracker, &[(0, budget * 7 / 10), (1, budget / 2)]); + tracker.lanes.push(LaneLoad::default()); + let now = Instant::now(); + tracker.add(market(2)); + assert_eq!(tracker.markets["0x2"].lane, 1); + report(&mut tracker, &[(2, budget / 10)]); + // Market 0 moves to lane 1, which then turns busy. + assert!(matches!( + tracker.rebalance(now).as_slice(), + [Move { from: 0, to: 1, .. }] + )); + report(&mut tracker, &[(0, budget * 7 / 10), (2, budget * 6 / 10)]); + let moves = tracker.rebalance(now + Duration::from_secs(60)); + assert!(matches!( + moves.as_slice(), + [Move { market, from: 1, to: 2 }] if market.condition_id == "0x2" + )); + } + + #[test] + fn a_completed_rescan_removes_only_the_known_markets_it_did_not_see() { + let mut tracker = Tracker::default(); + for id in 0..3 { + tracker.add(market(id)); + } + tracker.scan_started(); + tracker.add(market(0)); + // Added by a minute check during the scan, on a page already passed. + tracker.add(market(9)); + // Closed during the scan. + tracker.remove("0x1"); + let removed = tracker.scan_done(); + + assert!(matches!( + removed.as_slice(), + [(0, LaneCommand::Remove(market))] if market.condition_id == "0x2" + )); + let mut tracked: Vec<_> = tracker.markets.keys().cloned().collect(); + tracked.sort(); + assert_eq!(tracked, ["0x0", "0x9"]); + assert_eq!(tokens(&tracker), [4]); + } + + #[test] + fn a_failed_rescan_removes_nothing() { + let mut tracker = Tracker::default(); + tracker.add(market(0)); + tracker.scan_started(); + tracker.scan_failed(); + assert!(tracker.scan_done().is_empty()); + assert_eq!(tracker.markets.len(), 1); + } +} diff --git a/src/polymarket/socket.rs b/src/polymarket/socket.rs index b2b2155..bfc72a8 100644 --- a/src/polymarket/socket.rs +++ b/src/polymarket/socket.rs @@ -1,6 +1,8 @@ -//! The socket thread: reads the market channel and stamps each frame's receive -//! time, and nothing else. Parsing is the book thread's. +//! A lane's connection: reads the market channel and stamps each frame's +//! receive time, and sends what its book thread asks for (subscriptions, +//! resyncs). Parsing is the book thread's. +use std::collections::BTreeSet; use std::io::ErrorKind; use std::net::{TcpStream, ToSocketAddrs}; use std::sync::atomic::{AtomicBool, Ordering}; @@ -10,42 +12,66 @@ use std::time::{Duration, Instant}; use chrono::{DateTime, Utc}; use crossbeam::channel::{Receiver, Sender}; use serde_json::json; +use tungstenite::http::Uri; use tungstenite::stream::MaybeTlsStream; use tungstenite::{Message, Utf8Bytes, WebSocket}; -const MARKET_CHANNEL_HOST: &str = "ws-subscriptions-clob.polymarket.com"; -const MARKET_CHANNEL_URL: &str = "wss://ws-subscriptions-clob.polymarket.com/ws/market"; +pub const MARKET_CHANNEL_URL: &str = "wss://ws-subscriptions-clob.polymarket.com/ws/market"; /// For the TCP connect and the TLS + websocket handshake, and for every send. const CONNECT_TIMEOUT: Duration = Duration::from_secs(10); /// How often the loop wakes when no frame comes: tungstenite has no `split`, -/// so the reads and the writes below (PING, resyncs, close) share this thread. +/// so the reads and the writes below (PING, commands, close) share this thread. const READ_TIMEOUT: Duration = Duration::from_millis(200); const PING_EVERY: Duration = Duration::from_secs(10); const RECONNECT_AFTER: Duration = Duration::from_secs(1); const CLOSE_TIMEOUT: Duration = Duration::from_secs(2); +/// Tokens per subscribe or unsubscribe message: a first message of more than +/// 500 gets no opening books (docs/notes.md). +const CHUNK: usize = 500; /// A frame's text with its receive time. pub type Frame = (DateTime, Utf8Bytes); +/// What the book thread asks its connection for. +#[derive(Debug, PartialEq, Eq)] +pub enum Command { + Subscribe(Vec), + Unsubscribe(Vec), + /// Unsubscribe and subscribe again: the server then sends a fresh book. + Resync(String), +} + type Socket = WebSocket>; /// Runs until `stop` is set or the book thread is gone, reconnecting (and -/// subscribing every token again) whenever the connection drops. `resyncs` -/// carries the tokens to unsubscribe and subscribe again. +/// subscribing its tokens again) whenever the connection drops. pub fn run( - tokens: &[String], + url: &str, + lane: usize, frames: &Sender, - resyncs: &Receiver, + commands: &Receiver, stop: &AtomicBool, ) { + let mut tokens = BTreeSet::new(); while !stop.load(Ordering::Relaxed) { - match connect(tokens) { - Ok(mut socket) => match read(&mut socket, frames, resyncs, stop) { - Ok(Ended::Stopped) => return close(&mut socket, frames), - Ok(Ended::BookThreadGone) => return, - Err(error) => log!("polymarket: connection lost: {error}"), - }, - Err(error) => log!("polymarket: connect failed: {error}"), + // Asked for while disconnected: the subscribe on connect covers them, + // and its books are fresh, so resyncs have nothing left to do. + for command in commands.try_iter() { + _ = apply(&mut tokens, &command); + } + match connect(url, &tokens) { + Ok(mut socket) => { + log!( + "polymarket: lane {lane}: connected, {} symbols", + tokens.len() + ); + match read(&mut socket, &mut tokens, frames, commands, stop) { + Ok(Ended::Stopped) => return close(&mut socket, lane, frames), + Ok(Ended::BookThreadGone) => return, + Err(error) => log!("polymarket: lane {lane}: connection lost: {error}"), + } + } + Err(error) => log!("polymarket: lane {lane}: connect failed: {error}"), } thread::sleep(RECONNECT_AFTER); } @@ -56,17 +82,57 @@ enum Ended { BookThreadGone, } -fn connect(tokens: &[String]) -> anyhow::Result { +/// Updates the token set, and returns the messages that carry `command` out +/// on an open connection: none for a resync of a token not in the set. +#[must_use] +fn apply(tokens: &mut BTreeSet, command: &Command) -> Vec { + match command { + Command::Subscribe(added) => { + tokens.extend(added.iter().cloned()); + operations("subscribe", added) + } + Command::Unsubscribe(removed) => { + removed.iter().for_each(|token| _ = tokens.remove(token)); + operations("unsubscribe", removed) + } + // Subscribing a token already subscribed sends nothing back but + // `NO NEW ASSETS`: unsubscribing first makes the server send a fresh + // book. + Command::Resync(token) if tokens.contains(token) => [ + operations("unsubscribe", [token]), + operations("subscribe", [token]), + ] + .concat(), + Command::Resync(_) => Vec::new(), + } +} + +/// The messages that subscribe or unsubscribe `tokens` on an open connection. +fn operations<'a>(operation: &str, tokens: impl IntoIterator) -> Vec { + let tokens: Vec<_> = tokens.into_iter().collect(); + tokens + .chunks(CHUNK) + .map(|chunk| json!({"assets_ids": chunk, "operation": operation}).to_string()) + .collect() +} + +fn connect(url: &str, tokens: &BTreeSet) -> anyhow::Result { + let uri: Uri = url.parse()?; + let host = uri + .host() + .ok_or_else(|| anyhow::anyhow!("{url} has no host"))?; + // The market channel is `wss://` on TLS's port; the tests name theirs. + let port = uri.port_u16().unwrap_or(443); // Timed, unlike `tungstenite::connect`, so a stalled connect or handshake // can't keep SIGINT waiting. The name lookup itself has no timeout. - let address = (MARKET_CHANNEL_HOST, 443) + let address = (host, port) .to_socket_addrs()? .next() - .ok_or_else(|| anyhow::anyhow!("{MARKET_CHANNEL_HOST} has no address"))?; + .ok_or_else(|| anyhow::anyhow!("{host} has no address"))?; let stream = TcpStream::connect_timeout(&address, CONNECT_TIMEOUT)?; stream.set_read_timeout(Some(CONNECT_TIMEOUT))?; stream.set_write_timeout(Some(CONNECT_TIMEOUT))?; - let (mut socket, _response) = tungstenite::client_tls(MARKET_CHANNEL_URL, stream) + let (mut socket, _response) = tungstenite::client_tls(url, stream) .map_err(|error| anyhow::anyhow!("handshake failed: {error}"))?; let stream = match socket.get_ref() { MaybeTlsStream::Plain(stream) => stream, @@ -74,17 +140,21 @@ fn connect(tokens: &[String]) -> anyhow::Result { _ => anyhow::bail!("unexpected TLS stream"), }; stream.set_read_timeout(Some(READ_TIMEOUT))?; + // Opened empty, then subscribed in chunks: one rule for every token. socket.send(Message::text( - json!({"type": "market", "assets_ids": tokens}).to_string(), + json!({"type": "market", "assets_ids": []}).to_string(), ))?; - log!("polymarket: connected, subscribed {} symbols", tokens.len()); + for message in operations("subscribe", tokens) { + socket.send(Message::text(message))?; + } Ok(socket) } fn read( socket: &mut Socket, + tokens: &mut BTreeSet, frames: &Sender, - resyncs: &Receiver, + commands: &Receiver, stop: &AtomicBool, ) -> anyhow::Result { let mut last_ping = Instant::now(); @@ -111,25 +181,19 @@ fn read( socket.send(Message::text("PING"))?; last_ping = Instant::now(); } - for token in resyncs.try_iter() { - // Subscribing a token already subscribed sends nothing back but - // `NO NEW ASSETS`: unsubscribing first makes the server send a - // fresh book. - socket.send(Message::text( - json!({"assets_ids": [&token], "operation": "unsubscribe"}).to_string(), - ))?; - socket.send(Message::text( - json!({"assets_ids": [&token], "operation": "subscribe"}).to_string(), - ))?; + for command in commands.try_iter() { + for message in apply(tokens, &command) { + socket.send(Message::text(message))?; + } } } } /// Closes the connection, forwarding what still arrives until the server /// confirms or [`CLOSE_TIMEOUT`] passes. -fn close(socket: &mut Socket, frames: &Sender) { +fn close(socket: &mut Socket, lane: usize, frames: &Sender) { if let Err(error) = socket.close(None) { - return log!("polymarket: close failed: {error}"); + return log!("polymarket: lane {lane}: close failed: {error}"); } let deadline = Instant::now() + CLOSE_TIMEOUT; while Instant::now() < deadline { @@ -143,7 +207,6 @@ fn close(socket: &mut Socket, frames: &Sender) { Err(_) => break, } } - log!("polymarket: socket closed"); } /// A read timeout shows up as `WouldBlock` on macOS and `TimedOut` elsewhere; @@ -151,3 +214,108 @@ fn close(socket: &mut Socket, frames: &Sender) { fn timed_out(error: &std::io::Error) -> bool { matches!(error.kind(), ErrorKind::WouldBlock | ErrorKind::TimedOut) } + +#[cfg(test)] +mod tests { + use std::net::TcpListener; + use std::sync::Arc; + + use crossbeam::channel::unbounded; + + use super::*; + + /// The tokens each message of one connection subscribed, after the first. + fn subscribed(socket: &mut WebSocket, tokens: usize) -> Vec> { + let first = socket.read().unwrap().into_text().unwrap(); + let first: serde_json::Value = serde_json::from_str(&first).unwrap(); + assert_eq!(first, json!({"type": "market", "assets_ids": []})); + let mut chunks = Vec::new(); + while chunks.iter().map(Vec::len).sum::() < tokens { + let text = socket.read().unwrap().into_text().unwrap(); + let message: serde_json::Value = serde_json::from_str(&text).unwrap(); + assert_eq!(message["operation"], "subscribe"); + chunks.push(serde_json::from_value(message["assets_ids"].clone()).unwrap()); + } + chunks + } + + #[test] + fn a_reconnect_subscribes_every_token_again_in_chunks() { + let listener = TcpListener::bind("127.0.0.1:0").unwrap(); + let url = format!("ws://{}/ws/market", listener.local_addr().unwrap()); + let tokens: Vec = (0..1_200).map(|token| token.to_string()).collect(); + let total = tokens.len(); + let server = thread::spawn(move || { + let mut connections = Vec::new(); + for _ in 0..2 { + let (stream, _) = listener.accept().unwrap(); + // A client that never sends fails the test rather than hangs it. + stream + .set_read_timeout(Some(Duration::from_secs(5))) + .unwrap(); + let mut socket = tungstenite::accept(stream).unwrap(); + connections.push(subscribed(&mut socket, total)); + // The client reconnects after this close; after the second, + // it finds no server and stops. + socket.close(None).unwrap(); + while socket.read().is_ok() {} + } + connections + }); + + let (frames_tx, _frames_rx) = unbounded(); + let (commands_tx, commands_rx) = unbounded(); + commands_tx + .send(Command::Subscribe(tokens.clone())) + .unwrap(); + let stop = Arc::new(AtomicBool::new(false)); + let client_stop = Arc::clone(&stop); + let client = thread::spawn(move || run(&url, 0, &frames_tx, &commands_rx, &client_stop)); + + let connections = server.join().unwrap(); + stop.store(true, Ordering::Relaxed); + client.join().unwrap(); + let mut tokens = tokens; + tokens.sort(); + for chunks in connections { + assert!(chunks.iter().all(|chunk| chunk.len() <= CHUNK)); + let mut subscribed = chunks.concat(); + subscribed.sort(); + assert_eq!(subscribed, tokens); + } + } + + #[test] + fn commands_update_the_token_set_and_say_what_to_send() { + let mut tokens = BTreeSet::new(); + let operation = |message: &String| -> (String, Vec) { + let message: serde_json::Value = serde_json::from_str(message).unwrap(); + let tokens = serde_json::from_value(message["assets_ids"].clone()).unwrap(); + (message["operation"].as_str().unwrap().to_string(), tokens) + }; + let sent = |messages: Vec| messages.iter().map(operation).collect::>(); + let op = |name: &str, tokens: &[&str]| { + ( + name.to_string(), + tokens.iter().map(|token| token.to_string()).collect(), + ) + }; + + let subscribe = Command::Subscribe(vec!["a".into(), "b".into()]); + assert_eq!( + sent(apply(&mut tokens, &subscribe)), + [op("subscribe", &["a", "b"])] + ); + assert_eq!( + sent(apply(&mut tokens, &Command::Resync("a".into()))), + [op("unsubscribe", &["a"]), op("subscribe", &["a"])] + ); + let unsubscribe = Command::Unsubscribe(vec!["a".into()]); + assert_eq!( + sent(apply(&mut tokens, &unsubscribe)), + [op("unsubscribe", &["a"])] + ); + assert!(apply(&mut tokens, &Command::Resync("a".into())).is_empty()); + assert_eq!(tokens.into_iter().collect::>(), ["b"]); + } +} diff --git a/src/reconcile.rs b/src/reconcile.rs index 707a8ff..bf195c8 100644 --- a/src/reconcile.rs +++ b/src/reconcile.rs @@ -1,6 +1,6 @@ //! Checks an output file against itself: per symbol, the Deltas written since //! a Snapshot, applied to it, must give the next Snapshot. A mismatch is a loss -//! or a resync gap (the fresh book differs), each logged by the collector. +//! or a subscription gap (the fresh book differs), each logged by the collector. use std::collections::HashMap; use std::path::Path; diff --git a/tests/fixtures/README.md b/tests/fixtures/README.md index 5dec406..07b6b14 100644 --- a/tests/fixtures/README.md +++ b/tests/fixtures/README.md @@ -1,10 +1,16 @@ # Test fixtures A recording of Polymarket's market channel, replayed by the tests in -`src/polymarket/feed.rs` (no network in tests). +`src/polymarket/feed.rs`, and Gamma pages read by `src/polymarket/gamma.rs` +(no network in tests). - `gamma-markets.json`: Gamma's top 20 markets by 24 h volume on 2026-11-04, - trimmed to the fields the collector reads. + trimmed to the fields the collector reads. `conditionId`, `active`, `closed` + and `acceptingOrders` were added in sprint 03 (the ids from the recorded + books' `market`). +- `gamma-created.json`, `gamma-closed.json`: one `/markets/keyset` page each + (`limit=10`) of the newest live markets by `createdAt` and the latest closed + by `closedTime`, 2026-11-04 ~17:00 UTC, trimmed the same way. - `polymarket-frames.jsonl`: the first 90 s (14:24:44 to 14:26:14 UTC, so two minute boundaries) of the market channel for those markets' 40 tokens, one line per frame: `{"received_at_us": , "text": }`. diff --git a/tests/fixtures/gamma-closed.json b/tests/fixtures/gamma-closed.json new file mode 100644 index 0000000..2249655 --- /dev/null +++ b/tests/fixtures/gamma-closed.json @@ -0,0 +1,135 @@ +{ + "markets": [ + { + "question": "Populovskyi Eduard vs. Kovalenko Anatolii", + "conditionId": "0xefab23442b429fa445b8e52394c86caf3f092bf3f71300829b1c1eb0ce7f0b50", + "clobTokenIds": "[\"44826501889331181717150371523936754626679193531166396487334749539805478460209\", \"36302256322938166611393018282312068242391943224849867955841298097733298988891\"]", + "orderMinSize": 5, + "negRisk": false, + "active": true, + "closed": true, + "enableOrderBook": true, + "acceptingOrders": false, + "createdAt": "2026-11-03T19:53:35.292723Z", + "closedTime": "2026-11-04 16:57:50.384485+00" + }, + { + "question": "Set Handicap: Niels Peter Van Noord (-1.5) vs Keshav Chopra (+1.5)", + "conditionId": "0x2e131dc2a29bf2d276d34d7e3c0cdfca4e3b4cd3512f4b04a1a006e626f4ae61", + "clobTokenIds": "[\"39895080082116076744071586278013201425211458703422956002772510408434190677884\", \"19051851145542724727656665058814185905321034954829010093680773644546996513535\"]", + "orderMinSize": 5, + "negRisk": false, + "active": true, + "closed": true, + "enableOrderBook": true, + "acceptingOrders": false, + "createdAt": "2026-11-04T03:44:14.574104Z", + "closedTime": "2026-11-04 16:57:50.358086+00" + }, + { + "question": "Set Handicap: Kayden Colombo (-1.5) vs Johan Alexander Rodriguez (+1.5)", + "conditionId": "0x5353ac2ea7d3f4e21acb4d0861cb5b44ac779a096de9f0ffc546ff8140feb5fa", + "clobTokenIds": "[\"94958295102858405885818380458434883996828517060091040376117137374766737083914\", \"7671610100222080411744632394529476881255214291291880601575663863683487114654\"]", + "orderMinSize": 5, + "negRisk": false, + "active": true, + "closed": true, + "enableOrderBook": true, + "acceptingOrders": false, + "createdAt": "2026-11-04T03:44:13.474846Z", + "closedTime": "2026-11-04 16:57:22.344836+00" + }, + { + "question": "Set Handicap: Maria Jose Sanchez Uribe (-1.5) vs Luciana Agudelo (+1.5)", + "conditionId": "0xe9a4d2cc5013a9ef930769a06067bf1793b44e277262cedec9382605d1f4d789", + "clobTokenIds": "[\"21745108597700168910860076616721481120538218170946858631858304920364048593626\", \"88297511931567368303913767416363723944767307572967321208894173417589979621188\"]", + "orderMinSize": 5, + "negRisk": false, + "active": true, + "closed": true, + "enableOrderBook": true, + "acceptingOrders": false, + "createdAt": "2026-11-03T21:44:03.719514Z", + "closedTime": "2026-11-04 16:57:22.316215+00" + }, + { + "question": "Kuzyshyn Oleh vs. Blashkun Oleksandr", + "conditionId": "0xf9a84b5f47465e3256630198c3e593819ff00978d660842dc4f259f23d45b852", + "clobTokenIds": "[\"97554300278659417619290655189631481316508823986205552306096670194286646500988\", \"20252670317414328146520571690921965527846507756708657221097476588874672725450\"]", + "orderMinSize": 5, + "negRisk": false, + "active": true, + "closed": true, + "enableOrderBook": true, + "acceptingOrders": false, + "createdAt": "2026-11-03T19:52:18.066777Z", + "closedTime": "2026-11-04 16:57:22.288775+00" + }, + { + "question": "Set Handicap: Weronika Ewald (-1.5) vs Ida Wobker (+1.5)", + "conditionId": "0x4367cfbecaee2995e81fb1514ccf3ab26502f41cd9b8d588136fdecc3160bdb7", + "clobTokenIds": "[\"13297249568103066689520348076229200996848266474396001529378944683098932755914\", \"89894550493893133858598989669017150578230792898149665317182501688736361564700\"]", + "orderMinSize": 5, + "negRisk": false, + "active": true, + "closed": true, + "enableOrderBook": true, + "acceptingOrders": false, + "createdAt": "2026-11-03T21:44:04.165837Z", + "closedTime": "2026-11-04 16:57:22.261927+00" + }, + { + "question": "Will Tesla, Inc. (TSLA) hit (LOW) $390 Week of November 2 2026?", + "conditionId": "0xadd174f7eeb05feba859b8d4a4990b6fdd62de6382d6b0b34a5278aa0cec9e93", + "clobTokenIds": "[\"53111618025983699124066570152720566574297036520169668823377484893951216981918\", \"1738729912475637393699992934041272180198757392022311537253739693114313017539\"]", + "orderMinSize": 5, + "negRisk": false, + "active": true, + "closed": true, + "enableOrderBook": true, + "acceptingOrders": false, + "createdAt": "2026-10-30T22:00:27.078976Z", + "closedTime": "2026-11-04 16:57:22.234043+00" + }, + { + "question": "W35 Liberec: Ida Wobker vs Weronika Ewald", + "conditionId": "0xd45f79466a8883b774390fc1abb30f63a791ba3e9a85b7d6d92c9015acb115c0", + "clobTokenIds": "[\"37553866556598063681604558627906332172694310397407486121862876197091898752599\", \"77201422525387544766857381326308690506358974601454997287230251810366877318676\"]", + "orderMinSize": 5, + "negRisk": false, + "active": true, + "closed": true, + "enableOrderBook": true, + "acceptingOrders": false, + "createdAt": "2026-11-03T21:14:12.671977Z", + "closedTime": "2026-11-04 16:57:22.204847+00" + }, + { + "question": "W50 Heraklion: Carlota Martinez Cirez vs Daria Yesypchuk", + "conditionId": "0x76bce54b6da6ec1663e6c7a9fe3eb3e4ee57ccb6e4bd64aec2cb0c925ff5708d", + "clobTokenIds": "[\"67085659236189310000851293394749121730386584794981034590417392334188473111344\", \"64682819352472807351567187431129959173116373378073567759284036383866483667934\"]", + "orderMinSize": 5, + "negRisk": false, + "active": true, + "closed": true, + "enableOrderBook": true, + "acceptingOrders": false, + "createdAt": "2026-11-03T21:14:11.443782Z", + "closedTime": "2026-11-04 16:57:06.115352+00" + }, + { + "question": "Djokovic vs. Hijikata: Match O/U 22.5", + "conditionId": "0x8c8b46f1b0bfaa40e45de277b1aa0cae3f50e5752dd11492b68a6bcd66af7d79", + "clobTokenIds": "[\"69853942453264725463023948212497122784872761667832747694549450815793109801231\", \"99642552909265075117113081841538849880069795592738032133677294961975308569760\"]", + "orderMinSize": 5, + "negRisk": false, + "active": true, + "closed": true, + "enableOrderBook": true, + "acceptingOrders": false, + "createdAt": "2026-11-03T18:44:14.155967Z", + "closedTime": "2026-11-04 16:57:06.086161+00" + } + ], + "next_cursor": "YM0JZysiu5w7xwmDjxuLOiH9N_GyhP7sKFS4xl4RPWB7InYiOjEsImsiOiJtYXJrZXRzIiwib2giOiJjNmI2MzAwYWRmM2Q3NWFmYTZhYzAzNDUwNTM3MzZhMDk2ODc3Mzc2NDM5OTQ0MjIzYjA0MDRiMjU0MTVkMDVkIiwia2V5cyI6W3sidCI6InN0cmluZyIsInYiOiIyMDI2LTExLTA0IDE2OjU3OjA2LjA4NjE2MSswMCJ9LHsidCI6InN0cmluZyIsInYiOiI2NjQ3OTE4In1dfQ" +} \ No newline at end of file diff --git a/tests/fixtures/gamma-created.json b/tests/fixtures/gamma-created.json new file mode 100644 index 0000000..18ada0a --- /dev/null +++ b/tests/fixtures/gamma-created.json @@ -0,0 +1,125 @@ +{ + "markets": [ + { + "question": "XRP Up or Down - November 5, 11:50AM-11:55AM ET", + "conditionId": "0x7af9827676c3790cce6662e5a335d6f017ae99ecd9bd22b1d6f02bd4cf686f9b", + "clobTokenIds": "[\"115574995017995341868238121220540154732508720721227808951068953932623414227263\", \"42196855944990128389362302048239777171495224595711316446705919987418523770734\"]", + "orderMinSize": 5, + "negRisk": false, + "active": true, + "closed": false, + "enableOrderBook": true, + "acceptingOrders": true, + "createdAt": "2026-11-04T16:57:29.989119Z" + }, + { + "question": "Exact Score: Any Other Score?", + "conditionId": "0xa19ecbe2c5ee9809b9902ddadf27c002171e91a4dc1eff5f3df515072ac63580", + "clobTokenIds": "[\"89984761009969090606365049502308215774844729861018229163736541869010668521883\", \"109098292314077657939935494027719386110066867173218583078610092308306402652377\"]", + "orderMinSize": 5, + "negRisk": true, + "active": true, + "closed": false, + "enableOrderBook": true, + "acceptingOrders": true, + "createdAt": "2026-11-04T16:54:27.166917Z" + }, + { + "question": "Exact Score: South Shields 5 - 5 Southport FC?", + "conditionId": "0x526a2e209ecdc996126f7b0ce8aca970e238d00a5e8e1462583b319f161a81e2", + "clobTokenIds": "[\"23383971224876511286751474417288589641928550057174153221853770194514531463145\", \"102514263000521269570516172503620470436693609695432994526595167508530804022833\"]", + "orderMinSize": 5, + "negRisk": true, + "active": true, + "closed": false, + "enableOrderBook": true, + "acceptingOrders": true, + "createdAt": "2026-11-04T16:54:27.084521Z" + }, + { + "question": "Exact Score: South Shields 5 - 4 Southport FC?", + "conditionId": "0xef7541f4318da1290833e7d4228eec646c8b8c3dc56c1e54f19aab5999b79eac", + "clobTokenIds": "[\"9590662984010730528349011455067967517680765388249930403038178415563136879677\", \"27306691953532238445589487371954210070154006783856682982476322761457890551225\"]", + "orderMinSize": 5, + "negRisk": true, + "active": true, + "closed": false, + "enableOrderBook": true, + "acceptingOrders": true, + "createdAt": "2026-11-04T16:54:27.011508Z" + }, + { + "question": "Exact Score: South Shields 4 - 5 Southport FC?", + "conditionId": "0x650c95e95ded1b214ffd5ce3807ebbce5b53a5652e8d9bc4e5fc81a9b88e7e91", + "clobTokenIds": "[\"20427555995993676408608258992371178601145040240934408396949232679359337375578\", \"103063473308836065441608442719693527177410664791261885269576628238500009671761\"]", + "orderMinSize": 5, + "negRisk": true, + "active": true, + "closed": false, + "enableOrderBook": true, + "acceptingOrders": true, + "createdAt": "2026-11-04T16:54:26.943636Z" + }, + { + "question": "Exact Score: South Shields 5 - 3 Southport FC?", + "conditionId": "0x6b8873a1138c840f212b1232decf41022d3c3d9152fc3630ac208ec373d223e4", + "clobTokenIds": "[\"113885867652625558735658932195210301609406698665198534717816428669601426385071\", \"111530920921450186157632707389308324089042969410875400892559036758646303736272\"]", + "orderMinSize": 5, + "negRisk": true, + "active": true, + "closed": false, + "enableOrderBook": true, + "acceptingOrders": true, + "createdAt": "2026-11-04T16:54:26.857401Z" + }, + { + "question": "Exact Score: South Shields 4 - 4 Southport FC?", + "conditionId": "0xead8378060d14693e9b389460df33f4744f00624abebcac49f065fe4df71a8e1", + "clobTokenIds": "[\"74671658663122677156313407534212107516599429188436460342260016431503072138377\", \"98530071447174348550020247148894521999579673609192238144726023516722974096996\"]", + "orderMinSize": 5, + "negRisk": true, + "active": true, + "closed": false, + "enableOrderBook": true, + "acceptingOrders": true, + "createdAt": "2026-11-04T16:54:26.793546Z" + }, + { + "question": "Exact Score: South Shields 3 - 5 Southport FC?", + "conditionId": "0x2b1ce687525f18ad61a59e36f23450d21f054f63f1e45f8ad1280599bcec5a9c", + "clobTokenIds": "[\"57945964795574745726585942326101714792053389461524739487276980421631849920286\", \"49065546046569817187747663996196121254480922109385243099359350773652868328963\"]", + "orderMinSize": 5, + "negRisk": true, + "active": true, + "closed": false, + "enableOrderBook": true, + "acceptingOrders": true, + "createdAt": "2026-11-04T16:54:26.720058Z" + }, + { + "question": "Exact Score: South Shields 5 - 2 Southport FC?", + "conditionId": "0x73e7aad3c25b27f6ea5690533fbadda5232275bc3b2093be105bf2e8f0451f86", + "clobTokenIds": "[\"7212691718059316105797204867601393556290090412559254640981266981182975515579\", \"61803842573819523268874917567084084298265802112323835114562959941367399972826\"]", + "orderMinSize": 5, + "negRisk": true, + "active": true, + "closed": false, + "enableOrderBook": true, + "acceptingOrders": true, + "createdAt": "2026-11-04T16:54:26.635393Z" + }, + { + "question": "Exact Score: South Shields 4 - 3 Southport FC?", + "conditionId": "0x39774e1c4ea209f84a503fd99093e2ba0857b1e4ac9850e7fc7aea843f9ab68f", + "clobTokenIds": "[\"95978542179162501111386271847728617693164545763728410170416353787697899447782\", \"82831019763051352453486760727930593092008868235043819951383814785887774411178\"]", + "orderMinSize": 5, + "negRisk": true, + "active": true, + "closed": false, + "enableOrderBook": true, + "acceptingOrders": true, + "createdAt": "2026-11-04T16:54:26.595371Z" + } + ], + "next_cursor": "HD40wpBf4vc1G7nlWBWBeQio70FL4a-CS_Q9LExFBo17InYiOjEsImsiOiJtYXJrZXRzIiwib2giOiI5ZGExMThlZTZhMDViZmVkYTVjM2E1MzFjY2RhZWNkZTY0ZDlkYjQyZWQyMzQxZmE0NDcwNDBjOGViYWI1YzAxIiwia2V5cyI6W3sidCI6InRpbWUiLCJ2IjoiMjAyNi0xMS0wNFQxNjo1NDoyNi41OTUzNzFaIn0seyJ0Ijoic3RyaW5nIiwidiI6IjY2ODMyMTkifV19" +} \ No newline at end of file diff --git a/tests/fixtures/gamma-markets.json b/tests/fixtures/gamma-markets.json index cf896fe..e0a2b0a 100644 --- a/tests/fixtures/gamma-markets.json +++ b/tests/fixtures/gamma-markets.json @@ -4,139 +4,219 @@ "clobTokenIds": "[\"34722410608062854697106861099776685947172185964394483545370684749662285977831\", \"24483563228660028121020430722896740519524131851526579619586176931221901842798\"]", "orderMinSize": 5, "negRisk": true, - "enableOrderBook": true + "enableOrderBook": true, + "conditionId": "0x16c63b7cc37f012b9f59ee164ec03877914c701d06d48291ae8d6fc08a088b0d", + "active": true, + "closed": false, + "acceptingOrders": true }, { "question": "2026 Balance of Power: R Senate, R House", "clobTokenIds": "[\"103704141773947678931823410030956181918562062788486034785782641603149828893320\", \"25869296386911654049715489068824757745174681288397238521780118949553760488529\"]", "orderMinSize": 5, "negRisk": true, - "enableOrderBook": true + "enableOrderBook": true, + "conditionId": "0xc5eae79d1ffe716572353962eb926b1e3964c500a4880a7a94f58408218ee76b", + "active": true, + "closed": false, + "acceptingOrders": true }, { "question": "Will Republicans lose the Alaska, Iowa, Kansas, Maine, Michigan, Ohio, and Texas Senate seats?", "clobTokenIds": "[\"13535941900723044344907002451427688051727855016747811259193368015899108383450\", \"80727480162707694817992062873857913056698208056368499870278974630056151503113\"]", "orderMinSize": 5, "negRisk": true, - "enableOrderBook": true + "enableOrderBook": true, + "conditionId": "0xa32317ba83c3c83dc43ef260c3744c22ed8000db53d30bc09372c45cff5fb07a", + "active": true, + "closed": false, + "acceptingOrders": true }, { "question": "2026 Balance of Power: R Senate, D House", "clobTokenIds": "[\"6302359956133594764084277082169634158291609371270652093164054687145970756151\", \"558600218792430158020964384658762756367786275998757133304987611939841005404\"]", "orderMinSize": 5, "negRisk": true, - "enableOrderBook": true + "enableOrderBook": true, + "conditionId": "0x998bc71817b2d76921d1999ce0f3431cfd5945583667a371280ca2b430b0c06e", + "active": true, + "closed": false, + "acceptingOrders": true }, { "question": "Will the Republicans win the Mississippi Senate race in 2026?", "clobTokenIds": "[\"36589074670521348842786325911391532620430247097981345860356484520500173888573\", \"4639340586725578716005500791739559303082924048072534329825226706547465941088\"]", "orderMinSize": 5, "negRisk": true, - "enableOrderBook": true + "enableOrderBook": true, + "conditionId": "0xc8c5760c26490b0cd0bedcf1c61aa86c5e09c8b2886b84ff4ea2db81d5f29388", + "active": true, + "closed": false, + "acceptingOrders": true }, { "question": "Will Mary Peltola win the Alaska Senate race in 2026?", "clobTokenIds": "[\"72139607190315861034411702195742538105117960260100800877125381826122747504258\", \"38274379050640062523611849065721372699838119672720475300345842413156391540232\"]", "orderMinSize": 5, "negRisk": true, - "enableOrderBook": true + "enableOrderBook": true, + "conditionId": "0x19d2ff72e389d190c183e6b114817301d29eba651d67d75a38ee201a05188fd9", + "active": true, + "closed": false, + "acceptingOrders": true }, { "question": "Will the Democrats win the Mississippi Senate race in 2026?", "clobTokenIds": "[\"77001639760095266374534198365421436593389036687975321536722226607654949914567\", \"109329484956764506378760583725617100193692863947320202695087573398990069238209\"]", "orderMinSize": 5, "negRisk": true, - "enableOrderBook": true + "enableOrderBook": true, + "conditionId": "0x8c9b8ad3a64088a06d45e0b296a9a836203bc7f3ef35d134c57abead446a8b72", + "active": true, + "closed": false, + "acceptingOrders": true }, { "question": "2026 Balance of Power: D Senate, R House", "clobTokenIds": "[\"70997927349469817841862065582625658840347600365813612622959588796331622340305\", \"45720947308530468765087874228584729250108790729495800420104302263408503667446\"]", "orderMinSize": 5, "negRisk": true, - "enableOrderBook": true + "enableOrderBook": true, + "conditionId": "0x0808de4f0cfd47947f2d1be51f9a9c52ea0fec76f73a75cfbe79ddec98d8a908", + "active": true, + "closed": false, + "acceptingOrders": true }, { "question": "Will Republicans win Iowa, and lose Alaska, Kansas, Maine, Michigan, Ohio, and Texas?", "clobTokenIds": "[\"106253100341058888227154298197221161777912145262343559280892425406157624656915\", \"38883886691124225989332035378913105158890603529288875957747834557386418386450\"]", "orderMinSize": 5, "negRisk": true, - "enableOrderBook": true + "enableOrderBook": true, + "conditionId": "0xd116777a710b6a7c336fe12cbf742fafb0d8ac508cf400446159973cec9aec29", + "active": true, + "closed": false, + "acceptingOrders": true }, { "question": "Will the US confirm that aliens exist before 2027?", "clobTokenIds": "[\"107505882767731489358349912513945399560393482969656700824895970500493757150417\", \"7305630249804085635496399869905769372294302716159034447326228509068694952392\"]", "orderMinSize": 5, "negRisk": false, - "enableOrderBook": true + "enableOrderBook": true, + "conditionId": "0x747dc809fb79e1b05be09c42d6179459a58de2ef3e40f02484a4e1260f741f75", + "active": true, + "closed": false, + "acceptingOrders": true }, { "question": "Rolex Paris Masters: Alexander Zverev vs Lorenzo Musetti", "clobTokenIds": "[\"97202122435166371596632600271207731098197012320781592658111953138552052790871\", \"64561849525628736112915971493239643546218846988377083760471198523625509677016\"]", "orderMinSize": 5, "negRisk": false, - "enableOrderBook": true + "enableOrderBook": true, + "conditionId": "0x4b5ff3a152f6535ac82fbefb2034a8e5cf1444f6458b6f0491efabe9a7c73289", + "active": true, + "closed": false, + "acceptingOrders": true }, { "question": "Will Republicans win Alaska, and lose Iowa, Kansas, Maine, Michigan, Ohio, and Texas?", "clobTokenIds": "[\"83965128865532878903206942328238684643656639972076348062693215949819184139617\", \"93889370639599634853686597701962997508099936018794651953480468245159849951085\"]", "orderMinSize": 5, "negRisk": true, - "enableOrderBook": true + "enableOrderBook": true, + "conditionId": "0xe3b51058c3bebfb100c5e1d59ab89d395c8938150a788f1417157fe360bea48a", + "active": true, + "closed": false, + "acceptingOrders": true }, { "question": "US x Iran ceasefire continues through October 31?", "clobTokenIds": "[\"67102598924726349989742049657609063097299442010642912942032681022413402597017\", \"114475889232765203212032217024645308497828327658421967586050553616813506019119\"]", "orderMinSize": 5, "negRisk": false, - "enableOrderBook": true + "enableOrderBook": true, + "conditionId": "0x3738ad9665480991affe3ba286cd035516e17d1f8b7703e05088ec19c70c21a4", + "active": true, + "closed": false, + "acceptingOrders": true }, { "question": "Will Real Madrid CF win on 2026-11-04?", "clobTokenIds": "[\"39515423851331707664526519001996875757487006788622502797081364834883774387767\", \"3771084942539439798998256191956824579778203065651547449472385427482459623087\"]", "orderMinSize": 5, "negRisk": true, - "enableOrderBook": true + "enableOrderBook": true, + "conditionId": "0x8941f985cce9ac4a175ce7a0a3d75f38536a43de46517d75aa9658ee2cf55574", + "active": true, + "closed": false, + "acceptingOrders": true }, { "question": "Will Republicans win Iowa and Kansas, and lose Alaska, Maine, Michigan, Ohio, and Texas?", "clobTokenIds": "[\"105070040449310495327131730101841566321530681945885747881686068698995174349681\", \"111381299800799485509083912497813429956512687399857260440199290536659860679263\"]", "orderMinSize": 5, "negRisk": true, - "enableOrderBook": true + "enableOrderBook": true, + "conditionId": "0x564959d4c61eeb7c568d6f8e4f5a0dcf3622434d4f264d418d9674102e5a7a88", + "active": true, + "closed": false, + "acceptingOrders": true }, { "question": "Will the U.S. invade Iran before 2027?", "clobTokenIds": "[\"55115078421062885512539156303747803058407616201213034911037320915726138659123\", \"1910830010387565971650098373488592514702818137344973088263643820608151819241\"]", "orderMinSize": 5, "negRisk": false, - "enableOrderBook": true + "enableOrderBook": true, + "conditionId": "0x5db999fad322cea2914535aae5517060c3f80ad6d8c0231cde2124a434d16846", + "active": true, + "closed": false, + "acceptingOrders": true }, { "question": "Will Benjamin Netanyahu be the next Prime Minister of Israel?", "clobTokenIds": "[\"20006732765674855733524007935991362439352594042415161039767275461950712817548\", \"54533569528927082430133047556849600511985191158927477533124630301934489498530\"]", "orderMinSize": 5, "negRisk": true, - "enableOrderBook": true + "enableOrderBook": true, + "conditionId": "0x7586a96520578acaaaa4ea84a2582f197f84255da1f3392a7aa300386c187b37", + "active": true, + "closed": false, + "acceptingOrders": true }, { "question": "Will Dan Sullivan win the Alaska Senate race in 2026?", "clobTokenIds": "[\"2618138322856109565904469085316346602627882727483900432328802655317079880335\", \"64264976954058187238724333554507208255303776435657387002861226761784169752307\"]", "orderMinSize": 5, "negRisk": true, - "enableOrderBook": true + "enableOrderBook": true, + "conditionId": "0x741bb46008b72234857c5118c8365ab3c2f69dbf28b30b59de04f173b8a778de", + "active": true, + "closed": false, + "acceptingOrders": true }, { "question": "Will Nithya Raman win the 2026 Los Angeles mayoral election?", "clobTokenIds": "[\"50292933818751585369672690878774599873941191925538325333138798248932182222864\", \"58507084608782512565480280053882807505189448738098588563704773804344223603355\"]", "orderMinSize": 5, "negRisk": true, - "enableOrderBook": true + "enableOrderBook": true, + "conditionId": "0x694595d0a008f10e4701285d071e4b08407495f6f8a6bd8cbdd7de522fa9b463", + "active": true, + "closed": false, + "acceptingOrders": true }, { "question": "Will John McEntee win the 2028 Republican presidential nomination?", "clobTokenIds": "[\"50730508396692945396787154191308995513666087024795374376582570929738968294242\", \"7808428608295632540409928244677767077616163699109217296255297216219773833140\"]", "orderMinSize": 5, "negRisk": true, - "enableOrderBook": true + "enableOrderBook": true, + "conditionId": "0x90ba5eeb935da5a2e7d16ee635ae441f95b5b62fd0db9336eb9f8a59645bf714", + "active": true, + "closed": false, + "acceptingOrders": true } ] \ No newline at end of file