Archived
feat(polymarket): hash-verified order books for a subset of symbols
Collects the order books of Polymarket's 20 busiest markets (40 symbols) into the template's SBE format: a Delta per level change, plus Snapshots on subscribe, every minute, and whenever a fresh book corrects ours. Every change is checked against Polymarket's own book hash; a failed check resyncs the symbol on the open socket, and the log counts losses apart from the few ms our own resync hides. Runs until SIGINT, which closes the socket and flushes the output; reconnects when the server drops it. A `reconcile` binary checks an output file against itself.
This commit is contained in:
22 files changed
+6423
-84
No files matched your search
@@ -14,6 +14,11 @@ outside this repo.
|
||||
- `docs/adr/`: hard-to-reverse decisions (0001: threads, shared-nothing).
|
||||
- `docs/notes.md`: domain rules and exchange protocol facts (delta semantics,
|
||||
loss detection, message shapes). Read before touching exchange parsing.
|
||||
- `GLOSSARY.md`: the terms (symbol, resync, loss, resync gap, Snapshot, Delta).
|
||||
- `src/polymarket/feed.rs`: the book thread and its loss rules (the subtle
|
||||
part); `src/polymarket/socket.rs`: the socket thread; `src/book.rs`: the
|
||||
book and fixed point; `src/reconcile.rs` + `src/bin/reconcile.rs`: the output
|
||||
check. `tests/fixtures/`: a 90 s recording of the live feed the tests replay.
|
||||
- `src/util/` and `sbe_schemas/`: the template's SBE machinery, used as given so
|
||||
output stays readable by the template's `AlignedSbeReader`. Our code is the
|
||||
rest of `src/`.
|
||||
@@ -31,9 +36,11 @@ and let the user settle it. Claude owns the data model.
|
||||
plain threads and channels. Speed work waits for the profiler, after V1.
|
||||
- An abstraction arrives with its second implementation (the exchange trait in
|
||||
M6).
|
||||
- Stamp receive time immediately after the socket read, and rebuild a book from
|
||||
a fresh snapshot whenever a lost message is detected (how depends on the
|
||||
exchange: `docs/notes.md`).
|
||||
- Stamp receive time immediately after the socket read, and resync a book (a
|
||||
fresh full book from the exchange) whenever a loss is detected (how depends
|
||||
on the exchange: `docs/notes.md`).
|
||||
- Use `GLOSSARY.md`'s terms: a **Snapshot** is only the output message; getting
|
||||
a fresh book is a **resync**.
|
||||
- A corner cut for speed carries a comment naming its ceiling and upgrade path.
|
||||
|
||||
## Environment traps
|
||||
@@ -42,12 +49,21 @@ and let the user settle it. Claude owns the data model.
|
||||
resolver answers `195.186.4.210` / `2001:4d98:4:b10c::1` (a Swisscom block
|
||||
page). The user pins the hosts with `sudo ../host.sh on` (workspace root,
|
||||
outside this repo; `off` removes them). A timeout or the block address again
|
||||
means the CDN addresses rotated: ask the user to rerun it.
|
||||
means the CDN addresses rotated: ask the user to rerun it. A host not yet in
|
||||
`host.sh`'s `HOSTS` list fails the same way: add it there and ask the user to
|
||||
rerun it. A 403 is not DNS (see `docs/notes.md`, REST from scripts).
|
||||
- **Write 64 MiB chunks**: `AlignedSbeReader::open` hardcodes 64 MiB
|
||||
(`open_with_chunk_size` is private), so any other chunk size is unreadable.
|
||||
- **The binary runs until the server closes the socket.** macOS has no
|
||||
`timeout`; bound a run with
|
||||
`perl -e 'alarm 10; exec @ARGV' ./target/debug/engineering-challenge`.
|
||||
- **The collector runs until SIGINT, and only SIGINT flushes its output.**
|
||||
macOS has no `timeout`, and a `perl alarm` kill (SIGALRM) leaves the last
|
||||
64 MiB chunk unwritten. Bound a run with SIGINT, then check the output:
|
||||
`./target/release/engineering-challenge out.sbe 2> run.log & pid=$!;
|
||||
sleep 600; kill -INT $pid; wait $pid; ./target/release/reconcile out.sbe`.
|
||||
`reconcile` counts resync gaps as mismatches too: compare its list with the
|
||||
log's "loss" and "resync gap" lines.
|
||||
- **Live data beats the docs**: Polymarket's market channel has rules its
|
||||
docs don't state (`docs/notes.md`); check a protocol assumption on a
|
||||
recording of the real feed (`tests/fixtures/record.py`) before relying on it.
|
||||
|
||||
## Workflow differences
|
||||
|
||||
|
||||
Generated
+179
-1
@@ -47,7 +47,7 @@ dependencies = [
|
||||
"addr2line",
|
||||
"cfg-if",
|
||||
"libc",
|
||||
"miniz_oxide",
|
||||
"miniz_oxide 0.8.9",
|
||||
"object",
|
||||
"rustc-demangle",
|
||||
"windows-link",
|
||||
@@ -62,6 +62,12 @@ dependencies = [
|
||||
"backtrace",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "base64"
|
||||
version = "0.23.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "ac07cdecf99051d9a5238b80f35af32cdeba5b336e55d957b318b50137e18da5"
|
||||
|
||||
[[package]]
|
||||
name = "bitflags"
|
||||
version = "2.13.2"
|
||||
@@ -77,6 +83,15 @@ dependencies = [
|
||||
"hybrid-array",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "block2"
|
||||
version = "0.6.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "cdeb9d870516001442e364c5220d3574d2da8dc765554b4a617230d33fa58ef5"
|
||||
dependencies = [
|
||||
"objc2",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "bumpalo"
|
||||
version = "3.20.3"
|
||||
@@ -105,6 +120,12 @@ version = "1.0.5"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "4e7648175b45a9a48536d676f68d918270699102aa8dab5496df06904c914600"
|
||||
|
||||
[[package]]
|
||||
name = "cfg_aliases"
|
||||
version = "0.2.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f079e83a288787bcd14a6aea84cee5c87a67c5a3e660c30f557a3d24761b3527"
|
||||
|
||||
[[package]]
|
||||
name = "chacha20"
|
||||
version = "0.10.2"
|
||||
@@ -150,6 +171,15 @@ dependencies = [
|
||||
"libc",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "crc32fast"
|
||||
version = "1.5.4"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "5f97d3416d5337d15c4b0d78c8c91df922ca143f57ec20e2d1c188333047ef04"
|
||||
dependencies = [
|
||||
"cfg-if",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "crossbeam"
|
||||
version = "0.8.5"
|
||||
@@ -215,6 +245,17 @@ dependencies = [
|
||||
"hybrid-array",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "ctrlc"
|
||||
version = "3.5.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "e0b1fab2ae45819af2d0731d60f2afe17227ebb1a1538a236da84c93e9a60162"
|
||||
dependencies = [
|
||||
"dispatch2",
|
||||
"nix",
|
||||
"windows-sys 0.61.2",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "data-encoding"
|
||||
version = "2.11.1"
|
||||
@@ -232,6 +273,18 @@ dependencies = [
|
||||
"crypto-common",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "dispatch2"
|
||||
version = "0.3.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "1e0e367e4e7da84520dedcac1901e4da967309406d1e51017ae1abfb97adbd38"
|
||||
dependencies = [
|
||||
"bitflags",
|
||||
"block2",
|
||||
"libc",
|
||||
"objc2",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "displaydoc"
|
||||
version = "0.2.7"
|
||||
@@ -250,12 +303,16 @@ dependencies = [
|
||||
"anyhow",
|
||||
"chrono",
|
||||
"crossbeam",
|
||||
"ctrlc",
|
||||
"ergo-sbe",
|
||||
"indexmap",
|
||||
"rustls",
|
||||
"serde",
|
||||
"serde_json",
|
||||
"serde_yaml_ng",
|
||||
"sha1",
|
||||
"tungstenite",
|
||||
"ureq",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -296,6 +353,17 @@ version = "0.1.15"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "1c66915cc5391097de7a2b0b972e8505aa33c0df401fd3a6811dbdb32881ba48"
|
||||
|
||||
[[package]]
|
||||
name = "flate2"
|
||||
version = "1.1.10"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "6e634e2e0ebac1ee034020da1ca582e17ffe4e0f5e985823721e168928136dcb"
|
||||
dependencies = [
|
||||
"crc32fast",
|
||||
"miniz_oxide 0.9.1",
|
||||
"zlib-rs",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "futures-core"
|
||||
version = "0.3.34"
|
||||
@@ -595,6 +663,28 @@ dependencies = [
|
||||
"adler2",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "miniz_oxide"
|
||||
version = "0.9.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "b63fbc4a50860e98e7b2aa7804ded1db5cbc3aff9193adaff57a6931bf7c4b4c"
|
||||
dependencies = [
|
||||
"adler2",
|
||||
"simd-adler32",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "nix"
|
||||
version = "0.31.3"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "cf20d2fde8ff38632c426f1165ed7436270b44f199fc55284c38276f9db47c3d"
|
||||
dependencies = [
|
||||
"bitflags",
|
||||
"cfg-if",
|
||||
"cfg_aliases",
|
||||
"libc",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "num-traits"
|
||||
version = "0.2.19"
|
||||
@@ -604,6 +694,21 @@ dependencies = [
|
||||
"autocfg",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "objc2"
|
||||
version = "0.6.5"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "08849bbd4767dfae9457696856ae1c84fe4e0281bbe4a7abff2d0e06fb7981f8"
|
||||
dependencies = [
|
||||
"objc2-encode",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "objc2-encode"
|
||||
version = "4.1.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "ef25abbcd74fb2609453eb695bd2f860d389e457f67dc17cafc8b8cbc89d0c33"
|
||||
|
||||
[[package]]
|
||||
name = "object"
|
||||
version = "0.37.3"
|
||||
@@ -625,6 +730,12 @@ version = "4.4.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "13c45bb4a6ae1280ec0803b1ef9d3455eb50f01efbbe1447ab020f1d54fba9d8"
|
||||
|
||||
[[package]]
|
||||
name = "percent-encoding"
|
||||
version = "2.3.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "9b4f627cb1b25917193a259e49bdad08f671f8d9708acfd5fe0a8c1455d87220"
|
||||
|
||||
[[package]]
|
||||
name = "pin-project-lite"
|
||||
version = "0.2.17"
|
||||
@@ -741,6 +852,7 @@ version = "0.23.46"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "48e13bd8c0e9365c43cfa5c9e8f9ad49d3c8444926c9aac819e0e4dc503c8fdf"
|
||||
dependencies = [
|
||||
"log",
|
||||
"once_cell",
|
||||
"ring",
|
||||
"rustls-pki-types",
|
||||
@@ -811,6 +923,19 @@ dependencies = [
|
||||
"syn 3.0.9",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "serde_json"
|
||||
version = "1.0.154"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "e7e9cc8b1b85264074fbcc02a88680c4096b1e47df8f739dceb03bf482f04bd6"
|
||||
dependencies = [
|
||||
"itoa",
|
||||
"memchr",
|
||||
"serde",
|
||||
"serde_core",
|
||||
"zmij",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "serde_yaml_ng"
|
||||
version = "0.9.36"
|
||||
@@ -852,6 +977,12 @@ version = "2.0.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f8fadd59c855ef2080decdef8ff161eb6661b86933c9d82e5ba29dc602a55aba"
|
||||
|
||||
[[package]]
|
||||
name = "simd-adler32"
|
||||
version = "0.3.10"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "3a219298ac11a56ea9a6d2120044824d6f01aeb034955e7af7bc16858527deea"
|
||||
|
||||
[[package]]
|
||||
name = "slab"
|
||||
version = "0.4.12"
|
||||
@@ -1036,6 +1167,41 @@ version = "0.9.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "8ecb6da28b8a351d773b68d5825ac39017e680750f980f3a1a85cd8dd28a47c1"
|
||||
|
||||
[[package]]
|
||||
name = "ureq"
|
||||
version = "3.5.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "e7be76686bdf7198002062423ca0d8cf9b75a27c8e1de804513b97e97b555d9d"
|
||||
dependencies = [
|
||||
"base64",
|
||||
"flate2",
|
||||
"log",
|
||||
"percent-encoding",
|
||||
"rustls",
|
||||
"rustls-pki-types",
|
||||
"ureq-proto",
|
||||
"utf8-zero",
|
||||
"webpki-roots 1.0.11",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "ureq-proto"
|
||||
version = "0.7.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "622dce33c252115466d83f63e9744a18f12b4406aae51a01c15ad8fcf1185eb3"
|
||||
dependencies = [
|
||||
"base64",
|
||||
"http",
|
||||
"httparse",
|
||||
"log",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "utf8-zero"
|
||||
version = "0.8.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "b8c0a043c9540bae7c578c88f91dda8bd82e59ae27c21baca69c8b191aaf5a6e"
|
||||
|
||||
[[package]]
|
||||
name = "utf8_iter"
|
||||
version = "1.0.4"
|
||||
@@ -1340,3 +1506,15 @@ dependencies = [
|
||||
"quote",
|
||||
"syn 3.0.9",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "zlib-rs"
|
||||
version = "0.6.8"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "b268e58e7c693d7c271f93ffc4ba3b380412554231c85bf61ca7af91042a4112"
|
||||
|
||||
[[package]]
|
||||
name = "zmij"
|
||||
version = "1.0.23"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "29666d0abbfad1e3dc4dcf6144730dd3a3ab225bbbdac83319345b1b44ccfc1b"
|
||||
@@ -3,16 +3,22 @@ name = "engineering-challenge"
|
||||
version = "0.1.0"
|
||||
edition = "2024"
|
||||
build = "src/util/build.rs"
|
||||
# `cargo run` runs the collector, not `src/bin/reconcile.rs`.
|
||||
default-run = "engineering-challenge"
|
||||
|
||||
[dependencies]
|
||||
anyhow = "1"
|
||||
chrono = "0.4"
|
||||
crossbeam = "0.8.4"
|
||||
ctrlc = "3"
|
||||
# Not used directly: tungstenite leaves rustls's crypto provider to the app, and
|
||||
# rustls panics at runtime unless exactly one (here ring) is enabled.
|
||||
rustls = { version = "0.23", default-features = false, features = ["ring"] }
|
||||
serde = { version = "1", features = ["derive"] }
|
||||
serde_json = "1"
|
||||
sha1 = "0.11"
|
||||
tungstenite = { version = "0.30", features = ["rustls-tls-webpki-roots"] }
|
||||
ureq = "3"
|
||||
|
||||
[build-dependencies]
|
||||
ergo-sbe = "=0.1.16"
|
||||
|
||||
+58
@@ -0,0 +1,58 @@
|
||||
# Order-book collector
|
||||
|
||||
The collector's language: what it tracks on each exchange and what it writes
|
||||
for the backtests that read its output.
|
||||
|
||||
## Exchange side
|
||||
|
||||
**Market**:
|
||||
A tradable question on an exchange. On Polymarket it has two outcomes (Yes,
|
||||
No), each traded as its own symbol, and their books mirror each other.
|
||||
_Avoid_: Event, contract
|
||||
|
||||
**Symbol**:
|
||||
The unit that has one order book: on Polymarket a token (one outcome of a
|
||||
market), named by its token id.
|
||||
_Avoid_: Asset, instrument, ticker (for Polymarket)
|
||||
|
||||
**Book hash**:
|
||||
The exchange's fingerprint of a symbol's full book state, sent with every
|
||||
full book and every change.
|
||||
|
||||
## Collector side
|
||||
|
||||
**Book**:
|
||||
The collector's in-memory order book for one symbol: its bid and ask price
|
||||
levels, each with its absolute quantity.
|
||||
_Avoid_: Order book state, depth
|
||||
|
||||
**Receive time**:
|
||||
When a message came off the socket, by the local clock.
|
||||
_Avoid_: Arrival time, local timestamp
|
||||
|
||||
**Loss**:
|
||||
Any message for a symbol that the collector did not apply, which leaves its
|
||||
book silently wrong.
|
||||
_Avoid_: Gap (implies sequence numbers)
|
||||
|
||||
**Resync**:
|
||||
Asking the exchange for a fresh full book of a symbol and replacing ours with
|
||||
it, whenever our book can't be verified. Our book was wrong (a loss) only if
|
||||
the fresh one differs.
|
||||
_Avoid_: Re-snapshot, rebuild, refresh
|
||||
|
||||
**Resync gap**:
|
||||
The changes a symbol's resync itself keeps from us: the exchange holds them in
|
||||
the fresh book but never sends them as changes.
|
||||
_Avoid_: calling it a loss (the state stays right; only those updates are missing)
|
||||
|
||||
## Output
|
||||
|
||||
**Snapshot**:
|
||||
The output message holding one symbol's whole book at a point in time.
|
||||
_Avoid_: using it for the action of fetching a fresh book (that is a resync)
|
||||
|
||||
**Delta**:
|
||||
The output message changing one price level of one symbol's book to a new
|
||||
absolute quantity (`PriceLevelUpdate` in the schema).
|
||||
_Avoid_: Increment, diff
|
||||
+1
-1
@@ -25,7 +25,7 @@ The two claims split apart:
|
||||
|
||||
- **Completeness is cleanly measurable on recorded data.** Treat our output as
|
||||
ground truth, make a lossy copy (drop a % of messages, skip some
|
||||
re-snapshots), run the same strategy on both, compare P&L. The gap = cost of
|
||||
resyncs), run the same strategy on both, compare P&L. The gap = cost of
|
||||
dropped data. No live trading needed.
|
||||
- **Latency is NOT measurable from recorded data.** Collection latency is baked
|
||||
into a recording; a backtest can't show "we'd have been a few ms faster." A
|
||||
|
||||
+57
-16
@@ -6,12 +6,13 @@ A delta carries the **new total quantity** at a price level, not a change to it.
|
||||
never add. `quantity: 0` means delete the level. (Schema confirms: "absolute
|
||||
available quantity".)
|
||||
|
||||
## A missed message means a wrong book: rebuild it
|
||||
## A missed message means a wrong book: resync it
|
||||
If a message is lost, the in-memory book is silently wrong and cannot be
|
||||
patched — throw it away and re-snapshot that symbol. This is the
|
||||
patched — throw it away and resync that symbol from a fresh full book. This is the
|
||||
"completeness" requirement made mechanical. *How* a loss is detected depends on
|
||||
the exchange: a sequence-number gap where one exists (Kalshi, to verify in M6),
|
||||
something else where it doesn't — Polymarket has no sequence number (below).
|
||||
something else where it doesn't — Polymarket has no sequence number, but its
|
||||
book hash does the job (below).
|
||||
The challenge brief prescribes no mechanism; it only asks for complete data.
|
||||
|
||||
## Polymarket market channel (observed 2026-11-04)
|
||||
@@ -40,33 +41,73 @@ From a 60 s run of the M1 spike on one market, both tokens subscribed, and the
|
||||
then a `last_trade_price` — a natural resync point.
|
||||
- **`book`** (snapshot): `event_type`, `market`, `asset_id`, `timestamp`,
|
||||
`hash`, `bids`, `asks`, `tick_size`, `last_trade_price`. `bids`/`asks` are lists
|
||||
of `{price, size}` sorted **ascending by price**, so the best bid is the
|
||||
*last* bid. Its `timestamp` is the time of the book's **last change**, not the
|
||||
send time: 19 s old on receipt.
|
||||
of `{price, size}`, bids sorted **ascending** and asks **descending** by
|
||||
price, so the best level of each side comes *last*. Its `timestamp` is the
|
||||
time of the book's **last change**, not the send time: 19 s old on receipt.
|
||||
Books sent mid-stream omit `tick_size` and `last_trade_price`.
|
||||
- **`price_change`** (delta): `event_type`, `market`, `timestamp`,
|
||||
`price_changes[]`, each entry `{asset_id, price, size, side, hash, best_bid,
|
||||
best_ask}`. `side` is `BUY` (bid) or `SELL` (ask); `size` is the absolute
|
||||
level size (docs: `"0"` removes the level).
|
||||
- **Numbers are decimal strings** (`"0.77"`, `"26350.28"`): parse them exactly
|
||||
into fixed point (`1_000_000` = 1), never through a float.
|
||||
into fixed point (`1_000_000` = 1), never through a float. Prices and sizes
|
||||
are canonical (no trailing zeros: every one in 40 REST books, 2026-11-04);
|
||||
`last_trade_price` is not (`"0.080"`).
|
||||
- **Timestamps are milliseconds** since the epoch (the schema stores
|
||||
microseconds).
|
||||
- **No sequence number**, confirmed by the docs: none, and no option enables
|
||||
one, for the market channel (the `seq` in Polymarket's docs belongs to
|
||||
PolyBolt, a separate price feed). The docs describe no gap detection or
|
||||
resync either. Candidates for loss detection, to settle in M2:
|
||||
- the `hash` (40 hex chars, SHA-1 length) on every book and every change. A
|
||||
change's `hash` equals that of the `book` sent right after it for the same
|
||||
token, so it identifies the **full book state after the change**: a full
|
||||
check if we can reproduce it from our own book. The docs don't say how it is
|
||||
computed (M2's opening question);
|
||||
- `best_bid`/`best_ask` on every change — a cheap top-of-book check after
|
||||
applying it.
|
||||
resync either.
|
||||
- **The book hash identifies the full book state, and we can reproduce it**
|
||||
(found 2026-11-04; the docs don't describe it). It is the SHA-1, as 40 lowercase
|
||||
hex chars, of this compact JSON (no spaces), keys in this order:
|
||||
`{"market","asset_id","timestamp","hash":"","bids","asks","min_order_size","tick_size","neg_risk","last_trade_price"}`
|
||||
— the REST `/book` response with `hash` emptied. Bids ascending, asks
|
||||
descending, prices and sizes as the exchange writes them. On a change, the
|
||||
`timestamp` is the change message's. `min_order_size` (`"5"`) and `neg_risk`
|
||||
(`true`) are not on the websocket: Gamma's `orderMinSize` / `negRisk` give
|
||||
them (matched REST in 40 of 40 markets). Checked live: 14 books and 182
|
||||
changes on 6 tokens, every one matched after folding the changes into our
|
||||
own book. A change's `hash` therefore also equals that of the next `book`.
|
||||
- **One update can span several frames** (seen in a 90 s recording of 40
|
||||
tokens): `price_change` frames with the same `timestamp`, one change each
|
||||
for the same token, all carrying the hash of the book after the *whole*
|
||||
batch. Check a hash once its batch is over.
|
||||
- **`last_trade_price` belongs to the market**: both tokens of a market hash
|
||||
the same value (20 of 20 markets). A trade changes it, and the hashes use the
|
||||
new value up to ~680 ms before the `last_trade_price` event arrives; that
|
||||
event names one token, and the hashed value was its `price` (16 cases), its
|
||||
complement 1 − `price` (8 cases), or neither (1 case, a trade that probably
|
||||
filled at two prices). So a hash check fails after most trades with nothing
|
||||
lost; a fresh book (from a resync) carries the right value.
|
||||
- **Resync on the open connection:** `unsubscribe` then `subscribe` for a token
|
||||
makes the server send its full `book` at once. That book can lag the changes
|
||||
already sent by a few ms: in a 2-minute run, 14 of 14 fresh books that
|
||||
differed from ours had the hash of one of our own recent changes, i.e. a
|
||||
state we had already passed. Its timestamp and `last_trade_price` can be
|
||||
newer than its levels: in a 15-minute run, 16 of 27 fresh books that
|
||||
differed from ours equalled an earlier state of ours by levels, while their
|
||||
hash matched none of our changes (two of them with a timestamp ending in
|
||||
`.000`, as if cut to the second). So recognise a lagging book by its levels.
|
||||
It can also be *ahead*: between handling the
|
||||
`unsubscribe` and the `subscribe`, the server sends no changes for the
|
||||
token, so the book holds changes we never receive (6 of 302 resyncs in a
|
||||
10-minute run, 1 to 48 ms ahead of our last change; 39 others were behind). A `subscribe` for a token
|
||||
already subscribed is answered with the plain text `NO NEW ASSETS` (not
|
||||
JSON).
|
||||
- **Keepalive (docs):** send the text frame `PING` every 10 s; the server
|
||||
answers `PONG`. The spike survived 60 s without it, but it is required.
|
||||
- **Server restarts:** the server can close the socket with code `Away`,
|
||||
reason `server shutting down` (seen once in about 8 minutes of connections).
|
||||
- **`last_trade_price`:** `asset_id`, `price`, `size`, `side`, `fee_rate_bps`,
|
||||
`timestamp`, `transaction_hash` (one per trade, for one token).
|
||||
- **Not observed:** `tick_size_change` and the custom-feature events.
|
||||
- **Not observed:** `tick_size_change` and the custom-feature events. The docs
|
||||
now show these only in an SDK's wrapped form (`topic` / `payload`,
|
||||
camelCase), not the raw wire shape above.
|
||||
- **REST from scripts:** `clob.polymarket.com` answers Python's `urllib` with
|
||||
403 (curl and `ureq` get 200), most likely Cloudflare filtering by user agent.
|
||||
Fetch with curl in throwaway scripts.
|
||||
|
||||
## Clock accuracy bounds time accuracy
|
||||
`received_at` comes from the local clock, so it carries the clock's error. On
|
||||
|
||||
+17
-16
@@ -18,7 +18,7 @@ A running service that, for a chosen exchange:
|
||||
(subscribe to new, unsubscribe from gone);
|
||||
- maintains a correct, complete, time-accurate order book per symbol;
|
||||
- emits SBE `PriceLevelUpdate` deltas continuously and a full `Snapshot` per
|
||||
symbol every minute, on subscribe, and after any re-snapshot;
|
||||
symbol every minute, on subscribe, and after any resync;
|
||||
- shuts down cleanly on SIGINT (close sockets, flush/close output).
|
||||
|
||||
Full target (all four extensions): redundant websockets, subscription
|
||||
@@ -61,11 +61,11 @@ must guarantee:
|
||||
1. **Receive time stamped at the socket edge**, before any queue or processing.
|
||||
2. **A per-symbol book folded from absolute deltas** (overwrite the level;
|
||||
quantity 0 deletes it).
|
||||
3. **Loss detection → discard-and-re-snapshot** (a lost message means the book
|
||||
is silently corrupt; rebuilding is the only safe response). How a loss is
|
||||
detected depends on the exchange: a sequence gap where one exists; Polymarket
|
||||
has none, so its `hash` or best bid/ask instead
|
||||
([notes.md](notes.md)) — settled in M2.
|
||||
3. **Loss detection → resync** (a lost message means the book is silently
|
||||
corrupt; replacing it with a fresh full book is the only safe response). How
|
||||
a loss is detected depends on the exchange: a sequence gap where one exists;
|
||||
on Polymarket, which has none, recomputing its book hash after every change
|
||||
(settled in M2; the formula is in [notes.md](notes.md)).
|
||||
|
||||
These are proven first, before redundancy or scale.
|
||||
|
||||
@@ -92,7 +92,7 @@ function, symbols shardable later) are also the simplest, clearest design.
|
||||
not.** A channel, a function boundary, or a `trait` with two real impls keeps
|
||||
the door open; the clever internals are what we refuse.
|
||||
- **Never simplified away, even in V1:** receive-time stamping at the socket
|
||||
edge, and loss → re-snapshot. These are correctness, not readability tradeoffs.
|
||||
edge, and loss → resync. These are correctness, not readability tradeoffs.
|
||||
- **If a corner is genuinely cut for speed later** (a global lock, an O(n²)
|
||||
scan), mark it with a comment naming the ceiling and the upgrade path, so the
|
||||
debt is visible rather than silently clever.
|
||||
@@ -109,10 +109,14 @@ Their operational form, for writing code, is in the project
|
||||
modules under `src/`. `website/` stays separate as the brief.
|
||||
- **Scope of markets:** regular markets only; MVEs excluded; websocket exchanges
|
||||
only.
|
||||
- **Output destination** (settled in sprint 01): `AlignedSbeWriter` takes a file
|
||||
path, and stdout is just the path `/dev/stdout` (or a tiny writer tweak), so
|
||||
one code path covers both. Chunks must be 64 MiB: `AlignedSbeReader::open`
|
||||
reads nothing else.
|
||||
- **Output destination** (settled in sprints 01 and 02): `AlignedSbeWriter`
|
||||
takes a file path; the program's only argument is that path, default
|
||||
`/dev/stdout`, and logs go to stderr. No rotation: one output per run
|
||||
(rotation can't apply to stdout; a consumer that wants files rotates them).
|
||||
Chunks must be 64 MiB: `AlignedSbeReader::open` reads nothing else.
|
||||
- **Crash ceiling** (accepted in sprint 02): the writer holds the current
|
||||
64 MiB chunk in memory until it is full, so a hard crash loses it; flushing
|
||||
more often pads each flush to 64 MiB. Taken up with extension 1 (M4).
|
||||
|
||||
## Workflow differences
|
||||
|
||||
@@ -175,14 +179,11 @@ trusted source is chosen and switched, and emitting a fresh snapshot on switch).
|
||||
|
||||
## Open questions
|
||||
|
||||
- **Loss detection per exchange.** Polymarket has no sequence number: can we
|
||||
reproduce its book `hash`, or do we check best bid/ask after each change? M2's
|
||||
opening question. Kalshi's sequence semantics: checked in M6.
|
||||
- **Loss detection on Kalshi.** Its sequence semantics: checked in M6.
|
||||
(Polymarket: settled in M2, by its book hash.)
|
||||
- **Symbol lifecycle signal.** What marks a market "no longer trading"?
|
||||
Polymarket's `new_market` / `market_resolved` events (with
|
||||
`custom_feature_enabled`) are candidates; settle in M3.
|
||||
- **File rotation policy.** When to roll to a new output file (size, time), given
|
||||
64 MiB chunks. Resolve in M2 with the output writer.
|
||||
- **Redundancy source-selection.** How to pick/switch the trusted source
|
||||
(freshness, message rate, which source is ahead). Designed in the redundancy
|
||||
sprint.
|
||||
+27
-9
@@ -20,14 +20,14 @@ abstraction is *extracted* once a second exchange exists, not designed up front.
|
||||
Connect to Polymarket and log raw messages.
|
||||
*Ends:* CI green; raw Polymarket messages flowing.
|
||||
|
||||
2. **Single-source book pipeline (Polymarket, subset of symbols)** — *not started*
|
||||
The central bet in its first real form: normalize events → per-symbol book
|
||||
(fold absolute deltas) → loss-detect → re-snapshot → SBE output (deltas +
|
||||
snapshot every minute, on subscribe, on re-snapshot) → receive-time stamping
|
||||
→ clean SIGINT, with the `PING` keepalive every 10 s. Polymarket has no
|
||||
sequence number, so loss detection (its book `hash`, or best bid/ask) is the
|
||||
opening question. One/few sockets, a fixed subset of symbols. Deterministic
|
||||
replay tests + snapshot↔delta reconciliation.
|
||||
2. **Single-source book pipeline (Polymarket, subset of symbols)** — *done (2026-11-04)*
|
||||
The central bet in its first real form: parse → per-symbol book (fold
|
||||
absolute deltas) → loss-detect → resync → SBE output (deltas + snapshot
|
||||
every minute, on subscribe, after a resync) → receive-time stamping → clean
|
||||
SIGINT, with the `PING` keepalive every 10 s. Loss detection: Polymarket's
|
||||
book hash, reproduced after every change. One socket, a fixed subset of
|
||||
symbols. Deterministic replay tests + snapshot↔delta reconciliation.
|
||||
Sprint: [../roadmap/02. book pipeline.md](../roadmap/02.%20book%20pipeline.md).
|
||||
*Ends:* correct SBE output for a symbol subset; tests green.
|
||||
|
||||
3. **Full coverage: discovery + sharding (base challenge complete)** — *not started*
|
||||
@@ -35,6 +35,15 @@ abstraction is *extracted* once a second exchange exists, not designed up front.
|
||||
Polymarket's `new_market` / `market_resolved` events — using subscribe /
|
||||
unsubscribe on open connections; load-balance all live regular-market symbols
|
||||
across N sockets.
|
||||
**Opening questions (from M2):** what "live" means, and so how many symbols:
|
||||
Gamma lists at least 100,000 markets `active` and not `closed` (2026-11-04,
|
||||
count capped), at least 13,000 of them traded in the last 24 h, against the
|
||||
40 symbols M2 runs. At that size the minute Snapshots alone (~2 KB each)
|
||||
would be ~0.5 TB a day for 200,000 symbols, mostly unchanged books; and
|
||||
memory (~10 KB a book), sockets per connection, resyncs per minute and the
|
||||
book thread's throughput (M2: ~30,000 frames/s on one thread, 43 frames/s
|
||||
of load) all need measuring. Gamma pages past 10,000 markets only through
|
||||
`/markets/keyset` with `after_cursor`.
|
||||
*Ends:* every live Polymarket symbol tracked — base challenge satisfied for
|
||||
one exchange.
|
||||
|
||||
@@ -44,7 +53,12 @@ abstraction is *extracted* once a second exchange exists, not designed up front.
|
||||
**Async re-evaluation checkpoint:** before building this, reassess whether the
|
||||
socket-management complexity (reconnect/timeout/select, redundancy) warrants
|
||||
swapping the thread-only runtime for tokio (see ADR 0001).
|
||||
*Ends:* redundancy demonstrably reduces data loss; source-switch re-snapshots.
|
||||
Also: surviving a hard crash without losing the writer's in-memory chunk,
|
||||
and closing the resync gap (the few ms of a symbol's changes the server
|
||||
doesn't send between our unsubscribe and subscribe), both accepted as
|
||||
ceilings in M2.
|
||||
*Ends:* redundancy demonstrably reduces data loss; a Snapshot on every source
|
||||
switch.
|
||||
|
||||
5. **Extension 2 — subscription monitoring** — *not started*
|
||||
Per-symbol activity monitoring (data points per second/minute); unsubscribe +
|
||||
@@ -92,3 +106,7 @@ abstraction is *extracted* once a second exchange exists, not designed up front.
|
||||
- Observability: metrics, structured logging, dashboards.
|
||||
- Config-driven symbol filtering and per-exchange tuning.
|
||||
- Containerized packaging.
|
||||
- Counting a reconnect's differing books as losses rather than resync gaps
|
||||
(a reconnect marker through the frames channel; M2 labels them gaps).
|
||||
- A `reconcile` that reads the collector's log, to tell resync gaps from
|
||||
losses in its own report.
|
||||
@@ -0,0 +1,407 @@
|
||||
# 02. Single-source book pipeline
|
||||
|
||||
**Status: done (2026-11-04)**
|
||||
|
||||
The central bet in its first real form, on one Polymarket socket and a fixed
|
||||
subset of symbols: stamp each frame at the socket, fold changes into a book per
|
||||
symbol, detect any loss through Polymarket's book hash and resync, and write
|
||||
Deltas plus Snapshots (on subscribe, every minute, after a resync) with the
|
||||
template's SBE writer, until SIGINT closes everything cleanly. Design:
|
||||
[../docs/project.md](../docs/project.md) (central bet, pipeline sketch),
|
||||
[../docs/notes.md](../docs/notes.md) (protocol facts),
|
||||
[../GLOSSARY.md](../GLOSSARY.md) (terms), ADR 0001 (threads, shared-nothing).
|
||||
|
||||
## Pipeline
|
||||
|
||||
Three threads, joined by unbounded `crossbeam` channels:
|
||||
|
||||
```
|
||||
socket thread ──(receive time, frame text)──▶ book thread ──(Snapshot | Delta)──▶ writer thread
|
||||
▲ │
|
||||
└──────────── resync requests ───────────────┘
|
||||
```
|
||||
|
||||
- **Socket thread**: connects, subscribes to every symbol, reads. The receive
|
||||
time is stamped right after `read()` returns, before anything else. The
|
||||
socket has a short read timeout (200 ms) so the loop wakes up to send `PING`
|
||||
every 10 s, to send the unsubscribe / subscribe pairs the book thread asks for,
|
||||
and to see the SIGINT flag. It doesn't parse frames.
|
||||
- **Book thread**: parses frames, folds them into books, checks hashes, decides
|
||||
resyncs, emits a Delta per applied level change and Snapshots. It waits on
|
||||
the channel with a timeout so it also wakes for the minute Snapshots.
|
||||
- **Writer thread**: owns the `AlignedSbeWriter` (64 MiB chunks) and writes what
|
||||
it receives, in order.
|
||||
|
||||
Why the read timeout and not a fourth thread for writes: tungstenite 0.30 has no
|
||||
`split`, and `read()` takes `&mut self` and itself writes (pong replies, the
|
||||
close handshake), so read and write can't live on two threads without a mutex.
|
||||
The timeout was tried live: 1,081 timeouts in 4 minutes, every frame intact,
|
||||
23 PINGs answered by 23 PONGs.
|
||||
|
||||
## Data model
|
||||
|
||||
A **symbol** is a Polymarket token, named in the output by its token id; both
|
||||
tokens of a market are tracked, as two symbols. Per symbol, the book thread
|
||||
keeps:
|
||||
|
||||
- `bids` and `asks`: `BTreeMap<i64, i64>`, price → quantity in fixed point
|
||||
(1,000,000 = 1), parsed exactly from the decimal strings, never through a
|
||||
float. More than 6 decimals is a protocol error for that symbol: it resyncs.
|
||||
- the other inputs of the book hash, as the exchange's exact strings:
|
||||
`market`, `tick_size`, `last_trade_price`, `min_order_size`, `neg_risk`.
|
||||
- the last exchange timestamp (a Snapshot's `exchange_timestamp`).
|
||||
- a state: `AwaitingBook` (no book yet since subscribing: changes are
|
||||
dropped), `Live` (every change checked), `Resyncing` (a fresh book is
|
||||
asked for: changes are applied and written unchecked until it comes), or
|
||||
`Unverifiable` (see below).
|
||||
- the hash of a batch not yet checked, and the last 64 changes with their
|
||||
hashes (see below).
|
||||
|
||||
The subset: at startup, Gamma's `/markets` top 20 by 24 h volume (`active`,
|
||||
`closed=false`, order book enabled), which gives the token ids
|
||||
(`clobTokenIds`, a JSON array inside a string), `orderMinSize` and `negRisk`.
|
||||
40 symbols (fewer if one of the 20 has no order book). Fixed for the run;
|
||||
discovery is M3.
|
||||
|
||||
Output fields: a Delta's `backtest_timestamp` and a Snapshot's `received_at`
|
||||
are both the receive time. A minute Snapshot carries the receive time of the
|
||||
last frame the book thread applied, not the wall clock: the book is exact as of
|
||||
that frame, while frames still queued aren't in it yet.
|
||||
|
||||
No `NormalizedEvent` yet: Polymarket's messages parse into Polymarket types,
|
||||
the book code knows nothing of Polymarket, and the hash check is Polymarket
|
||||
code. The shared seam is extracted in M6 from two exchanges.
|
||||
|
||||
## Loss detection and resync
|
||||
|
||||
Checked live on 6 symbols for 2 minutes (14 books, 182 changes, all matched):
|
||||
the book hash is the SHA-1 of the compact JSON
|
||||
|
||||
```
|
||||
{"market","asset_id","timestamp","hash":"","bids","asks","min_order_size","tick_size","neg_risk","last_trade_price"}
|
||||
```
|
||||
|
||||
in that key order, with bids ascending and asks descending by price, prices and
|
||||
sizes as canonical decimals (true of every price and size in 40 books checked),
|
||||
and, for a change, the change message's `timestamp`. Full books sent
|
||||
mid-stream omit `tick_size` and `last_trade_price`: the last known values are
|
||||
used.
|
||||
|
||||
Two things the 2-minute check didn't show, found in the 90 s test recording
|
||||
(40 symbols, 3,850 changes; `docs/notes.md`):
|
||||
|
||||
- **Batches**: one update can arrive split over several frames, all carrying
|
||||
the hash of the book after the whole batch. A change's hash is therefore
|
||||
checked when its batch is over: right away if it matches, else when the
|
||||
symbol's next change carries a different hash (before applying it).
|
||||
- **The last trade price can't be known in time**: trades change it, the hash
|
||||
picks up the new value up to ~680 ms before the `last_trade_price` event,
|
||||
and that event gives it, its complement (1 − p), or neither. So after a
|
||||
trade the check fails although nothing was lost; only a fresh book (which
|
||||
carries the last trade price) tells the two apart.
|
||||
|
||||
The rules:
|
||||
|
||||
1. **A complete full book checks itself** (one that carries
|
||||
`last_trade_price`: on subscribe and after a resync): hash it as received.
|
||||
A mismatch means our formula or metadata is wrong for that symbol, not a
|
||||
loss: log it loudly, mark the symbol `Unverifiable`, and keep writing its
|
||||
Deltas, unchecked. No resync, which would loop forever.
|
||||
2. **A failed check asks for a fresh book** (a change, after its batch, or a
|
||||
mid-stream book, whose last trade price may be stale): the symbol goes
|
||||
`Resyncing`, the book thread asks the socket thread to unsubscribe and
|
||||
subscribe it again (tried live: the fresh book arrives at once), and its
|
||||
changes keep being applied and written, unchecked, until that book arrives.
|
||||
A symbol still `Resyncing` at a wall-clock minute is asked for again, so a
|
||||
lost answer doesn't leave it unchecked for the rest of the run. A full book
|
||||
that can't be read is a protocol error: the symbol resyncs too (it keeps
|
||||
`AwaitingBook` if it had no book yet).
|
||||
3. **A full book replaces ours** (on subscribe, mid-stream after trades, after
|
||||
a resync or a reconnect). It is written as a Snapshot if it is the symbol's
|
||||
first, or if its levels differ from ours: that is a **loss**, logged and
|
||||
counted. A fresh book with the same levels (the usual case after a trade)
|
||||
writes nothing. Two timings of a resync's fresh book need care:
|
||||
- **Behind our stream**: it can lag the changes already sent by a few ms
|
||||
(live: 14 of 14 differing books in 2 minutes), and its timestamp or last
|
||||
trade price can be newer than its levels, so its hash may match none of
|
||||
our changes. Checked in order, against the symbol's last 64 changes:
|
||||
1. if it equals our book as it was after one of them (ours rewound,
|
||||
each change undone back to the quantity it replaced), it is behind:
|
||||
ours is kept, its tick size and last trade price are taken, nothing
|
||||
is written. That past state of ours was the exchange's own, and the
|
||||
changes since are what it sent us. (Comparing the book *plus our
|
||||
later changes* with ours instead would be unsound: a change we lost
|
||||
at a price we had changed before is overwritten by that earlier
|
||||
change, and the wrong book passes.)
|
||||
2. else, if its hash is that of one of our changes (the latest
|
||||
included), it is the exchange's state after that change and ours is
|
||||
wrong (a loss): the changes received since are applied to it, and it
|
||||
replaces ours.
|
||||
- **Ahead of our stream (a resync gap)**: between the server handling our
|
||||
`unsubscribe` and our `subscribe`, it sends no changes for that symbol,
|
||||
so the fresh book holds changes we never got. A fresh book answering
|
||||
our resync that isn't behind, whose levels differ and whose exchange
|
||||
time is at or after our last change, is counted as a **resync gap**, not
|
||||
a loss: its Snapshot corrects the
|
||||
state, the Deltas inside the gap are lost. M4's redundant sockets can
|
||||
close it.
|
||||
4. **`tick_size_change`** resyncs the symbol: the fresh book carries the new
|
||||
tick size. Its wire shape was never observed (the docs now show an SDK's
|
||||
wrapped form), and this needs only `event_type` and `asset_id`.
|
||||
5. **`last_trade_price` events are ignored**: the hash takes the new price
|
||||
before the event and not always the event's value; the fresh book after
|
||||
the failed check carries the right one. Trades aren't in the schema.
|
||||
|
||||
`PONG` and `NO NEW ASSETS` (the server's answer to a subscribe for a symbol
|
||||
already subscribed) are skipped, as are unknown event types; any other frame
|
||||
that isn't readable JSON is logged and skipped.
|
||||
|
||||
## Output, shutdown and reconnect
|
||||
|
||||
- **Destination**: the program's only argument is the output path, default
|
||||
`/dev/stdout`. Logs go to stderr. No rotation: one output per run.
|
||||
- **Minute Snapshots**: every symbol with a book (not `AwaitingBook`), at each
|
||||
wall-clock minute.
|
||||
- **SIGINT**: `ctrlc` sets a flag; the socket thread sees it within 200 ms
|
||||
when connected, within 10 s while connecting (the TCP connect and the TLS
|
||||
handshake have a 10 s timeout), and 1 s while waiting to reconnect. It
|
||||
closes the websocket and drops its channel; the book thread drains and drops
|
||||
its own; the writer drains, flushes the last chunk and closes the file. Each
|
||||
stage stops because the previous one did.
|
||||
- **Reconnect**: when the server closes the socket or it errors, the socket
|
||||
thread waits 1 s, connects again and subscribes every symbol; the fresh
|
||||
books then go through rule 3. Seen once in about 8 minutes of test
|
||||
connections: a close with "server shutting down". Ceiling: a symbol that
|
||||
was `Resyncing` when the connection dropped has its reconnect book counted
|
||||
as a resync gap rather than a loss (the state and output are right either
|
||||
way; only the count is mislabelled, next to the logged reconnect).
|
||||
- **Crash ceiling**: `AlignedSbeWriter` keeps the current 64 MiB chunk in memory
|
||||
and writes it only when full (`src/util/aligned_sbe_writer.rs:48`) or on
|
||||
`flush`, which pads it to 64 MiB with zeros (`:68-70`). A hard crash loses
|
||||
the unwritten chunk, hours of data at 40 symbols; flushing every minute would
|
||||
write ~90 GB a day of padding. Accepted for M2; the brief lists "we go down
|
||||
unexpectedly" under extension 1, so M4 takes it up.
|
||||
|
||||
## Libraries
|
||||
|
||||
New: `serde_json` (parsing), `ureq` 3 (Gamma; tried: it reaches Gamma and the
|
||||
CLOB, and its TLS uses `ring`, already locked, with no `aws-lc-rs`), `sha1` 0.11
|
||||
(RustCrypto), `ctrlc` 3.
|
||||
|
||||
## Decisions
|
||||
|
||||
- **Loss detection by the book hash, after every change.** It identifies the
|
||||
full book state, so a check after each change catches any loss at once.
|
||||
*Rejected:* comparing best bid / ask, which misses losses deeper in the book.
|
||||
- **Resync by unsubscribe + subscribe on the open socket.** The fresh book
|
||||
arrives at once, on the stream we're already reading. *Rejected:* REST
|
||||
`/book` (a second path whose timing relative to the stream is unknown).
|
||||
- **A failed check asks for a fresh book but keeps applying changes; a loss is
|
||||
a fresh book that differs** (supersedes "a mismatch is a loss, changes are
|
||||
dropped until the fresh book", because the recording showed checks failing
|
||||
after trades with nothing lost: the hash includes a last trade price we
|
||||
can't know in time). After a trade it costs one book on the socket and
|
||||
nothing in the output; after a real loss the fresh book's Snapshot corrects
|
||||
the output. *Rejected:* guessing the last trade price from the trade event
|
||||
(its value, its complement, the batch's prices), a heuristic over
|
||||
undocumented behaviour that already missed a case in 90 s; dropping changes
|
||||
and writing a Snapshot on every trade; checking only best bid / ask, which
|
||||
misses deeper losses.
|
||||
- **A full book whose hash we saw on a recent change is brought forward with
|
||||
the changes received since, then compared** (supersedes "a book whose
|
||||
levels differ is a loss", because live, fresh books lagged our stream by a
|
||||
few ms; and supersedes "keep ours", agreed in between, because our book may
|
||||
be the wrong one). The hash identifies the full state, so the test is
|
||||
exact. *Rejected:* comparing exchange timestamps (ties at the same
|
||||
millisecond were common); REST `/book` (may lag the same way, and is a
|
||||
second path); keeping our book.
|
||||
- **"Behind" is recognised by levels first, by hash second** (supersedes the
|
||||
hash match alone, because live, fresh books were behind by levels while
|
||||
carrying a newer timestamp or last trade price: both losses of a 15-minute
|
||||
run and 14 books counted as resync gaps, which then replaced our newer book
|
||||
with a stale one). The hash match stays second: when our book is the wrong
|
||||
one, it still brings the fresh book forward to the right state at once.
|
||||
The level test rewinds ours (supersedes "the book plus our later changes
|
||||
equals ours", found unsound at the close review: a lost change at a price
|
||||
we had changed before is masked). *Rejected:* levels only (a loss would
|
||||
first write the stale book); hashes only (the misclassifications above).
|
||||
- **A requested resync is asked again at each minute while unanswered; the
|
||||
connect and TLS handshake time out after 10 s** (both found at the close
|
||||
review: a lost answer left a symbol unchecked for the rest of the run, and
|
||||
SIGINT could hang on a stalled connect). *Rejected:* a per-symbol resync
|
||||
timer (more state for the same effect at this scale).
|
||||
- **A resync gap is counted apart from losses and accepted in M2.** Our own
|
||||
unsubscribe / subscribe opens a few ms in which the server sends us no
|
||||
changes for the symbol; the fresh book's Snapshot keeps the state right.
|
||||
*Rejected:* resyncing through a short-lived second connection (M4's
|
||||
machinery, and its book could be ahead of our stream, a case not handled);
|
||||
fewer resyncs by guessing the last trade price (rejected above).
|
||||
- **A change's hash is checked when its batch is over.** One update can span
|
||||
frames that all carry its final hash. *Rejected:* checking after each
|
||||
change, which fails mid-batch.
|
||||
- **A self-check on full books, `Unverifiable` on failure.** Separates "our
|
||||
formula is wrong" from "we lost a message". *Rejected:* resyncing on any
|
||||
mismatch, which loops forever on a formula error.
|
||||
- **Levels as fixed-point integers, other hash inputs as exact strings.** The
|
||||
output needs integers, and integer keys make `"0.5"` and `"0.50"` one level;
|
||||
`last_trade_price` is not canonical (`"0.080"` seen), so it stays a string.
|
||||
*Rejected:* keying levels by the exchange's strings.
|
||||
- **Both tokens of a market are symbols.** The brief asks for a book per symbol,
|
||||
and consumers shouldn't need to know Polymarket mirrors them. *Rejected:* one
|
||||
token per market (half the output).
|
||||
- **Three threads, socket / book / writer.** The socket thread only reads and
|
||||
stamps; the writer survives a book thread crash and still flushes, and its
|
||||
64 MiB writes don't pause the book thread; M3 needs it anyway. *Rejected:*
|
||||
two threads (book thread owning the writer), and four (no `split` in
|
||||
tungstenite without a mutex).
|
||||
- **Unbounded channels.** Blocking stalls the socket read and hurts time
|
||||
accuracy; dropping breaks completeness. Their ceiling (memory, if a consumer
|
||||
stays behind) is named in a comment, and the book thread's minute log
|
||||
reports the most frames that waited for it that minute (supersedes "logs the
|
||||
queue length past 10,000", whose check could only run when the queue was
|
||||
empty). Switching to bounded is a constructor change. *Rejected:* bounded,
|
||||
for now.
|
||||
- **Subset from Gamma at startup.** A hardcoded list goes stale in days (M1's
|
||||
did), and M3 needs the HTTP client anyway. *Rejected:* hardcoded token ids.
|
||||
- **Simple reconnect in M2.** A collector that dies at each server restart
|
||||
isn't one, and rule 3 already handles the fresh books. *Rejected:* exit on
|
||||
disconnect until M4, which keeps redundancy.
|
||||
- **Output to a path, default stdout, no rotation.** The brief says stdout;
|
||||
rotation can't apply to it. *Rejected:* rotating files by size or time.
|
||||
- **`sha1` (RustCrypto).** The usual choice, and faster: 1.8 µs per book
|
||||
against 4.6 µs for `sha1_smol` (measured, ~5 KB). *Rejected:* `sha1_smol`,
|
||||
`ring`'s SHA-1.
|
||||
|
||||
## Tests
|
||||
|
||||
No network in tests. A fixture of a few minutes of real frames for the subset,
|
||||
each with its receive time, plus the Gamma metadata the hash needs, recorded
|
||||
once by a throwaway script and committed (a few MB at most; its size is
|
||||
checked when recorded).
|
||||
|
||||
- **Replay**: the fixture replays with no loss and no `Unverifiable` symbol,
|
||||
split batches and trades included.
|
||||
- **Induced loss**: the replay with one frame dropped: a resync of that
|
||||
symbol is requested where the full replay requests none, and a fresh book
|
||||
that differs is written as a Snapshot and counted as a loss.
|
||||
- **Self-check**: a full book with wrong metadata marks the symbol
|
||||
`Unverifiable` and requests no resync.
|
||||
- **Reconciliation**: book and writer threads run on the fixture into a temp
|
||||
file (`std::env::temp_dir()`), read back with `AlignedSbeReader`; per symbol,
|
||||
the Deltas applied to a Snapshot equal the next Snapshot.
|
||||
- **Parsing**: fixed-point edges (`"0"`, integers, 6 decimals, more than 6,
|
||||
empty sides), non-JSON frames.
|
||||
|
||||
## Not in this sprint
|
||||
|
||||
- Discovery, the full symbol set, several sockets (M3).
|
||||
- Redundant sockets, and losing nothing during a reconnect or a resync gap
|
||||
(M4).
|
||||
- Activity monitoring, socket recycling (M5).
|
||||
- Surviving a hard crash without losing the in-memory chunk (M4).
|
||||
- Kalshi, the generic core (M6); latency work (M7).
|
||||
|
||||
## Done when
|
||||
|
||||
- `scripts/ci.sh` is green.
|
||||
- A 10-minute live run on the top 20 markets, ended by SIGINT, leaves an output
|
||||
that reads back with `AlignedSbeReader` and passes the reconciliation check.
|
||||
- That run's log shows no `Unverifiable` symbol, and every loss explained
|
||||
(e.g. by a reconnect); resync gaps are counted. The reconciliation's
|
||||
mismatches are exactly those losses and gaps.
|
||||
- `docs/notes.md` corrected (asks descending) and completed (the hash formula,
|
||||
resync on the open socket, `NO NEW ASSETS`, mid-stream books' missing fields,
|
||||
the server's restart close); `docs/project.md` records loss detection, output
|
||||
destination and rotation as settled.
|
||||
|
||||
## Outcome
|
||||
|
||||
**Built.** Three threads joined by unbounded channels:
|
||||
- **Socket thread:** connects with a 10 s timeout, stamps each frame's receive
|
||||
time right after the read, sends `PING`, resync requests and the close on
|
||||
SIGINT, and reconnects after 1 s.
|
||||
- **Book thread** (`src/polymarket/feed.rs`): parses, folds into a `Book` per
|
||||
symbol, checks Polymarket's book hash once each batch is over, resyncs on a
|
||||
failed check, and writes Deltas and Snapshots (on subscribe, at each
|
||||
wall-clock minute, when a fresh book changes our state).
|
||||
- **Writer thread:** owns the template's `AlignedSbeWriter`.
|
||||
|
||||
Also: a `reconcile` binary that checks an output against itself; 16 tests on a
|
||||
90 s recording of the live feed (replay, induced loss, self-check, books
|
||||
behind and ahead of our stream, the reconciliation through the real book and
|
||||
writer threads, parsing). The final 10-minute live run on the top 20 markets
|
||||
(40 symbols), ended by SIGINT: 97,763 changes verified, 266 resyncs, 29 fresh
|
||||
books behind ours, 4 resync gaps, 0 losses, no `Unverifiable` symbol. Its 4
|
||||
reconciliation mismatches are exactly the 4 logged gaps, and at most 20
|
||||
frames waited for the book thread in any minute. CI is green.
|
||||
|
||||
**Diverged from the plan, and why.** The plan's loss detection rested on a
|
||||
2-minute check of 6 symbols. Larger recordings and live runs showed the feed
|
||||
breaks that model in four ways, each settled with the user and recorded in
|
||||
Decisions:
|
||||
1. **Batches.** One update can span several frames carrying the same hash, so
|
||||
a hash is checked once its batch is over.
|
||||
2. **The last trade price.** It is a market-wide hash input that changes
|
||||
before its event, sometimes as the event's complement. A failed check
|
||||
therefore asks for a fresh book and keeps writing changes ("soft resync"),
|
||||
rather than calling it a loss.
|
||||
3. **Fresh books can lag our stream.** They are recognised by levels (ours
|
||||
rewound), then by hash.
|
||||
4. **Fresh books can lead our stream.** Our own unsubscribe/subscribe hides a
|
||||
few ms of changes: a **resync gap**, counted apart from losses and
|
||||
accepted for M2.
|
||||
|
||||
Two more changes came out of the close review:
|
||||
- **A soundness hole.** "The book plus our later changes" masked a change lost
|
||||
at a level we had changed before, so the level test now rewinds ours instead.
|
||||
- **Resyncs no longer wait forever.** An unanswered or unreadable fresh book is
|
||||
asked for again at each minute.
|
||||
|
||||
Smaller changes:
|
||||
- `book.rs` moved into the library, so `reconcile` uses `Book`.
|
||||
- The queue warning, which could only fire when the queue was empty, became a
|
||||
per-minute peak in the stats line.
|
||||
- `last_trade_price` events are ignored, as their value isn't reliable.
|
||||
|
||||
**Left out, and where it went:**
|
||||
- **Resync gaps and the hard-crash chunk loss:** M4.
|
||||
- **Scale** (40 symbols here, against ≥100,000 live markets, ≥13,000 traded in
|
||||
24 h): M3's opening questions in the roadmap.
|
||||
- **Counting a reconnect's differing books as losses rather than gaps,** and a
|
||||
`reconcile` that reads the log: Possible features.
|
||||
- **The rest, recorded as ceilings:**
|
||||
- a book that stays unreadable is asked for once per answer;
|
||||
- the name lookup has no timeout;
|
||||
- the read loop is written twice in `socket.rs`;
|
||||
- `open_batch` is a pair of Strings.
|
||||
|
||||
### Retrospective
|
||||
|
||||
- **Went well:**
|
||||
- Checking every claim against live data: the hash formula and each of the
|
||||
four surprises above were found by recordings and runs, not assumed.
|
||||
- Mutation checks (disable the rule, watch its test fail) on every new
|
||||
loss-detection test.
|
||||
- Doc-first changes, each settled with the user before code.
|
||||
- The two-axis close review found a real soundness hole the tests missed.
|
||||
- **Went wrong, or took too long:**
|
||||
- The sprint was agreed on a 6-symbol, 2-minute sample, so its central rule
|
||||
was rewritten four times during implementation.
|
||||
- My estimate of resyncs per trade was off (28 estimated, 8 measured).
|
||||
- The queue warning sat where it could never fire.
|
||||
- The scale of the real target (100,000+ markets against 40 symbols) came up
|
||||
only when the user asked.
|
||||
- **Tested enough?** The rules are tested on a real recording and on built
|
||||
cases. Not covered by tests:
|
||||
- the reconnect path (seen live once, before this code);
|
||||
- SIGINT during a connect;
|
||||
- the book thread's wall-clock tick when no frame comes;
|
||||
- the minute re-ask and the unreadable-book path;
|
||||
- `tick_size_change` (never observed);
|
||||
- a live `Unverifiable` symbol.
|
||||
- **Next time:**
|
||||
- During planning, record a realistic sample of the external feed and keep
|
||||
it as the fixture.
|
||||
- Count the target scale before fixing a test subset.
|
||||
- Have the close review hunt for adversarial cases against each correctness
|
||||
rule.
|
||||
@@ -0,0 +1,24 @@
|
||||
//! `reconcile OUTPUT`: runs the reconciliation check on a collector's output
|
||||
//! and fails if any Snapshot disagrees with the Deltas before it.
|
||||
|
||||
fn main() -> anyhow::Result<()> {
|
||||
let path = std::env::args()
|
||||
.nth(1)
|
||||
.ok_or_else(|| anyhow::anyhow!("usage: reconcile OUTPUT"))?;
|
||||
let report = engineering_challenge::reconcile::reconcile(path)?;
|
||||
println!(
|
||||
"{} snapshots, {} deltas, {} orphan deltas, {} mismatches",
|
||||
report.snapshots,
|
||||
report.deltas,
|
||||
report.orphan_deltas,
|
||||
report.mismatches.len()
|
||||
);
|
||||
for (symbol, received_at) in &report.mismatches {
|
||||
println!("mismatch: {symbol} at {received_at}");
|
||||
}
|
||||
anyhow::ensure!(
|
||||
report.mismatches.is_empty() && report.orphan_deltas == 0,
|
||||
"the output doesn't reconcile"
|
||||
);
|
||||
Ok(())
|
||||
}
|
||||
+172
@@ -0,0 +1,172 @@
|
||||
//! A symbol's book, the exact decimal ↔ fixed-point conversion of its levels,
|
||||
//! and the output messages built from it.
|
||||
|
||||
use std::collections::BTreeMap;
|
||||
|
||||
use crate::Side;
|
||||
use crate::util::dtos::challenge_orderbook::{
|
||||
PriceLevelUpdateDomain, SnapshotAsksEntryDomain, SnapshotBidsEntryDomain, SnapshotDomain,
|
||||
};
|
||||
use chrono::{DateTime, Utc};
|
||||
|
||||
/// The schema's fixed point: 1_000_000 = 1 USDC, or 1 share.
|
||||
const SCALE: i64 = 1_000_000;
|
||||
const DECIMALS: usize = 6;
|
||||
|
||||
/// What the book thread hands the writer thread.
|
||||
#[derive(Debug)]
|
||||
pub enum Output {
|
||||
Snapshot(SnapshotDomain),
|
||||
Delta(PriceLevelUpdateDomain),
|
||||
}
|
||||
|
||||
/// Parses a non-negative decimal string (`"0.77"`, `"26350.28"`, `"5"`)
|
||||
/// exactly, never through a float. `None` if it isn't one, has more than 6
|
||||
/// decimals, or overflows.
|
||||
pub fn parse_fixed(text: &str) -> Option<i64> {
|
||||
let (integer, fraction) = text.split_once('.').unwrap_or((text, ""));
|
||||
let digits = |s: &str| !s.is_empty() && s.bytes().all(|b| b.is_ascii_digit());
|
||||
if !digits(integer) || (text.contains('.') && !digits(fraction)) || fraction.len() > DECIMALS {
|
||||
return None;
|
||||
}
|
||||
let fraction = if fraction.is_empty() {
|
||||
0
|
||||
} else {
|
||||
format!("{fraction:0<DECIMALS$}").parse::<i64>().ok()?
|
||||
};
|
||||
integer
|
||||
.parse::<i64>()
|
||||
.ok()?
|
||||
.checked_mul(SCALE)?
|
||||
.checked_add(fraction)
|
||||
}
|
||||
|
||||
/// The canonical decimal for a fixed-point value: no trailing zeros, no `.`
|
||||
/// for an integer. The inverse of [`parse_fixed`] on canonical input.
|
||||
pub fn format_fixed(value: i64) -> String {
|
||||
let (integer, fraction) = (value / SCALE, value % SCALE);
|
||||
if fraction == 0 {
|
||||
integer.to_string()
|
||||
} else {
|
||||
format!("{integer}.{fraction:0DECIMALS$}")
|
||||
.trim_end_matches('0')
|
||||
.to_string()
|
||||
}
|
||||
}
|
||||
|
||||
/// Bids and asks, each price → absolute quantity, in fixed point.
|
||||
#[derive(Debug, Default, Clone, PartialEq, Eq)]
|
||||
pub struct Book {
|
||||
pub bids: BTreeMap<i64, i64>,
|
||||
pub asks: BTreeMap<i64, i64>,
|
||||
}
|
||||
|
||||
impl Book {
|
||||
/// Sets a level to its new absolute quantity, 0 removing it, and returns
|
||||
/// the quantity it replaced (0 if there was none).
|
||||
pub fn set(&mut self, side: Side, price: i64, quantity: i64) -> i64 {
|
||||
let levels = match side {
|
||||
Side::Buy => &mut self.bids,
|
||||
Side::Sell => &mut self.asks,
|
||||
};
|
||||
let previous = if quantity == 0 {
|
||||
levels.remove(&price)
|
||||
} else {
|
||||
levels.insert(price, quantity)
|
||||
};
|
||||
previous.unwrap_or(0)
|
||||
}
|
||||
|
||||
/// Both sides in ascending price order.
|
||||
pub fn snapshot(
|
||||
&self,
|
||||
symbol: &str,
|
||||
received_at: DateTime<Utc>,
|
||||
exchange_timestamp: DateTime<Utc>,
|
||||
) -> Output {
|
||||
Output::Snapshot(SnapshotDomain {
|
||||
received_at,
|
||||
exchange_timestamp,
|
||||
bids: self
|
||||
.bids
|
||||
.iter()
|
||||
.map(|(&price_level, &quantity)| SnapshotBidsEntryDomain {
|
||||
price_level,
|
||||
quantity,
|
||||
})
|
||||
.collect(),
|
||||
asks: self
|
||||
.asks
|
||||
.iter()
|
||||
.map(|(&price_level, &quantity)| SnapshotAsksEntryDomain {
|
||||
price_level,
|
||||
quantity,
|
||||
})
|
||||
.collect(),
|
||||
symbol: symbol.to_string(),
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn parses_exactly() {
|
||||
assert_eq!(parse_fixed("0"), Some(0));
|
||||
assert_eq!(parse_fixed("5"), Some(5_000_000));
|
||||
assert_eq!(parse_fixed("0.001"), Some(1_000));
|
||||
assert_eq!(parse_fixed("26350.28"), Some(26_350_280_000));
|
||||
assert_eq!(parse_fixed("0.123456"), Some(123_456));
|
||||
// A float would turn this into ...0.099999...
|
||||
assert_eq!(parse_fixed("4351919.1"), Some(4_351_919_100_000));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rejects_what_is_not_an_exact_decimal() {
|
||||
for text in [
|
||||
"",
|
||||
".",
|
||||
"5.",
|
||||
".5",
|
||||
"-1",
|
||||
"1e3",
|
||||
"0.1234567",
|
||||
" 1",
|
||||
"1,5",
|
||||
"99999999999999",
|
||||
] {
|
||||
assert_eq!(parse_fixed(text), None, "{text:?}");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn formats_canonically() {
|
||||
for text in [
|
||||
"0",
|
||||
"5",
|
||||
"0.001",
|
||||
"0.5",
|
||||
"26350.28",
|
||||
"4351919.06",
|
||||
"0.123456",
|
||||
] {
|
||||
assert_eq!(format_fixed(parse_fixed(text).unwrap()), text);
|
||||
}
|
||||
assert_eq!(format_fixed(parse_fixed("0.50").unwrap()), "0.5");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_zero_quantity_removes_the_level() {
|
||||
let mut book = Book::default();
|
||||
assert_eq!(book.set(Side::Buy, 500_000, 10), 0);
|
||||
assert_eq!(book.set(Side::Buy, 500_000, 20), 10);
|
||||
book.set(Side::Sell, 600_000, 5);
|
||||
assert_eq!(book.bids, BTreeMap::from([(500_000, 20)]));
|
||||
assert_eq!(book.set(Side::Buy, 500_000, 0), 20);
|
||||
assert_eq!(book.set(Side::Buy, 400_000, 0), 0);
|
||||
assert!(book.bids.is_empty());
|
||||
assert_eq!(book.asks.len(), 1);
|
||||
}
|
||||
}
|
||||
@@ -1,3 +1,5 @@
|
||||
pub mod book;
|
||||
pub mod reconcile;
|
||||
pub mod util;
|
||||
|
||||
mod side;
|
||||
|
||||
+76
-1
@@ -1,5 +1,80 @@
|
||||
//! The collector: three threads joined by channels (roadmap/02):
|
||||
//! socket (read, stamp) → book (parse, fold, check, resync) → writer (SBE).
|
||||
//! Usage: `engineering-challenge [OUTPUT]`, OUTPUT defaulting to stdout; logs
|
||||
//! go to stderr.
|
||||
|
||||
use std::sync::Arc;
|
||||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
use std::thread;
|
||||
|
||||
use crossbeam::channel::{Receiver, unbounded};
|
||||
use engineering_challenge::util::AlignedSbeWriter;
|
||||
|
||||
/// Prints a line to stderr, prefixed with the UTC time.
|
||||
macro_rules! log {
|
||||
($($arg:tt)*) => {
|
||||
eprintln!("{} {}", chrono::Utc::now().format("%H:%M:%S%.3f"), format_args!($($arg)*))
|
||||
};
|
||||
}
|
||||
|
||||
mod polymarket;
|
||||
|
||||
use engineering_challenge::book::Output;
|
||||
use polymarket::feed::Feed;
|
||||
|
||||
/// `AlignedSbeReader::open` reads nothing else.
|
||||
const CHUNK: usize = 64 << 20;
|
||||
const MARKETS: u32 = 20;
|
||||
|
||||
fn main() -> anyhow::Result<()> {
|
||||
polymarket::run_spike()
|
||||
let output = std::env::args()
|
||||
.nth(1)
|
||||
.unwrap_or_else(|| "/dev/stdout".to_string());
|
||||
let markets = polymarket::top_markets(MARKETS)?;
|
||||
let feed = Feed::new(&markets);
|
||||
let tokens = feed.tokens();
|
||||
|
||||
let stop = Arc::new(AtomicBool::new(false));
|
||||
let on_sigint = Arc::clone(&stop);
|
||||
ctrlc::set_handler(move || on_sigint.store(true, Ordering::Relaxed))?;
|
||||
|
||||
let (frames_tx, frames_rx) = unbounded();
|
||||
let (resyncs_tx, resyncs_rx) = unbounded();
|
||||
let (outputs_tx, outputs_rx) = unbounded();
|
||||
// Each stage stops when the one before it does, so SIGINT only reaches the
|
||||
// socket thread; the writer flushes whatever came before.
|
||||
let socket =
|
||||
thread::spawn(move || polymarket::socket::run(&tokens, &frames_tx, &resyncs_rx, &stop));
|
||||
let book =
|
||||
thread::spawn(move || polymarket::feed::run(feed, frames_rx, outputs_tx, resyncs_tx));
|
||||
let writer = thread::spawn(move || write(&output, outputs_rx));
|
||||
|
||||
// Every thread is joined, so the writer flushes even after a panic
|
||||
// upstream; the run still fails then.
|
||||
let socket_panicked = socket.join().is_err();
|
||||
let book_panicked = book.join().is_err();
|
||||
let written = writer
|
||||
.join()
|
||||
.map_err(|_| anyhow::anyhow!("the writer thread panicked"))?;
|
||||
log!("output closed");
|
||||
written?;
|
||||
anyhow::ensure!(!socket_panicked, "the socket thread panicked");
|
||||
anyhow::ensure!(!book_panicked, "the book thread panicked");
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// The writer thread: writes outputs in arrival order until the book thread
|
||||
/// is gone, then flushes the last chunk.
|
||||
// ponytail: the chunk being filled lives only in memory until it is full, so
|
||||
// a hard crash loses it (hours of data); flushing more often pads every flush
|
||||
// to 64 MiB. Taken up in M4.
|
||||
fn write(path: &str, outputs: Receiver<Output>) -> anyhow::Result<()> {
|
||||
let mut writer = AlignedSbeWriter::create(path, CHUNK)?;
|
||||
for output in outputs {
|
||||
match output {
|
||||
Output::Snapshot(snapshot) => writer.write_msg(&snapshot)?,
|
||||
Output::Delta(delta) => writer.write_msg(&delta)?,
|
||||
}
|
||||
}
|
||||
writer.flush()
|
||||
}
|
||||
@@ -1,33 +0,0 @@
|
||||
//! Connectivity spike: connect to Polymarket's public market channel, subscribe
|
||||
//! to one market, and log every raw frame with its receive time.
|
||||
|
||||
use chrono::{SecondsFormat, Utc};
|
||||
use tungstenite::Message;
|
||||
|
||||
const MARKET_CHANNEL_URL: &str = "wss://ws-subscriptions-clob.polymarket.com/ws/market";
|
||||
|
||||
/// Both tokens (Yes, No) of one market, hardcoded for the spike (discovery comes
|
||||
/// in M3): "2026 Balance of Power: D Senate, D House", which closes 2026-11-04,
|
||||
/// so pick a fresh market to rerun this.
|
||||
const SUBSCRIBE_MESSAGE: &str = r#"{"type":"market","assets_ids":["34722410608062854697106861099776685947172185964394483545370684749662285977831","24483563228660028121020430722896740519524131851526579619586176931221901842798"]}"#;
|
||||
|
||||
/// Connects, sends [`SUBSCRIBE_MESSAGE`], and logs frames until the server
|
||||
/// closes the connection.
|
||||
pub fn run_spike() -> anyhow::Result<()> {
|
||||
let (mut socket, _response) = tungstenite::connect(MARKET_CHANNEL_URL)?;
|
||||
socket.send(Message::text(SUBSCRIBE_MESSAGE))?;
|
||||
|
||||
loop {
|
||||
let message = socket.read()?;
|
||||
// Stamp as soon as the frame is off the wire, before any processing.
|
||||
let received_at = Utc::now().to_rfc3339_opts(SecondsFormat::Micros, true);
|
||||
match message {
|
||||
Message::Text(text) => println!("{received_at} {text}"),
|
||||
Message::Close(frame) => {
|
||||
println!("{received_at} close {frame:?}");
|
||||
return Ok(());
|
||||
}
|
||||
other => println!("{received_at} {other:?}"),
|
||||
}
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,76 @@
|
||||
//! Polymarket: which markets to track (Gamma), the market-channel socket, and
|
||||
//! the feed that turns its frames into books and output.
|
||||
|
||||
pub mod feed;
|
||||
pub mod socket;
|
||||
|
||||
use serde::Deserialize;
|
||||
|
||||
const GAMMA_MARKETS_URL: &str = "https://gamma-api.polymarket.com/markets";
|
||||
|
||||
/// A market's symbols and the two inputs of their book hash that the
|
||||
/// websocket doesn't carry (docs/notes.md).
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct Market {
|
||||
/// Its token ids, one per outcome: each is a symbol.
|
||||
pub tokens: Vec<String>,
|
||||
/// Exactly as the hash expects it, e.g. `"5"`.
|
||||
pub min_order_size: String,
|
||||
pub neg_risk: bool,
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
struct GammaMarket {
|
||||
/// A JSON array inside a string.
|
||||
clob_token_ids: String,
|
||||
order_min_size: serde_json::Number,
|
||||
neg_risk: bool,
|
||||
enable_order_book: bool,
|
||||
}
|
||||
|
||||
/// The `limit` live markets with the most 24 h volume, keeping those with an
|
||||
/// order book. Fixed for the run: discovery is M3.
|
||||
pub fn top_markets(limit: u32) -> anyhow::Result<Vec<Market>> {
|
||||
let body = ureq::get(GAMMA_MARKETS_URL)
|
||||
.query("limit", limit.to_string())
|
||||
.query("active", "true")
|
||||
.query("closed", "false")
|
||||
.query("order", "volume24hr")
|
||||
.query("ascending", "false")
|
||||
.call()?
|
||||
.body_mut()
|
||||
.read_to_string()?;
|
||||
parse_markets(&body)
|
||||
}
|
||||
|
||||
fn parse_markets(body: &str) -> anyhow::Result<Vec<Market>> {
|
||||
let markets: Vec<GammaMarket> = serde_json::from_str(body)?;
|
||||
markets
|
||||
.into_iter()
|
||||
.filter(|market| market.enable_order_book)
|
||||
.map(|market| {
|
||||
Ok(Market {
|
||||
tokens: serde_json::from_str(&market.clob_token_ids)?,
|
||||
min_order_size: market.order_min_size.to_string(),
|
||||
neg_risk: market.neg_risk,
|
||||
})
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
pub(crate) fn fixture_markets() -> Vec<Market> {
|
||||
parse_markets(include_str!("../../tests/fixtures/gamma-markets.json")).unwrap()
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
#[test]
|
||||
fn parses_gamma_markets() {
|
||||
let markets = super::fixture_markets();
|
||||
assert_eq!(markets.len(), 20);
|
||||
assert!(markets.iter().all(|market| market.tokens.len() == 2));
|
||||
assert_eq!(markets[0].min_order_size, "5");
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,153 @@
|
||||
//! The socket thread: reads the market channel and stamps each frame's receive
|
||||
//! time, and nothing else. Parsing is the book thread's.
|
||||
|
||||
use std::io::ErrorKind;
|
||||
use std::net::{TcpStream, ToSocketAddrs};
|
||||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
use std::thread;
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use chrono::{DateTime, Utc};
|
||||
use crossbeam::channel::{Receiver, Sender};
|
||||
use serde_json::json;
|
||||
use tungstenite::stream::MaybeTlsStream;
|
||||
use tungstenite::{Message, Utf8Bytes, WebSocket};
|
||||
|
||||
const MARKET_CHANNEL_HOST: &str = "ws-subscriptions-clob.polymarket.com";
|
||||
const MARKET_CHANNEL_URL: &str = "wss://ws-subscriptions-clob.polymarket.com/ws/market";
|
||||
/// For the TCP connect and the TLS + websocket handshake, and for every send.
|
||||
const CONNECT_TIMEOUT: Duration = Duration::from_secs(10);
|
||||
/// How often the loop wakes when no frame comes: tungstenite has no `split`,
|
||||
/// so the reads and the writes below (PING, resyncs, close) share this thread.
|
||||
const READ_TIMEOUT: Duration = Duration::from_millis(200);
|
||||
const PING_EVERY: Duration = Duration::from_secs(10);
|
||||
const RECONNECT_AFTER: Duration = Duration::from_secs(1);
|
||||
const CLOSE_TIMEOUT: Duration = Duration::from_secs(2);
|
||||
|
||||
/// A frame's text with its receive time.
|
||||
pub type Frame = (DateTime<Utc>, Utf8Bytes);
|
||||
|
||||
type Socket = WebSocket<MaybeTlsStream<TcpStream>>;
|
||||
|
||||
/// Runs until `stop` is set or the book thread is gone, reconnecting (and
|
||||
/// subscribing every token again) whenever the connection drops. `resyncs`
|
||||
/// carries the tokens to unsubscribe and subscribe again.
|
||||
pub fn run(
|
||||
tokens: &[String],
|
||||
frames: &Sender<Frame>,
|
||||
resyncs: &Receiver<String>,
|
||||
stop: &AtomicBool,
|
||||
) {
|
||||
while !stop.load(Ordering::Relaxed) {
|
||||
match connect(tokens) {
|
||||
Ok(mut socket) => match read(&mut socket, frames, resyncs, stop) {
|
||||
Ok(Ended::Stopped) => return close(&mut socket, frames),
|
||||
Ok(Ended::BookThreadGone) => return,
|
||||
Err(error) => log!("polymarket: connection lost: {error}"),
|
||||
},
|
||||
Err(error) => log!("polymarket: connect failed: {error}"),
|
||||
}
|
||||
thread::sleep(RECONNECT_AFTER);
|
||||
}
|
||||
}
|
||||
|
||||
enum Ended {
|
||||
Stopped,
|
||||
BookThreadGone,
|
||||
}
|
||||
|
||||
fn connect(tokens: &[String]) -> anyhow::Result<Socket> {
|
||||
// Timed, unlike `tungstenite::connect`, so a stalled connect or handshake
|
||||
// can't keep SIGINT waiting. The name lookup itself has no timeout.
|
||||
let address = (MARKET_CHANNEL_HOST, 443)
|
||||
.to_socket_addrs()?
|
||||
.next()
|
||||
.ok_or_else(|| anyhow::anyhow!("{MARKET_CHANNEL_HOST} has no address"))?;
|
||||
let stream = TcpStream::connect_timeout(&address, CONNECT_TIMEOUT)?;
|
||||
stream.set_read_timeout(Some(CONNECT_TIMEOUT))?;
|
||||
stream.set_write_timeout(Some(CONNECT_TIMEOUT))?;
|
||||
let (mut socket, _response) = tungstenite::client_tls(MARKET_CHANNEL_URL, stream)
|
||||
.map_err(|error| anyhow::anyhow!("handshake failed: {error}"))?;
|
||||
let stream = match socket.get_ref() {
|
||||
MaybeTlsStream::Plain(stream) => stream,
|
||||
MaybeTlsStream::Rustls(stream) => stream.get_ref(),
|
||||
_ => anyhow::bail!("unexpected TLS stream"),
|
||||
};
|
||||
stream.set_read_timeout(Some(READ_TIMEOUT))?;
|
||||
socket.send(Message::text(
|
||||
json!({"type": "market", "assets_ids": tokens}).to_string(),
|
||||
))?;
|
||||
log!("polymarket: connected, subscribed {} symbols", tokens.len());
|
||||
Ok(socket)
|
||||
}
|
||||
|
||||
fn read(
|
||||
socket: &mut Socket,
|
||||
frames: &Sender<Frame>,
|
||||
resyncs: &Receiver<String>,
|
||||
stop: &AtomicBool,
|
||||
) -> anyhow::Result<Ended> {
|
||||
let mut last_ping = Instant::now();
|
||||
loop {
|
||||
let message = socket.read();
|
||||
// Stamp as soon as the frame is off the wire, before any processing.
|
||||
let received_at = Utc::now();
|
||||
match message {
|
||||
Ok(Message::Text(text)) => {
|
||||
if frames.send((received_at, text)).is_err() {
|
||||
return Ok(Ended::BookThreadGone);
|
||||
}
|
||||
}
|
||||
Ok(Message::Close(frame)) => anyhow::bail!("closed by the server: {frame:?}"),
|
||||
Ok(_) => {}
|
||||
Err(tungstenite::Error::Io(error)) if timed_out(&error) => {}
|
||||
Err(error) => return Err(error.into()),
|
||||
}
|
||||
|
||||
if stop.load(Ordering::Relaxed) {
|
||||
return Ok(Ended::Stopped);
|
||||
}
|
||||
if last_ping.elapsed() >= PING_EVERY {
|
||||
socket.send(Message::text("PING"))?;
|
||||
last_ping = Instant::now();
|
||||
}
|
||||
for token in resyncs.try_iter() {
|
||||
// Subscribing a token already subscribed sends nothing back but
|
||||
// `NO NEW ASSETS`: unsubscribing first makes the server send a
|
||||
// fresh book.
|
||||
socket.send(Message::text(
|
||||
json!({"assets_ids": [&token], "operation": "unsubscribe"}).to_string(),
|
||||
))?;
|
||||
socket.send(Message::text(
|
||||
json!({"assets_ids": [&token], "operation": "subscribe"}).to_string(),
|
||||
))?;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Closes the connection, forwarding what still arrives until the server
|
||||
/// confirms or [`CLOSE_TIMEOUT`] passes.
|
||||
fn close(socket: &mut Socket, frames: &Sender<Frame>) {
|
||||
if let Err(error) = socket.close(None) {
|
||||
return log!("polymarket: close failed: {error}");
|
||||
}
|
||||
let deadline = Instant::now() + CLOSE_TIMEOUT;
|
||||
while Instant::now() < deadline {
|
||||
let message = socket.read();
|
||||
let received_at = Utc::now();
|
||||
match message {
|
||||
Ok(Message::Text(text)) => _ = frames.send((received_at, text)),
|
||||
Ok(_) => {}
|
||||
Err(tungstenite::Error::Io(error)) if timed_out(&error) => {}
|
||||
// ConnectionClosed: the close handshake is done.
|
||||
Err(_) => break,
|
||||
}
|
||||
}
|
||||
log!("polymarket: socket closed");
|
||||
}
|
||||
|
||||
/// A read timeout shows up as `WouldBlock` on macOS and `TimedOut` elsewhere;
|
||||
/// tungstenite keeps a partly read frame, so the next read resumes it.
|
||||
fn timed_out(error: &std::io::Error) -> bool {
|
||||
matches!(error.kind(), ErrorKind::WouldBlock | ErrorKind::TimedOut)
|
||||
}
|
||||
@@ -0,0 +1,65 @@
|
||||
//! Checks an output file against itself: per symbol, the Deltas written since
|
||||
//! a Snapshot, applied to it, must give the next Snapshot. A mismatch is a loss
|
||||
//! or a resync gap (the fresh book differs), each logged by the collector.
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::path::Path;
|
||||
|
||||
use crate::book::Book;
|
||||
use crate::util::dtos::challenge_orderbook::{AnyMessage, PriceLevelUpdateDomain, SnapshotDomain};
|
||||
use crate::util::{AlignedSbeReader, OrderbookSchema};
|
||||
|
||||
#[derive(Debug, Default)]
|
||||
pub struct Report {
|
||||
pub snapshots: usize,
|
||||
pub deltas: usize,
|
||||
/// Deltas for a symbol with no Snapshot yet: the collector never writes one.
|
||||
pub orphan_deltas: usize,
|
||||
/// Each Snapshot that didn't match its symbol's book: (symbol, its `received_at`).
|
||||
pub mismatches: Vec<(String, chrono::DateTime<chrono::Utc>)>,
|
||||
}
|
||||
|
||||
pub fn reconcile(path: impl AsRef<Path>) -> anyhow::Result<Report> {
|
||||
let mut reader = AlignedSbeReader::<OrderbookSchema>::open(path)?;
|
||||
let mut books: HashMap<String, Book> = HashMap::new();
|
||||
let mut report = Report::default();
|
||||
while let Some(message) = reader.read_msg()? {
|
||||
match message {
|
||||
AnyMessage::Snapshot(decoder) => {
|
||||
let snapshot = SnapshotDomain::try_from_decoder(decoder)?;
|
||||
report.snapshots += 1;
|
||||
let book = Book {
|
||||
bids: snapshot
|
||||
.bids
|
||||
.iter()
|
||||
.map(|level| (level.price_level, level.quantity))
|
||||
.collect(),
|
||||
asks: snapshot
|
||||
.asks
|
||||
.iter()
|
||||
.map(|level| (level.price_level, level.quantity))
|
||||
.collect(),
|
||||
};
|
||||
if books
|
||||
.get(&snapshot.symbol)
|
||||
.is_some_and(|ours| *ours != book)
|
||||
{
|
||||
report
|
||||
.mismatches
|
||||
.push((snapshot.symbol.clone(), snapshot.received_at));
|
||||
}
|
||||
books.insert(snapshot.symbol, book);
|
||||
}
|
||||
AnyMessage::PriceLevelUpdate(decoder) => {
|
||||
let delta = PriceLevelUpdateDomain::try_from_decoder(decoder)?;
|
||||
report.deltas += 1;
|
||||
match books.get_mut(&delta.symbol) {
|
||||
Some(book) => _ = book.set(delta.side, delta.price_level, delta.quantity),
|
||||
None => report.orphan_deltas += 1,
|
||||
}
|
||||
}
|
||||
_ => anyhow::bail!("unknown message in the output"),
|
||||
}
|
||||
}
|
||||
Ok(report)
|
||||
}
|
||||
Vendored
+13
@@ -0,0 +1,13 @@
|
||||
# Test fixtures
|
||||
|
||||
A recording of Polymarket's market channel, replayed by the tests in
|
||||
`src/polymarket/feed.rs` (no network in tests).
|
||||
|
||||
- `gamma-markets.json`: Gamma's top 20 markets by 24 h volume on 2026-11-04,
|
||||
trimmed to the fields the collector reads.
|
||||
- `polymarket-frames.jsonl`: the first 90 s (14:24:44 to 14:26:14 UTC, so two
|
||||
minute boundaries) of the market channel for those markets' 40 tokens, one
|
||||
line per frame: `{"received_at_us": <receive time, µs>, "text": <frame>}`.
|
||||
Opening books, 3,850 changes, 28 mid-stream books, 14 trades, 8 `PONG`s.
|
||||
- `record.py`: the recorder, if the fixtures need redoing:
|
||||
`uvx --with websockets python -I record.py gamma-markets.json frames.jsonl 90`.
|
||||
Vendored
+142
@@ -0,0 +1,142 @@
|
||||
[
|
||||
{
|
||||
"question": "2026 Balance of Power: D Senate, D House",
|
||||
"clobTokenIds": "[\"34722410608062854697106861099776685947172185964394483545370684749662285977831\", \"24483563228660028121020430722896740519524131851526579619586176931221901842798\"]",
|
||||
"orderMinSize": 5,
|
||||
"negRisk": true,
|
||||
"enableOrderBook": true
|
||||
},
|
||||
{
|
||||
"question": "2026 Balance of Power: R Senate, R House",
|
||||
"clobTokenIds": "[\"103704141773947678931823410030956181918562062788486034785782641603149828893320\", \"25869296386911654049715489068824757745174681288397238521780118949553760488529\"]",
|
||||
"orderMinSize": 5,
|
||||
"negRisk": true,
|
||||
"enableOrderBook": true
|
||||
},
|
||||
{
|
||||
"question": "Will Republicans lose the Alaska, Iowa, Kansas, Maine, Michigan, Ohio, and Texas Senate seats?",
|
||||
"clobTokenIds": "[\"13535941900723044344907002451427688051727855016747811259193368015899108383450\", \"80727480162707694817992062873857913056698208056368499870278974630056151503113\"]",
|
||||
"orderMinSize": 5,
|
||||
"negRisk": true,
|
||||
"enableOrderBook": true
|
||||
},
|
||||
{
|
||||
"question": "2026 Balance of Power: R Senate, D House",
|
||||
"clobTokenIds": "[\"6302359956133594764084277082169634158291609371270652093164054687145970756151\", \"558600218792430158020964384658762756367786275998757133304987611939841005404\"]",
|
||||
"orderMinSize": 5,
|
||||
"negRisk": true,
|
||||
"enableOrderBook": true
|
||||
},
|
||||
{
|
||||
"question": "Will the Republicans win the Mississippi Senate race in 2026?",
|
||||
"clobTokenIds": "[\"36589074670521348842786325911391532620430247097981345860356484520500173888573\", \"4639340586725578716005500791739559303082924048072534329825226706547465941088\"]",
|
||||
"orderMinSize": 5,
|
||||
"negRisk": true,
|
||||
"enableOrderBook": true
|
||||
},
|
||||
{
|
||||
"question": "Will Mary Peltola win the Alaska Senate race in 2026?",
|
||||
"clobTokenIds": "[\"72139607190315861034411702195742538105117960260100800877125381826122747504258\", \"38274379050640062523611849065721372699838119672720475300345842413156391540232\"]",
|
||||
"orderMinSize": 5,
|
||||
"negRisk": true,
|
||||
"enableOrderBook": true
|
||||
},
|
||||
{
|
||||
"question": "Will the Democrats win the Mississippi Senate race in 2026?",
|
||||
"clobTokenIds": "[\"77001639760095266374534198365421436593389036687975321536722226607654949914567\", \"109329484956764506378760583725617100193692863947320202695087573398990069238209\"]",
|
||||
"orderMinSize": 5,
|
||||
"negRisk": true,
|
||||
"enableOrderBook": true
|
||||
},
|
||||
{
|
||||
"question": "2026 Balance of Power: D Senate, R House",
|
||||
"clobTokenIds": "[\"70997927349469817841862065582625658840347600365813612622959588796331622340305\", \"45720947308530468765087874228584729250108790729495800420104302263408503667446\"]",
|
||||
"orderMinSize": 5,
|
||||
"negRisk": true,
|
||||
"enableOrderBook": true
|
||||
},
|
||||
{
|
||||
"question": "Will Republicans win Iowa, and lose Alaska, Kansas, Maine, Michigan, Ohio, and Texas?",
|
||||
"clobTokenIds": "[\"106253100341058888227154298197221161777912145262343559280892425406157624656915\", \"38883886691124225989332035378913105158890603529288875957747834557386418386450\"]",
|
||||
"orderMinSize": 5,
|
||||
"negRisk": true,
|
||||
"enableOrderBook": true
|
||||
},
|
||||
{
|
||||
"question": "Will the US confirm that aliens exist before 2027?",
|
||||
"clobTokenIds": "[\"107505882767731489358349912513945399560393482969656700824895970500493757150417\", \"7305630249804085635496399869905769372294302716159034447326228509068694952392\"]",
|
||||
"orderMinSize": 5,
|
||||
"negRisk": false,
|
||||
"enableOrderBook": true
|
||||
},
|
||||
{
|
||||
"question": "Rolex Paris Masters: Alexander Zverev vs Lorenzo Musetti",
|
||||
"clobTokenIds": "[\"97202122435166371596632600271207731098197012320781592658111953138552052790871\", \"64561849525628736112915971493239643546218846988377083760471198523625509677016\"]",
|
||||
"orderMinSize": 5,
|
||||
"negRisk": false,
|
||||
"enableOrderBook": true
|
||||
},
|
||||
{
|
||||
"question": "Will Republicans win Alaska, and lose Iowa, Kansas, Maine, Michigan, Ohio, and Texas?",
|
||||
"clobTokenIds": "[\"83965128865532878903206942328238684643656639972076348062693215949819184139617\", \"93889370639599634853686597701962997508099936018794651953480468245159849951085\"]",
|
||||
"orderMinSize": 5,
|
||||
"negRisk": true,
|
||||
"enableOrderBook": true
|
||||
},
|
||||
{
|
||||
"question": "US x Iran ceasefire continues through October 31?",
|
||||
"clobTokenIds": "[\"67102598924726349989742049657609063097299442010642912942032681022413402597017\", \"114475889232765203212032217024645308497828327658421967586050553616813506019119\"]",
|
||||
"orderMinSize": 5,
|
||||
"negRisk": false,
|
||||
"enableOrderBook": true
|
||||
},
|
||||
{
|
||||
"question": "Will Real Madrid CF win on 2026-11-04?",
|
||||
"clobTokenIds": "[\"39515423851331707664526519001996875757487006788622502797081364834883774387767\", \"3771084942539439798998256191956824579778203065651547449472385427482459623087\"]",
|
||||
"orderMinSize": 5,
|
||||
"negRisk": true,
|
||||
"enableOrderBook": true
|
||||
},
|
||||
{
|
||||
"question": "Will Republicans win Iowa and Kansas, and lose Alaska, Maine, Michigan, Ohio, and Texas?",
|
||||
"clobTokenIds": "[\"105070040449310495327131730101841566321530681945885747881686068698995174349681\", \"111381299800799485509083912497813429956512687399857260440199290536659860679263\"]",
|
||||
"orderMinSize": 5,
|
||||
"negRisk": true,
|
||||
"enableOrderBook": true
|
||||
},
|
||||
{
|
||||
"question": "Will the U.S. invade Iran before 2027?",
|
||||
"clobTokenIds": "[\"55115078421062885512539156303747803058407616201213034911037320915726138659123\", \"1910830010387565971650098373488592514702818137344973088263643820608151819241\"]",
|
||||
"orderMinSize": 5,
|
||||
"negRisk": false,
|
||||
"enableOrderBook": true
|
||||
},
|
||||
{
|
||||
"question": "Will Benjamin Netanyahu be the next Prime Minister of Israel?",
|
||||
"clobTokenIds": "[\"20006732765674855733524007935991362439352594042415161039767275461950712817548\", \"54533569528927082430133047556849600511985191158927477533124630301934489498530\"]",
|
||||
"orderMinSize": 5,
|
||||
"negRisk": true,
|
||||
"enableOrderBook": true
|
||||
},
|
||||
{
|
||||
"question": "Will Dan Sullivan win the Alaska Senate race in 2026?",
|
||||
"clobTokenIds": "[\"2618138322856109565904469085316346602627882727483900432328802655317079880335\", \"64264976954058187238724333554507208255303776435657387002861226761784169752307\"]",
|
||||
"orderMinSize": 5,
|
||||
"negRisk": true,
|
||||
"enableOrderBook": true
|
||||
},
|
||||
{
|
||||
"question": "Will Nithya Raman win the 2026 Los Angeles mayoral election?",
|
||||
"clobTokenIds": "[\"50292933818751585369672690878774599873941191925538325333138798248932182222864\", \"58507084608782512565480280053882807505189448738098588563704773804344223603355\"]",
|
||||
"orderMinSize": 5,
|
||||
"negRisk": true,
|
||||
"enableOrderBook": true
|
||||
},
|
||||
{
|
||||
"question": "Will John McEntee win the 2028 Republican presidential nomination?",
|
||||
"clobTokenIds": "[\"50730508396692945396787154191308995513666087024795374376582570929738968294242\", \"7808428608295632540409928244677767077616163699109217296255297216219773833140\"]",
|
||||
"orderMinSize": 5,
|
||||
"negRisk": true,
|
||||
"enableOrderBook": true
|
||||
}
|
||||
]
|
||||
+3901
File diff suppressed because it is too large.
Load diff
Vendored
+19
@@ -0,0 +1,19 @@
|
||||
# Records the market channel for the markets in gamma-markets.json: one JSON
|
||||
# line per frame, {"received_at_us": <micros>, "text": <frame text>}.
|
||||
import asyncio, json, sys, time
|
||||
import websockets
|
||||
markets = json.load(open(sys.argv[1]))
|
||||
tokens = [t for m in markets if m["enableOrderBook"] for t in json.loads(m["clobTokenIds"])]
|
||||
async def main(out, seconds):
|
||||
async with websockets.connect("wss://ws-subscriptions-clob.polymarket.com/ws/market", max_size=None) as ws:
|
||||
await ws.send(json.dumps({"type": "market", "assets_ids": tokens}))
|
||||
loop = asyncio.get_event_loop(); end = loop.time() + seconds; last_ping = loop.time()
|
||||
while loop.time() < end:
|
||||
if loop.time() - last_ping >= 10:
|
||||
await ws.send("PING"); last_ping = loop.time()
|
||||
try: text = await asyncio.wait_for(ws.recv(), 1)
|
||||
except asyncio.TimeoutError: continue
|
||||
received_at_us = time.time_ns() // 1000
|
||||
out.write(json.dumps({"received_at_us": received_at_us, "text": text}) + "\n")
|
||||
with open(sys.argv[2], "w") as out:
|
||||
asyncio.run(main(out, float(sys.argv[3])))
|
||||
Reference in new issue
Block a user