feat(polymarket): skeleton and market-channel connectivity spike

Replaces the template's demo main.rs with a spike that connects to
Polymarket's public market channel, subscribes to one market, and logs every
frame with its receive time. Adds scripts/ci.sh (fmt, clippy, tests), the
project CLAUDE.md, ADR 0001 (OS threads, shared-nothing), and the market
channel's message shapes in docs/notes.md: no sequence number, a book hash on
every change, mirrored Yes/No books.
This commit is contained in:
pascal committed 2026-11-04 14:47:14 +01:00
1 parent adcf205f0b
commit 0e687e7eba
11 files changed
+713 -138

No files matched your search

+63
View File
@@ -0,0 +1,63 @@
# Order-book collector (Voluntas engineering challenge)
A collector for prediction-market order books (Polymarket, then Kalshi) that
emits SBE-encoded updates and per-minute snapshots, built inside the challenge
template. The brief is `../website/Engineering Challenge _ Voluntas.html`,
outside this repo.
## Where things are
- `docs/project.md`: definition, central bet (the concurrency pipeline), and the
reasons behind the design principles. Read before any design discussion.
- `docs/roadmap.md` and `roadmap/NN. name.md`: milestones and each sprint's
agreed plan. Read the current sprint file before writing code.
- `docs/adr/`: hard-to-reverse decisions (0001: threads, shared-nothing).
- `docs/notes.md`: domain rules and exchange protocol facts (delta semantics,
loss detection, message shapes). Read before touching exchange parsing.
- `src/util/` and `sbe_schemas/`: the template's SBE machinery, used as given so
output stays readable by the template's `AlignedSbeReader`. Our code is the
rest of `src/`.
## Working together
The user drives the pipeline / concurrency architecture: propose and debate it,
and let the user settle it. Claude owns the data model.
## Writing code
- Shared-nothing threads: each stage owns its state on one thread or is sharded,
and stages talk over channels. Mutexes stay out of the pipeline.
- Boring and readable: std collections (`BTreeMap` for a price-ordered side),
plain threads and channels. Speed work waits for the profiler, after V1.
- An abstraction arrives with its second implementation (the exchange trait in
M6).
- Stamp receive time immediately after the socket read, and rebuild a book from
a fresh snapshot whenever a lost message is detected (how depends on the
exchange: `docs/notes.md`).
- A corner cut for speed carries a comment naming its ceiling and upgrade path.
## Environment traps
- **Polymarket and Kalshi are DNS-blocked on the dev network**: the local
resolver answers `195.186.4.210` / `2001:4d98:4:b10c::1` (a Swisscom block
page). The user pins the hosts with `sudo ../host.sh on` (workspace root,
outside this repo; `off` removes them). A timeout or the block address again
means the CDN addresses rotated: ask the user to rerun it.
- **Write 64 MiB chunks**: `AlignedSbeReader::open` hardcodes 64 MiB
(`open_with_chunk_size` is private), so any other chunk size is unreadable.
- **The binary runs until the server closes the socket.** macOS has no
`timeout`; bound a run with
`perl -e 'alarm 10; exec @ARGV' ./target/debug/engineering-challenge`.
## Workflow differences
A job-challenge repo cloned from the template: no Gitea, Woodpecker or release
pipeline.
- **Kept**: `docs/`, `roadmap/` sprint files, `feat/*` branches, and
`scripts/ci.sh` (fmt, clippy `-D warnings`, tests).
- **Dropped**: the dependency audit (supply-chain checks are for deployed
products), Woodpecker, `release.sh`, versioning and changelog, rc images, and
the `prod` / `dev` branches. CI targets application quality, not prod-grade.
- **Branches**: `main` plays the role of `dev`; `feat/*` branches are squashed
into `main`.
Generated
+266 -23
View File
@@ -83,6 +83,12 @@ version = "3.20.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "72f5acc6cb2ba439de613abc23857ec3d78374d8ed5ac84e9d11336e87da8649"
[[package]]
name = "bytes"
version = "1.12.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fc652a48c352aef3ea3aed32080501cf3ef6ed5da78602a020c991775b0aff04"
[[package]]
name = "cc"
version = "1.9.0"
@@ -99,6 +105,17 @@ version = "1.0.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4e7648175b45a9a48536d676f68d918270699102aa8dab5496df06904c914600"
[[package]]
name = "chacha20"
version = "0.10.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "65c35e4b699c7e15ccbe7ee35c005e4fc0a278d22238a2857e6ce2dadeda1b06"
dependencies = [
"cfg-if",
"cpufeatures",
"rand_core",
]
[[package]]
name = "chrono"
version = "0.4.45"
@@ -198,6 +215,12 @@ dependencies = [
"hybrid-array",
]
[[package]]
name = "data-encoding"
version = "2.11.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4583a4551df46e2792f82ceeac45e850d2e2d5debba0b91f102385cda5b11f06"
[[package]]
name = "digest"
version = "0.11.3"
@@ -229,9 +252,10 @@ dependencies = [
"crossbeam",
"ergo-sbe",
"indexmap",
"rustls",
"serde",
"serde_yaml_ng",
"tempfile",
"tungstenite",
]
[[package]]
@@ -263,15 +287,9 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb"
dependencies = [
"libc",
"windows-sys",
"windows-sys 0.61.2",
]
[[package]]
name = "fastrand"
version = "2.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "da7c62ceae207dd37ea5b845da6a0696c799f85e97da1ab5b7910be3c1c80223"
[[package]]
name = "find-msvc-tools"
version = "0.1.15"
@@ -302,6 +320,17 @@ dependencies = [
"slab",
]
[[package]]
name = "getrandom"
version = "0.2.17"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ff2abc00be7fca6ebc474524697ae276ad847ad0a6b3faa4bcb027e9a4614ad0"
dependencies = [
"cfg-if",
"libc",
"wasi",
]
[[package]]
name = "getrandom"
version = "0.4.3"
@@ -311,6 +340,7 @@ dependencies = [
"cfg-if",
"libc",
"r-efi",
"rand_core",
]
[[package]]
@@ -325,6 +355,22 @@ version = "0.17.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ed5909b6e89a2db4456e54cd5f673791d7eca6732202bbf2a9cc504fe2f9b84a"
[[package]]
name = "http"
version = "1.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "918d3568bebf352712bc2ef3d46a8bcf1a75b373be6539de198e9105cbbf9ce0"
dependencies = [
"bytes",
"itoa",
]
[[package]]
name = "httparse"
version = "1.10.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6dbf3de79e51f3d586ab4cb9d5c3e2c14aa28ed23d180cf89b4df0454a69cc87"
[[package]]
name = "hybrid-array"
version = "0.4.15"
@@ -630,6 +676,37 @@ version = "6.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf"
[[package]]
name = "rand"
version = "0.10.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "65c9fb96cbc91e3478eaae79a69fcd3f1ae4ad052e471fe6732fff548984b4af"
dependencies = [
"chacha20",
"getrandom 0.4.3",
"rand_core",
]
[[package]]
name = "rand_core"
version = "0.10.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "63b8176103e19a2643978565ca18b50549f6101881c443590420e4dc998a3c69"
[[package]]
name = "ring"
version = "0.17.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a4689e6c2294d81e88dc6261c768b63bc4fcdb852be6d1352498b114f61383b7"
dependencies = [
"cc",
"cfg-if",
"getrandom 0.2.17",
"libc",
"untrusted",
"windows-sys 0.52.0",
]
[[package]]
name = "roxmltree"
version = "0.21.1"
@@ -655,7 +732,41 @@ dependencies = [
"errno",
"libc",
"linux-raw-sys",
"windows-sys",
"windows-sys 0.61.2",
]
[[package]]
name = "rustls"
version = "0.23.46"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "48e13bd8c0e9365c43cfa5c9e8f9ad49d3c8444926c9aac819e0e4dc503c8fdf"
dependencies = [
"once_cell",
"ring",
"rustls-pki-types",
"rustls-webpki",
"subtle",
"zeroize",
]
[[package]]
name = "rustls-pki-types"
version = "1.15.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2f4925028c7eb5d1fcdaf196971378ed9d2c1c4efc7dc5d011256f76c99c0a96"
dependencies = [
"zeroize",
]
[[package]]
name = "rustls-webpki"
version = "0.103.16"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4443b1b6d39d7c663139942cc4558262562d2c84123bb6282ef177009ca8b281"
dependencies = [
"ring",
"rustls-pki-types",
"untrusted",
]
[[package]]
@@ -713,6 +824,17 @@ dependencies = [
"unsafe-libyaml",
]
[[package]]
name = "sha1"
version = "0.11.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "aacc4cc499359472b4abe1bf11d0b12e688af9a805fa5e3016f9a386dc2d0214"
dependencies = [
"cfg-if",
"cpufeatures",
"digest",
]
[[package]]
name = "sha2"
version = "0.11.1"
@@ -748,6 +870,12 @@ version = "1.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6ce2be8dc25455e1f91df71bfa12ad37d7af1092ae736f3a6cd0e37bc7810596"
[[package]]
name = "subtle"
version = "2.6.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "13c2bddecc57b384dee18652358fb23172facb8a2c51ccc10d74c157bdea3292"
[[package]]
name = "supports-color"
version = "3.0.2"
@@ -802,19 +930,6 @@ dependencies = [
"syn 3.0.9",
]
[[package]]
name = "tempfile"
version = "3.27.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "32497e9a4c7b38532efcdebeef879707aa9f794296a4f0244f6f69e9bc8574bd"
dependencies = [
"fastrand",
"getrandom",
"once_cell",
"rustix",
"windows-sys",
]
[[package]]
name = "terminal_size"
version = "0.4.4"
@@ -822,7 +937,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "230a1b821ccbd75b185820a1f1ff7b14d21da1e442e22c0863ea5f08771a8874"
dependencies = [
"rustix",
"windows-sys",
"windows-sys 0.61.2",
]
[[package]]
@@ -866,6 +981,25 @@ dependencies = [
"zerovec",
]
[[package]]
name = "tungstenite"
version = "0.30.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e48ac77174b19c110a50ab2128b24215ac9cb40e0e12e093fb602d175c569d22"
dependencies = [
"bytes",
"data-encoding",
"http",
"httparse",
"log",
"rand",
"rustls",
"rustls-pki-types",
"sha1",
"thiserror",
"webpki-roots 0.26.11",
]
[[package]]
name = "typenum"
version = "1.20.1"
@@ -896,12 +1030,24 @@ version = "0.2.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "673aac59facbab8a9007c7f6108d11f63b603f7cabff99fabf650fea5c32b861"
[[package]]
name = "untrusted"
version = "0.9.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8ecb6da28b8a351d773b68d5825ac39017e680750f980f3a1a85cd8dd28a47c1"
[[package]]
name = "utf8_iter"
version = "1.0.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b6c140620e7ffbb22c2dee59cafe6084a59b5ffc27a8859a5f0d494b5d52b6be"
[[package]]
name = "wasi"
version = "0.11.1+wasi-snapshot-preview1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ccf3ec651a847eb01de73ccad15eb7d99f80485de043efb2f370cd654f4ea44b"
[[package]]
name = "wasm-bindgen"
version = "0.2.129"
@@ -947,6 +1093,24 @@ dependencies = [
"unicode-ident",
]
[[package]]
name = "webpki-roots"
version = "0.26.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "521bc38abb08001b01866da9f51eb7c5d647a19260e00054a8c7fd5f9e57f7a9"
dependencies = [
"webpki-roots 1.0.11",
]
[[package]]
name = "webpki-roots"
version = "1.0.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f5f280bbb2b287bc8e7de7e3852945a0ff50d1bbcb5ed42b6c7a568c546df171"
dependencies = [
"rustls-pki-types",
]
[[package]]
name = "windows-core"
version = "0.62.2"
@@ -1006,6 +1170,15 @@ dependencies = [
"windows-link",
]
[[package]]
name = "windows-sys"
version = "0.52.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "282be5f36a8ce781fad8c8ae18fa3f9beff57ec1b52cb3de0789201425d9a33d"
dependencies = [
"windows-targets",
]
[[package]]
name = "windows-sys"
version = "0.61.2"
@@ -1015,6 +1188,70 @@ dependencies = [
"windows-link",
]
[[package]]
name = "windows-targets"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9b724f72796e036ab90c1021d4780d4d3d648aca59e491e6b98e725b84e99973"
dependencies = [
"windows_aarch64_gnullvm",
"windows_aarch64_msvc",
"windows_i686_gnu",
"windows_i686_gnullvm",
"windows_i686_msvc",
"windows_x86_64_gnu",
"windows_x86_64_gnullvm",
"windows_x86_64_msvc",
]
[[package]]
name = "windows_aarch64_gnullvm"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "32a4622180e7a0ec044bb555404c800bc9fd9ec262ec147edd5989ccd0c02cd3"
[[package]]
name = "windows_aarch64_msvc"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "09ec2a7bb152e2252b53fa7803150007879548bc709c039df7627cabbd05d469"
[[package]]
name = "windows_i686_gnu"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8e9b5ad5ab802e97eb8e295ac6720e509ee4c243f69d781394014ebfe8bbfa0b"
[[package]]
name = "windows_i686_gnullvm"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0eee52d38c090b3caa76c563b86c3a4bd71ef1a819287c19d586d7334ae8ed66"
[[package]]
name = "windows_i686_msvc"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "240948bc05c5e7c6dabba28bf89d89ffce3e303022809e73deaefe4f6ec56c66"
[[package]]
name = "windows_x86_64_gnu"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "147a5c80aabfbf0c7d901cb5895d1de30ef2907eb21fbbab29ca94c5b08b1a78"
[[package]]
name = "windows_x86_64_gnullvm"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "24d5b23dc417412679681396f2b49f3de8c1473deb516bd34410872eff51ed0d"
[[package]]
name = "windows_x86_64_msvc"
version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "589f6da84c646204747d1270a2a5661ea66ed1cced2631d546fdfb155959f9ec"
[[package]]
name = "writeable"
version = "0.6.4"
@@ -1065,6 +1302,12 @@ dependencies = [
"synstructure",
]
[[package]]
name = "zeroize"
version = "1.9.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e13084392c5e4bc371903e2935a5eaeed24905a7511356b883835e18a78f6879"
[[package]]
name = "zerotrie"
version = "0.2.5"
+4 -1
View File
@@ -8,8 +8,11 @@ build = "src/util/build.rs"
anyhow = "1"
chrono = "0.4"
crossbeam = "0.8.4"
# Not used directly: tungstenite leaves rustls's crypto provider to the app, and
# rustls panics at runtime unless exactly one (here ring) is enabled.
rustls = { version = "0.23", default-features = false, features = ["ring"] }
serde = { version = "1", features = ["derive"] }
tempfile = "3"
tungstenite = { version = "0.30", features = ["rustls-tls-webpki-roots"] }
[build-dependencies]
ergo-sbe = "=0.1.16"
+35
View File
@@ -0,0 +1,35 @@
# 0001. Concurrency model: threads, shared-nothing
## Context
The collector moves a firehose of websocket messages from many sockets to a
single output sink. This is the project's central bet (see
[../project.md](../project.md)): it must be correct (no dropped messages, no
races) and time-accurate (receive time stamped at the socket edge) before it is
fast. The runtime is the one foundational, costly-to-change choice. The template
already uses `std` threads + `crossbeam` channels and pulls in no tokio.
## Decision
V1 uses **OS threads**, not an async runtime — a sync websocket client
(`tungstenite`), with one read loop per socket. State is **shared-nothing**: no
mutexes. Each pipeline stage either owns its state on a single thread or is
sharded so each shard owns its slice, and stages communicate over channels.
Async (tokio) is deferred as a possible optimization, re-evaluated at the start
of extension 1 (M4), when socket-management complexity (reconnect, timeouts,
redundancy) makes async ergonomics worth their cost.
## Why
Thread-per-socket is comfortable at this scale (symbols shard across tens of
sockets, not thousands) and is the simplest, most readable option — correctness
first, speed later. Shared-nothing + channels removes lock contention and data
races by construction, and keeps the runtime pluggable: only the socket layer is
runtime-specific, so a later async swap is localized to that boundary rather than
a pipeline rewrite. Deferring async is therefore safe through the base challenge.
Trade-off: thread-per-socket does not scale to very large socket counts, and the
extension-era socket-management logic (select / timeout / reconnect) is where
threads and async diverge most — hence the explicit M4 re-evaluation point rather
than "never."
+68 -5
View File
@@ -6,8 +6,71 @@ A delta carries the **new total quantity** at a price level, not a change to it.
never add. `quantity: 0` means delete the level. (Schema confirms: "absolute
available quantity".)
## Sequence numbers are the thread you must not drop
Each symbol's messages carry a monotonically increasing `seq`. If you receive a
gap (e.g. 101 then 103), you missed a message, so your in-memory book is now
wrong. You cannot patch it — throw the book away and re-snapshot that symbol.
This is the "completeness" requirement made mechanical.
## A missed message means a wrong book: rebuild it
If a message is lost, the in-memory book is silently wrong and cannot be
patched — throw it away and re-snapshot that symbol. This is the
"completeness" requirement made mechanical. *How* a loss is detected depends on
the exchange: a sequence-number gap where one exists (Kalshi, to verify in M6),
something else where it doesn't — Polymarket has no sequence number (below).
The challenge brief prescribes no mechanism; it only asks for complete data.
## Polymarket market channel (observed 2026-11-04)
From a 60 s run of the M1 spike on one market, both tokens subscribed, and the
[market channel docs](https://docs.polymarket.com/market-data/websocket/market-channel.md)
(read 2026-11-04).
- **Endpoint:** `wss://ws-subscriptions-clob.polymarket.com/ws/market`, public,
no auth. Subscribe: `{"type":"market","assets_ids":["<token id>", ...]}`.
On an open connection, `{"assets_ids":[...],"operation":"subscribe"}` (or
`"unsubscribe"`) changes the token set without reconnecting.
- **`custom_feature_enabled: true`** in the subscribe message adds
`best_bid_ask`, `new_market` and `market_resolved` events — candidate signals
for symbol discovery (M3), possibly replacing REST polling.
- **Symbol = token.** A market (`market`, a `0x…` condition id) has two tokens
(`asset_id`: Yes, No), each with its own book. The two books are **exact
mirrors**: Yes had only bids (best 0.999), No only asks (best 0.001), and
every change arrives as a pair in one message — Yes `BUY p` with No `SELL 1−p`,
same size.
- **Frames:** on subscribe, one frame holding a JSON **array** of `book`
messages (one per token); afterwards, one **object** per frame. A frame's text
can end with a newline.
- **Full books also arrive mid-stream.** After a trade the server sent, all with
one timestamp: the `price_change`, then a full `book` object for each token,
then a `last_trade_price` — a natural resync point.
- **`book`** (snapshot): `event_type`, `market`, `asset_id`, `timestamp`,
`hash`, `bids`, `asks`, `tick_size`, `last_trade_price`. `bids`/`asks` are lists
of `{price, size}` sorted **ascending by price**, so the best bid is the
*last* bid. Its `timestamp` is the time of the book's **last change**, not the
send time: 19 s old on receipt.
- **`price_change`** (delta): `event_type`, `market`, `timestamp`,
`price_changes[]`, each entry `{asset_id, price, size, side, hash, best_bid,
best_ask}`. `side` is `BUY` (bid) or `SELL` (ask); `size` is the absolute
level size (docs: `"0"` removes the level).
- **Numbers are decimal strings** (`"0.77"`, `"26350.28"`): parse them exactly
into fixed point (`1_000_000` = 1), never through a float.
- **Timestamps are milliseconds** since the epoch (the schema stores
microseconds).
- **No sequence number**, confirmed by the docs: none, and no option enables
one, for the market channel (the `seq` in Polymarket's docs belongs to
PolyBolt, a separate price feed). The docs describe no gap detection or
resync either. Candidates for loss detection, to settle in M2:
- the `hash` (40 hex chars, SHA-1 length) on every book and every change. A
change's `hash` equals that of the `book` sent right after it for the same
token, so it identifies the **full book state after the change**: a full
check if we can reproduce it from our own book. The docs don't say how it is
computed (M2's opening question);
- `best_bid`/`best_ask` on every change — a cheap top-of-book check after
applying it.
- **Keepalive (docs):** send the text frame `PING` every 10 s; the server
answers `PONG`. The spike survived 60 s without it, but it is required.
- **`last_trade_price`:** `asset_id`, `price`, `size`, `side`, `fee_rate_bps`,
`timestamp`, `transaction_hash` (one per trade, for one token).
- **Not observed:** `tick_size_change` and the custom-feature events.
## Clock accuracy bounds time accuracy
`received_at` comes from the local clock, so it carries the clock's error. On
2026-11-04 the dev machine ran ~30 ms slow (`sntp`: +30.6 ms ± 13 ms), which
made raw `received_at − exchange timestamp` read −14 to +41 ms; corrected, the
real lag was ~+16 to +71 ms. Latency figures need a synced clock, or the offset
measured and reported alongside them.
+40 -36
View File
@@ -37,10 +37,9 @@ saves the data.
The interacting decisions it is made of:
- **Threading / runtime:** threads + channels vs an async runtime. The template
already uses `std` threads + bounded `crossbeam` channels and pulls in no
tokio — a strong hint toward threads + channels. Pivotal sub-decision, settled
in the first sprint.
- **Threading / runtime:** settled in sprint 01 — OS threads + channels,
shared-nothing, async re-evaluated at M4
([ADR 0001](adr/0001-concurrency-model.md)).
- **Stamping point:** in the socket-read path, immediately, so as little as
possible sits between the wire and the receive timestamp.
- **Book ownership:** shard symbols so each book is owned by exactly one worker
@@ -51,8 +50,8 @@ The interacting decisions it is made of:
- **Output serialization:** one writer owns the single sink and consumes from
all workers.
- **Redundancy fit:** multiple sockets feeding one symbol's worker, which
dedups / selects by source + seq — the topology must leave room for this from
the start.
dedups / selects by source — the topology must leave room for this from the
start.
## Requirements the pipeline must satisfy
@@ -62,8 +61,11 @@ must guarantee:
1. **Receive time stamped at the socket edge**, before any queue or processing.
2. **A per-symbol book folded from absolute deltas** (overwrite the level;
quantity 0 deletes it).
3. **Sequence-gap detection → discard-and-re-snapshot** (a gap means the book is
silently corrupt; rebuilding is the only safe response).
3. **Loss detection → discard-and-re-snapshot** (a lost message means the book
is silently corrupt; rebuilding is the only safe response). How a loss is
detected depends on the exchange: a sequence gap where one exists; Polymarket
has none, so its `hash` or best bid/ask instead
([notes.md](notes.md)) — settled in M2.
These are proven first, before redundancy or scale.
@@ -90,12 +92,13 @@ function, symbols shardable later) are also the simplest, clearest design.
not.** A channel, a function boundary, or a `trait` with two real impls keeps
the door open; the clever internals are what we refuse.
- **Never simplified away, even in V1:** receive-time stamping at the socket
edge, and gap → re-snapshot. These are correctness, not readability tradeoffs.
edge, and loss → re-snapshot. These are correctness, not readability tradeoffs.
- **If a corner is genuinely cut for speed later** (a global lock, an O(n²)
scan), mark it with a comment naming the ceiling and the upgrade path, so the
debt is visible rather than silently clever.
Restated in the project `CLAUDE.md` at the skeleton milestone (M1).
Their operational form, for writing code, is in the project
[CLAUDE.md](../CLAUDE.md#writing-code).
## Constraints (already decided)
@@ -106,15 +109,15 @@ Restated in the project `CLAUDE.md` at the skeleton milestone (M1).
modules under `src/`. `website/` stays separate as the brief.
- **Scope of markets:** regular markets only; MVEs excluded; websocket exchanges
only.
- **Output destination** (settled in sprint 01): `AlignedSbeWriter` takes a file
path, and stdout is just the path `/dev/stdout` (or a tiny writer tweak), so
one code path covers both. Chunks must be 64 MiB: `AlignedSbeReader::open`
reads nothing else.
## Workflow differences
This is a job-challenge repo (cloned template, no Gitea/Woodpecker/release
pipeline). From the global workflow we **keep**: `docs/`, `roadmap/`, sprint
files, feature branches, and `scripts/ci.sh` (fmt + clippy `-D warnings` +
tests + dependency audit). We **drop**: Woodpecker pipelines, `release.sh`,
versioning/changelog, rc images, and the `prod`/`dev`/release branch flow.
Work on `feat/*` branches squashed into `main`.
See the project [CLAUDE.md](../CLAUDE.md#workflow-differences): what we keep
and drop from the global workflow for a job-challenge repo.
## Not in scope (v1)
@@ -133,14 +136,15 @@ discovery ──▶ subscription manager ──▶ N socket readers (stamp recei
│
route by symbol
▼
per-symbol book workers (fold, gap-detect, dedup sources)
per-symbol book workers (fold, loss-detect, dedup sources)
│
▼
single output writer (SBE sink)
```
Each stage and the channels between them are open until settled in a sprint; the
async-vs-threads call is the first one.
Each stage and the channels between them are open until settled in a sprint.
Settled so far: OS threads, shared-nothing, no mutexes
([ADR 0001](adr/0001-concurrency-model.md)).
## Data model, first sketch
@@ -148,15 +152,17 @@ Supporting detail (mostly Claude's to settle), serving the pipeline above.
Not a DB — an in-memory domain plus the fixed SBE output schema. Main entities:
- **Symbol / market** (from discovery): id, trading status, any price/tick
metadata the exchange gives.
- **Symbol** (from discovery): one order book, with its trading status and any
price/tick metadata the exchange gives. On Polymarket that is a token, not a
market: a market has two (Yes / No), each with its own, mirrored book.
- **NormalizedEvent** (internal, exchange-agnostic): the thing each adapter
produces so the core never sees exchange formats. Either a full snapshot or a
single delta, each carrying `symbol`, `seq`, `exchange_timestamp`,
`received_at`, and payload (side/price/quantity, or the full book).
single delta, each carrying `symbol`, `exchange_timestamp`, `received_at`,
whatever the exchange offers for loss detection (a sequence number, a book
hash), and payload (side/price/quantity, or the full book).
- **Book** (one per symbol): bids and asks as ordered price→quantity maps, plus
`last_seq`, last exchange timestamp, and the id of the source currently
trusted (for redundancy).
the last loss-detection marker seen, last exchange timestamp, and the id of the
source currently trusted (for redundancy).
- **Connection** (one websocket): the set of symbols it carries.
- **Source / redundancy:** several connections may feed the same symbol; one is
the trusted source at a time.
@@ -169,16 +175,14 @@ trusted source is chosen and switched, and emitting a fresh snapshot on switch).
## Open questions
- **Output: stdout vs file.** Brief says stdout; `AlignedSbeWriter` takes a file
path and writes fixed 64 MiB zero-padded chunks. Likely: write to a file
(possibly rotated); resolve in the skeleton milestone. *Recommendation: file
output, treat "stdout" as loose wording; confirm.*
- **Sequence semantics per exchange.** Is `seq` per-symbol or per-connection, and
how are snapshots vs deltas sequenced? Differs by exchange; needs doc check per
adapter.
- **Symbol lifecycle signal.** What marks a market "no longer trading" (drops
from the list vs a status field)? Per-exchange, check docs.
- **Loss detection per exchange.** Polymarket has no sequence number: can we
reproduce its book `hash`, or do we check best bid/ask after each change? M2's
opening question. Kalshi's sequence semantics: checked in M6.
- **Symbol lifecycle signal.** What marks a market "no longer trading"?
Polymarket's `new_market` / `market_resolved` events (with
`custom_feature_enabled`) are candidates; settle in M3.
- **File rotation policy.** When to roll to a new output file (size, time), given
64 MiB chunks. Resolve with the output decision.
64 MiB chunks. Resolve in M2 with the output writer.
- **Redundancy source-selection.** How to pick/switch the trusted source
(freshness, message rate, seq lead). Designed in the redundancy sprint.
(freshness, message rate, which source is ahead). Designed in the redundancy
sprint.
+21 -10
View File
@@ -11,31 +11,39 @@ abstraction is *extracted* once a second exchange exists, not designed up front.
## Milestones
1. **Skeleton + connectivity** — *not started*
1. **Skeleton + connectivity** — *done (2026-11-04)*
Project scaffolding inside the template: module layout (replace `main.rs`),
`scripts/ci.sh` (fmt + clippy `-D warnings` + tests + audit), project
CLAUDE.md, chosen websocket/HTTP libraries. Settle the pivotal
async-vs-threads call. Resolve the output destination (stdout vs file).
`scripts/ci.sh` (fmt + clippy `-D warnings` + tests), project CLAUDE.md, the
websocket library (`tungstenite` + `rustls`). Runtime settled: thread-only V1
(async deferred to M4; see [adr/0001-concurrency-model.md](adr/0001-concurrency-model.md)).
Output destination is a non-issue (an output path; `/dev/stdout` for stdout).
Connect to Polymarket and log raw messages.
*Ends:* CI green; raw Polymarket messages flowing.
2. **Single-source book pipeline (Polymarket, subset of symbols)** — *not started*
The central bet in its first real form: normalize events → per-symbol book
(fold absolute deltas) → gap-detect → re-snapshot → SBE output (deltas +
(fold absolute deltas) → loss-detect → re-snapshot → SBE output (deltas +
snapshot every minute, on subscribe, on re-snapshot) → receive-time stamping
→ clean SIGINT. One/few sockets, a fixed subset of symbols. Deterministic
→ clean SIGINT, with the `PING` keepalive every 10 s. Polymarket has no
sequence number, so loss detection (its book `hash`, or best bid/ask) is the
opening question. One/few sockets, a fixed subset of symbols. Deterministic
replay tests + snapshot↔delta reconciliation.
*Ends:* correct SBE output for a symbol subset; tests green.
3. **Full coverage: discovery + sharding (base challenge complete)** — *not started*
REST discovery loop with diffing (subscribe new, unsubscribe gone);
load-balance all live regular-market symbols across N sockets.
Discovery with diffing (subscribe new, unsubscribe gone) — REST polling, or
Polymarket's `new_market` / `market_resolved` events — using subscribe /
unsubscribe on open connections; load-balance all live regular-market symbols
across N sockets.
*Ends:* every live Polymarket symbol tracked — base challenge satisfied for
one exchange.
4. **Extension 1 — redundant websockets** — *not started*
Multiple connections per symbol; choose a trusted source and switch on
loss/quiet/delay; emit a fresh snapshot on every source switch.
**Async re-evaluation checkpoint:** before building this, reassess whether the
socket-management complexity (reconnect/timeout/select, redundancy) warrants
swapping the thread-only runtime for tokio (see ADR 0001).
*Ends:* redundancy demonstrably reduces data loss; source-switch re-snapshots.
5. **Extension 2 — subscription monitoring** — *not started*
@@ -54,11 +62,14 @@ abstraction is *extracted* once a second exchange exists, not designed up front.
7. **Extension 4 — latency experiments + writeup** — *not started*
Experiment with faster connections (first-after-maintenance, connection
racing, source-port tricks) on both exchanges; document findings in a
markdown file.
markdown file, with the local clock's offset measured alongside (the dev
machine ran ~30 ms slow; see notes.md).
*Ends:* experiments documented.
8. **Polish + submission** — *not started*
Final review, README for the solution, packaging/handoff for the challenge.
Final review, README for the solution (the template's README still says the
helpers are "demonstrated in `main.rs`", whose demo M1 replaced),
packaging/handoff for the challenge.
*Ends:* ready to submit.
## Why this order
+171
View File
@@ -0,0 +1,171 @@
# 01. Skeleton + connectivity
**Status: done (2026-11-04)**
Stand up the solution skeleton inside the template and prove we can connect to
Polymarket's public market websocket — the foundation the correctness pipeline
(M2) is built on. No order book, normalization, or SBE output yet. Design: see
[../docs/project.md](../docs/project.md) (central bet, design principles,
pipeline sketch).
## Concurrency model
- **Thread-only V1** (sync `tungstenite`). Async (tokio) is a possible future
optimization, re-evaluated at the start of extension 1 (M4), where
socket-management complexity (reconnect / timeout / redundancy) makes async
ergonomics matter. The swap stays cheap because the socket layer sits behind a
channel boundary and everything downstream is runtime-agnostic.
- **Shared-nothing, no mutexes.** Any stage with potential contention is sharded
(each shard owns its slice) or limited to a single owning thread, with channels
between stages.
## Module layout
Only modules for code we write — no speculative scaffolding: `main.rs`
(entry / wiring) + a `polymarket` module for the connect / subscribe / log spike.
`book` / `output` / `discovery` appear in the milestones that need them.
## The connectivity spike
- Connect to `wss://ws-subscriptions-clob.polymarket.com/ws/market` (public, no
auth).
- Subscribe by hardcoded asset/token id(s), message shape roughly
`{"type":"market","assets_ids":["<id>"]}` — exact schema confirmed against live
behavior during the spike. Find one live, active market id first.
- Read loop; **stamp receive time at the socket edge**; log raw frames.
- Observe and record message shapes in [../docs/notes.md](../docs/notes.md): the
`book` snapshot (bids/asks structure), `price_change` (`size:"0"` = deletion),
and crucially **whether Polymarket provides a per-message sequence number, a
book hash, and/or an exchange timestamp** — these drive the M2 gap-detection
design and are a known open risk for the central bet.
## Libraries
`tungstenite` (sync websocket) + `rustls` TLS, locked in `Cargo.lock` (caret
ranges in `Cargo.toml`, the template's style). `rustls`
is also a direct dependency with only its `ring` feature: tungstenite leaves the
crypto provider to the application, and rustls panics at runtime ("Could not
automatically determine the process-level CryptoProvider") unless exactly one is
enabled. `ring` over `aws-lc-rs`, which needs a C/cmake build. No other
dependencies this sprint: subscribe is a fixed JSON string and frames are logged
as raw text, so no JSON crate, no HTTP client, no CLI.
## CI
`scripts/ci.sh`: `cargo fmt --check`,
`cargo clippy --all-targets -- -D warnings`, `cargo test` — each on its own line (or `|| exit 1`), never `&&`-chained, so a
failing check can't hide. No dependency audit (see Decisions). Woodpecker /
release files unused (workflow differences in project.md).
## Tests
None this sprint: nothing with logic worth testing yet, and connectivity isn't
unit-testable. `cargo test` runs and passes trivially. Soundness/completeness
tests begin in M2.
## Decisions
- **Thread-only V1; async deferred to a possible M4 optimization.** Thread-per-
socket is comfortable at tens of sockets and is the simplest, most readable
option; the channel-based shared-nothing design keeps a later async swap
localized to the socket layer, so deferring is safe through the base challenge.
*Rejected:* async/tokio or hybrid now — speculative complexity before the
extensions need it.
- **No mutexes; shard-or-single-thread + channels.** Avoids lock contention and
data races by construction, and is what makes the runtime pluggable. *Rejected:*
shared state behind `Mutex` — simpler to write, but contention and lock risk on
the hot path.
- **`tungstenite` + `rustls`.** The standard sync WS crate; rustls avoids a
system OpenSSL dependency. *Rejected:* `tokio-tungstenite` (implies async),
native-tls (system dep).
- **No `cargo deny` / audit in CI.** Supply-chain auditing targets deployed
products with users; this is a throwaway challenge, so CI targets application
quality, not prod-grade. *Rejected:* keeping the audit — noise, and the
unmaintained-but-not-vulnerable RUSTSEC-2024-0449 lives in vendor codegen we
don't ship, for no benefit here.
- **Output destination settled at M2, not now.** The mandatory `AlignedSbeWriter`
takes a file path; a path can name stdout (`/dev/stdout`), or a tiny writer
tweak. 64 MiB chunks are mandatory to stay readable by the template's
`AlignedSbeReader::open`. *Rejected:* a generic sink abstraction — unnecessary.
- **No CLI this sprint.** Everything hardcoded for the spike; a config lib would
be speculative. *Rejected:* adding `clap` now.
## Not in this sprint
- Order book, event normalization, SBE output.
- REST symbol discovery and the subscribe/unsubscribe diff loop.
- Sharding across multiple sockets.
- Redundancy, monitoring, SIGINT handling beyond the process default.
- Kalshi, the generic exchange abstraction, latency work.
## Done when
- `scripts/ci.sh` is green (fmt + clippy `-D warnings` + test).
- The program connects to Polymarket's market channel, subscribes to the
hardcoded market, and logs raw frames to the terminal.
- Polymarket's message shapes — including whether a sequence number, book hash,
and/or exchange timestamp exist — are recorded in `docs/notes.md`.
- The project `CLAUDE.md` is written: what it is, layout, commands, environment
traps, design principles, workflow differences.
## Outcome
**Built.** `src/main.rs` now runs `src/polymarket.rs`, which connects to
Polymarket's public market channel, subscribes to both tokens of one market,
stamps the receive time right after each socket read, and logs every frame. Run
against the live feed (60 s, then 15 s at close), it received the opening
snapshot, price changes, a mid-stream book resend, and a trade. `scripts/ci.sh`
(fmt, clippy `--all-targets -D warnings`, test) is green, and was checked to fail
and stop on a violation. Also: the project `CLAUDE.md`, ADR 0001, and the market
channel's message shapes in `docs/notes.md`.
**Diverged from the plan, and why.**
- A direct `rustls` dependency with only `ring`: tungstenite leaves the crypto
provider to the application, and rustls panicked at the first TLS connection
without one. An echo-server smoke test caught it before Polymarket was
reachable.
- The subscribe message carries both tokens of the market, to see how the Yes and
No books relate (they are mirrors).
- `tempfile` removed: only the template's demo used it.
- `## Workflow differences` moved from `docs/project.md` into `CLAUDE.md`, where
the workflow's skills look for it.
- Requirement 3 reworded from "sequence-gap detection" to "loss detection" in
`docs/project.md` and `CLAUDE.md`. The sequence-number assumption came from
Claude's early explanation, not from the brief, and Polymarket has none; the
mechanism stays open.
- Both exchanges turned out DNS-blocked on the dev network. The user pins their
hosts with `../host.sh` (workspace root, outside the repo). Not planned; most
of the sprint's unplanned time went here.
**Found, for later milestones** (in `docs/notes.md` and the roadmap): no
sequence number on Polymarket, and a change's `hash` equals the next book's
(M2); a `PING` keepalive every 10 s is required (M2); a symbol is a token, and the
Yes/No books mirror each other (M2's data model); `new_market` /
`market_resolved` events and subscribe/unsubscribe on an open connection (M3);
the local clock runs ~30 ms slow (M7).
**Decided at close (the user):** research into how the `hash` is computed waits
for M2.
**Left out.** The template's README still says the helpers are "demonstrated in
`main.rs`" (moved to M8, which writes the solution README). The hardcoded market
closes 2026-11-04, so rerunning the spike needs a fresh token id (its comment
says so); M2 picks its own subset and M3 replaces hardcoding.
### Retrospective
- **Went well:** testing the transport against a neutral echo server caught the
rustls panic while the exchange was unreachable. Checking the docs' claims
against the live feed found what the docs don't say: books resent mid-stream,
and a change's hash identifying the next book.
- **Went wrong:** the network block surfaced only at implementation; a
reachability check during planning would have found it a day earlier. And a
general pattern (exchanges carry sequence numbers) entered the central bet's
requirements as if it came from the brief; the live feed and the user's
challenge caught it, not a review.
- **Tested enough?** No tests, as agreed: nothing with logic yet. Not covered: a
long-running connection (keepalive), reconnects, malformed frames — all M2.
`host.sh` was tested on copies, then by its own check on the real file.
- **Next time:** before a claim goes into a requirement, say where it comes from
(the brief, the user, or a general pattern to verify); check the reachability
of external services when planning.
Executable
+10
View File
@@ -0,0 +1,10 @@
#!/bin/sh
# The single CI entry point; run it before every commit.
# One check per line: `set -e` doesn't apply on the left of `&&` / `||`, so
# chaining would let a failing check pass silently.
set -eu
cd "$(dirname "$0")/.."
cargo fmt --check
cargo clippy --all-targets -- -D warnings
cargo test
+2 -63
View File
@@ -1,66 +1,5 @@
use chrono::DateTime;
use engineering_challenge::Side;
use engineering_challenge::util::dtos::challenge_orderbook::{
AnyMessage, PriceLevelUpdateDomain, SnapshotAsksEntryDomain, SnapshotBidsEntryDomain,
SnapshotDomain,
};
use engineering_challenge::util::{AlignedSbeReader, AlignedSbeWriter, OrderbookSchema};
mod polymarket;
const CHUNK_SIZE_64_MEBIBYTES: usize = 64 << 20;
/// This is a dummy example where we define a snapshot and price level update,
/// write them to a file using [`AlignedSbeWriter`], and then read them back
/// from the same file using [`AlignedSbeReader`], asserting that what we read
/// back is the exact same as what we wrote.
///
/// You should replace this main function with your actual solution, and split
/// up your code into modules in `src/` in a way that makes sense to you.
fn main() -> anyhow::Result<()> {
let timestamp = DateTime::from_timestamp_micros(1_700_000_000_000_000).unwrap();
let symbol = "EXAMPLE".to_owned();
let snapshot = SnapshotDomain {
received_at: timestamp,
exchange_timestamp: timestamp,
bids: vec![SnapshotBidsEntryDomain {
price_level: 400_000,
quantity: 2_000_000,
}],
asks: vec![SnapshotAsksEntryDomain {
price_level: 600_000,
quantity: 3_500_000,
}],
symbol: symbol.clone(),
};
let update = PriceLevelUpdateDomain {
backtest_timestamp: timestamp,
exchange_timestamp: timestamp,
price_level: 400_000,
quantity: 1_250_000,
side: Side::Buy,
symbol: symbol.clone(),
};
let file = tempfile::NamedTempFile::new()?;
let mut writer = AlignedSbeWriter::create(file.path(), CHUNK_SIZE_64_MEBIBYTES)?;
writer.write_msg(&snapshot)?;
writer.write_msg(&update)?;
writer.flush()?;
drop(writer);
let mut reader = AlignedSbeReader::<OrderbookSchema>::open(file.path())?;
let Some(AnyMessage::Snapshot(decoded)) = reader.read_msg()? else {
anyhow::bail!("expected snapshot");
};
assert_eq!(SnapshotDomain::try_from_decoder(decoded)?, snapshot);
let Some(AnyMessage::PriceLevelUpdate(decoded)) = reader.read_msg()? else {
anyhow::bail!("expected price level update");
};
assert_eq!(PriceLevelUpdateDomain::try_from_decoder(decoded)?, update);
assert!(reader.read_msg()?.is_none());
println!("Verification success!");
Ok(())
polymarket::run_spike()
}
+33
View File
@@ -0,0 +1,33 @@
//! Connectivity spike: connect to Polymarket's public market channel, subscribe
//! to one market, and log every raw frame with its receive time.
use chrono::{SecondsFormat, Utc};
use tungstenite::Message;
const MARKET_CHANNEL_URL: &str = "wss://ws-subscriptions-clob.polymarket.com/ws/market";
/// Both tokens (Yes, No) of one market, hardcoded for the spike (discovery comes
/// in M3): "2026 Balance of Power: D Senate, D House", which closes 2026-11-04,
/// so pick a fresh market to rerun this.
const SUBSCRIBE_MESSAGE: &str = r#"{"type":"market","assets_ids":["34722410608062854697106861099776685947172185964394483545370684749662285977831","24483563228660028121020430722896740519524131851526579619586176931221901842798"]}"#;
/// Connects, sends [`SUBSCRIBE_MESSAGE`], and logs frames until the server
/// closes the connection.
pub fn run_spike() -> anyhow::Result<()> {
let (mut socket, _response) = tungstenite::connect(MARKET_CHANNEL_URL)?;
socket.send(Message::text(SUBSCRIBE_MESSAGE))?;
loop {
let message = socket.read()?;
// Stamp as soon as the frame is off the wire, before any processing.
let received_at = Utc::now().to_rfc3339_opts(SecondsFormat::Micros, true);
match message {
Message::Text(text) => println!("{received_at} {text}"),
Message::Close(frame) => {
println!("{received_at} close {frame:?}");
return Ok(());
}
other => println!("{received_at} {other:?}"),
}
}
}