feat(polymarket): skeleton and market-channel connectivity spike

Replaces the template's demo main.rs with a spike that connects to
Polymarket's public market channel, subscribes to one market, and logs every
frame with its receive time. Adds scripts/ci.sh (fmt, clippy, tests), the
project CLAUDE.md, ADR 0001 (OS threads, shared-nothing), and the market
channel's message shapes in docs/notes.md: no sequence number, a book hash on
every change, mirrored Yes/No books.
This commit is contained in:
pascal committed 2026-11-04 14:47:14 +01:00
1 parent adcf205f0b
commit 0e687e7eba
11 files changed
+713 -138

No files matched your search

+35
View File
@@ -0,0 +1,35 @@
# 0001. Concurrency model: threads, shared-nothing
## Context
The collector moves a firehose of websocket messages from many sockets to a
single output sink. This is the project's central bet (see
[../project.md](../project.md)): it must be correct (no dropped messages, no
races) and time-accurate (receive time stamped at the socket edge) before it is
fast. The runtime is the one foundational, costly-to-change choice. The template
already uses `std` threads + `crossbeam` channels and pulls in no tokio.
## Decision
V1 uses **OS threads**, not an async runtime — a sync websocket client
(`tungstenite`), with one read loop per socket. State is **shared-nothing**: no
mutexes. Each pipeline stage either owns its state on a single thread or is
sharded so each shard owns its slice, and stages communicate over channels.
Async (tokio) is deferred as a possible optimization, re-evaluated at the start
of extension 1 (M4), when socket-management complexity (reconnect, timeouts,
redundancy) makes async ergonomics worth their cost.
## Why
Thread-per-socket is comfortable at this scale (symbols shard across tens of
sockets, not thousands) and is the simplest, most readable option — correctness
first, speed later. Shared-nothing + channels removes lock contention and data
races by construction, and keeps the runtime pluggable: only the socket layer is
runtime-specific, so a later async swap is localized to that boundary rather than
a pipeline rewrite. Deferring async is therefore safe through the base challenge.
Trade-off: thread-per-socket does not scale to very large socket counts, and the
extension-era socket-management logic (select / timeout / reconnect) is where
threads and async diverge most — hence the explicit M4 re-evaluation point rather
than "never."
+68 -5
View File
@@ -6,8 +6,71 @@ A delta carries the **new total quantity** at a price level, not a change to it.
never add. `quantity: 0` means delete the level. (Schema confirms: "absolute
available quantity".)
## Sequence numbers are the thread you must not drop
Each symbol's messages carry a monotonically increasing `seq`. If you receive a
gap (e.g. 101 then 103), you missed a message, so your in-memory book is now
wrong. You cannot patch it — throw the book away and re-snapshot that symbol.
This is the "completeness" requirement made mechanical.
## A missed message means a wrong book: rebuild it
If a message is lost, the in-memory book is silently wrong and cannot be
patched — throw it away and re-snapshot that symbol. This is the
"completeness" requirement made mechanical. *How* a loss is detected depends on
the exchange: a sequence-number gap where one exists (Kalshi, to verify in M6),
something else where it doesn't — Polymarket has no sequence number (below).
The challenge brief prescribes no mechanism; it only asks for complete data.
## Polymarket market channel (observed 2026-11-04)
From a 60 s run of the M1 spike on one market, both tokens subscribed, and the
[market channel docs](https://docs.polymarket.com/market-data/websocket/market-channel.md)
(read 2026-11-04).
- **Endpoint:** `wss://ws-subscriptions-clob.polymarket.com/ws/market`, public,
no auth. Subscribe: `{"type":"market","assets_ids":["<token id>", ...]}`.
On an open connection, `{"assets_ids":[...],"operation":"subscribe"}` (or
`"unsubscribe"`) changes the token set without reconnecting.
- **`custom_feature_enabled: true`** in the subscribe message adds
`best_bid_ask`, `new_market` and `market_resolved` events — candidate signals
for symbol discovery (M3), possibly replacing REST polling.
- **Symbol = token.** A market (`market`, a `0x…` condition id) has two tokens
(`asset_id`: Yes, No), each with its own book. The two books are **exact
mirrors**: Yes had only bids (best 0.999), No only asks (best 0.001), and
every change arrives as a pair in one message — Yes `BUY p` with No `SELL 1−p`,
same size.
- **Frames:** on subscribe, one frame holding a JSON **array** of `book`
messages (one per token); afterwards, one **object** per frame. A frame's text
can end with a newline.
- **Full books also arrive mid-stream.** After a trade the server sent, all with
one timestamp: the `price_change`, then a full `book` object for each token,
then a `last_trade_price` — a natural resync point.
- **`book`** (snapshot): `event_type`, `market`, `asset_id`, `timestamp`,
`hash`, `bids`, `asks`, `tick_size`, `last_trade_price`. `bids`/`asks` are lists
of `{price, size}` sorted **ascending by price**, so the best bid is the
*last* bid. Its `timestamp` is the time of the book's **last change**, not the
send time: 19 s old on receipt.
- **`price_change`** (delta): `event_type`, `market`, `timestamp`,
`price_changes[]`, each entry `{asset_id, price, size, side, hash, best_bid,
best_ask}`. `side` is `BUY` (bid) or `SELL` (ask); `size` is the absolute
level size (docs: `"0"` removes the level).
- **Numbers are decimal strings** (`"0.77"`, `"26350.28"`): parse them exactly
into fixed point (`1_000_000` = 1), never through a float.
- **Timestamps are milliseconds** since the epoch (the schema stores
microseconds).
- **No sequence number**, confirmed by the docs: none, and no option enables
one, for the market channel (the `seq` in Polymarket's docs belongs to
PolyBolt, a separate price feed). The docs describe no gap detection or
resync either. Candidates for loss detection, to settle in M2:
- the `hash` (40 hex chars, SHA-1 length) on every book and every change. A
change's `hash` equals that of the `book` sent right after it for the same
token, so it identifies the **full book state after the change**: a full
check if we can reproduce it from our own book. The docs don't say how it is
computed (M2's opening question);
- `best_bid`/`best_ask` on every change — a cheap top-of-book check after
applying it.
- **Keepalive (docs):** send the text frame `PING` every 10 s; the server
answers `PONG`. The spike survived 60 s without it, but it is required.
- **`last_trade_price`:** `asset_id`, `price`, `size`, `side`, `fee_rate_bps`,
`timestamp`, `transaction_hash` (one per trade, for one token).
- **Not observed:** `tick_size_change` and the custom-feature events.
## Clock accuracy bounds time accuracy
`received_at` comes from the local clock, so it carries the clock's error. On
2026-11-04 the dev machine ran ~30 ms slow (`sntp`: +30.6 ms ± 13 ms), which
made raw `received_at − exchange timestamp` read −14 to +41 ms; corrected, the
real lag was ~+16 to +71 ms. Latency figures need a synced clock, or the offset
measured and reported alongside them.
+40 -36
View File
@@ -37,10 +37,9 @@ saves the data.
The interacting decisions it is made of:
- **Threading / runtime:** threads + channels vs an async runtime. The template
already uses `std` threads + bounded `crossbeam` channels and pulls in no
tokio — a strong hint toward threads + channels. Pivotal sub-decision, settled
in the first sprint.
- **Threading / runtime:** settled in sprint 01 — OS threads + channels,
shared-nothing, async re-evaluated at M4
([ADR 0001](adr/0001-concurrency-model.md)).
- **Stamping point:** in the socket-read path, immediately, so as little as
possible sits between the wire and the receive timestamp.
- **Book ownership:** shard symbols so each book is owned by exactly one worker
@@ -51,8 +50,8 @@ The interacting decisions it is made of:
- **Output serialization:** one writer owns the single sink and consumes from
all workers.
- **Redundancy fit:** multiple sockets feeding one symbol's worker, which
dedups / selects by source + seq — the topology must leave room for this from
the start.
dedups / selects by source — the topology must leave room for this from the
start.
## Requirements the pipeline must satisfy
@@ -62,8 +61,11 @@ must guarantee:
1. **Receive time stamped at the socket edge**, before any queue or processing.
2. **A per-symbol book folded from absolute deltas** (overwrite the level;
quantity 0 deletes it).
3. **Sequence-gap detection → discard-and-re-snapshot** (a gap means the book is
silently corrupt; rebuilding is the only safe response).
3. **Loss detection → discard-and-re-snapshot** (a lost message means the book
is silently corrupt; rebuilding is the only safe response). How a loss is
detected depends on the exchange: a sequence gap where one exists; Polymarket
has none, so its `hash` or best bid/ask instead
([notes.md](notes.md)) — settled in M2.
These are proven first, before redundancy or scale.
@@ -90,12 +92,13 @@ function, symbols shardable later) are also the simplest, clearest design.
not.** A channel, a function boundary, or a `trait` with two real impls keeps
the door open; the clever internals are what we refuse.
- **Never simplified away, even in V1:** receive-time stamping at the socket
edge, and gap → re-snapshot. These are correctness, not readability tradeoffs.
edge, and loss → re-snapshot. These are correctness, not readability tradeoffs.
- **If a corner is genuinely cut for speed later** (a global lock, an O(n²)
scan), mark it with a comment naming the ceiling and the upgrade path, so the
debt is visible rather than silently clever.
Restated in the project `CLAUDE.md` at the skeleton milestone (M1).
Their operational form, for writing code, is in the project
[CLAUDE.md](../CLAUDE.md#writing-code).
## Constraints (already decided)
@@ -106,15 +109,15 @@ Restated in the project `CLAUDE.md` at the skeleton milestone (M1).
modules under `src/`. `website/` stays separate as the brief.
- **Scope of markets:** regular markets only; MVEs excluded; websocket exchanges
only.
- **Output destination** (settled in sprint 01): `AlignedSbeWriter` takes a file
path, and stdout is just the path `/dev/stdout` (or a tiny writer tweak), so
one code path covers both. Chunks must be 64 MiB: `AlignedSbeReader::open`
reads nothing else.
## Workflow differences
This is a job-challenge repo (cloned template, no Gitea/Woodpecker/release
pipeline). From the global workflow we **keep**: `docs/`, `roadmap/`, sprint
files, feature branches, and `scripts/ci.sh` (fmt + clippy `-D warnings` +
tests + dependency audit). We **drop**: Woodpecker pipelines, `release.sh`,
versioning/changelog, rc images, and the `prod`/`dev`/release branch flow.
Work on `feat/*` branches squashed into `main`.
See the project [CLAUDE.md](../CLAUDE.md#workflow-differences): what we keep
and drop from the global workflow for a job-challenge repo.
## Not in scope (v1)
@@ -133,14 +136,15 @@ discovery ──▶ subscription manager ──▶ N socket readers (stamp recei
│
route by symbol
▼
per-symbol book workers (fold, gap-detect, dedup sources)
per-symbol book workers (fold, loss-detect, dedup sources)
│
▼
single output writer (SBE sink)
```
Each stage and the channels between them are open until settled in a sprint; the
async-vs-threads call is the first one.
Each stage and the channels between them are open until settled in a sprint.
Settled so far: OS threads, shared-nothing, no mutexes
([ADR 0001](adr/0001-concurrency-model.md)).
## Data model, first sketch
@@ -148,15 +152,17 @@ Supporting detail (mostly Claude's to settle), serving the pipeline above.
Not a DB — an in-memory domain plus the fixed SBE output schema. Main entities:
- **Symbol / market** (from discovery): id, trading status, any price/tick
metadata the exchange gives.
- **Symbol** (from discovery): one order book, with its trading status and any
price/tick metadata the exchange gives. On Polymarket that is a token, not a
market: a market has two (Yes / No), each with its own, mirrored book.
- **NormalizedEvent** (internal, exchange-agnostic): the thing each adapter
produces so the core never sees exchange formats. Either a full snapshot or a
single delta, each carrying `symbol`, `seq`, `exchange_timestamp`,
`received_at`, and payload (side/price/quantity, or the full book).
single delta, each carrying `symbol`, `exchange_timestamp`, `received_at`,
whatever the exchange offers for loss detection (a sequence number, a book
hash), and payload (side/price/quantity, or the full book).
- **Book** (one per symbol): bids and asks as ordered price→quantity maps, plus
`last_seq`, last exchange timestamp, and the id of the source currently
trusted (for redundancy).
the last loss-detection marker seen, last exchange timestamp, and the id of the
source currently trusted (for redundancy).
- **Connection** (one websocket): the set of symbols it carries.
- **Source / redundancy:** several connections may feed the same symbol; one is
the trusted source at a time.
@@ -169,16 +175,14 @@ trusted source is chosen and switched, and emitting a fresh snapshot on switch).
## Open questions
- **Output: stdout vs file.** Brief says stdout; `AlignedSbeWriter` takes a file
path and writes fixed 64 MiB zero-padded chunks. Likely: write to a file
(possibly rotated); resolve in the skeleton milestone. *Recommendation: file
output, treat "stdout" as loose wording; confirm.*
- **Sequence semantics per exchange.** Is `seq` per-symbol or per-connection, and
how are snapshots vs deltas sequenced? Differs by exchange; needs doc check per
adapter.
- **Symbol lifecycle signal.** What marks a market "no longer trading" (drops
from the list vs a status field)? Per-exchange, check docs.
- **Loss detection per exchange.** Polymarket has no sequence number: can we
reproduce its book `hash`, or do we check best bid/ask after each change? M2's
opening question. Kalshi's sequence semantics: checked in M6.
- **Symbol lifecycle signal.** What marks a market "no longer trading"?
Polymarket's `new_market` / `market_resolved` events (with
`custom_feature_enabled`) are candidates; settle in M3.
- **File rotation policy.** When to roll to a new output file (size, time), given
64 MiB chunks. Resolve with the output decision.
64 MiB chunks. Resolve in M2 with the output writer.
- **Redundancy source-selection.** How to pick/switch the trusted source
(freshness, message rate, seq lead). Designed in the redundancy sprint.
(freshness, message rate, which source is ahead). Designed in the redundancy
sprint.
+21 -10
View File
@@ -11,31 +11,39 @@ abstraction is *extracted* once a second exchange exists, not designed up front.
## Milestones
1. **Skeleton + connectivity** — *not started*
1. **Skeleton + connectivity** — *done (2026-11-04)*
Project scaffolding inside the template: module layout (replace `main.rs`),
`scripts/ci.sh` (fmt + clippy `-D warnings` + tests + audit), project
CLAUDE.md, chosen websocket/HTTP libraries. Settle the pivotal
async-vs-threads call. Resolve the output destination (stdout vs file).
`scripts/ci.sh` (fmt + clippy `-D warnings` + tests), project CLAUDE.md, the
websocket library (`tungstenite` + `rustls`). Runtime settled: thread-only V1
(async deferred to M4; see [adr/0001-concurrency-model.md](adr/0001-concurrency-model.md)).
Output destination is a non-issue (an output path; `/dev/stdout` for stdout).
Connect to Polymarket and log raw messages.
*Ends:* CI green; raw Polymarket messages flowing.
2. **Single-source book pipeline (Polymarket, subset of symbols)** — *not started*
The central bet in its first real form: normalize events → per-symbol book
(fold absolute deltas) → gap-detect → re-snapshot → SBE output (deltas +
(fold absolute deltas) → loss-detect → re-snapshot → SBE output (deltas +
snapshot every minute, on subscribe, on re-snapshot) → receive-time stamping
→ clean SIGINT. One/few sockets, a fixed subset of symbols. Deterministic
→ clean SIGINT, with the `PING` keepalive every 10 s. Polymarket has no
sequence number, so loss detection (its book `hash`, or best bid/ask) is the
opening question. One/few sockets, a fixed subset of symbols. Deterministic
replay tests + snapshot↔delta reconciliation.
*Ends:* correct SBE output for a symbol subset; tests green.
3. **Full coverage: discovery + sharding (base challenge complete)** — *not started*
REST discovery loop with diffing (subscribe new, unsubscribe gone);
load-balance all live regular-market symbols across N sockets.
Discovery with diffing (subscribe new, unsubscribe gone) — REST polling, or
Polymarket's `new_market` / `market_resolved` events — using subscribe /
unsubscribe on open connections; load-balance all live regular-market symbols
across N sockets.
*Ends:* every live Polymarket symbol tracked — base challenge satisfied for
one exchange.
4. **Extension 1 — redundant websockets** — *not started*
Multiple connections per symbol; choose a trusted source and switch on
loss/quiet/delay; emit a fresh snapshot on every source switch.
**Async re-evaluation checkpoint:** before building this, reassess whether the
socket-management complexity (reconnect/timeout/select, redundancy) warrants
swapping the thread-only runtime for tokio (see ADR 0001).
*Ends:* redundancy demonstrably reduces data loss; source-switch re-snapshots.
5. **Extension 2 — subscription monitoring** — *not started*
@@ -54,11 +62,14 @@ abstraction is *extracted* once a second exchange exists, not designed up front.
7. **Extension 4 — latency experiments + writeup** — *not started*
Experiment with faster connections (first-after-maintenance, connection
racing, source-port tricks) on both exchanges; document findings in a
markdown file.
markdown file, with the local clock's offset measured alongside (the dev
machine ran ~30 ms slow; see notes.md).
*Ends:* experiments documented.
8. **Polish + submission** — *not started*
Final review, README for the solution, packaging/handoff for the challenge.
Final review, README for the solution (the template's README still says the
helpers are "demonstrated in `main.rs`", whose demo M1 replaced),
packaging/handoff for the challenge.
*Ends:* ready to submit.
## Why this order