Archived
feat(polymarket): skeleton and market-channel connectivity spike
Replaces the template's demo main.rs with a spike that connects to Polymarket's public market channel, subscribes to one market, and logs every frame with its receive time. Adds scripts/ci.sh (fmt, clippy, tests), the project CLAUDE.md, ADR 0001 (OS threads, shared-nothing), and the market channel's message shapes in docs/notes.md: no sequence number, a book hash on every change, mirrored Yes/No books.
This commit is contained in:
11 files changed
+713
-138
No files matched your search
@@ -0,0 +1,35 @@
|
||||
# 0001. Concurrency model: threads, shared-nothing
|
||||
|
||||
## Context
|
||||
|
||||
The collector moves a firehose of websocket messages from many sockets to a
|
||||
single output sink. This is the project's central bet (see
|
||||
[../project.md](../project.md)): it must be correct (no dropped messages, no
|
||||
races) and time-accurate (receive time stamped at the socket edge) before it is
|
||||
fast. The runtime is the one foundational, costly-to-change choice. The template
|
||||
already uses `std` threads + `crossbeam` channels and pulls in no tokio.
|
||||
|
||||
## Decision
|
||||
|
||||
V1 uses **OS threads**, not an async runtime — a sync websocket client
|
||||
(`tungstenite`), with one read loop per socket. State is **shared-nothing**: no
|
||||
mutexes. Each pipeline stage either owns its state on a single thread or is
|
||||
sharded so each shard owns its slice, and stages communicate over channels.
|
||||
|
||||
Async (tokio) is deferred as a possible optimization, re-evaluated at the start
|
||||
of extension 1 (M4), when socket-management complexity (reconnect, timeouts,
|
||||
redundancy) makes async ergonomics worth their cost.
|
||||
|
||||
## Why
|
||||
|
||||
Thread-per-socket is comfortable at this scale (symbols shard across tens of
|
||||
sockets, not thousands) and is the simplest, most readable option — correctness
|
||||
first, speed later. Shared-nothing + channels removes lock contention and data
|
||||
races by construction, and keeps the runtime pluggable: only the socket layer is
|
||||
runtime-specific, so a later async swap is localized to that boundary rather than
|
||||
a pipeline rewrite. Deferring async is therefore safe through the base challenge.
|
||||
|
||||
Trade-off: thread-per-socket does not scale to very large socket counts, and the
|
||||
extension-era socket-management logic (select / timeout / reconnect) is where
|
||||
threads and async diverge most — hence the explicit M4 re-evaluation point rather
|
||||
than "never."
|
||||
+68
-5
@@ -6,8 +6,71 @@ A delta carries the **new total quantity** at a price level, not a change to it.
|
||||
never add. `quantity: 0` means delete the level. (Schema confirms: "absolute
|
||||
available quantity".)
|
||||
|
||||
## Sequence numbers are the thread you must not drop
|
||||
Each symbol's messages carry a monotonically increasing `seq`. If you receive a
|
||||
gap (e.g. 101 then 103), you missed a message, so your in-memory book is now
|
||||
wrong. You cannot patch it — throw the book away and re-snapshot that symbol.
|
||||
This is the "completeness" requirement made mechanical.
|
||||
## A missed message means a wrong book: rebuild it
|
||||
If a message is lost, the in-memory book is silently wrong and cannot be
|
||||
patched — throw it away and re-snapshot that symbol. This is the
|
||||
"completeness" requirement made mechanical. *How* a loss is detected depends on
|
||||
the exchange: a sequence-number gap where one exists (Kalshi, to verify in M6),
|
||||
something else where it doesn't — Polymarket has no sequence number (below).
|
||||
The challenge brief prescribes no mechanism; it only asks for complete data.
|
||||
|
||||
## Polymarket market channel (observed 2026-11-04)
|
||||
|
||||
From a 60 s run of the M1 spike on one market, both tokens subscribed, and the
|
||||
[market channel docs](https://docs.polymarket.com/market-data/websocket/market-channel.md)
|
||||
(read 2026-11-04).
|
||||
|
||||
- **Endpoint:** `wss://ws-subscriptions-clob.polymarket.com/ws/market`, public,
|
||||
no auth. Subscribe: `{"type":"market","assets_ids":["<token id>", ...]}`.
|
||||
On an open connection, `{"assets_ids":[...],"operation":"subscribe"}` (or
|
||||
`"unsubscribe"`) changes the token set without reconnecting.
|
||||
- **`custom_feature_enabled: true`** in the subscribe message adds
|
||||
`best_bid_ask`, `new_market` and `market_resolved` events — candidate signals
|
||||
for symbol discovery (M3), possibly replacing REST polling.
|
||||
- **Symbol = token.** A market (`market`, a `0x…` condition id) has two tokens
|
||||
(`asset_id`: Yes, No), each with its own book. The two books are **exact
|
||||
mirrors**: Yes had only bids (best 0.999), No only asks (best 0.001), and
|
||||
every change arrives as a pair in one message — Yes `BUY p` with No `SELL 1−p`,
|
||||
same size.
|
||||
- **Frames:** on subscribe, one frame holding a JSON **array** of `book`
|
||||
messages (one per token); afterwards, one **object** per frame. A frame's text
|
||||
can end with a newline.
|
||||
- **Full books also arrive mid-stream.** After a trade the server sent, all with
|
||||
one timestamp: the `price_change`, then a full `book` object for each token,
|
||||
then a `last_trade_price` — a natural resync point.
|
||||
- **`book`** (snapshot): `event_type`, `market`, `asset_id`, `timestamp`,
|
||||
`hash`, `bids`, `asks`, `tick_size`, `last_trade_price`. `bids`/`asks` are lists
|
||||
of `{price, size}` sorted **ascending by price**, so the best bid is the
|
||||
*last* bid. Its `timestamp` is the time of the book's **last change**, not the
|
||||
send time: 19 s old on receipt.
|
||||
- **`price_change`** (delta): `event_type`, `market`, `timestamp`,
|
||||
`price_changes[]`, each entry `{asset_id, price, size, side, hash, best_bid,
|
||||
best_ask}`. `side` is `BUY` (bid) or `SELL` (ask); `size` is the absolute
|
||||
level size (docs: `"0"` removes the level).
|
||||
- **Numbers are decimal strings** (`"0.77"`, `"26350.28"`): parse them exactly
|
||||
into fixed point (`1_000_000` = 1), never through a float.
|
||||
- **Timestamps are milliseconds** since the epoch (the schema stores
|
||||
microseconds).
|
||||
- **No sequence number**, confirmed by the docs: none, and no option enables
|
||||
one, for the market channel (the `seq` in Polymarket's docs belongs to
|
||||
PolyBolt, a separate price feed). The docs describe no gap detection or
|
||||
resync either. Candidates for loss detection, to settle in M2:
|
||||
- the `hash` (40 hex chars, SHA-1 length) on every book and every change. A
|
||||
change's `hash` equals that of the `book` sent right after it for the same
|
||||
token, so it identifies the **full book state after the change**: a full
|
||||
check if we can reproduce it from our own book. The docs don't say how it is
|
||||
computed (M2's opening question);
|
||||
- `best_bid`/`best_ask` on every change — a cheap top-of-book check after
|
||||
applying it.
|
||||
- **Keepalive (docs):** send the text frame `PING` every 10 s; the server
|
||||
answers `PONG`. The spike survived 60 s without it, but it is required.
|
||||
- **`last_trade_price`:** `asset_id`, `price`, `size`, `side`, `fee_rate_bps`,
|
||||
`timestamp`, `transaction_hash` (one per trade, for one token).
|
||||
- **Not observed:** `tick_size_change` and the custom-feature events.
|
||||
|
||||
## Clock accuracy bounds time accuracy
|
||||
`received_at` comes from the local clock, so it carries the clock's error. On
|
||||
2026-11-04 the dev machine ran ~30 ms slow (`sntp`: +30.6 ms ± 13 ms), which
|
||||
made raw `received_at − exchange timestamp` read −14 to +41 ms; corrected, the
|
||||
real lag was ~+16 to +71 ms. Latency figures need a synced clock, or the offset
|
||||
measured and reported alongside them.
|
||||
+40
-36
@@ -37,10 +37,9 @@ saves the data.
|
||||
|
||||
The interacting decisions it is made of:
|
||||
|
||||
- **Threading / runtime:** threads + channels vs an async runtime. The template
|
||||
already uses `std` threads + bounded `crossbeam` channels and pulls in no
|
||||
tokio — a strong hint toward threads + channels. Pivotal sub-decision, settled
|
||||
in the first sprint.
|
||||
- **Threading / runtime:** settled in sprint 01 — OS threads + channels,
|
||||
shared-nothing, async re-evaluated at M4
|
||||
([ADR 0001](adr/0001-concurrency-model.md)).
|
||||
- **Stamping point:** in the socket-read path, immediately, so as little as
|
||||
possible sits between the wire and the receive timestamp.
|
||||
- **Book ownership:** shard symbols so each book is owned by exactly one worker
|
||||
@@ -51,8 +50,8 @@ The interacting decisions it is made of:
|
||||
- **Output serialization:** one writer owns the single sink and consumes from
|
||||
all workers.
|
||||
- **Redundancy fit:** multiple sockets feeding one symbol's worker, which
|
||||
dedups / selects by source + seq — the topology must leave room for this from
|
||||
the start.
|
||||
dedups / selects by source — the topology must leave room for this from the
|
||||
start.
|
||||
|
||||
## Requirements the pipeline must satisfy
|
||||
|
||||
@@ -62,8 +61,11 @@ must guarantee:
|
||||
1. **Receive time stamped at the socket edge**, before any queue or processing.
|
||||
2. **A per-symbol book folded from absolute deltas** (overwrite the level;
|
||||
quantity 0 deletes it).
|
||||
3. **Sequence-gap detection → discard-and-re-snapshot** (a gap means the book is
|
||||
silently corrupt; rebuilding is the only safe response).
|
||||
3. **Loss detection → discard-and-re-snapshot** (a lost message means the book
|
||||
is silently corrupt; rebuilding is the only safe response). How a loss is
|
||||
detected depends on the exchange: a sequence gap where one exists; Polymarket
|
||||
has none, so its `hash` or best bid/ask instead
|
||||
([notes.md](notes.md)) — settled in M2.
|
||||
|
||||
These are proven first, before redundancy or scale.
|
||||
|
||||
@@ -90,12 +92,13 @@ function, symbols shardable later) are also the simplest, clearest design.
|
||||
not.** A channel, a function boundary, or a `trait` with two real impls keeps
|
||||
the door open; the clever internals are what we refuse.
|
||||
- **Never simplified away, even in V1:** receive-time stamping at the socket
|
||||
edge, and gap → re-snapshot. These are correctness, not readability tradeoffs.
|
||||
edge, and loss → re-snapshot. These are correctness, not readability tradeoffs.
|
||||
- **If a corner is genuinely cut for speed later** (a global lock, an O(n²)
|
||||
scan), mark it with a comment naming the ceiling and the upgrade path, so the
|
||||
debt is visible rather than silently clever.
|
||||
|
||||
Restated in the project `CLAUDE.md` at the skeleton milestone (M1).
|
||||
Their operational form, for writing code, is in the project
|
||||
[CLAUDE.md](../CLAUDE.md#writing-code).
|
||||
|
||||
## Constraints (already decided)
|
||||
|
||||
@@ -106,15 +109,15 @@ Restated in the project `CLAUDE.md` at the skeleton milestone (M1).
|
||||
modules under `src/`. `website/` stays separate as the brief.
|
||||
- **Scope of markets:** regular markets only; MVEs excluded; websocket exchanges
|
||||
only.
|
||||
- **Output destination** (settled in sprint 01): `AlignedSbeWriter` takes a file
|
||||
path, and stdout is just the path `/dev/stdout` (or a tiny writer tweak), so
|
||||
one code path covers both. Chunks must be 64 MiB: `AlignedSbeReader::open`
|
||||
reads nothing else.
|
||||
|
||||
## Workflow differences
|
||||
|
||||
This is a job-challenge repo (cloned template, no Gitea/Woodpecker/release
|
||||
pipeline). From the global workflow we **keep**: `docs/`, `roadmap/`, sprint
|
||||
files, feature branches, and `scripts/ci.sh` (fmt + clippy `-D warnings` +
|
||||
tests + dependency audit). We **drop**: Woodpecker pipelines, `release.sh`,
|
||||
versioning/changelog, rc images, and the `prod`/`dev`/release branch flow.
|
||||
Work on `feat/*` branches squashed into `main`.
|
||||
See the project [CLAUDE.md](../CLAUDE.md#workflow-differences): what we keep
|
||||
and drop from the global workflow for a job-challenge repo.
|
||||
|
||||
## Not in scope (v1)
|
||||
|
||||
@@ -133,14 +136,15 @@ discovery ──▶ subscription manager ──▶ N socket readers (stamp recei
|
||||
│
|
||||
route by symbol
|
||||
▼
|
||||
per-symbol book workers (fold, gap-detect, dedup sources)
|
||||
per-symbol book workers (fold, loss-detect, dedup sources)
|
||||
│
|
||||
▼
|
||||
single output writer (SBE sink)
|
||||
```
|
||||
|
||||
Each stage and the channels between them are open until settled in a sprint; the
|
||||
async-vs-threads call is the first one.
|
||||
Each stage and the channels between them are open until settled in a sprint.
|
||||
Settled so far: OS threads, shared-nothing, no mutexes
|
||||
([ADR 0001](adr/0001-concurrency-model.md)).
|
||||
|
||||
## Data model, first sketch
|
||||
|
||||
@@ -148,15 +152,17 @@ Supporting detail (mostly Claude's to settle), serving the pipeline above.
|
||||
|
||||
Not a DB — an in-memory domain plus the fixed SBE output schema. Main entities:
|
||||
|
||||
- **Symbol / market** (from discovery): id, trading status, any price/tick
|
||||
metadata the exchange gives.
|
||||
- **Symbol** (from discovery): one order book, with its trading status and any
|
||||
price/tick metadata the exchange gives. On Polymarket that is a token, not a
|
||||
market: a market has two (Yes / No), each with its own, mirrored book.
|
||||
- **NormalizedEvent** (internal, exchange-agnostic): the thing each adapter
|
||||
produces so the core never sees exchange formats. Either a full snapshot or a
|
||||
single delta, each carrying `symbol`, `seq`, `exchange_timestamp`,
|
||||
`received_at`, and payload (side/price/quantity, or the full book).
|
||||
single delta, each carrying `symbol`, `exchange_timestamp`, `received_at`,
|
||||
whatever the exchange offers for loss detection (a sequence number, a book
|
||||
hash), and payload (side/price/quantity, or the full book).
|
||||
- **Book** (one per symbol): bids and asks as ordered price→quantity maps, plus
|
||||
`last_seq`, last exchange timestamp, and the id of the source currently
|
||||
trusted (for redundancy).
|
||||
the last loss-detection marker seen, last exchange timestamp, and the id of the
|
||||
source currently trusted (for redundancy).
|
||||
- **Connection** (one websocket): the set of symbols it carries.
|
||||
- **Source / redundancy:** several connections may feed the same symbol; one is
|
||||
the trusted source at a time.
|
||||
@@ -169,16 +175,14 @@ trusted source is chosen and switched, and emitting a fresh snapshot on switch).
|
||||
|
||||
## Open questions
|
||||
|
||||
- **Output: stdout vs file.** Brief says stdout; `AlignedSbeWriter` takes a file
|
||||
path and writes fixed 64 MiB zero-padded chunks. Likely: write to a file
|
||||
(possibly rotated); resolve in the skeleton milestone. *Recommendation: file
|
||||
output, treat "stdout" as loose wording; confirm.*
|
||||
- **Sequence semantics per exchange.** Is `seq` per-symbol or per-connection, and
|
||||
how are snapshots vs deltas sequenced? Differs by exchange; needs doc check per
|
||||
adapter.
|
||||
- **Symbol lifecycle signal.** What marks a market "no longer trading" (drops
|
||||
from the list vs a status field)? Per-exchange, check docs.
|
||||
- **Loss detection per exchange.** Polymarket has no sequence number: can we
|
||||
reproduce its book `hash`, or do we check best bid/ask after each change? M2's
|
||||
opening question. Kalshi's sequence semantics: checked in M6.
|
||||
- **Symbol lifecycle signal.** What marks a market "no longer trading"?
|
||||
Polymarket's `new_market` / `market_resolved` events (with
|
||||
`custom_feature_enabled`) are candidates; settle in M3.
|
||||
- **File rotation policy.** When to roll to a new output file (size, time), given
|
||||
64 MiB chunks. Resolve with the output decision.
|
||||
64 MiB chunks. Resolve in M2 with the output writer.
|
||||
- **Redundancy source-selection.** How to pick/switch the trusted source
|
||||
(freshness, message rate, seq lead). Designed in the redundancy sprint.
|
||||
(freshness, message rate, which source is ahead). Designed in the redundancy
|
||||
sprint.
|
||||
+21
-10
@@ -11,31 +11,39 @@ abstraction is *extracted* once a second exchange exists, not designed up front.
|
||||
|
||||
## Milestones
|
||||
|
||||
1. **Skeleton + connectivity** — *not started*
|
||||
1. **Skeleton + connectivity** — *done (2026-11-04)*
|
||||
Project scaffolding inside the template: module layout (replace `main.rs`),
|
||||
`scripts/ci.sh` (fmt + clippy `-D warnings` + tests + audit), project
|
||||
CLAUDE.md, chosen websocket/HTTP libraries. Settle the pivotal
|
||||
async-vs-threads call. Resolve the output destination (stdout vs file).
|
||||
`scripts/ci.sh` (fmt + clippy `-D warnings` + tests), project CLAUDE.md, the
|
||||
websocket library (`tungstenite` + `rustls`). Runtime settled: thread-only V1
|
||||
(async deferred to M4; see [adr/0001-concurrency-model.md](adr/0001-concurrency-model.md)).
|
||||
Output destination is a non-issue (an output path; `/dev/stdout` for stdout).
|
||||
Connect to Polymarket and log raw messages.
|
||||
*Ends:* CI green; raw Polymarket messages flowing.
|
||||
|
||||
2. **Single-source book pipeline (Polymarket, subset of symbols)** — *not started*
|
||||
The central bet in its first real form: normalize events → per-symbol book
|
||||
(fold absolute deltas) → gap-detect → re-snapshot → SBE output (deltas +
|
||||
(fold absolute deltas) → loss-detect → re-snapshot → SBE output (deltas +
|
||||
snapshot every minute, on subscribe, on re-snapshot) → receive-time stamping
|
||||
→ clean SIGINT. One/few sockets, a fixed subset of symbols. Deterministic
|
||||
→ clean SIGINT, with the `PING` keepalive every 10 s. Polymarket has no
|
||||
sequence number, so loss detection (its book `hash`, or best bid/ask) is the
|
||||
opening question. One/few sockets, a fixed subset of symbols. Deterministic
|
||||
replay tests + snapshot↔delta reconciliation.
|
||||
*Ends:* correct SBE output for a symbol subset; tests green.
|
||||
|
||||
3. **Full coverage: discovery + sharding (base challenge complete)** — *not started*
|
||||
REST discovery loop with diffing (subscribe new, unsubscribe gone);
|
||||
load-balance all live regular-market symbols across N sockets.
|
||||
Discovery with diffing (subscribe new, unsubscribe gone) — REST polling, or
|
||||
Polymarket's `new_market` / `market_resolved` events — using subscribe /
|
||||
unsubscribe on open connections; load-balance all live regular-market symbols
|
||||
across N sockets.
|
||||
*Ends:* every live Polymarket symbol tracked — base challenge satisfied for
|
||||
one exchange.
|
||||
|
||||
4. **Extension 1 — redundant websockets** — *not started*
|
||||
Multiple connections per symbol; choose a trusted source and switch on
|
||||
loss/quiet/delay; emit a fresh snapshot on every source switch.
|
||||
**Async re-evaluation checkpoint:** before building this, reassess whether the
|
||||
socket-management complexity (reconnect/timeout/select, redundancy) warrants
|
||||
swapping the thread-only runtime for tokio (see ADR 0001).
|
||||
*Ends:* redundancy demonstrably reduces data loss; source-switch re-snapshots.
|
||||
|
||||
5. **Extension 2 — subscription monitoring** — *not started*
|
||||
@@ -54,11 +62,14 @@ abstraction is *extracted* once a second exchange exists, not designed up front.
|
||||
7. **Extension 4 — latency experiments + writeup** — *not started*
|
||||
Experiment with faster connections (first-after-maintenance, connection
|
||||
racing, source-port tricks) on both exchanges; document findings in a
|
||||
markdown file.
|
||||
markdown file, with the local clock's offset measured alongside (the dev
|
||||
machine ran ~30 ms slow; see notes.md).
|
||||
*Ends:* experiments documented.
|
||||
|
||||
8. **Polish + submission** — *not started*
|
||||
Final review, README for the solution, packaging/handoff for the challenge.
|
||||
Final review, README for the solution (the template's README still says the
|
||||
helpers are "demonstrated in `main.rs`", whose demo M1 replaced),
|
||||
packaging/handoff for the challenge.
|
||||
*Ends:* ready to submit.
|
||||
|
||||
## Why this order
|
||||
|
||||
Reference in new issue
Block a user