feat(polymarket): skeleton and market-channel connectivity spike

Replaces the template's demo main.rs with a spike that connects to
Polymarket's public market channel, subscribes to one market, and logs every
frame with its receive time. Adds scripts/ci.sh (fmt, clippy, tests), the
project CLAUDE.md, ADR 0001 (OS threads, shared-nothing), and the market
channel's message shapes in docs/notes.md: no sequence number, a book hash on
every change, mirrored Yes/No books.
This commit is contained in:
pascal committed 2026-11-04 14:47:14 +01:00
1 parent adcf205f0b
commit 0e687e7eba
11 files changed
+713 -138

No files matched your search

+40 -36
View File
@@ -37,10 +37,9 @@ saves the data.
The interacting decisions it is made of:
- **Threading / runtime:** threads + channels vs an async runtime. The template
already uses `std` threads + bounded `crossbeam` channels and pulls in no
tokio — a strong hint toward threads + channels. Pivotal sub-decision, settled
in the first sprint.
- **Threading / runtime:** settled in sprint 01 — OS threads + channels,
shared-nothing, async re-evaluated at M4
([ADR 0001](adr/0001-concurrency-model.md)).
- **Stamping point:** in the socket-read path, immediately, so as little as
possible sits between the wire and the receive timestamp.
- **Book ownership:** shard symbols so each book is owned by exactly one worker
@@ -51,8 +50,8 @@ The interacting decisions it is made of:
- **Output serialization:** one writer owns the single sink and consumes from
all workers.
- **Redundancy fit:** multiple sockets feeding one symbol's worker, which
dedups / selects by source + seq — the topology must leave room for this from
the start.
dedups / selects by source — the topology must leave room for this from the
start.
## Requirements the pipeline must satisfy
@@ -62,8 +61,11 @@ must guarantee:
1. **Receive time stamped at the socket edge**, before any queue or processing.
2. **A per-symbol book folded from absolute deltas** (overwrite the level;
quantity 0 deletes it).
3. **Sequence-gap detection → discard-and-re-snapshot** (a gap means the book is
silently corrupt; rebuilding is the only safe response).
3. **Loss detection → discard-and-re-snapshot** (a lost message means the book
is silently corrupt; rebuilding is the only safe response). How a loss is
detected depends on the exchange: a sequence gap where one exists; Polymarket
has none, so its `hash` or best bid/ask instead
([notes.md](notes.md)) — settled in M2.
These are proven first, before redundancy or scale.
@@ -90,12 +92,13 @@ function, symbols shardable later) are also the simplest, clearest design.
not.** A channel, a function boundary, or a `trait` with two real impls keeps
the door open; the clever internals are what we refuse.
- **Never simplified away, even in V1:** receive-time stamping at the socket
edge, and gap → re-snapshot. These are correctness, not readability tradeoffs.
edge, and loss → re-snapshot. These are correctness, not readability tradeoffs.
- **If a corner is genuinely cut for speed later** (a global lock, an O(n²)
scan), mark it with a comment naming the ceiling and the upgrade path, so the
debt is visible rather than silently clever.
Restated in the project `CLAUDE.md` at the skeleton milestone (M1).
Their operational form, for writing code, is in the project
[CLAUDE.md](../CLAUDE.md#writing-code).
## Constraints (already decided)
@@ -106,15 +109,15 @@ Restated in the project `CLAUDE.md` at the skeleton milestone (M1).
modules under `src/`. `website/` stays separate as the brief.
- **Scope of markets:** regular markets only; MVEs excluded; websocket exchanges
only.
- **Output destination** (settled in sprint 01): `AlignedSbeWriter` takes a file
path, and stdout is just the path `/dev/stdout` (or a tiny writer tweak), so
one code path covers both. Chunks must be 64 MiB: `AlignedSbeReader::open`
reads nothing else.
## Workflow differences
This is a job-challenge repo (cloned template, no Gitea/Woodpecker/release
pipeline). From the global workflow we **keep**: `docs/`, `roadmap/`, sprint
files, feature branches, and `scripts/ci.sh` (fmt + clippy `-D warnings` +
tests + dependency audit). We **drop**: Woodpecker pipelines, `release.sh`,
versioning/changelog, rc images, and the `prod`/`dev`/release branch flow.
Work on `feat/*` branches squashed into `main`.
See the project [CLAUDE.md](../CLAUDE.md#workflow-differences): what we keep
and drop from the global workflow for a job-challenge repo.
## Not in scope (v1)
@@ -133,14 +136,15 @@ discovery ──▶ subscription manager ──▶ N socket readers (stamp recei
│
route by symbol
▼
per-symbol book workers (fold, gap-detect, dedup sources)
per-symbol book workers (fold, loss-detect, dedup sources)
│
▼
single output writer (SBE sink)
```
Each stage and the channels between them are open until settled in a sprint; the
async-vs-threads call is the first one.
Each stage and the channels between them are open until settled in a sprint.
Settled so far: OS threads, shared-nothing, no mutexes
([ADR 0001](adr/0001-concurrency-model.md)).
## Data model, first sketch
@@ -148,15 +152,17 @@ Supporting detail (mostly Claude's to settle), serving the pipeline above.
Not a DB — an in-memory domain plus the fixed SBE output schema. Main entities:
- **Symbol / market** (from discovery): id, trading status, any price/tick
metadata the exchange gives.
- **Symbol** (from discovery): one order book, with its trading status and any
price/tick metadata the exchange gives. On Polymarket that is a token, not a
market: a market has two (Yes / No), each with its own, mirrored book.
- **NormalizedEvent** (internal, exchange-agnostic): the thing each adapter
produces so the core never sees exchange formats. Either a full snapshot or a
single delta, each carrying `symbol`, `seq`, `exchange_timestamp`,
`received_at`, and payload (side/price/quantity, or the full book).
single delta, each carrying `symbol`, `exchange_timestamp`, `received_at`,
whatever the exchange offers for loss detection (a sequence number, a book
hash), and payload (side/price/quantity, or the full book).
- **Book** (one per symbol): bids and asks as ordered price→quantity maps, plus
`last_seq`, last exchange timestamp, and the id of the source currently
trusted (for redundancy).
the last loss-detection marker seen, last exchange timestamp, and the id of the
source currently trusted (for redundancy).
- **Connection** (one websocket): the set of symbols it carries.
- **Source / redundancy:** several connections may feed the same symbol; one is
the trusted source at a time.
@@ -169,16 +175,14 @@ trusted source is chosen and switched, and emitting a fresh snapshot on switch).
## Open questions
- **Output: stdout vs file.** Brief says stdout; `AlignedSbeWriter` takes a file
path and writes fixed 64 MiB zero-padded chunks. Likely: write to a file
(possibly rotated); resolve in the skeleton milestone. *Recommendation: file
output, treat "stdout" as loose wording; confirm.*
- **Sequence semantics per exchange.** Is `seq` per-symbol or per-connection, and
how are snapshots vs deltas sequenced? Differs by exchange; needs doc check per
adapter.
- **Symbol lifecycle signal.** What marks a market "no longer trading" (drops
from the list vs a status field)? Per-exchange, check docs.
- **Loss detection per exchange.** Polymarket has no sequence number: can we
reproduce its book `hash`, or do we check best bid/ask after each change? M2's
opening question. Kalshi's sequence semantics: checked in M6.
- **Symbol lifecycle signal.** What marks a market "no longer trading"?
Polymarket's `new_market` / `market_resolved` events (with
`custom_feature_enabled`) are candidates; settle in M3.
- **File rotation policy.** When to roll to a new output file (size, time), given
64 MiB chunks. Resolve with the output decision.
64 MiB chunks. Resolve in M2 with the output writer.
- **Redundancy source-selection.** How to pick/switch the trusted source
(freshness, message rate, seq lead). Designed in the redundancy sprint.
(freshness, message rate, which source is ahead). Designed in the redundancy
sprint.