feat(polymarket): skeleton and market-channel connectivity spike

Replaces the template's demo main.rs with a spike that connects to
Polymarket's public market channel, subscribes to one market, and logs every
frame with its receive time. Adds scripts/ci.sh (fmt, clippy, tests), the
project CLAUDE.md, ADR 0001 (OS threads, shared-nothing), and the market
channel's message shapes in docs/notes.md: no sequence number, a book hash on
every change, mirrored Yes/No books.
This commit is contained in:
pascal committed 2026-11-04 14:47:14 +01:00
1 parent adcf205f0b
commit 0e687e7eba
11 files changed
+713 -138

No files matched your search

+171
View File
@@ -0,0 +1,171 @@
# 01. Skeleton + connectivity
**Status: done (2026-11-04)**
Stand up the solution skeleton inside the template and prove we can connect to
Polymarket's public market websocket — the foundation the correctness pipeline
(M2) is built on. No order book, normalization, or SBE output yet. Design: see
[../docs/project.md](../docs/project.md) (central bet, design principles,
pipeline sketch).
## Concurrency model
- **Thread-only V1** (sync `tungstenite`). Async (tokio) is a possible future
optimization, re-evaluated at the start of extension 1 (M4), where
socket-management complexity (reconnect / timeout / redundancy) makes async
ergonomics matter. The swap stays cheap because the socket layer sits behind a
channel boundary and everything downstream is runtime-agnostic.
- **Shared-nothing, no mutexes.** Any stage with potential contention is sharded
(each shard owns its slice) or limited to a single owning thread, with channels
between stages.
## Module layout
Only modules for code we write — no speculative scaffolding: `main.rs`
(entry / wiring) + a `polymarket` module for the connect / subscribe / log spike.
`book` / `output` / `discovery` appear in the milestones that need them.
## The connectivity spike
- Connect to `wss://ws-subscriptions-clob.polymarket.com/ws/market` (public, no
auth).
- Subscribe by hardcoded asset/token id(s), message shape roughly
`{"type":"market","assets_ids":["<id>"]}` — exact schema confirmed against live
behavior during the spike. Find one live, active market id first.
- Read loop; **stamp receive time at the socket edge**; log raw frames.
- Observe and record message shapes in [../docs/notes.md](../docs/notes.md): the
`book` snapshot (bids/asks structure), `price_change` (`size:"0"` = deletion),
and crucially **whether Polymarket provides a per-message sequence number, a
book hash, and/or an exchange timestamp** — these drive the M2 gap-detection
design and are a known open risk for the central bet.
## Libraries
`tungstenite` (sync websocket) + `rustls` TLS, locked in `Cargo.lock` (caret
ranges in `Cargo.toml`, the template's style). `rustls`
is also a direct dependency with only its `ring` feature: tungstenite leaves the
crypto provider to the application, and rustls panics at runtime ("Could not
automatically determine the process-level CryptoProvider") unless exactly one is
enabled. `ring` over `aws-lc-rs`, which needs a C/cmake build. No other
dependencies this sprint: subscribe is a fixed JSON string and frames are logged
as raw text, so no JSON crate, no HTTP client, no CLI.
## CI
`scripts/ci.sh`: `cargo fmt --check`,
`cargo clippy --all-targets -- -D warnings`, `cargo test` — each on its own line (or `|| exit 1`), never `&&`-chained, so a
failing check can't hide. No dependency audit (see Decisions). Woodpecker /
release files unused (workflow differences in project.md).
## Tests
None this sprint: nothing with logic worth testing yet, and connectivity isn't
unit-testable. `cargo test` runs and passes trivially. Soundness/completeness
tests begin in M2.
## Decisions
- **Thread-only V1; async deferred to a possible M4 optimization.** Thread-per-
socket is comfortable at tens of sockets and is the simplest, most readable
option; the channel-based shared-nothing design keeps a later async swap
localized to the socket layer, so deferring is safe through the base challenge.
*Rejected:* async/tokio or hybrid now — speculative complexity before the
extensions need it.
- **No mutexes; shard-or-single-thread + channels.** Avoids lock contention and
data races by construction, and is what makes the runtime pluggable. *Rejected:*
shared state behind `Mutex` — simpler to write, but contention and lock risk on
the hot path.
- **`tungstenite` + `rustls`.** The standard sync WS crate; rustls avoids a
system OpenSSL dependency. *Rejected:* `tokio-tungstenite` (implies async),
native-tls (system dep).
- **No `cargo deny` / audit in CI.** Supply-chain auditing targets deployed
products with users; this is a throwaway challenge, so CI targets application
quality, not prod-grade. *Rejected:* keeping the audit — noise, and the
unmaintained-but-not-vulnerable RUSTSEC-2024-0449 lives in vendor codegen we
don't ship, for no benefit here.
- **Output destination settled at M2, not now.** The mandatory `AlignedSbeWriter`
takes a file path; a path can name stdout (`/dev/stdout`), or a tiny writer
tweak. 64 MiB chunks are mandatory to stay readable by the template's
`AlignedSbeReader::open`. *Rejected:* a generic sink abstraction — unnecessary.
- **No CLI this sprint.** Everything hardcoded for the spike; a config lib would
be speculative. *Rejected:* adding `clap` now.
## Not in this sprint
- Order book, event normalization, SBE output.
- REST symbol discovery and the subscribe/unsubscribe diff loop.
- Sharding across multiple sockets.
- Redundancy, monitoring, SIGINT handling beyond the process default.
- Kalshi, the generic exchange abstraction, latency work.
## Done when
- `scripts/ci.sh` is green (fmt + clippy `-D warnings` + test).
- The program connects to Polymarket's market channel, subscribes to the
hardcoded market, and logs raw frames to the terminal.
- Polymarket's message shapes — including whether a sequence number, book hash,
and/or exchange timestamp exist — are recorded in `docs/notes.md`.
- The project `CLAUDE.md` is written: what it is, layout, commands, environment
traps, design principles, workflow differences.
## Outcome
**Built.** `src/main.rs` now runs `src/polymarket.rs`, which connects to
Polymarket's public market channel, subscribes to both tokens of one market,
stamps the receive time right after each socket read, and logs every frame. Run
against the live feed (60 s, then 15 s at close), it received the opening
snapshot, price changes, a mid-stream book resend, and a trade. `scripts/ci.sh`
(fmt, clippy `--all-targets -D warnings`, test) is green, and was checked to fail
and stop on a violation. Also: the project `CLAUDE.md`, ADR 0001, and the market
channel's message shapes in `docs/notes.md`.
**Diverged from the plan, and why.**
- A direct `rustls` dependency with only `ring`: tungstenite leaves the crypto
provider to the application, and rustls panicked at the first TLS connection
without one. An echo-server smoke test caught it before Polymarket was
reachable.
- The subscribe message carries both tokens of the market, to see how the Yes and
No books relate (they are mirrors).
- `tempfile` removed: only the template's demo used it.
- `## Workflow differences` moved from `docs/project.md` into `CLAUDE.md`, where
the workflow's skills look for it.
- Requirement 3 reworded from "sequence-gap detection" to "loss detection" in
`docs/project.md` and `CLAUDE.md`. The sequence-number assumption came from
Claude's early explanation, not from the brief, and Polymarket has none; the
mechanism stays open.
- Both exchanges turned out DNS-blocked on the dev network. The user pins their
hosts with `../host.sh` (workspace root, outside the repo). Not planned; most
of the sprint's unplanned time went here.
**Found, for later milestones** (in `docs/notes.md` and the roadmap): no
sequence number on Polymarket, and a change's `hash` equals the next book's
(M2); a `PING` keepalive every 10 s is required (M2); a symbol is a token, and the
Yes/No books mirror each other (M2's data model); `new_market` /
`market_resolved` events and subscribe/unsubscribe on an open connection (M3);
the local clock runs ~30 ms slow (M7).
**Decided at close (the user):** research into how the `hash` is computed waits
for M2.
**Left out.** The template's README still says the helpers are "demonstrated in
`main.rs`" (moved to M8, which writes the solution README). The hardcoded market
closes 2026-11-04, so rerunning the spike needs a fresh token id (its comment
says so); M2 picks its own subset and M3 replaces hardcoding.
### Retrospective
- **Went well:** testing the transport against a neutral echo server caught the
rustls panic while the exchange was unreachable. Checking the docs' claims
against the live feed found what the docs don't say: books resent mid-stream,
and a change's hash identifying the next book.
- **Went wrong:** the network block surfaced only at implementation; a
reachability check during planning would have found it a day earlier. And a
general pattern (exchanges carry sequence numbers) entered the central bet's
requirements as if it came from the brief; the live feed and the user's
challenge caught it, not a review.
- **Tested enough?** No tests, as agreed: nothing with logic yet. Not covered: a
long-running connection (keepalive), reconnects, malformed frames — all M2.
`host.sh` was tested on copies, then by its own check on the real file.
- **Next time:** before a claim goes into a requirement, say where it comes from
(the brief, the user, or a general pattern to verify); check the reachability
of external services when planning.