Archived
feat(polymarket): skeleton and market-channel connectivity spike
Replaces the template's demo main.rs with a spike that connects to Polymarket's public market channel, subscribes to one market, and logs every frame with its receive time. Adds scripts/ci.sh (fmt, clippy, tests), the project CLAUDE.md, ADR 0001 (OS threads, shared-nothing), and the market channel's message shapes in docs/notes.md: no sequence number, a book hash on every change, mirrored Yes/No books.
This commit is contained in:
11 files changed
+713
-138
No files matched your search
@@ -0,0 +1,171 @@
|
||||
# 01. Skeleton + connectivity
|
||||
|
||||
**Status: done (2026-11-04)**
|
||||
|
||||
Stand up the solution skeleton inside the template and prove we can connect to
|
||||
Polymarket's public market websocket — the foundation the correctness pipeline
|
||||
(M2) is built on. No order book, normalization, or SBE output yet. Design: see
|
||||
[../docs/project.md](../docs/project.md) (central bet, design principles,
|
||||
pipeline sketch).
|
||||
|
||||
## Concurrency model
|
||||
|
||||
- **Thread-only V1** (sync `tungstenite`). Async (tokio) is a possible future
|
||||
optimization, re-evaluated at the start of extension 1 (M4), where
|
||||
socket-management complexity (reconnect / timeout / redundancy) makes async
|
||||
ergonomics matter. The swap stays cheap because the socket layer sits behind a
|
||||
channel boundary and everything downstream is runtime-agnostic.
|
||||
- **Shared-nothing, no mutexes.** Any stage with potential contention is sharded
|
||||
(each shard owns its slice) or limited to a single owning thread, with channels
|
||||
between stages.
|
||||
|
||||
## Module layout
|
||||
|
||||
Only modules for code we write — no speculative scaffolding: `main.rs`
|
||||
(entry / wiring) + a `polymarket` module for the connect / subscribe / log spike.
|
||||
`book` / `output` / `discovery` appear in the milestones that need them.
|
||||
|
||||
## The connectivity spike
|
||||
|
||||
- Connect to `wss://ws-subscriptions-clob.polymarket.com/ws/market` (public, no
|
||||
auth).
|
||||
- Subscribe by hardcoded asset/token id(s), message shape roughly
|
||||
`{"type":"market","assets_ids":["<id>"]}` — exact schema confirmed against live
|
||||
behavior during the spike. Find one live, active market id first.
|
||||
- Read loop; **stamp receive time at the socket edge**; log raw frames.
|
||||
- Observe and record message shapes in [../docs/notes.md](../docs/notes.md): the
|
||||
`book` snapshot (bids/asks structure), `price_change` (`size:"0"` = deletion),
|
||||
and crucially **whether Polymarket provides a per-message sequence number, a
|
||||
book hash, and/or an exchange timestamp** — these drive the M2 gap-detection
|
||||
design and are a known open risk for the central bet.
|
||||
|
||||
## Libraries
|
||||
|
||||
`tungstenite` (sync websocket) + `rustls` TLS, locked in `Cargo.lock` (caret
|
||||
ranges in `Cargo.toml`, the template's style). `rustls`
|
||||
is also a direct dependency with only its `ring` feature: tungstenite leaves the
|
||||
crypto provider to the application, and rustls panics at runtime ("Could not
|
||||
automatically determine the process-level CryptoProvider") unless exactly one is
|
||||
enabled. `ring` over `aws-lc-rs`, which needs a C/cmake build. No other
|
||||
dependencies this sprint: subscribe is a fixed JSON string and frames are logged
|
||||
as raw text, so no JSON crate, no HTTP client, no CLI.
|
||||
|
||||
## CI
|
||||
|
||||
`scripts/ci.sh`: `cargo fmt --check`,
|
||||
`cargo clippy --all-targets -- -D warnings`, `cargo test` — each on its own line (or `|| exit 1`), never `&&`-chained, so a
|
||||
failing check can't hide. No dependency audit (see Decisions). Woodpecker /
|
||||
release files unused (workflow differences in project.md).
|
||||
|
||||
## Tests
|
||||
|
||||
None this sprint: nothing with logic worth testing yet, and connectivity isn't
|
||||
unit-testable. `cargo test` runs and passes trivially. Soundness/completeness
|
||||
tests begin in M2.
|
||||
|
||||
## Decisions
|
||||
|
||||
- **Thread-only V1; async deferred to a possible M4 optimization.** Thread-per-
|
||||
socket is comfortable at tens of sockets and is the simplest, most readable
|
||||
option; the channel-based shared-nothing design keeps a later async swap
|
||||
localized to the socket layer, so deferring is safe through the base challenge.
|
||||
*Rejected:* async/tokio or hybrid now — speculative complexity before the
|
||||
extensions need it.
|
||||
- **No mutexes; shard-or-single-thread + channels.** Avoids lock contention and
|
||||
data races by construction, and is what makes the runtime pluggable. *Rejected:*
|
||||
shared state behind `Mutex` — simpler to write, but contention and lock risk on
|
||||
the hot path.
|
||||
- **`tungstenite` + `rustls`.** The standard sync WS crate; rustls avoids a
|
||||
system OpenSSL dependency. *Rejected:* `tokio-tungstenite` (implies async),
|
||||
native-tls (system dep).
|
||||
- **No `cargo deny` / audit in CI.** Supply-chain auditing targets deployed
|
||||
products with users; this is a throwaway challenge, so CI targets application
|
||||
quality, not prod-grade. *Rejected:* keeping the audit — noise, and the
|
||||
unmaintained-but-not-vulnerable RUSTSEC-2024-0449 lives in vendor codegen we
|
||||
don't ship, for no benefit here.
|
||||
- **Output destination settled at M2, not now.** The mandatory `AlignedSbeWriter`
|
||||
takes a file path; a path can name stdout (`/dev/stdout`), or a tiny writer
|
||||
tweak. 64 MiB chunks are mandatory to stay readable by the template's
|
||||
`AlignedSbeReader::open`. *Rejected:* a generic sink abstraction — unnecessary.
|
||||
- **No CLI this sprint.** Everything hardcoded for the spike; a config lib would
|
||||
be speculative. *Rejected:* adding `clap` now.
|
||||
|
||||
## Not in this sprint
|
||||
|
||||
- Order book, event normalization, SBE output.
|
||||
- REST symbol discovery and the subscribe/unsubscribe diff loop.
|
||||
- Sharding across multiple sockets.
|
||||
- Redundancy, monitoring, SIGINT handling beyond the process default.
|
||||
- Kalshi, the generic exchange abstraction, latency work.
|
||||
|
||||
## Done when
|
||||
|
||||
- `scripts/ci.sh` is green (fmt + clippy `-D warnings` + test).
|
||||
- The program connects to Polymarket's market channel, subscribes to the
|
||||
hardcoded market, and logs raw frames to the terminal.
|
||||
- Polymarket's message shapes — including whether a sequence number, book hash,
|
||||
and/or exchange timestamp exist — are recorded in `docs/notes.md`.
|
||||
- The project `CLAUDE.md` is written: what it is, layout, commands, environment
|
||||
traps, design principles, workflow differences.
|
||||
|
||||
## Outcome
|
||||
|
||||
**Built.** `src/main.rs` now runs `src/polymarket.rs`, which connects to
|
||||
Polymarket's public market channel, subscribes to both tokens of one market,
|
||||
stamps the receive time right after each socket read, and logs every frame. Run
|
||||
against the live feed (60 s, then 15 s at close), it received the opening
|
||||
snapshot, price changes, a mid-stream book resend, and a trade. `scripts/ci.sh`
|
||||
(fmt, clippy `--all-targets -D warnings`, test) is green, and was checked to fail
|
||||
and stop on a violation. Also: the project `CLAUDE.md`, ADR 0001, and the market
|
||||
channel's message shapes in `docs/notes.md`.
|
||||
|
||||
**Diverged from the plan, and why.**
|
||||
- A direct `rustls` dependency with only `ring`: tungstenite leaves the crypto
|
||||
provider to the application, and rustls panicked at the first TLS connection
|
||||
without one. An echo-server smoke test caught it before Polymarket was
|
||||
reachable.
|
||||
- The subscribe message carries both tokens of the market, to see how the Yes and
|
||||
No books relate (they are mirrors).
|
||||
- `tempfile` removed: only the template's demo used it.
|
||||
- `## Workflow differences` moved from `docs/project.md` into `CLAUDE.md`, where
|
||||
the workflow's skills look for it.
|
||||
- Requirement 3 reworded from "sequence-gap detection" to "loss detection" in
|
||||
`docs/project.md` and `CLAUDE.md`. The sequence-number assumption came from
|
||||
Claude's early explanation, not from the brief, and Polymarket has none; the
|
||||
mechanism stays open.
|
||||
- Both exchanges turned out DNS-blocked on the dev network. The user pins their
|
||||
hosts with `../host.sh` (workspace root, outside the repo). Not planned; most
|
||||
of the sprint's unplanned time went here.
|
||||
|
||||
**Found, for later milestones** (in `docs/notes.md` and the roadmap): no
|
||||
sequence number on Polymarket, and a change's `hash` equals the next book's
|
||||
(M2); a `PING` keepalive every 10 s is required (M2); a symbol is a token, and the
|
||||
Yes/No books mirror each other (M2's data model); `new_market` /
|
||||
`market_resolved` events and subscribe/unsubscribe on an open connection (M3);
|
||||
the local clock runs ~30 ms slow (M7).
|
||||
|
||||
**Decided at close (the user):** research into how the `hash` is computed waits
|
||||
for M2.
|
||||
|
||||
**Left out.** The template's README still says the helpers are "demonstrated in
|
||||
`main.rs`" (moved to M8, which writes the solution README). The hardcoded market
|
||||
closes 2026-11-04, so rerunning the spike needs a fresh token id (its comment
|
||||
says so); M2 picks its own subset and M3 replaces hardcoding.
|
||||
|
||||
### Retrospective
|
||||
|
||||
- **Went well:** testing the transport against a neutral echo server caught the
|
||||
rustls panic while the exchange was unreachable. Checking the docs' claims
|
||||
against the live feed found what the docs don't say: books resent mid-stream,
|
||||
and a change's hash identifying the next book.
|
||||
- **Went wrong:** the network block surfaced only at implementation; a
|
||||
reachability check during planning would have found it a day earlier. And a
|
||||
general pattern (exchanges carry sequence numbers) entered the central bet's
|
||||
requirements as if it came from the brief; the live feed and the user's
|
||||
challenge caught it, not a review.
|
||||
- **Tested enough?** No tests, as agreed: nothing with logic yet. Not covered: a
|
||||
long-running connection (keepalive), reconnects, malformed frames — all M2.
|
||||
`host.sh` was tested on copies, then by its own check on the real file.
|
||||
- **Next time:** before a claim goes into a requirement, say where it comes from
|
||||
(the brief, the user, or a general pattern to verify); check the reachability
|
||||
of external services when planning.
|
||||
Reference in new issue
Block a user