This repository has been archived on 2026-11-16. You can view files and clone it. You cannot open issues or pull requests or push a commit.
Files
pascal 0e687e7eba feat(polymarket): skeleton and market-channel connectivity spike
Replaces the template's demo main.rs with a spike that connects to
Polymarket's public market channel, subscribes to one market, and logs every
frame with its receive time. Adds scripts/ci.sh (fmt, clippy, tests), the
project CLAUDE.md, ADR 0001 (OS threads, shared-nothing), and the market
channel's message shapes in docs/notes.md: no sequence number, a book hash on
every change, mirrored Yes/No books.
2026-11-04 14:47:14 +01:00

8.9 KiB

01. Skeleton + connectivity

Status: done (2026-11-04)

Stand up the solution skeleton inside the template and prove we can connect to Polymarket's public market websocket — the foundation the correctness pipeline (M2) is built on. No order book, normalization, or SBE output yet. Design: see ../docs/project.md (central bet, design principles, pipeline sketch).

Concurrency model

  • Thread-only V1 (sync tungstenite). Async (tokio) is a possible future optimization, re-evaluated at the start of extension 1 (M4), where socket-management complexity (reconnect / timeout / redundancy) makes async ergonomics matter. The swap stays cheap because the socket layer sits behind a channel boundary and everything downstream is runtime-agnostic.
  • Shared-nothing, no mutexes. Any stage with potential contention is sharded (each shard owns its slice) or limited to a single owning thread, with channels between stages.

Module layout

Only modules for code we write — no speculative scaffolding: main.rs (entry / wiring) + a polymarket module for the connect / subscribe / log spike. book / output / discovery appear in the milestones that need them.

The connectivity spike

  • Connect to wss://ws-subscriptions-clob.polymarket.com/ws/market (public, no auth).
  • Subscribe by hardcoded asset/token id(s), message shape roughly {"type":"market","assets_ids":["<id>"]} — exact schema confirmed against live behavior during the spike. Find one live, active market id first.
  • Read loop; stamp receive time at the socket edge; log raw frames.
  • Observe and record message shapes in ../docs/notes.md: the book snapshot (bids/asks structure), price_change (size:"0" = deletion), and crucially whether Polymarket provides a per-message sequence number, a book hash, and/or an exchange timestamp — these drive the M2 gap-detection design and are a known open risk for the central bet.

Libraries

tungstenite (sync websocket) + rustls TLS, locked in Cargo.lock (caret ranges in Cargo.toml, the template's style). rustls is also a direct dependency with only its ring feature: tungstenite leaves the crypto provider to the application, and rustls panics at runtime ("Could not automatically determine the process-level CryptoProvider") unless exactly one is enabled. ring over aws-lc-rs, which needs a C/cmake build. No other dependencies this sprint: subscribe is a fixed JSON string and frames are logged as raw text, so no JSON crate, no HTTP client, no CLI.

CI

scripts/ci.sh: cargo fmt --check, cargo clippy --all-targets -- -D warnings, cargo test — each on its own line (or || exit 1), never &&-chained, so a failing check can't hide. No dependency audit (see Decisions). Woodpecker / release files unused (workflow differences in project.md).

Tests

None this sprint: nothing with logic worth testing yet, and connectivity isn't unit-testable. cargo test runs and passes trivially. Soundness/completeness tests begin in M2.

Decisions

  • Thread-only V1; async deferred to a possible M4 optimization. Thread-per- socket is comfortable at tens of sockets and is the simplest, most readable option; the channel-based shared-nothing design keeps a later async swap localized to the socket layer, so deferring is safe through the base challenge. Rejected: async/tokio or hybrid now — speculative complexity before the extensions need it.
  • No mutexes; shard-or-single-thread + channels. Avoids lock contention and data races by construction, and is what makes the runtime pluggable. Rejected: shared state behind Mutex — simpler to write, but contention and lock risk on the hot path.
  • tungstenite + rustls. The standard sync WS crate; rustls avoids a system OpenSSL dependency. Rejected: tokio-tungstenite (implies async), native-tls (system dep).
  • No cargo deny / audit in CI. Supply-chain auditing targets deployed products with users; this is a throwaway challenge, so CI targets application quality, not prod-grade. Rejected: keeping the audit — noise, and the unmaintained-but-not-vulnerable RUSTSEC-2024-0449 lives in vendor codegen we don't ship, for no benefit here.
  • Output destination settled at M2, not now. The mandatory AlignedSbeWriter takes a file path; a path can name stdout (/dev/stdout), or a tiny writer tweak. 64 MiB chunks are mandatory to stay readable by the template's AlignedSbeReader::open. Rejected: a generic sink abstraction — unnecessary.
  • No CLI this sprint. Everything hardcoded for the spike; a config lib would be speculative. Rejected: adding clap now.

Not in this sprint

  • Order book, event normalization, SBE output.
  • REST symbol discovery and the subscribe/unsubscribe diff loop.
  • Sharding across multiple sockets.
  • Redundancy, monitoring, SIGINT handling beyond the process default.
  • Kalshi, the generic exchange abstraction, latency work.

Done when

  • scripts/ci.sh is green (fmt + clippy -D warnings + test).
  • The program connects to Polymarket's market channel, subscribes to the hardcoded market, and logs raw frames to the terminal.
  • Polymarket's message shapes — including whether a sequence number, book hash, and/or exchange timestamp exist — are recorded in docs/notes.md.
  • The project CLAUDE.md is written: what it is, layout, commands, environment traps, design principles, workflow differences.

Outcome

Built. src/main.rs now runs src/polymarket.rs, which connects to Polymarket's public market channel, subscribes to both tokens of one market, stamps the receive time right after each socket read, and logs every frame. Run against the live feed (60 s, then 15 s at close), it received the opening snapshot, price changes, a mid-stream book resend, and a trade. scripts/ci.sh (fmt, clippy --all-targets -D warnings, test) is green, and was checked to fail and stop on a violation. Also: the project CLAUDE.md, ADR 0001, and the market channel's message shapes in docs/notes.md.

Diverged from the plan, and why.

  • A direct rustls dependency with only ring: tungstenite leaves the crypto provider to the application, and rustls panicked at the first TLS connection without one. An echo-server smoke test caught it before Polymarket was reachable.
  • The subscribe message carries both tokens of the market, to see how the Yes and No books relate (they are mirrors).
  • tempfile removed: only the template's demo used it.
  • ## Workflow differences moved from docs/project.md into CLAUDE.md, where the workflow's skills look for it.
  • Requirement 3 reworded from "sequence-gap detection" to "loss detection" in docs/project.md and CLAUDE.md. The sequence-number assumption came from Claude's early explanation, not from the brief, and Polymarket has none; the mechanism stays open.
  • Both exchanges turned out DNS-blocked on the dev network. The user pins their hosts with ../host.sh (workspace root, outside the repo). Not planned; most of the sprint's unplanned time went here.

Found, for later milestones (in docs/notes.md and the roadmap): no sequence number on Polymarket, and a change's hash equals the next book's (M2); a PING keepalive every 10 s is required (M2); a symbol is a token, and the Yes/No books mirror each other (M2's data model); new_market / market_resolved events and subscribe/unsubscribe on an open connection (M3); the local clock runs ~30 ms slow (M7).

Decided at close (the user): research into how the hash is computed waits for M2.

Left out. The template's README still says the helpers are "demonstrated in main.rs" (moved to M8, which writes the solution README). The hardcoded market closes 2026-11-04, so rerunning the spike needs a fresh token id (its comment says so); M2 picks its own subset and M3 replaces hardcoding.

Retrospective

  • Went well: testing the transport against a neutral echo server caught the rustls panic while the exchange was unreachable. Checking the docs' claims against the live feed found what the docs don't say: books resent mid-stream, and a change's hash identifying the next book.
  • Went wrong: the network block surfaced only at implementation; a reachability check during planning would have found it a day earlier. And a general pattern (exchanges carry sequence numbers) entered the central bet's requirements as if it came from the brief; the live feed and the user's challenge caught it, not a review.
  • Tested enough? No tests, as agreed: nothing with logic yet. Not covered: a long-running connection (keepalive), reconnects, malformed frames — all M2. host.sh was tested on copies, then by its own check on the real file.
  • Next time: before a claim goes into a requirement, say where it comes from (the brief, the user, or a general pattern to verify); check the reachability of external services when planning.