Archived
Replaces the template's demo main.rs with a spike that connects to Polymarket's public market channel, subscribes to one market, and logs every frame with its receive time. Adds scripts/ci.sh (fmt, clippy, tests), the project CLAUDE.md, ADR 0001 (OS threads, shared-nothing), and the market channel's message shapes in docs/notes.md: no sequence number, a book hash on every change, mirrored Yes/No books.
172 lines
8.9 KiB
Markdown
172 lines
8.9 KiB
Markdown
# 01. Skeleton + connectivity
|
|
|
|
**Status: done (2026-11-04)**
|
|
|
|
Stand up the solution skeleton inside the template and prove we can connect to
|
|
Polymarket's public market websocket — the foundation the correctness pipeline
|
|
(M2) is built on. No order book, normalization, or SBE output yet. Design: see
|
|
[../docs/project.md](../docs/project.md) (central bet, design principles,
|
|
pipeline sketch).
|
|
|
|
## Concurrency model
|
|
|
|
- **Thread-only V1** (sync `tungstenite`). Async (tokio) is a possible future
|
|
optimization, re-evaluated at the start of extension 1 (M4), where
|
|
socket-management complexity (reconnect / timeout / redundancy) makes async
|
|
ergonomics matter. The swap stays cheap because the socket layer sits behind a
|
|
channel boundary and everything downstream is runtime-agnostic.
|
|
- **Shared-nothing, no mutexes.** Any stage with potential contention is sharded
|
|
(each shard owns its slice) or limited to a single owning thread, with channels
|
|
between stages.
|
|
|
|
## Module layout
|
|
|
|
Only modules for code we write — no speculative scaffolding: `main.rs`
|
|
(entry / wiring) + a `polymarket` module for the connect / subscribe / log spike.
|
|
`book` / `output` / `discovery` appear in the milestones that need them.
|
|
|
|
## The connectivity spike
|
|
|
|
- Connect to `wss://ws-subscriptions-clob.polymarket.com/ws/market` (public, no
|
|
auth).
|
|
- Subscribe by hardcoded asset/token id(s), message shape roughly
|
|
`{"type":"market","assets_ids":["<id>"]}` — exact schema confirmed against live
|
|
behavior during the spike. Find one live, active market id first.
|
|
- Read loop; **stamp receive time at the socket edge**; log raw frames.
|
|
- Observe and record message shapes in [../docs/notes.md](../docs/notes.md): the
|
|
`book` snapshot (bids/asks structure), `price_change` (`size:"0"` = deletion),
|
|
and crucially **whether Polymarket provides a per-message sequence number, a
|
|
book hash, and/or an exchange timestamp** — these drive the M2 gap-detection
|
|
design and are a known open risk for the central bet.
|
|
|
|
## Libraries
|
|
|
|
`tungstenite` (sync websocket) + `rustls` TLS, locked in `Cargo.lock` (caret
|
|
ranges in `Cargo.toml`, the template's style). `rustls`
|
|
is also a direct dependency with only its `ring` feature: tungstenite leaves the
|
|
crypto provider to the application, and rustls panics at runtime ("Could not
|
|
automatically determine the process-level CryptoProvider") unless exactly one is
|
|
enabled. `ring` over `aws-lc-rs`, which needs a C/cmake build. No other
|
|
dependencies this sprint: subscribe is a fixed JSON string and frames are logged
|
|
as raw text, so no JSON crate, no HTTP client, no CLI.
|
|
|
|
## CI
|
|
|
|
`scripts/ci.sh`: `cargo fmt --check`,
|
|
`cargo clippy --all-targets -- -D warnings`, `cargo test` — each on its own line (or `|| exit 1`), never `&&`-chained, so a
|
|
failing check can't hide. No dependency audit (see Decisions). Woodpecker /
|
|
release files unused (workflow differences in project.md).
|
|
|
|
## Tests
|
|
|
|
None this sprint: nothing with logic worth testing yet, and connectivity isn't
|
|
unit-testable. `cargo test` runs and passes trivially. Soundness/completeness
|
|
tests begin in M2.
|
|
|
|
## Decisions
|
|
|
|
- **Thread-only V1; async deferred to a possible M4 optimization.** Thread-per-
|
|
socket is comfortable at tens of sockets and is the simplest, most readable
|
|
option; the channel-based shared-nothing design keeps a later async swap
|
|
localized to the socket layer, so deferring is safe through the base challenge.
|
|
*Rejected:* async/tokio or hybrid now — speculative complexity before the
|
|
extensions need it.
|
|
- **No mutexes; shard-or-single-thread + channels.** Avoids lock contention and
|
|
data races by construction, and is what makes the runtime pluggable. *Rejected:*
|
|
shared state behind `Mutex` — simpler to write, but contention and lock risk on
|
|
the hot path.
|
|
- **`tungstenite` + `rustls`.** The standard sync WS crate; rustls avoids a
|
|
system OpenSSL dependency. *Rejected:* `tokio-tungstenite` (implies async),
|
|
native-tls (system dep).
|
|
- **No `cargo deny` / audit in CI.** Supply-chain auditing targets deployed
|
|
products with users; this is a throwaway challenge, so CI targets application
|
|
quality, not prod-grade. *Rejected:* keeping the audit — noise, and the
|
|
unmaintained-but-not-vulnerable RUSTSEC-2024-0449 lives in vendor codegen we
|
|
don't ship, for no benefit here.
|
|
- **Output destination settled at M2, not now.** The mandatory `AlignedSbeWriter`
|
|
takes a file path; a path can name stdout (`/dev/stdout`), or a tiny writer
|
|
tweak. 64 MiB chunks are mandatory to stay readable by the template's
|
|
`AlignedSbeReader::open`. *Rejected:* a generic sink abstraction — unnecessary.
|
|
- **No CLI this sprint.** Everything hardcoded for the spike; a config lib would
|
|
be speculative. *Rejected:* adding `clap` now.
|
|
|
|
## Not in this sprint
|
|
|
|
- Order book, event normalization, SBE output.
|
|
- REST symbol discovery and the subscribe/unsubscribe diff loop.
|
|
- Sharding across multiple sockets.
|
|
- Redundancy, monitoring, SIGINT handling beyond the process default.
|
|
- Kalshi, the generic exchange abstraction, latency work.
|
|
|
|
## Done when
|
|
|
|
- `scripts/ci.sh` is green (fmt + clippy `-D warnings` + test).
|
|
- The program connects to Polymarket's market channel, subscribes to the
|
|
hardcoded market, and logs raw frames to the terminal.
|
|
- Polymarket's message shapes — including whether a sequence number, book hash,
|
|
and/or exchange timestamp exist — are recorded in `docs/notes.md`.
|
|
- The project `CLAUDE.md` is written: what it is, layout, commands, environment
|
|
traps, design principles, workflow differences.
|
|
|
|
## Outcome
|
|
|
|
**Built.** `src/main.rs` now runs `src/polymarket.rs`, which connects to
|
|
Polymarket's public market channel, subscribes to both tokens of one market,
|
|
stamps the receive time right after each socket read, and logs every frame. Run
|
|
against the live feed (60 s, then 15 s at close), it received the opening
|
|
snapshot, price changes, a mid-stream book resend, and a trade. `scripts/ci.sh`
|
|
(fmt, clippy `--all-targets -D warnings`, test) is green, and was checked to fail
|
|
and stop on a violation. Also: the project `CLAUDE.md`, ADR 0001, and the market
|
|
channel's message shapes in `docs/notes.md`.
|
|
|
|
**Diverged from the plan, and why.**
|
|
- A direct `rustls` dependency with only `ring`: tungstenite leaves the crypto
|
|
provider to the application, and rustls panicked at the first TLS connection
|
|
without one. An echo-server smoke test caught it before Polymarket was
|
|
reachable.
|
|
- The subscribe message carries both tokens of the market, to see how the Yes and
|
|
No books relate (they are mirrors).
|
|
- `tempfile` removed: only the template's demo used it.
|
|
- `## Workflow differences` moved from `docs/project.md` into `CLAUDE.md`, where
|
|
the workflow's skills look for it.
|
|
- Requirement 3 reworded from "sequence-gap detection" to "loss detection" in
|
|
`docs/project.md` and `CLAUDE.md`. The sequence-number assumption came from
|
|
Claude's early explanation, not from the brief, and Polymarket has none; the
|
|
mechanism stays open.
|
|
- Both exchanges turned out DNS-blocked on the dev network. The user pins their
|
|
hosts with `../host.sh` (workspace root, outside the repo). Not planned; most
|
|
of the sprint's unplanned time went here.
|
|
|
|
**Found, for later milestones** (in `docs/notes.md` and the roadmap): no
|
|
sequence number on Polymarket, and a change's `hash` equals the next book's
|
|
(M2); a `PING` keepalive every 10 s is required (M2); a symbol is a token, and the
|
|
Yes/No books mirror each other (M2's data model); `new_market` /
|
|
`market_resolved` events and subscribe/unsubscribe on an open connection (M3);
|
|
the local clock runs ~30 ms slow (M7).
|
|
|
|
**Decided at close (the user):** research into how the `hash` is computed waits
|
|
for M2.
|
|
|
|
**Left out.** The template's README still says the helpers are "demonstrated in
|
|
`main.rs`" (moved to M8, which writes the solution README). The hardcoded market
|
|
closes 2026-11-04, so rerunning the spike needs a fresh token id (its comment
|
|
says so); M2 picks its own subset and M3 replaces hardcoding.
|
|
|
|
### Retrospective
|
|
|
|
- **Went well:** testing the transport against a neutral echo server caught the
|
|
rustls panic while the exchange was unreachable. Checking the docs' claims
|
|
against the live feed found what the docs don't say: books resent mid-stream,
|
|
and a change's hash identifying the next book.
|
|
- **Went wrong:** the network block surfaced only at implementation; a
|
|
reachability check during planning would have found it a day earlier. And a
|
|
general pattern (exchanges carry sequence numbers) entered the central bet's
|
|
requirements as if it came from the brief; the live feed and the user's
|
|
challenge caught it, not a review.
|
|
- **Tested enough?** No tests, as agreed: nothing with logic yet. Not covered: a
|
|
long-running connection (keepalive), reconnects, malformed frames — all M2.
|
|
`host.sh` was tested on copies, then by its own check on the real file.
|
|
- **Next time:** before a claim goes into a requirement, say where it comes from
|
|
(the brief, the user, or a general pattern to verify); check the reachability
|
|
of external services when planning.
|