Archived
feat(polymarket): skeleton and market-channel connectivity spike
Replaces the template's demo main.rs with a spike that connects to Polymarket's public market channel, subscribes to one market, and logs every frame with its receive time. Adds scripts/ci.sh (fmt, clippy, tests), the project CLAUDE.md, ADR 0001 (OS threads, shared-nothing), and the market channel's message shapes in docs/notes.md: no sequence number, a book hash on every change, mirrored Yes/No books.
This commit is contained in:
11 files changed
+713
-138
No files matched your search
@@ -0,0 +1,63 @@
|
||||
# Order-book collector (Voluntas engineering challenge)
|
||||
|
||||
A collector for prediction-market order books (Polymarket, then Kalshi) that
|
||||
emits SBE-encoded updates and per-minute snapshots, built inside the challenge
|
||||
template. The brief is `../website/Engineering Challenge _ Voluntas.html`,
|
||||
outside this repo.
|
||||
|
||||
## Where things are
|
||||
|
||||
- `docs/project.md`: definition, central bet (the concurrency pipeline), and the
|
||||
reasons behind the design principles. Read before any design discussion.
|
||||
- `docs/roadmap.md` and `roadmap/NN. name.md`: milestones and each sprint's
|
||||
agreed plan. Read the current sprint file before writing code.
|
||||
- `docs/adr/`: hard-to-reverse decisions (0001: threads, shared-nothing).
|
||||
- `docs/notes.md`: domain rules and exchange protocol facts (delta semantics,
|
||||
loss detection, message shapes). Read before touching exchange parsing.
|
||||
- `src/util/` and `sbe_schemas/`: the template's SBE machinery, used as given so
|
||||
output stays readable by the template's `AlignedSbeReader`. Our code is the
|
||||
rest of `src/`.
|
||||
|
||||
## Working together
|
||||
|
||||
The user drives the pipeline / concurrency architecture: propose and debate it,
|
||||
and let the user settle it. Claude owns the data model.
|
||||
|
||||
## Writing code
|
||||
|
||||
- Shared-nothing threads: each stage owns its state on one thread or is sharded,
|
||||
and stages talk over channels. Mutexes stay out of the pipeline.
|
||||
- Boring and readable: std collections (`BTreeMap` for a price-ordered side),
|
||||
plain threads and channels. Speed work waits for the profiler, after V1.
|
||||
- An abstraction arrives with its second implementation (the exchange trait in
|
||||
M6).
|
||||
- Stamp receive time immediately after the socket read, and rebuild a book from
|
||||
a fresh snapshot whenever a lost message is detected (how depends on the
|
||||
exchange: `docs/notes.md`).
|
||||
- A corner cut for speed carries a comment naming its ceiling and upgrade path.
|
||||
|
||||
## Environment traps
|
||||
|
||||
- **Polymarket and Kalshi are DNS-blocked on the dev network**: the local
|
||||
resolver answers `195.186.4.210` / `2001:4d98:4:b10c::1` (a Swisscom block
|
||||
page). The user pins the hosts with `sudo ../host.sh on` (workspace root,
|
||||
outside this repo; `off` removes them). A timeout or the block address again
|
||||
means the CDN addresses rotated: ask the user to rerun it.
|
||||
- **Write 64 MiB chunks**: `AlignedSbeReader::open` hardcodes 64 MiB
|
||||
(`open_with_chunk_size` is private), so any other chunk size is unreadable.
|
||||
- **The binary runs until the server closes the socket.** macOS has no
|
||||
`timeout`; bound a run with
|
||||
`perl -e 'alarm 10; exec @ARGV' ./target/debug/engineering-challenge`.
|
||||
|
||||
## Workflow differences
|
||||
|
||||
A job-challenge repo cloned from the template: no Gitea, Woodpecker or release
|
||||
pipeline.
|
||||
|
||||
- **Kept**: `docs/`, `roadmap/` sprint files, `feat/*` branches, and
|
||||
`scripts/ci.sh` (fmt, clippy `-D warnings`, tests).
|
||||
- **Dropped**: the dependency audit (supply-chain checks are for deployed
|
||||
products), Woodpecker, `release.sh`, versioning and changelog, rc images, and
|
||||
the `prod` / `dev` branches. CI targets application quality, not prod-grade.
|
||||
- **Branches**: `main` plays the role of `dev`; `feat/*` branches are squashed
|
||||
into `main`.
|
||||
Generated
+266
-23
@@ -83,6 +83,12 @@ version = "3.20.3"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "72f5acc6cb2ba439de613abc23857ec3d78374d8ed5ac84e9d11336e87da8649"
|
||||
|
||||
[[package]]
|
||||
name = "bytes"
|
||||
version = "1.12.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "fc652a48c352aef3ea3aed32080501cf3ef6ed5da78602a020c991775b0aff04"
|
||||
|
||||
[[package]]
|
||||
name = "cc"
|
||||
version = "1.9.0"
|
||||
@@ -99,6 +105,17 @@ version = "1.0.5"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "4e7648175b45a9a48536d676f68d918270699102aa8dab5496df06904c914600"
|
||||
|
||||
[[package]]
|
||||
name = "chacha20"
|
||||
version = "0.10.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "65c35e4b699c7e15ccbe7ee35c005e4fc0a278d22238a2857e6ce2dadeda1b06"
|
||||
dependencies = [
|
||||
"cfg-if",
|
||||
"cpufeatures",
|
||||
"rand_core",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "chrono"
|
||||
version = "0.4.45"
|
||||
@@ -198,6 +215,12 @@ dependencies = [
|
||||
"hybrid-array",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "data-encoding"
|
||||
version = "2.11.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "4583a4551df46e2792f82ceeac45e850d2e2d5debba0b91f102385cda5b11f06"
|
||||
|
||||
[[package]]
|
||||
name = "digest"
|
||||
version = "0.11.3"
|
||||
@@ -229,9 +252,10 @@ dependencies = [
|
||||
"crossbeam",
|
||||
"ergo-sbe",
|
||||
"indexmap",
|
||||
"rustls",
|
||||
"serde",
|
||||
"serde_yaml_ng",
|
||||
"tempfile",
|
||||
"tungstenite",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -263,15 +287,9 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb"
|
||||
dependencies = [
|
||||
"libc",
|
||||
"windows-sys",
|
||||
"windows-sys 0.61.2",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "fastrand"
|
||||
version = "2.5.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "da7c62ceae207dd37ea5b845da6a0696c799f85e97da1ab5b7910be3c1c80223"
|
||||
|
||||
[[package]]
|
||||
name = "find-msvc-tools"
|
||||
version = "0.1.15"
|
||||
@@ -302,6 +320,17 @@ dependencies = [
|
||||
"slab",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "getrandom"
|
||||
version = "0.2.17"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "ff2abc00be7fca6ebc474524697ae276ad847ad0a6b3faa4bcb027e9a4614ad0"
|
||||
dependencies = [
|
||||
"cfg-if",
|
||||
"libc",
|
||||
"wasi",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "getrandom"
|
||||
version = "0.4.3"
|
||||
@@ -311,6 +340,7 @@ dependencies = [
|
||||
"cfg-if",
|
||||
"libc",
|
||||
"r-efi",
|
||||
"rand_core",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -325,6 +355,22 @@ version = "0.17.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "ed5909b6e89a2db4456e54cd5f673791d7eca6732202bbf2a9cc504fe2f9b84a"
|
||||
|
||||
[[package]]
|
||||
name = "http"
|
||||
version = "1.5.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "918d3568bebf352712bc2ef3d46a8bcf1a75b373be6539de198e9105cbbf9ce0"
|
||||
dependencies = [
|
||||
"bytes",
|
||||
"itoa",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "httparse"
|
||||
version = "1.10.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "6dbf3de79e51f3d586ab4cb9d5c3e2c14aa28ed23d180cf89b4df0454a69cc87"
|
||||
|
||||
[[package]]
|
||||
name = "hybrid-array"
|
||||
version = "0.4.15"
|
||||
@@ -630,6 +676,37 @@ version = "6.0.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf"
|
||||
|
||||
[[package]]
|
||||
name = "rand"
|
||||
version = "0.10.3"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "65c9fb96cbc91e3478eaae79a69fcd3f1ae4ad052e471fe6732fff548984b4af"
|
||||
dependencies = [
|
||||
"chacha20",
|
||||
"getrandom 0.4.3",
|
||||
"rand_core",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "rand_core"
|
||||
version = "0.10.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "63b8176103e19a2643978565ca18b50549f6101881c443590420e4dc998a3c69"
|
||||
|
||||
[[package]]
|
||||
name = "ring"
|
||||
version = "0.17.14"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "a4689e6c2294d81e88dc6261c768b63bc4fcdb852be6d1352498b114f61383b7"
|
||||
dependencies = [
|
||||
"cc",
|
||||
"cfg-if",
|
||||
"getrandom 0.2.17",
|
||||
"libc",
|
||||
"untrusted",
|
||||
"windows-sys 0.52.0",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "roxmltree"
|
||||
version = "0.21.1"
|
||||
@@ -655,7 +732,41 @@ dependencies = [
|
||||
"errno",
|
||||
"libc",
|
||||
"linux-raw-sys",
|
||||
"windows-sys",
|
||||
"windows-sys 0.61.2",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "rustls"
|
||||
version = "0.23.46"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "48e13bd8c0e9365c43cfa5c9e8f9ad49d3c8444926c9aac819e0e4dc503c8fdf"
|
||||
dependencies = [
|
||||
"once_cell",
|
||||
"ring",
|
||||
"rustls-pki-types",
|
||||
"rustls-webpki",
|
||||
"subtle",
|
||||
"zeroize",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "rustls-pki-types"
|
||||
version = "1.15.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "2f4925028c7eb5d1fcdaf196971378ed9d2c1c4efc7dc5d011256f76c99c0a96"
|
||||
dependencies = [
|
||||
"zeroize",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "rustls-webpki"
|
||||
version = "0.103.16"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "4443b1b6d39d7c663139942cc4558262562d2c84123bb6282ef177009ca8b281"
|
||||
dependencies = [
|
||||
"ring",
|
||||
"rustls-pki-types",
|
||||
"untrusted",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -713,6 +824,17 @@ dependencies = [
|
||||
"unsafe-libyaml",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "sha1"
|
||||
version = "0.11.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "aacc4cc499359472b4abe1bf11d0b12e688af9a805fa5e3016f9a386dc2d0214"
|
||||
dependencies = [
|
||||
"cfg-if",
|
||||
"cpufeatures",
|
||||
"digest",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "sha2"
|
||||
version = "0.11.1"
|
||||
@@ -748,6 +870,12 @@ version = "1.2.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "6ce2be8dc25455e1f91df71bfa12ad37d7af1092ae736f3a6cd0e37bc7810596"
|
||||
|
||||
[[package]]
|
||||
name = "subtle"
|
||||
version = "2.6.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "13c2bddecc57b384dee18652358fb23172facb8a2c51ccc10d74c157bdea3292"
|
||||
|
||||
[[package]]
|
||||
name = "supports-color"
|
||||
version = "3.0.2"
|
||||
@@ -802,19 +930,6 @@ dependencies = [
|
||||
"syn 3.0.9",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "tempfile"
|
||||
version = "3.27.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "32497e9a4c7b38532efcdebeef879707aa9f794296a4f0244f6f69e9bc8574bd"
|
||||
dependencies = [
|
||||
"fastrand",
|
||||
"getrandom",
|
||||
"once_cell",
|
||||
"rustix",
|
||||
"windows-sys",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "terminal_size"
|
||||
version = "0.4.4"
|
||||
@@ -822,7 +937,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "230a1b821ccbd75b185820a1f1ff7b14d21da1e442e22c0863ea5f08771a8874"
|
||||
dependencies = [
|
||||
"rustix",
|
||||
"windows-sys",
|
||||
"windows-sys 0.61.2",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -866,6 +981,25 @@ dependencies = [
|
||||
"zerovec",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "tungstenite"
|
||||
version = "0.30.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "e48ac77174b19c110a50ab2128b24215ac9cb40e0e12e093fb602d175c569d22"
|
||||
dependencies = [
|
||||
"bytes",
|
||||
"data-encoding",
|
||||
"http",
|
||||
"httparse",
|
||||
"log",
|
||||
"rand",
|
||||
"rustls",
|
||||
"rustls-pki-types",
|
||||
"sha1",
|
||||
"thiserror",
|
||||
"webpki-roots 0.26.11",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "typenum"
|
||||
version = "1.20.1"
|
||||
@@ -896,12 +1030,24 @@ version = "0.2.11"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "673aac59facbab8a9007c7f6108d11f63b603f7cabff99fabf650fea5c32b861"
|
||||
|
||||
[[package]]
|
||||
name = "untrusted"
|
||||
version = "0.9.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "8ecb6da28b8a351d773b68d5825ac39017e680750f980f3a1a85cd8dd28a47c1"
|
||||
|
||||
[[package]]
|
||||
name = "utf8_iter"
|
||||
version = "1.0.4"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "b6c140620e7ffbb22c2dee59cafe6084a59b5ffc27a8859a5f0d494b5d52b6be"
|
||||
|
||||
[[package]]
|
||||
name = "wasi"
|
||||
version = "0.11.1+wasi-snapshot-preview1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "ccf3ec651a847eb01de73ccad15eb7d99f80485de043efb2f370cd654f4ea44b"
|
||||
|
||||
[[package]]
|
||||
name = "wasm-bindgen"
|
||||
version = "0.2.129"
|
||||
@@ -947,6 +1093,24 @@ dependencies = [
|
||||
"unicode-ident",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "webpki-roots"
|
||||
version = "0.26.11"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "521bc38abb08001b01866da9f51eb7c5d647a19260e00054a8c7fd5f9e57f7a9"
|
||||
dependencies = [
|
||||
"webpki-roots 1.0.11",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "webpki-roots"
|
||||
version = "1.0.11"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f5f280bbb2b287bc8e7de7e3852945a0ff50d1bbcb5ed42b6c7a568c546df171"
|
||||
dependencies = [
|
||||
"rustls-pki-types",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "windows-core"
|
||||
version = "0.62.2"
|
||||
@@ -1006,6 +1170,15 @@ dependencies = [
|
||||
"windows-link",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "windows-sys"
|
||||
version = "0.52.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "282be5f36a8ce781fad8c8ae18fa3f9beff57ec1b52cb3de0789201425d9a33d"
|
||||
dependencies = [
|
||||
"windows-targets",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "windows-sys"
|
||||
version = "0.61.2"
|
||||
@@ -1015,6 +1188,70 @@ dependencies = [
|
||||
"windows-link",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "windows-targets"
|
||||
version = "0.52.6"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "9b724f72796e036ab90c1021d4780d4d3d648aca59e491e6b98e725b84e99973"
|
||||
dependencies = [
|
||||
"windows_aarch64_gnullvm",
|
||||
"windows_aarch64_msvc",
|
||||
"windows_i686_gnu",
|
||||
"windows_i686_gnullvm",
|
||||
"windows_i686_msvc",
|
||||
"windows_x86_64_gnu",
|
||||
"windows_x86_64_gnullvm",
|
||||
"windows_x86_64_msvc",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "windows_aarch64_gnullvm"
|
||||
version = "0.52.6"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "32a4622180e7a0ec044bb555404c800bc9fd9ec262ec147edd5989ccd0c02cd3"
|
||||
|
||||
[[package]]
|
||||
name = "windows_aarch64_msvc"
|
||||
version = "0.52.6"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "09ec2a7bb152e2252b53fa7803150007879548bc709c039df7627cabbd05d469"
|
||||
|
||||
[[package]]
|
||||
name = "windows_i686_gnu"
|
||||
version = "0.52.6"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "8e9b5ad5ab802e97eb8e295ac6720e509ee4c243f69d781394014ebfe8bbfa0b"
|
||||
|
||||
[[package]]
|
||||
name = "windows_i686_gnullvm"
|
||||
version = "0.52.6"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "0eee52d38c090b3caa76c563b86c3a4bd71ef1a819287c19d586d7334ae8ed66"
|
||||
|
||||
[[package]]
|
||||
name = "windows_i686_msvc"
|
||||
version = "0.52.6"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "240948bc05c5e7c6dabba28bf89d89ffce3e303022809e73deaefe4f6ec56c66"
|
||||
|
||||
[[package]]
|
||||
name = "windows_x86_64_gnu"
|
||||
version = "0.52.6"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "147a5c80aabfbf0c7d901cb5895d1de30ef2907eb21fbbab29ca94c5b08b1a78"
|
||||
|
||||
[[package]]
|
||||
name = "windows_x86_64_gnullvm"
|
||||
version = "0.52.6"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "24d5b23dc417412679681396f2b49f3de8c1473deb516bd34410872eff51ed0d"
|
||||
|
||||
[[package]]
|
||||
name = "windows_x86_64_msvc"
|
||||
version = "0.52.6"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "589f6da84c646204747d1270a2a5661ea66ed1cced2631d546fdfb155959f9ec"
|
||||
|
||||
[[package]]
|
||||
name = "writeable"
|
||||
version = "0.6.4"
|
||||
@@ -1065,6 +1302,12 @@ dependencies = [
|
||||
"synstructure",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "zeroize"
|
||||
version = "1.9.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "e13084392c5e4bc371903e2935a5eaeed24905a7511356b883835e18a78f6879"
|
||||
|
||||
[[package]]
|
||||
name = "zerotrie"
|
||||
version = "0.2.5"
|
||||
|
||||
+4
-1
@@ -8,8 +8,11 @@ build = "src/util/build.rs"
|
||||
anyhow = "1"
|
||||
chrono = "0.4"
|
||||
crossbeam = "0.8.4"
|
||||
# Not used directly: tungstenite leaves rustls's crypto provider to the app, and
|
||||
# rustls panics at runtime unless exactly one (here ring) is enabled.
|
||||
rustls = { version = "0.23", default-features = false, features = ["ring"] }
|
||||
serde = { version = "1", features = ["derive"] }
|
||||
tempfile = "3"
|
||||
tungstenite = { version = "0.30", features = ["rustls-tls-webpki-roots"] }
|
||||
|
||||
[build-dependencies]
|
||||
ergo-sbe = "=0.1.16"
|
||||
|
||||
@@ -0,0 +1,35 @@
|
||||
# 0001. Concurrency model: threads, shared-nothing
|
||||
|
||||
## Context
|
||||
|
||||
The collector moves a firehose of websocket messages from many sockets to a
|
||||
single output sink. This is the project's central bet (see
|
||||
[../project.md](../project.md)): it must be correct (no dropped messages, no
|
||||
races) and time-accurate (receive time stamped at the socket edge) before it is
|
||||
fast. The runtime is the one foundational, costly-to-change choice. The template
|
||||
already uses `std` threads + `crossbeam` channels and pulls in no tokio.
|
||||
|
||||
## Decision
|
||||
|
||||
V1 uses **OS threads**, not an async runtime — a sync websocket client
|
||||
(`tungstenite`), with one read loop per socket. State is **shared-nothing**: no
|
||||
mutexes. Each pipeline stage either owns its state on a single thread or is
|
||||
sharded so each shard owns its slice, and stages communicate over channels.
|
||||
|
||||
Async (tokio) is deferred as a possible optimization, re-evaluated at the start
|
||||
of extension 1 (M4), when socket-management complexity (reconnect, timeouts,
|
||||
redundancy) makes async ergonomics worth their cost.
|
||||
|
||||
## Why
|
||||
|
||||
Thread-per-socket is comfortable at this scale (symbols shard across tens of
|
||||
sockets, not thousands) and is the simplest, most readable option — correctness
|
||||
first, speed later. Shared-nothing + channels removes lock contention and data
|
||||
races by construction, and keeps the runtime pluggable: only the socket layer is
|
||||
runtime-specific, so a later async swap is localized to that boundary rather than
|
||||
a pipeline rewrite. Deferring async is therefore safe through the base challenge.
|
||||
|
||||
Trade-off: thread-per-socket does not scale to very large socket counts, and the
|
||||
extension-era socket-management logic (select / timeout / reconnect) is where
|
||||
threads and async diverge most — hence the explicit M4 re-evaluation point rather
|
||||
than "never."
|
||||
+68
-5
@@ -6,8 +6,71 @@ A delta carries the **new total quantity** at a price level, not a change to it.
|
||||
never add. `quantity: 0` means delete the level. (Schema confirms: "absolute
|
||||
available quantity".)
|
||||
|
||||
## Sequence numbers are the thread you must not drop
|
||||
Each symbol's messages carry a monotonically increasing `seq`. If you receive a
|
||||
gap (e.g. 101 then 103), you missed a message, so your in-memory book is now
|
||||
wrong. You cannot patch it — throw the book away and re-snapshot that symbol.
|
||||
This is the "completeness" requirement made mechanical.
|
||||
## A missed message means a wrong book: rebuild it
|
||||
If a message is lost, the in-memory book is silently wrong and cannot be
|
||||
patched — throw it away and re-snapshot that symbol. This is the
|
||||
"completeness" requirement made mechanical. *How* a loss is detected depends on
|
||||
the exchange: a sequence-number gap where one exists (Kalshi, to verify in M6),
|
||||
something else where it doesn't — Polymarket has no sequence number (below).
|
||||
The challenge brief prescribes no mechanism; it only asks for complete data.
|
||||
|
||||
## Polymarket market channel (observed 2026-11-04)
|
||||
|
||||
From a 60 s run of the M1 spike on one market, both tokens subscribed, and the
|
||||
[market channel docs](https://docs.polymarket.com/market-data/websocket/market-channel.md)
|
||||
(read 2026-11-04).
|
||||
|
||||
- **Endpoint:** `wss://ws-subscriptions-clob.polymarket.com/ws/market`, public,
|
||||
no auth. Subscribe: `{"type":"market","assets_ids":["<token id>", ...]}`.
|
||||
On an open connection, `{"assets_ids":[...],"operation":"subscribe"}` (or
|
||||
`"unsubscribe"`) changes the token set without reconnecting.
|
||||
- **`custom_feature_enabled: true`** in the subscribe message adds
|
||||
`best_bid_ask`, `new_market` and `market_resolved` events — candidate signals
|
||||
for symbol discovery (M3), possibly replacing REST polling.
|
||||
- **Symbol = token.** A market (`market`, a `0x…` condition id) has two tokens
|
||||
(`asset_id`: Yes, No), each with its own book. The two books are **exact
|
||||
mirrors**: Yes had only bids (best 0.999), No only asks (best 0.001), and
|
||||
every change arrives as a pair in one message — Yes `BUY p` with No `SELL 1−p`,
|
||||
same size.
|
||||
- **Frames:** on subscribe, one frame holding a JSON **array** of `book`
|
||||
messages (one per token); afterwards, one **object** per frame. A frame's text
|
||||
can end with a newline.
|
||||
- **Full books also arrive mid-stream.** After a trade the server sent, all with
|
||||
one timestamp: the `price_change`, then a full `book` object for each token,
|
||||
then a `last_trade_price` — a natural resync point.
|
||||
- **`book`** (snapshot): `event_type`, `market`, `asset_id`, `timestamp`,
|
||||
`hash`, `bids`, `asks`, `tick_size`, `last_trade_price`. `bids`/`asks` are lists
|
||||
of `{price, size}` sorted **ascending by price**, so the best bid is the
|
||||
*last* bid. Its `timestamp` is the time of the book's **last change**, not the
|
||||
send time: 19 s old on receipt.
|
||||
- **`price_change`** (delta): `event_type`, `market`, `timestamp`,
|
||||
`price_changes[]`, each entry `{asset_id, price, size, side, hash, best_bid,
|
||||
best_ask}`. `side` is `BUY` (bid) or `SELL` (ask); `size` is the absolute
|
||||
level size (docs: `"0"` removes the level).
|
||||
- **Numbers are decimal strings** (`"0.77"`, `"26350.28"`): parse them exactly
|
||||
into fixed point (`1_000_000` = 1), never through a float.
|
||||
- **Timestamps are milliseconds** since the epoch (the schema stores
|
||||
microseconds).
|
||||
- **No sequence number**, confirmed by the docs: none, and no option enables
|
||||
one, for the market channel (the `seq` in Polymarket's docs belongs to
|
||||
PolyBolt, a separate price feed). The docs describe no gap detection or
|
||||
resync either. Candidates for loss detection, to settle in M2:
|
||||
- the `hash` (40 hex chars, SHA-1 length) on every book and every change. A
|
||||
change's `hash` equals that of the `book` sent right after it for the same
|
||||
token, so it identifies the **full book state after the change**: a full
|
||||
check if we can reproduce it from our own book. The docs don't say how it is
|
||||
computed (M2's opening question);
|
||||
- `best_bid`/`best_ask` on every change — a cheap top-of-book check after
|
||||
applying it.
|
||||
- **Keepalive (docs):** send the text frame `PING` every 10 s; the server
|
||||
answers `PONG`. The spike survived 60 s without it, but it is required.
|
||||
- **`last_trade_price`:** `asset_id`, `price`, `size`, `side`, `fee_rate_bps`,
|
||||
`timestamp`, `transaction_hash` (one per trade, for one token).
|
||||
- **Not observed:** `tick_size_change` and the custom-feature events.
|
||||
|
||||
## Clock accuracy bounds time accuracy
|
||||
`received_at` comes from the local clock, so it carries the clock's error. On
|
||||
2026-11-04 the dev machine ran ~30 ms slow (`sntp`: +30.6 ms ± 13 ms), which
|
||||
made raw `received_at − exchange timestamp` read −14 to +41 ms; corrected, the
|
||||
real lag was ~+16 to +71 ms. Latency figures need a synced clock, or the offset
|
||||
measured and reported alongside them.
|
||||
+40
-36
@@ -37,10 +37,9 @@ saves the data.
|
||||
|
||||
The interacting decisions it is made of:
|
||||
|
||||
- **Threading / runtime:** threads + channels vs an async runtime. The template
|
||||
already uses `std` threads + bounded `crossbeam` channels and pulls in no
|
||||
tokio — a strong hint toward threads + channels. Pivotal sub-decision, settled
|
||||
in the first sprint.
|
||||
- **Threading / runtime:** settled in sprint 01 — OS threads + channels,
|
||||
shared-nothing, async re-evaluated at M4
|
||||
([ADR 0001](adr/0001-concurrency-model.md)).
|
||||
- **Stamping point:** in the socket-read path, immediately, so as little as
|
||||
possible sits between the wire and the receive timestamp.
|
||||
- **Book ownership:** shard symbols so each book is owned by exactly one worker
|
||||
@@ -51,8 +50,8 @@ The interacting decisions it is made of:
|
||||
- **Output serialization:** one writer owns the single sink and consumes from
|
||||
all workers.
|
||||
- **Redundancy fit:** multiple sockets feeding one symbol's worker, which
|
||||
dedups / selects by source + seq — the topology must leave room for this from
|
||||
the start.
|
||||
dedups / selects by source — the topology must leave room for this from the
|
||||
start.
|
||||
|
||||
## Requirements the pipeline must satisfy
|
||||
|
||||
@@ -62,8 +61,11 @@ must guarantee:
|
||||
1. **Receive time stamped at the socket edge**, before any queue or processing.
|
||||
2. **A per-symbol book folded from absolute deltas** (overwrite the level;
|
||||
quantity 0 deletes it).
|
||||
3. **Sequence-gap detection → discard-and-re-snapshot** (a gap means the book is
|
||||
silently corrupt; rebuilding is the only safe response).
|
||||
3. **Loss detection → discard-and-re-snapshot** (a lost message means the book
|
||||
is silently corrupt; rebuilding is the only safe response). How a loss is
|
||||
detected depends on the exchange: a sequence gap where one exists; Polymarket
|
||||
has none, so its `hash` or best bid/ask instead
|
||||
([notes.md](notes.md)) — settled in M2.
|
||||
|
||||
These are proven first, before redundancy or scale.
|
||||
|
||||
@@ -90,12 +92,13 @@ function, symbols shardable later) are also the simplest, clearest design.
|
||||
not.** A channel, a function boundary, or a `trait` with two real impls keeps
|
||||
the door open; the clever internals are what we refuse.
|
||||
- **Never simplified away, even in V1:** receive-time stamping at the socket
|
||||
edge, and gap → re-snapshot. These are correctness, not readability tradeoffs.
|
||||
edge, and loss → re-snapshot. These are correctness, not readability tradeoffs.
|
||||
- **If a corner is genuinely cut for speed later** (a global lock, an O(n²)
|
||||
scan), mark it with a comment naming the ceiling and the upgrade path, so the
|
||||
debt is visible rather than silently clever.
|
||||
|
||||
Restated in the project `CLAUDE.md` at the skeleton milestone (M1).
|
||||
Their operational form, for writing code, is in the project
|
||||
[CLAUDE.md](../CLAUDE.md#writing-code).
|
||||
|
||||
## Constraints (already decided)
|
||||
|
||||
@@ -106,15 +109,15 @@ Restated in the project `CLAUDE.md` at the skeleton milestone (M1).
|
||||
modules under `src/`. `website/` stays separate as the brief.
|
||||
- **Scope of markets:** regular markets only; MVEs excluded; websocket exchanges
|
||||
only.
|
||||
- **Output destination** (settled in sprint 01): `AlignedSbeWriter` takes a file
|
||||
path, and stdout is just the path `/dev/stdout` (or a tiny writer tweak), so
|
||||
one code path covers both. Chunks must be 64 MiB: `AlignedSbeReader::open`
|
||||
reads nothing else.
|
||||
|
||||
## Workflow differences
|
||||
|
||||
This is a job-challenge repo (cloned template, no Gitea/Woodpecker/release
|
||||
pipeline). From the global workflow we **keep**: `docs/`, `roadmap/`, sprint
|
||||
files, feature branches, and `scripts/ci.sh` (fmt + clippy `-D warnings` +
|
||||
tests + dependency audit). We **drop**: Woodpecker pipelines, `release.sh`,
|
||||
versioning/changelog, rc images, and the `prod`/`dev`/release branch flow.
|
||||
Work on `feat/*` branches squashed into `main`.
|
||||
See the project [CLAUDE.md](../CLAUDE.md#workflow-differences): what we keep
|
||||
and drop from the global workflow for a job-challenge repo.
|
||||
|
||||
## Not in scope (v1)
|
||||
|
||||
@@ -133,14 +136,15 @@ discovery ──▶ subscription manager ──▶ N socket readers (stamp recei
|
||||
│
|
||||
route by symbol
|
||||
▼
|
||||
per-symbol book workers (fold, gap-detect, dedup sources)
|
||||
per-symbol book workers (fold, loss-detect, dedup sources)
|
||||
│
|
||||
▼
|
||||
single output writer (SBE sink)
|
||||
```
|
||||
|
||||
Each stage and the channels between them are open until settled in a sprint; the
|
||||
async-vs-threads call is the first one.
|
||||
Each stage and the channels between them are open until settled in a sprint.
|
||||
Settled so far: OS threads, shared-nothing, no mutexes
|
||||
([ADR 0001](adr/0001-concurrency-model.md)).
|
||||
|
||||
## Data model, first sketch
|
||||
|
||||
@@ -148,15 +152,17 @@ Supporting detail (mostly Claude's to settle), serving the pipeline above.
|
||||
|
||||
Not a DB — an in-memory domain plus the fixed SBE output schema. Main entities:
|
||||
|
||||
- **Symbol / market** (from discovery): id, trading status, any price/tick
|
||||
metadata the exchange gives.
|
||||
- **Symbol** (from discovery): one order book, with its trading status and any
|
||||
price/tick metadata the exchange gives. On Polymarket that is a token, not a
|
||||
market: a market has two (Yes / No), each with its own, mirrored book.
|
||||
- **NormalizedEvent** (internal, exchange-agnostic): the thing each adapter
|
||||
produces so the core never sees exchange formats. Either a full snapshot or a
|
||||
single delta, each carrying `symbol`, `seq`, `exchange_timestamp`,
|
||||
`received_at`, and payload (side/price/quantity, or the full book).
|
||||
single delta, each carrying `symbol`, `exchange_timestamp`, `received_at`,
|
||||
whatever the exchange offers for loss detection (a sequence number, a book
|
||||
hash), and payload (side/price/quantity, or the full book).
|
||||
- **Book** (one per symbol): bids and asks as ordered price→quantity maps, plus
|
||||
`last_seq`, last exchange timestamp, and the id of the source currently
|
||||
trusted (for redundancy).
|
||||
the last loss-detection marker seen, last exchange timestamp, and the id of the
|
||||
source currently trusted (for redundancy).
|
||||
- **Connection** (one websocket): the set of symbols it carries.
|
||||
- **Source / redundancy:** several connections may feed the same symbol; one is
|
||||
the trusted source at a time.
|
||||
@@ -169,16 +175,14 @@ trusted source is chosen and switched, and emitting a fresh snapshot on switch).
|
||||
|
||||
## Open questions
|
||||
|
||||
- **Output: stdout vs file.** Brief says stdout; `AlignedSbeWriter` takes a file
|
||||
path and writes fixed 64 MiB zero-padded chunks. Likely: write to a file
|
||||
(possibly rotated); resolve in the skeleton milestone. *Recommendation: file
|
||||
output, treat "stdout" as loose wording; confirm.*
|
||||
- **Sequence semantics per exchange.** Is `seq` per-symbol or per-connection, and
|
||||
how are snapshots vs deltas sequenced? Differs by exchange; needs doc check per
|
||||
adapter.
|
||||
- **Symbol lifecycle signal.** What marks a market "no longer trading" (drops
|
||||
from the list vs a status field)? Per-exchange, check docs.
|
||||
- **Loss detection per exchange.** Polymarket has no sequence number: can we
|
||||
reproduce its book `hash`, or do we check best bid/ask after each change? M2's
|
||||
opening question. Kalshi's sequence semantics: checked in M6.
|
||||
- **Symbol lifecycle signal.** What marks a market "no longer trading"?
|
||||
Polymarket's `new_market` / `market_resolved` events (with
|
||||
`custom_feature_enabled`) are candidates; settle in M3.
|
||||
- **File rotation policy.** When to roll to a new output file (size, time), given
|
||||
64 MiB chunks. Resolve with the output decision.
|
||||
64 MiB chunks. Resolve in M2 with the output writer.
|
||||
- **Redundancy source-selection.** How to pick/switch the trusted source
|
||||
(freshness, message rate, seq lead). Designed in the redundancy sprint.
|
||||
(freshness, message rate, which source is ahead). Designed in the redundancy
|
||||
sprint.
|
||||
+21
-10
@@ -11,31 +11,39 @@ abstraction is *extracted* once a second exchange exists, not designed up front.
|
||||
|
||||
## Milestones
|
||||
|
||||
1. **Skeleton + connectivity** — *not started*
|
||||
1. **Skeleton + connectivity** — *done (2026-11-04)*
|
||||
Project scaffolding inside the template: module layout (replace `main.rs`),
|
||||
`scripts/ci.sh` (fmt + clippy `-D warnings` + tests + audit), project
|
||||
CLAUDE.md, chosen websocket/HTTP libraries. Settle the pivotal
|
||||
async-vs-threads call. Resolve the output destination (stdout vs file).
|
||||
`scripts/ci.sh` (fmt + clippy `-D warnings` + tests), project CLAUDE.md, the
|
||||
websocket library (`tungstenite` + `rustls`). Runtime settled: thread-only V1
|
||||
(async deferred to M4; see [adr/0001-concurrency-model.md](adr/0001-concurrency-model.md)).
|
||||
Output destination is a non-issue (an output path; `/dev/stdout` for stdout).
|
||||
Connect to Polymarket and log raw messages.
|
||||
*Ends:* CI green; raw Polymarket messages flowing.
|
||||
|
||||
2. **Single-source book pipeline (Polymarket, subset of symbols)** — *not started*
|
||||
The central bet in its first real form: normalize events → per-symbol book
|
||||
(fold absolute deltas) → gap-detect → re-snapshot → SBE output (deltas +
|
||||
(fold absolute deltas) → loss-detect → re-snapshot → SBE output (deltas +
|
||||
snapshot every minute, on subscribe, on re-snapshot) → receive-time stamping
|
||||
→ clean SIGINT. One/few sockets, a fixed subset of symbols. Deterministic
|
||||
→ clean SIGINT, with the `PING` keepalive every 10 s. Polymarket has no
|
||||
sequence number, so loss detection (its book `hash`, or best bid/ask) is the
|
||||
opening question. One/few sockets, a fixed subset of symbols. Deterministic
|
||||
replay tests + snapshot↔delta reconciliation.
|
||||
*Ends:* correct SBE output for a symbol subset; tests green.
|
||||
|
||||
3. **Full coverage: discovery + sharding (base challenge complete)** — *not started*
|
||||
REST discovery loop with diffing (subscribe new, unsubscribe gone);
|
||||
load-balance all live regular-market symbols across N sockets.
|
||||
Discovery with diffing (subscribe new, unsubscribe gone) — REST polling, or
|
||||
Polymarket's `new_market` / `market_resolved` events — using subscribe /
|
||||
unsubscribe on open connections; load-balance all live regular-market symbols
|
||||
across N sockets.
|
||||
*Ends:* every live Polymarket symbol tracked — base challenge satisfied for
|
||||
one exchange.
|
||||
|
||||
4. **Extension 1 — redundant websockets** — *not started*
|
||||
Multiple connections per symbol; choose a trusted source and switch on
|
||||
loss/quiet/delay; emit a fresh snapshot on every source switch.
|
||||
**Async re-evaluation checkpoint:** before building this, reassess whether the
|
||||
socket-management complexity (reconnect/timeout/select, redundancy) warrants
|
||||
swapping the thread-only runtime for tokio (see ADR 0001).
|
||||
*Ends:* redundancy demonstrably reduces data loss; source-switch re-snapshots.
|
||||
|
||||
5. **Extension 2 — subscription monitoring** — *not started*
|
||||
@@ -54,11 +62,14 @@ abstraction is *extracted* once a second exchange exists, not designed up front.
|
||||
7. **Extension 4 — latency experiments + writeup** — *not started*
|
||||
Experiment with faster connections (first-after-maintenance, connection
|
||||
racing, source-port tricks) on both exchanges; document findings in a
|
||||
markdown file.
|
||||
markdown file, with the local clock's offset measured alongside (the dev
|
||||
machine ran ~30 ms slow; see notes.md).
|
||||
*Ends:* experiments documented.
|
||||
|
||||
8. **Polish + submission** — *not started*
|
||||
Final review, README for the solution, packaging/handoff for the challenge.
|
||||
Final review, README for the solution (the template's README still says the
|
||||
helpers are "demonstrated in `main.rs`", whose demo M1 replaced),
|
||||
packaging/handoff for the challenge.
|
||||
*Ends:* ready to submit.
|
||||
|
||||
## Why this order
|
||||
|
||||
@@ -0,0 +1,171 @@
|
||||
# 01. Skeleton + connectivity
|
||||
|
||||
**Status: done (2026-11-04)**
|
||||
|
||||
Stand up the solution skeleton inside the template and prove we can connect to
|
||||
Polymarket's public market websocket — the foundation the correctness pipeline
|
||||
(M2) is built on. No order book, normalization, or SBE output yet. Design: see
|
||||
[../docs/project.md](../docs/project.md) (central bet, design principles,
|
||||
pipeline sketch).
|
||||
|
||||
## Concurrency model
|
||||
|
||||
- **Thread-only V1** (sync `tungstenite`). Async (tokio) is a possible future
|
||||
optimization, re-evaluated at the start of extension 1 (M4), where
|
||||
socket-management complexity (reconnect / timeout / redundancy) makes async
|
||||
ergonomics matter. The swap stays cheap because the socket layer sits behind a
|
||||
channel boundary and everything downstream is runtime-agnostic.
|
||||
- **Shared-nothing, no mutexes.** Any stage with potential contention is sharded
|
||||
(each shard owns its slice) or limited to a single owning thread, with channels
|
||||
between stages.
|
||||
|
||||
## Module layout
|
||||
|
||||
Only modules for code we write — no speculative scaffolding: `main.rs`
|
||||
(entry / wiring) + a `polymarket` module for the connect / subscribe / log spike.
|
||||
`book` / `output` / `discovery` appear in the milestones that need them.
|
||||
|
||||
## The connectivity spike
|
||||
|
||||
- Connect to `wss://ws-subscriptions-clob.polymarket.com/ws/market` (public, no
|
||||
auth).
|
||||
- Subscribe by hardcoded asset/token id(s), message shape roughly
|
||||
`{"type":"market","assets_ids":["<id>"]}` — exact schema confirmed against live
|
||||
behavior during the spike. Find one live, active market id first.
|
||||
- Read loop; **stamp receive time at the socket edge**; log raw frames.
|
||||
- Observe and record message shapes in [../docs/notes.md](../docs/notes.md): the
|
||||
`book` snapshot (bids/asks structure), `price_change` (`size:"0"` = deletion),
|
||||
and crucially **whether Polymarket provides a per-message sequence number, a
|
||||
book hash, and/or an exchange timestamp** — these drive the M2 gap-detection
|
||||
design and are a known open risk for the central bet.
|
||||
|
||||
## Libraries
|
||||
|
||||
`tungstenite` (sync websocket) + `rustls` TLS, locked in `Cargo.lock` (caret
|
||||
ranges in `Cargo.toml`, the template's style). `rustls`
|
||||
is also a direct dependency with only its `ring` feature: tungstenite leaves the
|
||||
crypto provider to the application, and rustls panics at runtime ("Could not
|
||||
automatically determine the process-level CryptoProvider") unless exactly one is
|
||||
enabled. `ring` over `aws-lc-rs`, which needs a C/cmake build. No other
|
||||
dependencies this sprint: subscribe is a fixed JSON string and frames are logged
|
||||
as raw text, so no JSON crate, no HTTP client, no CLI.
|
||||
|
||||
## CI
|
||||
|
||||
`scripts/ci.sh`: `cargo fmt --check`,
|
||||
`cargo clippy --all-targets -- -D warnings`, `cargo test` — each on its own line (or `|| exit 1`), never `&&`-chained, so a
|
||||
failing check can't hide. No dependency audit (see Decisions). Woodpecker /
|
||||
release files unused (workflow differences in project.md).
|
||||
|
||||
## Tests
|
||||
|
||||
None this sprint: nothing with logic worth testing yet, and connectivity isn't
|
||||
unit-testable. `cargo test` runs and passes trivially. Soundness/completeness
|
||||
tests begin in M2.
|
||||
|
||||
## Decisions
|
||||
|
||||
- **Thread-only V1; async deferred to a possible M4 optimization.** Thread-per-
|
||||
socket is comfortable at tens of sockets and is the simplest, most readable
|
||||
option; the channel-based shared-nothing design keeps a later async swap
|
||||
localized to the socket layer, so deferring is safe through the base challenge.
|
||||
*Rejected:* async/tokio or hybrid now — speculative complexity before the
|
||||
extensions need it.
|
||||
- **No mutexes; shard-or-single-thread + channels.** Avoids lock contention and
|
||||
data races by construction, and is what makes the runtime pluggable. *Rejected:*
|
||||
shared state behind `Mutex` — simpler to write, but contention and lock risk on
|
||||
the hot path.
|
||||
- **`tungstenite` + `rustls`.** The standard sync WS crate; rustls avoids a
|
||||
system OpenSSL dependency. *Rejected:* `tokio-tungstenite` (implies async),
|
||||
native-tls (system dep).
|
||||
- **No `cargo deny` / audit in CI.** Supply-chain auditing targets deployed
|
||||
products with users; this is a throwaway challenge, so CI targets application
|
||||
quality, not prod-grade. *Rejected:* keeping the audit — noise, and the
|
||||
unmaintained-but-not-vulnerable RUSTSEC-2024-0449 lives in vendor codegen we
|
||||
don't ship, for no benefit here.
|
||||
- **Output destination settled at M2, not now.** The mandatory `AlignedSbeWriter`
|
||||
takes a file path; a path can name stdout (`/dev/stdout`), or a tiny writer
|
||||
tweak. 64 MiB chunks are mandatory to stay readable by the template's
|
||||
`AlignedSbeReader::open`. *Rejected:* a generic sink abstraction — unnecessary.
|
||||
- **No CLI this sprint.** Everything hardcoded for the spike; a config lib would
|
||||
be speculative. *Rejected:* adding `clap` now.
|
||||
|
||||
## Not in this sprint
|
||||
|
||||
- Order book, event normalization, SBE output.
|
||||
- REST symbol discovery and the subscribe/unsubscribe diff loop.
|
||||
- Sharding across multiple sockets.
|
||||
- Redundancy, monitoring, SIGINT handling beyond the process default.
|
||||
- Kalshi, the generic exchange abstraction, latency work.
|
||||
|
||||
## Done when
|
||||
|
||||
- `scripts/ci.sh` is green (fmt + clippy `-D warnings` + test).
|
||||
- The program connects to Polymarket's market channel, subscribes to the
|
||||
hardcoded market, and logs raw frames to the terminal.
|
||||
- Polymarket's message shapes — including whether a sequence number, book hash,
|
||||
and/or exchange timestamp exist — are recorded in `docs/notes.md`.
|
||||
- The project `CLAUDE.md` is written: what it is, layout, commands, environment
|
||||
traps, design principles, workflow differences.
|
||||
|
||||
## Outcome
|
||||
|
||||
**Built.** `src/main.rs` now runs `src/polymarket.rs`, which connects to
|
||||
Polymarket's public market channel, subscribes to both tokens of one market,
|
||||
stamps the receive time right after each socket read, and logs every frame. Run
|
||||
against the live feed (60 s, then 15 s at close), it received the opening
|
||||
snapshot, price changes, a mid-stream book resend, and a trade. `scripts/ci.sh`
|
||||
(fmt, clippy `--all-targets -D warnings`, test) is green, and was checked to fail
|
||||
and stop on a violation. Also: the project `CLAUDE.md`, ADR 0001, and the market
|
||||
channel's message shapes in `docs/notes.md`.
|
||||
|
||||
**Diverged from the plan, and why.**
|
||||
- A direct `rustls` dependency with only `ring`: tungstenite leaves the crypto
|
||||
provider to the application, and rustls panicked at the first TLS connection
|
||||
without one. An echo-server smoke test caught it before Polymarket was
|
||||
reachable.
|
||||
- The subscribe message carries both tokens of the market, to see how the Yes and
|
||||
No books relate (they are mirrors).
|
||||
- `tempfile` removed: only the template's demo used it.
|
||||
- `## Workflow differences` moved from `docs/project.md` into `CLAUDE.md`, where
|
||||
the workflow's skills look for it.
|
||||
- Requirement 3 reworded from "sequence-gap detection" to "loss detection" in
|
||||
`docs/project.md` and `CLAUDE.md`. The sequence-number assumption came from
|
||||
Claude's early explanation, not from the brief, and Polymarket has none; the
|
||||
mechanism stays open.
|
||||
- Both exchanges turned out DNS-blocked on the dev network. The user pins their
|
||||
hosts with `../host.sh` (workspace root, outside the repo). Not planned; most
|
||||
of the sprint's unplanned time went here.
|
||||
|
||||
**Found, for later milestones** (in `docs/notes.md` and the roadmap): no
|
||||
sequence number on Polymarket, and a change's `hash` equals the next book's
|
||||
(M2); a `PING` keepalive every 10 s is required (M2); a symbol is a token, and the
|
||||
Yes/No books mirror each other (M2's data model); `new_market` /
|
||||
`market_resolved` events and subscribe/unsubscribe on an open connection (M3);
|
||||
the local clock runs ~30 ms slow (M7).
|
||||
|
||||
**Decided at close (the user):** research into how the `hash` is computed waits
|
||||
for M2.
|
||||
|
||||
**Left out.** The template's README still says the helpers are "demonstrated in
|
||||
`main.rs`" (moved to M8, which writes the solution README). The hardcoded market
|
||||
closes 2026-11-04, so rerunning the spike needs a fresh token id (its comment
|
||||
says so); M2 picks its own subset and M3 replaces hardcoding.
|
||||
|
||||
### Retrospective
|
||||
|
||||
- **Went well:** testing the transport against a neutral echo server caught the
|
||||
rustls panic while the exchange was unreachable. Checking the docs' claims
|
||||
against the live feed found what the docs don't say: books resent mid-stream,
|
||||
and a change's hash identifying the next book.
|
||||
- **Went wrong:** the network block surfaced only at implementation; a
|
||||
reachability check during planning would have found it a day earlier. And a
|
||||
general pattern (exchanges carry sequence numbers) entered the central bet's
|
||||
requirements as if it came from the brief; the live feed and the user's
|
||||
challenge caught it, not a review.
|
||||
- **Tested enough?** No tests, as agreed: nothing with logic yet. Not covered: a
|
||||
long-running connection (keepalive), reconnects, malformed frames — all M2.
|
||||
`host.sh` was tested on copies, then by its own check on the real file.
|
||||
- **Next time:** before a claim goes into a requirement, say where it comes from
|
||||
(the brief, the user, or a general pattern to verify); check the reachability
|
||||
of external services when planning.
|
||||
Executable
+10
@@ -0,0 +1,10 @@
|
||||
#!/bin/sh
|
||||
# The single CI entry point; run it before every commit.
|
||||
# One check per line: `set -e` doesn't apply on the left of `&&` / `||`, so
|
||||
# chaining would let a failing check pass silently.
|
||||
set -eu
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
cargo fmt --check
|
||||
cargo clippy --all-targets -- -D warnings
|
||||
cargo test
|
||||
+2
-63
@@ -1,66 +1,5 @@
|
||||
use chrono::DateTime;
|
||||
use engineering_challenge::Side;
|
||||
use engineering_challenge::util::dtos::challenge_orderbook::{
|
||||
AnyMessage, PriceLevelUpdateDomain, SnapshotAsksEntryDomain, SnapshotBidsEntryDomain,
|
||||
SnapshotDomain,
|
||||
};
|
||||
use engineering_challenge::util::{AlignedSbeReader, AlignedSbeWriter, OrderbookSchema};
|
||||
mod polymarket;
|
||||
|
||||
const CHUNK_SIZE_64_MEBIBYTES: usize = 64 << 20;
|
||||
|
||||
/// This is a dummy example where we define a snapshot and price level update,
|
||||
/// write them to a file using [`AlignedSbeWriter`], and then read them back
|
||||
/// from the same file using [`AlignedSbeReader`], asserting that what we read
|
||||
/// back is the exact same as what we wrote.
|
||||
///
|
||||
/// You should replace this main function with your actual solution, and split
|
||||
/// up your code into modules in `src/` in a way that makes sense to you.
|
||||
fn main() -> anyhow::Result<()> {
|
||||
let timestamp = DateTime::from_timestamp_micros(1_700_000_000_000_000).unwrap();
|
||||
let symbol = "EXAMPLE".to_owned();
|
||||
let snapshot = SnapshotDomain {
|
||||
received_at: timestamp,
|
||||
exchange_timestamp: timestamp,
|
||||
bids: vec![SnapshotBidsEntryDomain {
|
||||
price_level: 400_000,
|
||||
quantity: 2_000_000,
|
||||
}],
|
||||
asks: vec![SnapshotAsksEntryDomain {
|
||||
price_level: 600_000,
|
||||
quantity: 3_500_000,
|
||||
}],
|
||||
symbol: symbol.clone(),
|
||||
};
|
||||
let update = PriceLevelUpdateDomain {
|
||||
backtest_timestamp: timestamp,
|
||||
exchange_timestamp: timestamp,
|
||||
price_level: 400_000,
|
||||
quantity: 1_250_000,
|
||||
side: Side::Buy,
|
||||
symbol: symbol.clone(),
|
||||
};
|
||||
|
||||
let file = tempfile::NamedTempFile::new()?;
|
||||
let mut writer = AlignedSbeWriter::create(file.path(), CHUNK_SIZE_64_MEBIBYTES)?;
|
||||
writer.write_msg(&snapshot)?;
|
||||
writer.write_msg(&update)?;
|
||||
writer.flush()?;
|
||||
drop(writer);
|
||||
|
||||
let mut reader = AlignedSbeReader::<OrderbookSchema>::open(file.path())?;
|
||||
|
||||
let Some(AnyMessage::Snapshot(decoded)) = reader.read_msg()? else {
|
||||
anyhow::bail!("expected snapshot");
|
||||
};
|
||||
assert_eq!(SnapshotDomain::try_from_decoder(decoded)?, snapshot);
|
||||
|
||||
let Some(AnyMessage::PriceLevelUpdate(decoded)) = reader.read_msg()? else {
|
||||
anyhow::bail!("expected price level update");
|
||||
};
|
||||
assert_eq!(PriceLevelUpdateDomain::try_from_decoder(decoded)?, update);
|
||||
assert!(reader.read_msg()?.is_none());
|
||||
|
||||
println!("Verification success!");
|
||||
|
||||
Ok(())
|
||||
polymarket::run_spike()
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
//! Connectivity spike: connect to Polymarket's public market channel, subscribe
|
||||
//! to one market, and log every raw frame with its receive time.
|
||||
|
||||
use chrono::{SecondsFormat, Utc};
|
||||
use tungstenite::Message;
|
||||
|
||||
const MARKET_CHANNEL_URL: &str = "wss://ws-subscriptions-clob.polymarket.com/ws/market";
|
||||
|
||||
/// Both tokens (Yes, No) of one market, hardcoded for the spike (discovery comes
|
||||
/// in M3): "2026 Balance of Power: D Senate, D House", which closes 2026-11-04,
|
||||
/// so pick a fresh market to rerun this.
|
||||
const SUBSCRIBE_MESSAGE: &str = r#"{"type":"market","assets_ids":["34722410608062854697106861099776685947172185964394483545370684749662285977831","24483563228660028121020430722896740519524131851526579619586176931221901842798"]}"#;
|
||||
|
||||
/// Connects, sends [`SUBSCRIBE_MESSAGE`], and logs frames until the server
|
||||
/// closes the connection.
|
||||
pub fn run_spike() -> anyhow::Result<()> {
|
||||
let (mut socket, _response) = tungstenite::connect(MARKET_CHANNEL_URL)?;
|
||||
socket.send(Message::text(SUBSCRIBE_MESSAGE))?;
|
||||
|
||||
loop {
|
||||
let message = socket.read()?;
|
||||
// Stamp as soon as the frame is off the wire, before any processing.
|
||||
let received_at = Utc::now().to_rfc3339_opts(SecondsFormat::Micros, true);
|
||||
match message {
|
||||
Message::Text(text) => println!("{received_at} {text}"),
|
||||
Message::Close(frame) => {
|
||||
println!("{received_at} close {frame:?}");
|
||||
return Ok(());
|
||||
}
|
||||
other => println!("{received_at} {other:?}"),
|
||||
}
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user