Archived
feat(polymarket): hash-verified order books for a subset of symbols
Collects the order books of Polymarket's 20 busiest markets (40 symbols) into the template's SBE format: a Delta per level change, plus Snapshots on subscribe, every minute, and whenever a fresh book corrects ours. Every change is checked against Polymarket's own book hash; a failed check resyncs the symbol on the open socket, and the log counts losses apart from the few ms our own resync hides. Runs until SIGINT, which closes the socket and flushes the output; reconnects when the server drops it. A `reconcile` binary checks an output file against itself.
This commit is contained in:
22 files changed
+6423
-84
No files matched your search
@@ -14,6 +14,11 @@ outside this repo.
|
||||
- `docs/adr/`: hard-to-reverse decisions (0001: threads, shared-nothing).
|
||||
- `docs/notes.md`: domain rules and exchange protocol facts (delta semantics,
|
||||
loss detection, message shapes). Read before touching exchange parsing.
|
||||
- `GLOSSARY.md`: the terms (symbol, resync, loss, resync gap, Snapshot, Delta).
|
||||
- `src/polymarket/feed.rs`: the book thread and its loss rules (the subtle
|
||||
part); `src/polymarket/socket.rs`: the socket thread; `src/book.rs`: the
|
||||
book and fixed point; `src/reconcile.rs` + `src/bin/reconcile.rs`: the output
|
||||
check. `tests/fixtures/`: a 90 s recording of the live feed the tests replay.
|
||||
- `src/util/` and `sbe_schemas/`: the template's SBE machinery, used as given so
|
||||
output stays readable by the template's `AlignedSbeReader`. Our code is the
|
||||
rest of `src/`.
|
||||
@@ -31,9 +36,11 @@ and let the user settle it. Claude owns the data model.
|
||||
plain threads and channels. Speed work waits for the profiler, after V1.
|
||||
- An abstraction arrives with its second implementation (the exchange trait in
|
||||
M6).
|
||||
- Stamp receive time immediately after the socket read, and rebuild a book from
|
||||
a fresh snapshot whenever a lost message is detected (how depends on the
|
||||
exchange: `docs/notes.md`).
|
||||
- Stamp receive time immediately after the socket read, and resync a book (a
|
||||
fresh full book from the exchange) whenever a loss is detected (how depends
|
||||
on the exchange: `docs/notes.md`).
|
||||
- Use `GLOSSARY.md`'s terms: a **Snapshot** is only the output message; getting
|
||||
a fresh book is a **resync**.
|
||||
- A corner cut for speed carries a comment naming its ceiling and upgrade path.
|
||||
|
||||
## Environment traps
|
||||
@@ -42,12 +49,21 @@ and let the user settle it. Claude owns the data model.
|
||||
resolver answers `195.186.4.210` / `2001:4d98:4:b10c::1` (a Swisscom block
|
||||
page). The user pins the hosts with `sudo ../host.sh on` (workspace root,
|
||||
outside this repo; `off` removes them). A timeout or the block address again
|
||||
means the CDN addresses rotated: ask the user to rerun it.
|
||||
means the CDN addresses rotated: ask the user to rerun it. A host not yet in
|
||||
`host.sh`'s `HOSTS` list fails the same way: add it there and ask the user to
|
||||
rerun it. A 403 is not DNS (see `docs/notes.md`, REST from scripts).
|
||||
- **Write 64 MiB chunks**: `AlignedSbeReader::open` hardcodes 64 MiB
|
||||
(`open_with_chunk_size` is private), so any other chunk size is unreadable.
|
||||
- **The binary runs until the server closes the socket.** macOS has no
|
||||
`timeout`; bound a run with
|
||||
`perl -e 'alarm 10; exec @ARGV' ./target/debug/engineering-challenge`.
|
||||
- **The collector runs until SIGINT, and only SIGINT flushes its output.**
|
||||
macOS has no `timeout`, and a `perl alarm` kill (SIGALRM) leaves the last
|
||||
64 MiB chunk unwritten. Bound a run with SIGINT, then check the output:
|
||||
`./target/release/engineering-challenge out.sbe 2> run.log & pid=$!;
|
||||
sleep 600; kill -INT $pid; wait $pid; ./target/release/reconcile out.sbe`.
|
||||
`reconcile` counts resync gaps as mismatches too: compare its list with the
|
||||
log's "loss" and "resync gap" lines.
|
||||
- **Live data beats the docs**: Polymarket's market channel has rules its
|
||||
docs don't state (`docs/notes.md`); check a protocol assumption on a
|
||||
recording of the real feed (`tests/fixtures/record.py`) before relying on it.
|
||||
|
||||
## Workflow differences
|
||||
|
||||
|
||||
Reference in new issue
Block a user