diff --git a/CLAUDE.md b/CLAUDE.md index 5d415eb..950f59d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -2,8 +2,7 @@ A collector for prediction-market order books (Polymarket, then Kalshi) that emits SBE-encoded updates and per-minute snapshots, built inside the challenge -template. The brief is `../website/Engineering Challenge _ Voluntas.html`, -outside this repo. +template. The brief is https://voluntas.io/engineering-challenge. ## Where things are @@ -80,11 +79,8 @@ and let the user settle it. Claude owns the data model. ## Environment traps -- **DNS: the exchanges resolve normally on the server** (a remote-claude - workspace, since 2026-11-09). The Mac's dev network blocks them (its resolver - answers `195.186.4.210` / `2001:4d98:4:b10c::1`, a Swisscom block page); - there the user pins the hosts with `sudo ../host.sh on` (workspace root, - outside this repo; `off` removes them), rerun when the CDN addresses rotate. +- **DNS: a network can block the exchanges** (its resolver answers a block + page's address): check what the hosts resolve to before blaming the code. A 403 is not DNS (see `docs/notes.md`, REST from scripts). - **Write 64 MiB chunks**: `AlignedSbeReader::open` hardcodes 64 MiB (`open_with_chunk_size` is private), so any other chunk size is unreadable. @@ -120,9 +116,8 @@ and let the user settle it. Claude owns the data model. Check `cat /proc/loadavg` before a live run whose numbers are compared, and alternate the builds in short rounds rather than running them back to back. - **Kalshi needs a key; nothing else does**: `KALSHI_KEY_ID_FILE` and - `KALSHI_PRIVATE_KEY_FILE` name its two files (on the server - `/run/secrets/kalshi-demo-key-id` and `kalshi-demo-private-key`; never read - or print them). The key is Ed25519 or RSA (RSA-PSS-SHA256), told apart + `KALSHI_PRIVATE_KEY_FILE` name its two files (never read or print + them). The key is Ed25519 or RSA (RSA-PSS-SHA256), told apart by the parsed key. Opening ~100 connections at once gets some handshakes refused with 429; each retries after 1 s and gets in. - **cargo keeps a build whose source got an older mtime**: a script that diff --git a/LATENCY.md b/LATENCY.md index 366c091..5398e66 100644 --- a/LATENCY.md +++ b/LATENCY.md @@ -1,7 +1,7 @@ # Latency experiments Extension 4 of the challenge: ways to get the exchanges' changes sooner, -tried on Polymarket and Kalshi. Every trick the brief names (connecting +tried on Polymarket and Kalshi's demo. Every trick the brief names (connecting first after maintenance, spamming connections until a fast one comes, changing the TCP source port) rests on one question: **is a connection that is fast now still fast later?** If its speed is noise from message to @@ -177,8 +177,8 @@ as the median lag), but at the median it wins 9 ms, a tenth of the lag. - **Connections differ, and keep their speed**: strongly on Polymarket (rank correlation 0.96 and 0.98), less on Kalshi's demo (0.66 and 0.78). - Nothing we could choose at connect sets it (the address doesn't): it is - drawn per connection, mostly when it opens, though it can shift mid-way + The one thing we could choose at connect, the address, doesn't set it + (the source port could not be tried): it is drawn per connection, mostly when it opens, though it can shift mid-way (Kalshi's connection 4 went from 75 to 3 ms over its second run). - **On Polymarket it is worth under a millisecond.** Keeping the fastest pair wins at the median, not in the tail, where bursts delay every @@ -211,8 +211,8 @@ as the median lag), but at the median it wins 9 ms, a tenth of the lag. connections, on the 100 busiest markets, differed far more than the collector's; assumed to be because each of the collector's shares its markets with many others and is judged against all their copies, which - evens the means out. So the mechanism is tested but was not seen - to fire live, and its gain on the collector was not measured; how the + evens the means out. So the mechanism is tested and was seen to fire + live, but its gain on the collector was not measured; how the demo's figures carry to production, which runs elsewhere, is unknown. - **Distance weighs more**: 14 ms of lag to Polymarket, about 100 ms to Kalshi's demo, against a median gain from racing under 1 ms and of 9 ms. diff --git a/README.md b/README.md index c3f5217..f7194e6 100644 --- a/README.md +++ b/README.md @@ -4,15 +4,17 @@ My answer to the Voluntas engineering challenge. The collector subscribes to every live market of one exchange (Polymarket, or Kalshi), keeps an order book for each symbol, and writes the template's SBE messages: a Delta for every change, and a Snapshot of every book each minute and whenever a book -is replaced. All four extensions are done: redundant connections, +is replaced. The four extensions are built: redundant connections, subscription monitoring, one core for both exchanges, and latency -experiments ([LATENCY.md](LATENCY.md)). +experiments ([LATENCY.md](LATENCY.md); of the brief's three ideas, two +could not be tried from here). What each leaves out is under +[Limits](#limits). -Built with Claude Code. I set the architecture (the threading model, -redundancy, the exchange seam) and settled every design decision; Claude -proposed the options, wrote the code and ran the experiments. Each -decision and its reasons are in [`roadmap/`](roadmap) and -[`docs/adr/`](docs/adr). +The design is mine: the threading model, the redundancy (two connections +per symbol, first one wins), the exchange seam, and every design decision, +each with its reasons in [`roadmap/`](roadmap) and [`docs/adr/`](docs/adr). +I built it with Claude Code, which proposed the options, wrote the code and +ran the experiments. ## Quick start @@ -156,8 +158,9 @@ code relies on, several found only in live data, are in - **Subscription monitoring.** A subscription is quiet when it delivered nothing for a minute while the other connection delivered at least 3 changes of the same symbol: it is resubscribed, and moved if quiet - again. Judged against the other connection, not a rate, it holds for - the quietest markets too. Every + again. Judged against the other connection, not a rate, it needs no + threshold per market; it can't judge a market under 3 changes a minute, + nor one quiet on both connections at once (Limits). Every connection is recycled about once an hour, one at a time, never two of a market's; one silent for 30 s reconnects. - **Generic core.** `src/core/` is the same for both exchanges. An exchange @@ -166,14 +169,28 @@ code relies on, several found only in live data, are in decoding, since telling a duplicate from a loss depends on what the exchange sends (a book hash, or a sequence number and a clock). - **Latency.** [LATENCY.md](LATENCY.md): a connection that delivers - changes sooner than the others keeps doing so, on both exchanges. - Keeping the fastest gains under a millisecond on Polymarket, and tens of - milliseconds on Kalshi's demo, where the collector now recycles a - connection that stays slow. + changes sooner than the others keeps doing so, on Polymarket and on + Kalshi's demo. Keeping the fastest gains under a millisecond on + Polymarket, and tens of milliseconds on Kalshi's demo, where the + collector now recycles a connection that stays slow. Connecting first + after maintenance and choosing the TCP source port were not tried; the + file says why, and how each would be. ## Limits - A hard crash (SIGKILL, power) loses the chunk being filled, up to 64 MiB. +- Nothing covers the process itself going down, the fourth failure the + brief's extension 1 names: it is one process, and starting it again on + the same output path truncates the file. A second collector on another + host, the two outputs merged downstream, would cover it; not built. +- Subscription monitoring compares a symbol's two connections, so it + doesn't see a market under 3 changes a minute, nor a symbol quiet on both + at once: only the next fresh book shows what was missed. A rate per + symbol, as the brief describes, would see the second. +- On Kalshi, a change that one connection skipped without a gap in its + sequence numbers would be dropped as a duplicate when the other delivers + it, and the book would stay wrong until the next fresh book. Never + observed; the rule rests on the demo's behaviour. - When a symbol is subscribed, the exchange's opening book holds a few changes it never sends as changes: logged as a subscription gap. - Polymarket changes held for ordering when the run stops (at most 1 s) are diff --git a/scripts/test_watch.sh b/scripts/test_watch.sh index 63d1f9a..1422ce0 100755 --- a/scripts/test_watch.sh +++ b/scripts/test_watch.sh @@ -10,19 +10,24 @@ set -eu cd "$(dirname "$0")/.." failed=0 -# check NAME LOG EXPECTED [AWK OPTION...] +# The sample logs are files and what the view must show comes on stdin: no +# here-document sits inside `$( )`, where the bash 3.2 that is macOS's sh +# stops at an apostrophe. +tmp=$(mktemp -d) +trap 'rm -rf "$tmp"' EXIT +# check NAME LOG [AWK OPTION...], the expected view on stdin check() { - name=$1 log=$2 expected=$3 - shift 3 - actual=$(printf '%s\n' "$log" | awk -v book_threads=2 "$@" -f scripts/watch.awk) + name=$1 log=$2 + shift 2 + expected=$(cat) + actual=$(awk -v book_threads=2 "$@" -f scripts/watch.awk <"$tmp/$log") if [ "$actual" != "$expected" ]; then printf 'watch.awk, %s: printed\n%s\nexpected\n%s\n' "$name" "$actual" "$expected" failed=1 fi } -polymarket=$( - cat <<'EOF' +cat >"$tmp/polymarket" <<'EOF' 12:00:00.100 polymarket: orchestrator: first scan, listing the exchange's markets 12:00:30.200 polymarket: orchestrator: first scan done, 40 live markets, subscribing 12:00:30.300 polymarket: connection 0: connected, 40 symbols @@ -37,10 +42,8 @@ polymarket=$( 12:02:11.000 polymarket: connection 1: connected, 40 symbols 12:02:30.400 polymarket: orchestrator: 40 markets on 2 connections; 0 added, 0 removed, 0 moved (0 quiet), 0 splits, 1 recycled (0 slow), 0 silent this minute; busiest connection at 1300 bytes/s EOF -) -check "Polymarket, live" "$polymarket" "$( - cat <<'EOF' +check "Polymarket, live" polymarket <<'EOF' 12:00:00 polymarket: orchestrator: first scan, listing the exchange's markets 12:00:30 polymarket: orchestrator: first scan done, 40 live markets, subscribing 12:00:30 connections opening (counted in the minute line) @@ -50,78 +53,60 @@ check "Polymarket, live" "$polymarket" "$( 12:02:10 polymarket: connection 1: recycled 12:02:30 40 markets on 2 connections; 0 added, 0 removed, 0 moved (0 quiet), 0 splits, 1 recycled (0 slow), 0 silent this minute; busiest connection at 1300 bytes/s; 1 connected (0 refused, retried), 0 Snapshots with a cause EOF -)" -check "Polymarket, final" "$polymarket" \ - "12:02 books (2 threads): 40 symbols, 660 changes written, 330 duplicates, 9 rescued, 5 resyncs, 1 subscription gaps, 3 losses" \ - -v final=1 +check "Polymarket, final" polymarket -v final=1 <<'EOF' +12:02 books (2 threads): 40 symbols, 660 changes written, 330 duplicates, 9 rescued, 5 resyncs, 1 subscription gaps, 3 losses +EOF # A minute with one of its two book threads' lines: printed with the first # line 10 s after it, before the orchestrator's. -missing=$( - cat <<'EOF' +cat >"$tmp/missing" <<'EOF' 12:01:00.001 polymarket: book thread 0: 10 symbols; so far 100 changes written (90 verified), 50 duplicates, 2 rescued, 0 held dropped, 1 chains broken, 3 resyncs, 0 full books behind ours, 1 subscription gaps, 2 losses, 0 switches, 0 quiet subscriptions resubscribed, 0 markets reported quiet again, 4 reconnect books judged (1 losses); duplicates' lateness 1.5 ms at the median, 9.0 ms at p99; this minute 150 changes received, at most 4 events waited 12:01:00.500 polymarket: connection 1: recycled 12:01:30.400 polymarket: orchestrator: 40 markets on 2 connections; 0 added, 0 removed, 0 moved (0 quiet), 0 splits, 1 recycled (0 slow), 0 silent this minute; busiest connection at 1200 bytes/s EOF -) -check "a book thread missing" "$missing" "$( - cat <<'EOF' +check "a book thread missing" missing <<'EOF' 12:01:00 polymarket: connection 1: recycled 12:01 books (1 threads): 10 symbols, 100 changes written, 50 duplicates, 2 rescued, 3 resyncs, 1 subscription gaps, 2 losses 12:01:30 40 markets on 2 connections; 0 added, 0 removed, 0 moved (0 quiet), 0 splits, 1 recycled (0 slow), 0 silent this minute; busiest connection at 1200 bytes/s; 0 connected (0 refused, retried), 0 Snapshots with a cause EOF -)" # A thread's line 1.6 s after the first, past a line between them: still # the same minute's, summed whole. -late=$( - cat <<'EOF' +cat >"$tmp/late" <<'EOF' 12:01:00.075 polymarket: book thread 0: 10 symbols; so far 100 changes written (90 verified), 50 duplicates, 2 rescued, 0 held dropped, 1 chains broken, 3 resyncs, 0 full books behind ours, 1 subscription gaps, 2 losses, 0 switches, 0 quiet subscriptions resubscribed, 0 markets reported quiet again, 4 reconnect books judged (1 losses); duplicates' lateness 1.5 ms at the median, 9.0 ms at p99; this minute 150 changes received, at most 4 events waited 12:01:01.600 polymarket: connection 1: recycled 12:01:01.672 polymarket: book thread 1: 30 symbols; so far 300 changes written (290 verified), 150 duplicates, 4 rescued, 0 held dropped, 0 chains broken, 1 resyncs, 0 full books behind ours, 0 subscription gaps, 1 losses, 0 switches, 0 quiet subscriptions resubscribed, 0 markets reported quiet again, 0 reconnect books judged (0 losses); duplicates' lateness 2.0 ms at the median, 8.0 ms at p99; this minute 450 changes received, at most 9 events waited EOF -) -check "a thread 1.6 s late" "$late" "$( - cat <<'EOF' +check "a thread 1.6 s late" late <<'EOF' 12:01:01 polymarket: connection 1: recycled 12:01 books (2 threads): 40 symbols, 400 changes written, 200 duplicates, 6 rescued, 4 resyncs, 1 subscription gaps, 3 losses EOF -)" -check "a thread 1.6 s late, final" "$late" \ - "12:01 books (2 threads): 40 symbols, 400 changes written, 200 duplicates, 6 rescued, 4 resyncs, 1 subscription gaps, 3 losses" \ - -v final=1 +check "a thread 1.6 s late, final" late -v final=1 <<'EOF' +12:01 books (2 threads): 40 symbols, 400 changes written, 200 duplicates, 6 rescued, 4 resyncs, 1 subscription gaps, 3 losses +EOF # Across midnight: a minute begun at 23:59 still ends 10 s on. -midnight=$( - cat <<'EOF' +cat >"$tmp/midnight" <<'EOF' 23:59:00.001 polymarket: book thread 0: 10 symbols; so far 100 changes written (90 verified), 50 duplicates, 2 rescued, 0 held dropped, 1 chains broken, 3 resyncs, 0 full books behind ours, 1 subscription gaps, 2 losses, 0 switches, 0 quiet subscriptions resubscribed, 0 markets reported quiet again, 4 reconnect books judged (1 losses); duplicates' lateness 1.5 ms at the median, 9.0 ms at p99; this minute 150 changes received, at most 4 events waited 00:00:30.000 polymarket: connection 1: recycled EOF -) -check "across midnight" "$midnight" "$( - cat <<'EOF' +check "across midnight" midnight <<'EOF' 23:59 books (1 threads): 10 symbols, 100 changes written, 50 duplicates, 2 rescued, 3 resyncs, 1 subscription gaps, 2 losses 00:00:30 polymarket: connection 1: recycled EOF -)" # A line between a minute's book threads: the minute is still summed whole. -interleaved=$( - cat <<'EOF' +cat >"$tmp/interleaved" <<'EOF' 12:01:00.001 polymarket: book thread 0: 10 symbols; so far 100 changes written (90 verified), 50 duplicates, 2 rescued, 0 held dropped, 1 chains broken, 3 resyncs, 0 full books behind ours, 1 subscription gaps, 2 losses, 0 switches, 0 quiet subscriptions resubscribed, 0 markets reported quiet again, 4 reconnect books judged (1 losses); duplicates' lateness 1.5 ms at the median, 9.0 ms at p99; this minute 150 changes received, at most 4 events waited 12:01:00.002 polymarket: connection 1: recycled 12:01:00.003 polymarket: book thread 1: 30 symbols; so far 300 changes written (290 verified), 150 duplicates, 4 rescued, 0 held dropped, 0 chains broken, 1 resyncs, 0 full books behind ours, 0 subscription gaps, 1 losses, 0 switches, 0 quiet subscriptions resubscribed, 0 markets reported quiet again, 0 reconnect books judged (0 losses); duplicates' lateness 2.0 ms at the median, 8.0 ms at p99; this minute 450 changes received, at most 9 events waited EOF -) -check "a line between book threads" "$interleaved" "$( - cat <<'EOF' +check "a line between book threads" interleaved <<'EOF' 12:01:00 polymarket: connection 1: recycled 12:01 books (2 threads): 40 symbols, 400 changes written, 200 duplicates, 6 rescued, 4 resyncs, 1 subscription gaps, 3 losses EOF -)" -kalshi=$( - cat <<'EOF' +cat >"$tmp/kalshi" <<'EOF' 12:00:00.100 kalshi: orchestrator: first scan, listing the exchange's markets 12:00:10.100 kalshi: orchestrator: first scan, 1000 live markets listed so far 12:00:20.200 kalshi: orchestrator: first scan done, 1500 live markets, subscribing @@ -136,10 +121,8 @@ kalshi=$( 12:02:00.002 kalshi: book thread 1: 800 symbols; so far 260 changes written, 250 duplicates, 0 before their connection's book, 1 ahead of the book, 2 fresh books (0 losses), 1 below zero, 1 resyncs, 0 quiet subscriptions resubscribed, 0 markets reported quiet again; this minute 120 changes received, at most 4 events waited 12:02:20.400 kalshi: orchestrator: 1500 markets on 2 connections; 0 added, 0 removed, 0 moved (0 quiet), 0 splits, 0 recycled (0 slow), 0 silent this minute; busiest connection at 8000 bytes/s EOF -) -check "Kalshi, live" "$kalshi" "$( - cat <<'EOF' +check "Kalshi, live" kalshi <<'EOF' 12:00:00 kalshi: orchestrator: first scan, listing the exchange's markets 12:00:10 kalshi: orchestrator: first scan, 1000 live markets listed so far 12:00:20 kalshi: orchestrator: first scan done, 1500 live markets, subscribing @@ -149,9 +132,8 @@ check "Kalshi, live" "$kalshi" "$( 12:02 books (2 threads): 1500 symbols, 410 changes written, 390 duplicates, 3 resyncs, 1 losses 12:02:20 1500 markets on 2 connections; 0 added, 0 removed, 0 moved (0 quiet), 0 splits, 0 recycled (0 slow), 0 silent this minute; busiest connection at 8000 bytes/s; 0 connected (0 refused, retried), 0 Snapshots with a cause EOF -)" -check "Kalshi, final" "$kalshi" \ - "12:02 books (2 threads): 1500 symbols, 410 changes written, 390 duplicates, 3 resyncs, 1 losses" \ - -v final=1 +check "Kalshi, final" kalshi -v final=1 <<'EOF' +12:02 books (2 threads): 1500 symbols, 410 changes written, 390 duplicates, 3 resyncs, 1 losses +EOF exit "$failed"