feat: ready to submit, a run to watch, the README, the review's fixes

Sprint 08 (roadmap/08. ready to submit.md), the last milestone.

- scripts/watch.sh runs the collector with a live view of its log, stops
  cleanly on Ctrl-C, then checks the output (reconcile) and shows one
  symbol's book as text (show, a new binary).
- The README for the solution, with the final runs: Polymarket through an
  outage it recovered from on its own, Kalshi's demo with two slow
  connections recycled.
- From a review of the whole codebase: Kalshi takes RSA keys too
  (RSA-PSS-SHA256); a slow Kalshi connection takes the hourly recycle's
  next turn, which it nearly never got before; a market tracked again no
  longer fails reconcile; Book::add checks for overflow; usage and --help
  name the key variables; the first scan says how far it got.
- Tests for every gap the reviews found, an offline run of the whole
  orchestrator among them; ShellCheck in CI.
This commit is contained in:
pascal committed 2026-11-12 22:32:39 +01:00
1 parent 515e85941d
commit cf6c16c9cf
35 files changed
+2791 -207

No files matched your search

+101
View File
@@ -0,0 +1,101 @@
#!/bin/sh
# watch.sh [EXCHANGE] [MINUTES]: runs the collector and shows what it does.
# EXCHANGE is polymarket (the default, no key needed), kalshi or kalshi-demo
# (KALSHI_KEY_ID_FILE and KALSHI_PRIVATE_KEY_FILE exported); MINUTES
# defaults to 3. Ctrl-C stops the run early, cleanly.
#
# The output and the full log go to watch/; the terminal shows, each minute,
# the orchestrator's line and the book threads' summed (scripts/watch.awk),
# and the notable events between. Then the output is checked against the
# log (reconcile) and one symbol is shown as text (show). It exits non-zero
# when the collector failed or the output doesn't reconcile.
set -eu
cd "$(dirname "$0")/.."
exchange=${1:-polymarket}
minutes=${2:-3}
case $exchange in
polymarket | kalshi | kalshi-demo) ;;
*)
echo "usage: scripts/watch.sh [polymarket|kalshi|kalshi-demo] [MINUTES]" >&2
exit 2
;;
esac
echo "building (cargo build --release)"
cargo build --release --quiet
mkdir -p watch
out=watch/$exchange.sbe log=watch/$exchange.log memory=watch/$exchange.memory
rm -f "$out" "$log" "$memory"
: >"$log"
echo "load before the run: $(cut -d' ' -f1-3 /proc/loadavg) (1, 5, 15 min)"
echo "collecting $exchange for $minutes minutes into $out (log: $log); Ctrl-C stops it"
echo "the markets are listed first (a minute or two), then subscribed"
timeout -s INT "${minutes}m" ./target/release/engineering-challenge "$exchange" "$out" 2>>"$log" &
run=$!
# The view ends by itself once the run is over and the log read to its end.
# mawk reads a pipe a buffer at a time unless interactive: the view would
# show nothing until the end.
live=
if awk -W version 2>/dev/null | grep -q mawk; then live='-W interactive'; fi
# shellcheck disable=SC2086 # $live is empty or two words, both wanted.
tail -n +1 -F --pid="$run" "$log" 2>/dev/null | awk $live -f scripts/watch.awk &
view=$!
# A job started with & ignores Ctrl-C (sh without job control), so the
# script passes it on: timeout hands SIGINT to the collector, which stops
# cleanly, and the script carries on to the checks. The view, also a job,
# keeps showing the end.
trap 'echo; echo "stopping (Ctrl-C)"; kill -INT "$run" 2>/dev/null || true' INT
# The collector's peak memory (VmHWM, the kernel's own peak) and most
# threads, sampled every 10 s while it runs, from /proc: /usr/bin/time isn't
# installed everywhere. The last sample may fall in the shutdown, when the
# connections' threads are gone: the most seen is kept.
(
sleep 1
pid=$(pgrep -P "$run" || true)
most=0
while [ -n "$pid" ] && kill -0 "$pid" 2>/dev/null; do
proc=$(cat "/proc/$pid/status" 2>/dev/null || true)
peak=$(printf '%s\n' "$proc" | awk '/^VmHWM:/ {print $2}')
threads=$(printf '%s\n' "$proc" | awk '/^Threads:/ {print $2}')
if [ -n "$threads" ] && [ "$threads" -gt "$most" ]; then most=$threads; fi
if [ -n "$peak" ]; then echo "$peak kB, at most $most threads" >"$memory"; fi
sleep 10
done
) &
# wait returns early when Ctrl-C interrupts it: wait again until the
# collector has flushed and exited.
status=0
while kill -0 "$run" 2>/dev/null; do
wait "$run" && status=0 || status=$?
done
trap - INT
wait "$view" || true
# 124: stopped by timeout, at the end or on Ctrl-C (passed on by it).
failed=0
case $status in
0 | 124 | 130) ;;
*)
echo "the collector failed (exit $status); its log: $log"
failed=1
;;
esac
if [ ! -e "$out" ]; then
echo "no output was written"
exit 1
fi
echo
echo "== totals at the last minute"
awk -v final=1 -f scripts/watch.awk "$log"
if [ -s "$memory" ]; then echo "peak memory $(cat "$memory")"; fi
echo
echo "== the output against the log (reconcile)"
./target/release/reconcile "$out" "$log" || failed=1
echo
echo "== one symbol, as text (show)"
# An output stopped before any book is empty: said, not a failure.
./target/release/show "$out" || true
exit "$failed"