Archived
feat: ready to submit, a run to watch, the README, the review's fixes
Sprint 08 (roadmap/08. ready to submit.md), the last milestone. - scripts/watch.sh runs the collector with a live view of its log, stops cleanly on Ctrl-C, then checks the output (reconcile) and shows one symbol's book as text (show, a new binary). - The README for the solution, with the final runs: Polymarket through an outage it recovered from on its own, Kalshi's demo with two slow connections recycled. - From a review of the whole codebase: Kalshi takes RSA keys too (RSA-PSS-SHA256); a slow Kalshi connection takes the hourly recycle's next turn, which it nearly never got before; a market tracked again no longer fails reconcile; Book::add checks for overflow; usage and --help name the key variables; the first scan says how far it got. - Tests for every gap the reviews found, an offline run of the whole orchestrator among them; ShellCheck in CI.
This commit is contained in:
35 files changed
+2791
-207
No files matched your search
Executable
+101
@@ -0,0 +1,101 @@
|
||||
#!/bin/sh
|
||||
# watch.sh [EXCHANGE] [MINUTES]: runs the collector and shows what it does.
|
||||
# EXCHANGE is polymarket (the default, no key needed), kalshi or kalshi-demo
|
||||
# (KALSHI_KEY_ID_FILE and KALSHI_PRIVATE_KEY_FILE exported); MINUTES
|
||||
# defaults to 3. Ctrl-C stops the run early, cleanly.
|
||||
#
|
||||
# The output and the full log go to watch/; the terminal shows, each minute,
|
||||
# the orchestrator's line and the book threads' summed (scripts/watch.awk),
|
||||
# and the notable events between. Then the output is checked against the
|
||||
# log (reconcile) and one symbol is shown as text (show). It exits non-zero
|
||||
# when the collector failed or the output doesn't reconcile.
|
||||
set -eu
|
||||
cd "$(dirname "$0")/.."
|
||||
exchange=${1:-polymarket}
|
||||
minutes=${2:-3}
|
||||
case $exchange in
|
||||
polymarket | kalshi | kalshi-demo) ;;
|
||||
*)
|
||||
echo "usage: scripts/watch.sh [polymarket|kalshi|kalshi-demo] [MINUTES]" >&2
|
||||
exit 2
|
||||
;;
|
||||
esac
|
||||
|
||||
echo "building (cargo build --release)"
|
||||
cargo build --release --quiet
|
||||
mkdir -p watch
|
||||
out=watch/$exchange.sbe log=watch/$exchange.log memory=watch/$exchange.memory
|
||||
rm -f "$out" "$log" "$memory"
|
||||
: >"$log"
|
||||
|
||||
echo "load before the run: $(cut -d' ' -f1-3 /proc/loadavg) (1, 5, 15 min)"
|
||||
echo "collecting $exchange for $minutes minutes into $out (log: $log); Ctrl-C stops it"
|
||||
echo "the markets are listed first (a minute or two), then subscribed"
|
||||
timeout -s INT "${minutes}m" ./target/release/engineering-challenge "$exchange" "$out" 2>>"$log" &
|
||||
run=$!
|
||||
# The view ends by itself once the run is over and the log read to its end.
|
||||
# mawk reads a pipe a buffer at a time unless interactive: the view would
|
||||
# show nothing until the end.
|
||||
live=
|
||||
if awk -W version 2>/dev/null | grep -q mawk; then live='-W interactive'; fi
|
||||
# shellcheck disable=SC2086 # $live is empty or two words, both wanted.
|
||||
tail -n +1 -F --pid="$run" "$log" 2>/dev/null | awk $live -f scripts/watch.awk &
|
||||
view=$!
|
||||
# A job started with & ignores Ctrl-C (sh without job control), so the
|
||||
# script passes it on: timeout hands SIGINT to the collector, which stops
|
||||
# cleanly, and the script carries on to the checks. The view, also a job,
|
||||
# keeps showing the end.
|
||||
trap 'echo; echo "stopping (Ctrl-C)"; kill -INT "$run" 2>/dev/null || true' INT
|
||||
|
||||
# The collector's peak memory (VmHWM, the kernel's own peak) and most
|
||||
# threads, sampled every 10 s while it runs, from /proc: /usr/bin/time isn't
|
||||
# installed everywhere. The last sample may fall in the shutdown, when the
|
||||
# connections' threads are gone: the most seen is kept.
|
||||
(
|
||||
sleep 1
|
||||
pid=$(pgrep -P "$run" || true)
|
||||
most=0
|
||||
while [ -n "$pid" ] && kill -0 "$pid" 2>/dev/null; do
|
||||
proc=$(cat "/proc/$pid/status" 2>/dev/null || true)
|
||||
peak=$(printf '%s\n' "$proc" | awk '/^VmHWM:/ {print $2}')
|
||||
threads=$(printf '%s\n' "$proc" | awk '/^Threads:/ {print $2}')
|
||||
if [ -n "$threads" ] && [ "$threads" -gt "$most" ]; then most=$threads; fi
|
||||
if [ -n "$peak" ]; then echo "$peak kB, at most $most threads" >"$memory"; fi
|
||||
sleep 10
|
||||
done
|
||||
) &
|
||||
|
||||
# wait returns early when Ctrl-C interrupts it: wait again until the
|
||||
# collector has flushed and exited.
|
||||
status=0
|
||||
while kill -0 "$run" 2>/dev/null; do
|
||||
wait "$run" && status=0 || status=$?
|
||||
done
|
||||
trap - INT
|
||||
wait "$view" || true
|
||||
# 124: stopped by timeout, at the end or on Ctrl-C (passed on by it).
|
||||
failed=0
|
||||
case $status in
|
||||
0 | 124 | 130) ;;
|
||||
*)
|
||||
echo "the collector failed (exit $status); its log: $log"
|
||||
failed=1
|
||||
;;
|
||||
esac
|
||||
|
||||
if [ ! -e "$out" ]; then
|
||||
echo "no output was written"
|
||||
exit 1
|
||||
fi
|
||||
echo
|
||||
echo "== totals at the last minute"
|
||||
awk -v final=1 -f scripts/watch.awk "$log"
|
||||
if [ -s "$memory" ]; then echo "peak memory $(cat "$memory")"; fi
|
||||
echo
|
||||
echo "== the output against the log (reconcile)"
|
||||
./target/release/reconcile "$out" "$log" || failed=1
|
||||
echo
|
||||
echo "== one symbol, as text (show)"
|
||||
# An output stopped before any book is empty: said, not a failure.
|
||||
./target/release/show "$out" || true
|
||||
exit "$failed"
|
||||
Reference in new issue
Block a user