Archived
feat: ready to submit, a run to watch, the README, the review's fixes
Sprint 08 (roadmap/08. ready to submit.md), the last milestone. - scripts/watch.sh runs the collector with a live view of its log, stops cleanly on Ctrl-C, then checks the output (reconcile) and shows one symbol's book as text (show, a new binary). - The README for the solution, with the final runs: Polymarket through an outage it recovered from on its own, Kalshi's demo with two slow connections recycled. - From a review of the whole codebase: Kalshi takes RSA keys too (RSA-PSS-SHA256); a slow Kalshi connection takes the hourly recycle's next turn, which it nearly never got before; a market tracked again no longer fails reconcile; Book::add checks for overflow; usage and --help name the key variables; the first scan says how far it got. - Tests for every gap the reviews found, an offline run of the whole orchestrator among them; ShellCheck in CI.
This commit is contained in:
35 files changed
+2791
-207
No files matched your search
@@ -0,0 +1,107 @@
|
||||
# The view of scripts/watch.sh: reads the collector's log and prints, each
|
||||
# minute, the orchestrator's line and the book threads' lines summed into one,
|
||||
# and between them only the notable events. The Snapshot cause lines and the
|
||||
# connections coming up are counted, not shown.
|
||||
#
|
||||
# With -v final=1 it prints nothing live, only the last minute's sums: the
|
||||
# run's totals, since the book threads count "so far". A minute's sum is
|
||||
# printed once all its book threads' lines are in (-v book_threads=N, 16 by
|
||||
# default, src/core/mod.rs's BOOK_THREADS), or with the first other line 10 s
|
||||
# or more after the minute's first, or when the next minute begins. A book
|
||||
# thread that received nothing logs no line; the others' come within a few
|
||||
# seconds of the minute (1.6 s apart at most in a 10-minute Polymarket run,
|
||||
# 2026-11-12), and other lines often fall between them.
|
||||
|
||||
# The book threads' counters shown, in this order; a name missing from an
|
||||
# exchange's line (Kalshi has no "rescued") is left out.
|
||||
BEGIN {
|
||||
if (book_threads == "") book_threads = 16
|
||||
shown = "symbols|changes written|duplicates|rescued|resyncs|subscription gaps|losses"
|
||||
count = split(shown, names, "|")
|
||||
}
|
||||
|
||||
function flush( line, i) {
|
||||
if (threads == 0) return
|
||||
line = minute " books (" threads " threads):"
|
||||
for (i = 1; i <= count; i++)
|
||||
if (names[i] in sum) line = line " " sum[names[i]] " " names[i] ","
|
||||
sub(/,$/, "", line)
|
||||
last = line
|
||||
event(line)
|
||||
threads = 0
|
||||
split("", sum)
|
||||
}
|
||||
|
||||
# Printed at once: the view is read while the run goes on, and mawk
|
||||
# buffers its output otherwise.
|
||||
function event(text) {
|
||||
if (final) return
|
||||
print text
|
||||
fflush()
|
||||
}
|
||||
|
||||
{
|
||||
time = substr($1, 1, 8)
|
||||
text = substr($0, index($0, " ") + 1)
|
||||
# To the millisecond: $1 is HH:MM:SS.mmm.
|
||||
seconds = substr($1, 1, 2) * 3600 + substr($1, 4, 2) * 60 + substr($1, 7)
|
||||
}
|
||||
|
||||
/: book thread [0-9]+: / {
|
||||
this = substr(time, 1, 5)
|
||||
if (threads > 0 && this != minute) flush()
|
||||
if (threads == 0) first_at = seconds
|
||||
minute = this
|
||||
threads++
|
||||
body = text
|
||||
sub(/; this minute .*/, "", body)
|
||||
split("", seen)
|
||||
# Each "N name": a count and its name, up to the next comma, semicolon
|
||||
# or parenthesis. A name is summed once a line (its first occurrence).
|
||||
while (match(body, /[0-9]+ [a-z][a-z' ]*[a-z]/)) {
|
||||
pair = substr(body, RSTART, RLENGTH)
|
||||
body = substr(body, RSTART + RLENGTH)
|
||||
n = substr(pair, 1, index(pair, " ") - 1)
|
||||
name = substr(pair, index(pair, " ") + 1)
|
||||
sub(/^so far /, "", name)
|
||||
if (name in seen) continue
|
||||
seen[name] = 1
|
||||
sum[name] += n
|
||||
}
|
||||
if (threads == book_threads) flush()
|
||||
next
|
||||
}
|
||||
|
||||
# A minute with a book thread's line missing: printed once it is clearly over
|
||||
# (a day's seconds added back across midnight).
|
||||
threads > 0 && (seconds - first_at + 86400) % 86400 >= 10 { flush() }
|
||||
|
||||
/: orchestrator: [0-9]+ markets on/ {
|
||||
line = text
|
||||
sub(/^[a-z-]+: orchestrator: /, "", line)
|
||||
event(time " " line "; " up + 0 " connected (" refused + 0 " refused, retried), " snapshots + 0 " Snapshots with a cause")
|
||||
up = 0
|
||||
refused = 0
|
||||
snapshots = 0
|
||||
next
|
||||
}
|
||||
|
||||
/: connection [0-9]+: connected, / {
|
||||
if (!opened) event(time " connections opening (counted in the minute line)")
|
||||
opened = 1
|
||||
up++
|
||||
next
|
||||
}
|
||||
# Kalshi refuses some of ~100 handshakes opened at once (429); each retries.
|
||||
/: connect failed: handshake refused: 429/ { refused++; next }
|
||||
/: Snapshot of / { snapshots++; next }
|
||||
/ changes go unchecked$| tick size changed, resync$| unreadable / { next }
|
||||
|
||||
# Everything else is an event worth seeing: a connection lost, recycled,
|
||||
# split, moved off when quiet, a refused handshake, a failed fetch, the end.
|
||||
{ event(time " " text) }
|
||||
|
||||
END {
|
||||
flush()
|
||||
if (final && last != "") print last
|
||||
}
|
||||
Reference in new issue
Block a user