Archived
Every live market is subscribed on two connections chosen at random, and its book takes each change from whichever connection delivers it first: the book hash says whether a change is a duplicate, the next one, or one to hold until the change before it arrives. A change one connection misses costs nothing, each Delta carries the earliest receive time, and a connection that drops is covered by the other. When a change fits on no connection in time, the book follows one connection unchecked and asks the other for a fresh book. Connections now decode and route events to 16 book threads, independent of them; a move leaves its old connection only once the new one delivers, so it costs no change; a connection closed with 1013 twice within ten minutes is split. reconcile reads the collector's log and fails only on a mismatch no cause line explains. SIGTERM and SIGHUP stop the collector cleanly. Breaking: lanes are gone (connections and book threads are separate), and reconcile takes the log as a second argument.
58 lines
2.8 KiB
Markdown
58 lines
2.8 KiB
Markdown
# Evaluation
|
||
|
||
How we judge the finished software. This is a **post-completion evaluation**, not
|
||
a metric we develop against. Correctness (soundness/completeness tests),
|
||
benchmarks, and profiling drive development; this is the "does the data have
|
||
financial value" story told afterward.
|
||
|
||
## The idea: a trading simulator on top of the collector
|
||
|
||
Write a *separate* program that consumes our collector's output as its source of
|
||
truth, makes trading decisions, runs for a while, and reports P&L — a tangible
|
||
demonstration of the "faster/more complete makes more money" claim.
|
||
|
||
## The trap, and how to design around it
|
||
|
||
A trading sim measures the **strategy**, not the collector. Absolute P&L is
|
||
mostly "was the bot any good," which is noise for our purposes.
|
||
|
||
**Fix: make it differential — same strategy, two feeds.** Run the identical bot
|
||
on (a) our full feed and (b) a deliberately degraded copy, and compare the P&L
|
||
*difference*. That delta is attributable to data quality, which is what the
|
||
challenge claims matters.
|
||
|
||
The two claims split apart:
|
||
|
||
- **Completeness is cleanly measurable on recorded data.** Treat our output as
|
||
ground truth, make a lossy copy (drop a % of messages, skip some
|
||
resyncs), run the same strategy on both, compare P&L. The gap = cost of
|
||
dropped data. No live trading needed.
|
||
- **Latency is NOT measurable from recorded data.** Collection latency is baked
|
||
into a recording; a backtest can't show "we'd have been a few ms faster." A
|
||
dollar figure on latency needs a live/forward A/B (two collectors, one slower,
|
||
same strategy in real time, or injected delay). Much bigger apparatus.
|
||
|
||
**The fill model is the hard part.** A naive "see best ask → buy → book profit"
|
||
bot overstates P&L massively: it ignores queue position, the level vanishing
|
||
before the order arrives, latency to the exchange, and market impact. A credible
|
||
fill model for Kalshi/Polymarket microstructure is arguably bigger than the
|
||
challenge itself. Scope it carefully or not at all.
|
||
|
||
## Cheaper metrics that target the claims directly
|
||
|
||
Prefer these; they can't be gamed by strategy luck:
|
||
|
||
- **Time-accuracy:** distribution of `received_at − exchange_timestamp`, and
|
||
separately our *internal* added latency (edge-stamp → output byte).
|
||
- **Completeness:** reconciliation error rate (snapshot↔delta divergence); count
|
||
of missed deletes / stale levels under induced loss.
|
||
- **Redundancy value:** changes one connection missed and the other delivered
|
||
(rescued) — literal "data loss avoided."
|
||
|
||
## Recommendation
|
||
|
||
Keep the trading-sim as the narrative. For an actual P&L demo, build only the
|
||
**differential completeness** version (full vs lossy, recorded data, same
|
||
strategy) — the one variant that's both doable and meaningful. Treat the full
|
||
live-latency-P&L rig as out of scope. Revisit at the polish milestone.
|