This repository has been archived on 2026-11-16. You can view files and clone it. You cannot open issues or pull requests or push a commit.
Files
orderbook-collector/docs/evaluation.md
T
pascal adb7fee8d8 feat(polymarket): hash-verified order books for a subset of symbols
Collects the order books of Polymarket's 20 busiest markets (40 symbols)
into the template's SBE format: a Delta per level change, plus Snapshots on
subscribe, every minute, and whenever a fresh book corrects ours. Every
change is checked against Polymarket's own book hash; a failed check
resyncs the symbol on the open socket, and the log counts losses apart from
the few ms our own resync hides. Runs until SIGINT, which closes the socket
and flushes the output; reconnects when the server drops it. A `reconcile`
binary checks an output file against itself.
2026-11-04 17:10:17 +01:00

2.8 KiB
Raw Blame History

Evaluation

How we judge the finished software. This is a post-completion evaluation, not a metric we develop against. Correctness (soundness/completeness tests), benchmarks, and profiling drive development; this is the "does the data have financial value" story told afterward.

The idea: a trading simulator on top of the collector

Write a separate program that consumes our collector's output as its source of truth, makes trading decisions, runs for a while, and reports P&L — a tangible demonstration of the "faster/more complete makes more money" claim.

The trap, and how to design around it

A trading sim measures the strategy, not the collector. Absolute P&L is mostly "was the bot any good," which is noise for our purposes.

Fix: make it differential — same strategy, two feeds. Run the identical bot on (a) our full feed and (b) a deliberately degraded copy, and compare the P&L difference. That delta is attributable to data quality, which is what the challenge claims matters.

The two claims split apart:

  • Completeness is cleanly measurable on recorded data. Treat our output as ground truth, make a lossy copy (drop a % of messages, skip some resyncs), run the same strategy on both, compare P&L. The gap = cost of dropped data. No live trading needed.
  • Latency is NOT measurable from recorded data. Collection latency is baked into a recording; a backtest can't show "we'd have been a few ms faster." A dollar figure on latency needs a live/forward A/B (two collectors, one slower, same strategy in real time, or injected delay). Much bigger apparatus.

The fill model is the hard part. A naive "see best ask → buy → book profit" bot overstates P&L massively: it ignores queue position, the level vanishing before the order arrives, latency to the exchange, and market impact. A credible fill model for Kalshi/Polymarket microstructure is arguably bigger than the challenge itself. Scope it carefully or not at all.

Cheaper metrics that target the claims directly

Prefer these; they can't be gamed by strategy luck:

  • Time-accuracy: distribution of received_at − exchange_timestamp, and separately our internal added latency (edge-stamp → output byte).
  • Completeness: reconciliation error rate (snapshot↔delta divergence); count of missed deletes / stale levels under induced loss.
  • Redundancy value (once ext1 exists): messages the backup socket caught that the primary missed — literal "data loss avoided."

Recommendation

Keep the trading-sim as the narrative. For an actual P&L demo, build only the differential completeness version (full vs lossy, recorded data, same strategy) — the one variant that's both doable and meaningful. Treat the full live-latency-P&L rig as out of scope. Revisit at the polish milestone.