This repository has been archived on 2026-11-16. You can view files and clone it. You cannot open issues or pull requests or push a commit.
Files
orderbook-collector/docs/evaluation.md
T
pascal adb7fee8d8 feat(polymarket): hash-verified order books for a subset of symbols
Collects the order books of Polymarket's 20 busiest markets (40 symbols)
into the template's SBE format: a Delta per level change, plus Snapshots on
subscribe, every minute, and whenever a fresh book corrects ours. Every
change is checked against Polymarket's own book hash; a failed check
resyncs the symbol on the open socket, and the log counts losses apart from
the few ms our own resync hides. Runs until SIGINT, which closes the socket
and flushes the output; reconnects when the server drops it. A `reconcile`
binary checks an output file against itself.
2026-11-04 17:10:17 +01:00

58 lines
2.8 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Evaluation
How we judge the finished software. This is a **post-completion evaluation**, not
a metric we develop against. Correctness (soundness/completeness tests),
benchmarks, and profiling drive development; this is the "does the data have
financial value" story told afterward.
## The idea: a trading simulator on top of the collector
Write a *separate* program that consumes our collector's output as its source of
truth, makes trading decisions, runs for a while, and reports P&L — a tangible
demonstration of the "faster/more complete makes more money" claim.
## The trap, and how to design around it
A trading sim measures the **strategy**, not the collector. Absolute P&L is
mostly "was the bot any good," which is noise for our purposes.
**Fix: make it differential — same strategy, two feeds.** Run the identical bot
on (a) our full feed and (b) a deliberately degraded copy, and compare the P&L
*difference*. That delta is attributable to data quality, which is what the
challenge claims matters.
The two claims split apart:
- **Completeness is cleanly measurable on recorded data.** Treat our output as
ground truth, make a lossy copy (drop a % of messages, skip some
resyncs), run the same strategy on both, compare P&L. The gap = cost of
dropped data. No live trading needed.
- **Latency is NOT measurable from recorded data.** Collection latency is baked
into a recording; a backtest can't show "we'd have been a few ms faster." A
dollar figure on latency needs a live/forward A/B (two collectors, one slower,
same strategy in real time, or injected delay). Much bigger apparatus.
**The fill model is the hard part.** A naive "see best ask → buy → book profit"
bot overstates P&L massively: it ignores queue position, the level vanishing
before the order arrives, latency to the exchange, and market impact. A credible
fill model for Kalshi/Polymarket microstructure is arguably bigger than the
challenge itself. Scope it carefully or not at all.
## Cheaper metrics that target the claims directly
Prefer these; they can't be gamed by strategy luck:
- **Time-accuracy:** distribution of `received_at − exchange_timestamp`, and
separately our *internal* added latency (edge-stamp → output byte).
- **Completeness:** reconciliation error rate (snapshot↔delta divergence); count
of missed deletes / stale levels under induced loss.
- **Redundancy value (once ext1 exists):** messages the backup socket caught that
the primary missed — literal "data loss avoided."
## Recommendation
Keep the trading-sim as the narrative. For an actual P&L demo, build only the
**differential completeness** version (full vs lossy, recorded data, same
strategy) — the one variant that's both doable and meaningful. Treat the full
live-latency-P&L rig as out of scope. Revisit at the polish milestone.