Archived
Collects the order books of Polymarket's 20 busiest markets (40 symbols) into the template's SBE format: a Delta per level change, plus Snapshots on subscribe, every minute, and whenever a fresh book corrects ours. Every change is checked against Polymarket's own book hash; a failed check resyncs the symbol on the open socket, and the log counts losses apart from the few ms our own resync hides. Runs until SIGINT, which closes the socket and flushes the output; reconnects when the server drops it. A `reconcile` binary checks an output file against itself.
58 lines
2.8 KiB
Markdown
58 lines
2.8 KiB
Markdown
# Evaluation
|
||
|
||
How we judge the finished software. This is a **post-completion evaluation**, not
|
||
a metric we develop against. Correctness (soundness/completeness tests),
|
||
benchmarks, and profiling drive development; this is the "does the data have
|
||
financial value" story told afterward.
|
||
|
||
## The idea: a trading simulator on top of the collector
|
||
|
||
Write a *separate* program that consumes our collector's output as its source of
|
||
truth, makes trading decisions, runs for a while, and reports P&L — a tangible
|
||
demonstration of the "faster/more complete makes more money" claim.
|
||
|
||
## The trap, and how to design around it
|
||
|
||
A trading sim measures the **strategy**, not the collector. Absolute P&L is
|
||
mostly "was the bot any good," which is noise for our purposes.
|
||
|
||
**Fix: make it differential — same strategy, two feeds.** Run the identical bot
|
||
on (a) our full feed and (b) a deliberately degraded copy, and compare the P&L
|
||
*difference*. That delta is attributable to data quality, which is what the
|
||
challenge claims matters.
|
||
|
||
The two claims split apart:
|
||
|
||
- **Completeness is cleanly measurable on recorded data.** Treat our output as
|
||
ground truth, make a lossy copy (drop a % of messages, skip some
|
||
resyncs), run the same strategy on both, compare P&L. The gap = cost of
|
||
dropped data. No live trading needed.
|
||
- **Latency is NOT measurable from recorded data.** Collection latency is baked
|
||
into a recording; a backtest can't show "we'd have been a few ms faster." A
|
||
dollar figure on latency needs a live/forward A/B (two collectors, one slower,
|
||
same strategy in real time, or injected delay). Much bigger apparatus.
|
||
|
||
**The fill model is the hard part.** A naive "see best ask → buy → book profit"
|
||
bot overstates P&L massively: it ignores queue position, the level vanishing
|
||
before the order arrives, latency to the exchange, and market impact. A credible
|
||
fill model for Kalshi/Polymarket microstructure is arguably bigger than the
|
||
challenge itself. Scope it carefully or not at all.
|
||
|
||
## Cheaper metrics that target the claims directly
|
||
|
||
Prefer these; they can't be gamed by strategy luck:
|
||
|
||
- **Time-accuracy:** distribution of `received_at − exchange_timestamp`, and
|
||
separately our *internal* added latency (edge-stamp → output byte).
|
||
- **Completeness:** reconciliation error rate (snapshot↔delta divergence); count
|
||
of missed deletes / stale levels under induced loss.
|
||
- **Redundancy value (once ext1 exists):** messages the backup socket caught that
|
||
the primary missed — literal "data loss avoided."
|
||
|
||
## Recommendation
|
||
|
||
Keep the trading-sim as the narrative. For an actual P&L demo, build only the
|
||
**differential completeness** version (full vs lossy, recorded data, same
|
||
strategy) — the one variant that's both doable and meaningful. Treat the full
|
||
live-latency-P&L rig as out of scope. Revisit at the polish milestone.
|