An orchestrator keeps the tracked set equal to Gamma's live markets (a scan at startup and every hour, new and closed markets every minute, reaching back 15 minutes for Gamma's indexing delay) and spreads them over lanes, each a connection and a book thread, all writing through the one writer. Lanes report each market's load in bytes per second; the orchestrator puts new markets on the least-loaded lane and moves the busiest markets off lanes over a 300 KB/s budget, busiest lane first. Differing books our own subscribe explains are subscription gaps, not losses, and every first book after a move is logged, so reconcile's mismatches are all accounted for in the log. A 75-minute live run tracked ~195k markets (~390k symbols) in 89 lanes.
Read before getting started!!
First, read src/main.rs. Then, some background on this repo:
This repo uses the FIX SBE standard
for writing binary messages. The rust code abstracts this away for you, and you can simply use the helpers, as demonstrated in main.rs.
More background, in case you are curious!
FIX SBE is defined in XML, which I find extremely cumbersome to edit, especially when updating schemas frequently. To solve this, I have also introduced some helper code that takes our own custom YAML format, and turns it into FIX SBE XML, which in turn is used to generate rust code to encode and decode the messages. You don't need to edit any of these files. This explanation is simply to help you understand the origins of the various files in this repo.
If you're curious to see the YAML schema, take a look at challenge_orderbook.yaml.
We also use a custom timestamp field, so that the Rust struct can use
chrono::DateTime in the generated struct, and in SBE schema, it is a uint64
representing the timestamp as the number of microseconds since unix epoch.