Building a Crypto Order Book Dataset for Machine Learning

Most "crypto machine learning" datasets on the internet are OHLCV bars. That is fine for a price-direction toy model and useless for anything that trades against a book: execution models, short-horizon signals, market-making policies, reinforcement-learning environments. Those need historical order-book data — the Level-2 depth path, event by event — and they need it in a shape a training loop can consume. This guide walks through building that dataset from the CryptoStruct tick archive: what a day file contains, how to turn it into fixed-grid Parquet, which features and labels are worth computing, how to split without leaking, and what it costs. Every command below runs on the free sample days before you buy anything.

Why L2 ticks, not candles

A candle is a lossy summary of trades. It throws away the order flow that produced those trades and everything that did not trade at all: resting size at each level, how the two sides were imbalanced, how deep the book was when a large order arrived, and the cancellations that pulled liquidity a second before the move. For a model whose job is to predict the next few seconds or minutes — or to decide how to execute a parent order — those are the features that carry the signal. You cannot reconstruct them from bars, and you cannot reconstruct them from a venue API after the fact either: venue APIs can backfill some historical trade and price data, but they generally do not let you replay the full tick-by-tick order-book path (check the current docs). That path has to be recorded live, which is what the archive does at every venue.

The archive records this co-located at the venue: every order-book snapshot, every incremental update and every trade, with both the venue's timestamp and the recorder's receive time in integer nanoseconds UTC, in one schema across spot, perpetuals, futures, options and the Kalshi/Polymarket event contracts. A parser written against a Binance file reads a Bybit or BitMEX file unchanged — which is what makes a multi-venue dataset a loop rather than a project.

What a day file holds

One file is one instrument for one UTC day: zstd-compressed JSON lines. Line 1 is the instrument's masterdata (tick size, lot size, contract multiplier, inverse flag); every other line is one event in a fixed envelope — [msgType, instrumentId, prevEventId, eventId, adapterTs, exchangeTs, data, …]. The message types you will meet:

Scale is the first thing to plan around. A BitMEX XRPUSDT perpetual day is about 59 MB compressed and 1.87 million events, 1.83 million of them book updates; a Binance BTCUSDT perpetual day is 1–2 GB and tens of millions of events. Compression is roughly 4–8×, so a flagship day decompresses to many gigabytes of text. Never read a file into memory — stream it through a zstd decoder line by line, and filter on the message type before parsing JSON where you can. The free reader does both, and it handles the one trap that catches hand-written decoders: day files are multi-frame zstd, and some stream decoders stop silently after the first frame. The full field-by-field layout is in the format guide.

Dataset shape: fixed-grid depth snapshots

Raw events are irregular in time; a model wants rows. The standard transformation is a fixed time grid: at every boundary, take the last book state at or before it and write the top N levels per side. The reader's book command emits exactly this schema — ts, mid, spread, spread_bps, bid_px_1..20, bid_qty_1..20, ask_px_1..20, ask_qty_1..20, n_events, suspect — with epoch-aligned boundaries, so rows from different instruments and different days line up on the same clock without a resample step.

• Grid interval: 1 s is the usual default (86,400 rows per day); 100 ms for execution and market-making work (864,000 rows per day); 1 min if you are joining with the minute statistics API.
• Depth: 20 levels per side matches the archive's top-20 depth statistics; 5 or 10 levels are fine for signal models and shrink the row by a factor of two to four.
• n_events is the number of book events since the previous sample — 0 means a quiet second with an unchanged book, which is itself a feature.
• suspect flags samples after an event-chain gap and before the next snapshot; drop or mask them rather than train on a book you cannot trust.
• Event-driven alternative: sample on every k-th update or on every trade instead of on the clock. Better for volatility-adaptive models, harder to align across instruments — build the clock grid first.

Build the pipeline: one Parquet file per day

Convert each day file once, then treat the directory of Parquet files as the dataset. Days are independent, so the conversion loop parallelises trivially:

ls data/*.txt.zst | xargs -P 4 -I{} sh -c '
  out="parquet/book_$(basename {} .txt.zst).parquet"
  [ -f "$out" ] || python3 cryptostruct_reader.py book {} \
      --every 1s --depth 20 --out "$out"
'

# trades as a second table (labels + flow features)
ls data/*.txt.zst | xargs -P 4 -I{} sh -c '
  out="parquet/trades_$(basename {} .txt.zst).parquet"
  [ -f "$out" ] || python3 cryptostruct_reader.py trades {} --out "$out"
'

import polars as pl

book = pl.scan_parquet("parquet/book_*.parquet")
trades = pl.scan_parquet("parquet/trades_*.parquet").unique(
    subset=["trade_id", "exchange_ts"]
)

# keep each row inside its own UTC day (drops the pre-midnight margin)
book = book.with_columns(day=pl.col("ts").dt.date()).unique(subset=["ts"])

# signed aggressor flow per second (side 0 = buy), then CVD
flow = (
    trades.sort("ts")
    .group_by_dynamic("ts", every="1s")
    .agg(
        net_flow=(pl.when(pl.col("side") == 0)
                    .then(pl.col("quote_value"))
                    .otherwise(-pl.col("quote_value"))).sum(),
        n_trades=pl.len(),
    )
    .with_columns(cvd=pl.col("net_flow").cum_sum())
)
ds = book.join(flow, on="ts", how="left").fill_null(0)

Features and labels

With the grid in place, the classic limit-order-book features are one expression each. Keep the first version boring — the value is in the data, not in the feature count:

• Queue imbalance at level 1: (bid_qty_1 − ask_qty_1) / (bid_qty_1 + ask_qty_1), and the same over the top 5 or 10 levels.
• Microprice: (bid_px_1 × ask_qty_1 + ask_px_1 × bid_qty_1) / (bid_qty_1 + ask_qty_1) minus mid — the queue-weighted fair value.
• Depth within N bps of mid, per side, as quote value — the liquidity a market order of a given size would actually meet (recipe 4 in the reader's references).
• Trade-flow imbalance and cumulative volume delta from the trades table, using the aggressor side (0 = buy) — identical to the minute API's buy/sell split, so it cross-checks.
• Book activity: n_events per sample, and its rolling mean — a cheap regime feature.
• Slow context from the same file: funding rate (type 9) and mark–index basis (types 7/8) on perpetuals.

Labels for crypto instruments come from the future of the same grid: the mid move over a horizon of h samples, thresholded into up / flat / down, or the realised spread cost of a simulated fill for execution models. Choose h in the same units as the grid and store it with the dataset — the split logic below depends on it.

Prediction markets add a label crypto never has: a settlement. Every Kalshi and Polymarket contract in the archive is a book that resolves to yes or no, and the series-day bundles give you all of them for a day. In the recorded data the outcome is *implied* by the last prints — a close at or above 0.97 reads as yes/up, at or below 0.03 as no/down — and the venue's official resolution is a separate fact you should take from its settlement metadata for production labels. How many labelled windows a series yields per day, and why that budget is fixed, is worked through in How much data does an up/down bot need?.

Splits, leakage and auditing the input

Book samples one second apart are almost the same observation. A random train/test split puts near-duplicates on both sides and reports an out-of-sample accuracy that evaporates in production. Split by time — walk-forward, oldest days for training, newest for testing — and purge a gap of at least the feature look-back plus the label horizon around every boundary, so no training row can see a test label's future. Treat each UTC day as the atomic unit of the split; it is also the unit you buy.

Before a day enters the dataset, audit the file. The reader's stats --deep pass reports what matters:

python3 cryptostruct_reader.py stats data/2449_2026-09-01.txt.zst --deep
# 1870832 events, 0 parse errors
# coverage 2026-08-31T23:51:55 → 2026-09-02T00:00:59  (86944 s)
# fills 3423   snapshots 3   chain gaps: book 0, trades 0
# crossed states 0/1836542 (0.0%)   max levels 1247

• parse errors > 0 — damaged in transit; re-download before doing anything else.
• book chain gaps > 0 — the book is unreliable between the gap and the next snapshot; the grid marks those rows suspect.
• crossed states above ~0.1 % — treat spread and depth features from that day with suspicion.
• Coverage well below ~86,400 s plus margins — a recording gap that day; check the instrument's calendar in the shop and consider excluding the day.
• More than one snapshot is normal (recorder failover); every snapshot resets the book cleanly.

Sizing and cost

Row counts are deterministic: a 1-second grid yields 86,400 rows per instrument-day, a 100-millisecond grid 864,000. At 20 levels per side that is roughly 84 columns of floats per row — a 1 s day of one instrument is a few tens of megabytes of Parquet, a year of one instrument fits on a laptop, and a year of ten instruments is a small object-store bucket. The raw input costs €1 per instrument-day in the data shop; credit packs bring that down (€100 buys 140 credits, €250 buys 375, €1,000 buys 1,750 — credits never expire), and Premium at €20 per month includes 50 credits plus the minute-statistics exports. A year of one flagship perpetual is therefore about 365 credits; a multi-venue BTC dataset over the top five venues for a quarter, about 450.

Delivery scales the same way: single files per instrument-day, one ZIP per series-day for prediction bundles, and a resumable tar stream for whole date ranges of a bundle (up to 100 days per request). Prototype first on the free full-day samples on the downloads page — they include a pinned high-volatility Binance BTCUSDT perpetual day from 2026-07-06 (~1.56 GB), the busiest Kalshi and Polymarket contracts of a recent day, and smaller spot files that convert in seconds. They are byte-identical to shop files, so the pipeline you build on them runs unchanged on purchased days.

Licensing note

Purchased data is for your own use: research, backtesting and training your own models — including commercial ones — are all fine, and the models you train are yours. What you may not do is redistribute, resell or publish the raw data itself. The training-data page states the terms in one place, together with the dataset sizes and the bulk-purchase options.

Limitations

Quantities in the files are venue-native contract units, not base-asset amounts: read the header's multiplier, is_inverse and contract_value before summing depth into notionals (the shipped format reference has worked examples). Not every venue publishes a separate top-of-book stream — BitMEX, Coinbase, Kraken spot, Kalshi and Polymarket record depth only, so top-of-book there comes from the L2 replay, which the reader does for you. The receive timestamp is the recorder's clock at the venue's co-location, not the matching engine's; the venue timestamp, where present, is the venue's own. The scale figures above are from real production files of specific days and vary with market activity. This is a data-engineering guide, not investment advice.