Training Data for AI & Machine-Learning Trading Models — Crypto Order Books, Trades and Event Contracts

Across the whole archive our capture records about 634.1M trade prints per UTC day (19.0B in the 30 days to 2026-09-16, $5.7T of USD turnover across 31 venues) — every one of them a labelled event with its side, size, price and the order-book state around it. The rows below are what a single instrument contributes.

For crypto instruments the label is yours to define — a forward mid-price move over a horizon with a dead-band, the sign of the next N-second return, whether a level gets consumed — and the L2 grid gives you the inputs at any sampling rate. Because every event carries both the venue timestamp and the co-located receive timestamp, features can be built strictly from information available before the label's start, which is where most leakage in published crypto models comes from.

For event contracts the label comes with the data: every Kalshi and Polymarket up/down window settles, and the last print of a contract implies the outcome (in our statistics a close at or above 0.97 reads as Yes/Up, at or below 0.03 as No/Down — always described as "implied", the venue's official resolution is a separate record you should join). A 5-minute series produces 288 labeled windows per UTC day, 105,120 per year; the training-data guide works through that budget, walk-forward splits and the purge/embargo needed around each window.

Split by time, never at random: adjacent windows share regime and order flow, and a random split leaks that structure across train and test. Trim the ~8-minute overlap at the start of every day file before concatenating days, audit each file (stats FILE --deep: parse errors, chain gaps, crossed states, coverage close to 86,400 s) and drop grid rows flagged suspect before drawing conclusions from microstructure features.

The free reader from the AI toolkit streams a multi-gigabyte day file and writes a fixed-grid order-book table or the trade tape as Parquet; polars or pandas takes it from there. Days are independent, so the conversion parallelizes trivially.

# one Parquet order-book grid per day: 20 levels, sampled every second
for f in data/*.txt.zst; do
  out="parquet/book_$(basename "$f" .txt.zst).parquet"
  [ -f "$out" ] || python3 cryptostruct_reader.py book "$f" --every 1s --depth 20 --out "$out"
done

# trades of the same days, for flow features and labels
for f in data/*.txt.zst; do
  out="parquet/trades_$(basename "$f" .txt.zst).parquet"
  [ -f "$out" ] || python3 cryptostruct_reader.py trades "$f" --out "$out"
done

import polars as pl

book = pl.scan_parquet("parquet/book_*.parquet")
# columns: ts, mid, spread, spread_bps, bid_px_1..20, bid_qty_1..20,
#          ask_px_1..20, ask_qty_1..20, n_events, suspect
feat = (
    book.filter(pl.col("suspect") == 0)
        .with_columns(
            imb1=(pl.col("bid_qty_1") - pl.col("ask_qty_1"))
                / (pl.col("bid_qty_1") + pl.col("ask_qty_1")),
            micro=(pl.col("bid_px_1") * pl.col("ask_qty_1") + pl.col("ask_px_1") * pl.col("bid_qty_1"))
                / (pl.col("bid_qty_1") + pl.col("ask_qty_1")),
        )
        .with_columns(y=(pl.col("mid").shift(-60) / pl.col("mid") - 1))  # 60 s forward return
)
df = feat.collect()

The full walkthrough — grid choice, features, labels, splits and audit — is the order-book dataset guide.

1. Start with the free samples: a pinned high-volatility Bitcoin day (Binance BTCUSDT perpetual, 2026-07-06, ~1.5 GB), rolling full days on major venues and the busiest Kalshi and Polymarket contracts of the day are on the downloads page — build the pipeline against them before buying anything.
2. Size the set from the label budget, not from gigabytes: pick the instruments and the number of days that give you enough independent labeled windows, then buy exactly those instrument-days in the Data Shop at €1 each, or whole series-days of an event-contract family on the bundles page.
3. Buy in bulk when the set is large: "Select to buy" in the instrument scanner adds a shared date window across many instruments, credit packs bring the effective per-day price down (€100 → 140 credits · €250 → 375 credits · €1,000 → 1,750 credits) and never expire, and Premium adds 50 credits a month for €20.
4. Download everything you own as a batch ZIP from your account, bundle days as one archive each, or date ranges as one resumable tar; convert with the free reader (grids, trades, funding, greeks to Parquet) or use ?format=trades.parquet directly on the download link.
5. Let an agent do it: the Agent Skill teaches Claude Code or Cursor the format and recipes, and the MCP server searches instruments and bundles, previews and — with your consent — buys and downloads on your behalf.

Purchased data is for your own use — backtesting, research, analytics and training your own models, in-house or in your own products; the trained models and derived data you build from it are yours. The raw files and tick streams themselves may not be redistributed, resold or published. The full terms are on the data license page.

The archive itself — venues, message types, prices and bulk delivery — is described on the historical data overview.