Challenges facing quantitative research teams
Quantitative teams are rarely short of ideas. The constraint is often the data, the scale and the repetitive engineering around them.
Most effort goes to data preparation
Teams spend much of their time preparing data, rebuilding pipelines and recomputing features rather than investigating market behaviour.
Market data is fragmented
Tick data, order books and cross-venue feeds arrive in incompatible schemas, so each source needs its own transformation logic.
Microstructure data is large
Fine-grained data means large volumes across instruments and time, creating trade-offs between data fidelity and runtime.
Research and production diverge
Features built in research notebooks are often rewritten for streaming production. The two implementations can drift, and results diverge.
Architecture
Pipeline orchestration
Declarative pipeline definition with automated dependency resolution, scheduling and monitoring via Dagster.
Platform Components
Four components, one pipeline — each stage communicates through the QuantFlow Market Data Model, with no tight coupling.
DataInfra
Metadata-driven data infrastructure — ingest, normalize, and validate market data into a unified QuantFlow Market Data Model
- Multi-source ingestion & normalization
- Automatic dbt pipeline generation
- Schema-validated configuration & dbt data tests
- Engine-agnostic — DuckDB, BigQuery, Snowflake, Databricks, Iceberg/Trino, DolphinDB
MarketState
Market structure reconstruction — bars, order books, and supervised labeling from raw market data
- 9 configurable bar types (fixed + event-driven)
- Order book snapshot reconstruction
- Triple barrier & CUSUM-based labels
- Single-pass Numba fused kernel
FeatureDAG
Compiler-based feature engine — define features in YAML, generate engine-agnostic DAG, compile to batch and streaming execution
- Formula DSL — 40+ math functions compiled to an optimized DAG
- Compiler pipeline — AST → IR → Lowering → Execution
- 133 FeatureTypes across 6 dimensions
- Schema inference across 76 operators at DAG construction
Execution Layer
Dual-backend execution — Polars for batch research, DolphinDB for live streaming. Same feature definitions, two runtimes.
- Single definition, dual runtime — no duplicate implementations
- Streaming engines deployed server-side in DolphinDB
- Streaming modes: tick / bar / tick_to_bar
Where QuantFlow is applied
These are platform capabilities and research applications. They are not claims of proven investment performance.
Low-Code Quantitative Feature Engineering
Define and reuse quantitative analytical features through declarative YAML specifications, without manually implementing each calculation pipeline. QuantFlow's FeatureDAG compiler translates feature definitions into executable analytical workflows for supported batch and streaming environments.
- Declarative feature configuration
- Reusable feature definitions
- Dependency management
- Automated compilation
- Batch and streaming analytical execution
Market Microstructure Research
Analyse order-book behaviour, trade activity, liquidity conditions and market state transitions.
Quantitative Factor Research
Develop and evaluate quantitative indicators based on price, volume, volatility, liquidity and order-flow characteristics.
Trading Signal Investigation
Explore whether market-derived analytical features contain useful information for potential trading strategies.
Historical and Streaming Analysis
Designed to apply the same feature definitions across historical analytical workflows and streaming calculations.
Systematic Trading Research Infrastructure
Provide reusable analytical inputs and software components for developing and evaluating systematic trading research workflows.
Architecture, reference and worked examples
Compiler design, engine configuration, the Feature Library and tutorials are documented in detail.