QuantFlow currently supports three types of data engines:

• Local engine — DuckDB, mainly for local development, debugging, and lightweight research workflows.
• Cloud warehouse engine — commercial data platforms such as Databricks, BigQuery, and Snowflake.
• Open lakehouse engine — the QuantFlow embedded data engine built on top of S3-compatible object storage + Parquet + Iceberg + Trino.

Why an Open Lakehouse Engine at All?​

I have to admit that I have always believed that self-managed systems built on top of open-source products tend to cost more overall than commercial platforms, especially when considering engineering labour, operational issues, maintenance overhead, and opportunity cost. For most routine data processing and analytics workloads, commercial cloud data platforms are actually quite reasonable when managed properly.

However, I become much more hesitant when dealing with quant research over market data, especially with the current trend toward microstructure-level research using tick and order book data. It is not only the sheer scale of market data required today, but more importantly the highly iterative nature of quantitative research and experimentation, that can make usage-based pricing models much more expensive than expected.

Market data is naturally high-volume, time-sensitive, append-heavy, and repeatedly scanned during research. A single symbol can generate a surprisingly large amount of data when working with tick trades or order book updates. Once you move from one symbol to a cross-sectional strategy, the numbers grow very quickly. For example, one year of QQQ MBP-1 data can already be around 117 GB. That is just one symbol, one schema, and one year.

The cost problem is not one query. The cost problem is repeated experimentation, such as:

• try one feature set
• try another feature set
• change the sampling method
• change the label horizon
• change the universe
• change the lookback window

S3 + Parquet + Iceberg + Trino​

The open lakehouse architecture is simple in concept: store large market data files in cheap S3-compatible object storage, use Parquet as the physical file format, use Iceberg as the table format, and use Trino as the SQL query engine.

The important point is that the platform is no longer a single product. It becomes a set of replaceable layers.

Parquet matters because market data is naturally columnar. Query engines can read only the required columns instead of scanning entire files. Iceberg matters because Parquet files alone do not make a table — Iceberg adds snapshots, schema evolution, partition management, and atomic commits. Trino sits on top of Iceberg and executes distributed SQL queries across many Parquet files in parallel. For Python-native state and feature engineering, I still prefer Ray + Polars over SQL-based transformations.

Example Architecture Cost Breakdown​

Below is a simplified monthly infrastructure breakdown for the open lakehouse setup used in QuantFlow:

This is obviously not a complete production cost model. It does not include engineering labour, monitoring systems, backup infrastructure, or operational overhead. The point is simply to show that the raw infrastructure layer for large-scale market data research can be surprisingly affordable when storage and compute are separated properly.

Cost Comparison​

To make the cost discussion more concrete, let's work through one practical example: one year of QQQ MBP-1 data at around 117 GB. One scan of 117 GB does not sound expensive. The problem is that market data research rarely scans it once. A cross-sectional strategy may scan many symbols, and a research workflow may scan the same data repeatedly while changing features, labels, horizons, and sampling rules.

A simple way to think about it is this: 117 GB is about 0.114 TiB. If we scan that dataset 1,000 times during research, that is around 114 TiB of scanned data. If we scale from one symbol to a 10-symbol research universe with similar order-book data size, one full scan is already around 1.17 TB, and 100 research iterations becomes around 117 TB of scanned data. The cost problem is not the single QQQ query; it is repeated experimentation over a growing universe.

Below is an indicative monthly comparison for a QQQ-style workload. Assume QQQ one-year MBP-1 data is 117 GB, a 10-symbol universe has similar data size per symbol, and the research workflow scans that universe 100 times in a month.

117 GB × 10 symbols × 100 scans ≈ 117 TB scanned ≈ 114–117 TiB scanned per month

The exact numbers will obviously vary depending on compression ratio, pruning efficiency, warehouse size, concurrency, cloud provider, and research behaviour. The important point is not the precise dollar amount, but how the cost scales with repeated scans and experimentation.

A More Detailed Breakdown​

Open lakehouse:

• R2 storage for ~1.17 TB active dataset: $40/month
• 1 Trino coordinator + 4 worker VMs (16 GB each): $250/month
• R2 egress: $0
• Estimated total: $290/month

BigQuery on-demand:

• Capability-matched repeated research scans and larger concurrent workloads
• Effective monthly scanned data: ~232 TiB
• 232 × $6.25 ≈ $1,450/month
• Storage for ~1.17 TB: relatively small compared with scan cost

Databricks Jobs:

• Underlying cloud VMs + DBU charges
• 1 driver + 4 workers 16 GB cluster, always-on equivalent: about $1,350/month

Databricks All-Purpose:

• Same cluster shape, higher interactive DBU rate
• About $3,200/month if kept running heavily

Snowflake:

• Medium warehouse with sustained research usage
• 6 credits/hour × 100 hours × ~$3/credit ≈ $1,800
• Plus storage, usually smaller than compute in this example

The main point is not that the open lakehouse is always cheaper for every workload. It is that for repeated market-data scans, its cost grows much more slowly. Once the VMs are running, scanning the same Parquet/Iceberg data repeatedly does not create a new per-TiB query bill in the same way as BigQuery on-demand, and it does not add a Databricks or Snowflake platform charge on top of every hour of managed compute.

For the open lakehouse version, the cost is more predictable. Using Cloudflare R2 as active storage and low-cost 16 GB VMs for Trino/Ray workers, the monthly cost can be roughly in the low hundreds of dollars rather than scaling directly with every TiB scanned. The storage cost is mostly object storage, and the compute cost is mostly the fixed VM bill. If the workload scans the same market data many times, this fixed-compute model can be attractive.

BigQuery is different. With on-demand pricing, the query cost is linked to the amount of data scanned. That model is very convenient and often perfectly reasonable for normal analytics, but market data research can generate many repeated scans. A single 117 GB QQQ scan is small; hundreds or thousands of scans across many symbols are not.

Databricks has a different shape again. It is not simply "per query". The cost comes from the underlying cloud infrastructure plus Databricks DBU usage. It gives you Spark, notebooks, managed jobs, collaboration, and a very productive platform, but if the target workload is mainly Ray/Polars-style ingestion and repeated market-data processing, a small self-managed VM cluster can be much cheaper.

Snowflake is also not exactly "per query". It is mainly warehouse-credit based: you pay for the virtual warehouse size and how long it runs. This is excellent for managed SQL workloads and enterprise analytics, but repeated order-book scans and backtest-style research can keep warehouses running and consuming credits.

Why DolphinDB for Streaming?​

We chose DolphinDB as the streaming engine because of one reason: speed with complicated computation that requires chained steps. Most streaming engines are fast at simple aggregations but fall apart when you need rolling windows, lags, cross-sectional operations, and conditional logic chained across multiple steps. DolphinDB's ReactiveStateEngine handles this natively.

But a streaming engine alone only solves half the problem. You also need:

• Market state reconstruction (bars, order books) from raw exchange data
• A way to define features declaratively and compile them to streaming operators
• A visualization layer that queries live data without adding latency

QuantFlow bridges all three. By combining DolphinDB with QuantFlow's MarketState engine and FeatureDAG compiler, and leveraging Grafana's visualization capability, you can set up a real-time market monitor dashboard with almost no effort.

The Streaming Architecture​

FeatureDAG parses your YAML definitions and generates a DAG representing the full feature computation graph — rolling windows, lags, arithmetic expressions, conditional logic, order book array extractions. The same DAG compiles to Polars expressions for batch research and DolphinDB reactive engine scripts for live trading.

Each computation step becomes a metric expression inside a ReactiveStateEngine. The compiler consolidates multiple features sharing the same input into a single engine, inlines intermediate expressions, and merges compatible features together. Instead of one engine per feature, a handful of consolidated engines produce multiple output columns in one pass.

Engines communicate through shared stream tables — an upstream engine writes rows; a downstream engine subscribes and reacts. Everything stays in-memory within the same DolphinDB process. No serialization between steps. No disk. No context switches. Python steps aside.

Trades + LOB → MarketState → Stream Tables → Feature Engines → Grafana
   (raw)        (bars)       (in-memory)    (consolidated)   (WebSocket)

Setting Up the Dashboard​

Step 1 — Pull the Grafana Image​

docker pull dolphindb/dolphindb-grafana:9.1.0
docker run -d --name ddb_gra -p 5000:3000 dolphindb/dolphindb-grafana:9.1.0

This bundles Grafana 9.1.0 with the DolphinDB plugin pre-installed. No separate plugin installation needed.

Step 2 — Add the Data Source​

Log in at http://localhost:5000 (default credentials: admin/admin). Go to Configuration → Data Sources → Add, search for "dolphindb." The connection URL uses WebSocket format:

ws://host.docker.internal:8848

Use host.docker.internal if DolphinDB runs on the host machine. If DolphinDB is also containerized, use the container name instead.

Step 3 — Build the Dashboard​

Create panels on a Grafana dashboard, write DolphinDB queries to read from the stream tables generated by QuantFlow. Each panel queries a live stream table — the data updates in real-time as the streaming pipeline processes new ticks.

Step 4 — Start the QuantFlow Streaming Pipeline​

Once the pipeline is running, the dashboard comes alive. Grafana talks WebSocket directly to the DolphinDB server — every dashboard query executes inside the same DolphinDB process that's computing the features. No REST API, no middleware, no serialization overhead between computation and visualization.

What Makes This Fast​

The latency comes from eliminating every non-essential hop:

The result: sub-millisecond feature computation with live visualization that refreshes as fast as your data arrives.

Beyond the Dashboard​

This same architecture powers QuantFlow's entire streaming capability. The YAML definitions you write for research (batch/Polars) compile to the exact same streaming operators in DolphinDB — define once, run anywhere. The dashboard is just the most visible surface of a pipeline that can feed live trading signals, risk monitors, and alerting systems simultaneously.

Ready to try it? Check out the Quickstart Guide or explore the Feature Library to see what's available out of the box.

The common assumption is that AI will reduce the need for systems like QuantFlow because:

• models can learn features automatically
• raw data can be fed directly into neural networks
• end-to-end learning replaces feature engineering

But this misses a key point:

AI changes how we model markets — it does not remove the need to define what the market is in the first place.

⚙️ What AI actually changes (and what it doesn't)​

AI is extremely good at:

• learning patterns from complex data
• extracting latent structure from sequences
• reducing manual feature engineering
• generalising across regimes (to some extent)

But it does not eliminate core structural problems:

1. Markets are still not clean inputs​

Market data remains:

• event-driven (trades, quotes, order books)
• irregular in time
• fragmented across venues
• inconsistent in representation

AI does not fix this — it learns on top of it.

2. Representation still matters more than model power​

Even the best AI model only sees:

• the representation of the market you give it

If two systems define liquidity, order flow, or imbalance differently, then:

• the model learns different worlds
• research ≠ live behaviour
• performance becomes unstable

So the real bottleneck becomes: consistency of market representation, not model sophistication

3. Research and production still diverge​

Even in AI-native systems:

• training is batch-based
• production is streaming-based
• latency constraints still exist
• execution feedback loops are unavoidable

This gap is structural — not model-dependent.

🏗️ Where QuantFlow fits in an AI world​

QuantFlow is not competing with AI.

It sits underneath it.

Its role is to define a consistent bridge between:

raw market data → AI-ready representation → live execution

But importantly, it does this in a specific way:

users define the features they want, and QuantFlow automatically generates them from raw market data using a built-in library of microstructure primitives

So it is not a feature store.

It is not a pipeline tool.

It is:

a declarative system that converts raw market data into consistent, production-grade feature representations

🚀 Why this becomes more important in the AI era​

As AI models become more powerful:

1. They become more sensitive to input consistency​

Small representation differences create large performance divergence.

2. They become easier to overfit on inconsistent pipelines​

Especially in high-frequency / microstructure settings.

3. They increase iteration speed — but amplify infrastructure weaknesses​

More experiments expose more pipeline inconsistency.

So the bottleneck shifts:

from model quality → to data representation and feature consistency

🧠 What QuantFlow actually provides in an AI system​

QuantFlow ensures:

✔ Consistent market representation​

The same definitions of:

• order flow
• liquidity
• spread
• microstructure features

across research and live systems.

✔ Production-aligned feature generation​

Features are not manually re-implemented.

They are:

• generated consistently from a shared definition layer

✔ A stable foundation for AI models​

AI systems no longer learn from:

• slightly different pipelines
• inconsistent feature logic
• ad-hoc research code

They learn from:

• a unified, production-grade representation of the market

📌 Final answer​

Yes — QuantFlow is still useful in the AI era.

But more precisely:

AI reduces the need for manual feature engineering, but increases the need for consistent, production-aligned market representation systems.

QuantFlow becomes more important because:

it is the layer that makes AI systems actually reliable in real trading environments — not just powerful in research.

Explore QuantFlow: System Overview | Contact

— The QuantFlow Team

Series Overview​

This article concludes our series on the topic of market microstructure. If you're new to this series, I recommend starting with:

Part 1: Introduction to Market Microstructure - The discussion on how modern financial markets operate at the micro level.

After exploring order flow, liquidity, impact, regimes, and cross-asset structure, one conclusion becomes increasingly clear:

microstructure trading is not primarily a modelling problem — it is a data representation problem.

Most strategies don't fail because the model is weak. They fail because the market is not represented correctly in the first place.

That is the problem QuantFlow is designed to address.

🧠 The real issue in systematic trading​

Most quant workflows still look like this:

raw data → ad-hoc cleaning → feature engineering → model → research → execution

The problem is not the model.

It's everything before it.

Three structural issues appear repeatedly:

1. Inconsistent data​

Different vendors, different timestamps, different definitions of trades and events.

2. Fragmented features​

Core microstructure signals like OFI, spread, imbalance are often re-implemented differently across teams.

3. Research vs production drift​

Research logic and live trading logic diverge over time.

The root cause: there is no single, consistent representation of market microstructure data.

⚙️ QuantFlow's core idea​

QuantFlow is a financial data intelligence system built on one principle:

market data should be structured, versioned, and reproducible across the entire research-to-execution pipeline.

Not just cleaned data. Not just a feature store.

But a shared language for market structure.

🏗️ Architecture: two layers, one shared foundation​

QuantFlow is built as two layers:

• QuantFlow Research (offline analysis layer)
• QuantFlow Streaming (live market layer)

But the key design principle is:

both layers use the same metadata-driven feature definitions

This ensures:

• features are defined once
• reused consistently everywhere
• no divergence between research and production
• identical logic across historical and live systems

🧠 Why this system must be layered​

This architecture is not an implementation preference — it is a structural requirement of how markets and computation behave.

Markets are simultaneously:

• historical (fully observable after the fact)
• real-time (incomplete, streaming, latency-sensitive)
• structurally consistent (same microstructure rules apply)
• operationally different (constraints change completely across time)

Because of this, no single system can optimise all dimensions at once.

🧪 Research layer exists for understanding​

The research layer is designed to:

• reconstruct full market history
• test hypotheses on large datasets
• evaluate signals and regimes
• explore statistical structure of order flow and liquidity

Its constraints are relaxed:

• latency does not matter
• recomputation is acceptable
• completeness of data is critical

In short: research optimises for correctness and completeness of market understanding

⚡ Streaming layer exists for interaction​

The streaming layer is designed to:

• process live tick and order book data
• compute features in real time
• support execution and decision systems
• operate under strict latency constraints

Its constraints are strict:

• every millisecond matters
• computation must be incremental
• partial information is the norm

In short: streaming optimises for speed and real-time responsiveness

🧾 Metadata layer exists for consistency​

Between these two sits the most important layer:

the metadata definition layer

This layer defines:

• what a feature actually means
• how it should be computed
• how events should be interpreted
• how time alignment should behave

Its only job is: ensure that "market structure" has a single consistent definition everywhere

🔁 Why separation is essential (and not optional)​

If research and streaming are forced into a single system, one of two things always breaks:

• either research becomes constrained by real-time limitations
• or production becomes inconsistent with research assumptions

In practice: you either lose correctness or you lose performance

QuantFlow avoids this trade-off by separating concerns while unifying meaning.

⚠️ What breaks without this structure​

Without layering, systems typically suffer from:

• silent divergence between research and live execution
• inconsistent feature implementations across teams
• latency assumptions leaking into research logic
• execution constraints distorting signal design
• non-reproducible research pipelines

These issues are not edge cases — they are structural.

🧩 Metadata-driven pipeline generation (core capability)​

QuantFlow is fundamentally a metadata-driven system.

Instead of manually coding pipelines, users define:

• what market data means and how it should be transformed into features

From this, the system automatically generates:

✔ Data processing pipelines​

• ingestion logic
• event alignment
• timestamp normalization
• missing data handling

✔ Feature computation graphs​

• dependency resolution
• shared computation reuse
• optimized execution ordering

✔ Execution modes​

• batch pipelines for research
• streaming pipelines for live markets
• incremental computation for real-time updates

✔ Versioned and reproducible logic​

• every feature is version-controlled
• transformations are fully traceable
• research and production share identical semantics

A single metadata definition becomes the source of truth for both research and production systems.

🚀 System architecture capabilities​

QuantFlow is designed to operate across multiple scales of market data and system complexity — from historical research to high-frequency live execution.

1. Large-scale research data handling​

QuantFlow supports industrial-scale historical processing:

• multi-year tick datasets
• multi-asset universes
• high-frequency order book reconstruction
• cross-sectional research at scale

2. High-frequency / HFT-grade data processing​

QuantFlow processes event-driven microstructure data:

• tick-by-tick trade streams
• L2 order book updates
• real-time event sequencing
• streaming feature computation

3. Customisable and extensible feature system​

QuantFlow is modular by design:

• custom features via metadata definitions
• extensible microstructure representations
• reusable logic across research and streaming
• integration of new data sources without pipeline rewrites

🧠 What QuantFlow actually changes​

QuantFlow does not aim to improve prediction directly.

Instead, it changes something more fundamental:

how market data is structured, standardized, and operationalized across research and execution.

This leads to:

• consistent feature definitions
• reproducible research pipelines
• reduced research-to-production drift
• scalable cross-asset analysis
• unified logic across all trading environments

🧠 Final thought​

Across this entire series, we moved from:

price → order flow → liquidity → impact → regimes → cross-asset structure → systems

And ended here:

markets are not a prediction problem — they are a representation problem.

QuantFlow is the attempt to formalise that representation layer.

Not as a trading system.

But as:

the infrastructure layer that makes microstructure research and execution consistent, scalable, and production-ready

Read the full series starting with Part 1

Explore QuantFlow: System Overview | Contact

— The QuantFlow Team

This is not just a product site. It is a place where system design, quantitative research, and practical implementation come together.

Why a New Website?​

QuantFlow sits at the intersection of data engineering, quantitative research, and machine learning.

To properly understand its value, it's not enough to describe features — we also need to explain:

• how the system is designed
• why certain architectural decisions were made
• how it fits into real-world quantitative workflows

What We'll Share​

1. System Design​

We will provide detailed insights into how QuantFlow is built across all four components:

DataInfra — the engine-agnostic data foundation:

• Multi-source ingestion with declarative feed provider YAML
• Common Data Model (CDM) with Pydantic validation
• QFSQL — an engine-agnostic SQL dialect for field mappings, compiling to BigQuery, Snowflake, DuckDB, and PostgreSQL
• Auto-generated dbt pipelines and four-layer data quality enforcement

MarketState — market structure reconstruction:

• 8 bar types (fixed + information-driven) via a single-pass Numba fused kernel
• Order book snapshot reconstruction from tick data
• Label Engine with triple barrier, fixed horizon return, trend scanning, and time-series labeling

FeatureDAG — the compiler for quantitative features:

• Formula Language — a mathematical DSL with ~40 functions compiled to an IR DAG via Python's ast module
• 125+ FeatureTypes and 14 MFP packs across 6 dimensions
• 4-stage pipeline: AST compiler → IR DAG → lowering → execution
• 50+ compile-time schema contracts catch errors before any data is touched

Execution Layer — dual-backend runtime:

• Batch (Polars) for research — lazy evaluation, Arrow zero-copy, in-process deployment
• Streaming (DolphinDB) for live trading — deploy-and-forget, sub-ms latency, consolidated engines
• Mode polymorphism: tick / bar / tick_to_bar
• Dagster orchestrates the batch pipeline with 5-stage asset lineage and per-stage retries

2. Product and Business Perspective​

Beyond the system itself, we will discuss:

• how quantitative teams build and scale research pipelines
• the challenges of data fragmentation and feature engineering
• where QuantFlow fits within the broader quant ecosystem
• design trade-offs between flexibility, performance, and usability

3. Theoretical Foundations​

We will also explore the underlying concepts that inform the system:

• market microstructure and event-driven data
• financial data modeling and time alignment
• feature engineering for machine learning
• causality and leakage prevention

Our Goal​

The goal of this platform is to bridge:

• system design and real-world usage
• practical engineering and theoretical understanding

We aim to make QuantFlow not only a tool, but also a reference point for how modern quantitative systems are built.

Explore​

• System Overview — architecture and component design
• Feature Library — 125+ FeatureTypes and 14 MFP packs
• QFDSL Reference — QFSQL and Formula Language references
• Quickstart — get started in 5 minutes

We're building QuantFlow as both a system and a framework for thinking about quantitative finance.

— The QuantFlow Team