Skip to main content

QuantFlow FeatureDAG

Declarative YAML → IR DAG → Polars (batch) + DolphinDB (streaming)

🔗 Where It Fits

DataInfra → MarketState → FeatureDAG → Research / Trading

📋 Overview

FeatureDAG is a compiler for quantitative finance features. It eliminates the research-to-production gap by treating financial features as declarative computation graphs, not hand-coded functions. A single YAML specification compiles through a 4-layer pipeline and produces both batch (Polars) and streaming (DolphinDB) execution from the same definition.

Quant researchers write features in notebooks with pandas and one-off scripts. Production engineers rewrite them for streaming pipelines. The two diverge — results drift, bugs creep in, and every new feature requires a full rewrite cycle. FeatureDAG eliminates this gap entirely.

[Formula YAML] → [AST Compiler] → [IR DAG] → [Lowering] → [Execution]
┌───────────────┴───────────────┐
Polars (batch)        DolphinDB (streaming)

🏗️ 4-Stage Compilation Pipeline

Stage 1

AST Compiler

Formula string → IR nodes. Parses Python AST, dispatches ~40 built-in functions.

Stage 2

IR DAG

Frozen IR nodes. 50+ schema contracts. Column aliasing. Rustworkx DAG.

Stage 3

Lowering

Backend protocol. Decorator-based registry. 30+ agnostic ops.

Stage 4

Execution

Polars lazy pipeline (batch) + DolphinDB stream engines (streaming).

🧩 6 Computation Primitives

Every feature computation falls into one of six primitives. The compiler understands the semantics of each and optimizes accordingly.

PrimitiveNature
SOURCEData ingestion from Market Data Model tables
TRANSFORMStateless row-wise mapping (mid_price, trade_sign, depth_imbalance)
WINDOWRolling/windowed aggregation (rolling_vol, rolling_corr, pct_change, lag)
STATERecursive computations with memory (ema, decay_accum, rolling_zscore)
SINKMarks output columns — final feature values written to storage
EVENTBar trigger generation for information-driven sampling

🔄 Batch + Streaming Consistency

Same feature definitions, two execution models. The divergence happens only at the expression-generation layer.

BatchStreaming
RuntimePolars (Python)DolphinDB cluster
DataStatic Arrow tablesUnbounded stream tables
TriggerExplicit run()Continuous — data arrival
GroupingSequential per-featureConsolidated engines (~60% fewer)
UseResearchLive trading, real-time signals

Same definitions, two runtimes — no duplicate implementations.

💡 Key Design Decisions

  • Feature definitions are declarative, not imperative — YAML specs compile to IR. No hand-coded computation logic in the pipeline.
  • Single IR, multiple backends — The IR is backend-agnostic. Lowering functions produce engine-specific expressions for Polars and DolphinDB from the same IR nodes.
  • Feature-level error isolation — A misconfigured feature is logged and skipped. The rest of the pipeline continues — critical for research iteration speed.
  • LazyFrame chaining in batch — Polars expressions are chained via successive with_columns calls, letting the query optimizer fuse operations and minimize allocations.
  • Consolidated engine deployment in streaming — Multiple features sharing the same input table deploy as a single DolphinDB engine, reducing engine count by ~60%.
  • Expression folding in streaming — Intermediate computation steps are inlined into terminal expressions via regex substitution. Deployed engines only see final outputs.

📚 Standard Feature Library

FeatureDAG ships with 133 FeatureTypes across 6 dimensions (Signal, Execution, Quality, Regime, Stability, Technical).

Each feature is documented with its inputs, parameters, computation type, and economic meaning. Features are consumed directly by the compiler — no manual implementation needed.

Browse the Standard Feature Library →

📖 Design Docs

← Back to Home