Frequently Asked Questions¶
What is ETLantic?¶
ETLantic is a Python framework for defining typed, contract-driven data pipelines. It validates them, generates portable contracts, creates deterministic plans, and can execute registered Python implementations through its local runtime.
Is ETLantic an orchestration framework?¶
No.
ETLantic models pipelines and produces resolved PipelinePlan objects. It
includes a built-in LocalScheduler for development and tests, and
intentionally delegates durable scheduling and external workflow platforms to
optional plugins (for example Airflow compilation via etlantic-airflow, or
Prefect direct execution via the shipped etlantic-prefect
ExecutionScheduler local MVP).
Is ETLantic 0.21 production-supported?¶
ETLantic 0.21.0 is stable for documented single-tenant reference deployments (not unrestricted enterprise production). See Capabilities and Production readiness. Multi-tenant isolation, deployment topology, compliance, and advanced supply-chain controls remain adopter-owned.
What is the difference between Stable and Experimental?¶
Stable APIs and behaviors are supported within the documented 0.21 reference
envelope. Features explicitly labeled Experimental, currently including
Structured Streaming foundations and etlantic-datafusion, may change and are
outside that stable claim. A page describing a shipped feature does not make
every feature on that page stable; check its status label and
Capabilities.
Is ETLantic an ETL engine?¶
No.
ETLantic does not implement a dataframe engine, database clients, scheduling, or distributed cluster management. It includes an in-process local runtime with memory, callable, JSON, CSV, and no-write storage, plus optional Polars, Pandas, SQL, and PySpark plugins that execute through versioned protocols.
Instead, it coordinates existing tools through a common typed model.
Why create ETLantic instead of using Airflow or Dagster?¶
Airflow and Dagster are excellent orchestration systems.
ETLantic solves a different problem.
Its focus is:
- typed pipeline modeling
- contract generation
- validation
- portability
- execution abstraction
ETLantic's architecture is designed so plugins can consume the same
logical model. Use Airflow (via etlantic-airflow) to compile plans into DAG
artifacts; use a direct-execution scheduler (built-in LocalScheduler, or
the shipped etlantic-prefect local MVP) to run resolved plans in process; use ETLantic for
typed contracts and fail-closed planning.
How does ETLantic compare to dbt, Prefect, or Pandera?¶
| Tool | Primary job | ETLantic relationship |
|---|---|---|
| dbt | SQL transformation project / warehouse analytics | Complementary. ETLantic models typed Python pipelines and multi-engine plans; dbt owns SQL project workflows. |
| Prefect / Dagster / Airflow | Orchestration and scheduling | Complementary. Airflow compiles plans to DAG artifacts; the shipped etlantic-prefect local MVP directly executes resolved plans (it is not a DAG compiler). Prefect deployment/serve and Dagster remain future. |
| Pandera / Great Expectations | Dataframe / table validation libraries | Complementary. ETLantic validates wiring and contracts before run; row-level suites remain engine-side or library-side. |
If you need only SQL analytics projects, start with dbt. If you need only schedulers, start with Airflow/Dagster/Prefect. If you need typed pipeline composition across engines with secret-free plans, evaluate ETLantic.
Why is ETLantic inspired by FastAPI?¶
FastAPI demonstrated that Python type annotations can become the foundation for an outstanding developer experience.
ETLantic applies the same philosophy to data engineering.
Types define interfaces.
Everything else can be inferred.
What are Data Contracts?¶
Data contracts describe datasets.
ETLantic uses ContractModel-compatible Pydantic models as the source of truth and generates Open Data Contract Standard (ODCS) documents from those models.
What are Transformation Contracts?¶
Transformation contracts describe the logical interface of a transformation.
They specify:
- inputs
- outputs
- parameters
- metadata
They intentionally do not specify implementation details.
Can one transformation run on Polars, PySpark, Pandas, and SQL?¶
Yes, when you author with @Transformation.portable and install the matching
engine plugin. Support differs by engine—see the
Portable Compiler Matrix.
Use a native @implementation(...) for anything outside that matrix.
What are Pipeline Contracts?¶
Pipeline contracts describe how data contracts and transformation contracts are connected together.
ETLantic can generate Data Pipeline Contract Standard (DPCS) documents directly from pipeline classes.
Why are execution engines separate?¶
Keeping execution separate allows the same logical pipeline to execute through different runtimes.
Examples include:
- local Python
- Polars / Pandas (optional plugins)
- SQL (
etlantic-sql) - PySpark (
etlantic-pyspark) - Airflow compile (
etlantic-airflow) and Prefect local scheduler (etlantic-prefect) - Optional SQLModel / keyring packages (
etlantic-sqlmodel,etlantic-keyring)
The transformation contract and pipeline wiring remain unchanged; native implementation bodies may still differ by engine.
Which dataframe engine should I use?¶
ETLantic is dataframe-engine neutral.
Install etlantic-polars or etlantic-pandas and set
Profile.dataframe_engine accordingly. Prefer Polars when you need lazy
preservation or portable relational compilation; use Pandas when you need the
Pandas ecosystem (eager portable relational compilation is available).
SQL is available via etlantic-sql and Profile.sql_engine="sql". Spark is
available via etlantic-pyspark and Profile.spark_engine="pyspark".
Which engine should I start with?¶
Start with the built-in local Python engine and memory or JSON/CSV storage; it has no optional engine dependency and makes validation and wiring easiest to understand. Add Polars for a first dataframe engine, Pandas for ecosystem compatibility, SQL when work should remain in PostgreSQL, or PySpark only when you need Spark semantics and have a working Java environment.
Must core and plugin versions match?¶
Yes. Keep core and optional plugins on the same minor release. For a reproducible 0.21.0 environment, pin both exactly, for example:
A mismatched plugin may fail discovery, protocol checks, validation, or planning even when both packages install successfully.
Why do validate and plan work but CLI run has no input data?¶
Validation and planning inspect definitions and do not need source rows. The
quickstart seeds memory inside its Python process; a later etlantic run
process has a fresh in-memory store and cannot see those records. Run that
example with python pipeline.py, or configure JSON/CSV or another durable
storage binding for cross-process CLI execution.
Does ETLantic support asynchronous execution?¶
Yes.
Users may write synchronous (def) or asynchronous (async def)
implementations.
ETLantic normalizes invocation internally so authors do not need to manage event loops, worker threads, or execution scheduling.
Do I have to write YAML contracts?¶
No.
The preferred workflow is code-first.
ETLantic generates ODCS, DTCS, and DPCS contracts automatically from Python definitions.
Existing contracts can also be loaded and consumed.
Can I use existing ODCS contracts?¶
Yes.
ETLantic supports loading existing ODCS contracts and integrating them into typed pipeline definitions.
Is validation optional?¶
Validation is a core feature.
ETLantic validates contracts, pipeline wiring, parameter types, and implementation compatibility before execution whenever possible.
Can one transformation have multiple implementations?¶
Yes.
For example, the same transformation contract may have:
- a local Python implementation
- a Polars implementation
- a Pandas implementation
- a SQL implementation (
@….implementation("sql")withProfile.sql_engine) - a PySpark implementation (
@….implementation("pyspark")withProfile.spark_engine)
The logical transformation remains unchanged.
Is ETLantic tied to a specific cloud provider?¶
No.
ETLantic is designed to be cloud-agnostic.
Cloud-specific integrations are implemented through plugins.
Can ETLantic generate documentation?¶
Yes.
ETLantic generates or exposes:
- contract documentation
- pipeline documentation
- lineage diagrams (including Graphviz DOT and HTML exporters)
- Mermaid graphs
- execution plans
Does ETLantic include SparkForge / medallion layers?¶
No.
Bronze / silver / gold stay in SparkForge. Optional package
etlantic-sparkforge maps medallion IR onto ordinary ETLantic nodes,
profiles, and reports. See Migrating 0.9 → 0.10.
Who should use ETLantic?¶
ETLantic is intended for:
- data engineers
- analytics engineers
- platform engineers
- ETL framework authors
- organizations adopting contract-first data engineering
Where should I go next?¶
- Installation → Quickstart → First Pipeline → Engine selection
- Capabilities for the shipped boundary
- Runnable examples from a checkout (see Examples); pip users stay on paste-ready Quickstart
- CLI reference for
etlantic validate|plan|run|compile|viz - Foundations (philosophy / architecture) when you want deeper design context