Typed, contract-driven pipelines

Design once.
Validate everywhere.

Model data pipelines in Python, validate them as contracts, and run them where your engines already are.

ETLantic is a Python framework for defining typed, contract-driven data pipelines and coordinating their execution through the tools users already choose.

Its central idea is simple:

Define data, transformations, and pipelines with typed Python classes. Validate and plan them once. Execute them through interchangeable backends.

ETLantic also treats validation as a continuous envelope around ETL. ETLantic validates at every typed boundary—before extract, between steps, and at load—without treating validation as an ordinary transform. See Validation Everywhere.

This is the practical meaning of the name: ETL is the data flow; ETLantic is ETL surrounded by typed contracts, validation, planning, and evidence from source to publication.

ETLantic is inspired by FastAPI's type-driven developer experience, but it does not turn ETL into a web API metaphor. It applies the same principle—types as executable interface declarations—to data engineering.

Project status¶

ETLantic 0.21.0 is stable for documented single-tenant reference deployments. It models, validates, and plans typed Python data pipelines, then runs them locally or through optional engine plugins.

Use today: single-tenant reference deployments (see Capabilities).
Not included: multi-tenant control plane, managed Spark, unrestricted enterprise compliance attestations beyond shipped SBOM/attestations (adopter-owned for broader compliance programs).
Experimental: Structured Streaming; etlantic-datafusion (Gate B stub).

Install · Quickstart · Engine selection · Capabilities

Green path (start here only)

Installation — pip install etlantic==0.21.0
Quickstart — etlantic init (five-minute success)
First Pipeline — evolve the generated project
Engine selection — then an engine tutorial

Diligence (after first success): Capabilities, What's new in 0.21, What's new in 0.20, Evaluator, Compare. Pages marked Future design are not APIs. Design studies under Examples are aspirational—not installable APIs.

Minimal working example¶

The canonical first success is CLI-first:

pip install 'etlantic==0.21.0'
mkdir my-pipeline && cd my-pipeline
etlantic init --with-toml
etlantic validate pipeline.py:SamplePipeline --profile development
etlantic plan pipeline.py:SamplePipeline --profile development
etlantic run pipeline.py:SamplePipeline --profile development
cat data/out.json

Follow the full Quickstart, then First Pipeline to evolve the generated contracts. The PyPI wheel does not include examples/; from a checkout an optional in-memory SDK demo is examples/memory_customers.py.

From the same typed definitions ETLantic can also generate ODCS / DTCS / DPCS contracts, Mermaid lineage, and a secret-free PipelinePlan. Optional plugins add Polars, Pandas, SQL, PySpark, and Airflow compilation.

The transformation implementation remains separate—register "polars", "sql", and other engines the same way after installing the matching plugin.

The Architecture in One View¶

Typed Python authoring or portable contracts
                    │
                    ▼
          Typed logical pipeline model
                    │
                    ▼
     Introspection and semantic validation
                    │
                    ▼
        Profile and capability resolution
                    │
                    ▼
        Immutable PipelinePlan (resolved IR)
                    │
          ┌─────────┼──────────┐
          ▼         ▼          ▼
      Execute    Compile    Generate
          │         │          │
          ▼         ▼          ▼
      Plugins   Airflow/SQL  Docs/graphs

Runtime execution adds contract checks around extracts, transformation inputs and outputs, engine/interchange boundaries, and loads. Those checks remain visible in plan decisions and run evidence even when a backend safely fuses their physical execution.

ETLantic owns modeling, validation, planning, and coordination.

Standards own contract meaning. ContractModel owns data-contract operationalization. Plugins and external systems perform the work.

The Three Contract Authorities¶

ETLantic intentionally recognizes only three top-level contract families:

Contract	Authority	Answers
Data contract	ODCS and ContractModel	What is valid data?
Transformation contract	DTCS	How is data expected to change?
Pipeline contract	DPCS	How are data and transformations composed?

Profiles, plugins, resources, callbacks, artifacts, and execution plans are implementation or runtime concepts—not additional contract standards.

Choose Your Path¶

Follow the Green path above for first success. Optional persona forks:

I want to run something in five minutes¶

Same as the Green path: Installation → Quickstart (etlantic init) → then Capabilities. From a checkout, optional companions include examples/memory_customers.py and file_storage.py.

I want to understand the idea¶

I want to author pipelines¶

I want to understand execution (shipped)¶

Execution Model
Local Python
Secrets Management (env + file; optional etlantic-keyring)
Polars / Pandas
SQL
Run Reports

I want runnable examples¶

examples/memory_customers.py — in-memory SDK demo
examples/file_storage.py — JSON/CSV storage
examples/dataframe_parity.py — Polars/Pandas
examples/sql_to_sql.py — SQL (etlantic-sql)
examples/sql_boundary_hybrid.py — SQL → Python boundary
examples/sql_transactional_write.py — insert-select publication
examples/sql_failure_recovery.py — unsupported merge fails closed

I want design studies (aspirational)¶

These pages describe intended 1.0 workflows. They are not current API guides:

CSV to CSV (design narrative; prefer examples/file_storage.py)
SQL to SQL (design narrative; prefer examples/sql_to_sql.py)
Airflow Pipeline
PySpark to Delta

I want to build a dataframe or SQL plugin¶

I want to extend plugins (shipped protocols)¶

Plugin SDK overview — shipped dataframe, SQL, Spark, orchestration, and transform-compiler protocols
Testing Plugins
Future (not shipped): broader storage/resource/observability protocol catalogs — see Storage today vs Future-bannered pages

I am integrating or migrating SparkForge¶

Documentation Map¶

Section	Purpose
Getting Started	Learn the core workflow
Foundations	Understand product philosophy and architecture
Data Contracts	Define and operationalize typed datasets
Transformations	Define typed transformation interfaces
Pipelines	Compose transformations into portable graphs
Execution	Local runtime, secrets, dataframe and SQL engines
Orchestration	Airflow compiler (`etlantic-airflow`); other orchestrators future
Visualization	Mermaid, Graphviz DOT, HTML lineage
Examples	Runnable pointers + design studies
Reference	CLI, API, compatibility
Development	Contributing, roadmap, release
Specifications	Normative DTCS and DPCS documents

Non-Goals¶

ETLantic is not intended to become:

A dataframe engine
A distributed scheduler
A storage system
A secret manager
A proprietary pipeline contract format
A replacement for Pandas, Polars, SQL engines, Spark, Airflow, or Dagster

It is the typed control and interoperability layer that connects those systems without allowing any one of them to define the pipeline's portable meaning.

Design once.Validate everywhere.