Architecture¶

ETLantic is a typed modeling, validation, planning, and coordination framework for data pipelines.

It does not implement dataframe computation, distributed scheduling, storage, or infrastructure. It defines a portable logical pipeline, resolves that model for a selected environment, and delegates the resulting plan to plugins and external systems.

Architectural Boundary¶

Standards own meaning.
ContractModel operationalizes data contracts.
ETLantic owns the logical model and resolved plan.
Plugins own backend adaptation and execution.
External systems perform the work.

This boundary is the primary defense against ETLantic becoming another monolithic ETL framework.

Security is a cross-cutting architectural constraint, not a plugin feature. See the Security Model.

System Overview¶

┌──────────────────────────────────────────────────────────────┐
│ Authoring and Interchange                                   │
│                                                              │
│ ContractModel classes   Transformation classes   Pipelines   │
│ Portable expressions   ODCS / DTCS / DPCS documents          │
└──────────────────────────────┬───────────────────────────────┘
                               ▼
┌──────────────────────────────────────────────────────────────┐
│ Typed Logical Model                                          │
│                                                              │
│ Contracts • ports • steps • edges • parameters • identities │
└──────────────────────────────┬───────────────────────────────┘
                               ▼
┌──────────────────────────────────────────────────────────────┐
│ Analysis                                                     │
│                                                              │
│ Introspection • references • validation • diagnostics        │
└──────────────────────────────┬───────────────────────────────┘
                               ▼
┌──────────────────────────────────────────────────────────────┐
│ Planning                                                     │
│                                                              │
│ Profiles • bindings • capabilities • execution regions       │
│ resources • materialization boundaries                       │
│ Gate A interchange descriptors (Polars ↔ Pandas, 0.18)       │
└──────────────────────────────┬───────────────────────────────┘
                               ▼
┌──────────────────────────────────────────────────────────────┐
│ PipelinePlan                                                 │
│                                                              │
│ Immutable • resolved • deterministic • secret-free           │
└──────────────────────┬───────────────┬───────────────────────┘
                       ▼               ▼
              Direct execution     Compilation / generation
                       │               │
                       ▼               ▼
              Runtime plugins      Backend artifacts, docs,
                                   diagrams, lineage

Authoring Layer¶

ETLantic supports complementary authoring paths.

Code-first¶

Data classes define data contracts.
Transformation classes define typed interfaces.
Portable definitions optionally describe relational behavior once through a PySpark-inspired symbolic API (shipped in 0.11+) that normalizes to the published DTCS 3.0 dtcs.transform-plan/2 representation (v1 readable).
Pipeline classes connect extracts, steps, loads, and subpipelines.

Contract-first¶

ContractModel loads ODCS data contracts.
ETLantic integrations load DTCS transformations.
ETLantic loads DPCS pipelines.

Both paths converge on semantically equivalent domain models and one typed logical pipeline graph. ETLantic does not flatten ODCS, DTCS, and DPCS into one universal contract object.

Typed Logical Model¶

The logical model captures portable meaning:

Stable identities
Data-contract references
Typed transformation ports
Parameters and defaults
Extracts and Loads (entry and publication boundaries)
Step instances
Edges and dependencies
Subpipeline interfaces
Callbacks and declared failure policy
Lifespan, middleware, and typed resource requirements
Typed outbound event declarations

It excludes resolved credentials, dataframe objects, scheduler tasks, database connections, and cluster handles.

Beginning with the shipped 0.11+ portable authoring surface, the logical model may also contain a closed dtcs.transform-plan/2 expression graph (v1 remains readable). This graph is data-only and backend-independent; native Polars, Pandas, SQL, and Spark objects remain outside core.

Available in ETLantic 0.11 (authoring)

The canonical expression and Transformation Plan models belong to the dtcs package. etlantic.transform is a PySpark-inspired authoring facade over those public models—not a parallel semantic model. Portable compiler execution remains 0.12+. See Portable Transformations.

Validation Architecture¶

Validation is phased so tools can provide precise diagnostics:

Definition validation — annotations, metadata, identities, and class declarations are internally valid.
Contract validation — ODCS, DTCS, and DPCS artifacts satisfy their authorities.
Graph validation — dependencies, ports, cycles, fan-in, fan-out, and subpipeline boundaries are valid.
Compatibility validation — producer outputs satisfy consumer inputs.
Portable-expression validation — columns, types, outputs, operations, and bounded structure are valid when a portable definition is present.
Profile validation — bindings and resources are complete.
Capability validation — selected plugins and compilers can preserve required semantics.
Runtime data validation — actual inputs and outputs satisfy their data contracts at configured boundaries.

The first seven phases occur before execution. Runtime data validation occurs through ContractModel and backend integrations.

Planning Architecture¶

Planning combines a valid logical pipeline with a profile.

Logical pipeline
      +
Profile
      +
Installed plugin capabilities
      ↓
Resolved PipelinePlan

The planner resolves:

Transformation implementations
Portable transformation compiler selection and operation requirements
Extract and Load asset bindings
Gate A tabular interchange descriptors when engines cross at a boundary (etlantic.interchange/1, Polars ↔ Pandas in 0.18)
Orchestrator selection
Resource-provider references
Execution modes
Artifact boundaries
Retry and timeout requirements
Backend capability constraints
Portable IR and compiler fingerprints
SQL or Spark execution regions

Planning must not execute transformations, acquire live credentials, or materialize data.

Planning should also avoid importing or executing untrusted user modules when a static discovery path is available.

PipelinePlan¶

PipelinePlan is the resolved intermediate representation between authoring and execution.

It should be:

Immutable after construction
Deterministic for equivalent inputs
Serializable where practical
Fully resolved
Inspectable
Versioned
Free of resolved secrets

The plan preserves mappings between logical nodes and physical execution units. This is essential because a backend may fuse several logical transformations into one SQL statement or Spark plan while ETLantic still needs step-level lineage, diagnostics, and failure attribution.

Logical and Physical Graphs¶

ETLantic distinguishes:

Logical graph
User-visible extracts, steps, loads, ports, and contracts

Physical graph
Backend tasks, fused queries, Spark stages, materializations, and submissions

Optimizations may change the physical graph. They must not silently change the observable semantics of the logical graph.

Execution Regions¶

Adjacent compatible nodes may be grouped into an execution region:

A SQL region compiled into one or more statements
A Polars lazy region collected at a sink
A Spark region represented by one logical Spark plan
A local Python region coordinated by the reference orchestrator

Region formation depends on:

Available implementations
Shared execution environment
Validation boundaries
Retry and failure boundaries
Reuse and fan-out
Backend capabilities
Required materialization
Portable-expression compatibility and compiler support

Plugin Architecture¶

The core depends on public protocols rather than backend packages.

Shipped extension families (0.21):

Extension	Responsibility
Dataframe plugin	Execute transformation implementations with a dataframe engine
Portable transformation compiler	Compile DTCS Transformation Plans to native backend operations
SQL plugin and dialect	Compile and execute SQL-native regions
PySpark plugin	Build and submit Spark-native regions
Orchestrator / scheduler plugin	Compile plans (Airflow) or execute plans (Prefect local MVP)
Built-in storage bindings	Memory, callable, JSON, CSV, and no-write asset resolution

Future design (not shipped as public plugin protocols—see Design Proposals):

Extension	Responsibility
Storage plugin protocol catalog	Pluggable persistent dataset providers beyond built-ins
Resource provider protocol	Acquire managed runtime dependencies
Observability provider protocol	Route logs, metrics, traces, and lifecycle events
Notification provider protocol	Deliver typed outbound events beyond built-in outbound policy

Plugins advertise capabilities. The planner selects them only when those capabilities satisfy the logical model.

Runtime Architecture¶

The runtime boundary is async-first.

async def callable
    → await directly

def callable
    → managed worker boundary

CPU-heavy Python
    → process or external mode when declared

Airflow, Spark, dbt, remote service
    → plugin-managed external execution

ETLantic coordinates invocation, concurrency limits, cancellation, timeouts, context propagation, and cleanup. It does not assume worker threads make CPU-heavy Python parallel.

Lifecycle Extension Architecture¶

ETLantic uses separate mechanisms for distinct lifecycle concerns:

Runtime lifespan
    └── initialize and clean up shared runtime state

Run and step middleware
    └── wrap matching logical operations

Resource injection
    └── acquire typed services required by callables

Lifecycle callbacks
    └── respond to specific outcomes with declarative actions

Outbound event declarations
    └── document and deliver typed external notifications

These mechanisms remain separate so ordering, cleanup, portability, and failure semantics are predictable.

Resource Architecture¶

Logical models refer to named resources. Profiles bind those names to resource providers.

Transformation requires "warehouse"
                 ↓
Production profile selects SQLAlchemy provider
                 ↓
Provider resolves credentials at runtime
                 ↓
Managed connection is injected and cleaned up

Resolved secrets must never enter contracts, generated documentation, or a serialized PipelinePlan.

Generation Architecture¶

Validated models and plans can generate:

ODCS, DTCS, and DPCS artifacts
Mermaid and Graphviz diagrams
HTML documentation
Lineage
Pipeline interface descriptions
SQL scripts
Airflow DAGs
Plugin-defined deployment artifacts

Generation must be deterministic and suitable for a CI --check workflow.

Repository and Dependency Direction¶

The intended dependency direction is:

identities + typing + diagnostics
              ↓
authoring + contract integrations
              ↓
logical graph + validation
              ↓
profiles + planning + PipelinePlan
              ↓
Plugin SDK
              ↓
runtime + compilers + CLI

The core package must not require Pandas, Polars, PySpark, Airflow, or a SQL engine.

Architectural Invariants¶

Importing a pipeline never executes it.
Planning never materializes user data.
Profiles do not redefine portable semantics.
Plugins preserve logical meaning or planning fails.
Generated artifacts derive from validated models or plans.
Resolved secrets never enter portable artifacts.
Physical optimization retains logical identity mappings.
Sync and async implementations produce equivalent framework behavior.
Domain standards remain authoritative for contract meaning.
Execution technology never becomes the source of truth.

0.20 trust and safe I/O delta¶

ETLantic 0.20.0 adds fail-closed trust boundaries that apply before plugins load and before profile or artifact files are read from disk.

Pre-import plugin trust¶

Static plugin manifests ship inside each distribution; tampered manifests fail with diagnostic PMPLUG411.
Production profiles require a non-empty plugin_allowlist; unauthorized plugins never reach entry_point.load().
Optional capability probes (require_plugin_probe) validate entry points in an isolated subprocess before load.

Plugin discovery diagnostics surface in validate / SARIF under the plugin_discovery phase (alongside plugin_trust).

Safe filesystem I/O¶

:class:~etlantic.io_policy.SafeIoPolicyconfines reads and writes to **approved roots**, rejects symlink escapes (symlink_policy="reject"), bounds file size, and can enforce atomic writes with integrity digests. Profile JSON load/save, reports, schema history, and cache paths use this policy — not unboundedopen()`.

Artifact and cache isolation¶

Plan artifacts and compiler caches include security domain, tenant, and environment dimensions so cross-tenant reuse fails closed at identity check boundaries.

Outbound policy¶

Runtime outbound HTTP(S) calls evaluate against an explicit host/scheme allowlist; metadata endpoints and private ranges are denied by default.

Fingerprint verification¶

Portable transform plans and interchange payloads carry fingerprints verified at trust boundaries (planning, compilation, and artifact handoff).