DI² generates readable, maintainable and auditable ETL processes in pure SQL - automatically, from the metadata of your source system. No proprietary pipeline editor. No black box. Just SQL your team can understand, review and test.
Which platform, which engine, which cloud vendor. Yet the real question is a different one: do you understand the data you're working with?
The pipeline is built in the UI of a proprietary platform. Logic lives in connectors, transformations are clicked together via drag & drop, code exists only as an export. Reviews, tests and auditability fall by the wayside - and with every tool change the work starts over.
The process is derived from the source system's metadata: tables, columns, data types, relationships. From that, pure SQL is generated - versioned, review-ready, testable. The result runs in any SQL engine you choose, not locked into a tool.
An ETL process is not a one-off project. A year later it must still be understood, extended and reviewed - even by someone who wasn't there.
No custom DSL, no drag & drop artefact. Pure SQL that data engineers and analysts can follow without onboarding.
Every procedure is its own clearly named SQL file. Changing a table doesn't trigger a UI click marathon - it's a diff in the Git repo.
Three logging levels - from the overall run down to the full stack trace. Execution time, step type, table, rows processed: every run is provable in SQL tables.
Read-only access to the source system's schema - tables, columns, types, keys.
DI² reads information_schema and builds a complete model of the source from it.
You add what the DBMS doesn't know: allowed values, ranges, business and alternate keys, historization rules. Domain knowledge becomes metadata - not comments in the code.
Three stages: extraction, historization, transformation. The first two are generated fully automatically from the metadata. The transformation logic into the target structure - DWH or OLTP - stays deliberately hand-written: DI² provides the scaffold, you write the domain logic.
Two modes, one result: synchronous directly in SQL for simple chains, asynchronous via Python for parallel or scheduled runs.
Data quality doesn't come from a dashboard at the end. It comes when every step of an ETL process is provable, reproducible and auditable - and when domain rules (allowed values, ranges, business keys) live as metadata in the model, not as comments in the code.
DI² logs every run on three levels: from the overall process to the full stack trace. Execution time, step type, affected table, number of records processed - all lands directly in SQL tables. Reviewable in the pull request. Auditable via SELECT.
Overall run: start, end, status, duration, run ID, trigger.
Per procedure: type (Extract / Historize / Transform), target table, rows read & written, duration, return value.
On errors: full SQL and procedure-call stack, SQLSTATE, message, context variables - without access to external logs.
A dynamic validation process checks all extracted data against your domain rules - as WHERE clauses that, like every other rule, are stored as metadata. If a record violates a rule, it lands in an error table - readable and auditable like any other table - and is excluded from further processing. No silent pass-through. No surprises in the DWH.
The DI² demo environment hosts six established sample databases - classic training and test datasets from the PostgreSQL, MSSQL and MySQL worlds. Realistic data instead of Lorem Ipsum, with jsonb, geometry, triggers and stored procedures as a generator test surface.
DVD-Rental, jsonb · partitioned tables · FTS
Geospatial - neighborhoods, subway, census
B2B wholesaler, geography · temporal tables
Retail / HR / Production OLTP
DVD-Rental, triggers · views · stored procs
Country / City worldwide, UTF-8 diacritics
DI² reads the metadata of your source system - and generates a first runnable pipeline in PostgreSQL from it.