Ab Initio Data Quality May 2026

Go ab initio , or go home. [Your Name] writes about the intersection of rigorous engineering and practical data science. Disagree with the zero-NULL policy? [Link to comments or Twitter.]

Ab initio (Latin for "from the beginning") means starting from first principles. In a quantum simulation, you don't patch errors later—you define the laws of physics upfront. If your initial conditions are wrong, the simulation is worthless.

Replace NULL with explicit semantics. Use -999 for "offline," -9999 for "out of range," or better—split the column into value and value_metadata_flag . 3. The Referential Integrity Illusion Modern data lakes love "schema on read." This is the enemy of ab initio . You are essentially saying, “Let’s store the garbage, and we’ll figure out what kind of garbage it is later.” ab initio data quality

If you work in data long enough, you’ve heard the mantra: “Garbage In, Garbage Out.” We all nod in agreement. Then, we build complex pipelines with 47 validation steps, six months of cleaning scripts, and a "trust but verify" dashboard that nobody actually reads.

Stop cleaning the swamp. Stop building the bridge. Stop the garbage at the gate. Go ab initio , or go home

Ab Initio Data Quality: Why You Can’t Fix Rubbish Later

Use tools like pydantic (Python), Great Expectations (with expect_column_values_to_not_be_null set to fatal ), or dbt 's constraints (enforced, not just documented). If the contract fails, the pipe breaks. Loudly. [Link to comments or Twitter

Audit your warehouse. Pick one critical table. Enforce NOT NULL on every single column. If you truly need a missing value, use a sentinel row (e.g., id = 0 , name = "UNKNOWN" ). You will be shocked how many bugs disappear.