The Four Silent Data Errors That Pass Every Cleaning Pipeline

Your pipeline ran clean. No nulls, no duplicate rows, no type mismatches. Every column passed every check, and the table landed in the warehouse with a green light. Then the model came back wrong, or the dashboard moved a number nobody could explain, and the investigation started from a place of false confidence: the data was clean, so the problem must be somewhere else.

Usually it isn’t. A quarter of organizations say poor data quality costs them more than $5 million a year, and 7% put the figure above $25 million, according to a 2021 Forrester finding that IBM cites in its own data quality research. Most of that damage never shows up on a dashboard. It hides in tables that passed every automated gate.

The four bug classes below all clear null checks, type checks, and range checks. They are duplicated entities that don’t look identical, values that sit inside range but are wrong for the row, timestamps that leak information from the future, and denominators that quietly drop the people who would have changed the answer. Each one is the kind of thing that ships to production because nothing in the pipeline was built to catch it, and each one is worth recognizing before it reaches a feature table.

A three-layer test framework borrowed from linguistics

The cleanest way to think about data quality is to borrow a distinction from linguistics. Syntax asks whether a value is well formed. Semantics asks whether it makes sense for what it represents. Pragmatics asks whether the same kind of value needs different tests depending on who is using it.

A five-digit zip code passes syntax. A 214-year-old customer passes syntax and fails semantics. A 3 am login at a store that closes at 10 pm passes syntax, passes a naive semantic check on the timestamp format, and fails the moment you ask what the timestamp means. Pragmatics is where things get genuinely hard: in a pharmaceutical company, the drug discovery team and the manufacturing team disagree about what counts as a valid record, and both are right for their own purposes.

Most production pipelines stop at syntax. That is not laziness so much as architecture. The tools were built to answer the questions that don’t require knowing the business.

Why cleaning pipelines stop too early

Standard checks answer a narrow question: is this value well formed? Is the column non-null, is the date parseable, is the zip code five digits? Those checks are cheap, generic, and genuinely useful. They are also the end of the line in most production pipelines, and the end of the line is well before the interesting failures begin.

Pandera and dbt tests, the SQL queries that check data quality, are excellent at type, null, and range checks. They default to those because those are the checks a generic library can write without knowing anything about what the data means. A library can tell you a column is an integer. It cannot tell you that a salary far below the range of every peer at that level and location is probably a data entry error.

Semantic checks need a person who understands the data’s meaning, and that gap is exactly where the four bugs live. The tools aren’t failing. They are doing precisely what they were designed to do, and the design boundary is the problem.

The gap matters more as models take on more decisions, because a silent error in a feature table doesn’t stay in the feature table. It propagates into training data, into offline metrics, into the finance report that someone presents on a Monday. The same discipline that makes a research pipeline reproducible is what catches these bugs, and it is worth reading how the field thinks about scale and rigor in venues like Inside arXiv’s cs.LG Firehose: What the Machine Learning Listing Page Reveals About Research at Scale before assuming your own checks are exhaustive.

Bug 1: Duplicate entities that aren’t identical rows

drop_duplicates() catches “John Smith, 123 Main St” appearing twice. It does not catch that string sitting alongside “J. Smith, 123 Main Street.” Two rows, one customer, and a lifetime-value model that now counts two people. The check passes because the row isn’t duplicated. The entity is.

This happens unintentionally all the time when two source systems ship the same table into a warehouse with no dedup step before the join. Every downstream SUM or COUNT doubles, and no null check flags it. The same grain confusion shows up in production when a metric is defined at the wrong level: one user firing 10 events looks like 10 users if the aggregation counts rows instead of distinct entities, and COUNT(DISTINCT user_id) and COUNT(*) give different answers for the same table.

The fixes work in both directions. Canonicalize strings first: lowercase, strip punctuation, normalize addresses. Then score similarity instead of testing equality, and route the in-between cases to a review queue rather than forcing a binary decision.

Bug 2: Values in range but wrong for the row

A 214-year-old customer passes an integer check. A negative delivered quantity passes a numeric check. Catching either requires a rule that knows what the number means, not just its type.

Salary is the classic case. Take a query that flags every manager earning less than twice the average salary of their direct reports. The query is correct. Then a single bad salary, whether zero, negative, or entered in the wrong unit, shifts the average and silently adds or drops managers from the result. The bug sits in one cell, and every syntax check waved it through.

Two layers of defense help. Business-aware constraints at ingestion encode what is possible for the business rather than what fits the data type: age between 0 and 120, order quantity greater than zero. Peer-group plausibility checks go further by comparing a value against the distribution for the same level and location, which catches a number that is plausible on its own but wrong for its row.

Bug 3: Timestamp leakage

Leakage happens when a feature carries information from after the moment it is supposed to describe. The cleanest definition, as Wikipedia puts it, is information that would not actually be available at prediction time.

This is the failure mode that inflates offline metrics and then disappears in production, which makes it especially dangerous because the model looks better right up until it meets reality. A timestamp column can pass every format check, sit in a perfectly reasonable range, and still encode the future. Catching it requires asking a question no generic test can ask: at the moment this row was created, would this value have existed yet? The same forecasting discipline shows up in domains where timing is everything, as in work that AI Forecasts Geomagnetic Storm Risk Across 66,935 Substations With 30 – 60 Minutes of Warning, where a feature built from post-event data would be worse than useless.

Bug 4: Denominator bias

Denominator bias is the subtlest of the four: denominators that drop the people who would have changed the answer. The rows that vanish from a metric are often the ones that would have moved it most, and nothing in a syntax check flags an absent population.

A churn rate computed only over customers who have a complete profile is not a churn rate. A conversion metric that quietly excludes sessions where tracking failed is not measuring conversion. (These are illustrative examples; the source material describes denominator bias only in general terms.) The table is clean. The number is wrong. No null check will ever tell you, because the missing rows aren’t missing from the table. They were never there.

What to do next

Move semantic checks into the ingestion path rather than bolting them on downstream, where the damage is already done. Write constraints that encode what is possible for the business, not what fits the data type, and pair them with peer-group comparisons so a value has to be plausible relative to its neighbors, not just to zero. Treat entity resolution as a pipeline stage with its own canonicalization, similarity scoring, and review queue, not as an afterthought you add when someone notices the counts are off.

The broader point is about who builds the checks. Generic libraries will keep getting better at syntax, and they should. The semantic layer requires someone who knows what the data means, and that person is not optional. If you are building that skill set from scratch, resources like 5 Free Courses to Learn AI Engineering in 2026: From LLM Basics to Production Systems cover evaluation and monitoring alongside the modeling work.

A pipeline that passes every check is not the same as a pipeline that is right. The four bugs above live in the space between those two statements, and closing that space is the work.