Free tools Windows power users keep installed
One-click scans. No signup required.
For uncertain data, flagging is often safer than automatically changing the value. Keep the received record recoverable, test it against rules that reflect the field’s purpose, and route failures for review or quarantine. That does not mean cleaning is always wrong: deterministic, documented corrections can be appropriate. The key is not to treat every failed check as proof that a value is invalid.
Why I prefer flags when a value is uncertain
“Bad” data is not a property a value has on its own. It depends on what the field means and how the data will be used. A missing primary key may break record identity; a missing value in a field where absence is meaningful may be perfectly acceptable. A blanket rule can turn a legitimate exception into a silent data change.
As an Amazon Associate I earn from qualifying purchases.
Common quality problems include missing values, duplicates, and schema drift. Depending on where they occur, they can distort analytics, break jobs, or reduce the accuracy of models. Those risks make validation important, but they do not make automatic replacement the right response in every case. Great Expectations’ ingestion guidance describes these issues and their potential downstream effects.
A failed check is evidence that a record needs attention under a stated expectation. It is not, by itself, proof that the observed value should be overwritten. Great Expectations describes an expectation as a verifiable assertion about data and notes that expectations can change as data and understanding evolve. Its definition comes from the legacy 0.18.21 documentation, so it is best treated as a useful concept rather than a current-version feature claim: Great Expectations 0.18.21: Expectation.
#1 Best Overall
What should happen when a check fails?
Separate detection from disposition. First record what failed and preserve enough context to investigate it. Then decide whether to let the record continue, hold it for review, or block the affected downstream step. This keeps unusual-but-valid cases visible while still preventing serious integrity failures from silently contaminating later work.
Keep the input recoverable
Retain an immutable raw input, or another recoverable representation, separately from derived or corrected data. This is an implementation recommendation, not a claim that Great Expectations provides a particular preservation mechanism. GX documents validation of raw or staged data; your storage and lineage design should make it possible to inspect what arrived and trace any later transformation.
Make each flag actionable
A useful flag should identify the row or key, field, observed value, failed rule, source or batch, timestamp, severity, and current disposition. This is a practical design suggestion, not a required schema. The purpose is to give an owner enough context to distinguish a bad source value from a valid exception or a rule that needs revision.
Quarantine or block only where the risk warrants it
Route hard integrity failures to quarantine or stop the affected downstream step when continuing would make the result unsafe. For lower-risk anomalies, log a warning and permit processing if the pipeline can safely continue. Great Expectations documents validating raw data before warehouse loading so records can be quarantined and source-system bugs identified, as well as conditioning later pipeline steps on validation outcomes. See GX ingestion guidance and GX validation in a data pipeline.
Correct only with a defensible rule
Automatic correction is reasonable when the transformation is deterministic and justified—for example, a documented formatting normalization whose meaning is unambiguous. Keep lineage to the original and record what changed. If multiple interpretations are plausible, flag the record and seek context or an owner rather than guessing.
Which data quality checks should you run?
Choose checks with the data owner and tie each one to the field’s business purpose. dbt Labs identifies five useful baseline dimensions: uniqueness, non-nullness, accepted values, referential integrity, and freshness. Schema expectations can also help catch drift. Not every check belongs on every field; in particular, non-nullness is inappropriate when absence is valid. See Kira Furuichi’s dbt Labs overview of essential data quality checks.
- Requiredness: Is a value mandatory for this field and this use, or is absence meaningful?
- Uniqueness: Should a key or combination of fields identify only one record?
- Accepted values or ranges: Which values are valid according to a documented business rule?
- Referential integrity: Does a value that points to another record have a valid match?
- Freshness: Has the data arrived or been updated within the expected interval?
- Schema: Are expected fields and structures present, and should changes trigger review?
Set thresholds and consequences deliberately. A uniqueness failure on a primary key may warrant a hard stop; an unfamiliar category may instead need a warning if new legitimate values can appear. The rule should state what is expected, and the response should reflect what could go wrong if that expectation is violated.
Recommended Free Tools
Where should validation run?
Validation can run before data is loaded into a warehouse or after raw data has been staged. It can also run alongside transformed warehouse models. The right placement depends on where a failure is cheapest and safest to handle, as well as the source, compute environment, orchestration, and team responsible for maintaining rules.
| Approach | Where it fits | What the cited documentation supports |
|---|---|---|
| Great Expectations | Ingestion or staged raw data, and validation that conditions later pipeline steps | GX documents pre-load validation, quarantine of failing records, identifying source-system bugs, and using validation outcomes to condition subsequent steps. Ingestion; Data pipeline validation. |
| dbt tests | Checks associated with transformed warehouse models and data sources | dbt Labs describes uniqueness, non-nullness, accepted-value, relationship, and source-freshness checks, and cautions that not every column should be non-null. dbt Labs overview. |
These approaches are not a universal either-or choice, and the cited pages do not establish a winner, pricing comparison, or independent benchmark. Compare them against your actual source and compute environment, where you need checks to run, how failures are surfaced and routed, how they fit your orchestration, and who owns rule maintenance. A practitioner asking why they would use Great Expectations when dbt tests already exist is raising a useful design question: add a tool when it covers a specific validation or routing need your existing setup does not, not merely to duplicate checks.
A practical flag-first workflow
- Preserve the received data. Keep raw input recoverable and separate from transformed outputs so a flag or correction can be traced back to what arrived.
- Agree on expectations with the owner. Define requiredness, uniqueness, accepted values or ranges, relationships, freshness, and schema rules where they fit the field’s purpose.
- Validate at an appropriate pipeline stage. Run checks before loading, after staging, or with transformed models according to the failure risk and the work needed to investigate it.
- Emit a contextual flag. Capture the record identifier, field, observed value, failed rule, batch or source, timestamp, severity, and disposition.
- Choose a response by severity. Quarantine or block hard integrity failures; allow safe processing with warnings when a failure needs review but does not make downstream use unsafe.
- Correct only when justified. Apply a documented, deterministic transformation, preserve lineage to the original, and record the change. Escalate ambiguous cases to an owner.
- Review patterns and address causes. If flags recur systematically, investigate the upstream source as well as the rule. GX explicitly identifies source-system bugs as a reason to validate and quarantine records.
What a flag can—and cannot—tell you
A flag makes a mismatch visible: this value did not meet this expectation in this context. It does not settle whether the value is wrong, whether the expectation is incomplete, or whether a correction is safe. That judgment belongs to the people who understand the field and its downstream use. Treating the flag as a triage signal preserves the distinction between detecting a problem and deciding what the data should become.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Scan for outdated or missing drivers - takes under a minuteDriver Scan →
