Amazon Deequ

Overview

Amazon Deequ is a data-quality testing library for Apache Spark, aimed at engineering teams whose datasets already run through Spark. It is a practical choice for adding validation to large-scale data workflows, but teams looking for checks without a Spark dependency should look elsewhere.

Key features

Deequ can check row counts, completeness, uniqueness, allowed and nonnegative values, patterns, and approximate quantiles. That range addresses common data-quality failures, while the Spark foundation makes it relevant to datasets measured in billions of rows. The trade-off is operational: Deequ is a library to use within a Spark environment, not a separate managed testing service.

Its examples extend beyond individual checks to profiling, metric persistence and querying, anomaly detection over time, automatic constraint suggestions, and incremental metric computation. Together these support recurring quality workflows, though teams must incorporate them into their own pipeline. DQDL provides declarative rules for counts, completeness, uniqueness, statistics, schema matching, freshness, and custom SQL.

Row-level evaluation identifies records that pass or fail supported rules. It does not cover every kind of check in that mode: dataset-level rules such as RowCount and Mean are skipped. Teams that need record-level outcomes should account for this distinction when designing their checks.

Deequ releases target particular Apache Spark versions, so selecting a compatible release is essential. The README lists Spark 3.1 through 3.5 compatibility for Deequ 2.x, and versions 2.1.0 and later require Java 11. Maven and sbt dependencies are provided. Python users can use PyDeequ, a Python API for Deequ. The library is Apache 2.0 licensed, and its maintainers welcome feedback and contributions.

Pricing

Deequ is free and open source under Apache 2.0. The Apache 2.0 open-source library plan costs nothing. There are no paid-plan trade-offs to weigh, but the free library requires Apache Spark and a release matched to that Spark version.

Platforms

Deequ is listed for API, Linux, macOS, self-hosted, and Windows use. It runs as a Spark library, with Maven artifacts and a Python interface through PyDeequ. Test execution is self-hosted; supported test languages are Scala, Java, DQDL, and SQL.

It supports schema migration tests, data-quality checks, pre-commit scanning, and custom detection rules. Git is the supported version-control system. It does not provide CI/CD scanning, pull-request scanning, or push protection, so those looking for checks integrated at those stages will need another approach.

Who it's for

Deequ suits data engineering teams that already operate Spark and want to catch quality problems before data reaches downstream systems or machine-learning algorithms. Its scale, multiple rule styles, and metrics workflows make it a strong fit for distributed data pipelines. It is less suitable for teams without Spark or those seeking a managed service with built-in CI/CD, pull-request, or push scanning.

Pros and cons

• Pros: A broad set of checks, profiling, and metrics workflows supports both validation and ongoing monitoring.
• Pros: DQDL offers declarative rules, while Scala, Java, and SQL provide other ways to define tests.
• Pros: Apache 2.0 licensing makes the library free to use and adapt.
• Cons: Teams must run Spark and pair Deequ with the appropriate Spark release; newer releases also require Java 11.
• Cons: Row-level evaluation skips dataset-level checks such as RowCount and Mean.
• Cons: No CI/CD, pull-request, or push-protection scanning is provided.

Alternatives

For a different balance of data-quality tooling and plan limits, compare Great Expectations: its free Developer plan covers up to 5 data assets under test per month, up to 3 users, and unlimited rows per asset, while its Team plan uses custom pricing.

DataKitchen TestGen is another option, with a free Open Source plan limited to 1 user, 1 connection, and 1 project, but unlimited tables and data volume and community support. Choose it if those caps fit better than a Spark-based library.

Atlas is an open-core alternative: its core engine is Apache 2.0 licensed, with advanced features available in proprietary editions. Consider it if an open-core model suits your needs better.

pgroll is a free option for PostgreSQL 14.0 or later, distributed as a single cross-platform binary. Soda offers a free plan with pipeline testing, metrics observability, and alerting and ticketing integrations. For Oracle Database PL/SQL unit testing, utPLSQL is a free and open-source framework. SQLancer is free under the MIT License, and SQLFluff is a free, open-source SQL linter.

Browse Database Testing Tools or Secrets Scanning Software for related categories.

Verdict

Choose Deequ if your team already runs Spark and wants a free, Apache 2.0 library with broad checks and metrics workflows for large datasets. Look elsewhere if Spark compatibility, self-hosted execution, or the absence of CI/CD and pull-request scanning makes it a poor fit for your pipeline.