Dataform

What Dataform is

Dataform is a cloud service for data analysts and engineers developing SQL workflows that transform data in BigQuery. Teams can develop, test, version, and schedule workflows in its web environment, which also supports Git connections, continuous integration and deployment, and workflow execution. Dataform Core adds dependency management, data quality assertions, and documentation to SQL. Repositories can connect to GitHub, GitLab, Azure DevOps Services, or Bitbucket. Workflow scheduling can use Dataform configurations, Managed Service for Apache Airflow, or Workflows with Cloud Scheduler; Cloud Build triggers can automate workflows. Dataform Core is open source and can be compiled and run locally with the Dataform CLI. In Google Cloud, compilation takes place in a sandbox without internet access and cannot call external APIs. Each repository must connect to a custom service account. Dataform itself is free, but BigQuery queries, Cloud Logging, and other services used to execute pipelines may incur charges. Google Cloud compilation uses a plain V8 runtime without additional Node.js capabilities or modules.

Who it suits

Dataform suits data analysts and engineers building and collaborating on BigQuery transformation pipelines. It can also suit teams that want to compile and run Dataform Core locally with the CLI.

Limits

The Dataform service is free, but BigQuery queries, Cloud Logging, and other services used to execute pipelines may incur charges. A repository compilation is limited to 5,000 actions, with at most 50 dependencies per action; Google Cloud compilation cannot access the internet or call external APIs.

Getting started

1. Open the Dataform page in Google Cloud.
2. Create a repository and connect it to a custom service account.
3. Connect a GitHub, GitLab, Azure DevOps Services, or Bitbucket repository if needed.
4. Develop SQL workflows with Dataform Core and use the web environment to execute them.
5. Configure scheduling with Dataform, Managed Service for Apache Airflow, Workflows and Cloud Scheduler, or Cloud Build triggers.
6. For local work outside Google Cloud, compile and run open-source Dataform Core with the Dataform CLI.

Verdict

Dataform combines SQL workflow development, versioning, quality checks, and scheduling for BigQuery teams. The service is free, though execution-related charges and its compilation and runtime limits are important to account for.