Driver FixRecommendedSound, Wi-Fi or graphics acting up? Check drivers firstFind missing or outdated drivers fast.Check DriversHalloween DealsAmazon USHalloween week deals: useful picks before month-endAmazon US: current deals, useful picks and tech finds.Check DealsWindows FixRecommendedWindows errors stealing your time? Find the fix fastScan stability, cleanup and performance issues.Fix Now×
FREEDOM251

Freedom251 › Blog › Apache Spark

Recover Bad Pipeline Runs and Reproduce ML Data with Delta Lake Time Travel

By Freedom251 Team · · 4 min read

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Delta Lake time travel lets you query an earlier table snapshot by version or timestamp, so you can inspect a bad pipeline result or identify the dataset behind an ML run without changing the table’s current state. It is only a recovery window, not an indefinite archive: the transaction-log history and the snapshot’s data files must both still be available.

What time travel gives you

A Delta table’s transaction log tracks changes, while its data files hold the table’s contents. Together, retained log entries and files let Delta Lake present a snapshot from an earlier point in the table’s history. You can query that snapshot to compare results, investigate a data mistake, audit a past state, or reconstruct which data an analysis or model used. The current table remains unchanged by a historical read. Delta Lake describes these time-travel uses.

As an Amazon Associate I earn from qualifying purchases.

That distinction matters during an incident: querying a known-good version can help you understand or rerun work, but it does not by itself roll the live table back. A separate operation is needed to change the current state.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Read a specific snapshot

For repeatable work, prefer an explicit version number when one is available. A timestamp can be convenient for exploration, but timestamps are tied to file and log metadata; copying a table directory can affect timestamp-based reads. The Delta Lake documentation explains both selection methods and their considerations.

#1 Best Overall
Seagate 2TB Portable Hard Drive | USB 3.0 (STGX2000400) | Everyday HDD storage. Plug-and-play backup. Windows & macOS. Adobe CC trial included. 1-Year Warranty
  • Easily store and access 2TB to content on the go with the Seagate Portable Drive, a USB external hard drive
  • Designed to work with Windows or Mac computers, this external hard drive makes backup a snap just drag and drop
  • To get set up, connect the portable hard drive to a computer for automatic recognition no software required
  • This USB drive provides plug and play simplicity with the included 18 inch USB 3.0 cable
  • The available storage capacity may vary.

In Spark

Use the DataFrame reader options versionAsOf or timestampAsOf. For example, after choosing a version that exists and is still retained:

df = (spark.read.format("delta")
      .option("versionAsOf", 42)
      .load("/path/to/table"))

The version number here is illustrative; substitute the version you intend to query. The Delta Lake batch documentation covers historical reads. Confirm that the Spark and Delta Lake versions deployed in your environment support the APIs you use. The current quick start demonstrates setup compatible with Delta Lake 4.0.0, but compatibility depends on the target environment. See the Delta Lake quick start.

Rank #2
Seagate Portable 5TB External Hard Drive HDD – USB 3.0 for PC, Mac, PS4, & Xbox - 1-Year Rescue Service (STGX5000400), Black
  • Easily store and access 5TB of content on the go with the Seagate portable drive, a USB external hard Drive
  • Designed to work with Windows or Mac computers, this external hard drive makes backup a snap just drag and drop
  • To get set up, connect the portable hard drive to a computer for automatic recognition software required
  • This USB drive provides plug and play simplicity with the included 18 inch USB 3.0 cable
  • The available storage capacity may vary.

In SQL

Delta SQL supports VERSION AS OF and timestamp-based historical reads. Consult the documentation for the SQL dialect and syntax supported by your runtime before adopting a query in production. The time-travel guide documents snapshot queries.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Make an ML run reproducible

Record the Delta table identity and selected version in the experiment’s metadata alongside the model code and other run inputs. Then a later run can request that exact snapshot with versionAsOf, rather than silently using whatever happens to be current. This is a practical use of version-addressed reads; Delta Lake supplies the snapshot mechanism, while recording it as experiment metadata is an application-level practice.

Rank #3
Seagate Portable 1TB External Hard Drive HDD – USB 3.0 for PC, Mac, PlayStation, & Xbox, 1-Year Rescue Service (STGX1000400) , Black
  • Easily store and access 1TB to content on the go with the Seagate Portable Drive, a USB external hard drive.Specific uses: Personal
  • Designed to work with Windows or Mac computers, this external hard drive makes backup a snap just drag and drop. Reformatting may be required for Mac
  • To get set up, connect the portable hard drive to a computer for automatic recognition no software required
  • This USB drive provides plug and play simplicity with the included 18 inch USB 3.0 cable
  • The available storage capacity may vary.

If you need to preserve an earlier training dataset separately for testing, a shallow clone can be an option. A clone has its own history, however, so its version identifiers do not directly correspond to the source table’s versions. Track the clone’s identity and the source snapshot from which it was created. Delta Lake’s clone documentation describes shallow cloning.

Know the recovery window—and its limits

Historical reads work only while both the needed transaction-log history and the files that implement the selected snapshot remain available. Delta Lake’s current documentation lists these default table properties:

Rank #4
Seagate Portable 4TB External Hard Drive HDD – USB 3.0, 1-Year Rescue | Everyday HDD storage. Plug-and-play backup. Windows & macOS. Adobe CC trial included. 1-Year Warranty
  • Easily store and access 4TB of content on the go with the Seagate Portable Drive, a USB external hard drive.Specific uses: Personal
  • Designed to work with Windows or Mac computers, this external hard drive makes backup a snap just drag and drop
  • To get set up, connect the portable hard drive to a computer for automatic recognition no software required
  • This USB drive provides plug and play simplicity with the included 18 inch USB 3.0 cable
  • The available storage capacity may vary.
Property Documented default What it controls
delta.logRetentionDuration interval 30 days How long transaction-log history is retained.
delta.deletedFileRetentionDuration interval 7 days The minimum age before logically deleted data files become eligible for VACUUM.

These are defaults in the Delta Lake documentation, not guarantees that every deployment keeps every snapshot for those periods. Table properties and deployment behavior matter. The two settings serve different purposes, and preserving log entries alone cannot restore data files that have been removed. The retention documentation explains the properties.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

VACUUM is not automatic: it removes eligible files no longer referenced by the table. After it removes files required by an older snapshot, that snapshot can no longer be read. Log cleanup is separate and can also limit which versions can be reconstructed. Longer retention can extend the recovery window, but may increase storage use.

Best Value
UnionSine 500GB Ultra Slim Portable External Hard Drive HDD-USB 3.0 | for PC, Mac, Laptop, PS4, Xbox one, Xbox 360-(Black)
  • [Upgraded Version] - This external hard drive features a mirrored logo stripe combined with a striped anti-slip design, and the rounded corners of the casing make it easier to grip. The stripes also have a heat dissipation function, ensuring stable and fast data transfer.
  • 【Ultra-thin and quiet】 - The motherboard adopts JMicron 578 noise-free solution, giving you a quiet working environment. Lightweight and portable size designed to fit in your pocket for easy portability.
  • 【Ultra-Fast Data Transfers】 - Pairing this external hard drive with JMicron 578 solution USB 3.0 and USB 2.0 interfaces enables blazing-fast data transfer. It boasts theoretical read speeds of up to 125MB/s and write speeds of up to 103MB/s.
  • 【Plug and Play】 - With no software to install, just plug it in and the drive is ready to use.The hard disk chip is wrapped with an aluminum anti-interference layer to increase heat dissipation and protect data.
  • 【What You Get】 - 1 x Portable Hard Drive, 1 x USB 3.0 Cable, 1 x User Manual, Gift-type shell packaging ,Three-year manufacturer's warranty and free technical support services.

Set retention around real workloads

Choose deleted-file retention to exceed the longest concurrent reader or writer operation and the longest stream lag or outage you need to tolerate. The Delta Lake utility guide recommends keeping at least seven days for VACUUM safety; a deployment with longer-running work or a longer recovery target may need more. Check the behavior and supported configuration for your deployed Delta Lake version before changing defaults. See the Delta Lake VACUUM guidance.

Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Support on Ko-Fi

Choose between a historical read and RESTORE

A historical read examines a prior snapshot while leaving the current table state alone. RESTORE, by contrast, commits a new operation that makes the table’s current contents correspond to an earlier state. It does not erase the intervening history; it creates another data-changing commit. Delta Lake documents RESTORE behavior.

Before restoring, assess downstream consumers. A restore commit is marked as a data change, and streaming applications may interpret its records as new changes and process data again. That can cause duplicate downstream processing unless consumers account for the restore. Validate downstream semantics and recovery procedures before using RESTORE on a live pipeline.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Time travel is not a permanent change log

Time travel answers questions about retained snapshots; it does not automatically provide a perpetual record of every row-level change. Delta Lake Change Data Feed is a separate feature that must be explicitly enabled, and it captures changes only after enablement. Its change records are subject to table retention and can be removed through VACUUM. Read the Change Data Feed documentation before treating it as part of an audit or downstream-processing design.

Quick Recap

Bestseller No. 1
Seagate 2TB Portable Hard Drive | USB 3.0 (STGX2000400) | Everyday HDD storage. Plug-and-play backup. Windows & macOS. Adobe CC trial included. 1-Year Warranty
Seagate 2TB Portable Hard Drive | USB 3.0 (STGX2000400) | Everyday HDD storage. Plug-and-play backup. Windows & macOS. Adobe CC trial included. 1-Year Warranty
This USB drive provides plug and play simplicity with the included 18 inch USB 3.0 cable; The available storage capacity may vary.
$129.99
Bestseller No. 2
Seagate Portable 5TB External Hard Drive HDD – USB 3.0 for PC, Mac, PS4, & Xbox - 1-Year Rescue Service (STGX5000400), Black
Seagate Portable 5TB External Hard Drive HDD – USB 3.0 for PC, Mac, PS4, & Xbox - 1-Year Rescue Service (STGX5000400), Black
This USB drive provides plug and play simplicity with the included 18 inch USB 3.0 cable; The available storage capacity may vary.
$219.99
Bestseller No. 3
Seagate Portable 1TB External Hard Drive HDD – USB 3.0 for PC, Mac, PlayStation, & Xbox, 1-Year Rescue Service (STGX1000400) , Black
Seagate Portable 1TB External Hard Drive HDD – USB 3.0 for PC, Mac, PlayStation, & Xbox, 1-Year Rescue Service (STGX1000400) , Black
This USB drive provides plug and play simplicity with the included 18 inch USB 3.0 cable; The available storage capacity may vary.
$119.80
Bestseller No. 4
Seagate Portable 4TB External Hard Drive HDD – USB 3.0, 1-Year Rescue | Everyday HDD storage. Plug-and-play backup. Windows & macOS. Adobe CC trial included. 1-Year Warranty
Seagate Portable 4TB External Hard Drive HDD – USB 3.0, 1-Year Rescue | Everyday HDD storage. Plug-and-play backup. Windows & macOS. Adobe CC trial included. 1-Year Warranty
This USB drive provides plug and play simplicity with the included 18 inch USB 3.0 cable; The available storage capacity may vary.
$189.99

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.

Apache SparkData EngineeringData pipelinesDelta LakeMachine Learning

Leave a Reply

Your email address will not be published. Required fields are marked *

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Recommended PC Tool
Recommended PC Tool
Crashes, No Sound, or Screen Glitches?Free driver scan
PC Slower Than It Used to Be?Free scan - under a minute

Two free Windows tools

One Free Minute Could Fix That PC

Before you go - each of these free tools takes about a minute and tackles what quietly slows a Windows PC down.

Special offer. View Outbyte info, uninstall instructions, EULA, and Privacy Policy.