Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Choose capacity from the bytes your project must keep at the same time—not from a row count or a single rule such as “twice the dataset.” Start with the source data on disk, then add simultaneously retained cleaned or transformed copies, staging and cache space, checkpoints and outputs, local backups and versions, expected growth, and working headroom. After that, test whether the drive can deliver the required throughput and concurrency; a capacity number alone cannot prove that one hard drive is suitable for training.
Use this planning identity: required capacity = retained source data + simultaneously retained processed or staged copies + retained checkpoints and outputs + backup and version copies + expected growth + working headroom. Calculate every term from your own pipeline. There is no universal AI-dataset drive size, and published platform figures are starting estimates for particular environments rather than consumer-drive recommendations.
As an Amazon Associate I earn from qualifying purchases.
Start with the data you will actually retain
Measure bytes on disk
Export or inventory the files and database extracts that the training job will read, then record their real size on disk. A row count, token count, or uncompressed theoretical size can be misleading because encoding, metadata, compression, and duplicate files change the footprint. Record both the current total and the date of the measurement so growth can be calculated later.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Separate the modalities because they behave differently. Text and structured records may be relatively compact but can expand during tokenization or feature generation. Images, audio, and especially video can dominate capacity. Keep a list by dataset partition (for example, train, validation, and test) so a later retention decision does not accidentally delete an evaluation set.
#1 Best Overall
- High-capacity add-on storage.Compatibility : Windows 10 plus, Reformatting required for use with MacOS.
- Fast data transfers
- Plug-and-play ready for Windows PCs
- WD quality inside and out
Define what “stored” means for this project
Decide whether the source is a permanent local copy, a temporary download, or an authoritative copy kept elsewhere. If the local disk is only a cache and the pipeline can recreate files safely, its required capacity may be smaller. If the disk is the only copy, include the full source and a backup plan rather than treating it as disposable cache space.
- Record file count and total bytes for each modality.
- Note compression format, shard size, and whether files are deduplicated.
- Identify data that must remain immutable for audit or reproducibility.
- Mark files that can be regenerated and the time and network cost of regenerating them.
Count every representation that coexists at peak use
Raw, cleaned, transformed, and augmented data
Many pipelines keep raw files while producing cleaned, normalized, tokenized, resized, or augmented versions. Add each representation that exists simultaneously. If a conversion is strictly in-place and the old file is deleted before the next stage starts, count only the peak overlap; if both versions remain for rollback, count both.
Build a peak-space timeline rather than adding every intermediate ever created. For example, a job might download raw images, write decoded shards, and then remove the raw shard after validation. The required space is the largest simultaneous combination, not the sum of all historical temporary files. Conversely, a failed run can leave partial shards, so include a cleanup allowance if your process does not remove them automatically.
Staging, cache, and shuffle space
Training frameworks often stage files locally, cache decoded samples, or create shuffled shards. Cache policy matters: a bounded cache has a known ceiling, while an unbounded cache can consume all free space. Document the configured cache limit and whether multiple workers each create a private copy. A single multi-worker node can therefore need more working space than the source dataset suggests.
Add checkpoints, logs, and model outputs
Checkpoint size is model- and workflow-specific
Checkpoints can contain weights, optimizer state, scheduler state, random-number state, and metadata. Their size depends on parameter count, precision, and the framework. Google Cloud gives an estimate of approximately 12–16 bytes per parameter for FP16 plus optimizer state, while noting that optimizer states and different precisions require a buffer. Treat that as a planning reference for the documented TPU workflow, not as a universal hard-drive multiplier. See Google Cloud’s storage best practices for AI/ML workloads on TPU VMs.
Estimate your own checkpoint by creating one representative save and measuring it. Multiply by the number retained at the same time, then add space for temporary checkpoint files written during an atomic save. If you keep “best,” “latest,” and periodic snapshots, count the retention policy rather than assuming only one file exists.
Logs, metrics, and generated outputs
Include TensorBoard or equivalent event files, profiling traces, evaluation predictions, sample generations, exported models, and packaging artifacts. These may be small compared with raw video but can accumulate across many experiments. Set a retention period and an automated deletion or archival rule; otherwise the disk estimate becomes obsolete after repeated runs.
Rank #2
- Easy-to-use desktop hard drive—simply plug in the power adapter and USB cable
- Fast file transfers with USB 3.3
- Drag-and-drop file saving right out of the box
- Automatic recognition of Windows and Mac computers for simple setup (Reformatting required for use with Time Machine)
- Enjoy peace of mind with the included limited warranty and Rescue Data Recovery Services
Include backups, versions, growth, and usable capacity
Retention and backup are separate requirements
A second copy is a durability decision, not a mathematical law that every dataset must be doubled. Ask the same questions AWS poses in its storage guidance: what capacity, durability, and availability do the training data and model artifacts require, and what are the retention and backup requirements? A local mirror, snapshots, or an object-storage backup each has different space and recovery characteristics.
Count retained revisions explicitly. If you keep three dated source releases, two processed releases, and monthly checkpoints, use the actual expected footprints. If older versions are archived elsewhere and removed locally, they do not belong in the local working-capacity figure, but the archive’s capacity and retrieval time still belong in the architecture plan.
Plan for growth and formatted capacity
Estimate incoming data per week or month and multiply by the retention period. Include expected changes in resolution, sampling rate, labeling, or shard format; a “same dataset” can grow when preprocessing becomes richer. Use the drive’s usable formatted capacity rather than its advertised decimal capacity, and leave working headroom for failed jobs, filesystem metadata, and maintenance. The reviewed guidance does not establish a universal free-space percentage, so choose a margin from measured peak usage and your recovery procedures.
A practical sizing worksheet
Fill in one row for each item and use the largest simultaneous total:
Recommended Free Tools
| Component | What to measure | When to include it |
|---|---|---|
| Retained source | Actual bytes of raw or authoritative files | Always, unless the local copy is safely reproducible cache |
| Processed or staged copies | Peak overlap of cleaned, tokenized, decoded, augmented, or shuffled data | Include every representation present together |
| Checkpoints and outputs | Measured representative checkpoint × retained count, plus exports and logs | Include temporary save space and experiment retention |
| Backups and versions | Expected local replicas, snapshots, and revisions | Include copies kept on the same storage system |
| Growth | New data and format changes over the planning period | Use the retention horizon, not a one-day snapshot |
| Working headroom | Peak scratch, cleanup failures, and maintenance allowance | Set from observed peaks; no universal percentage is established |
Add the measured values, then compare the total with usable formatted capacity. Recalculate whenever the model, worker count, preprocessing format, or retention policy changes.
What published AI-storage figures do—and do not—tell you
Official platform documentation illustrates how widely requirements vary. Google Cloud’s table for TPU virtual machines lists the following starting estimates:
| Workload | Published starting estimate | Scope |
|---|---|---|
| LLM pre-training | 2 TB per TPU | Google Cloud TPU VM guidance; year is not stated on the documentation page |
| Multimodal training | 12 TB per TPU | Google Cloud TPU VM guidance; year is not stated on the documentation page |
| Inference | 1 TB per TPU | Google Cloud TPU VM guidance; year is not stated on the documentation page |
These entries describe the documented Google Cloud environment. They are not a claim that a typical personal dataset needs 2 TB, 12 TB, or 1 TB, and they should not be converted into a recommendation for one consumer hard drive. A project using several TPUs, a different modality, or a different retention policy will have a different total. The source is Google Cloud’s TPU storage guidance.
Rank #3
- Easy-to-use desktop hard drive — simply plug in the power adapter and USB cable.Specific uses: Business, personal
- Fast file transfers with USB 3.0
- Drag-and-drop file saving right out of the box
- Automatic recognition of Windows and Mac computers for simple setup (reformatting required for use with Time Machine)
- Enjoy peace of mind with the included limited warranty and Rescue Data Recovery Services
NVIDIA’s B300 reference architecture gives a different sense of scale: high-resolution computer-vision datasets can exceed 30 TB, and one described case requires 4 GB/s per GPU read performance. Those are examples for a particular high-performance architecture, not ordinary desktop requirements. See NVIDIA’s High Performance Storage Architecture for DGX B300 systems.
Capacity and performance are different decisions
When a hard drive can be adequate
A hard disk may work for a small or moderate, mostly sequential workload when the dataset fits locally, one or a few workers read it, preprocessing is not continuously waiting on storage, and checkpoints can be written without disrupting training. A bounded cache or prefetch layer can hide some latency. Validate this with a representative run rather than assuming that capacity implies adequate speed.
When storage becomes the bottleneck
Random reads, many concurrent workers, frequent checkpoint writes, network sharing, and large decoded samples can expose seek latency and queueing. If GPUs sit idle while the input pipeline waits, more capacity will not fix the problem. Measure read throughput, write throughput, latency, queue depth, and utilization during a training step. NVIDIA’s guidance recommends characterizing the workload before finalizing storage, and its scaling discussion explains why access pattern and concurrency matter: Tips on Scaling Storage for AI Training and Inferencing.
Do not treat GPU VRAM, system RAM, and disk capacity as interchangeable. VRAM holds active tensors and batches; RAM may buffer or cache samples; the drive stores persistent data and artifacts. A model that fits in VRAM can still require terabytes of dataset and checkpoint storage, while a large disk cannot compensate for insufficient memory.
Choose an architecture, not just a drive size
| Option | Strengths | Questions to answer |
|---|---|---|
| Single local hard drive | Simple, portable, and economical for a bounded working set | Can it sustain the measured throughput, and what happens if it fails? |
| Local solid-state storage plus bulk disk | Fast cache or active-set storage with larger, slower capacity | Which shards and checkpoints need low latency, and how will they be staged? |
| Shared file storage or NAS | Several machines can access one retained dataset | What aggregate throughput, network bandwidth, concurrency, and permissions are required? |
| Cloud or object storage | Elastic capacity, lifecycle policies, and geographically separate copies | What are transfer time, access frequency, durability, retention, and recurring costs? |
AWS separates capacity, durability, availability, and retention as distinct design requirements. Use that framework when comparing local and shared systems rather than selecting on terabytes alone. Microsoft’s Design Training Data for AI Workloads on Azure also emphasizes organizing data for the training lifecycle, including retention and access considerations.
Step-by-step method for selecting capacity
- Inventory the source. Export a manifest and measure bytes by modality and partition. Record compression and the date.
- Map the pipeline. List every stage from download through training and deletion. Mark which representations overlap at peak.
- Measure one real artifact. Save a representative checkpoint, processed shard, cache segment, and output bundle. Record their sizes instead of guessing.
- Apply retention rules. Count the number of checkpoints, experiment runs, source revisions, and local backups kept concurrently.
- Project growth. Add expected new data and format changes over the period you plan to keep files locally.
- Add working headroom. Use observed peak scratch usage, incomplete-job cleanup, and maintenance needs to set a margin.
- Compare with usable capacity. Check formatted capacity and filesystem limits, not only the headline drive number.
- Test performance. Run the intended worker count and batch pattern while monitoring storage throughput, latency, queueing, and GPU utilization.
- Choose the architecture. If one drive cannot meet capacity, durability, or throughput requirements, split the active set and archive, or move to shared or cloud storage.
- Document recovery. Verify that a failed drive, corrupted shard, or interrupted checkpoint can be restored within the project’s acceptable time.
Worked planning examples
Compact text fine-tuning project
Suppose the source export is 180 GB. Tokenized shards occupy 220 GB and coexist with the raw export during validation. A run retains four 35 GB checkpoints, 20 GB of logs and evaluation outputs, and one 180 GB local backup. The measured peak scratch allowance is 60 GB, and six months of new data is expected to add 120 GB. The planning total is 180 + 220 + (4 × 35) + 20 + 180 + 60 + 120 = 900 GB. Because this is a peak calculation, deleting the raw export before tokenization completes would change the number; doing so may also remove rollback capability.
Image training with parallel workers
Assume 2 TB of raw images, 2.4 TB of resized shards, a 600 GB decoded cache, six 80 GB checkpoints, 2 TB of backup copies, 500 GB of projected growth, and 700 GB of measured scratch and failed-job allowance. The total is 2 + 2.4 + 0.6 + (6 × 0.08) + 2 + 0.5 + 0.7 = 8.68 TB. If four workers each create private decoded caches, the 600 GB cache estimate must be replaced by the measured aggregate peak. The capacity decision and the throughput test are separate: the disk may hold 8.68 TB yet still starve GPUs.
Rank #4
- Easily store and access 1TB to content on the go with the Seagate Portable Drive, a USB external hard drive.Specific uses: Personal
- Designed to work with Windows or Mac computers, this external hard drive makes backup a snap just drag and drop. Reformatting may be required for Mac
- To get set up, connect the portable hard drive to a computer for automatic recognition no software required
- This USB drive provides plug and play simplicity with the included 18 inch USB 3.0 cable
- The available storage capacity may vary.
Reduce storage without losing reproducibility
- Deduplicate before training. Remove byte-identical files and near-duplicates when your labeling policy permits it, and record what was removed.
- Choose an appropriate representation. Compression, aggregation, and efficient sharding can reduce volume, but the achievable reduction depends on the data and must be measured.
- Use lifecycle rules. Move immutable raw data or old checkpoints to archival storage and keep only active versions locally.
- Bound caches. Set explicit limits and monitor them so a prefetch process cannot consume the working headroom.
- Delete safely. Remove intermediates only after validation and backup checks; retain manifests, hashes, and transformation metadata so a result remains reproducible.
The UK Government’s Data and AI Ethics Framework is a useful reminder that retention and minimization are governance choices as well as storage optimizations. Do not keep sensitive or unnecessary copies merely because space is available.
Common sizing mistakes and fixes
Using a universal multiplier
Rules such as “buy twice the dataset” or “buy three times the dataset” ignore modality, cache behavior, checkpoint retention, and backup policy. Replace the multiplier with the worksheet and a measured peak.
Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchPC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Confusing a published example with a recommendation
The Google TPU figures and NVIDIA B300 example are scoped platform references. They do not establish a typical dataset size or an HDD speed threshold. Label every external number with its workload and environment.
Counting only the source files
A training run can need processed shards, temporary staging, checkpoints, logs, versions, and backups at the same time. Map the pipeline and measure overlap.
Ignoring concurrency
One reader streaming sequentially is not equivalent to many workers issuing reads and writes. Benchmark with the actual worker count, shard layout, batch size, and checkpoint schedule.
Filling the disk to its advertised limit
Formatted capacity is lower than the headline decimal figure, and a full filesystem leaves no room for cleanup or atomic writes. Keep a measured operating margin and alert before it is consumed.
Decision checklist
- Have you measured the retained source in bytes?
- Have you listed every representation that coexists at peak?
- Have you measured a real checkpoint and counted retained copies?
- Are logs, exports, backups, and revisions included?
- Is growth tied to a stated retention period?
- Is usable formatted capacity, rather than advertised capacity, being compared?
- Have you tested throughput and latency with the intended concurrency?
- Do durability, availability, and recovery requirements exceed what one drive can provide?
- Are compression, deduplication, archival, and deletion policies documented?
If the worksheet total fits comfortably within usable capacity and a representative run meets its I/O target, a local hard drive can be a reasonable part of the design. If capacity, recovery, concurrency, or throughput fails any test, use a tiered, shared, or cloud architecture instead of purchasing a larger drive blindly.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
