Hadoop Distributed File System

Overview

Hadoop Distributed File System (HDFS) provides distributed storage for Hadoop applications, separating filesystem metadata from data across a cluster. It is a strong fit for teams running large, throughput-oriented workloads on infrastructure they operate themselves. Its central trade-off is clear: HDFS is built for scale and resilience, not low-latency access.

Key features

A NameNode manages filesystem metadata while DataNodes store the data. HDFS is tuned for files from gigabytes to terabytes, clusters of hundreds of nodes, and tens of millions of files in one instance. That makes it compelling for large batch jobs and streaming reads, but a poor match for applications dominated by small files or rapid individual responses.

Fault tolerance and replication are designed to keep storage resilient on low-cost commodity hardware. Operators can use shell-like commands, NameNode and DataNode web servers, fsck diagnostics, safemode, and a cluster balancer to administer and inspect the system. These are useful controls, though the self-hosted deployment means teams must take responsibility for operating the cluster.

Permissions follow much of the POSIX model, with optional POSIX access control lists. Hadoop supports operating-system identity or Kerberos authentication. When configured, encryption zones provide transparent encryption at rest and in transit through the Hadoop Key Management Server. The WebHDFS REST API supports the complete FileSystem and FileContext interfaces; access options also include Java and C APIs, HTTP browser access, an NFS gateway, and the FS shell.

HDFS is written in Java, supported on major platforms, and can integrate with Amazon S3, Azure Blob Storage, Azure Data Lake Storage, Tencent COS, and other Hadoop-compatible filesystems. Apache distributes source and binary tarballs and recommends checking releases with GPG signatures or SHA-512 checksums. Support comes through project mailing lists maintained by non-paid volunteers, so organizations needing guaranteed response times should plan accordingly.

Pricing

Apache Hadoop (open source): free, billed no charge. The plan is licensed under Apache License 2.0 and is for self-hosted cluster deployment. It suits teams prepared to deploy and operate their own infrastructure; there are no paid tiers or seat and quota limits in this plan.

Platforms

HDFS supports API, Linux, macOS, self-hosted, and Windows environments. The Java and C APIs, REST, HTTP browser, NFS gateway, and FS shell provide several routes to file access, while multi-site support can serve deployments spanning sites.

Who it's for

Choose HDFS when your work involves large files, batch processing, or high-throughput streaming across a sizable cluster, and you have the operational capacity to manage distributed infrastructure. Look elsewhere for latency-sensitive applications or teams that need a managed service or guaranteed support response.

Pros and cons

• Pro: Built for large files and clusters of hundreds of nodes, making it well suited to substantial processing workloads.
• Pro: Replication and fault tolerance target resilience on commodity hardware.
• Pro: Kerberos, optional POSIX access control lists, and configurable encryption cover important identity and data-protection needs.
• Con: The throughput-first design sacrifices fit for low-latency workloads.
• Con: Self-hosting places cluster operations on the adopting team, while project mailing-list support has no guaranteed response time.

Alternatives

For a broader comparison, see Distributed File System Software.

• MooseFS is worth comparing if its freemium plans suit your deployment; its Community plan is free and has a single active master with single-parity erasure coding.
• SeaweedFS offers a free Community support plan for any cluster size, with GitHub issues, discussions, and public documentation but no guaranteed response time.
• Ceph is another free open-source distributed storage option, with block storage requiring a running Ceph cluster.
• JuiceFS offers a self-hosted Community Edition with community support via GitHub and Discord.
• OpenAFS is a free, open-source distributed filesystem to consider alongside HDFS.
• OrangeFS has a free open-source downloadable release.
• Quobyte may suit teams seeking a free edition capped at 150 TB of storage.
• XtreemFS is a free distributed filesystem under a BSD license.

Verdict

HDFS is a sound choice for teams that need resilient, high-throughput storage for large-scale Hadoop workloads and can run a self-hosted cluster. Its scale-oriented design and no-charge Apache-licensed deployment are the main reasons to choose it; latency-sensitive workloads and teams needing guaranteed support are reasons to look elsewhere.