Hadoop Distributed File System (HDFS) is the primary distributed storage used by Hadoop applications. A cluster has a NameNode that manages filesystem metadata and DataNodes that store data. HDFS is designed for fault tolerance on commodity hardware and for high-throughput streaming access and batch processing, rather than low-latency workloads. It is tuned for files from gigabytes to terabytes, clusters with hundreds of nodes, and tens of millions of files in one instance. Written in Java, Hadoop is supported on major platforms, and HDFS is deployed as a self-hosted cluster. Administration options include shell-like commands, web servers for NameNodes and DataNodes, fsck diagnostics, safemode, and a cluster balancer. The permissions model shares much with POSIX and can optionally support POSIX ACLs. Hadoop supports operating-system identity and Kerberos authentication. Configured encryption protects data at rest and in transit. WebHDFS provides REST access, and listed storage integrations include Amazon S3, Azure Blob Storage, Azure Data Lake Storage, and Tencent COS. HDFS is free under Apache License 2.0.
Who it is for
HDFS suits teams needing distributed storage for high-throughput streaming and batch processing, especially for large files and clusters. It is not aimed at low-latency workloads.
What is good
- Designed for fault tolerance on commodity hardware
- Supports high-throughput streaming and batch processing
- Offers REST access through WebHDFS
- Configured encryption covers data at rest and in transit
- Free under Apache License 2.0
What to know first
- Designed for high throughput rather than low latency
- Tuned for gigabyte-to-terabyte files
- Support is through volunteer-maintained project mailing lists
Verdict
HDFS is intended for large-scale distributed storage and throughput-focused processing, with administration, authentication, and encryption options. Its workload fit is less suitable when low latency is the priority.
Hadoop Distributed File System plans and pricing
All plansCompared on distributed file system software
- Deployment model
- self_hosted
- Data replication
- Yes
- Multi-site support
- Yes
- File access protocols
- Java API, C API, REST API, HTTP browser, NFS gateway, FS shell
- Encryption at rest
- Yes
