Connect Apache Ozone to Hadoop or Spark through one of two routes: use Ozone’s Hadoop filesystem connector with ofs:// for direct filesystem access, or use the Ozone S3 Gateway with Hadoop’s S3A connector and keep s3a:// paths in applications already built for S3. The right choice depends on the interface your application uses, the runtime libraries available to its clients, and how the cluster authenticates and deploys jobs.
Choose between ofs:// and s3a://
Both routes let Hadoop ecosystem applications work with Ozone, but they use different client interfaces and require different setup.
As an Amazon Associate I earn from qualifying purchases.
| Route | Use it when | Client-side requirements |
|---|---|---|
ofs:// |
You want Ozone’s Hadoop-compatible filesystem interface, including a rooted view across volumes and buckets. | Ozone filesystem client JAR and filesystem configuration. |
s3a:// |
Your application already uses the Hadoop S3A connector or an S3-compatible storage interface, and the Ozone S3 Gateway is available. | Compatible hadoop-aws dependency, gateway endpoint, path-style access, and Ozone S3 credentials. |
Ozone’s OFS filesystem documentation describes the direct filesystem route. Its S3A integration guide describes access through the gateway. The S3 route is useful when porting existing cloud-oriented Spark jobs: the Spark guide says S3A can be used without application code changes, provided the application already uses compatible S3A paths and APIs.
Connect with the Hadoop-compatible ofs:// filesystem
Use ofs:// when clients should access Ozone through Hadoop’s filesystem interface. The URI takes this form:
#1 Best Overall
ofs://<om-service-id>/<volume>/<bucket>/path/to/key
Here, <om-service-id> is the Ozone Manager service ID for the environment. Configure the filesystem implementation as org.apache.hadoop.fs.ozone.RootedOzoneFileSystem and make the ozone-filesystem-hadoop3 JAR available on the Hadoop client classpath.
Configure a Hadoop client
In the Hadoop configuration, set:
fs.ofs.impl=org.apache.hadoop.fs.ozone.RootedOzoneFileSystem- If Ozone should be the default filesystem, set
fs.defaultFSto the appropriate Ozone Manager URI for your cluster.
For commands and applications that accept a filesystem URI, pass an ofs:// path naming the service ID, volume, bucket, and key. The URI is the Hadoop filesystem path; it is not a separate Spark-specific API.
Understand ofs versus o3fs
ofs presents a rooted view that can address multiple Ozone volumes and buckets. o3fs is limited to one bucket. Ozone continues to support o3fs:// for legacy compatibility, but its Spark documentation recommends ofs:// for new deployments.
The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Connect through the Ozone S3 Gateway with S3A
Choose this route when existing Hadoop or Spark applications already use S3A. The Ozone S3 Gateway exposes an S3-compatible REST interface, while Hadoop S3A translates S3 operations into the Hadoop filesystem interface. Apache Ozone names Hive, Impala, and Spark among tools that can use Ozone through S3A.
Set up the Hadoop S3A client
- Match the Hadoop libraries. Add
hadoop-awsto the client and use the same version ashadoop-common, as required by the Ozone S3A guide. - Set the gateway endpoint. Configure
fs.s3a.endpointto the endpoint reachable from the application’s runtime. - Set a logical region. Configure
fs.s3a.endpoint.regionto a valid-looking region value; the guide’s example isus-east-1. - Enable path-style addressing. Set
fs.s3a.path.style.access=true, because the Ozone gateway uses path-style URLs. - Configure credentials. Supply the Ozone S3 access and secret keys, or use the AWS environment variables documented by Ozone. When security is enabled, the guide instructs users to obtain a key and secret with
ozone s3 getsecretusing Kerberos authentication. - Apply compatibility settings if needed. The guide identifies
fs.s3a.bucket.probe=0andfs.s3a.change.detection.mode=noneas settings to consider for Ozone.
Use s3a:// paths in the Hadoop or Spark job. The Ozone S3A guide also documents Hadoop filesystem commands, local-to-Ozone copies, and DistCp transfers between HDFS and Ozone, so this route can support data movement as well as application reads and writes.
Make the client libraries and configuration available to Spark
Spark must be able to load the selected filesystem implementation and its dependencies wherever file access occurs. Supply the Ozone client JAR and configuration to both the driver and executors; putting a JAR only on the submit host does not establish that the cluster runtime can load it.
Use ofs:// from Spark
For the direct route, Spark’s guide shows explicitly setting:
Free tools Windows power users keep installed
One-click scans. No signup required.
spark.hadoop.fs.ofs.impl=org.apache.hadoop.fs.ozone.RootedOzoneFileSystem
When required by the deployment, also provide the Ozone filesystem client JAR and the relevant Hadoop configuration to the driver and executors. Spark jobs can then use standard read and write APIs with an ofs:// URI, such as reading CSV input or writing Parquet output.
Check Spark, Hadoop, and Ozone versions together
Apache Ozone’s Spark integration guide says its examples were tested with Spark 3.5.x and Ozone 2.2.0. It also flags a compatibility issue: Ozone 2.1.0 and later require Hadoop 3.4.x classes for several classes removed from Ozone’s bundled copies, while Spark 3.5.x ships with Hadoop 3.3.4. Treat this as a version checkpoint, not a guarantee that every distribution or combination works. Verify the exact Spark, Hadoop, and Ozone versions and follow the relevant release guidance before rollout; do not casually replace a cluster’s bundled Hadoop libraries.
Kerberos on YARN
On a Kerberos-enabled cluster, Spark needs permission to obtain Ozone delegation tokens, and the submitting user needs a valid Kerberos ticket. The guide gives this YARN setting as an example:
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
spark.kerberos.access.hadoopFileSystems=ofs://ozone1/
Use the Ozone Manager service ID for the target environment rather than copying ozone1 literally.
Spark on Kubernetes
One documented deployment approach is to include the Ozone filesystem client JAR, any required Hadoop compatibility JAR, and core-site.xml in a custom Spark image. The configuration needs the filesystem implementation and Ozone Manager address, including settings such as fs.ofs.impl and ozone.om.address. Adapt addresses, versions, image locations, and security settings to the cluster, and ensure both the driver and executor pods receive what they need.
Validate the connection before rolling it out
- Confirm whether the application will use
ofs://ors3a://, and that its paths and APIs match the selected route. - Check that the required Ozone filesystem JAR, or the compatible
hadoop-awsdependency, is available to the actual client runtime. - For S3A, verify the endpoint, path-style setting, region value, and credentials; for direct filesystem access, verify the Ozone Manager service ID and filesystem configuration.
- For Spark, check the versions of Spark, Hadoop, and Ozone together, then confirm that configuration and dependencies reach the driver and executors.
- For Kerberos, confirm the submitting identity has a valid ticket and that Spark is configured to obtain Ozone delegation tokens.
These checks establish configuration readiness, not universal feature parity or performance. The official integration guidance does not establish that every Hadoop/Spark operation behaves identically to HDFS or AWS S3, or that one route is faster in every environment.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
