Moving an AWS Glue job to OCI Data Flow is two migrations at once. The Spark application has to run on Data Flow, and every Glue-provided service around it has to be rebuilt: job state such as bookmarks, connections and network access, IAM and secrets, dependency packaging, scheduling, and monitoring. The Spark transformation logic may port with little change. The Glue-specific pieces are where the work concentrates, and a run that starts and finishes is not proof of parity. Parity is shown by comparing outputs on representative workloads.
Oracle publishes a migration tutorial for existing Spark applications, and AWS documents Glue features such as job bookmarks and VPC connections that do not carry over automatically. The steps below follow the order most migrations need: map what Glue does for the job, build a minimal Data Flow baseline, replace each integration, move data and network access, size compute, validate, and cut over.
As an Amazon Associate I earn from qualifying purchases.
1. Inventory what the Glue job actually depends on
Start with the job definition, not the script. Record the job type (batch Spark or streaming ETL), the Glue version, the script entry point, job parameters, third-party libraries, every source and sink with its format, catalog reference, authentication method and network path, whether the job is incremental, its triggers and retry settings, its output mode, and where engineers read its logs today.
AWS’s guidance on migrating Apache Spark programs into Glue names versions, dependencies, credentials, Spark configuration and custom arguments as the areas that need attention. Read that list in reverse: those are the same categories that need an explicit answer when you leave Glue.
#1 Best Overall
Then search the code for Glue-specific markers: GlueContext, DynamicFrames, Data Catalog references, Glue connection names, job.init, job.commit, transformation_ctx, bookmark arguments, Glue-only transforms, and AWS SDK calls. Anything that matches belongs in the Glue-managed rows of the table below. The remaining transformation code is the part that can move with the least change.
2. Glue elements and their Data Flow counterparts
| Glue element | What to record | Data Flow side to design |
|---|---|---|
| Job type | Batch Spark or streaming ETL. AWS notes that some Spark job features do not apply to streaming ETL jobs. | Confirm streaming support for your configuration; not stated in the Oracle documentation cited in this article. |
| Glue and Spark/Python versions | Glue version and the Spark and Python versions it provides | Choose a Data Flow Spark runtime and compare versions; no general version mapping is documented. |
| GlueContext, DynamicFrames, Data Catalog | Every call and table reference | Spark DataFrames where you convert, plus a metastore decision (see below) |
Bookmarks (job.init, job.commit, transformation_ctx) |
Key or source-selection logic and restart expectations | An explicit replacement design; bookmark state is not documented as transferable. |
| Glue connections (JDBC, VPC) | Endpoint, authentication, subnet and security groups | OCI network path, private endpoints, and credential management |
| IAM role and secrets | What the job role can read and write, and where secrets come from | Run-user permissions for IAM-compatible services; separate credential or key management for others |
| Job parameters | Each argument the script receives | Application or run arguments; environment variables are not available |
| Triggers, workflows, retries, alerts | Schedule, event source, retry and alert settings | Your orchestrator; no one-to-one trigger conversion |
| Workers and worker count | Worker type, count, and measured runtime | Driver and executor shapes chosen by benchmark; no worker-count conversion is documented |
| Logs and metrics | Where engineers read them today | Data Flow run output, run statistics, Spark UI, and OCI Logging |
3. Build a minimal Data Flow baseline first
-
Choose a Data Flow Spark runtime and compare its Spark and Python versions with the Glue job’s. Oracle’s migration tutorial notes that Data Flow creates the Spark session before your application starts, so any session-level configuration in your code must be checked rather than assumed.
-
Test every custom Spark property against Data Flow’s supported settings. The same tutorial identifies Spark properties that cannot be set or overridden, so a setting that worked on Glue may not apply on Data Flow.
Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy. -
Remove environment-variable dependencies. Oracle’s tutorial states: “You can’t set environment variables in Data Flow jobs.” Move each value into a command-line argument or into configuration the application reads at startup.
-
Upload the application artifact to OCI Object Storage and confirm that the run principal can read the application and every asset it references. Oracle’s import guide for Spark applications describes this flow.
-
Package dependencies by language. For Java and Scala, bundle dependencies into an uber (assembly) JAR, which Oracle recommends, and watch for runtime library conflicts, applying shading where Oracle’s guidance calls for it. For Python, use Oracle’s documented mechanism for third-party packages. Keep the entry-point script as its own file. Do not upload a zipped application package as though Data Flow will run it directly.
-
Run the minimal application before you integrate the migrated code: one read from Object Storage, a trivial transformation, one write, and a check of the output and logs. This separates packaging and permission failures from logic failures.
The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
4. Replace each Glue integration
DynamicFrames and the Glue Data Catalog
Find every DynamicFrame and decide whether it becomes a Spark DataFrame. Each conversion changes how schema and null handling are expressed, so compare output after each conversion rather than only at the end.
Catalog metadata does not become OCI resources on its own. Data Flow can use a Hive-compatible metastore, and Oracle’s setup documentation identifies the managed and external table storage buckets involved. Decide where table definitions will live before you rewrite any read that refers to a catalog name.
Connections and secrets
A Glue connection bundles endpoint, authentication and network settings. On OCI there is no object to copy across, so translate each endpoint, authentication method and network path explicitly. In Data Flow, runs use the permissions of the user who starts them for IAM-compatible services. For services that are not IAM-compatible, Oracle points to credential and key management, as described in its security documentation. Keep secrets out of code and out of application arguments.
Rank #3
Bookmarks and restart behavior
Bookmarks are state that Glue manages. AWS documents that bookmark operation depends on job.init and job.commit being called and on a consistent transformation_ctx, as explained in its bookmark documentation. Two rules matter for migration: user-defined JDBC bookmark keys must be strictly monotonic, and changing a source or its transformation context can invalidate prior bookmark behavior. The vendor documentation cited here does not describe moving bookmark state into Data Flow, so plan it as an explicit migration or a rebuild.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallChoose the replacement before cutover. Common designs are a watermark kept in a table you control, an explicit key or date range passed as a run argument, or a deliberate full reload. Whichever you choose, test a rerun, a restart after a mid-run failure, late-arriving records, and duplicate output.
Triggers, workflows and retries
List every Glue trigger, workflow, schedule, event-driven start, retry setting and alert. These currently run inside Glue. On OCI they run wherever you orchestrate the pipeline. The vendor documentation cited here does not identify your orchestrator, so do not assume a one-to-one trigger conversion. Recreate ordering dependencies and retry counts explicitly, then force a failure in a test run to confirm that the retry and alert behavior is what you intended.
5. Move data and rebuild network access
Data Flow is optimized for OCI Object Storage. Oracle says access is highly performant when the application and the data are in the same OCI region, so co-locate them first and treat cross-region reads as a separate performance question. Data Flow can also read Spark-supported sources such as relational databases. For on-premises systems, Oracle’s import guide describes private endpoint access using an existing FastConnect configuration. For each source and sink, confirm the connector and driver, credentials, routing, firewall rules, DNS and region.
The AWS side is more specific than many teams expect. Glue jobs that reach private JDBC sources use elastic network interfaces in the selected subnet, and every JDBC store the job accesses must be reachable from that subnet, as described in AWS’s network access documentation. Write down the actual AWS path (subnet, security groups, routes and name resolution), then design the OCI path from that record. VPC security groups and OCI network policy are not interchangeable: a rule copied across does not carry the same meaning.
Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Repair Windows errors before they cause bigger problems3Scan for outdated or missing drivers - takes under a minute6. Translate arguments and size compute from measurements
Glue passes job parameters to the script as ---prefixed arguments, as described in AWS’s job parameters documentation. Give each one a home in the Data Flow run definition.
Re-select driver and executor shapes and counts. Do not convert Glue worker counts one-for-one: the vendor documentation gives no general conversion, because the two services size compute in different units. Start from the Glue run’s measured runtime and shuffle volume, then benchmark representative inputs, including skewed keys and the real output pattern.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.7. Validate output and operations
Run the job on representative small, typical and peak inputs, and compare source and target results on:
- row counts and key coverage
- schema, data types and null handling
- partition counts and file layout
- ordering that downstream consumers rely on
- incremental boundaries, meaning the exact range each run covers
- runtime and resource use, measured rather than assumed
Run output, statistics and logs
For each test run, check the run output, run statistics, Spark UI access, and driver and executor logs, as listed in Oracle’s run applications documentation. If logs must be centralized, configure OCI Logging policies and destinations; Oracle’s application logging guide covers that setup. Set this up before the first parallel run rather than after a failure.
Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Repair Windows errors before they cause bigger problemsFix Now →Maximum run duration
Oracle documents automatic stopping for long-running batch jobs. The maximum period depends on the authentication mode: runs authenticated with delegation tokens and runs using resource principals have different limits. Confirm the current figure for your chosen mode in Oracle’s run applications documentation before cutover, and compare it with the longest run you have measured, including backfills.
Best Value
8. Cut over in controlled stages
-
Run the Glue job and the Data Flow job in parallel on bounded inputs where the data allows it.
-
Reconcile outputs, watermarks and checkpoints before you move any schedule.
-
Write rollback conditions before the switch. Useful triggers include a reconciliation mismatch in counts or keys, a missed or duplicated input window, or a run that exceeds its duration budget.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy. -
Name an owner for each checkpoint, and decide the first-run policy: whether the first Data Flow run backfills from a stated boundary or starts from the current position.
-
Move the schedule only after reconciliation passes for the agreed window. Keep the Glue job disabled rather than deleted until the rollback window closes.
The right depth of parallel running depends on data volume, how mutable the source is, which downstream consumers read the output, and how much duplicate or missing data the business will tolerate. Those inputs belong to the team that owns the pipeline.
Quick Recap
What the documentation settles and what it does not
- Vendor documentation describes what each service can do. It does not establish that a particular Glue job will run unchanged on Data Flow. Its definitions, scripts, versions, data formats, region, and IAM and network model decide that.
- No general cost or performance winner is established, and the cited documentation gives no migration-time or savings figures. Estimate effort from your own inventory, and measure runtime and cost on your workloads.
- Oracle’s and AWS’s pages describe current behavior and can change. Check Spark property support, runtime versions and networking options for your target region and tenancy at the time you implement.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
