A production AWS Glue pipeline can run faster when each change targets a measured bottleneck—not simply when it gets more workers. In a case study published September 26, 2026, Kiran Gunturu reports cutting end-to-end runtime by about 56% across four workstreams: orchestration, output publishing, ZIP compression, and SFTP runtime choice. Two separate investigations exposed a JDBC partitioning problem and a missing socket timeout, but neither should be mistaken for a measured optimization.
What changed—and how strong is the result?
Gunturu describes an Oracle-to-S3 ingestion pipeline built with AWS Glue and PySpark, followed by output packaging, encryption, and transfer to a downstream analytics platform. He reports an approximately 56% end-to-end runtime reduction after changes to four parts of the workflow. The figures below are his results in a production environment whose details were genericized; they are not AWS benchmarks or independently verified measurements. The AWS tuning guidance offers a general method—set a goal, measure, identify bottlenecks, reduce their impact, and measure again—but does not validate this case’s results: AWS Glue performance tuning.
As an Amazon Associate I earn from qualifying purchases.
| Workstream | Reported result | Change |
|---|---|---|
| Ingestion orchestration | About 13 minutes 55 seconds to about 8 minutes | Raised Step Functions Map concurrency from 30 to 50 for 63 tables |
| Output publishing | About 25 minutes to 7 minutes | Wrote final output once and parallelized reconciliation |
| ZIP compression | About 9 minutes to 4 minutes | Threaded compression and reduced DEFLATE level to 1 |
| SFTP transfer | About 11 MB/s retained; estimated annual cost about $63 to $11 | Moved the single-stream transfer from Glue Spark to Glue Python Shell |
These are separate reported changes, not a guarantee that another pipeline will achieve the same reductions. The useful question is what consumed time in each stage, and whether the proposed change addresses that specific cost.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
How did orchestration concurrency affect ingestion?
The pipeline ran 63 tables through a Step Functions Map. At concurrency 30, Gunturu says the work completed in three waves and took about 13 minutes 55 seconds wall-clock. Increasing concurrency to 50 moved the run to two waves and about eight minutes. This is a wave-boundary effect in that workload: raising concurrency does not automatically make a job faster if the work is bottlenecked elsewhere or the new setting creates resource contention.
#1 Best Overall
When table sizes vary, task order can also affect the final completion time. Gunturu’s practical suggestion was to start larger tables early, reducing the chance that heavy work is left in the final wave. Check both the number of waves and whether a few slow tables determine when the Map finishes.
Why did publishing output take less time after a single write?
The original publish stage wrote output, then reread and rewrote it in separate passes for renaming, newline handling, and standardization. Those passes meant processing the same bytes repeatedly. Gunturu reports that writing final bytes once from executors and parallelizing reconciliation reduced the stage from about 25 minutes to seven.
The broader lesson is to look for full-data passes that can be combined with the producing stage. This is especially worth checking when a job’s transformations are modest but its output is large: moving or rewriting all the data can cost more than the transformation itself.
The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →When does parallel compression help—and when can encryption fail?
The outbound packaging job zipped files, GPG-encrypted each ZIP, and uploaded the result with SSE-KMS. Although it ran on AWS Glue Spark with six DPUs on G.2X, the ZIP and encryption work was serial on the driver, leaving Spark executors mostly idle. The reported results show why compression and encryption need separate diagnosis.
Thread ZIP work, while accounting for larger files
Gunturu reports that threading ZIP compression and lowering DEFLATE to level 1 cut compression time from about nine minutes to four, with slightly larger archives. Lower compression effort can trade CPU time for storage and transfer size. Whether that is worthwhile depends on downstream storage limits and available network bandwidth; the reported time is specific to this workload.
Avoid recompressing ZIP files with GPG
GPG’s default compression was recompressing already-compressed ZIP input. In one reported batch, the size rose from 5,246 MB to 5,312 MB while consuming extra CPU. The author says using --compress-algo none avoided that redundant compression. This setting addresses the unnecessary second compression pass; it does not make encryption itself free.
Do not parallelize encryption by multiplying driver memory use
Gunturu says parallel encryption caused driver out-of-memory failures because each multi-gigabyte ZIP and an armored copy were held in memory. His proposed safer design is to stream files, use isolated GPG home directories, and produce binary rather than armored output. He presents this as a design proposal, not a measured production result. The distinction matters: adding parallel tasks is only useful when the work can be distributed without exceeding the memory available to each process.
Why did a Python Shell job suit the SFTP transfer?
A separate job sent large files to an external SFTP endpoint. Gunturu characterizes the transfer as bandwidth-bound: roughly 5 GB files moved at about 11 MB/s, and increasing connector concurrency did not improve throughput. The job used Glue Spark with six DPUs on G.2X, but this single-stream I/O task did not make useful use of Spark’s distributed compute.
He moved the transfer to a one-DPU Glue Python Shell job and reports retaining the observed throughput. At the stated rate of $0.44 per DPU-hour, his estimated annual cost fell from about $63 to $11. Those cost estimates depend on his setup and assumptions; they are not general Glue pricing or a cost guarantee. The decision principle is to select a runtime that matches the work: distributed Spark computation may add cost without improving a transfer capped by one stream or the network path.
What did the JDBC partitioning investigation uncover?
One ingestion discrepancy led Gunturu to investigate a much slower production read. He reports that about 8.9 million rows took eight minutes in a smaller test, while a 25-million-row production read took 2 hours 45 minutes and continued climbing. He initially suspected skew, but says a bucket-distribution query showed a near-uniform distribution.
His reported diagnosis was a computed-expression partition column that Oracle could not index. Each JDBC connection then scanned the full time window to find its own slice, rather than efficiently pruning the data. The proposed remedy was to partition on a prunable column, but Gunturu explicitly says he had not implemented and measured that fix. Treat it as a plausible diagnosis in this case, not a demonstrated speedup or a universal explanation for slow JDBC reads.
AWS documentation provides relevant general context: pushdown applies filters closer to the data source, potentially reducing data scanned and transferred; JDBC parallel reads require suitable partitioning and query configuration. AWS’s guidance discusses partition pruning and parallelism, while its JDBC documentation explains custom SQL and parallel sample queries. These describe Glue capabilities and trade-offs, not the Oracle-specific behavior reported here:
- AWS Glue pushdown
- AWS Prescriptive Guidance on parallelizing tasks
- AWS Big Data Blog: optimize memory management in AWS Glue
- AWS Glue JDBC connections
Before increasing JDBC parallelism, verify that the partition key and source query let the database avoid repeatedly scanning the same range. More connections can increase work rather than reduce elapsed time if each connection performs a full scan.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.How can a missing socket timeout hold up a Spark stage?
Gunturu describes a run that hung for 2 hours 42 minutes, then completed in 14 minutes on an identical rerun. He attributes the long run to a JDBC socket read without a timeout: one connection silently died and its Spark task never returned, preventing the stage from completing. This is his diagnosis of that incident, not a finding established for other drivers or environments.
His recommended safeguards are to configure query and Oracle JDBC read/connect timeouts and enable Spark speculation so a slow or stuck read can be retried. Validate the precise property names and behavior against the Oracle JDBC driver and Glue/Spark versions in use. One configuration trap he highlights is that oracle.net.CONNECT_TIMEOUT is in milliseconds when set as a connection property, but seconds when supplied bare in the URL. Confirm the units for the exact configuration form you use; a timeout with the wrong unit may expire far sooner or later than intended.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Scan for outdated or missing drivers - takes under a minuteDriver Scan →How to apply the lessons to your own Glue pipeline
- Set a runtime goal and measure stages. Record elapsed time for orchestration, source reads, transformations, output writes, compression, encryption, and transfer. AWS recommends measuring before changing tuning parameters and repeating the measure-and-adjust cycle in its Glue performance guidance.
- Locate the limiting resource. Determine whether the stage is constrained by task waves or skew, repeated bytes read and written, CPU, driver memory, source-database scans, or network throughput. A change that helps one constraint can be irrelevant to another.
- Make one targeted change. For orchestration, inspect wave count and task duration. For publishing, eliminate avoidable full-data passes. For packaging, separate ZIP compression from encryption. For transfers, check whether additional workers can actually increase the available stream or bandwidth.
- Measure the same outcome again. Compare wall-clock time and resource use under comparable conditions, and check trade-offs such as archive size, database load, memory failures, and cost. Keep proposed fixes separate from changes that have been implemented and measured.
Gunturu’s summary of the approach is apt: “Mostly it’s proving what the bottleneck actually is before changing anything.”
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
