If an AWS Glue job read only half a CSV from S3, S3’s strong consistency is unlikely to be the explanation by itself. Strong consistency governs which complete object S3 returns; it does not guarantee that Glue selected the intended object, parsed every CSV record correctly, or kept every row through later transformations. Start by finding the earliest point in the job where the expected and actual counts differ.
What S3 strong consistency guarantees—and what it doesn’t
Amazon S3 provides strong read-after-write and LIST consistency: after a successful write, subsequent reads and listings reflect it. For an update to a single key, concurrent reads return either the old complete object or the new complete object—not a partially updated object. AWS describes this as atomic updates to a single key.
As an Amazon Associate I earn from qualifying purchases.
That guarantee is about the object S3 serves. It does not establish that a Glue job read the intended key or version, interpreted the file’s CSV structure as intended, or retained every record during processing. A “half-file” symptom can originate at any of those layers.
Free tools Windows power users keep installed
One-click scans. No signup required.
Trace the data in order, starting with the source
-
Verify the object and source selection
Record the bucket, key or prefix, AWS Region, object size, last-modified time, and version ID if versioning is enabled. Determine whether the job reads one key or a prefix that can match multiple objects. Check the Glue source path, recursive-read behavior, partition predicates, and any file or byte limits. Glue source options and job arguments can affect what input is selected.
-
Compare the exact object with Glue’s initial record count
Inspect the exact object version the run was meant to consume. Compare its byte size and, where available, checksum with the producer’s expected output. Count logical CSV records with a CSV-aware parser; counting newline characters is not reliable when quoted fields can contain embedded newlines. Then compare the expected count with the count immediately after Glue reads the source.
-
Check the CSV dialect and record boundaries
Confirm the delimiter, quote character, escape behavior, header handling, encoding, compression, and whether any quoted field spans multiple lines. AWS documents that the Glue CSV reader’s
multilineoption must be set totruewhen a record spans lines. Its default isfalse, which allows more aggressive file splitting. Review the CSV reader options for your Glue version. -
Identify which CSV reader Glue used
Record the Glue version and whether the optimized CSV reader is enabled. AWS documents that the optimized reader does not support multiline CSV and has limitations for other options; depending on the configuration, Glue can fall back to the row-based reader. Compare reader modes using a controlled sample while keeping the input and CSV settings constant. AWS lists the optimized reader’s supported options and limitations.
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Scan for outdated or missing drivers - takes under a minute3Repair Windows errors before they cause bigger problemsSpecial offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy. -
Locate the first processing stage that loses rows
Capture counts immediately after the source read and after each filter, mapping, join, deduplication, and output write. If the first count is already low, focus on source selection and parsing. If it falls later, inspect predicates, schema conversions, malformed-record handling, and join keys. Glue runs extraction, transformation, and loading with Spark, so stage-by-stage counts help distinguish a read issue from a later transformation. AWS explains Glue’s ETL model.
Rank #3
-
Check write timing separately
If the producer and Glue run concurrently, verify that the producer received success for its write—or for multipart upload completion—before the Glue read began. S3 exposes a completed multipart object after its parts have been uploaded and assembled, rather than serving a half-finished upload as a completed partial object. Identify the exact key and version the job accessed. AWS explains multipart upload completion.
Use the evidence to narrow the cause
| Where the first count diverges | What to investigate | Evidence to collect |
|---|---|---|
| Before or at source selection | Wrong key or prefix, unexpected extra or missing objects, recursive settings, partition predicates, or source bounds | Configured source path and options; matching object keys, versions, and sizes |
| Immediately after parsing | Multiline records, delimiter or quoting mismatch, header handling, encoding, compression, or reader limitations | CSV-aware count and sample; dialect settings; Glue version and reader mode |
| During transformation | Filters, schema conversion, malformed-record handling, joins, or deduplication | Counts before and after each stage; relevant predicates, schemas, and join keys |
| At output | Write logic or expectations about output files and record counts | Final pre-write count and output count |
Why multipart upload does not normally explain a half-object
A multipart upload is not visible as a completed object until the upload is completed and its parts are assembled. If Glue starts before that success, investigate whether it read an older existing version or a different object matching the source prefix; do not assume S3 exposed a partially assembled replacement. AWS’s multipart upload documentation describes the completion step.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.What to capture before changing the job
- The Glue job script, Glue version, run ID, and relevant logs.
- The configured S3 path and source options, including recursion, predicates, bounds, and bookmark behavior if applicable.
- The exact key and version, object size, last-modified time, and checksum if available.
- A representative CSV sample showing headers, quoting, and any multiline fields, with sensitive data removed.
- Expected and observed logical record counts at the source, each transformation stage, and output.
- Whether the writer had completed successfully before the Glue run read the object.
Without those details, the symptom alone cannot identify a specific cause or demonstrate a Glue bug. The most useful next step is to establish the earliest stage where the count diverges.
The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Quick Recap
Best Value
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
