October DealsAmazon USOctober deal check: compare before you payAmazon US: current deals, useful picks and tech finds.Check DealsSlow PC?RecommendedPC slow today? Run a repair scan before it gets worseResolve common Windows issues and optimize system performance.Scan NowOctober DealsAmazon USDeal season is back - check today's better picksAmazon US: current deals, useful picks and tech finds.See Picks×
Skip to content
World desk5 min

Bounded-Memory Streaming in Lioran S3: Reading Gigabytes Without Buffering Gigabytes

Lioran S3’s article describes processing objects through reusable chunks rather than object-sized buffers. Learn what that bounds, how concurrency changes memory use, and what remains unverified.
Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Lioran S3’s article describes object uploads as using a reusable, fixed-size buffer rather than loading an entire object into a byte vector. In its example, a normal PUT uses a default 256 KiB streaming buffer, so a 10 MiB object and a 100 GiB object can use the same primary buffer size. That bounds one part of memory use—not the total memory of a process handling many simultaneous streams or other buffered work.

The implementation details below are claims in Swaraj Puppalwar’s October 1, 2026 article, surfaced in its published search-result text. The DEV page could not be retrieved for independent inspection, and no source-code review or benchmark confirms the claims. Read the Lioran S3 article.

As an Amazon Associate I earn from qualifying purchases.

How the described upload path handles an object

The article says a normal object PUT passes an asynchronous reader and a staging file to a stream_to_staging helper, along with a durability mode and chunk size. The helper selects an effective chunk size, allocates one buffer with vec![0u8; effective_chunk_size], and reuses that buffer as it reads the input.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
  1. Read into the reusable buffer. A read returning n == 0 marks end of file.
  2. Hash only the bytes just read, &buffer[..n], using incremental SHA-256.
  3. Write that same slice to the staging file and add its length to a cumulative byte count.
  4. After EOF, flush the staging file.

This pattern can process a large object without retaining all of its contents in one object-sized heap allocation. The article states the invariant categorically: “Object uploads and downloads strictly utilize fixed-size buffer chunks. Under no circumstances are entire payloads buffered into heap memory.” That is the article’s own implementation claim, not an independently verified guarantee about every code path or the whole process’s memory use.

#1 Best Overall
Sandisk 2TB Extreme Portable SSD, Up to 1050MB/s, USB-C, USB 3.2 Gen 2, IP65 Water and Dust Resistance, Updated Firmware, External Solid State Drive, SDSSDE61-2T00-G25
  • Get NVMe solid state performance with up to 1050MB/s read and 1000MB/s write speeds in a portable, high-capacity drive(1) (Based on internal testing; performance may be lower depending on host device & other factors. 1MB=1,000,000 bytes.)
  • Up to 3-meter drop protection and IP65 water and dust resistance mean this tough drive can take a beating(3) (Previously rated for 2-meter drop protection and IP55 rating. Now qualified for the higher, stated specs.)
  • Use the handy carabiner loop to secure it to your belt loop or backpack for extra peace of mind.
  • Help keep private content private with the included password protection featuring 256‐bit AES hardware encryption.(3)
  • Easily manage files and automatically free up space with the SanDisk Memory Zone app.(5). Non-Operating Temperature -20°C to 85°C

What “bounded memory” means—and what it does not

The article gives 256 KiB as the default buffer size for a normal PUT. On the described path, the primary buffer is tied to the configured chunk size, not to the object’s total size. Thus, the article’s 10 MiB-versus-100 GiB comparison concerns the primary per-stream buffer, not every allocation made while transferring either object.

Concurrent streams multiply those buffers. The article’s illustrative arithmetic is 100 simultaneous streams × 256 KiB, or approximately 25 MiB for those buffers alone. It is not a measurement of Lioran S3’s process memory. Other allocations, filesystem or runtime buffering, multipart work, and the rest of the application can add to total use.

Rank #2
Sandisk 1TB Portable SSD, Up to 800MB/s Read Speeds, Black (Old Model)
  • Solid state performance with up to 800MB/s read speeds in a portable drive. (Based on internal testing; performance may be lower depending on host device, interface, usage conditions and other factors. 1MB=1,000,000 bytes.)
  • Back up your content and memories on a storage solution that fits seamlessly into your mobile lifestyle.
  • Take it with you on your adventures—up to two-meter drop protection means this durable drive can take a beating. (Based on internal testing.)
  • Secure it to your belt loop or backpack for extra peace of mind thanks to the tough rubber hook.
  • From Sandisk, a brand professional photographers trust to take on assignments.
  • Per-stream bound: the described reusable buffer is sized by the effective chunk size.
  • Concurrency effect: more active streams can mean more buffers at once.
  • System-wide caveat: bounded stream buffers do not establish a fixed ceiling for total process memory.

Progress reporting, timing, and flush behavior

The article describes progress reports every five seconds with bytes received, elapsed time, total MiB received, and effective MiB/s. It also says receive duration and write duration are accumulated separately. That instrumentation can help investigate the author’s question, “Is the server slow, or is the server waiting for the client/network?” It does not by itself identify a bottleneck or demonstrate a particular throughput.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

The article says the staging file is flushed after EOF and explicitly distinguishes a flush from fsync. A flush should not be described as equivalent to a durability guarantee from syncing data to storage; the precise durability behavior depends on the implementation and its selected mode.

Rank #3
Sale
Seagate 2TB Portable Hard Drive | USB 3.0 (STGX2000400) | Everyday HDD storage. Plug-and-play backup. Windows & macOS. Adobe CC trial included. 1-Year Warranty
  • Easily store and access 2TB to content on the go with the Seagate Portable Drive, a USB external hard drive
  • Designed to work with Windows or Mac computers, this external hard drive makes backup a snap just drag and drop
  • To get set up, connect the portable hard drive to a computer for automatic recognition no software required
  • This USB drive provides plug and play simplicity with the included 18 inch USB 3.0 cable
  • The available storage capacity may vary.

Multipart work and the boundary of the claim

For multipart handling, the article reports a separate 128 KiB copy buffer and global concurrency control. These details matter because multipart parts and concurrent work can consume memory in addition to the normal PUT buffer. The available article text does not establish a process-wide memory cap, nor does it supply measured RAM use under a specified workload.

The article’s architectural explanation is that the object engine consumes an asynchronous reader without needing to know whether the bytes came from an HTTP request body, a file, a test stream, or another producer. That separation lets the engine operate on a stream abstraction; it does not alone establish retry, replay, validation, timeout, or cancellation behavior.

Rank #4
Sale
Sandisk 1TB Extreme Portable SSD, Up to 2000MB/s Transfer Speeds-New Model
  • NEARLY 2X FASTER THAN OUR PREVIOUS GENERATION(8) – move 1,000 high-res photos in under 60 seconds(6) with up to 2000MB/s transfer speeds(2).
  • IP65 RATING AND UP TO 3M DROP PROTECTION(3) – protects against spills and drops.
  • POCKET-SIZED – fits easily in pockets and small bags.
  • SPACE TO OWN YOUR AI CONTENT – speed and capacity to download your high-res clips and photo edits.
  • 256-BIT AES ENCRYPTION(4) – helps keep private files secure with password protection.

How this compares with other streaming designs

The separate s3-wire 0.3.1 API documentation describes an async S3-compatible client as an “Async, streaming S3-compatible client with explicit resource bounds.” Its API illustrates why “streaming” is not one universal contract: source replayability, checksums, backpressure, timeouts, and cleanup are separate design choices. These are s3-wire behaviors, not independent confirmation of Lioran S3.

What’s actually slowing this PC down?

Pick the symptom - the matching free tool is one click away.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
Approach in s3-wire documentation Relevant behavior described by its docs Practical distinction
ByteStream::from_bytes Replayable byte source Replayability is available, though this description alone does not specify all memory costs.
ByteStream::from_path Replayable path-based source A path can serve as a source without treating it as the same one-shot stream contract.
ByteStream::from_stream One-shot stream requiring an exact length and SHA-256 digest The caller must provide metadata the API requires; a consumed stream is not automatically replayable.
Download ResponseStream Applies backpressure and validates declared length, configured deadlines, and supported checksums while consumed Streaming can include validation and flow control as well as incremental reads.
Managed multipart upload Bounds concurrent part buffers and handles abort cleanup after an upload ID exists Part concurrency and cleanup are distinct from a single reusable upload buffer.

The s3-wire documentation is for version 0.3.1, whose project listing dates versions 0.3.0 and 0.3.1 to September 10, 2026. Its docs say pinned MinIO, RustFS, and SeaweedFS suites run in CI, while an opt-in AWS suite had not yet run for that release. Those compatibility checks should not be read as proof of AWS compatibility.

Best Value
Seagate Portable 5TB External Hard Drive HDD – USB 3.0 for PC, Mac, PS4, & Xbox - 1-Year Rescue Service (STGX5000400), Black
  • Easily store and access 5TB of content on the go with the Seagate portable drive, a USB external hard Drive
  • Designed to work with Windows or Mac computers, this external hard drive makes backup a snap just drag and drop
  • To get set up, connect the portable hard drive to a computer for automatic recognition software required
  • This USB drive provides plug and play simplicity with the included 18 inch USB 3.0 cable
  • The available storage capacity may vary.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Support on Ko-Fi

Keep Amazon S3 limits separate from Lioran S3

Amazon’s documentation gives useful scale context, but its limits and buffering details apply to AWS operations and SDK paths—not automatically to Lioran S3 or every S3-compatible service. Amazon S3’s upload guide states that a single PutObject operation can upload up to 5 GB, while multipart upload supports a single object from 5 MB to 50 TB.

The same AWS guide says that, for large objects uploaded from a memory stream, AWS CRT buffers each part up to 5 GB and throughput is limited by allocated memory; for uploads from disk, CRT can use direct disk streaming instead of intermediate part buffering. This is AWS-specific documentation and should not be used to infer Lioran S3’s memory behavior.

What to check when evaluating a streaming implementation

A fixed-size read buffer answers only one resource question. To compare upload paths or assess a real workload, check the details that determine memory use and transfer behavior:

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
  • Buffer size per active stream and the maximum number of active streams.
  • For multipart, per-part buffer size and the concurrency limit on parts.
  • Whether the source is replayable, and how retries work for a one-shot stream.
  • Whether exact length and a checksum are required, and when validation occurs.
  • Whether backpressure and deadlines are enforced while data is consumed.
  • Whether data is staged to disk, when it is flushed, and what durability the mode promises.
  • What happens on cancellation or failure, including multipart abort and cleanup.

Those checks distinguish a bounded buffer from a complete resource-management policy. The cited Lioran article describes chunk reuse and timing instrumentation; the separate s3-wire docs document additional API behaviors. Neither supplies an independent Lioran benchmark or a measured whole-process memory profile.

Quick Recap

Bestseller No. 2
Sandisk 1TB Portable SSD, Up to 800MB/s Read Speeds, Black (Old Model)
Sandisk 1TB Portable SSD, Up to 800MB/s Read Speeds, Black (Old Model)
From Sandisk, a brand professional photographers trust to take on assignments.
$188.90
SaleBestseller No. 3
Seagate 2TB Portable Hard Drive | USB 3.0 (STGX2000400) | Everyday HDD storage. Plug-and-play backup. Windows & macOS. Adobe CC trial included. 1-Year Warranty
Seagate 2TB Portable Hard Drive | USB 3.0 (STGX2000400) | Everyday HDD storage. Plug-and-play backup. Windows & macOS. Adobe CC trial included. 1-Year Warranty
This USB drive provides plug and play simplicity with the included 18 inch USB 3.0 cable; The available storage capacity may vary.
$129.99
SaleBestseller No. 4
Sandisk 1TB Extreme Portable SSD, Up to 2000MB/s Transfer Speeds-New Model
Sandisk 1TB Extreme Portable SSD, Up to 2000MB/s Transfer Speeds-New Model
IP65 RATING AND UP TO 3M DROP PROTECTION(3) – protects against spills and drops.; POCKET-SIZED – fits easily in pockets and small bags.
$249.17
Bestseller No. 5
Seagate Portable 5TB External Hard Drive HDD – USB 3.0 for PC, Mac, PS4, & Xbox - 1-Year Rescue Service (STGX5000400), Black
Seagate Portable 5TB External Hard Drive HDD – USB 3.0 for PC, Mac, PS4, & Xbox - 1-Year Rescue Service (STGX5000400), Black
This USB drive provides plug and play simplicity with the included 18 inch USB 3.0 cable; The available storage capacity may vary.
$229.99

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.

Leave a Reply

Your email address will not be published. Required fields are marked *

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

More from the Wire

  1. World desk4 min
    How to Spot an AI Voice Scam Before Sending MoneyDon’t rely on how a caller sounds. Pause, call back through a known number, and verify the emergency with another trusted person before sending money.
  2. Mountain View desk4 min
    Google’s SynthID Detector: How to Check AI-Generated Images, Video and AudioGoogle’s SynthID Detector looks for an embedded watermark in supported images, video and audio. Here is what its results do—and do not—show.
  3. Redmond desk20 min
    How to create a link to File or Folder in Windows 11Windows 11 gives you several ways to point to a file or folder without moving or duplicating it. You can create a desktop shortcut,…
Recommended PC Tool
Recommended PC Tool
Windows Errors? Fix Them Before They SpreadFree repair scan
Crashes, No Sound, or Screen Glitches?Free driver scan

Two free Windows tools

One Free Minute Could Fix That PC

Before you go - each of these free tools takes about a minute and tackles what quietly slows a Windows PC down.

Special offer. View Outbyte info, uninstall instructions, EULA, and Privacy Policy.