What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Muhammad Hammad’s attempt to collect nine years of his DEV.to publishing history ran into two separate problems: a large in-memory workload and a mismatch between the platform’s article count and his local database. His reported figures—847 articles in the dashboard, 612 database records, and a failed fetch that used more than 3.2 GB of Python heap—describe one project, not a general DEV.to benchmark. The architectural response he describes was to replace an accumulating in-memory workflow with bounded streaming, capacity-limited queues, and batched writes.
What did the nine-year DEV.to data pull uncover?
In a DEV Community article dated September 25, 2026, Muhammad Hammad describes collecting and analyzing his own publishing history. The effort revealed a count discrepancy as well as a resource problem in his first implementation.
As an Amazon Associate I earn from qualifying purchases.
| Figure | What Hammad reports |
|---|---|
| 847 articles | Listed in his DEV.to dashboard |
| 612 records | Present in his database |
| 235 articles | The difference between those counts |
| About 510 MB | Estimated size of the raw, uncompressed JSON |
| More than 3.2 GB | Python heap use in the failed in-memory attempt |
These are Hammad’s project-specific figures, not independently measured platform statistics. He attributes the missing records to soft deletion by DEV.to, but the available article excerpt does not show an audit trail or other confirmation of that explanation. A dashboard/database count difference establishes a discrepancy; it does not, on its own, establish what caused it. Read Hammad’s DEV Community article.
Why did the in-memory approach fail?
A straightforward paginated collector often fetches each page, appends the results to a growing collection, and then performs later transformations or joins. That can be convenient for a small dataset. As the collection grows, however, the process must retain more objects at once. Parsing JSON and joining related data can add further working copies, so memory consumption may exceed the size of the raw response files.
#1 Best Overall
Hammad says his naive fetch crashed at page 47. In his account, raw JSON was estimated at about 510 MB, while joined data could expand roughly fourfold. He reports Python heap use exceeding 3.2 GB during the failed attempt. These figures help explain the shape of the problem in this project, but they are not a benchmark for other archives: memory use depends on implementation, data shape, runtime, and what is retained during processing.
What architectural change did Hammad make?
Rather than treating the archive as one dataset to fetch and hold in memory, Hammad says he treated it as a streaming pipeline. He describes using Python’s standard library, bounded components, capacity-limited queues, and batched writes. As he puts it: “The fix was not adding more RAM. The fix was stopping the treatment of this like a data processing problem and starting to treat it like a streaming pipeline problem.”
Rank #2
- Wiley
- Language: english
- Book - storytelling with data: a data visualization guide for business professionals
The principle is to move data through stages in manageable portions instead of allowing an ever-growing in-memory collection to become the central store. A capacity-limited queue can constrain how much work accumulates between stages, while batched writes can group persistence operations. The excerpt does not provide enough detail to reconstruct Hammad’s full pipeline, persistence design, validation rules, checkpointing behavior, or a reproducible performance comparison.
When is streaming a better fit than accumulating everything?
| Consideration | Fetch and accumulate in memory | Bounded streaming |
|---|---|---|
| Memory as data grows | The retained collection grows with the fetched data; joins or copies can raise the working set. | Bounded stages can limit how much data is in flight, depending on their implementation. |
| Resume after failure | Not established by the approach alone; it depends on added persistence or checkpoint logic. | Not established by streaming alone; checkpoints and recovery rules must be designed. |
| Rate limits and throughput | Not quantified in Hammad’s excerpt. | Not quantified in Hammad’s excerpt; concurrency and batch size still need to respect source limits. |
| Implementation effort | Often simpler for a small, bounded dataset. | Requires coordinating stages and handling partial progress, failures, and backpressure. |
| Count reconciliation | Requires explicit comparison against source totals and record-level checks. | Also requires explicit comparison against source totals and record-level checks. |
Streaming is not automatically faster or more reliable. Its practical advantage is control over the amount of work held in memory. For a modest archive that fits comfortably in memory, the simpler approach may be entirely adequate. For a growing archive, especially one that must be joined or transformed, bounded processing can make resource use more predictable.
Quick Recap
Best Value
Rank #4
What should developers verify when collecting a paginated archive?
- Separate source counts from local counts. Record the platform’s reported total and the number of records actually persisted, then investigate any gap rather than assuming its cause.
- Account for expansion. The raw response size does not necessarily predict peak memory when data is parsed, copied, or joined.
- Bound work in progress. Set queue capacities and batch sizes deliberately so a slow downstream stage cannot allow unlimited data to accumulate.
- Plan recovery explicitly. Streaming does not itself provide checkpoints, retries, duplicate handling, or proof that all pages were processed.
- Measure the actual workload. Hammad’s memory figures describe his own failed run; they should not be used to size another developer’s machine without testing that workload.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
