A nightly job can report “success” while one of its sources has been frozen for almost two weeks. That is what happened in a 20-source trending-data pipeline described by hc_xshh in a September 28, 2026 DEV Community post (an English translation of the author’s original Chinese notes). From September 1 through September 12, the Reddit column on the author’s site kept showing August 31 data, and the job status never changed from green.
The counts are the author’s own: of 20 platforms, 8 had a directly callable API, 10 were reachable through RSS, and 2 needed workarounds. Everything below comes from that single implementation. These are one developer’s endpoint observations, not platform documentation, and routes, headers and geographic behavior can change. Treat the design lessons as the durable part and re-verify every endpoint before reusing it.
As an Amazon Associate I earn from qualifying purchases.
The short answer: how to see stale sources before readers do
The author’s central line is blunt: “The most deceptive status a collection script can report is ‘success’.” A process exit code only tells you the script finished. It says nothing about whether each source returned fresh, complete data. The fix is to give every source its own recorded outcome and to make the difference between “nothing came back,” “the request failed,” and “we’re showing old data” explicit, as the author put it: “Zero items returned” and “fetch failed” are different bugs.
The rest of this article covers how the author’s 20 sources split by access method, the four incidents that exposed the weaknesses, and a design you can adapt.
#1 Best Overall
How the 20 sources split: API, RSS, workaround
The pipeline runs nightly. It fetches trending data, translates selected items, summarizes them, builds a static site, commits it and deploys it. In the author’s example run of all sources, 480 items were collected, and the stretch from fetch through deployment took 64 seconds.
| Category | Count | Sources | What the author reported |
|---|---|---|---|
| Direct API | 8 | Zhihu, Bilibili, V2EX, Hupu, Maoyan, Hacker News, Lobsters, GitHub | Zhihu’s official CLI allowed two trending-list calls per day. Bilibili, Hupu and Maoyan routes behaved differently with proxies. Hupu needed several pages for a larger list. Maoyan returned 403 without a desktop User-Agent and a Referer. V2EX’s public list was smaller. Hacker News, Lobsters and GitHub used comparatively simple JSON or search routes. |
| RSS | 10 | sspai, ifanr, ITHome, Solidot, cnBeta, The Verge, Ars Technica, TechCrunch, arXiv, Reddit | Uniform format and no API keys, but each feed decides which fields and how many items it exposes. Reddit’s feed had titles and links but no scores or comment counts. |
| Workaround | 2 | 36Kr, YouTube | 36Kr pages needed a rendering service because a plain request returned an empty shell. YouTube went through a third-party aggregator after the official trending API returned an empty shell from the author’s datacenter IP. |
Read the “8 of 20” figure as a statement about this author’s setup, constraints and date, not as a verdict on those platforms. A different IP range, account type or month could move a source from one column to another. The useful takeaway is the maintenance profile: RSS sources were the most uniform, direct APIs came with per-route quirks (headers, proxies, pagination, call quotas), and the workarounds were the most fragile.
Incident 1: Reddit served August 31 data for 12 days
What happened
From September 1 to September 12, the Reddit column displayed data from August 31. A fetch-failure message did appear in the report, but it sat among many successful source lines and was easy to miss. The overall job status stayed “success,” and the page kept rendering, so nothing looked broken.
The author traced the break to a Google Translate proxy page that began answering with a 302 redirect in early September. The replacement was Reddit’s Atom RSS feed. In the author’s setup it needed the parameter geo_filter=GLOBAL to avoid localized results, plus a proxy.
Rank #2
The trade-off the author accepted
The RSS route gave titles and links but no scores or comment counts. The author tried a keyless .json endpoint to recover them and got 403. A login-cookie approach did return scores, but the author rejected it as unapproved automated access. This is an access decision by one developer, not a statement of Reddit’s current API policy. If you need engagement numbers from a platform, check its official API terms rather than relying on this account.
The failure policy behind it
The author’s stated rule is: “When a fetch fails, keep the previous data and tag it as stale; I don’t overwrite readable content with empty data.” Keeping old data is reasonable, since a column of old-but-readable items beats a blank one. The Reddit case shows the missing half of that policy: the staleness has to be visible, to the operator and ideally to readers, and it has to be monitored. Old data presented as current is the trap.
Incident 2: arXiv looked complete but skipped categories
The author reports that arXiv’s official API returned a body reading Rate exceeded., and switched to RSS, which worked. Later, an early return inside a loop over categories let the first category fill the 30-item limit before the others were requested. The result still contained a full 30 items, yet machine-learning and natural-language-processing papers were missing.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Clear out junk files and repair common Windows errorsFree Scan →The count looked right, so a count-based check passed. The lesson: validate the shape of the result (which categories are present, which fields are populated), not just how many rows came back.
Incident 3: 36Kr’s empty result that looked like a network error
36Kr required a rendering service. According to the author, below a certain service tier the response contained an empty result list and a failure reason instead of raising an error. The code then read the first result, hit an index error, and an outer exception handler quietly kept the old data. The symptom resembled a network failure, but the actual cause was a service parameter that had to be raised.
This is the clearest example of why “empty” and “failed” need separate handling. An HTTP 200 with an empty list plus a reason field is a failure with a success-shaped body. A broad catch-all handler then converted a diagnosable problem into silent fallback.
Incident 4: the one-hour timeout that left a 249-byte report
On September 15 the scheduled script ran into its 3,600-second limit and was killed. Python’s output was piped and block-buffered, so nothing useful had been flushed. The report was 249 bytes, with no progress information, and the live page did not update.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Fix the driver behind crashes, sound loss and screen glitches3Clear out junk files and repair common Windows errorsThe author’s response:
- Added
exec </dev/nullto the script. - Put timeouts on individual steps rather than only on the whole job.
- Ran Python with
python3 -ufor unbuffered output. - Printed one timestamped line per step.
Later audits added separate retry budgets for translation and for Reddit. Those exact limits are the author’s choices, not recommended defaults. The principle is that an unbounded retry can consume the entire job window and starve every source after it.
A source-level design that would have caught these
None of the four problems required exotic tooling. They needed each source to report its own state. The following is an illustrative sketch of that idea, not the author’s code.
Give each source one of four explicit states
| State | Meaning | What to do |
|---|---|---|
| fresh | Request succeeded, parsed items passed validation | Publish, record fetch time |
| empty | Request succeeded and the source legitimately has nothing new | Publish nothing new; log it as expected only if the source has a history of being empty |
| failed | Request error, non-200 status, or success-shaped body with a failure reason | Record the error and alert; do not treat as empty |
| stale | Fetch failed and previous data is being shown | Tag with the last successful fetch time, show age, alert after a threshold |
def collect(source):
try:
resp = source.fetch(timeout=source.timeout)
items = source.parse(resp)
except Exception as exc:
return stale_or_failed(source, reason=repr(exc))
if resp.failure_reason: # 200 with a failure body (the 36Kr case)
return stale_or_failed(source, reason=resp.failure_reason)
problems = source.validate(items) # categories, required fields, minimum count
if problems:
return stale_or_failed(source, reason=problems)
return Result(source.name, "fresh" if items else "empty", items)Two details matter. First, the catch is narrow in what it records: it stores the reason, so a failure never disappears into a generic fallback. Second, each source runs inside its own handler, so one failure cannot block the others, which matches the author’s goal of unrelated sources continuing independently.
Validate coverage, not just volume
For arXiv-style feeds, assert that every expected category appears at least once and that required fields (title, link, and any score you rely on) are non-empty. A check such as “30 items returned” would have passed during the author’s arXiv bug, while “every configured category contributed” would have failed it.
Recommended Free Tools
Make staleness loud
- Store a last-successful-fetch timestamp per source and render it next to the data.
- Put failed and stale sources at the top of the job summary, not interleaved with successful lines.
- Make the job’s final status reflect source health. A run with a stale source should not read the same as a clean run. Choose a threshold: a single stale source might warn, while several or an age beyond a day might fail the job.
- Send an alert on state transitions (fresh to stale) rather than on every run, so the signal stays readable.
Make the run observable even when killed
- Print a timestamp and step name at the start and end of each stage, so a killed job leaves a trail.
- Disable output buffering (
python3 -uorPYTHONUNBUFFERED=1) wherever a pipe or supervisor could swallow progress. - Give each stage and each retry loop a bounded budget so one slow source cannot consume the job window.
Smaller lessons from the same pipeline
Constrain the language model to choosing IDs
The author keeps model output limited to selecting item identifiers where possible, then fills titles and links from the collected records. That reduces invented or invalid links, since the model can never emit a URL that was not in the input.
Batch size affects output integrity
The author’s translation output was truncated at 30 items per requested batch and was fixed by dropping the batch to 15. That threshold applies to their model and prompt; the portable point is to check that the output count matches the input count after each batch.
Expect conflicting network requirements in one job
Within one run, some sources wanted a proxy, some behaved differently with one, some needed browser-like headers, and some blocked datacenter IPs. Store these as per-source configuration rather than global settings, and re-verify them periodically because they are observations about particular endpoints on particular dates.
Choosing a collection route for each source
The author’s account does not compare vendors, but the trade-offs it describes suggest these axes when deciding how to collect from a given platform:
| Axis | Official API | RSS or Atom | Workaround (renderer, aggregator) |
|---|---|---|---|
| Fields and coverage | Usually richest, but may be paged or capped (Hupu, V2EX, Zhihu quota) | Whatever the feed publishes; Reddit’s lacked scores | Depends on the third party |
| Access constraints | Headers, proxies, quotas, keys | Few, though geography parameters can matter | Service tiers and IP-based blocks |
| Maintenance burden | Moderate | Lowest in this pipeline | Highest; two of the four incidents involved fallbacks or routes that changed |
| Failure visibility | Often explicit errors, but “Rate exceeded” can arrive as a body | Silent staleness if the feed stops updating | Success-shaped empty bodies |
The pattern is that no route removes the need for validation. RSS is the cheapest to maintain but the easiest to leave quietly stale, and workarounds are the likeliest to return responses that look fine and contain nothing.
A checklist to audit your own pipeline
- Does every source have its own recorded state (fresh, empty, failed, stale) and last-success time?
- Can the overall job fail, or at least warn, when any source is stale beyond a set age?
- Do you treat an HTTP 200 with an empty list or a failure field as a failure?
- Does any broad exception handler fall back silently without recording the reason?
- Do validations check categories and required fields, not only the total count?
- Is stale data labelled, with its age, wherever readers can see it?
- Does each step have its own timeout, and each retry loop a bounded budget?
- If the job were killed at minute 30, would the log show which step it was in?
- Are proxy, header and region settings stored per source and re-checked on a schedule?
The author’s own record is the argument for the list: a source frozen for 12 days, a full-looking result missing whole categories, a failure disguised as an index error, and a one-hour hang that left 249 bytes of evidence.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
