More news websites are telling Internet Archive-associated crawlers not to collect their pages, but the reported numbers describe robots.txt rules—not proof that every crawler was successfully blocked. Nieman Journalism Lab’s May 20, 2026 analysis found such rules on 382 news sites in its sample, most of them local outlets. Publishers cite concerns about unauthorized use of journalism, including by AI companies; the same restrictions can make it harder to inspect how online reporting changes or disappears.
How widespread are the restrictions?
Nieman Journalism Lab’s May 20, 2026 update counted 382 news websites that disallowed at least one Internet Archive-associated crawler in their robots.txt files. Of those, 342 were local news sites, and 93% of the sites in the updated sample were based in the United States. The report’s headline emphasized the local-outlet figure; its overall count was 382.
As an Amazon Associate I earn from qualifying purchases.
The count grew from 241 sites in Nieman Lab’s January analysis: the May update added 141 sites and expanded the sample. These figures describe the sites covered by the analysis, not a census of all news outlets. Many of the local sites belonged to larger chains, but the report’s count does not mean that 382 major national publishers independently made the same decision.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
A robots.txt disallow rule is a request about crawler behavior. Its presence does not, by itself, establish that a crawler encountered the rule or was technically prevented from capturing a page. The reported count should therefore be read as a count of restrictions specified in the files Nieman Lab examined, not a count of verified, successful blocks.
#1 Best Overall
- Easily store and access 2TB to content on the go with the Seagate Portable Drive, a USB external hard drive
- Designed to work with Windows or Mac computers, this external hard drive makes backup a snap just drag and drop
- To get set up, connect the portable hard drive to a computer for automatic recognition no software required
- This USB drive provides plug and play simplicity with the included 18 inch USB 3.0 cable
- The available storage capacity may vary.
What did the analysis count?
For its January analysis, Nieman Lab used journalist Ben Welsh’s database of 1,167 news-site robots.txt files, then checked additional files for the May update. It counted a site when its file disallowed at least one of seven Internet Archive-associated crawler names.
There is an attribution caveat: Wayback Machine founder Mark Graham told the reporters that Wayback does not use “ia_archiver,” “ia_archiverbot” or “ia_archiver-web.archive.org.” Nieman Lab nevertheless included “ia_archiver-web.archive.org” in its count because publishers had disallowed that name under the assumption it was used by the Archive. The methodology is evidence of what publishers’ files specify, not confirmation that every named crawler belongs to or is used by Wayback.
Rank #2
- Easily store and access 5TB of content on the go with the Seagate portable drive, a USB external hard Drive
- Designed to work with Windows or Mac computers, this external hard drive makes backup a snap just drag and drop
- To get set up, connect the portable hard drive to a computer for automatic recognition software required
- This USB drive provides plug and play simplicity with the included 18 inch USB 3.0 cable
- The available storage capacity may vary.
Why are publishers restricting the crawlers?
The reasons reported by Nieman Lab vary by organization. Some publishers fear that AI companies could use archived journalism to train models without permission or compensation. Others want to protect the commercial value of their work, preserve licensing leverage or ensure AI products attribute information to the original publisher.
Some policies are broader than Wayback
Advance Local spokesperson Christine deWit described the company’s policy as a broader response to third-party use: “This is part of a broader effort to protect the value of our published work from unfair third‑party use. This decision is not specific to the Wayback Machine.” The Atlantic’s SVP of communications, Anna Bross, said: “Our default is to block: No one should be scraping The Atlantic’s journalism without permission, regardless of the use.”
Rank #3
- Easily store and access 1TB to content on the go with the Seagate Portable Drive, a USB external hard drive.Specific uses: Personal
- Designed to work with Windows or Mac computers, this external hard drive makes backup a snap just drag and drop. Reformatting may be required for Mac
- To get set up, connect the portable hard drive to a computer for automatic recognition no software required
- This USB drive provides plug and play simplicity with the included 18 inch USB 3.0 cable
- The available storage capacity may vary.
AI concerns are not proof of AI use of archived pages
Publishers’ stated concern about possible AI use should not be confused with evidence that it happened in these cases. As of Nieman Lab’s May 20, 2026 report, no news publisher had confirmed to the publication that an AI company had scraped its content from Wayback Machine captures.
The Baltimore Banner’s chief technology officer and AI strategist, Biswajit Ganguly, put the distinction plainly: “The threat is definitely not the Internet Archive,” he told Nieman Lab, emphasizing concern about whether AI products link back to original reporting. That is one publisher’s stated view, not a finding about how all AI products use archived material.
Why does Wayback access matter for journalism?
An archived page can help readers and researchers examine an earlier version of an online story, compare changes, or locate material that later disappears. Nieman Lab reported that working journalists use local news archives and described articles lost during site migrations as well as a defunct publication whose archive went offline. When a publisher’s current page changes or vanishes, a third-party archive may be one of the few accessible records of what was published before.
The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Rank #4
- Easily store and access 4TB of content on the go with the Seagate Portable Drive, a USB external hard drive.Specific uses: Personal
- Designed to work with Windows or Mac computers, this external hard drive makes backup a snap just drag and drop
- To get set up, connect the portable hard drive to a computer for automatic recognition no software required
- This USB drive provides plug and play simplicity with the included 18 inch USB 3.0 cable
- The available storage capacity may vary.
Edward McCain, journalism librarian at the University of Missouri, told Nieman Lab: “Blocking the Internet Archive’s web crawlers threatens one of the most effective ways that we capture and store news content for the long term,”
Internet Archive Europe argues that restrictions reduce access to a public historical record. That is the Archive’s institutional position. In a June 9, 2026 commentary, it said Wayback holds more than one trillion archived web pages and preserves permanent citations for nearly 5 million news articles referenced on Wikipedia; those figures are the organization’s own claims. It also said more than 250 journalists had signed an open letter by the date of its commentary.
Best Value
- [Upgraded Version] - This external hard drive features a mirrored logo stripe combined with a striped anti-slip design, and the rounded corners of the casing make it easier to grip. The stripes also have a heat dissipation function, ensuring stable and fast data transfer.
- 【Ultra-thin and quiet】 - The motherboard adopts JMicron 578 noise-free solution, giving you a quiet working environment. Lightweight and portable size designed to fit in your pocket for easy portability.
- 【Ultra-Fast Data Transfers】 - Pairing this external hard drive with JMicron 578 solution USB 3.0 and USB 2.0 interfaces enables blazing-fast data transfer. It boasts theoretical read speeds of up to 125MB/s and write speeds of up to 103MB/s.
- 【Plug and Play】 - With no software to install, just plug it in and the drive is ready to use.The hard disk chip is wrapped with an aluminum anti-interference layer to increase heat dissipation and protect data.
- 【What You Get】 - 1 x Portable Hard Drive, 1 x USB 3.0 Cable, 1 x User Manual, Gift-type shell packaging ,Three-year manufacturer's warranty and free technical support services.
What can publishers and researchers use instead?
Nieman Lab describes several options that can complement web archiving. They differ in who can access material, who controls retention and what resources preservation requires. The report does not establish that any option fully replaces the Wayback Machine.
| Option | Access | Coverage and continuity | Control and newsroom requirements | What it preserves or resolves |
|---|---|---|---|---|
| Internet Archive’s Wayback Machine | Public web archive; access to a particular page or capture can vary. | Nieman Lab discusses its role in preserving changing or disappearing pages. A completeness measure for news coverage is not stated (Nieman Lab, May 20, 2026). | Preservation is operated by the Internet Archive. A newsroom’s cost or technical workload is not stated in the report. | Can provide access to earlier page versions and persistent citations; no guarantee that every page or version is captured. |
| Publisher-owned archives | Access terms depend on the publisher; public availability is not stated (Nieman Lab, May 20, 2026). | Coverage and retention depend on the publisher’s archive; comparative breadth is not stated. | The publisher controls its own archive. Specific costs and technical requirements are not stated. | Can preserve a publisher’s material, but availability of earlier versions and citation behavior are not stated. |
| ProQuest and LexisNexis | Paid services, potentially available through libraries, universities or individual subscriptions. | Comparative breadth and continuity are not stated (Nieman Lab, May 20, 2026). | Access and preservation are mediated by commercial services and the institutions or subscribers using them; newsroom costs are not stated. | Can provide access to archived journalism, but the report does not establish that coverage or version history matches open web archives. |
| Newsroom archiving strategies and training | Access to preserved material depends on the newsroom’s approach; public access is not stated (Nieman Lab, May 20, 2026). | Continuity depends on what a newsroom captures and retains. | Requires newsroom planning and capacity. A partnership among the Internet Archive, Poynter Institute and Investigative Reporters and Editors aimed to train 300 newsrooms by the end of 2027, after an initial cohort of 33 outlets. | Can help newsrooms preserve their own reporting; the initiative is training, not evidence of a complete replacement archive. |
For readers trying to find an old article, check whether the publisher still offers an archive and whether a library or university provides access to a relevant database. If those routes do not have the material, a public web archive may have an earlier capture, but a page’s absence there does not prove it was never published.
Recommended Free Tools
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
