An ethical web scraper does more than obey robots.txt: it checks whether collection is authorized, limits what it gathers, protects personal data, respects rights reservations, and records where every dataset item came from. A path allowed by a crawler protocol is not, by itself, permission to copy or use the material. Before collecting data for AI, assess the source’s terms and access controls, the data and intended use, applicable privacy and intellectual-property rules, and the burden your crawler may place on the site.
Is web scraping legal if a page is public or robots.txt allows it?
Neither public availability nor a permissive robots.txt file settles whether a scrape is lawful. Depending on the jurisdiction and facts, separate questions can arise under copyright, database rights, privacy law, contract or site terms, and laws governing access to computer systems. The OECD notes that the legal effect of machine-readable scraping restrictions can depend on the circumstances, and that technical rules and site terms may not match. Its 2025 analysis is a useful overview, not a ruling on any particular collection (OECD, February 2025).
As an Amazon Associate I earn from qualifying purchases.
RFC 9309 describes rules that automated clients are requested to follow. It states: “These rules are not a form of access authorization.” So treat robots rules as one important crawler signal, not as a license or a substitute for reviewing the source’s terms and relevant rights (IETF RFC 9309, September 2022). A site’s terms, a rights reservation, restricted access, and the material and purpose of your collection may all affect the analysis. The available sources do not determine whether a particular scrape violates copyright, contract, database rights, or access law in a specific jurisdiction; high-impact or commercial projects warrant jurisdiction-specific legal review.
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Repair Windows errors before they cause bigger problemsFix Now →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →What should you check before collecting data?
Use these questions as a project gate. They are a practical framework, not a statutory checklist.
#1 Best Overall
- Purpose: Is the collection for model training, evaluation, indexing, or another defined use? Set that purpose before gathering data, and do not assume that permission or a legal basis for one use carries over to another.
- Source and access: Who controls the site or dataset? Review applicable terms, access restrictions, and machine-readable instructions. Consider whether the source offers an authorized way to obtain the material.
- Rights: What copyright, database-rights, or other rights may apply? Are there rights reservations, and what do they cover? Record the basis for proceeding rather than treating absence of a technical block as clearance.
- Data sensitivity: Will the scrape include information about identifiable people? Could it include special-category personal data? Decide which fields are necessary and how you will prevent or handle incidental collection.
- Scale and impact: Is the collection proportionate to the purpose, and can the source bear the traffic? Establish crawler identification, operational limits, monitoring, and a way to respond to source-side concerns.
- Accountability: Can you document what was collected, when, under what rules and purpose, how it was filtered and validated, and what happens when data must be removed?
How should robots.txt guide your crawler?
Identify your crawler, retrieve and parse the site’s current robots.txt, follow applicable disallow instructions, and keep a record of the policy version or fetch time. RFC 9309 specifies that parseable rules must be followed after successful retrieval. If the file is unreachable because of server or network errors, the protocol says the crawler must assume complete disallow. These are protocol behaviors; they do not establish legal authorization (RFC 9309).
Review site terms and access controls separately. The OECD’s 2025 analysis describes inconsistent implementation of scraping restrictions, including gaps between terms and technical rules. A robots file can therefore be neither the whole compliance review nor a reliable substitute for it (OECD analysis).
What changes when a scrape contains personal data?
In the EU, the GDPR applies when web scraping involves processing personal data—including collecting, storing, organizing, or retrieving it. The European Data Protection Board’s 2026 guidance highlights purpose limitation, transparency, accuracy, and data minimisation. Its announcement recommends using reliable sources, recording timestamps, and validating data before AI training (EDPB announcement, 2026).
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Build those requirements into collection rather than trying to fix everything after the dataset is assembled. Specify the purpose and fields in advance, collect only what is necessary, retain source and time information to support accuracy checks, and document how the data will be used and kept. The EDPB’s Guidelines 03/2026 were under public consultation until 30 October 2026 as of the guidance announcement; they are current guidance, but not a final post-consultation text.
Special-category personal data requires a separate assessment
For special-category personal data under the GDPR, the EDPB says processing is in principle prohibited unless both an Article 6 lawful basis and an Article 9(2) exception are available. It discusses possible incidental or residual collection only in limited circumstances and says applicability must be determined case by case. Incidental collection is not a general exemption. If your project cannot establish the required grounds, exclude that data or stop the collection rather than assuming that public visibility makes it usable (EDPB announcement).
How do copyright and AI-training obligations fit together?
Copyright and related rights questions are distinct from privacy questions. A dataset can contain no personal data and still raise rights issues. Review the material, the amount collected, the use you intend to make of it, applicable terms, and any rights reservations. The U.S. Copyright Office’s AI study page lists Part 3, Generative AI Training, as a pre-publication version released May 9, 2025, and says a final version is expected. That status describes an issue under study; it is not a definitive court ruling or a settled, general rule for all U.S. training uses (U.S. Copyright Office, Artificial Intelligence Study).
In the EU, the European Commission describes obligations for general-purpose AI providers to maintain a copyright-compliance policy, including identifying and respecting rights reservations, and to publish a sufficiently detailed summary of training content. Its guidance also describes downstream documentation about training, testing, and validation data, including data types, provenance, and curation methods. These are provider obligations described in the Commission’s guidance; do not assume every organization that collects web data has the same role or obligations (European Commission guidance on general-purpose AI providers).
The UK government’s 2026 report summarizes the EU training-content template as calling for information such as modalities, sizes, material types, languages, acquisition dates, major public datasets and identifiers, crawlers and their purposes, rights-reservation methods, and measures to remove illegal content. Use the Commission and applicable EU materials for primary compliance decisions; the UK report is a summary of the EU transparency requirements (UK Government report, 2026).
Best Value
How do you build a compliant collection and dataset pipeline?
Keep a record at each stage so the team can explain what it collected, why, and how it handled the data. The following workflow combines privacy, crawler-protocol, and AI-provenance considerations; it is a practical implementation approach, not a universal legal checklist.
- Define and approve the use. Write down whether the data is for training, evaluation, indexing, or another purpose, who will use it, and where the collection and downstream processing will occur. Identify the jurisdictions and legal questions that need review.
- Assess each source. Record the source URL, its terms and access conditions, relevant rights or reservations, and the results of your rights review. Identify the crawler and its purpose. If a source does not authorize the proposed access or use, do not treat a public page as permission.
- Check crawler rules before each collection. Fetch and parse
robots.txt, follow applicable instructions, and record when and what you checked. Apply RFC 9309’s complete-disallow behavior when the file is unreachable because of server or network errors. Keep this protocol check separate from the legal and terms review. - Specify fields and minimise. List the fields needed for the approved purpose, and configure collection to omit unnecessary material. Assess whether personal or special-category data may be present; define filtering and escalation steps before collection begins.
- Collect with source impact in view. Clearly identify the crawler, monitor its behavior, and establish operational safeguards appropriate to the source and collection. The Italian data protection authority gives registration-gated areas, anti-scraping terms, abnormal-traffic monitoring, and technical measures such as robots.txt as examples of possible source-side safeguards. It describes those measures as non-mandatory and for controllers to assess based on accountability, technology, and implementation costs—not as universal requirements for every site (Italian Data Protection Authority, May 30, 2024).
- Validate before use. Retain timestamps and source information, check that data is reliable and fit for the stated purpose, and document validation and filtering steps before training or evaluation. This supports accuracy and helps detect stale, incorrect, or out-of-scope material.
- Version, retain, and govern the dataset. Store dataset versions and provenance records, define retention and deletion decisions, and record how you will address corrections or removal requests. If you provide a general-purpose AI model in the EU, assess the Commission’s specific copyright-policy and public-summary obligations as well as applicable documentation expectations.
What should an audit trail contain?
Keep records detailed enough for another team member to reconstruct the collection and its decisions. A useful dataset record includes:
- Source URL and collection timestamp.
- Crawler identity, purpose, and the relevant
robots.txtpolicy version or fetch time. - Applicable source terms, access restrictions, and rights-reservation review.
- Fields collected and the reason each field was necessary.
- Privacy assessment, including the legal basis and handling decisions for personal data where applicable.
- Filtering, minimisation, and validation methods, with the results of material checks.
- Dataset version, retention period or decision, deletion handling, and downstream use.
- For applicable AI-provider disclosures, provenance and curation information needed to describe training, testing, and validation data.
This record is not a substitute for legal review, but it makes the project’s decisions and dataset lineage inspectable. The EDPB’s guidance emphasizes accuracy and minimisation, while the European Commission’s AI-provider guidance addresses provenance and curation documentation (EDPB; European Commission).
Recommended Free Tools
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
