Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Clear out junk files and repair common Windows errorsFree Scan →To collect public government data reliably, start with the official dataset record, check its reuse terms, and use the publisher’s documented API or bulk download when available. Scrape web pages only when the publisher permits automated access and no suitable structured route exists. Public visibility is not blanket permission, and service-specific rate limits and restrictions still apply.
This guide focuses on U.S. federal examples; state, local, and non-U.S. sources may have different access routes and terms.
1. Find the official dataset record
Begin with a catalog, but treat it as a way to discover data—not necessarily the place to retrieve or interpret it. Data.gov helps locate federal datasets and provides APIs for dataset search and metadata retrieval. Follow the catalog entry to the responsible agency or publisher, then use that publisher’s instructions and record as the authoritative starting point.
For government publications and selected legislative or regulatory collections, GovInfo documents API and bulk-data options. Availability varies by collection: do not assume every agency or catalog item has an API, bulk file, or identical formats.
What to capture from the record
- Publisher and the specific dataset or collection name.
- Coverage dates, update information, and any known gaps.
- Available formats, metadata, data dictionary, and documentation.
- Access method and the dataset-level “Access and Use Information.”
- Links to the source endpoint or file, plus the date you retrieved it.
Keeping these details with the downloaded data makes it easier to reproduce a collection and spot later changes.
2. Check reuse terms and service rules
Read both the dataset’s own access-and-use information and the terms for the service through which you retrieve it. Data.gov says federal data is generally offered free and without domestic copyright restrictions, but exceptions exist; non-federal records can have different licensing. A catalog listing does not establish that all its contents share one license or that automated collection is permitted in every context. See Data.gov’s catalog guidance and the federal Open Data Principles.
Rules can be endpoint-specific. For example, the U.S. Department of Commerce API terms require attribution, prohibit falsely representing API content, and allow access limitations (Commerce API Terms of Service). SAM.gov directs users to selected APIs and extracts for some information, prohibits bots from downloading or copying restricted or sensitive data, and states that automated gathering and scraping tools are prohibited on that service (SAM.gov entity information). These examples are not universal rules for every government website; check the exact service you plan to use.
This is practical guidance, not legal advice. If the terms are unclear or your use is sensitive, seek clarification from the publisher before collecting or republishing the data.
PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchRank #2
- Designed By Field Experts
- This Data Book Is Ideal For Police And Military Missions
- Country Of Origin: China
- Model Number: 12-8208000000
3. Choose an access route
Use the publisher’s intended structured route whenever it meets your needs. APIs are useful for targeted or recurring queries; bulk downloads can be simpler for large, one-time collections. Page scraping is a fallback for information that is only exposed on pages and whose terms allow automated retrieval.
| Route | Best fit | What to check |
|---|---|---|
| Documented API | Filtered queries, regular updates, or a publisher-defined programmatic interface. | Authentication, endpoint-specific limits, response format, attribution, and error handling. |
| Bulk download | A large snapshot or a collection provided as a downloadable package. | Coverage and update date, file format, checksums if supplied, and the documentation needed to interpret it. |
| Page-level scraping | Relevant information is available only in web pages and automated collection is allowed. | Terms, robots.txt guidance, request pace, page changes, and whether the result can be validated. |
GovInfo is one concrete example of a publisher offering API documentation and bulk XML for selected collections, including documented XML and JSON bulk endpoints (GovInfo bulk data). Confirm that a specific collection is covered before building your workflow around it.
4. Prepare API access and respect limits
Data.gov APIs use api.data.gov to manage authentication, rate limiting, and usage tracking. The Data.gov API page lists a free personal key with a limit of 1,000 requests per hour. Its DEMO_KEY is more restricted: 30 requests per IP per hour and 50 per IP per day. These are operating limits for those credentials, not general allowances for all government APIs; service-specific limits can differ. Check the live Data.gov API documentation before relying on a figure.
The api.data.gov developer manual recommends checking rate-limit headers and notes that limits may vary by service. Treat a 429 response or a rate-limit header as a signal to slow down, not as an invitation to rotate identities or keys to evade a limit.
Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Repair Windows errors before they cause bigger problems3Scan for outdated or missing drivers - takes under a minuteA conservative API request pattern
Use the exact endpoint, parameters, and authentication method documented by the publisher. The following Python template makes a single request to an example endpoint; replace the URL and parameters with those specified by the API you are authorized to use. It deliberately does not assume a particular API response format.
import time
import requests
url = "https://api.example.gov/v1/records"
params = {"limit": 100}
headers = {"User-Agent": "ResearchProject/1.0 contact: [email protected]"}
response = requests.get(url, params=params, headers=headers, timeout=30)
if response.status_code == 429:
retry_after = response.headers.get("Retry-After")
raise RuntimeError(f"Rate limited; follow Retry-After: {retry_after}")
response.raise_for_status()
print("Status:", response.status_code)
print("Rate limit remaining:", response.headers.get("X-RateLimit-Remaining"))
print(response.text[:1000])This is a request-handling pattern, not a universal API client: use the service’s documented headers, pagination, response parsing, and retry guidance. For a multi-page collection, request one page at a time, pause as needed, record progress, and resume from the last confirmed page rather than restarting blindly.
5. Scrape pages only when appropriate
Before automating a page, review its terms and inspect its robots.txt for crawling guidance. Digital.gov’s robots.txt guidance explains the file and crawl-delay directives. Robots.txt is not a complete permission grant and does not replace service terms or API documentation.
A GSA blog discussing agency scraping recommends considering robots.txt, terms, low-impact frameworks, and off-peak requests, while explicitly noting that the blog’s views are not official federal guidance (GSA blog: Web scraping government sites). Apply the same care, and honor any service-specific prohibition or restriction.
Page-scraping checklist
- Confirm there is no suitable documented API or bulk route, or explain why it does not meet the task.
- Check the service terms and robots.txt; stop if automated collection is prohibited or the applicable permission is unclear.
- Fetch only the pages and fields needed. Avoid aggressive concurrency, repeated downloads, and needless retries.
- Use a clear, truthful client identity where appropriate, and preserve the page URL and retrieval timestamp with your records.
- Handle redirects, errors, and layout changes explicitly. Do not treat a successful HTTP response as proof that the parsed data is correct.
When you need a screenshot of a page to document its appearance, a screenshot is not a substitute for structured data or permission to collect the underlying content. ScreenshotNeo is a website screenshot API and MCP server from Yorker Media; its capture options can remove consent banners, newsletter popups, and chat widgets before taking a shot. See ScreenshotNeo for details.
6. Validate before analysis or publication
A machine-readable file is not automatically well-defined. Consult the dataset description, data dictionary, format documentation, and stated limitations before interpreting fields. Federal open-data principles call for accessible, machine-readable formats and descriptions of data strengths, weaknesses, limitations, and processing needs (U.S. Federal Government Open Data Principles).
Practical validation checks
- Compare row or record counts with publisher documentation when available; investigate unexpected changes.
- Check field names, types, units, date formats, missing values, and known codes against documentation.
- Look for duplicate records, malformed rows, encoding issues, and inconsistent identifiers.
- Record the retrieval date, source URL, query or file name, and any cleaning transformations.
- Separate source values from your derived or cleaned values so decisions can be audited.
- Before publication, explain known coverage gaps or limitations that could affect the conclusion.
7. Troubleshoot common collection failures
The API returns 401 or 403
Check whether the endpoint requires a key, whether the key is correctly supplied, and whether the service grants access to that resource. A 403 can also reflect a service restriction; do not try to bypass it. Re-read the endpoint’s terms and authentication instructions.
The API returns 429
You have hit a rate limit or another request threshold. Follow the response’s rate-limit headers or Retry-After instruction, reduce request frequency, and avoid parallel retries. Check the current limits for that particular service.
Free tools Windows power users keep installed
One-click scans. No signup required.
The page has changed or the parser returns empty fields
Inspect the current page and its underlying documented data route. Page markup can change without a dataset changing. Update and test the parser against representative pages, and validate field counts and values before accepting a run.
The download is large, slow, or interrupted
See whether the publisher offers a narrower API query, partitioned download, or documented bulk process. Save progress safely, use the service’s recommended resume or pagination behavior, and avoid repeatedly restarting a large transfer without checking what has already arrived.
The data appears inconsistent with the catalog
Check coverage dates, update cadence, publisher notes, and version information. Preserve the retrieved file and metadata so you can distinguish a source update from a parsing or processing error.
8. Or skip the browser setup
For a screenshot—not a replacement for an API or dataset download—ScreenshotNeo can capture a URL in one GET request. It returns PNG, JPEG, WebP, or PDF; the example saves the response as WebP. Replace the example target URL with the page you need to document. Keep the access key private. See the ScreenshotNeo API documentation for request parameters and response details.
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpCookie banners, popups, and chat widgets are removed before the shot. Bot checks, blank pages, and failed loads are never billed; each response identifies the page verdict and billing status. Its MCP server lets AI agents use screenshot tools, and 1,000 screenshots a month are free with no card; paid plans start at $5 for 3,000. Sign up for ScreenshotNeo’s free plan.
9. Keep the collection reproducible
For recurring work, make the collection process explicit: save the publisher record, applicable terms, endpoint or file source, retrieval date, query parameters, and transformation steps. Recheck live documentation and terms when the source changes or before a new collection, because limits and access methods can change. For a project spanning multiple agencies, assess each service on its own rather than assuming one agency’s rules apply to another.
Frequently Asked Questions
Does public government data always have the same reuse license?
No. Dataset-level access and use information and the terms of the serving agency or service can differ; review both before collecting or reusing data.
Can I use robots.txt as permission to scrape a government site?
No. It communicates crawling guidance, but it does not replace the service’s terms, endpoint documentation, or other restrictions.
Are Data.gov API limits the same for every federal API?
No. The cited key limits concern api.data.gov credentials; individual services can set different limits, so check that API’s current documentation and response headers.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
