The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →To capture every page listed in a sitemap, parse its XML (including any nested sitemap files), deduplicate the page URLs, then visit each URL with one Selenium browser session and save a separate PNG. The script below writes a CSV manifest and continues after individual page failures. A sitemap is only an input list: its presence does not guarantee that a URL is reachable or that Google will crawl it.
What this script does—and what it does not
The workflow reads a sitemap URL, follows sitemap indexes, and captures the current browser window for each page URL. Each screenshot gets a distinct filename, and the CSV records the listed URL, final URL when available, output path, capture time, and any error.
As an Amazon Associate I earn from qualifying purchases.
Selenium’s documented save_screenshot() method saves the current window as a PNG; this script does not claim to produce a full-page image. The sitemap format and limits described by Google allow up to 50,000 URLs or 50 MB uncompressed per sitemap file, with sitemap indexes available for larger collections. Google also cautions that submitting a sitemap is a hint, not a guarantee that it will fetch the file or use its URLs for crawling. See Google’s sitemap documentation.
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Scan for outdated or missing drivers - takes under a minute3Repair Windows errors before they cause bigger problemsInstall the requirements
Use Python 3.10 or later with the Selenium Python bindings. Install Selenium and Requests:
#1 Best Overall
- 14" diagonal, 1366x768 resolution, HD BrightView LED, Glossy NON-TOUCH Display
python -m pip install selenium requestsCurrent Selenium documentation lists support for browsers including Chrome, Edge, Firefox, and Safari. This example uses Chrome. Selenium Manager generally handles driver installation for supported platforms, so a separate driver download is not normally needed. Check the current Selenium Manager documentation and Python API documentation for your platform and browser.
Save and run the batch script
Save this as sitemap_screenshots.py. It accepts either a sitemap URL or a local XML file, handles <urlset> and <sitemapindex>, and optionally discovers a sitemap declared in a site’s robots.txt when given a site URL. The script is intended for public, unauthenticated XML sitemaps and pages you are permitted to access.
import argparse
import csv
import hashlib
import re
import time
from datetime import datetime, timezone
from pathlib import Path
from urllib.parse import urlparse
import xml.etree.ElementTree as ET
import requests
from selenium import webdriver
from selenium.common.exceptions import WebDriverException
OUTPUT = Path("screenshots")
MANIFEST = OUTPUT / "manifest.csv"
USER_AGENT = "SitemapScreenshotBatch/1.0"
def fetch_bytes(location):
"""Read a local file or retrieve a remote sitemap/robots.txt."""
if location.startswith(("http://", "https://")):
response = requests.get(location, timeout=30, headers={"User-Agent": USER_AGENT})
response.raise_for_status()
return response.content
return Path(location).read_bytes()
def absolute_http_url(value):
"""Return an absolute HTTP(S) URL, or None for unusable locations."""
value = (value or "").strip()
parsed = urlparse(value)
if parsed.scheme not in ("http", "https") or not parsed.netloc:
return None
return value
def discover_sitemap(site_url):
"""Use the first Sitemap: declaration in robots.txt, if present."""
parsed = urlparse(site_url)
if parsed.scheme not in ("http", "https") or not parsed.netloc:
raise ValueError("For discovery, provide a complete site URL such as https://example.com")
robots_url = f"{parsed.scheme}://{parsed.netloc}/robots.txt"
response = requests.get(robots_url, timeout=30, headers={"User-Agent": USER_AGENT})
response.raise_for_status()
for line in response.text.splitlines():
key, separator, value = line.partition(":")
if separator and key.strip().lower() == "sitemap":
candidate = absolute_http_url(value)
if candidate:
return candidate
raise ValueError(f"No Sitemap: declaration found in {robots_url}; pass a sitemap URL or local file")
def parse_sitemap(location, seen_sitemaps=None):
"""Recursively collect page locations from sitemap XML and index files."""
if seen_sitemaps is None:
seen_sitemaps = set()
if location in seen_sitemaps:
return []
seen_sitemaps.add(location)
content = fetch_bytes(location)
root = ET.fromstring(content)
kind = root.tag.rsplit("}", 1)[-1].lower()
found = []
if kind == "sitemapindex":
for node in root:
if node.tag.rsplit("}", 1)[-1].lower() == "sitemap":
loc = next((child.text for child in node
if child.tag.rsplit("}", 1)[-1].lower() == "loc"), None)
child_url = absolute_http_url(loc)
if child_url:
found.extend(parse_sitemap(child_url, seen_sitemaps))
elif kind == "urlset":
for node in root:
if node.tag.rsplit("}", 1)[-1].lower() == "url":
loc = next((child.text for child in node
if child.tag.rsplit("}", 1)[-1].lower() == "loc"), None)
page_url = absolute_http_url(loc)
if page_url:
found.append(page_url)
else:
raise ValueError(f"Unrecognized sitemap XML root element: {root.tag}")
return found
def unique_in_order(urls):
seen = set()
result = []
for url in urls:
if url not in seen:
seen.add(url)
result.append(url)
return result
def filename_for(index, url):
parsed = urlparse(url)
slug = re.sub(r"[^a-zA-Z0-9._-]+", "_", parsed.path.strip("/"))[:70] or "home"
host = re.sub(r"[^a-zA-Z0-9._-]+", "_", parsed.netloc)
digest = hashlib.sha256(url.encode("utf-8")).hexdigest()[:10]
return f"{index:05d}_{host}_{slug}_{digest}.png"
def capture(urls, ready_selector=None, page_load_timeout=45, script_timeout=30):
OUTPUT.mkdir(parents=True, exist_ok=True)
results = []
driver = webdriver.Chrome()
try:
driver.set_page_load_timeout(page_load_timeout)
driver.set_script_timeout(script_timeout)
for index, url in enumerate(urls, start=1):
path = OUTPUT / filename_for(index, url)
row = {
"listed_url": url,
"final_url": "",
"screenshot_path": str(path),
"captured_at_utc": datetime.now(timezone.utc).isoformat(),
"saved": False,
"error": "",
}
try:
driver.get(url)
row["final_url"] = driver.current_url
if ready_selector:
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
WebDriverWait(driver, 20).until(
EC.presence_of_element_located((By.CSS_SELECTOR, ready_selector))
)
row["saved"] = bool(driver.save_screenshot(str(path)))
if not row["saved"]:
row["error"] = "WebDriver returned false when saving the screenshot"
except Exception as exc:
try:
row["final_url"] = driver.current_url
except WebDriverException:
pass
row["error"] = f"{type(exc).__name__}: {exc}"
results.append(row)
print(f"{index}/{len(urls)} {'OK' if row['saved'] else 'FAILED'} {url}")
finally:
driver.quit()
with MANIFEST.open("w", newline="", encoding="utf-8") as file:
writer = csv.DictWriter(file, fieldnames=[
"listed_url", "final_url", "screenshot_path", "captured_at_utc", "saved", "error"
])
writer.writeheader()
writer.writerows(results)
return results
def main():
parser = argparse.ArgumentParser(description="Capture a PNG for each URL in a sitemap.")
parser.add_argument("source", help="Sitemap URL, local XML file, or site URL for robots.txt discovery")
parser.add_argument("--discover", action="store_true", help="Treat source as a site URL and read its robots.txt")
parser.add_argument("--ready-selector", help="Optional CSS selector that must appear before capture")
args = parser.parse_args()
sitemap = discover_sitemap(args.source) if args.discover else args.source
urls = unique_in_order(parse_sitemap(sitemap))
if not urls:
raise SystemExit("No valid page URLs were found in the sitemap.")
print(f"Found {len(urls)} unique page URLs")
rows = capture(urls, ready_selector=args.ready_selector)
failures = sum(not row["saved"] for row in rows)
print(f"Finished: {len(rows) - failures} saved, {failures} failed. Manifest: {MANIFEST}")
if __name__ == "__main__":
main()
Run it with an explicit sitemap URL:
python sitemap_screenshots.py https://example.com/sitemap.xmlOr let it look for a Sitemap: declaration in the site’s robots.txt:
Free tools Windows power users keep installed
One-click scans. No signup required.
python sitemap_screenshots.py https://example.com --discoverA site can publish multiple sitemaps, and the script follows child sitemap URLs when the supplied XML is an index. For a private or local sitemap, pass its file path instead. XML parsing errors, HTTP errors retrieving sitemap files, or an unsupported root element stop sitemap collection; page navigation and screenshot errors are recorded per URL so the rest of the batch can continue.
How the URL and file handling works
Indexes, absolute URLs, and duplicates
Google’s sitemap format uses absolute URLs. The parser accepts only HTTP and HTTPS locations and recursively reads child sitemap files. It strips XML namespaces when checking element names, so the usual namespaced sitemap XML is accepted. URL deduplication preserves the first occurrence and its order; URLs that differ by query string or fragment remain distinct because they are not necessarily equivalent for the target site.
Rank #2
- 256 GB SSD of storage.
- Multitasking is easy with 16GB of RAM
- Equipped with a blazing fast Core i5 2.00 GHz processor.
This deliberately does not rewrite listed URLs or remove tracking parameters. If you want canonical-page-only captures, apply a site-specific normalization policy and preserve the original URL separately in the manifest. If a sitemap index contains a child file that is unavailable, collection raises an error rather than silently pretending that the batch is complete.
Distinct names and an auditable manifest
Filenames combine a sequential number, host, sanitized path, and a short SHA-256 digest of the complete URL. The digest reduces collisions for similar paths, while the index keeps output names distinct even if two source URLs normalize to a similar slug. The CSV records the listed URL separately from the final URL after redirects, which is useful when diagnosing unexpected destinations.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallThe timestamp is recorded when each URL is processed, not as a claim about an independently synchronized capture clock. The output directory is created automatically. Keep the manifest with the screenshots if you need to trace an image back to its source.
Wait for the right page state
driver.get() returning does not prove that a JavaScript-heavy page has finished rendering, that lazy images have loaded, or that the content you care about is visible. The optional --ready-selector waits for a CSS selector to appear before taking the screenshot:
python sitemap_screenshots.py https://example.com/sitemap.xml --ready-selector "main article"Choose a selector that signals readiness for the pages in your batch. The example uses a 20-second Selenium wait for that selector; the page-load and script timeouts are set separately in the script. A fixed sleep is simpler but is not a reliable readiness test: it can waste time on fast pages and still be too short on slow ones. A selector’s presence also does not guarantee that animations have stopped or every image has loaded, so visual-comparison workflows may need site-specific checks.
Rank #3
- WINDOWS 11 | STABLE PERFORMANCE: Powered by Intel Celeron N4020 processor and Windows 11 system, this laptop delivers stable performance for everyday computing tasks. It supports web browsing, online learning, document editing, email communication, and basic office work with optimized power efficiency, providing a practical and reliable experience for essential daily use for daily use.
- 15.6” FHD IPS DISPLAY: Features a 15.6-inch Full HD IPS display with narrow bezels, offering wider viewing angles and clearer image details compared to standard panels. The improved screen-to-body ratio enhances visual experience for study, reading, document work, and video playback, making it suitable for both productivity and entertainment use.
- 4GB DDR4 + 128GB eMMC STORAGE: Equipped with 4GB DDR4 memory and 128GB eMMC storage for everyday basics such as browsing, documents, email, and online learning platforms. The built-in TF card slot supports storage expansion up to 1TB, giving you more flexibility for files, photos, videos, and daily documents. TF card not included.
- CONNECTIVITY & PORTS: Includes 1× TF card slot, 2× USB 3.2 Gen1 ports, and 2× full-featured Type-C ports (USB 3.2 Gen1). The Type-C ports support data transfer, charging, and video output, enabling flexible connection with external devices such as monitors, storage, and peripherals for daily work and study use.
- LIGHTWEIGHT DESIGN | ONLINE COMMUNICATION: Designed with a slim, portable profile, this laptop is easy to carry for school, commuting, and travel. A built-in 1MP front camera supports online classes, video meetings, remote communication, and everyday conferencing. The 3300mAh battery works with the low-power system design to support practical daily use, while thermal optimization helps maintain quieter operation during extended tasks.
Selenium’s documented screenshot workflow navigates to a page and calls driver.save_screenshot('./image.png'). It captures the current window, not automatically the entire document. Set the browser window or viewport consistently if screenshots will be compared across runs; the script leaves Chrome’s default window size unchanged.
Reuse one browser or start a new one per page?
| Approach | Advantages | Trade-offs |
|---|---|---|
| Reuse one session (the script’s default) | Avoids browser startup for every URL and generally uses less overhead for a serial batch. | Cookies, local storage, and page state can carry between pages. Restart periodically or isolate pages if state changes the results. |
| Start a fresh session per URL | Provides stronger separation from cookies and browser state left by the previous page. | Repeated startup costs time and system resources. Add explicit cleanup for every browser instance. |
These are operational trade-offs, not guarantees about a particular site’s behavior. Begin with a reused session for ordinary public pages; choose isolation when state pollution is observed or when the pages require different authentication contexts.
Performance, scale, and repeatability
- Budget the run. Browser navigation is much heavier than fetching sitemap XML. A sitemap file can list up to 50,000 URLs under Google’s documented limit, but that does not mean a single machine can capture them quickly or store the resulting images comfortably. Estimate runtime and disk capacity on a small representative batch first.
- Keep traffic controlled. The script visits pages serially and does not start unbounded browsers. Large batches can still load a site heavily; use an agreed rate limit or split the run into smaller batches where appropriate.
- Record the environment. For visual comparisons, keep Selenium and browser versions, viewport, locale, timezone, and authentication state consistent. Current Selenium browser support and setup guidance are documented in the Selenium documentation.
- Plan storage. PNG file sizes vary with page content and viewport. Check output size during a small pilot and make sure the destination has room before running a large sitemap.
- Use controlled concurrency only when needed. Parallel browsers may reduce wall-clock time but increase CPU, memory, network traffic, and pressure on the target site. Set a finite worker count and respect the site’s access policies rather than launching a browser for every URL at once.
Troubleshooting common failures
| Symptom | Likely cause | What to do |
|---|---|---|
| Chrome fails to start or Selenium reports a driver/session error | Browser installation, platform support, or Selenium Manager setup is unavailable or misconfigured. | Install a supported browser, upgrade the Selenium Python package, and consult the Selenium Manager documentation for the platform. Confirm the browser can launch outside the batch script. |
robots.txt discovery finds no sitemap |
The file is missing, inaccessible, or has no Sitemap: line. |
Pass the sitemap URL directly or provide a local XML file. Discovery in this script uses the first declaration only. |
| XML parse error or unrecognized root | The supplied file is not a well-formed XML sitemap, or it is HTML, plain text, or compressed data rather than XML bytes the script can parse. | Verify the URL returns sitemap XML, not an error page; decompress compressed files before parsing or adapt the input handling for the actual format. This example does not implement plain-text or gzip sitemap parsing. |
| Some page rows show navigation timeout | A page is slow, blocked, inaccessible, or does not complete navigation within the configured timeout. | Check the URL in a browser, increase page_load_timeout for known slow pages, or use a site-specific readiness strategy. The per-page failure is retained in the manifest. |
| Screenshot is blank or content is missing | The page may be empty, JavaScript may not have rendered, images may be lazy-loaded, or bot protection may have intervened. | Inspect the page and its final URL, wait for a meaningful selector, and confirm the target permits automated access. A sitemap listing does not establish that a browser can access the page. |
| Images are clipped or only part of a long page appears | The WebDriver call captures the current window rather than promising full-page output. | Set a deliberate viewport for consistent window shots. If you need full-document captures, implement and validate a separate browser-specific approach rather than treating this PNG call as full-page. |
| Later screenshots differ from earlier ones unexpectedly | Cookies, local storage, personalization, time-dependent content, viewport, or locale may vary. | Use a controlled browser profile or restart sessions at chosen intervals, and record the relevant environment settings in your own run metadata. |
| Manifest is absent after an interruption | The script writes the CSV after the capture loop; a hard process termination can happen before it is written. | For long runs, adapt the script to append each result to the CSV as it completes or checkpoint results periodically. |
Or skip the browser setup
If you do not need a custom Selenium workflow, ScreenshotNeo can return a screenshot or PDF from one GET request. Its clean-shot flow accepts cookie or consent banners like a visitor and removes 60+ known consent platforms, newsletter popups, and chat widgets before capture; each step can be turned off. Bot checks or CAPTCHAs, blank pages, timeouts, failed loads, and cache hits are not billed, and response headers report the page verdict and billing status. ScreenshotNeo also has an MCP server with take_screenshot, get_page_info, and capture_pdf tools for AI agents.
For one URL, the cURL request is:
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://example.com -o shot.webpSee the ScreenshotNeo API documentation for the API options. The Free plan includes 1,000 screenshots per month with no card; paid plans start at $5 for 3,000. Sign up for 1,000 free screenshots a month, with no card.
Frequently Asked Questions
Does this script capture every URL in a sitemap index?
Yes, it recursively follows child sitemap files referenced by a sitemap index and collects URLs from their URL sets. A child file that cannot be fetched or parsed stops sitemap collection.
Recommended Free Tools
Can I save screenshots as JPEG or WebP with Selenium’s screenshot call?
The Selenium Python screenshot API used here saves PNG files. Converting the output to another image format requires a separate image-processing step.
Can I use it for authenticated pages?
Not as written: the example opens a fresh Chrome session without logging in. Add a site-authorized authentication flow or profile deliberately, and protect any credentials you use.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
