Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Repair Windows errors before they cause bigger problemsFix Now →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Track competitor URLs by collecting each site’s sitemap files, preserving dated snapshots, and comparing what is newly listed, removed, or retained. Treat that comparison as a discovery signal—not proof that a page is live, new, deleted, crawled, or indexed. Verify important changes with a separate crawl.
How do I find a competitor’s sitemap?
- Set the scope. Record each competitor’s canonical host and any subdomains you intend to monitor. Keep hosts separate, including when a site redirects between HTTP and HTTPS or between
wwwand non-www. - Fetch the host’s robots.txt. Check
https://example.com/robots.txtand parse everySitemap:directive. A site may declare more than one sitemap location. - Follow each declared location. Determine whether it is a sitemap index that points to other sitemap files or a URL-set file containing page URLs. Follow index references recursively and save the full chain of file URLs.
- If no location is declared, try discovery fallbacks. You can check common paths such as
/sitemap.xmlor inspect publicly linked feeds, but label these as fallbacks rather than confirmed locations. Not every site uses the same structure.
Google documents sitemap declarations in robots.txt and the use of sitemap indexes to organize multiple files. Its guidance also cautions that sitemap inclusion does not ensure a URL will be crawled or indexed: Learn about sitemaps.
As an Amazon Associate I earn from qualifying purchases.
How can I extract all URLs from a sitemap index?
Fetch each index and child file, identify the XML structure, and extract each listed URL along with its source and metadata. For large collections, keep processing at the child-file level so a failed file does not erase the coverage you obtained from the others.
Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Repair Windows errors before they cause bigger problems3Scan for outdated or missing drivers - takes under a minuteDistinguish indexes from URL sets
A sitemap index lists sitemap files; a URL-set sitemap lists URLs. Follow every index reference until you reach URL sets, while recording which index led to each child. XML is widely used and can carry additional metadata for images, video, news, and localized alternatives. Google also supports RSS or Atom feeds and plain-text sitemaps; a text sitemap is a simpler URL list.
#1 Best Overall
Preserve provenance and raw values
For every extracted record, retain the competitor host, source sitemap URL, URL exactly as listed, collection timestamp, and lastmod value if present. Store the original URL even if you create a normalized comparison key. That lets you investigate changes in casing, query strings, trailing slashes, or redirects instead of silently discarding them.
Sitemap URLs should be fully qualified. XML files use UTF-8 and must escape special characters correctly. The protocol permits a date-only lastmod value such as 2026-10-04 or a fuller W3C datetime. See the Sitemaps protocol.
Respect file limits and partial failures
Google’s stated limit for a single sitemap is 50 MB uncompressed or 50,000 URLs; split larger collections and organize the child files in an index. The Sitemaps protocol separately sets limits of 50,000 sitemap files and 50 MB for a sitemap index. These are per-file limits, not a guarantee that every site uses the maximum. Record fetch errors per file and report partial coverage rather than treating an inaccessible child as empty.
How do I compare sitemap snapshots?
- Collect consistently. Save a dated snapshot of each successfully fetched file and the extracted records. Record HTTP status, fetch time, and parse failures.
- Deduplicate exact repeats. Identify the same URL appearing in multiple child files, but preserve the raw entries and their sources for diagnosis.
- Normalize with a documented rule. Apply the same comparison rule every run. Keep the listed URL alongside any normalized key so transformations do not hide query-string, casing, slash, or redirect differences.
- Classify the difference. Mark URLs as newly listed, removed from the sitemap, or retained. Describe these as sitemap changes, not confirmed page launches or deletions.
- Use metadata cautiously. Treat
lastmodas a useful signal only where the site maintains it accurately. The protocol defines it as the modification date of the linked page, not the generation date of the sitemap. Google says it relies on the field when it is consistently and verifiably accurate. Do not usechangefreqorpriorityas observed change evidence; Google ignores both.
A new sitemap entry may be an older page newly exposed in the file; a removed entry may still resolve. Google describes sitemap submission as a hint, not a guarantee of crawling: Build and Submit a Sitemap.
Does a sitemap show every page on a website?
No. A sitemap shows URLs the site has chosen to expose in that file or feed. It is not an authoritative inventory of every live, internally linked, crawled, or indexed page. Google’s example says a small site of about 500 pages or fewer may not need a sitemap if its pages are well linked and meet the other conditions in its guidance; that is an example, not a universal cutoff for competitor monitoring.
To assess coverage, compare sitemap URLs with a separate crawl of the site. Check URLs found only in the sitemap and URLs discovered by crawling but absent from it. Validate important candidates for HTTP status, redirect destination, canonical URL, robots directives, indexability, and whether internal links make the page discoverable. Sitebulb documents sitemap-versus-crawl comparisons and checks for non-indexable, disallowed, canonicalized, and broken URLs: Sitemaps hints. This describes documented capability, not a recommendation for every team or budget.
How do I monitor competitor websites at scale?
Plan for more than URL count
Runtime depends on the number of domains, index depth, number and size of child files, fetch latency, parsing errors, and any follow-up crawl—not just the total number of URLs. Track each sitemap file independently, report unavailable files, and avoid allowing one failed download to conceal partial results.
Schedule repeatable, polite collection
- Choose a collection interval based on how quickly the information matters to your work; there is no universally safe request rate established here.
- Use conservative concurrency and bounded retries, and record status and timestamps so transient errors are visible.
- Where supported, use HTTP validators or timestamps to avoid repeatedly downloading unchanged large files.
- Keep crawl settings and URL normalization consistent between runs, or changes in method may look like changes on the competitor site.
- Review each site’s published access terms and instructions. A public sitemap does not settle questions of permission, terms, privacy, copyright, or jurisdiction. For high-stakes or sensitive monitoring, obtain appropriate legal advice.
Report the limits with the findings
For each report, state the domains included, sitemap files found, files that failed, extraction date, and whether supplementary crawling was performed. Make clear that sitemap monitoring only covers URLs exposed through the files you successfully collected. Separate observed sitemap changes from page-content changes confirmed by fetching the pages.
Best Value
What should I look for in a monitoring workflow?
| Need | What to check |
|---|---|
| Discovery coverage | Robots.txt declarations, index traversal, clearly labeled fallback paths, and optional internal-link crawling. |
| Scale handling | Recursive file traversal, compressed XML handling, per-file status and failure reporting, and enforcement of sitemap and index limits. |
| Validation | HTTP status, redirects, canonical URLs, robots restrictions, indexability, and internal-link crawl comparisons. |
| Change quality | Raw URL preservation, reliable lastmod handling, and page-level content checks when the question is whether content changed. |
| Repeatability | Scheduled collection, consistent normalization, provenance, exports, and alerts tied to defined change categories. |
Troubleshooting sitemap extraction
- Robots.txt has no Sitemap line: Try common sitemap paths and public feeds as discovery fallbacks, then label the source and confidence. Absence of a directive does not establish that no sitemap exists.
- An index downloads but few URLs appear: Confirm that your parser follows its child sitemap locations recursively; an index contains file references rather than the page URL set.
- A child file fails: Record the failed URL, response status, and time; retry conservatively and report the run as partial until it succeeds.
- XML parsing fails: Check encoding, malformed XML, and escaped special characters. Preserve the failed source file for diagnosis rather than dropping it silently.
- Duplicates inflate counts: Compare exact raw URLs across child files, deduplicate for totals, and retain source membership to identify overlapping sitemap exports.
- A sitemap change does not match what the page does: Fetch the URL and inspect redirects, status, canonical, robots directives, and page content. Sitemap inclusion or omission alone cannot establish page state.
- Every run reports many changes: Check whether host redirects, URL normalization rules, query strings, or crawl settings changed. Keep the same comparison rules and preserve raw values.
Or skip the browser setup
Once sitemap monitoring identifies a URL worth inspecting, a screenshot can help review its rendered state. ScreenshotNeo is a website screenshot API and MCP server. Its one-call API can return a screenshot or PDF; adapt the target URL as needed:
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpSee the ScreenshotNeo API documentation for parameters and response details. Before capture, it accepts cookie or consent banners like a visitor and removes more than 60 known consent platforms, newsletter popups, and chat widgets; each step can be turned off. Bot checks or CAPTCHAs, blank pages, timeouts, failed loads, and cache hits are not billed, with response headers indicating the page verdict and billing status. Its MCP server offers take_screenshot, get_page_info, and capture_pdf for Claude, Cursor, and other MCP clients. The free plan includes 1,000 screenshots a month with no card; paid plans start at $5 for 3,000 shots.
Sign up for 1,000 free screenshots a month—no card required.
The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
