Crawl Lab

What Crawl Lab is

Crawl Lab is a free web-based SEO toolbox for examining server logs and crawler activity, including visits from AI bots. It accepts Apache and Nginx common or combined logs, including compressed .gz files, and reports crawled pages, crawl frequency, wasted crawl budget and bot errors. It recognizes crawlers such as GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, Bytespider and Amazonbot, and checks crawler authenticity against official IP ranges. Analysis runs in the browser; log files and IP addresses stay on the user’s machine and are not uploaded or stored. No registration is required. Streaming processing can handle several gigabytes or a full month of logs without a server-imposed size limit. The tool is aimed at technical SEO practitioners working on search and generative-engine visibility. Its findings show what crawlers fetched, but do not establish whether pages were indexed or how they rank. An llms.txt generator and an AI citation audit are listed as coming soon.

Who it suits

Crawl Lab suits technical SEO practitioners who want to examine server logs and AI crawler activity locally. It can help describe crawler fetches, but not determine indexing or rankings.

Limits

The Free plan has no paid tier or server-imposed size limit. Log analysis shows what crawlers fetched, but not whether pages were indexed or how they rank.

Getting started

1. Open the Crawl Lab website.
2. Choose a supported Apache or Nginx log file, including a compressed .gz file if needed.
3. Upload the log file for analysis in the browser.

Verdict

Crawl Lab offers local log analysis and crawler identification at no cost. Its output concerns what bots fetched, not indexing or ranking outcomes.