Crawler Log Analyser is a free, open-source tool for inspecting crawler activity on self-hosted static sites that run nginx. It reads nginx access logs in combined or extended seo_crawl format, including rotated .gz files, and separates search and AI crawlers. From their observed responses, it assesses Googlebot and broader search-crawler health. It can also identify issues around a deployment that affect crawlers, such as robots.txt failures, bursts of content 404 responses, and crawl gaps after 404s. Other checks look for user-agent spoofing on sitemap paths and changing bot identity claims from one IP address. The tool produces text, Markdown, or JSON reports, and can return a non-zero exit code when it finds critical issues, supporting cron and CI workflows. Its maker describes it as a single-file Python script with no telemetry, dashboards, database, external services, or external dependencies. It runs on Linux and requires Python 3.11 or later along with nginx logs. The project is licensed under AGPL-3.0-or-later, which permits commercial use and internal modification; modifications used to provide a network service must make corresponding source available. The project is at v1.10.2, with feature work frozen through 31 August 2026; bug and security fixes remain welcome.
Who it is for
It suits operators of self-hosted static sites who want to inspect actual search and AI crawler activity in nginx logs. Linux users who can run Python 3.11 or later and prefer reports for cron or CI workflows are a particularly close fit.
What is good
- Reads combined and extended seo_crawl nginx logs, including rotated .gz files.
- Classifies search and AI crawlers and assesses crawler health.
- Flags several crawler-visible issues around deployments.
- Produces text, Markdown, and JSON reports.
- Can signal critical findings with a non-zero exit code.
- Single-file script with no telemetry or external dependencies.
What to know first
- Linux only, with no stated Apache or IIS support guarantee.
- Requires Python 3.11 or later and nginx logs.
- No Slack, Prometheus, Elasticsearch, or SaaS integrations.
- Feature work is frozen through 31 August 2026.
Verdict
Choose Crawler Log Analyser if you operate a self-hosted static site on nginx and need crawler-focused log reports or CI signals. Look elsewhere if you need Apache or IIS support, hosted dashboards, or integrations such as Slack, Prometheus, or Elasticsearch.
Get started with Crawler Log Analyser
- Open the project website on GitHub.
- Use a Linux environment with Python 3.11 or later.
- Provide nginx access logs in combined or extended seo_crawl format; rotated .gz logs are supported.
- Choose text, Markdown, or JSON output for reports.
- Use the non-zero exit behavior for critical findings in cron or CI workflows.
Limits to know first
The tool requires Python 3.11 or later and nginx logs; Apache and IIS support is not guaranteed. It has no Slack, Prometheus, Elasticsearch, or SaaS integrations.
Questions about Crawler Log Analyser
How much does Crawler Log Analyser cost?
It is free and open source.
What operating system does it support?
Linux is the listed platform.
What logs can it read?
It parses nginx access logs in combined or extended seo_crawl format, including rotated .gz logs.
Does it integrate with Slack or Prometheus?
No. Its stated integration boundary is JSON output, and it has no Slack, Prometheus, Elasticsearch, or SaaS integrations.
What license does it use?
It uses AGPL-3.0-or-later. Commercial use and internal modification are permitted; network-service modifications must make corresponding source available.
Who makes Crawler Log Analyser?
SpeyTech makes the project and lists its location as Inverness, Scottish Highlands, UK.
Compared on SEO log file analysis tools
- Crawler segmentation
- Yes
- Search Console integration
- No
