Crawlr

Free plan APIWeb

What Crawlr is

Crawlr is a website crawler and technical SEO audit tool for finding site issues, crawl inefficiencies and architecture problems. It discovers URLs through internal links, sitemaps, canonicals and JavaScript. Crawl controls include robots.txt compliance, rate limiting, include and exclude rules, and crawl depth. Its analysis covers response codes, indexability, metadata, headings, word counts, and internal and external links. Results can be viewed in interactive graphs, directory trees, 3D site explorations and content-cluster visualizations. Crawls can be scheduled daily, weekly or on custom schedules, with email completion notifications. The API can start crawls, report status, return structured JSON results, expose historical data and send webhook events. The Free plan includes 500 tokens per month, one concurrent crawl, CSV and JSON exports, and 15-day crawl history. Starter costs $8 a month, billed yearly, and includes 5,000 tokens per month and five concurrent crawls. Unauthenticated sandbox crawls are rate limited and capped at 25 pages.

Who it suits

Crawlr suits people auditing site structure and technical SEO, from single-domain users to teams running larger crawl workflows. Its published plans describe options for portfolios, growing teams and high-volume pipelines.

What it does well

Finds URLs through links, sitemaps, canonicals and JavaScript.
Analyzes indexability, metadata, headings and links.
Offers scheduled crawls with email completion notifications.
Free plan includes CSV and JSON exports.
API supports structured JSON results and webhooks.

What to know first

Free plan includes 500 tokens per month.
Free plan permits one concurrent crawl.
Unauthenticated sandbox crawls are limited to 25 pages.
Starter is billed yearly at $8 a month.

Limits

The Free plan has 500 tokens per month, one concurrent crawl, and 15-day crawl history. Unauthenticated sandbox crawls are rate limited and have a maximum of 25 pages.

Plans

Plan	Billed and limits	Price
Free	monthly · 500 tokens/month · 1 concurrent crawl · 500 pages/crawl · 15-day crawl history · full crawling · SEO intelligence and DIQ score · CSV/JSON exports	Free
Starter	yearly, 2 months free · 5,000 tokens/month · 5 concurrent crawls · 5,000 pages/crawl · 3-month crawl history · scheduled crawls · PageSpeed Insights · 8 visualizations	$8 a month
Standard	yearly, 2 months free · 35,000 tokens/month · 25 concurrent crawls · 10,000 pages/crawl · 1-year crawl history · LLM-ready markdown · advanced indexability · priority queues	$73 a month
Growth	yearly, 2 months free · 100,000 tokens/month · 100 concurrent crawls · 25,000 pages/crawl · 1-year crawl history · webhook notifications beta · bulk exports · dedicated account manager · custom crawl frequency	$291 a month
Enterprise	custom pricing · Unlimited pages · custom crawl logic · dedicated resources · SLA	On request

As crawlr.ai lists them, read 15 Oct 2026. The maker's own page has the final word.

Getting started

Open https://www.crawlr.ai/ in a browser.
Choose the Free plan or a paid plan; Starter is $8 a month billed yearly.
Set crawl options such as robots.txt compliance, rate limits, URL rules, and crawl depth.
Start a crawl through the web tool or API.
Review the SEO analysis and export CSV or JSON results on the Free plan.

Verdict

Crawlr combines configurable crawling, technical SEO analysis, visualizations and API access. Compare plan limits with the size and frequency of the crawls you need.

Questions

Does Crawlr have a free plan?

Yes. The Free plan costs nothing and includes 500 tokens per month, one concurrent crawl, and CSV and JSON exports.

What does Starter cost?

Starter is listed at $8 a month, billed yearly with two months free.

Which platforms are available?

Crawlr lists web and API access.

What can the API do?

It can start crawls, check status, return structured JSON results, access historical data, and send webhook events.

How many pages can an unauthenticated sandbox crawl cover?

Sandbox crawls without authentication are rate limited and capped at 25 pages.

What integrations are listed?

Google PageSpeed Insights is listed for Starter, and webhook notifications (Beta) are listed for Growth.

Facts from Crawlr's own pages

AI-oriented output: Crawlr provides token-aware markdown summaries intended for LLM and RAG ingestion. · crawlr.ai, 15 Oct 2026
API: The API supports starting crawls, checking status, retrieving structured JSON results, accessing historical data, and webhook events. · crawlr.ai, 28 Sept 2026
API access: Yes · crawlr.ai, 20 Sept 2026
API capabilities: The API can start crawls, poll status, retrieve JSON results, access historical crawls and emit webhook events. · crawlr.ai, 15 Oct 2026
API design: The maker describes a REST API using API-key authentication, standard HTTP methods, documented JSON structures, rate limits and descriptive errors. · crawlr.ai, 15 Oct 2026
Company listing: A LinkedIn page named Crawlr Labs describes a privately held technology company with 11-50 employees focused on competitor monitoring; the page does not identify crawlr.ai's SnappyTech maker. · linkedin.com, 15 Oct 2026
Crawl controls: Crawl settings include robots.txt compliance, rate limiting, include and exclude rules, and crawl depth controls. · crawlr.ai, 28 Sept 2026
Crawl graphs: Interactive 2D force graphs, 3D graphs, tree views and filtered views visualize URL relationships and site structure. · crawlr.ai, 15 Oct 2026
Data collected: The service collects account details, billing information, usage logs, device information and crawl data such as URLs, content, metadata, links, status codes and performance metrics. · crawlr.ai, 15 Oct 2026
Data retention: Personal data is retained while an account is active or needed for service delivery, while crawl-data retention follows the subscription plan. · crawlr.ai, 15 Oct 2026
Deployment: The official pages describe Crawlr as cloud-based and provide no self-hosting option. · crawlr.ai, 15 Oct 2026
DIQ score: Its Domain Intelligence Quotient combines indexability, technical health, content quality, link structure and mobile or speed signals into a score from 0 to 100. · crawlr.ai, 15 Oct 2026
Documentation depth: The API page provides endpoint-level capabilities, data-format descriptions, use cases and authentication, rate-limit and error-handling notes, but no separate public SDK is stated. · crawlr.ai, 15 Oct 2026
Exports: The Free plan includes CSV and JSON exports; paid plans add LLM-ready markdown exports and higher-tier bulk export capabilities. · crawlr.ai, 28 Sept 2026
Free plan: Yes · crawlr.ai, 20 Sept 2026
Full-site crawling: It crawls entire websites, including subdomains, parameterized URLs and XML-sitemap URLs. · crawlr.ai, 15 Oct 2026
HTTP analysis: The crawler tracks 2xx, 3xx, 4xx and 5xx responses and reports redirect chains, loops, soft 404s and broken links. · crawlr.ai, 15 Oct 2026
Independent listing status: G2 says Crawlr has not yet been reviewed and lacks enough reviews to provide buying insight. · g2.com, 15 Oct 2026
Independent pricing availability: G2 states that pricing details are unavailable on its listing and directs buyers to the vendor site. · g2.com, 15 Oct 2026
Independent product description: G2 describes Crawlr as a free web-scraping tool for extracting and managing website data, which differs from crawlr.ai's technical-SEO positioning. · g2.com, 15 Oct 2026
Independent seller identity: G2 attributes the listed Crawlr product to Crawlr Labs. · g2.com, 15 Oct 2026
Indexability analysis: It checks noindex, nofollow and canonical directives and flags conflicts such as noindex combined with another canonical target. · crawlr.ai, 15 Oct 2026
Integration: Google PageSpeed Insights is listed as the named third-party integration for performance analysis. · crawlr.ai, 15 Oct 2026
Integrations: The pricing page lists Google PageSpeed Insights for the Starter plan and webhook notifications (Beta) for Growth. · crawlr.ai, 28 Sept 2026