Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Repair Windows errors before they cause bigger problems3Scan for outdated or missing drivers - takes under a minuteYou can build a B2B crawler in Python with Scrapy, but replacing a $99-per-month scraping service does not automatically make the work cheaper. A maintainable crawler needs a narrow source allowlist, conservative request pacing, bounded retries, stable records, restartable storage, and monitoring. This guide shows how to structure those pieces—and why collecting business data and using it for outreach require separate review.
Decide what the crawler is allowed to collect
Start with a written scope, not a spider. List the specific sites or pages you are permitted to access, the business fields you need, how often you will refresh them, and how long you will retain the results. Keep the scope narrow: a crawler designed for a defined set of company pages is easier to operate and review than one that follows every link it encounters.
Define a record before writing selectors
A useful starting schema is:
company_namecompany_domainpublic_business_contact, if needed for the use casesource_urlretrieved_atvalidation_status
Keep provenance with each record so you can trace a value back to its page and retrieval time. Collect personal fields only after reviewing whether the specific collection and intended use are permitted. Public availability by itself does not settle that question.
Use a separate adapter for each source
Different sites have different markup, navigation, and access rules. Give each source its own Scrapy spider or adapter instead of assuming one set of selectors will work everywhere. Keep extraction separate from persistence: a markup change should be repairable and testable without overwriting or corrupting previously collected records.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
#1 Best Overall
Configure Scrapy for explicit policies and bounded retries
Scrapy 2.19.0 documents RetryMiddleware as enabled by default. Its documented RETRY_TIMES default is two additional attempts after the original request, and its default retryable HTTP status list includes 429, 408, and selected server errors. Treat those defaults as a starting point, not a production policy for every source.
Make important settings explicit in your project configuration:
ROBOTSTXT_OBEY = True
RETRY_ENABLED = True
RETRY_TIMES = 2
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
CONCURRENT_REQUESTS_PER_DOMAIN = 1
DOWNLOAD_DELAY = 1.0The values shown for concurrency and delay are conservative example settings, not a guarantee that a site will consider a given pace acceptable. Set them according to the source’s stated policies and observed behavior; reduce traffic or stop when a source signals throttling, blocking, or load. AutoThrottle adjusts delay using latency and target average concurrency per remote site, but Scrapy describes its concurrency target as an average it attempts to approach—not a hard maximum. Keep explicit concurrency ceilings as well.
Handle 429 as a signal to slow down
A 429 response means the source is throttling requests. Do not turn it into a rapid retry loop. Respect any retry timing the source supplies, reduce or pause requests to that domain, and resume only at a pace allowed by the source. Record the status and response details so repeated throttling is visible to an operator.
Retry only failures likely to be transient
Retries can help with temporary network problems and selected transient server responses. They cannot repair a removed page, a changed layout, a denied request, or invalid data. Keep retry counts bounded, record attempts, and log a request when its retries are exhausted. Scrapy’s request documentation also describes the per-request max_retry_times option in Request.meta, which can be used when an individual request needs a different cap.
A failed parse should be visible as a source or schema problem, not silently retried as though the network were at fault. Separate transport failures, HTTP statuses, extraction failures, and validation failures in logs or metrics.
Make crawling restartable and records reviewable
Resilience is more than retrying requests. A long-running job should make progress incrementally so a crash or deployment does not force a full restart. Make writes idempotent: processing the same page again should update or recognize the same business record rather than create an uncontrolled duplicate.
Choose a stable deduplication key
Use a canonical business identifier when the source provides one. If it does not, a normalized company domain may be a practical key, provided your use case accounts for subsidiaries, regional sites, and multiple brands sharing a domain. Keep the source URL as provenance even when records from multiple pages resolve to one company.
Validate before accepting a record
Check required fields, normalize values consistently, and send malformed or ambiguous records to a review queue instead of treating them as usable leads. Track validated records separately from raw pages fetched. Page count alone does not tell you whether the crawler is producing accurate, useful data.
- Persist results as they are processed, rather than holding an entire crawl in memory.
- Record the source, retrieval time, request outcome, and validation status.
- Keep failure logs that distinguish exhausted retries from parse or validation errors.
- Test source adapters against saved examples when page structure changes.
- Monitor per-domain request volume and throttle or pause a source when its behavior warrants it.
Choose between self-hosting and a managed API
A self-hosted Scrapy project offers control over source-specific parsing, validation, and operational policy. A managed scraping API may reduce some infrastructure and maintenance work, but it adds provider terms, data-handling questions, and its own subscription or usage charges. Compare the options against your own sources and volume rather than assuming either one is cheaper.
| Factor | Self-hosted Scrapy | Managed scraping API |
|---|---|---|
| Control | You own source-specific parsing and validation. | Depends on provider coverage, API options, and customization. |
| Operating effort | You maintain deployments, monitoring, retries, and repairs when sources change. | The provider may operate some infrastructure; confirm what remains your responsibility. |
| Cost structure | Engineering time, hosting, monitoring, and any proxy or browser needs. | Subscription and usage charges, plus any other applicable platform fees. |
| Observability | You can instrument source-specific requests, failures, and validation directly. | Review the provider’s execution and dataset reporting to see what you can inspect. |
| Data governance | You choose where your own system processes and stores data. | Confirm processing location, retention, contractual terms, and whether the provider may handle the intended fields. |
As one vendor example, Scrapy.io’s pricing page displayed Starter at $19 per month plus pay-as-you-go usage and Growth at $129 per month plus usage when checked on October 5, 2026. These are vendor-listed prices that may change, not a like-for-like comparison with a $99-per-month benchmark. Its product pages describe Python SDK and direct HTTP API use, along with executions, datasets, and schedules; verify current features and terms with the provider before deciding.
To estimate the self-hosted option, count the engineering and operations time needed for your specific sources, plus hosting and any extra services. For a managed option, include the subscription, expected usage charges, and the cost of checking provider coverage and governance terms. A small crawl with stable sources can favor a different choice than a high-volume workflow with frequent source changes.
Recommended Free Tools
Review data collection separately from outreach
Crawler mechanics do not establish whether a particular collection or marketing workflow is lawful. The answer can depend on jurisdiction, the exact fields, the source, storage and retention, recipients, and how the data will be used. Before collecting personal information or contacting people, obtain jurisdiction-specific legal review of the intended workflow.
Quick Recap
- Identify which fields are business information and which may identify a person.
- Review source terms and access restrictions as well as applicable privacy and marketing rules.
- Document where data is stored, who can access it, how long it is retained, and whether a vendor processes it.
- Assess collection and outreach as distinct uses rather than assuming permission for one covers the other.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
