If a website blocks an Apify crawl, first find out whether the proxy connection is failing or the target site is rejecting the request. Test Apify Proxy, inspect the response from the target, then choose an eligible proxy group and decide whether your workflow needs a steady session IP or rotation. A proxy can change how requests reach a site; it cannot guarantee access or bypass every protection.
First determine what is being blocked
A failed crawl can mean the proxy could not connect, the proxy credentials or configuration are wrong, or the target site returned a block page, CAPTCHA, or incomplete content. Those problems need different fixes. Start by testing the proxy independently and checking the actual response your crawler received.
As an Amazon Associate I earn from qualifying purchases.
- Use Apify’s proxy status test to check whether Apify Proxy is available to your account and responding.
- Route a request to Apify’s browser-info endpoint through the proxy. Confirm that the request succeeds and inspect the apparent client IP and location.
- Retry the target request and inspect its status, response body, and rendered page. A site-branded denial, challenge, CAPTCHA, or missing content points to a target-side response; an Apify proxy error points to the connection or configuration.
Apify Documentation’s “Proxy” page says, “Apify Proxy lets you rotate IP addresses when scraping to avoid geographic blocking.” That describes a capability, not a promise that a site will allow a crawl.
Configure Apify Proxy for an Apify Actor
For an Actor, use the Apify SDK proxy configuration rather than embedding proxy credentials in crawler code. The examples below use JavaScript and Python SDK configuration paths. Pass the resulting configuration to the crawler; the exact crawler option name depends on the crawler and SDK version you use.
JavaScript
const proxyConfiguration = await Actor.createProxyConfiguration({ groups: ['AUTO'] });
const crawler = new CheerioCrawler({
proxyConfiguration,
// Other crawler options...
});Python
proxy_configuration = await Actor.create_proxy_configuration(groups=['AUTO'])
crawler = BeautifulSoupCrawler(
proxy_configuration=proxy_configuration,
# Other crawler options...
)Use a group available to your account. If you select a country or a specific group, confirm that the group and geography are supported for your account in the current Apify Console and documentation. SDK APIs and crawler integration details can vary by version; check the reference for the SDK release installed in your Actor.
Configure an external client
For clients running outside Apify, Apify documents the proxy host as proxy.apify.com on port 8000. Use the username parameters to select a group, country, or session as documented for your account, and provide the proxy password through a secret store or environment variable rather than source code. Apify warns that proxy usage is charged to the account and that the HTTP proxy password is sent unencrypted by the HTTP protocol. Keep credentials private and use an appropriate secure network path.
Consult Apify’s proxy configuration documentation for the current username syntax and examples; do not guess parameter names. A generic client setup looks like this, with placeholders that are not usable credentials:
HTTP_PROXY=http://USERNAME:[email protected]:8000
HTTPS_PROXY=http://USERNAME:[email protected]:8000Apify SDK and Console workflows can also use customer-supplied proxy URLs. That is a configuration option, not evidence that another provider will work for a particular target.
Choose a proxy group for the failure you see
Group availability, eligibility, and charging depend on your Apify account. Compare the observed target response with the continuity, geography, speed, and billing characteristics your task needs. Apify’s current account UI and documentation are the place to verify access and volatile pricing.
Rank #2
- # Unlimited Bandwidth to use
- # Endless list of countries to connect to worldwide!
- # Simple one click to connect
- # Super fast speed proxy
- # Proxy any apps and sites in any country
| Option | When it may fit | Trade-offs to consider |
|---|---|---|
| Datacenter | Ordinary proxy use where speed and relatively low cost matter. | Apify describes health checks, rotation, country selection, persistent sessions, and shared or dedicated groups. Shared IP activity from other users can contribute to blocks. |
| Residential | A task where residential IP characteristics are appropriate and account terms allow their use. | IPs are assigned by ISPs to homes or offices and pricing is traffic-based. Speed can vary across host devices, and sessions can end when a device disconnects. |
Unblocker (groups-UNBLOCKER) |
A task where Apify’s managed routing for common protections and CAPTCHA challenges is appropriate. | It has unit-based billing, does not provide a session parameter, and is not a guarantee of access. Check current pricing and availability in Apify’s documentation or account UI. |
Do not switch groups blindly. First establish that the proxy itself works; then try an eligible group that matches the target’s observed response and your session and geography needs.
Decide whether to keep a session or rotate IPs
A session keeps requests associated with the same IP for continuity. Rotation changes IPs. Prefer continuity for workflows that depend on a logged-in state or related sequence of requests; consider rotation when changing IP is appropriate to the task. Neither choice repairs broken credentials or guarantees that a target will accept a request.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →For workflows that need coordinated session and cookie or fingerprint management, Apify’s SessionPool can manage sessions. Unblocker does not provide a session parameter. For browser crawlers, inspect the browser and crawler lifecycle: multiple page requests may use the same IP during a browser session, so do not assume an IP changes on every individual request.
Interpret Apify proxy errors before retrying
Apify documents proxy response codes in the 590–599 range. Use the error as a diagnostic clue, then check credentials, settings, and connectivity rather than repeatedly retrying the same failing request.
| Code | What it indicates | What to check |
|---|---|---|
| 593 | DNS or configuration problem | Verify the proxy hostname, port, and client configuration. |
| 594–596 | Connection, refusal, or reset problems | Check network reachability and proxy settings; retry later only if the failure appears transient. |
| 597 | Upstream authentication failed | Confirm the Apify account’s proxy password and username parameters, and make sure credentials have not expired or been mistyped. |
| 599 | Generic upstream error | Inspect the response and configuration; if it persists, use Apify’s support and current documentation to investigate. |
A target-site challenge or denial is not the same thing as one of these proxy connection errors. If the proxy status test succeeds but the target returns a block page, changing connection settings alone may not address the target’s decision.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Reduce avoidable failures and cost
- Test with a small number of requests before launching a large crawl; verify both the proxy and the target response.
- Use the least complex group and session behavior that fits the task. Residential traffic and Unblocker have their own billing models; check current account pricing before scaling.
- Do not run tight retry loops against a site that is returning blocks, challenges, or repeated errors. Retries can increase load without fixing the underlying cause.
- Keep proxy credentials out of logs, public repositories, and client-side code. Treat them as account credentials because proxy use can be charged to your account.
- Follow the target site’s terms, applicable law, and the authorization you have to access its content. A proxy changes request routing; it does not establish permission.
Or skip the browser setup
If your goal is simply to capture a page as an image or PDF rather than crawl it through Apify, ScreenshotNeo offers a one-request screenshot API. For setup and options, see the ScreenshotNeo documentation.
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Repair Windows errors before they cause bigger problemsFix Now →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpScreenshotNeo accepts cookie or consent banners like a visitor and removes more than 60 known consent platforms, newsletter popups, and chat widgets before capture; each step can be turned off. Bot checks/CAPTCHAs, blank pages, timeouts, failed loads, and cache hits are not billed, and the response identifies the page verdict and billing status in headers. Its MCP server provides take_screenshot, get_page_info, and capture_pdf tools for AI agents. The free plan includes 1,000 screenshots per month with no card; paid plans start at $5 for 3,000.
Sign up for ScreenshotNeo’s free plan.
FAQ
Why is Apify blocked?
“Blocked” can refer either to a proxy connection/configuration error or to a target site rejecting the request. Check the Apify proxy status and error code, then inspect the target’s actual response to distinguish them.
Can a proxy guarantee that a site will not block my crawler?
No. IP quality and request limits matter, but anti-scraping protections can extend beyond IP blocking. A proxy group may be suitable for a task without guaranteeing access to any particular site.
Can I use my own proxy URLs with Apify?
Yes. Apify SDK and Console support customer-supplied proxy URLs. You still need to configure and validate the provider endpoint and credentials for your workflow.
Recommended Free Tools
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
