Free tools Windows power users keep installed
One-click scans. No signup required.
To route Screaming Frog SEO Spider through a proxy, open File > Settings > Proxy on Windows or Linux, or Screaming Frog SEO Spider > Settings > Proxy on macOS. Turn on Use Proxy Server, enter the proxy address and port, add credentials if your proxy requires them, save the change, and restart SEO Spider. The application supports one proxy only: when enabled, every HTTP and HTTPS request uses it; when disabled, none do. There are no per-site exceptions.
These steps and limitations come from Screaming Frog’s official SEO Spider Configuration guide. A proxy changes network routing, not your authorization to crawl a site or SEO Spider’s robots.txt behavior.
Where are the proxy settings in Screaming Frog?
The menu location depends on your operating system:
| Operating system | Menu path |
|---|---|
| Windows | File > Settings > Proxy |
| Linux | File > Settings > Proxy |
| macOS | Screaming Frog SEO Spider > Settings > Proxy |
If the menu does not match, check that you are changing SEO Spider’s own settings rather than your operating system’s global network settings. The proxy configured here applies to SEO Spider traffic.
How to configure the proxy
- Open the Proxy screen. Use the operating-system-specific path above.
- Enable Use Proxy Server. This activates proxy routing for SEO Spider.
- Enter the server address and port. Use the hostname or IP address and the listening port supplied by your network administrator or proxy provider.
- Add credentials when required. Enter the username and password requested by the proxy. Screaming Frog’s guide confirms that credentials can be supplied, but does not specify a particular authentication protocol in the proxy section.
- Apply the setting and restart SEO Spider. Screaming Frog states that a restart is required before the change takes effect.
After restarting, begin with a small, authorized crawl. This makes it easier to distinguish a proxy problem from a target-site block, robots.txt restriction, DNS issue, or general connectivity failure.
Understand the all-or-none proxy scope
SEO Spider supports one proxy server at a time. It does not provide a list of proxy profiles or exceptions for selected domains. With the proxy enabled, all HTTP and HTTPS traffic generated by the application goes through that server. With it disabled, traffic does not use the configured proxy.
- You cannot send one client site through a proxy and another site directly from the same SEO Spider configuration.
- A proxy that is unreachable can therefore disrupt every request in a crawl, not just requests to one domain.
- If only one target needs a proxy, plan for the application-wide effect before starting a large crawl.
These constraints are documented in Screaming Frog’s configuration guide. They also mean that a proxy is not a substitute for site-owner permission or a way to bypass access controls.
HTTPS certificate errors from an inspecting proxy
Corporate proxies sometimes inspect TLS traffic and re-sign the destination certificate with an organization-controlled certificate authority. If that authority is not trusted by SEO Spider, the TLS connection is rejected and HTTPS URLs can fail.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallRecognize the symptom
Typical signs include certificate or trust errors on HTTPS requests, even though the same URL works in a browser managed by your organization. In the certificate details, the issuer may be your company’s proxy or security appliance rather than the public certificate authority used by the website.
Add the organization-provided certificate
- Ask your IT or network administrator for the correct proxy certificate. Do not download or trust an unknown certificate from an unverified source.
- Open SEO Spider’s Trusted Certificates settings.
- Use Discover to identify a certificate whose issuer has been changed by the network proxy.
- Add the supplied certificate file. The guide lists
.crt,.pem,.cer, and.deras accepted formats. - Restart SEO Spider and retry a small HTTPS crawl.
The certificate must represent the organization’s legitimate trust authority. Adding an unrelated certificate can weaken connection security and will not correctly fix the proxy’s TLS chain.
Robots.txt, authorization and responsible crawling
SEO Spider respects robots.txt by default. The general guide documents an option in the licensed version to ignore robots.txt and places responsibility for that decision on the user: SEO Spider General.
Using a proxy does not alter robots.txt processing, grant permission to crawl, or make a restricted site public. Crawl only properties you own or are authorized to assess, and follow the target site’s published access requirements. If you operate the site being crawled and your security platform is blocking the crawler, Screaming Frog’s FAQ recommends allowlisting the crawler’s IP address and the “Screaming Frog SEO Spider” user agent in your CDN security settings. That is a site-owner configuration, not a technique for bypassing another organization’s controls.
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Scan for outdated or missing drivers - takes under a minute3Repair Windows errors before they cause bigger problemsRank #3
Troubleshoot a crawl that fails after proxy setup
SEO Spider cannot connect to any URL
- Check whether the proxy was enabled accidentally. Screaming Frog’s FAQ specifically recommends checking this condition. Disable Use Proxy Server, restart SEO Spider, and test again.
- Verify the address and port. A typo, wrong protocol, closed port, or outdated endpoint prevents all routed requests from reaching their destination.
- Confirm credentials. Re-enter the username and password supplied for this proxy. A browser login or operating-system credential does not necessarily configure SEO Spider.
- Test reachability from the same machine. Ask your network administrator whether the proxy accepts connections from the computer running SEO Spider and whether your account is allowed to use it.
Only HTTPS URLs fail
This pattern commonly indicates TLS inspection or an incomplete trust chain. Use the Trusted Certificates workflow described above and obtain the certificate from your organization’s administrator. If the proxy does not inspect TLS, ask the administrator to verify that HTTPS tunneling is permitted.
Some sites are blocked while others work
Because SEO Spider has no per-site proxy exceptions, a selective failure is more likely to involve the destination site, its CDN rules, DNS, robots.txt, rate limits, or the proxy’s own filtering policy. Compare a URL that works with one that fails, then check the proxy logs or network policy with the responsible administrator.
The crawl is blocked or rate-limited
A proxy does not guarantee access. If you control the target site, follow Screaming Frog’s FAQ guidance and allowlist the crawler IP and user agent in your CDN security settings: SEO Spider FAQ. If you do not control the site, stop and request authorization rather than attempting to evade its controls.
Changes appear to have no effect
Restart SEO Spider after every proxy change. The official configuration guide says the restart is required; continuing an already running session can leave it using the previous network state.
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Repair Windows errors before they cause bigger problemsFix Now →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Operational checklist before a large crawl
- Confirm the proxy hostname or IP address and port with the administrator.
- Verify whether authentication is required and that the supplied credentials are current.
- Decide whether routing every HTTP and HTTPS request through one proxy is acceptable.
- Determine whether TLS inspection is enabled and obtain the organization’s trust certificate if necessary.
- Run a small crawl against an authorized site after restarting SEO Spider.
- Record the proxy setting so it can be disabled when you return to a network that does not use it.
- Check robots.txt and your written crawl authorization independently of the proxy configuration.
Performance and reliability considerations
The cited Screaming Frog documentation does not publish a proxy-specific speed benchmark, success rate, or capacity figure. In practice, the proxy becomes another network dependency: its latency, filtering rules, authentication service, and availability can affect crawl behavior. Treat a failed crawl as a routing problem first, then isolate destination-site and robots.txt causes.
Because one setting covers all requests, a dedicated crawl window and a small test run are safer than changing the proxy during a production crawl. Keep the administrator’s support details available if certificate or authentication errors appear.
Or skip the browser setup
If your goal is to create screenshots rather than crawl a site, ScreenshotNeo provides a one-request website screenshot API. It is separate from Screaming Frog and does not configure SEO Spider’s proxy; it is an alternative when you need a captured page instead of an SEO crawl.
Use the documented API examples at ScreenshotNeo documentation:
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpimport requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
open("shot.webp", "wb").write(r.content)const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);ScreenshotNeo removes cookie banners, newsletter popups and chat widgets before capture. Bot checks, blank pages, failed loads and cache hits are not billed, and response headers identify the page verdict and billing status. Its MCP server lets AI agents use take_screenshot, get_page_info and capture_pdf. The Free plan includes 1,000 screenshots each month with no card; paid plans start at $5 for 3,000 shots. Sign up for the free plan.
Frequently Asked Questions
Can I configure more than one proxy in SEO Spider?
No. The documented configuration supports one proxy server, with no per-site exceptions.
Do proxy settings change robots.txt compliance?
No. Proxy routing and robots.txt behavior are separate settings; SEO Spider respects robots.txt by default.
Which certificate file types can Trusted Certificates import?
Screaming Frog lists .crt, .pem, .cer and .der files.
Who should provide a certificate for a corporate TLS-inspecting proxy?
Your organization’s IT or network administrator should provide the correct trust certificate.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
