The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Use the headers argument on scrapy.Request when one request needs custom values. Use DEFAULT_REQUEST_HEADERS in settings.py for project-wide defaults. Scrapy’s default-header middleware fills only headers that are missing, so a value supplied on an individual request wins. Cookies, Referer, and request fingerprints have separate behavior that you must account for.
The examples below follow the current Scrapy 2.19.0 documentation. Check the linked API pages if you maintain a project across Scrapy releases.
Choose the right place to set a header
| Need | Use | What happens |
|---|---|---|
| A different value on one request | scrapy.Request(..., headers={...}) |
The value is attached at the call site and takes precedence over a project default. |
| The same fallback values throughout a project | DEFAULT_REQUEST_HEADERS |
DefaultHeadersMiddleware adds configured values only when the request does not already contain that header. |
| Cookie state managed by Scrapy | The request’s cookies argument |
Cookie middleware understands these cookies; a raw Cookie header is not treated as managed cookie state. |
Control or limit Referer |
Referer middleware, REFERER_POLICY, and request metadata |
Middleware can derive a Referer from the response that created a request, replacing a configured default in many cases. |
Scrapy’s request and response API is documented at the Requests and Responses reference. Its settings reference lists the built-in default values and middleware settings at the Scrapy settings documentation.
Add headers to one request
Pass a dictionary-like mapping to headers when creating the request. This is the most precise option when only one URL, endpoint, or API call needs a special value.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Fix the driver behind crashes, sound loss and screen glitches3Clear out junk files and repair common Windows errors#1 Best Overall
import scrapy
class ExampleSpider(scrapy.Spider):
name = "example"
start_urls = ["https://example.com"]
def start_requests(self):
yield scrapy.Request(
"https://example.com",
headers={
"Accept-Language": "fr",
"X-Client": "my-spider",
},
)If your spider already yields requests elsewhere, the essential form is:
yield scrapy.Request(
url,
headers={"X-Client": "my-spider"},
)Header values can be strings for single-valued headers or lists for multi-valued headers. The request exposes them through a dictionary-like scrapy.http.headers.Headers object. Passing None as a value means that header is not sent. These details are defined in the Request API reference.
Use a list for a multi-valued header
yield scrapy.Request(
"https://example.com/feed",
headers={
"Accept": ["application/xml", "text/xml"],
"X-Trace": "crawl-2026-09-29",
},
)Only use a list when the target service and the HTTP header semantics allow multiple values. For ordinary headers such as Accept-Language or an authorization token, a string is usually clearer.
Omit a value explicitly
yield scrapy.Request(
"https://example.com",
headers={"X-Optional": None},
)A None value tells Scrapy not to send that header. If a project default exists for the same name, verify the resulting request in your own downloader logs or instrumentation rather than assuming a default has been removed at every middleware stage.
Free tools Windows power users keep installed
One-click scans. No signup required.
Set project-wide default headers
Put common fallback values in the project’s settings.py:
DEFAULT_REQUEST_HEADERS = {
"Accept": "application/json",
"Accept-Language": "en",
"X-Client": "my-spider",
}Scrapy documents built-in defaults of Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 and Accept-Language: en. The DefaultHeadersMiddleware component applies values from DEFAULT_REQUEST_HEADERS. See the Downloader Middleware documentation and the Settings reference.
Per-request values override defaults
The middleware uses request.headers.setdefault(k, v). In practical terms, a configured default fills an absent key; it does not overwrite a value already placed on the request.
# settings.py
DEFAULT_REQUEST_HEADERS = {
"Accept": "application/json",
"X-Client": "default-client",
}
# spider
yield scrapy.Request(
"https://example.com/special",
headers={
"Accept": "text/csv",
"X-Client": "csv-exporter",
},
)The special request receives text/csv and csv-exporter; other requests without those keys receive the defaults. This precedence lets you keep authentication or content-negotiation policy centralized while making deliberate exceptions locally.
Recommended Free Tools
Cookies are not ordinary custom headers
When Scrapy’s cookie middleware should maintain session state, pass cookies through the request’s cookies argument:
yield scrapy.Request(
"https://example.com/account",
cookies={
"session_id": "abc123",
"language": "en",
},
)The settings documentation cautions that cookies supplied as a raw Cookie header are not considered by that middleware. A manually composed header may reach the server, but it will not provide the same cookie-jar behavior for subsequent requests. Use the Cookie header directly only when you intentionally need to control the wire value and do not expect cookie middleware to manage it.
Rank #3
Understand Referer middleware
RefererMiddleware derives a Referer header from the response that generated a new request. Consequently, a Referer placed in DEFAULT_REQUEST_HEADERS generally reaches requests for which the middleware does not set one, such as start requests; it is not a guaranteed override for follow-up requests.
Set the policy globally
Use the REFERER_POLICY setting to select the policy documented by Scrapy. The available policy controls how much of the originating URL may be sent and whether a referrer is sent at all. The behavior and setting names are described in the Spider Middleware documentation.
Set a policy for one request
Scrapy also supports a per-request referrer_policy metadata key:
yield scrapy.Request(
"https://example.com/next",
meta={"referrer_policy": "no-referrer"},
)Choose a policy that matches the target site’s documented requirements and your privacy obligations. Do not assume that adding a hand-written Referer header defeats middleware policy; inspect the final request behavior when the distinction matters.
Headers and request fingerprints
Adding a header does not automatically make two otherwise identical requests distinct to Scrapy’s default request fingerprinter. The request-fingerprinting reference says headers are ignored by default. If a selected header must participate in deduplication or HTTP cache identity, include it through the fingerprinter’s include_headers argument as described in Scrapy’s request utility reference.
This matters for language, authorization, tenant, or API-version headers: two requests with different logical identities can otherwise share a fingerprint. Decide deliberately whether that header should change duplicate filtering and cache keys; including volatile headers can reduce cache reuse.
A complete spider pattern
The following example combines project defaults, a per-request override, managed cookies, and a request-specific referrer policy:
# settings.py
DEFAULT_REQUEST_HEADERS = {
"Accept": "application/json",
"Accept-Language": "en",
"X-Client": "catalog-spider",
}
# spiders/catalog.py
import scrapy
class CatalogSpider(scrapy.Spider):
name = "catalog"
start_urls = ["https://example.com/catalog"]
def parse(self, response):
yield scrapy.Request(
"https://example.com/catalog/export",
headers={
"Accept": "text/csv",
"X-Export-Format": "csv",
},
cookies={"region": "us"},
meta={"referrer_policy": "same-origin"},
callback=self.parse_export,
)
def parse_export(self, response):
yield {"status": response.status, "length": len(response.body)}Here, Accept-Language and X-Client come from settings, while the export request replaces only Accept and adds X-Export-Format. The region cookie is handed to cookie middleware, and the referrer behavior is selected through metadata rather than by assuming a static default header.
Common failures and fixes
The custom value never appears
- Check that the header mapping is passed to the
Requestyou actually yield. Setting a variable on the spider does nothing unless it is attached to the request. - Confirm the spelling and capitalization of the key. HTTP field names are treated case-insensitively, but a typo creates a different key.
- Look for another downloader middleware or request callback that creates a new request without copying the headers.
The default overwrites my request value
Scrapy’s default-header middleware uses setdefault, so the documented behavior is the opposite: an existing per-request key is preserved. If you see a different result, another middleware, a redirect, or a newly generated request is likely involved. Trace the request object at the point where it is yielded and compare it with the final downloader instrumentation.
My manually set Referer changes on follow-up requests
That is expected when Referer middleware derives a value from the parent response. Configure REFERER_POLICY or the request’s referrer_policy metadata instead of relying on a project-wide static header. Refer to the official middleware rules.
Cookies do not persist
Move cookie data from a raw Cookie header to the request’s cookies argument when cookie middleware should maintain the session. Ensure you are not disabling or bypassing the cookie middleware for that request.
Requests with different headers are deduplicated
Default fingerprints ignore headers. Configure fingerprinting to include only the headers that define request identity; including every dynamic header can create unnecessary duplicates and reduce cache effectiveness. The implementation details are in the request utility source documentation.
The server rejects my User-Agent or Accept
There is no universal browser-style header set. Values depend on the target service, API contract, authentication method, and access rules. Read that service’s published guidance, send only values you need, and do not treat custom headers as a way to bypass robots rules, bot checks, authentication, or other access controls.
Reliability, performance, and maintenance
- Keep stable defaults small. A short set of project defaults is easier to audit than copying a browser dump into every request.
- Make exceptions local. Put endpoint-specific content negotiation, tenant IDs, or API versions beside the request that needs them.
- Protect secrets. Do not hard-code tokens in a spider committed to source control; load them from Scrapy settings populated by your deployment environment.
- Consider cache identity. If a header changes the representation or authorization context, decide whether it belongs in the request fingerprint and HTTP cache key.
- Expect middleware to transform requests. Cookies, redirects, retries, authentication middleware, and Referer handling can all make the final wire request differ from the object created in a callback.
- Use the target’s contract. Header names and values are not interchangeable across services. Follow the API documentation for required media types, authorization syntax, language tags, and rate-limit expectations.
Or skip the browser setup
If your goal is a rendered page image rather than extracting responses with Scrapy, ScreenshotNeo provides a single HTTP call. Before capture it accepts cookie or consent banners and removes more than 60 known consent platforms, newsletter popups, and chat widgets; each cleanup step can be disabled. Only clean shots are billed: bot checks or CAPTCHAs, blank pages, timeouts, failed loads, and cache hits cost nothing, and the response identifies the result with X-Page-Verdict and X-Billed headers. It also offers an MCP server with take_screenshot, get_page_info, and capture_pdf tools for Claude, Cursor, and other MCP clients.
For API details, see the ScreenshotNeo documentation. The following calls use the supplied API endpoint and save the returned WebP image:
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpimport requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
open("shot.webp", "wb").write(r.content)const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);The Free plan includes 1,000 shots per month with no card. Paid plans start at $5 for 3,000 shots; every feature is available on every plan, and yearly billing gives two months free. Create a free ScreenshotNeo account to start without a card.
Frequently Asked Questions
Should I copy a browser’s entire header set into Scrapy?
No. Send the headers required by the target service and your application. A universal browser header set is neither necessary nor guaranteed to work.
Does a header value affect Scrapy’s duplicate filter automatically?
No. Scrapy’s default request fingerprinter ignores headers unless you configure selected headers through its include_headers option.
PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchWhat is the correct spelling of the HTTP referrer field?
The HTTP header is spelled Referer. Scrapy’s RefererMiddleware and its referrer_policy metadata use that spelling.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
