To convert a cURL command to Python Requests, map each cURL option to the corresponding request argument: query values to params, headers to headers, JSON to json, form or raw bodies to data, credentials to auth, cookies to cookies, uploads to files, and time limits to timeout. Then check the response status and handle errors explicitly. This guide walks through that translation and the common reasons a command that works in cURL may fail in Python.
Install Requests and make a first request
Install Requests in the Python environment where your script will run:
python -m pip install requestsThe Requests overview lists Python 3.10+ support and identifies Requests 2.34.2 as its current release; these are version-sensitive details, so check the official overview for the current compatibility information.
A useful first request includes a timeout, checks for an unsuccessful HTTP status, and only parses JSON when the server says it returned JSON:
Free tools Windows power users keep installed
One-click scans. No signup required.
#1 Best Overall
import requests
url = "https://httpbin.org/get"
try:
response = requests.get(url, timeout=(3, 20))
response.raise_for_status()
except requests.exceptions.Timeout:
raise SystemExit("The request timed out")
except requests.exceptions.HTTPError as exc:
raise SystemExit(f"The server returned an unsuccessful status: {exc}")
print("Status:", response.status_code)
print("Content type:", response.headers.get("Content-Type"))
if "application/json" in response.headers.get("Content-Type", "").lower():
print(response.json())
else:
print(response.text[:500])Requests describes itself as “an elegant and simple HTTP library for Python, built for human beings.” Its quickstart demonstrates the basic request and response workflow.
Translate cURL options into Requests arguments
Start with the server’s contract: it determines the accepted URL, HTTP method, body format, authentication, redirect behavior, and response codes. A cURL command is a convenient way to express that contract, but not every cURL flag has a one-to-one Requests argument.
| cURL pattern | Requests equivalent | What to know |
|---|---|---|
-G with -d key=value |
params={...} |
Query parameters belong in the URL query string; Requests encodes them. |
-H 'Name: value' |
headers={...} |
Header names are strings; values should match the API’s requirements. |
-d '{...}' as JSON |
json={...} |
Requests serializes the object as JSON and sets a JSON content type. |
-d 'a=b' as form data, or a raw body |
data=... |
Use a dictionary for form fields or a string/bytes body when the API expects raw content. |
-u user:password |
auth=(user, password) |
This represents HTTP Basic authentication. Other schemes need their own handling. |
-F 'file=@path' |
files={...} |
Use multipart uploads when the endpoint expects them. |
-b or -c |
cookies=... or a Session |
Use a session when cookies must persist across calls. |
--max-time |
timeout=... |
Requests accepts a scalar or a (connect, read) pair; behavior is not a total wall-clock deadline. |
The corresponding arguments are documented in the Requests API reference. Be careful with cURL’s -d: paired with -G, it contributes query parameters; without -G, it normally sends a request body. In Python, express those two intentions separately with params and data or json.
Build requests with query strings, headers, and bodies
Query parameters
Pass query values as a mapping instead of manually concatenating and escaping them:
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
import requests
response = requests.get(
"https://api.example.com/search",
params={"q": "red shoes", "page": 2},
timeout=(3, 20),
)
response.raise_for_status()Requests handles URL encoding. This avoids common errors with spaces, ampersands, non-ASCII characters, and values that need escaping. If the API requires repeated keys, use a sequence of pairs, such as params=[("tag", "one"), ("tag", "two")].
Headers and JSON
Use headers for values such as an API token or a version header. Send JSON with json:
import os
import requests
response = requests.post(
"https://api.example.com/items",
headers={"Authorization": f"Bearer {os.environ['API_TOKEN']}"},
json={"name": "sample", "enabled": True},
timeout=(3, 20),
)
response.raise_for_status()Keep credentials in environment variables or a secret manager, not source code or logs. Do not assume every API uses Bearer tokens or accepts JSON; follow the endpoint’s authentication and content-type requirements.
Form data and raw request bodies
Use data with a dictionary for form-encoded fields:
The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →response = requests.post(
"https://api.example.com/login",
data={"username": "example", "remember": "yes"},
timeout=(3, 20),
)
response.raise_for_status()For an API that expects a specific raw body, pass a string or bytes to data and set the required content type yourself:
response = requests.post(
"https://api.example.com/ingest",
data=b"raw payload",
headers={"Content-Type": "application/octet-stream"},
timeout=(3, 20),
)
response.raise_for_status()Do not use data and json together to describe one body. Choose the representation the server expects.
Send authentication, cookies, and files
Authentication
For HTTP Basic authentication, pass a username and password with auth:
response = requests.get(
"https://api.example.com/private",
auth=("user", "password-from-secret-store"),
timeout=(3, 20),
)
response.raise_for_status()Requests also documents Digest authentication and use of .netrc. OAuth and OAuth 2/OpenID Connect typically require obtaining and refreshing tokens with an appropriate integration; they are not interchangeable with Basic authentication. Consult the authentication documentation and the API’s own token, scope, and refresh rules. Redact authorization values when logging request details.
Cookies
For a single request, pass cookies as a mapping:
response = requests.get(
"https://example.com/account",
cookies={"session_id": "value-from-a-secure-source"},
timeout=(3, 20),
)
response.raise_for_status()When a server sets cookies that later requests need, use a session rather than manually copying cookie headers. Cookie scope and behavior are controlled by the server and the cookie rules applied by the client.
File uploads
Use files for a multipart form upload. Open the file in binary mode and close it when the request finishes:
with open("report.pdf", "rb") as file_handle:
response = requests.post(
"https://api.example.com/upload",
files={"file": file_handle},
data={"category": "reports"},
timeout=(5, 60),
)
response.raise_for_status()The field name, additional form fields, and accepted file types are endpoint-specific. For very large payloads, check the service’s size limits and streaming guidance rather than assuming a small example is suitable.
Read responses and handle errors explicitly
A response exposes several useful views:
response.status_codeis the HTTP status code.response.headerscontains response headers, such as content type.response.textdecodes the body as text.response.contentgives the body as bytes, useful for binary data.response.json()parses JSON, but raises an error if the body is not valid JSON.
Call raise_for_status() when a non-success HTTP status should stop ordinary processing. It raises an HTTPError for unsuccessful responses; it does not mean every failure is an HTTP status. Connection errors, timeouts, invalid JSON, and application-level error objects need their own handling. The quickstart explains response handling and raised exceptions.
PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteimport requests
try:
response = requests.get("https://api.example.com/status", timeout=(3, 20))
response.raise_for_status()
content_type = response.headers.get("Content-Type", "").lower()
if "application/json" not in content_type:
raise ValueError(f"Expected JSON, got {content_type or 'unknown content type'}")
payload = response.json()
except requests.exceptions.Timeout:
print("The server did not respond within the configured timeout")
except requests.exceptions.ConnectionError:
print("Could not establish or maintain a connection")
except requests.exceptions.HTTPError as exc:
print(f"HTTP failure: {exc}")
except ValueError as exc:
print(f"Response was not usable JSON: {exc}")Set timeouts and make retries safe
Never rely on an implicit unlimited wait. A scalar timeout applies to the connection and response-wait behavior; a pair such as (3, 20) sets a connect timeout and a read timeout separately. The connect timeout limits the time spent establishing a connection. The read timeout limits how long the client waits for response bytes after connection; it is not necessarily a maximum for the entire operation. See the timeout parameter reference.
Catch requests.exceptions.Timeout or a narrower timeout subclass, then decide whether retrying is safe. Repeating a read-only request is often different from repeating a request that charges a card, creates an order, or otherwise changes server state. A timeout does not prove the server failed to perform the operation; it may have completed it while the response was lost. Use an API-provided idempotency mechanism where available, and follow the service’s retry guidance.
Reuse connections and cookies with a Session
For repeated calls to the same service, a Session persists cookies and reuses pooled connections. It can also carry shared headers. This reduces repeated setup and is useful for multi-request workflows such as login followed by authenticated requests. Requests describes these behaviors in its advanced usage guide.
import requests
with requests.Session() as session:
session.headers.update({"Accept": "application/json"})
first = session.get("https://api.example.com/start", timeout=(3, 20))
first.raise_for_status()
second = session.get("https://api.example.com/next", timeout=(3, 20))
second.raise_for_status()
print(second.json())The context manager closes the session and its connections when the block ends. Keep a session scoped to the work that needs its shared state; do not accidentally share user-specific cookies or authorization headers across unrelated users in a long-running service.
Why cURL may work while Python Requests fails
The difference is often not “cURL versus Python” but a detail hidden in the original command or environment. Compare the actual request components before changing libraries.
- Different URL encoding: Move query values into
paramsrather than hand-building the query string. Check whether cURL was using-G. - Wrong body format: cURL
-ddoes not by itself tell you whether the endpoint expects JSON, form encoding, or raw bytes. Usejsononly for JSON, anddatafor the other documented formats. - Missing headers: Reproduce required headers from
-H, especially content type, accept, and API version headers. Do not copy browser headers blindly. - Authentication mismatch: cURL
-umaps to Basic authentication, not automatically to an API key or OAuth flow. Verify the scheme and credential placement expected by the service. - Cookie state differs: cURL may be loading or saving cookies with
-band-c. Use a Requests session when cookies must carry across calls. - Redirect or status differences: Inspect
status_code, response headers, and the final URL. Confirm the endpoint’s redirect policy and do not treat a returned error page as a successful API response. - Local network or certificate setup: The two clients may run with different proxy, CA certificate, environment, or runtime configuration. Diagnose the actual environment instead of disabling TLS verification.
Requests verifies TLS certificates by default. Avoid verify=False as a routine workaround: it removes certificate verification rather than fixing an incorrect trust configuration. If an organization uses a private CA, configure the appropriate CA bundle deliberately.
When to choose Requests or curl_cffi
Requests is the straightforward default for ordinary Python HTTP clients. Choose an alternative when a concrete compatibility requirement calls for it, not simply because one request failed.
| Need | Requests | curl_cffi |
|---|---|---|
| Familiar Python request interface | Common methods and arguments such as get, post, params, json, and timeout. |
Requests-like interface, described in its quickstart. |
| Sessions and cookie persistence | Sessions persist cookies and reuse pooled connections. | Provides sessions; its maintainers say, “You should always use a session whenever possible.” |
| Basic request controls and errors | Documented request parameters, timeouts, TLS verification, and response handling. | Offers curl-oriented options alongside its Requests-like API; check the API reference for exact options. |
| Browser impersonation controls | Not the reason to choose Requests. | Provides an impersonate parameter for browser-oriented behavior. |
| Command-line use | Typically used as a Python library. | Its documentation gives CLI invocations including uv run curl-cffi and python -m curl_cffi; see the documentation PDF. |
Browser impersonation is not permission to bypass a site’s terms, access controls, or authorization. For either library, review its installation requirements, release policy, and deployment constraints before making it a production dependency. A similar API can ease migration, but TLS and HTTP behavior can still differ; verify the behavior your integration actually requires.
Or skip the browser setup
If your task is to capture a website as an image or PDF rather than build a general-purpose HTTP client, ScreenshotNeo provides a screenshot API and MCP server. One GET request can return a PNG, JPEG, WebP, or PDF. For example, from Python:
import requests
r = requests.get(
"https://api.screenshotneo.com/v1/shot",
params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"},
timeout=90,
)
open("shot.webp", "wb").write(r.content)See the ScreenshotNeo documentation for request options. It accepts cookie or consent banners and removes more than 60 known consent platforms, newsletter popups, and chat widgets before capture; each of those steps can be turned off. Bot checks and CAPTCHAs, blank pages, timeouts, failed loads, and cache hits are not billed, and response headers report the page verdict and billing status. Its MCP server offers take_screenshot, get_page_info, and capture_pdf for AI agents and MCP clients. The free plan includes 1,000 screenshots per month with no card; paid plans start at $5 for 3,000 shots.
Sign up free for 1,000 screenshots a month, with no card required.
Troubleshoot common failures
| Symptom | Likely cause | What to check or change |
|---|---|---|
| Connection hangs | No timeout was set, or the configured timeout is too long for the task. | Set a scalar timeout or a connect/read pair. Handle timeout exceptions and choose limits appropriate to the endpoint. |
| HTTP 400 or 415 | Malformed input or an unexpected body/content type. | Confirm whether the API expects JSON, form data, or raw bytes. Use json for JSON and inspect the server’s error response. |
| HTTP 401 or 403 | Missing, invalid, expired, or insufficient credentials; possibly the wrong authentication scheme. | Check the API’s credential format, token scopes, refresh rules, and required headers. Do not log secrets. |
| HTTP 404 | Wrong path, API version, or resource identifier. | Compare the full URL and path against the endpoint documentation and inspect redirects or the final URL. |
| JSON parsing error | The response is not JSON or contains invalid JSON, often because an error page was returned. | Check the status and content type before calling .json(); inspect a safe excerpt of .text. |
| Certificate verification failure | The trust store does not recognize the certificate chain, possibly due to a private CA or environment configuration. | Install or configure the appropriate CA bundle. Avoid turning off verification as a general fix. |
| Repeated request creates duplicates | A retry occurred after a timeout even though the server completed the first operation. | Use an idempotency key if the API supports one and follow its retry rules before repeating state-changing calls. |
FAQ
Does Python Requests run the cURL executable?
No. Requests is a Python HTTP library with its own request API; translating a command means reproducing its HTTP method, URL, headers, body, authentication, cookies, and relevant network behavior.
Recommended Free Tools
Can I use Requests for a screenshot API?
Yes. A screenshot service that exposes an HTTP endpoint can be called with Requests; use the service’s documented parameters and save the response bytes in the appropriate file format.
Should I switch to curl_cffi whenever a site rejects Requests?
No. First verify that your request is authorized and correctly formed. Consider curl_cffi only when its documented curl-oriented or browser-impersonation capabilities are a genuine requirement, and respect the site’s policies.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
