Free tools Windows power users keep installed
One-click scans. No signup required.
Postman can send, inspect, test and repeat requests to a web-scraping API, but it does not make unauthorized scraping permissible. Start with the target provider’s documentation: it defines the endpoint, HTTP method, query or path parameters, authentication, response format and limits. In Postman, configure those pieces, send the request, inspect the response, then save the request in a collection with variables and tests so the workflow is repeatable.
What Postman does—and what it does not do
Postman is an HTTP/API client. It builds a request and displays the response; it does not itself crawl websites, bypass access controls or grant permission to copy a site’s content. A scraping API provider normally performs the fetching and extraction, while Postman is the interface you use to call that provider.
- Postman does: send HTTP requests, show status codes, headers and bodies, run scripts, and organize requests into collections.
- The target API does: define valid endpoints, methods, parameters, authentication and returned data.
- You must do: confirm automated access is authorized, follow the target site’s terms and applicable law, and respect provider and website rate limits.
Postman’s own terms restrict unauthorized scraping, data mining, extraction, duplication or copying of other customers’ content. Those restrictions are separate from the target website’s rules and do not replace them.
Build your first scraping API request
1. Create the request
- Open Postman and select New → HTTP Request.
- Choose the method documented by the scraping provider, usually GET for retrieving data or POST when the provider expects a JSON job definition.
- Enter the complete endpoint URL. Do not substitute a website URL for the provider’s API endpoint unless the provider explicitly documents that behavior.
An HTTP request requires a URL and method. Parameters, authorization, body data, headers and cookies are optional but commonly required by scraping APIs.
#1 Best Overall
2. Add query and path parameters
Use the Params tab for query parameters. Add one key per row and enter the value in the adjacent column; Postman URL-encodes values when it constructs the request. Typical keys include a target URL, API key, output format, country, device or pagination cursor—but use the exact names in the provider’s documentation.
Path parameters belong in the URL itself, such as /jobs/:job_id. Replace the variable with a real ID or define it as a Postman variable. Check the generated URL above the request before sending; a misspelled key or unencoded ampersand can change the request the server receives.
3. Configure authentication and headers
Use the Authorization tab when the provider supports a standard scheme such as Bearer token, Basic Auth or an API-key helper. Postman then generates the corresponding header. If the provider requires a custom header, use Headers, for example:
| Purpose | Typical location | What to verify |
|---|---|---|
| Bearer token | Authorization: Bearer … |
Token is current and has the required scope |
| API key | Provider-specified header or query key | Name and case match the documentation |
| Content type | Content-Type: application/json |
Matches the body format |
| Accept format | Accept: application/json |
Provider supports the requested representation |
Do not add arbitrary browser headers to imitate a visitor unless the provider documents that requirement. Remove duplicate authorization headers: conflicting credentials can produce a 401 or 403 response.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
4. Add a body when the API expects one
For POST, PUT or PATCH requests, choose Body → raw → JSON, then enter the schema from the API documentation. A common job request might contain a target URL, output fields and callback settings, but field names differ between providers. Postman adds the JSON content-type header when you select the JSON editor; still confirm it in Headers.
5. Send and inspect the response
Select Send. Read the status code before parsing the body:
- 2xx: the request was accepted or completed. A 202 often means an asynchronous job was queued rather than finished.
- 400: malformed URL, missing parameter or invalid JSON.
- 401: missing, expired or incorrectly formatted credentials.
- 403: credentials lack permission, the provider rejected the request, or access policy blocks it.
- 404: wrong endpoint or resource ID.
- 429: rate limit exceeded; follow the provider’s retry guidance.
- 5xx: provider-side failure; capture the response ID and retry cautiously.
Use the response tabs to inspect the body, headers, cookies and timing. Save the response example when you need a stable fixture for later tests.
Make requests reusable with collections and variables
Collections and folders
Save related requests in a collection, such as Scraper API, with folders for authentication, single-page extraction, pagination and job status. Collections support collection-level authorization, pre-request scripts, post-response scripts and reusable variables. A collection runner can execute the same workflow repeatedly.
Environment and collection variables
Define variables for the API base URL, token, target URL, job ID and page cursor. Use double braces in requests:
{{base_url}}/extract?url={{target_url}}Keep development, staging and production values in separate environments. Collection variables are useful defaults; environment values let you switch targets without editing every request. Put secrets in Postman Vault or secure variables rather than hard-coding them in a shared request or collection.
Pre-request scripts
A pre-request script can generate a timestamp, calculate a signature required by the provider, or copy a value into a variable. Keep signing logic identical to the provider’s specification, including canonical ordering and encoding. Never log the secret itself to the Postman Console.
Post-response scripts and assertions
Post-response scripts run after the response arrives. They can assert properties, pass values to later requests and show outcomes in Test Results. A basic JSON test is:
pm.test("HTTP request succeeded", function () {
pm.expect(pm.response.code).to.be.within(200, 299);
});
pm.test("Response is JSON", function () {
pm.response.to.have.jsonBody();
});
const data = pm.response.json();
pm.test("Scraper returned a result", function () {
pm.expect(data).to.be.an("object");
});Adapt the final assertion to the documented schema—for example, require a non-empty result array or a job identifier. Do not assume a field exists merely because another provider uses that name.
Test asynchronous scraping jobs
Some APIs return a job ID immediately. Model that as a collection workflow:
- Create job: send the POST request and save the returned ID in a collection variable.
- Poll status: call the documented status endpoint with that ID.
- Stop on a terminal state: continue only while the status is pending or running; stop on success or failure.
- Fetch results: request the output endpoint after completion and assert the expected fields.
Use the provider’s recommended polling interval and maximum attempts. A 429 response is a signal to slow down, not to launch parallel retries. If the provider offers webhooks, prefer the callback flow for long jobs and verify the webhook signature before accepting data.
Pagination, rate limits and reliable runs
Pagination
Inspect the response for the provider’s documented next-page token, cursor or link. Store it in a variable and send it unchanged on the next request. Stop when the API omits the cursor or reports no additional records. Do not increment an assumed page number if the API uses opaque cursors.
Recommended Free Tools
Rate-limit behavior
Read rate-limit headers and documentation. Space requests, cap concurrency and use exponential backoff for transient 429 or 5xx responses. Preserve idempotency: retrying a GET is generally safer than retrying a POST that creates a job unless the provider documents idempotency keys.
Response size and sensitive data
Large HTML or extraction results can slow the Postman client and collection runner. Request only needed fields where the API supports field selection, and save representative examples rather than sensitive production responses. Treat API keys, cookies, authorization headers and scraped personal data as confidential.
Common Postman errors and fixes
| Symptom | Likely cause | Fix |
|---|---|---|
| 401 Unauthorized | Missing, expired or mislocated credential | Compare the Authorization tab and raw headers with the provider’s example; regenerate the key if necessary. |
| 403 Forbidden | Insufficient scope, blocked account or disallowed target | Check account permissions and target authorization; do not attempt to bypass the restriction. |
| 400 with “invalid URL” | Target URL was not encoded or required scheme is absent | Use the Params table, include https://, and inspect the generated URL. |
| 415 Unsupported Media Type | Body format and Content-Type disagree | Select raw JSON (or the documented format) and set the matching content type. |
| 429 Too Many Requests | Provider limit exceeded | Wait according to retry headers, reduce runner concurrency and cache results where permitted. |
| Successful status but empty data | Extraction selector, field name or target access issue | Validate the API’s selector syntax, inspect the raw response and test a page you are authorized to access. |
| Tests fail although the request is 2xx | Assertion assumes the wrong schema or an asynchronous response | Print a sanitized response, check whether the result is a job object, and assert documented fields only. |
Postman’s own API
If your workflow calls Postman’s API rather than a scraping provider, you need a valid Postman API key. Postman warns that rate and usage limits apply, and endpoint availability can vary by region and plan. Keep that key in Vault or a secure variable and apply the same collection, assertion and backoff practices.
Or skip the browser setup
If your actual goal is a clean screenshot rather than structured page extraction, ScreenshotNeo provides a website screenshot API and MCP server. One request returns a PNG, JPEG, WebP or PDF. It accepts cookie and consent banners like a visitor, then removes more than 60 known consent platforms, newsletter popups and chat widgets before capture; each cleanup step can be disabled. Bot checks or CAPTCHAs, blank pages, timeouts, failed loads and cache hits are not billed, and the response identifies the page verdict and billing result in X-Page-Verdict and X-Billed headers.
Using the documented endpoint, replace the target URL and key:
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpSee the ScreenshotNeo API documentation for options such as full-page capture, CSS-selector element capture, device and retina settings, custom JavaScript or CSS, waits, blocked resources, cookies, headers, geolocation, PDF output, resizing, caching, signed links, asynchronous jobs, webhooks and bulk capture.
Python
import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
open("shot.webp", "wb").write(r.content)Node.js
const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);An MCP server exposes take_screenshot, get_page_info and capture_pdf to Claude, Cursor and other MCP clients. The Free plan includes 1,000 shots per month with no card; paid plans start at $5 for 3,000 shots, and every feature is on every plan. Create a free ScreenshotNeo account.
FAQ
Can Postman scrape any website?
No. Postman sends the request; permission comes from the API provider, target site and applicable law.
Should an API key go in the URL?
Only when the provider documents a query-key scheme. Prefer the Authorization tab or required header, and store the secret securely.
Why does a 202 response contain no scraped records?
It commonly indicates an asynchronous job. Save the job ID, poll the documented status endpoint, then fetch results after completion.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
