Build the pipeline in stages: define a bounded collection target and schema, retrieve data through Bright Data’s JavaScript SDK or REST API, monitor longer jobs, validate and normalize the output, then store it with provenance before using it in an AI workflow. Bright Data handles collection mechanics; your application still needs to check data quality, permissions, and retention.
Choose how you will collect data
Bright Data documents two practical ways to connect a Node.js application to collection: use its JavaScript SDK for a client-oriented workflow, or call the REST API directly when you want explicit control over requests and job orchestration.
As an Amazon Associate I earn from qualifying purchases.
| Choice | Use it when | What to plan for |
|---|---|---|
| JavaScript SDK | You want a documented Node.js client for URL scraping, search, platform scrapers, Scraper Studio, datasets, or Browser API access. | Install @brightdata/sdk, configure a secret API key, and use the relevant client method. Bright Data JavaScript SDK documentation. |
| Direct REST API | You want to trigger dataset collection and implement job tracking explicitly in your own service. | Send an authenticated request, persist the returned snapshot ID, poll its documented progress endpoint, and retrieve results using the current snapshot API documentation. Trigger a collection and Monitor progress. |
Next, choose the scraper source. Bright Data’s Scrapers Library provides maintained scrapers for popular sites; Scraper Studio is the custom route when the library does not cover the needed target or data shape. Studio offers an AI Agent that can generate a scraper from a natural-language description and target URL, an IDE for JavaScript editing, and a managed-scraper option. These are alternatives, not guarantees of a particular data quality or legal permission.
Free tools Windows power users keep installed
One-click scans. No signup required.
Scraper Studio supports product-page, discovery, discovery-plus-detail, search, and sitemap patterns. Treat each scraper as a defined collection job rather than a universal crawler: the FAQ cautions that an AI Agent scraper is scoped to a data shape, while deeper discovery can be implemented as a multi-stage IDE scraper. Scraper Studio FAQs.
#1 Best Overall
Set up the Node.js connection
Install the official SDK with npm and initialize its client with an API key. The SDK documentation identifies BRIGHTDATA_API_KEY as an accepted environment variable. Do not paste a real key into source code or commit it to a repository; use your deployment environment’s secret-management facility.
npm install @brightdata/sdkA minimal SDK setup follows the documented import and client pattern:
import { bdclient } from "@brightdata/sdk";
const client = bdclient({
apiKey: process.env.BRIGHTDATA_API_KEY,
});
try {
const result = await client.scrapeUrl("https://example.com", {
// Add supported country or data-format options when needed.
});
console.log(result);
} finally {
await client.close();
}Use a real target only after defining what fields you need and confirming that your intended collection and use are permitted. The SDK guide also documents platform-specific methods and custom Studio execution through client.scraperStudio.run(...) or client.scraperStudio.trigger(...); consult the current reference for required arguments and return types rather than assuming every scraper accepts the same input. SDK methods and examples.
PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteSelect a worker based on how the page works
Bright Data’s guidance distinguishes workers by rendering needs: Browser is intended for JavaScript-rendered pages and interactions such as waiting, clicking, scrolling, or capturing background network calls; Code is intended for static HTML and HTTP responses, which Bright Data positions as faster and cheaper. This is vendor guidance, not an independent performance comparison. Worker documentation.
- Choose Browser when the content or state you need appears only after client-side rendering or interaction.
- Choose Code when the required information is present in a static response and does not require browser behavior.
Run short collections synchronously and long jobs asynchronously
A synchronous request is convenient when a collection is predictably small and its result can be returned in the request. For a dataset, Bright Data documents the synchronous /datasets/v3/scrape route. Its progress documentation says that if a synchronous request exceeds the documented one-minute timeout, it receives a snapshot ID; at that point, monitor progress and retrieve the result through the snapshot workflow. The timeout is a vendor-documented operational behavior and may change. Monitor progress.
For larger or unpredictable jobs, use asynchronous orchestration from the start. The documented dataset trigger endpoint is POST https://api.brightdata.com/datasets/v3/trigger, authenticated with a bearer token and a JSON input array. The response includes a snapshot ID. Save that ID in your job record so your process can resume monitoring after a restart. Bright Data’s documentation includes both Axios and built-in fetch examples. Dataset collection trigger.
- Submit the job. Send the target inputs and required configuration to the trigger endpoint with your API key in the bearer-token authorization header.
- Persist the snapshot ID. Associate it with your own request ID, input set, creation time, and collection configuration.
- Check progress. Call
GET https://api.brightdata.com/datasets/v3/progress/{snapshot_id}and handle the documented states:starting,running,ready,failed, andcanceled. Use a bounded polling interval and stop polling at a terminal state. - Retrieve results when ready. Use the current documented snapshot result or download API for the returned snapshot. Do not construct an endpoint path from memory; follow the live API reference.
- Handle unsuccessful outcomes. Record error details, failed inputs, and delivery issues. Retry only when the operation is safe to repeat, and make your own processing idempotent.
The progress reference documents errors including input validation failures, empty snapshots, delivery failures, and collector-trigger failures. A job reaching a terminal state is not the same as your application obtaining a useful, valid dataset; test both conditions. Progress states and error details.
Define a schema and parse the output defensively
Before collecting, decide which fields the downstream task actually needs. For example, a product research record might contain a source URL, retrieval timestamp, product name, price text, currency, and collection job ID. Treat this as your own schema, not a promise that a given scraper returns those exact fields.
Scraper Studio can deliver JSON, NDJSON, CSV, XLSX, and selected Parquet options; Parquet is not supported for every destination. Choose a format your consumer can parse reliably and verify the destination’s current format support. One input can produce multiple records, so never assume input count equals output-row count. Bright Data’s dashboard statistics count records rather than inputs. Output formats and record counts.
- Check required fields, types, nullability, and encoding before accepting a record.
- Detect duplicates and malformed or implausible values instead of passing them silently downstream.
- Track schema versions and flag unexpected field additions, removals, or type changes.
- Keep raw responses in a separate immutable layer where permitted; write normalized records to a distinct layer.
- Preserve provenance such as source URL, retrieval time, job or snapshot ID, locale, and query context where applicable.
These checks belong in your pipeline. The SDK and scraper output do not by themselves establish that results are complete, current, accurate, or suitable for a particular model task.
Persist results before the vendor snapshot expires
Bright Data’s current Scraper Studio FAQ states that batch snapshots are retained for 16 days and real-time snapshots for 7 days before permanent deletion; the documentation does not state a publication date for these values. Treat them as operational limits to verify against the live FAQ, not as an archive strategy. Arrange prompt download or automatic delivery into storage you control, and define your own retention and deletion policy. Snapshot retention.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Fix the driver behind crashes, sound loss and screen glitches3Clear out junk files and repair common Windows errorsFor AI use, keep collected source material separate from derived data. Normalize and validate before chunking and indexing content for retrieval-augmented generation. Label generated summaries, classifications, or other annotations as derived outputs so a later consumer can distinguish them from source facts.
Check permissions as well as technical access
Successful retrieval does not establish permission to collect or use a particular site’s data. Review the target site’s terms, applicable laws, privacy obligations, and the permissions relevant to your intended purpose. Public accessibility, robots directives, or a provider’s ability to retrieve a page do not, by themselves, resolve those questions. Bright Data’s technical documentation does not determine the requirements for your specific target or use case.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
