PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchInstall js-crawler from npm, create a crawler, and call crawl() with a starting URL. Set its depth and URL filters to control how far it follows links, then use callbacks to handle fetched pages, failures, and completion. The package documentation describes HTTP and HTTPS crawling; it does not establish that it runs page JavaScript like a browser.
Install js-crawler
The project README documents installation with npm:
As an Amazon Associate I earn from qualifying purchases.
npm install js-crawlerThe examples below use CommonJS, matching the README’s documented import pattern. Save the script in a Node.js project where the package is installed, then run it with Node.
Recommended Free Tools
Start a basic crawl
This example starts at one URL, follows links up to depth three, and prints each successfully fetched page’s URL:
#1 Best Overall
const Crawler = require("js-crawler").default;
new Crawler()
.configure({ depth: 3 })
.crawl("https://example.com", function onSuccess(page) {
console.log(page.url);
});configure() is optional. The documented default depth is 2, meaning the crawler follows links outward from the starting page to that depth. In the success callback, the page object includes the URL, response content (usually HTML), and HTTP status; the README also documents response-related fields and a referer.
Handle successes, failures, and completion
Use the options-based form when you need separate callbacks for successful pages, inaccessible pages, and the end of the crawl:
const Crawler = require("js-crawler").default;
const crawler = new Crawler();
crawler.configure({ depth: 2 });
crawler.crawl({
url: "https://example.com",
success(page) {
console.log("Fetched:", page.url, "status:", page.status);
},
failure(response) {
console.error("Could not access page:", response.url, "status:", response.status);
},
finished(urls) {
console.log("Crawl finished. URLs:", urls);
}
});The completion callback receives the collection of crawled URLs. A failure response’s status may be undefined, so do not assume every failed request includes an HTTP status code.
Rank #2
Control which URLs are visited
Depth limits how many links outward from the starting page are followed. For more precise scope, distinguish between whether a candidate URL may be requested and whether links on a fetched page should be added to the queue:
shouldCrawl(url)decides whether a candidate URL is requested.shouldCrawlLinksFrom(url)decides whether links found on a fetched page are added to the crawl queue.
For example, the following restricts requests to URLs on the example.com host while allowing link discovery only from its main page:
const Crawler = require("js-crawler").default;
const crawler = new Crawler();
crawler.configure({
depth: 4,
shouldCrawl(url) {
return new URL(url).hostname === "example.com";
},
shouldCrawlLinksFrom(url) {
return new URL(url).pathname === "/";
}
});
crawler.crawl("https://example.com", page => console.log(page.url));These are separate controls: allow a URL to be fetched with shouldCrawl, and decide whether it can lead the crawler to more URLs with shouldCrawlLinksFrom. Choose filters that match the site’s URL patterns and the pages you actually need.
Rank #3
Set request limits and other options
The README documents these options and defaults:
| Option | Documented default | Purpose |
|---|---|---|
depth |
2 | How many links outward from the start page are followed. |
ignoreRelative |
false | Whether relative URLs are skipped. |
userAgent |
crawler/js-crawler |
Request user-agent string. |
maxRequestsPerSecond |
100 | Upper request-rate limit. |
maxConcurrentRequests |
10 | Maximum number of active requests. |
Request rate and concurrency limit different things: one caps the number of requests issued per second, while the other caps simultaneous active requests. The actual rate also depends on network speed. For a gentler crawl, the README shows setting a rate of two requests per second:
const Crawler = require("js-crawler").default;
const crawler = new Crawler();
crawler.configure({
maxRequestsPerSecond: 2,
maxConcurrentRequests: 2
});
crawler.crawl("https://example.com", page => console.log(page.url));The rate setting is an upper limit, not a promise that the crawler will reach two requests per second. Set limits appropriate to the site and your use, and check applicable site terms and policies; technical throttling alone does not establish permission to crawl.
Reuse an instance deliberately
A crawler instance remembers URLs it has already crawled and, by default, will not crawl them again. To crawl those URLs again, clear that memory with forgetCrawled, as documented by the project, or create a new crawler instance. This matters when you run multiple crawl passes in the same process.
Rank #4
Know what js-crawler does—and what is not established
The project describes js-crawler as a Node.js web crawler supporting HTTP and HTTPS. Its documented callbacks expose response content, usually HTML. The README does not establish that it executes page JavaScript or renders browser-driven content. If important text or links appear only after client-side scripts run, the documented behavior is not enough to assume those elements will be available in the response.
For a page image rather than a link-and-content crawl, ScreenshotNeo is a separate option: it provides website screenshots through an API and MCP server. Learn more at ScreenshotNeo.
Troubleshoot common problems
- The import is undefined or fails: Check that
js-crawleris installed in the project from which you run Node and use the README’s CommonJS form,require("js-crawler").default. - The crawl finishes without the pages you expected: Check the configured depth and both URL-filter callbacks. Also check whether relative URLs are being skipped with
ignoreRelative. - A failure has no status code: The documented failure response may have an undefined
status. Treat failure as possible even when no HTTP status is available. - The crawl seems slower than the configured rate:
maxRequestsPerSecondis a ceiling, not a target; network speed and the concurrency cap can limit throughput. - JavaScript-rendered content is missing: The README does not document browser execution. Do not assume the crawler renders scripts; determine whether the needed content is present in the HTTP response.
Or skip the browser setup
If you need a screenshot rather than a crawl of links and response HTML, ScreenshotNeo takes a URL in one GET request. See the ScreenshotNeo API documentation.
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://example.com -o shot.webpScreenshotNeo accepts cookie and consent banners before capture and removes more than 60 known consent platforms, newsletter popups, and chat widgets; each of those steps can be turned off. Bot checks, blank pages, timeouts, failed loads, and cache hits cost nothing, and the response identifies the page verdict and billing status. Its MCP server offers screenshot tools for AI agents, and 1,000 screenshots a month are free with no card; paid plans start at $5 for 3,000. Sign up for the free plan.
Frequently Asked Questions
Does js-crawler support HTTPS?
Yes. The project describes support for both HTTP and HTTPS.
Can I crawl the same URLs again with the same instance?
Yes. Clear the instance’s remembered URLs with `forgetCrawled`, or make a new crawler instance.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
