Some links on this page are affiliate links: if you buy through them we may earn a commission, at no extra cost to you.
Choose the method that matches the page’s protection: use HTTPS Basic authentication for a username and password, an Authorization header for a bearer token, or send cookies after a login request for a cookie-based session. If sign-in depends on JavaScript, browser storage, or interactive authentication, use Playwright or Puppeteer instead of trying to reproduce a browser with a plain HTTP request.
This guide covers Node.js’s built-in HTTP and Fetch options, cookie handling, browser-based sessions, and the security and troubleshooting details that commonly cause authenticated requests to fail.
Choose the right authentication method
First identify what the server expects. A page that prompts for credentials before returning content may use HTTP Basic authentication. An API may require a bearer token or another custom header. A website may authenticate a form submission and then rely on cookies. A JavaScript application may need browser features that ordinary HTTP requests do not provide.
| What the site requires | Suitable Node.js approach | What it does not do for you |
|---|---|---|
| HTTP Basic username and password | https.request() or https.get() with auth |
It does not submit a website login form or maintain a browser session. |
| Bearer token or custom header | Built-in fetch or Undici with an Authorization or other header |
It does not make a token valid or prevent it from being exposed by unsafe logging. |
| Cookie-based session | Send a login request, retain relevant cookies, and include them on later requests; use a cookie jar if you need managed persistence | Undici’s cookie helpers do not implement a cookie jar or network activity. |
| JavaScript login or browser-only state | Playwright or Puppeteer | A plain HTTP request does not run page JavaScript or provide browser storage and interaction. |
Only automate accounts and pages you are authorized to access. Use HTTPS for credentials and tokens in transit, keep secrets outside source code, and treat saved session state as a credential.
#1 Best Overall
Use HTTP Basic authentication
Node’s http and https request APIs accept an auth option in the form user:password, which computes a Basic Authorization header. For a real protected site, use https so credentials are encrypted in transit. The Node.js HTTP API documents the auth option.
Example using environment variables, with the target URL supplied by an environment variable as well:
const https = require('node:https');
const target = new URL(process.env.TARGET_URL);
const user = process.env.BASIC_USER;
const password = process.env.BASIC_PASSWORD;
if (target.protocol !== 'https:') {
throw new Error('Use an HTTPS URL for Basic credentials.');
}
if (!user || !password) {
throw new Error('Set BASIC_USER and BASIC_PASSWORD.');
}
const req = https.request(target, {
method: 'GET',
auth: `${user}:${password}`,
}, (res) => {
console.log('status:', res.statusCode);
console.log('location:', res.headers.location || '(none)');
let body = '';
res.setEncoding('utf8');
res.on('data', chunk => { body += chunk; });
res.on('end', () => {
if (res.statusCode >= 300 && res.statusCode < 400) {
console.error('Redirect received; inspect Location and handle it deliberately.');
} else if (res.statusCode === 401) {
console.error('Unauthorized: verify credentials and the requested auth scheme.');
} else if (res.statusCode === 403) {
console.error('Forbidden: the account may lack permission.');
} else if (res.statusCode < 200 || res.statusCode >= 300) {
console.error('Request failed with status', res.statusCode);
} else {
console.log(body);
}
});
});
req.setTimeout(30000, () => req.destroy(new Error('Request timed out')));
req.on('error', err => console.error('Request error:', err.message));
req.end();Run it by setting TARGET_URL, BASIC_USER, and BASIC_PASSWORD in the process environment, then executing the file with Node.js. Do not paste actual credentials into a committed script or log them. Basic authentication is a transport-level HTTP scheme; it is not the same as filling in a website’s login form.
If you set an explicit Authorization header in the request options, it takes precedence over the auth option. Avoid setting both unless you intentionally want the explicit header to win.
Send a bearer token or custom header with fetch
Current Node.js releases provide a global fetch implementation based on Undici. Supply the token in the request headers and check the response status before assuming the body is the protected page. Undici documents Fetch-compatible request and response behavior.
Rank #2
const url = process.env.TARGET_URL;
const token = process.env.ACCESS_TOKEN;
if (!url || !token) {
throw new Error('Set TARGET_URL and ACCESS_TOKEN.');
}
const response = await fetch(url, {
method: 'GET',
headers: {
Authorization: `Bearer ${token}`,
Accept: 'text/html,application/json',
},
redirect: 'manual',
signal: AbortSignal.timeout(30000),
});
console.log('status:', response.status);
console.log('content type:', response.headers.get('content-type'));
if (response.status >= 300 && response.status < 400) {
console.log('redirect location:', response.headers.get('location'));
} else if (response.status === 401) {
throw new Error('401 Unauthorized: check token, expiry, and expected auth scheme.');
} else if (response.status === 403) {
throw new Error('403 Forbidden: token may be valid but lack access.');
} else if (!response.ok) {
throw new Error(`Request failed: HTTP ${response.status}`);
}
const contentType = response.headers.get('content-type') || '';
const body = contentType.includes('application/json')
? await response.json()
: await response.text();
console.log(body);Use a bearer token only when the service documents that scheme; some APIs require a different header name, token prefix, or request method. The example sets redirect: 'manual' so you can inspect redirects rather than silently forwarding sensitive authentication to an unexpected destination. If you choose automatic redirect handling, understand the runtime’s redirect behavior and ensure secrets cannot be forwarded across origins.
Log in and reuse a cookie session
A cookie-based login usually involves two requests: submit the login request, then send the session cookie with the protected-page request. Cookies have domain, path, and security scope. Do not copy every cookie blindly between unrelated hosts or send session cookies to a broader domain than necessary.
Undici provides helpers such as getSetCookies(), getCookies(), setCookie(), and parseCookie(). They parse and mutate supplied headers; they do not create a cookie jar, enforce persistence policy, or perform requests. The Undici cookie documentation explicitly notes that these helpers do not manage a jar or perform network activity.
A minimal illustration for a service that returns a single session cookie named session is below. Adapt the login URL, request body, and cookie name to the service’s documented contract; real sites may return multiple cookies or require CSRF tokens and additional headers.
const loginUrl = process.env.LOGIN_URL;
const pageUrl = process.env.PROTECTED_URL;
const username = process.env.LOGIN_USER;
const password = process.env.LOGIN_PASSWORD;
if (![loginUrl, pageUrl, username, password].every(Boolean)) {
throw new Error('Set LOGIN_URL, PROTECTED_URL, LOGIN_USER, LOGIN_PASSWORD.');
}
const login = await fetch(loginUrl, {
method: 'POST',
headers: { 'content-type': 'application/json', Accept: 'application/json' },
body: JSON.stringify({ username, password }),
redirect: 'manual',
signal: AbortSignal.timeout(30000),
});
if (!login.ok) {
throw new Error(`Login failed with HTTP ${login.status}`);
}
// This example assumes the service sets one cookie called "session".
const setCookie = login.headers.get('set-cookie');
if (!setCookie) {
throw new Error('No Set-Cookie received; inspect the login response and auth flow.');
}
const sessionPair = setCookie.split(';', 1)[0];
if (!sessionPair.startsWith('session=')) {
throw new Error('Expected session cookie was not found.');
}
const page = await fetch(pageUrl, {
headers: { Cookie: sessionPair },
redirect: 'manual',
signal: AbortSignal.timeout(30000),
});
console.log('protected-page status:', page.status);
if (page.status === 401 || page.status === 403) {
throw new Error(`Session was not accepted: HTTP ${page.status}`);
}
if (page.status >= 300 && page.status < 400) {
console.log('redirect location:', page.headers.get('location'));
} else if (!page.ok) {
throw new Error(`Page request failed with HTTP ${page.status}`);
} else {
console.log(await page.text());
}This deliberately narrow example is not a general-purpose cookie parser: a response can contain multiple Set-Cookie fields, and cookie values can include characters that make ad hoc splitting unsafe. For production flows, use a cookie-jar implementation that respects domain, path, expiry, Secure, and SameSite rules, or use Undici’s helpers to parse headers while implementing the persistence and scope policy yourself. Do not assume one headers.get('set-cookie') value represents every cookie in every runtime or response.
Rank #3
When the login flow uses CSRF protection, first fetch the login page and extract its CSRF token as required by the service, then submit it with the login request. If a valid-looking cookie is repeatedly rejected, check whether the login response also requires a second cookie or whether the site binds sessions to additional request state.
Use a real browser for JavaScript-based authentication
Use browser automation when login depends on running page JavaScript, interactive form behavior, local storage, IndexedDB, passkeys, or WebAuthn. Playwright can save and reuse authenticated storage state that includes cookies, local storage, IndexedDB, and passkey/WebAuthn-based authentication. Its authentication guide also cautions that session storage is domain-specific and is not persisted across page loads; save or initialize it separately if the application depends on it.
Here is a compact Playwright example that signs in through the UI, writes storage state, and then reuses it for a later run. Replace selectors and URLs with those from the site you are authorized to use:
const { chromium } = require('playwright');
(async () => {
const browser = await chromium.launch();
const context = await browser.newContext();
const page = await context.newPage();
try {
await page.goto(process.env.LOGIN_URL, { waitUntil: 'domcontentloaded' });
await page.getByLabel('Email').fill(process.env.LOGIN_USER);
await page.getByLabel('Password').fill(process.env.LOGIN_PASSWORD);
await page.getByRole('button', { name: /sign in|log in/i }).click();
await page.waitForURL(process.env.AUTHENTICATED_URL_PATTERN, { timeout: 30000 });
await context.storageState({ path: 'auth-state.json', indexedDB: true });
await page.goto(process.env.PROTECTED_URL, { waitUntil: 'domcontentloaded' });
console.log('page title:', await page.title());
} finally {
await browser.close();
}
})();For later use, create the context with browser.newContext({ storageState: 'auth-state.json' }). The state file grants access to the session: exclude it from version control, restrict file permissions, store it in an appropriate secret store for automation, and rotate or regenerate it when the session expires. Playwright’s option to include IndexedDB is available in current documentation; check the installed Playwright version’s API if that option is unavailable in an older project.
Seed a browser context with an API login
If the application’s documented login endpoint accepts a normal HTTP request, Playwright’s APIRequestContext can authenticate and save storage state that is interchangeable with a BrowserContext. This avoids UI interaction when a browser is still needed for the protected page.
Rank #4
const { request, chromium } = require('playwright');
(async () => {
const api = await request.newContext({
httpCredentials: process.env.BASIC_USER && process.env.BASIC_PASSWORD
? { username: process.env.BASIC_USER, password: process.env.BASIC_PASSWORD }
: undefined,
});
try {
const response = await api.post(process.env.LOGIN_URL, {
data: { username: process.env.LOGIN_USER, password: process.env.LOGIN_PASSWORD },
});
if (!response.ok()) {
throw new Error(`API login failed: HTTP ${response.status()}`);
}
await api.storageState({ path: 'auth-state.json' });
const browser = await chromium.launch();
try {
const context = await browser.newContext({ storageState: 'auth-state.json' });
const page = await context.newPage();
await page.goto(process.env.PROTECTED_URL);
console.log(await page.title());
} finally {
await browser.close();
}
} finally {
await api.dispose();
}
})();Use the API login shape the service actually specifies; this sample demonstrates the state handoff rather than asserting that a particular site accepts these fields. Playwright documents APIRequestContext support for HTTP credentials and storage-state reuse.
Use Puppeteer for HTTP authentication
Puppeteer’s page.authenticate(credentials) supplies credentials for HTTP authentication. It is useful when the task already requires a Chromium page, but it is not a general replacement for submitting a site’s login form. Puppeteer notes that authentication enables request interception behind the scenes, which may affect performance. See the Puppeteer API documentation.
const puppeteer = require('puppeteer');
(async () => {
const browser = await puppeteer.launch();
try {
const page = await browser.newPage();
await page.authenticate({
username: process.env.BASIC_USER,
password: process.env.BASIC_PASSWORD,
});
await page.goto(process.env.PROTECTED_URL, { waitUntil: 'domcontentloaded' });
console.log('status:', (await page.goto(process.env.PROTECTED_URL))?.status());
} finally {
await browser.close();
}
})();For a real implementation, navigate once and retain the result rather than calling goto twice; the important sequence is authenticate, navigate, then inspect the result. As with direct HTTP requests, use an HTTPS target for credentials and protect secrets through environment or secret-manager injection.
Redirects, status codes, and response handling
A response status is not proof that authentication succeeded or failed by itself. Inspect status, redirect destination, content type, and the body shape expected from the service.
- 401 Unauthorized: credentials may be missing, malformed, expired, or sent using the wrong scheme. A server may also issue a challenge that identifies the expected authentication mechanism.
- 403 Forbidden: the identity may have authenticated but lack permission for that resource, or the site may apply another access policy.
- 3xx redirect: inspect
Location. A redirect to a login page often means the session was not recognized. Avoid forwarding secrets to another origin unless that behavior is explicitly intended. - 200 with a login page: some sites return the sign-in page with a success status. Check the returned URL, title, content type, and expected page markers rather than relying on
response.okalone. - HTML when JSON was expected: a redirect or authentication failure may have delivered an HTML page. Check the content type before parsing JSON.
Security and operational considerations
- Protect credentials: inject tokens and passwords from environment variables or a secret manager, and redact authorization headers and cookie values from logs.
- Use TLS: send credentials and session cookies only over HTTPS. Do not disable certificate verification to work around a connection error.
- Limit session scope: send cookies only to the intended host and path. Store browser state files as secrets and do not commit them.
- Set timeouts: use a timeout or abort signal so a hung request does not hold the process indefinitely.
- Keep browser usage purposeful: direct HTTP is simpler and lighter when the server accepts HTTP authentication or an API token; browser automation carries browser startup and lifecycle overhead.
- Respect authorization and terms: do not bypass access controls or automate pages without permission.
Troubleshooting common failures
| Symptom | Likely cause | What to check or change |
|---|---|---|
| 401 from a Basic-auth endpoint | Wrong username/password, wrong host, or endpoint expects another scheme | Confirm the URL and credentials, inspect the server’s authentication challenge, and verify that an explicit Authorization header is not overriding auth. |
| 403 after a successful login | Authenticated identity lacks permission, or another policy blocks access | Check the account’s access rights and the specific resource policy; do not treat repeated password changes as a fix for an authorization denial. |
| Login works in a browser but fetch is redirected to sign-in | The login depends on JavaScript, CSRF, additional cookies, or browser-managed state | Inspect the login flow and response headers. Use a properly scoped cookie jar for a supported HTTP flow, or switch to Playwright/Puppeteer when browser behavior is required. |
| Cookie exists but protected page rejects it | Cookie omitted, wrong domain/path, expired session, missing companion cookie, or session requires further state | Inspect all relevant Set-Cookie headers and the request destination; use a jar that applies cookie scope and expiry rules. |
| JSON parsing throws | Response is HTML, often a login page or redirect target | Check status and Content-Type before choosing response.json(); inspect redirect handling. |
| Browser storage reuse fails | State expired, file is unreadable, or required state is session storage rather than persisted storage state | Regenerate state with a valid login, confirm the intended origin, and separately initialize session storage if the application requires it. |
| Request hangs or fails intermittently | No timeout, network issue, or slow page behavior | Set bounded request timeouts, surface errors, and for browser navigation wait for the page condition your task needs rather than assuming all network activity ends at the same moment. |
Or skip the browser setup
If your task is to capture a page rather than build a reusable authenticated application session, ScreenshotNeo is a website screenshot API and MCP server for developers. Its request supports custom headers, cookies, user agent, and Authorization, along with browser-oriented capture options; it is not a replacement for an interactive sign-in flow that requires you to establish a session first.
One-call cURL example for a public page:
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpFor protected captures, use the API’s documented request parameters to pass the authorization material appropriate to the page, and keep API keys and credentials out of shared scripts. See the ScreenshotNeo documentation for parameter details.
- Cookie banners, newsletter popups, and chat widgets are removed before the shot; each cleanup step can be turned off.
- Bot checks, blank pages, and failed loads are never billed; response headers identify the page verdict and billing status.
- An MCP server lets AI agents use the
take_screenshot,get_page_info, andcapture_pdftools. - 1,000 screenshots a month are free with no card; paid plans start at $5 for 3,000 shots.
Sign up for ScreenshotNeo free: 1,000 screenshots per month, no card required.
Frequently Asked Questions
Does Node.js fetch keep cookies between requests automatically?
No. Treat cookie persistence as application responsibility or use a cookie-jar solution; Undici’s cookie helpers parse and manipulate headers but do not maintain a jar.
Can I use a saved Playwright state file on another machine?
It may be usable if the runtime, origin, and session remain valid, but it is a credential-bearing file. Transfer it only through protected storage and expect sessions to expire or be invalidated.
Should I use fetch or Playwright for a protected page?
Use fetch when the server accepts a documented HTTP credential, token, or cookie flow; use Playwright when the login or page depends on browser execution or browser storage.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
