A dependable web monitor is a scheduled pipeline: fetch a page, extract the value that matters, normalize it, compare it with the previous observation, persist state, and deliver an alert. Use ordinary HTTP for static pages, conditional requests to avoid needless downloads, and a browser or underlying API for JavaScript-rendered content.
Start with the monitor pipeline
Model each monitor as a durable record rather than as a timer wrapped around a single script. A useful definition contains:
- Target: URL, HTTP method, selector or extraction rule, and the expected content type.
- Schedule: cadence, timezone, next due time, and a pause or maintenance window.
- Request context: descriptive user agent, headers, cookies or an authentication reference, proxy policy, and timeout limits.
- Change policy: the fields to compare, normalization rules, whether one or two consecutive observations must agree, and alert destinations.
- State: validators, last successful response, extracted value, content hash, error history, and emitted event identifiers.
A scheduler places due records on a durable queue. Workers fetch them with bounded per-host concurrency. An extractor produces a stable value, a comparator decides whether it changed, and an outbox sends notifications independently of the worker that found the change. This separation lets a mail or webhook outage be retried without fetching the site again or losing the event.
Choose HTTP or browser rendering
| Page type | First choice | Why | Typical failure |
|---|---|---|---|
| Server-rendered HTML, XML, CSV, or JSON | HTTP client | Lower resource use, simpler scaling, and straightforward conditional requests | The response contains a shell or placeholder instead of the value a visitor sees |
| Client-rendered application | Playwright browser context | Executes JavaScript and exposes the rendered DOM, requests, responses, and WebSockets | Selectors run before content appears, or a service worker hides the traffic you expected to intercept |
| Page backed by a documented JSON endpoint | Direct API request | More deterministic than rendering pixels and usually easier to rate-limit | Endpoint requires cookies, tokens, or a request sequence that your client does not reproduce |
Start with an HTTP request. Escalate only monitors whose meaningful value is absent from the response, or discover the request that supplies that value and monitor the endpoint instead. In Playwright, use a separate browser context per monitor or credential set. If request interception appears incomplete, account for service workers; disabling them for the monitoring context makes request and response events predictable.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Scan for outdated or missing drivers - takes under a minuteDriver Scan →#1 Best Overall
- Used Book in Good Condition
Use conditional requests before downloading again
Save the ETag and Last-Modified response headers with each successful observation. On the next check send them as If-None-Match and If-Modified-Since. A 304 Not Modified response means the body has not changed, so skip extraction and comparison. RFC 9110 describes conditional GET as the most efficient mechanism for HTTP cache updates; MDN notes that both validators support validation.
Validators are hints, not your only change detector. Some origins omit them, change them unnecessarily, or return weak validators while the selected field remains stable. If no validator is available, fetch the body and compare your normalized value. Never treat a 304 as an error, and replace saved validators only when the origin supplies new ones.
Extract a stable value, not an entire page
Selectors and structured fields
Prefer a product price, headline, availability flag, JSON property, or other narrow field over a whole-DOM snapshot. Whole pages contain navigation, timestamps, rotating recommendations, advertisement markup, randomized IDs, and whitespace that create false alerts.
Normalization
- Trim leading and trailing whitespace and collapse repeated internal whitespace.
- Remove known timestamps, rotating counters, ad containers, and tracking attributes only when they are irrelevant to the monitor.
- Parse numbers, currencies, dates, and booleans into a canonical representation before comparing them.
- Keep both the raw extraction and normalized value when an audit trail or human-readable diff matters.
Hashing and diffs
Hash the normalized value for a cheap equality check. When a change is found, retain the old and new values and generate a structured diff for the alert. For noisy pages, require the same new value in two consecutive successful checks; keep this rule per monitor so a rapidly changing stock count does not delay a price alert.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Rank #2
- Bookbound planner helps you keep track of passwords and favorite websites
- Room for over 200 entries; 3.5 x 6 inch page sizes
- User name and security questions field
- Tips for what makes a strong password; web resources; notes pages
- Printed on quality paper containing 30% post-consumer waste; black simulated leather cover; 3.63 x 6.13 x .21 inches
A runnable Python monitor worker
The following compact worker demonstrates SQLite state, conditional GET, a CSS selector, normalization, a two-check confirmation option, and a webhook notification. Install its two dependencies with pip install requests beautifulsoup4. Replace the sample URL, selector, and webhook with your own values.
import hashlib
import json
import sqlite3
import time
from datetime import datetime, timezone
from urllib.parse import urlparse
from urllib.robotparser import RobotFileParser
import requests
from bs4 import BeautifulSoup
DB = "monitors.db"
USER_AGENT = "ExampleWebMonitor/1.0 (+https://example.com/monitor-info)"
MONITOR = {
"name": "price",
"url": "https://example.com/product",
"selector": "[data-price]",
"cadence_seconds": 900,
"webhook": "https://hooks.example.net/monitor",
"confirmations": 2,
}
def db():
con = sqlite3.connect(DB)
con.execute("""CREATE TABLE IF NOT EXISTS state (
name TEXT PRIMARY KEY, etag TEXT, last_modified TEXT,
value TEXT, fingerprint TEXT, pending TEXT, pending_count INTEGER DEFAULT 0,
checked_at TEXT, error TEXT)""")
return con
def allowed(url):
parts = urlparse(url)
rp = RobotFileParser()
rp.set_url(f"{parts.scheme}://{parts.netloc}/robots.txt")
try:
rp.read()
return rp.can_fetch(USER_AGENT, url)
except Exception:
# An unreachable robots.txt is not authorization; apply your own policy.
return False
def normalize(text):
return " ".join(text.split())
def notify(url, old, new, hook):
payload = {"url": url, "old": old, "new": new,
"checked_at": datetime.now(timezone.utc).isoformat()}
r = requests.post(hook, json=payload, timeout=(5, 15))
r.raise_for_status()
def check(monitor):
if not allowed(monitor["url"]):
raise RuntimeError("robots.txt disallows this user agent or could not be read")
con = db()
row = con.execute("SELECT etag,last_modified,value,fingerprint,pending,pending_count FROM state WHERE name=?",
(monitor["name"],)).fetchone()
headers = {"User-Agent": USER_AGENT}
if row and row[0]: headers["If-None-Match"] = row[0]
if row and row[1]: headers["If-Modified-Since"] = row[1]
response = requests.get(monitor["url"], headers=headers,
timeout=(10, 30), allow_redirects=True)
if response.status_code == 304:
con.execute("UPDATE state SET checked_at=?, error=NULL WHERE name=?",
(datetime.now(timezone.utc).isoformat(), monitor["name"]))
con.commit(); con.close(); return "not-modified"
response.raise_for_status()
soup = BeautifulSoup(response.text, "html.parser")
node = soup.select_one(monitor["selector"])
if node is None:
raise RuntimeError("selector matched no element")
value = normalize(node.get_text(" "))
fingerprint = hashlib.sha256(value.encode()).hexdigest()
old_value = row[2] if row else None
pending = row[4] if row else None
pending_count = row[5] if row else 0
if old_value is None:
pending, pending_count = value, monitor["confirmations"]
elif fingerprint != (row[3] or ""):
pending_count = pending_count + 1 if pending == value else 1
pending = value
if pending_count >= monitor["confirmations"]:
notify(monitor["url"], old_value, value, monitor["webhook"])
old_value, pending, pending_count = value, None, 0
con.execute("""INSERT INTO state(name,etag,last_modified,value,fingerprint,pending,pending_count,checked_at,error)
VALUES(?,?,?,?,?,?,?,?,NULL)
ON CONFLICT(name) DO UPDATE SET etag=excluded.etag,
last_modified=excluded.last_modified,value=excluded.value,
fingerprint=excluded.fingerprint,pending=excluded.pending,
pending_count=excluded.pending_count,checked_at=excluded.checked_at,error=NULL""",
(monitor["name"], response.headers.get("ETag"),
response.headers.get("Last-Modified"), old_value, fingerprint,
pending, pending_count, datetime.now(timezone.utc).isoformat()))
con.commit(); con.close(); return "checked"
if __name__ == "__main__":
while True:
try:
print(check(MONITOR))
except Exception as exc:
print(f"check failed: {exc}")
time.sleep(MONITOR["cadence_seconds"])For production, move monitor definitions and secrets out of source code, put due checks on a durable queue, and record HTTP status, response size, extraction status, and notification delivery as separate fields. The example refuses an unreadable robots.txt; you may choose a different documented policy, but do not silently crawl when your compliance decision is unknown.
Add a Playwright worker for JavaScript pages
Install Playwright and its browser once, then use a browser context only for monitors that need it. The example waits for a selector, captures the rendered value, and observes responses so you can identify an underlying API.
import asyncio
from playwright.async_api import async_playwright
async def rendered_value(url, selector):
async with async_playwright() as p:
browser = await p.chromium.launch()
context = await browser.new_context(service_workers="block")
page = await context.new_page()
api_responses = []
page.on("response", lambda r: api_responses.append(r.url)
if "api" in r.url else None)
await page.goto(url, wait_until="domcontentloaded", timeout=45000)
await page.wait_for_selector(selector, state="visible", timeout=30000)
value = await page.locator(selector).inner_text()
await browser.close()
return " ".join(value.split()), api_responses
value, requests_seen = asyncio.run(
rendered_value("https://example.com/app", "[data-price]")
)
print(value)
print(requests_seen)Use a navigation timeout, a selector timeout, and a total job timeout. If the response list reveals a stable JSON endpoint, prefer an authenticated HTTP request to that endpoint. Keep cookies and authorization isolated by context and never print them in response logs.
Recommended Free Tools
Rank #3
Respect robots.txt, terms, and host capacity
Fetch /robots.txt for every host, match your monitor’s user agent to the appropriate group, and apply the most specific Allow or Disallow rule. RFC 9309 states that these rules are for crawlers to honor but are not access authorization. They do not override a login boundary, contract, copyright restriction, or published rate limit.
Cache a successful robots file according to its response headers. RFC 9309 says not to use a cached copy for more than 24 hours unless the file is unreachable. Use a descriptive user agent, a per-host concurrency limit, and a cadence that gives the site time to respond. A monitor that needs faster detection should negotiate an API, webhook, or feed rather than multiplying parallel polls.
Make failures boring and observable
Timeouts and retries
Set connect, read, navigation, and total-job limits. Retry 429 and transient 5xx responses with capped exponential backoff and jitter. Do not blindly retry authentication failures or permanent 4xx responses. Honor a server’s Retry-After value when present.
Deduplication and delivery
Give each observation a fingerprint and each notification an idempotency key such as monitor ID plus fingerprint. Write the event to an outbox in the same transaction that records the new state. A delivery worker can retry the outbox while preserving one logical alert.
The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Rank #4
- Used Book in Good Condition
Metrics and health checks
- Check latency and queue age, separated by host.
- HTTP status distribution, response size, and conditional-hit rate.
- Extraction failures, selector miss rate, and browser launch failures.
- Notification attempts, provider responses, and time to delivery.
- Consecutive monitor failures, which should alert separately from content changes.
Playwright’s web-server configuration is a useful pattern for worker dependencies: wait for a URL, reuse an existing server when possible, and set an explicit startup timeout. Apply the same principle to your browser pool and queue consumers.
Troubleshooting common monitor failures
| Symptom | Likely cause | Fix |
|---|---|---|
| Every poll downloads the full page | Validators are not saved or headers are misspelled | Persist response ETag/Last-Modified and send the exact conditional request headers; handle 304 explicitly |
| Selector matches nothing | Content is rendered later, selector changed, or an iframe contains the value | Wait for the selector in Playwright, inspect the rendered DOM, handle the frame, or monitor the backing API |
| Alerts fire on every check | Whitespace, timestamps, ads, or randomized IDs change | Normalize and remove only known non-semantic fields; hash the selected value instead of the whole DOM |
| Browser sees fewer requests than the page | A service worker handles them | Use a context with service workers blocked while diagnosing interception |
| Many 429 responses | Cadence or concurrency is too aggressive | Lower per-host concurrency, increase the interval, honor Retry-After, and add jitter |
| Duplicate alerts after a restart | State was updated but notification delivery was not recorded atomically | Use a transactional outbox and idempotency key |
| Credentials appear in logs | Full request URLs or headers are being logged | Redact authorization, cookies, query secrets, and page contents before persistence |
Or skip the browser setup
ScreenshotNeo provides a website screenshot API and MCP server when your monitor needs a rendered visual or PDF rather than a custom browser fleet. It accepts consent banners before capture and removes more than 60 known consent platforms, newsletter popups, and chat widgets; each cleanup step can be disabled. Bot checks, CAPTCHAs, blank pages, timeouts, failed loads, and cache hits are not billed, and every response reports its result in X-Page-Verdict and X-Billed headers.
One GET request returns PNG, JPEG, WebP, or PDF. The API supports full-page and CSS-selector captures, lazy-image loading, dark mode, device presets or custom viewports, retina scale, PDF paper settings and page ranges, custom CSS and JavaScript, clicks, waits, request blocking, headers, cookies, user agents, authorization, timezone, geolocation, transparent backgrounds, resizing, selectable cache TTL, signed image links, asynchronous jobs with signed webhooks, bulk capture of up to 100 URLs per call, a usage API, and an OpenAPI specification. Parameter names used by other screenshot APIs also work, which eases migration.
Use the ScreenshotNeo API documentation for authentication and the complete option list. A minimal call is:
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpPython:
import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
open("shot.webp", "wb").write(r.content)Node.js:
const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);
const body = Buffer.from(await res.arrayBuffer());ScreenshotNeo includes an MCP server with take_screenshot, get_page_info, and capture_pdf tools for Claude, Cursor, and other MCP clients. The Free plan includes 1,000 shots per month without a card; paid plans start at $5 for 3,000 shots. Current plan allowances are:
Best Value
- 【Featured A-Z Tabs & Untitle for Security】Our password books have recognizable alphabetical tabs with the colorful design allow you to locate quickly and save time. The anonymous cover of our password keeper is unobtrusive and stays secure.
- 【Premium Quality & Perfect Size】This password journal features a eco-leather hardcover and 100gsm no-bleed paper, equipped with an elastic band, inner pocket, pen loop and bookmark. It comes in medium format (5.3 x 7.7 inches) which is the perfect size you need.
- 【Clean Layout & Plenty of Space】 Each tab has 6 pages with 4 entries per page and contains more than 552 passwords in our password organizer. This password notebook also provides more password space in case you need to change your password.
- 【Perfect Organization & Safe Placement】We ensure this password log book provides you with a secure space to keep passwords and web addresses. You won't have to worry about passwords being leaked or hacked.
- 【Thoughtful Gift & Warm Heart】 Considering for practical gifts for family or friends? Our specially designed internet password book is sturdy and easy to use. Ideal for any occasion, it's a gift that truly shows care.
| Plan | Price | Shots |
|---|---|---|
| Free | $0 | 1,000 per month |
| Starter | $5 | 3,000 |
| Growth | $15 | 15,000 |
| Pro | $39 | 60,000 |
| Scale | $99 | 250,000 |
| Business | $249 | 1,000,000 |
Yearly billing gives two months free, and every feature is on every plan. Start with 1,000 free screenshots a month without a card.
FAQ
How should I handle a site’s planned maintenance window?
Pause the monitor or suppress alerts for that window, but continue recording checks and HTTP results. This preserves an audit trail and prevents a planned outage from looking like an unexpected change.
Should an alert contain the whole old and new page?
Usually no. Include the selected field, check time, URL, and a short diff. Store larger raw values behind authenticated access with a retention limit appropriate to the content.
When is a webhook better than polling?
Use a publisher-provided webhook or feed when available and contractually permitted. It can remove polling load and reduce alert delay, while a periodic monitor remains useful as a reconciliation check.
Frequently Asked Questions
How should I handle a site’s planned maintenance window?
Pause the monitor or suppress alerts for that window while recording checks and HTTP results, so planned downtime remains distinguishable from an unexpected change.
Should an alert contain the whole old and new page?
Usually not. Send the selected field, check time, URL, and a short diff; retain larger raw values privately with an appropriate retention limit.
When is a webhook better than polling?
Use a publisher-provided webhook or feed when available and permitted, with periodic polling as a reconciliation check.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
