Short answer: build a small stateful checker around a persistent requests.Session. Give every request explicit connect and read timeouts, classify HTTP and network outcomes, record latency and redirects, compare a normalized content region when needed, and alert only when the state changes. The complete example below monitors status, optional text, and content hashes without hanging your scheduler or flooding your inbox.
What the monitor should decide
A URL is not simply “up” or “down.” A useful result records the check time, requested URL, final URL, HTTP status, elapsed time, and any exception. It then assigns a state that matches your policy:
- healthy: an accepted status (usually 2xx) and any required text marker are present;
- redirect: a redirect is followed but is recorded separately when it matters to you;
- client_error (4xx) or server_error (5xx);
- timeout, dns_error, tls_error, or another network failure;
- content_changed: the response is reachable, but a normalized monitored region differs from the previous successful check.
Decide in advance whether redirects are healthy, degraded, or failures. Python’s HTTP handlers follow redirects by default, so always retain the final URL rather than treating a 301 or 302 as invisible.
Install and define a monitoring policy
Create an isolated environment and install Requests:
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Clear out junk files and repair common Windows errorsFree Scan →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →#1 Best Overall
- Includes Raspberry Pi 4 4GB Model B with 1.5GHz 64-bit quad-core CPU (4GB RAM)
- Includes Pre-Loaded 32GB EVO+ Micro SD Card (Class 10), USB MicroSD Card Reader
- CanaKit Premium High-Gloss Raspberry Pi 4 Case with Integrated Fan Mount, CanaKit Low Noise Bearing System Fan
- CanaKit 3.5A USB-C Raspberry Pi 4 Power Supply (US Plug) with Noise Filter, Set of Heat Sinks, Display Cable - 6 foot (Supports up to 4K60p)
- CanaKit USB-C PiSwitch (On/Off Power Switch for Raspberry Pi 4)
python -m venv .venv
. .venv/bin/activate # Windows: .venv\Scripts\activate
python -m pip install requestsStore targets as data, not scattered constants. Each target below accepts specific status codes, an optional required marker, and an optional CSS-like extraction rule that you can implement with an HTML parser if you need one. The baseline example hashes the whole response after removing obvious volatile text.
MONITOR_TARGETS = [
{
"url": "https://example.com/health",
"ok_statuses": {200},
"required_text": "OK",
"content": False,
},
{
"url": "https://example.com/status",
"ok_statuses": {200, 204},
"required_text": None,
"content": True,
},
]Use an honest User-Agent, keep polling within the site’s terms and rate limits, and do not monitor a URL you are not authorized to access.
A complete stateful Python monitor
This script uses one Session for connection reuse, separate connect/read timeouts, bounded retries for transient transport failures, JSON state on disk, and transition-based alerts. Replace send_alert with email, a webhook, or your incident system.
#!/usr/bin/env python3
import hashlib
import json
import os
import re
import time
from datetime import datetime, timezone
from pathlib import Path
from urllib.parse import urlparse
import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
STATE_FILE = Path(os.getenv("MONITOR_STATE", "monitor-state.json"))
CONNECT_TIMEOUT = 5
READ_TIMEOUT = 20
TARGETS = [
{"name": "home", "url": "https://example.com/", "ok_statuses": {200},
"required_text": None, "watch_content": False},
{"name": "health", "url": "https://example.com/health", "ok_statuses": {200},
"required_text": "OK", "watch_content": False},
]
def utc_now():
return datetime.now(timezone.utc).isoformat()
def make_session():
retry = Retry(
total=2,
connect=2,
read=2,
status=2,
backoff_factor=0.5,
status_forcelist={429, 500, 502, 503, 504},
allowed_methods={"GET", "HEAD"},
respect_retry_after_header=True,
)
session = requests.Session()
session.headers.update({"User-Agent": "freedom251-monitor/1.0"})
session.mount("https://", HTTPAdapter(max_retries=retry))
session.mount("http://", HTTPAdapter(max_retries=retry))
return session
def normalize(body):
"""Remove common volatile values before a content comparison."""
text = body.decode("utf-8", errors="replace")
text = re.sub(r"\s+", " ", text).strip()
text = re.sub(r"\b20\d{2}-\d{2}-\d{2}[^< ]*", "", text)
return text
def classify(response, target):
status = response.status_code
if status not in target["ok_statuses"]:
return "client_error" if 400 <= status < 500 else "server_error" if status >= 500 else "unexpected_status"
marker = target.get("required_text")
if marker and marker not in response.text:
return "unexpected_content"
return "healthy"
def check(session, target, previous):
started = time.perf_counter()
result = {"checked_at": utc_now(), "url": target["url"]}
try:
response = session.get(
target["url"],
timeout=(CONNECT_TIMEOUT, READ_TIMEOUT),
allow_redirects=True,
verify=True,
)
result.update({
"status_code": response.status_code,
"final_url": response.url,
"elapsed_ms": round((time.perf_counter() - started) * 1000, 1),
"state": classify(response, target),
})
if target.get("watch_content") and result["state"] == "healthy":
digest = hashlib.sha256(normalize(response.content).encode()).hexdigest()
result["content_sha256"] = digest
if previous and previous.get("content_sha256") and previous["content_sha256"] != digest:
result["state"] = "content_changed"
except requests.exceptions.ConnectTimeout:
result.update({"state": "timeout", "error": "connect timeout"})
except requests.exceptions.ReadTimeout:
result.update({"state": "timeout", "error": "read timeout"})
except requests.exceptions.SSLError as exc:
result.update({"state": "tls_error", "error": str(exc)})
except requests.exceptions.ConnectionError as exc:
result.update({"state": "dns_or_connection_error", "error": str(exc)})
except requests.exceptions.RequestException as exc:
result.update({"state": "request_error", "error": str(exc)})
result["elapsed_ms"] = round((time.perf_counter() - started) * 1000, 1)
return result
def send_alert(target, old_state, new_result):
print(f"ALERT {target['name']}: {old_state or 'unknown'} -> {new_result['state']} "
f"({new_result.get('status_code', new_result.get('error', ''))})")
def main():
try:
state = json.loads(STATE_FILE.read_text()) if STATE_FILE.exists() else {}
except (OSError, json.JSONDecodeError):
state = {}
session = make_session()
for target in TARGETS:
old = state.get(target["name"], {})
result = check(session, target, old)
if result["state"] != old.get("state"):
send_alert(target, old.get("state"), result)
state[target["name"]] = result
print(json.dumps(result, sort_keys=True))
STATE_FILE.write_text(json.dumps(state, indent=2))
if __name__ == "__main__":
main()Run it once with python monitor.py. The first run records a baseline; subsequent runs alert on healthy-to-failed, failed-to-healthy, or meaningful content transitions. Keep the state file on durable storage if the process runs from a container or ephemeral job.
Rank #2
- Includes Raspberry Pi 5 with 2.4Ghz 64-bit quad-core CPU (8GB RAM)
- Includes 128GB Micro SD Card pre-loaded with 64-bit Raspberry Pi OS, USB MicroSD Card Reader
- CanaKit Turbine Black Case for the Raspberry Pi 5
- CanaKit Low Noise Bearing System Fan
- Mega Heat Sink - Black Anodized
Scheduling without duplicate noise
Operating-system scheduler
On Linux, add a cron entry such as */5 * * * * /opt/monitor/.venv/bin/python /opt/monitor/monitor.py >> /var/log/monitor.log 2>&1. Use a service manager when you need restart policies, resource limits, and centralized logs. On Windows, create a Task Scheduler action that runs the virtual-environment Python executable and sets the working directory.
Prevent overlapping runs
If a check can exceed its schedule, use a lock (for example, a systemd service with one process or a platform-level single-instance setting). Overlapping writers can corrupt state and create duplicate alerts.
Content-change monitoring that survives real pages
Hashing raw HTML is noisy: timestamps, rotating ads, counters, personalization, and tracking markup change even when the page is functionally identical. Extract a stable region (such as an article body or a status element), remove known dynamic selectors, normalize whitespace, then hash the result. An HTML parser such as Beautiful Soup can select that region; store the selector and normalization rules alongside the target so a future edit is auditable. Treat a changed hash as a review signal, not automatically as an outage.
Retries, pooling, and scale
Retries should be bounded and selective. The example retries connection/read failures and 429/5xx responses twice with backoff, while deterministic 4xx responses are not retried. Respect a server’s Retry-After value. A persistent Session reuses connections; at higher concurrency, urllib3 provides pooling, thread-safe components, client-side TLS verification, redirect handling, compression, proxy support, and retry helpers. Use a pool sized for your actual concurrency rather than spawning an unbounded thread per URL.
Free tools Windows power users keep installed
One-click scans. No signup required.
Rank #3
- Includes Made in UK Raspberry Pi 3 B+ (B Plus) with 1.4 GHz 64-bit Quad-Core Processor, 1 GB RAM
- Dual Band 2.4GHz and 5GHz IEEE 802.11.b/g/n/ac Wireless LAN, Enhanced Ethernet Performance
- Includes 32 GB EVO+ Micro SD Card (Class 10) Pre-loaded with OS, USB MicroSD Card Reader
- CanaKit 2.5A USB Power Supply with Micro USB Cable and Noise Filter - Specially designed for the Raspberry Pi 3 B+ (UL Listed)
- Premium Raspberry Pi 3 B+ Case, Display Cable, 2 x Heat Sinks, GPIO Quick Reference Card, CanaKit Full Color Quick-Start Guide
For a small list, sequential checks are easiest to reason about. For many targets, use bounded workers, separate per-host rate limits, and a durable queue. Record each attempt and final classification so a transient failure is distinguishable from a persistent outage.
Security and operational safeguards
- Keep webhook, email, and API credentials in environment variables or a secret manager; never commit them to the script or state file.
- Leave TLS verification enabled. Disabling it hides certificate failures and creates a man-in-the-middle risk; only change it for a documented, controlled test.
- If users can submit URLs, allow only
http/https, resolve DNS, and block loopback, link-local, private, and other reserved destinations to prevent SSRF. Re-check after redirects because a public URL can redirect to an internal address. - Cap response size, reject unexpected schemes, and redact authorization headers and response bodies from logs.
- Check authorization, terms, robots guidance, and rate limits before polling. A monitor is not permission to bypass access controls or bot challenges.
Diagnosing common failures
| Symptom | Likely cause | Fix |
|---|---|---|
| Every check times out | Connect or read timeout too short, network path blocked, or host is overloaded | Inspect DNS and outbound firewall rules; measure connect versus read time; increase only the relevant timeout. |
| TLS error | Expired, mismatched, or untrusted certificate | Fix the certificate or trust chain. Do not set verify=False in production. |
| Unexpected 3xx | Redirect to login, HTTP-to-HTTPS, or a changed canonical URL | Review final_url and choose whether redirects are acceptable in the target policy. |
| Repeated 429/503 | Rate limiting or a temporary service failure | Honor Retry-After, reduce frequency, and use bounded exponential backoff. |
| Constant content changes | Dynamic timestamps, ads, counters, or personalization | Extract a stable element and normalize or exclude volatile fields before hashing. |
| Alert on every run | State is not persisted or alerts are based on results rather than transitions | Use durable state and alert only when the classified state changes. |
| Internal addresses are fetched | User-controlled URL or redirect creates SSRF | Validate schemes and resolved IP ranges before every request and after redirects. |
When to use a monitoring platform
A script is a good fit for a small URL list, a known schedule, and a simple rule you can maintain. Move to a platform when you need concurrent probes, persistent history, DNS/SSL/port checks, content-change detection, multiple alert routes, reports, Prometheus metrics, or centralized SSRF protections. The website-monitoring-automation project on PyPI describes bounded concurrency, persistence, change detection, alerts, reports, metrics, and SSRF guards; evaluate its current release and configuration before adopting it. A hosted service also removes the need to operate schedulers and durable state, but gives you less control over request logic and data location.
Or skip the browser setup
If your check needs a rendered page or a clean visual artifact, ScreenshotNeo is a website screenshot API and MCP server. It accepts consent banners before capture and removes more than 60 known consent platforms, newsletter popups, and chat widgets; bot checks, blank pages, timeouts, failed loads, and cache hits are not billed, and response headers identify the page verdict and billing result. Its MCP tools—take_screenshot, get_page_info, and capture_pdf—work with Claude, Cursor, and other MCP clients.
One request returns PNG, JPEG, WebP, or PDF. The API supports full-page lazy-image loading, CSS-selector element capture, dark mode, device presets and custom viewports, retina scale, PDF paper and page options, custom CSS/JavaScript, clicks, waits, blocking rules, headers, cookies, user agents, authorization, timezone, geolocation, transparent backgrounds, resizing, chosen cache TTLs, signed links, asynchronous jobs with signed webhooks, bulk capture of up to 100 URLs per call, a usage API, and an OpenAPI specification.
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpSee the ScreenshotNeo API documentation for options and response headers. Python and Node.js equivalents:
Rank #4
- Includes Raspberry Pi 5 with 2.4Ghz 64-bit quad-core CPU (4GB RAM)
- Includes 128GB Micro SD Card pre-loaded with 64-bit Raspberry Pi OS, USB MicroSD Card Reader
- CanaKit Turbine Black Case for the Raspberry Pi 5
- CanaKit Low Noise Bearing System Fan
- CanaKit Mega Heat Sink - Black Anodized
import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
open("shot.webp", "wb").write(r.content)const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);The Free plan includes 1,000 shots each month without a card; paid plans start at $5 for 3,000 shots. Sign up for the free plan.
FAQ
Should I use HEAD instead of GET?
Only when the endpoint reliably implements HEAD with the same availability semantics. GET is necessary for body markers and content hashes.
How should I handle authentication?
Use environment-backed headers or cookies, redact them from logs, and rotate credentials. Never place secrets in a URL that may be stored by proxies or logs.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Scan for outdated or missing drivers - takes under a minute3Clear out junk files and repair common Windows errorsCan a 200 response still be an outage?
Yes. A login page, maintenance message, missing marker, or changed content can return 200; combine status with the application-level checks your target requires.
Best Value
- 5 sets of code: Python (compatible with 2&3), C, Java, Scratch and Processing (Scratch and Processing code provide graphical interfaces)
- Detailed tutorial: Can be downloaded (in English, 962-page in total) or viewed online (original in English, can be translated into other languages by browsers) (The tutorial link can be found on the product box, no paper tutorial)
- 128 projects from simple to complex: Provides step-by-step guide with electronics and components knowledge, each project has schematics, wiring diagrams, complete code and detailed explanations
- 223 items in total: This ultimate kit includes the most commonly used electronic components, modules, sensors, wires and other compatible items
- Compatible models: Raspberry Pi 5 / 500 / 400 / 4B / 3B+ / 3B / 3A+ / 2B / 1B+ / 1A+ / Zero 2 W / Zero W / Zero (NOT included in this kit)
Frequently Asked Questions
Should I use HEAD instead of GET?
Only when the endpoint reliably implements HEAD with the same availability semantics. GET is necessary for body markers and content hashes.
How should I handle authentication?
Use environment-backed headers or cookies, redact them from logs, and rotate credentials. Never place secrets in a URL that may be stored by proxies or logs.
Can a 200 response still be an outage?
Yes. A login page, maintenance message, missing marker, or changed content can return 200; combine status with the application-level checks your target requires.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
