PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteThe dependable way to block a bad bot on Apache is to reject its request at the server with an Apache 2.4 access-control rule or a narrowly scoped mod_rewrite rule that returns HTTP 403. A robots.txt entry alone is only a voluntary request and will not stop a scraper that ignores it.
Antihost offers Linux VPS plans with full root access, unlimited traffic on a 1 Gbps port and a 14-day free trial.
Start with access-log evidence, match the smallest path and clearest identifying signal, and treat a User-Agent denylist as a coarse filter. User-Agent headers are supplied by the client and can be changed trivially; sustained or distributed attacks belong at a firewall, managed WAF, CDN security layer or another upstream control.
As an Amazon Associate I earn from qualifying purchases.
Choose the control that matches the problem
Apache gives you several enforcement points. They do not provide the same protection, and placing a rule closer to the application does not automatically make it stronger. The practical choice depends on whether the crawler is merely impolite, whether it identifies itself consistently, and whether the traffic is consuming enough resources to require upstream filtering.
| Control | What it does | Enforcement strength | Typical scope | Can the client evade it? | Apache resources used before rejection |
|---|---|---|---|---|---|
robots.txt |
Publishes a crawler-exclusion request | Voluntary only | Site or selected paths, as interpreted by a crawler | Yes. A hostile bot can ignore the file entirely | The request still reaches the web server to fetch the file, and later requests are not automatically rejected |
| Apache access control | Denies a request when an expression or environment flag matches | Stronger than a voluntary signal for the requests Apache evaluates | A directory, location, virtual host or other configured scope | Yes, if the bot changes the identifying header or otherwise avoids the match | Apache processes the request and then returns a denial |
mod_rewrite with [F] |
Matches request conditions and returns HTTP 403 | Useful coarse filter; strength depends on the conditions | Can be limited to one path or expanded to a virtual host | Yes, when the rule relies only on User-Agent text | Apache evaluates the rewrite conditions before rejecting |
| Firewall, managed WAF or CDN security layer | Filters traffic before it reaches the origin Apache process | Best category for sustained, volumetric or distributed attacks | Network, hostname, route, rate, IP and other upstream dimensions | Depends on the signals and policy; no single layer is perfect | Less origin capacity is consumed because unwanted traffic is stopped upstream |
For a bot that announces a stable name and is requesting one sensitive directory, Apache is often sufficient. For a bot that rotates names, changes IP addresses, opens many connections or keeps the origin busy despite denials, move the control higher in the stack.
#1 Best Overall
Inspect logs before writing a deny rule
A broad pattern can block legitimate visitors, monitoring systems or a useful crawler. Review the access log entries for the suspected bot and record the exact request path, User-Agent spelling and capitalization, source addresses, response status and request frequency. Look for a pattern that distinguishes the unwanted traffic without matching normal users.
- Confirm that the same User-Agent appears on the requests you intend to stop.
- Check whether the bot requests one route, a directory or the entire site.
- Compare several entries rather than relying on a single request; clients can send different headers on different requests.
- Note whether the traffic is coming from a small, stable address range. An IP condition can narrow a rule, but an address-only block can become stale when the source changes.
- Identify any legitimate automation that uses the same header before denying it.
Keep the first rule narrow. You can widen it after observing the result, but a mistake in a broad denylist can make an entire site or virtual host unavailable.
Block a User-Agent with Apache 2.4 access control
Apache 2.4 supports expression-based authorization and environment-variable conditions. The Apache documentation also shows a SetEnvIfNoCase pattern that sets a flag when the User-Agent matches, then denies that flag in a protected location.
Set an environment flag with SetEnvIfNoCase
This example marks requests whose User-Agent starts with NameOfBadRobot and denies those requests under /secret/files:
SetEnvIfNoCase User-Agent ^NameOfBadRobot goaway
<Location "/secret/files">
<RequireAll>
Require all granted
Require not env goaway
</RequireAll>
</Location>SetEnvIfNoCase performs a case-insensitive match, so capitalization changes in the bot’s spelling do not defeat this particular rule. The <RequireAll> container first permits normal requests and then excludes requests carrying the goaway environment flag.
Rank #2
- Used Book in Good Condition
The <Location> example is written for Apache configuration where a location container is allowed, such as the server or virtual-host configuration. Hosting providers may restrict those contexts. If you are using .htaccess, confirm that your host permits the directives you plan to use and that the relevant authorization modules are enabled; a syntactically correct rule in a disallowed context will produce a configuration error rather than a useful block.
Use an Apache 2.4 expression
For a single exact User-Agent value, Apache 2.4 can deny requests directly with an <If> expression:
<If "%{HTTP_USER_AGENT} == 'BadBot'">
Require all denied
</If>This expression compares the request’s HTTP_USER_AGENT value with BadBot. It is intentionally exact: a suffix, prefix or capitalization difference will not match. Use a regular-expression or case-insensitive approach when the observed spelling varies, and keep the protected scope as small as practical.
Both access-control approaches return a server-side denial when the condition matches. They are preferable to asking the bot to behave because the request is actually refused rather than merely discouraged.
Return HTTP 403 with mod_rewrite
mod_rewrite can inspect %{HTTP_USER_AGENT} and stop a request with the [F] flag. The documented example combines a User-Agent test with an IP pattern and limits the rule to /secret/files/:
Recommended Free Tools
RewriteEngine On
RewriteCond "%{HTTP_USER_AGENT}" "^NameOfBadRobot"
RewriteCond "%{REMOTE_ADDR}" "=123.45.67.[8-9]"
RewriteRule "^/secret/files/" "-" [F]When all conditions match, the rule returns HTTP 403 Forbidden. The hyphen means that no substitution is performed; the request is simply rejected. The first condition looks for a User-Agent beginning with the named string. The second limits the match to addresses whose final octet is 8 or 9 in the shown range.
Make a rewrite match case-insensitive
If log entries show inconsistent capitalization, add the case-insensitive condition flag to the relevant RewriteCond, for example:
RewriteCond "%{HTTP_USER_AGENT}" "^NameOfBadRobot" [NC]Use the flag only where it helps the observed pattern. A broad, case-insensitive expression can catch more traffic than intended if the name is common or ambiguous.
Scope the rewrite to the route that needs protection
A rule for one private directory is safer than a virtual-host-wide rule. Apply the narrow path first, then expand only when the logs demonstrate that the same bot is abusing additional routes. If the rule is moved between server configuration and per-directory configuration, check the path syntax: rewrite patterns are interpreted differently in a directory context, and a pattern copied without adjustment may never match or may match an unintended path.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Combine signals only when they are stable
The example’s User-Agent-and-IP combination reduces accidental matches, but it also fails when the bot changes either value. Conversely, blocking only the address can affect unrelated users behind the same network and will not follow a distributed bot. Treat combined conditions as a precision tool, not as authentication.
Using the rule in .htaccess
Many shared hosts expose .htaccess but do not let customers edit the main server or virtual-host configuration. Before adding a rule, make a backup and verify which directives the host allows in that file. Apache reads .htaccess rules for requests under the directory containing the file, so a rule placed in a subdirectory has a narrower natural scope than one placed at the document root.
- Save a copy of the current
.htaccessfile. - Place the rule in the directory that owns the path you intend to protect, or use the document-root file with an explicit path condition.
- Use the syntax appropriate to per-directory rewriting; do not assume that a server-context pattern beginning with
/has identical behavior in.htaccess. - Check the Apache configuration before reloading or changing traffic. A syntax or context error can prevent Apache from starting or make the site return an error.
- Request an allowed page and the protected page from a controlled test client, then verify the response and access log.
- Remove or narrow the rule immediately if a legitimate request is denied.
If the host does not permit the required directive, the remedy is not to keep changing the expression. Ask the host to apply an equivalent server-level rule or use an upstream control.
What robots.txt can and cannot do
robots.txt is a crawler-exclusion signal. Cooperative crawlers read it and may avoid the paths you list, which can reduce unwanted indexing and ordinary crawl traffic. It is not an access-control mechanism: a hostile scraper can ignore the file, fetch the disallowed URL anyway and continue making requests.
Free tools Windows power users keep installed
One-click scans. No signup required.
Use robots.txt when your objective is to communicate crawl preferences to compliant agents. Use Apache denial when the request itself must be refused. If the traffic remains harmful after an Apache rule, do not treat a more elaborate robots.txt file as an escalation; move the control to a firewall, managed WAF or comparable upstream layer.
Understand the User-Agent limitation
The User-Agent value is supplied by the client. A program can claim to be a browser, imitate a search crawler or generate a new token for every request. Apache’s documentation explicitly warns that any technique relying on this string can be trivially circumvented because the client can change it.
Best Value
- Used Book in Good Condition
That makes header blocking useful for self-identifying or low-effort bots, not for proving that a request is malicious and not for authenticating a legitimate crawler. Do not build an allowlist of important crawlers solely from their User-Agent strings; the same limitation applies to both “allow” and “deny” decisions.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Escalate sustained attacks upstream
An Apache rule still requires the request to reach the origin and be evaluated. If an attacker sends enough requests to consume connection slots, bandwidth, CPU or log capacity, denying each request at Apache may protect a route while leaving the underlying resource problem intact.
The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →The Apache guidance recommends considering a higher-level block when an attack is sustained. A network firewall, managed WAF or CDN security layer can reject traffic before it reaches the origin and can apply controls that Apache alone cannot conveniently provide. Selection depends on your traffic pattern, deployment and provider’s capabilities; verify the policy, coverage and failure behavior separately rather than assuming every service offers the same protection.
Escalate when these symptoms appear
- The bot changes User-Agent strings after the first deny rule.
- Requests arrive from many addresses or networks, making a small IP list ineffective.
- Apache logs show sustained volume even though the responses are 403.
- Legitimate users experience slow responses, connection failures or exhausted server capacity.
- The attack targets many routes and maintaining per-path rules has become unmanageable.
Test, observe and recover safely
Confirm that the intended request is denied
Use a controlled request that sends the exact User-Agent string and path in your rule. Confirm an HTTP 403 response and verify that the access log records the expected request. Then repeat with a normal User-Agent and an unprotected path to ensure the rule is not broader than intended.
Diagnose a rule that does not match
- Check capitalization and punctuation in the observed User-Agent.
- Confirm that the request path is inside the configured location or directory.
- Check whether the rule is loaded in a context where its directives are permitted.
- For
mod_rewrite, verify that the module is enabled and that the pattern matches the path form used in that context. - Inspect the error log for configuration or authorization errors after a reload.
Diagnose a rule that blocks too much
- Temporarily remove the broad expression and restore the last known-good configuration.
- Replace a short or common token with a more specific observed string.
- Restrict the rule to the smallest affected path.
- Remove an IP condition only if it is causing legitimate users to be denied; otherwise, document why the address condition is trusted.
- Retest normal traffic before re-enabling a wider rule.
Maintain the denylist
A deny rule is operational code, not a set-and-forget security boundary. Review logs after deployment to see whether the bot changed its header, moved to a different path or stopped. Remove obsolete entries so the configuration remains understandable, and record why each pattern exists and which path it protects.
Keep the response behavior consistent with your objective. A 403 tells the client that Apache understood the request but will not authorize it; it does not hide the existence of the route from a determined observer. If concealment, rate control, authentication or abuse investigation is required, those are separate controls and should be addressed at the appropriate layer.
A practical decision checklist
- Is the crawler cooperative? Publish a
robots.txtexclusion, but do not rely on it against hostile traffic. - Does it identify itself consistently? Use a case-insensitive Apache access rule or a narrowly scoped
mod_rewritecondition. - Is one sensitive path affected? Restrict the rule to that location or directory.
- Are source addresses stable? Add an IP condition only when it improves precision and will not block shared legitimate users.
- Can the bot change headers or addresses? Assume it can; treat the User-Agent rule as a coarse filter.
- Is the attack consuming origin capacity? Escalate to a firewall, managed WAF or CDN security layer.
- Did the change survive testing? Verify denied and allowed requests, inspect logs and keep a rollback copy.
Bottom line
For Apache 2.4, use SetEnvIfNoCase with Require or an expression-based <If> when access-control syntax fits your server context. Use mod_rewrite with the [F] flag when you need conditional matching, such as combining a User-Agent and IP range. Keep every match and path as narrow as possible, remember that clients can spoof User-Agent values, and move sustained attacks to an upstream firewall or WAF. robots.txt remains a courtesy signal, never a hostile-bot blocker.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
