Windows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallOutdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchTo ask search engines not to index an entire WordPress site, open Settings > Reading, enable Discourage search engines from indexing this site, and click Save Changes. Current WordPress adds a noindex,nofollow robots directive to the page head when the normal wp_head hook is present. The site remains publicly reachable, however: this checkbox is a search-exclusion request, not a password wall.
Antihost offers managed WordPress hosting with free migration, one-click staging and a 14-day free trial.
Use a page-level noindex directive when only selected posts, pages, archives, or files should stay out of search. If nobody should be able to retrieve the content, use login, password, or server-level access controls as well.
![WordPress SEO Plugins [2017 Edition]: Learn Search Engine Optimization With Smart Internet Marketing Plugins](https://m.media-amazon.com/images/I/51-ZxmPOSwL._SL160_.jpg)
As an Amazon Associate I earn from qualifying purchases.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
What the WordPress setting actually does
The Reading-screen checkbox changes the indexing signal WordPress publishes to crawlers. In a standard theme that calls wp_head(), WordPress outputs:
<meta name='robots' content='noindex,nofollow' />noindex asks a search engine not to include the URL in its index. nofollow asks it not to follow links on that page for discovery. The directive is placed in the HTML head so a crawler can read it while fetching the page.
#1 Best Overall
WordPress documentation cautions that neither this option nor the older crawl-blocking approach blocks access to the site; search engines must honor the request. A visitor who knows the address can still load the page, and a crawler can still request it. Do not use the checkbox as protection for unpublished client work, private records, paid content, or a staging database.
WordPress changed its core behavior in version 5.3. Before that release, enabling the option added Disallow: / to robots.txt. Since 5.3, core uses a robots meta directive instead, because a crawler that is barred by robots.txt may never see a noindex instruction.
Discourage indexing for the entire site
Use this procedure when every public URL on the WordPress installation should be excluded from search while the site is being built, reviewed, or temporarily taken offline from search.
- Sign in to the WordPress dashboard. You need an account with permission to change reading settings.
- Open Settings > Reading. Scroll to the section labelled Search Engine Visibility.
- Enable Discourage search engines from indexing this site. This is the current WordPress label.
- Click Save Changes. WordPress then publishes the robots directive on pages that use the normal head hook.
The setting does not erase URLs that a search engine has already indexed instantly. A crawler has to revisit the URL, observe noindex, and process the removal. Crawl timing varies, so leave the directive in place until the site is ready to be searchable.
When the checkbox appears to do nothing
View the rendered source of a front-end page after saving, then search for noindex. If the tag is absent, check whether the active theme calls wp_head(). A custom theme, template, optimization plugin, or server-side cache can remove or replace the output. Purge page caches and test a newly generated response rather than an old cached copy.
Exclude only one page, post, archive, or resource
For selective exclusion, do not enable the site-wide checkbox unless you truly want every URL discouraged. Google supports noindex in either an HTML meta tag or an HTTP X-Robots-Tag response header.
Recommended Free Tools
HTML meta directive
A basic page-level instruction is:
<meta name='robots' content='noindex'>Place it in the page’s <head>. In WordPress, a reputable SEO plugin can add the directive through its page or post settings, or a developer can add conditional theme logic for particular URLs, post types, taxonomies, or archives. Confirm that the resulting public HTML contains the tag; an editor setting that is overridden by a theme or cache has no effect.
HTTP response header
For a PDF, image, feed, or another non-HTML response, send an HTTP header instead:
X-Robots-Tag: noindexThe header is useful when the resource has no HTML head. It must be present in the response that the crawler receives, so inspect the actual URL and any redirects, CDN layer, or cache involved.
The crawler must be able to fetch the URL
A crawler cannot read a meta tag or response header on a URL it is forbidden to fetch. Keep the URL crawlable long enough for the engine to see noindex. If a page is both blocked by robots.txt and supposed to be removed with noindex, remove the crawl block first. Otherwise the crawler may report that it is blocked while retaining a URL discovered from links or other references.
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Repair Windows errors before they cause bigger problems3Fix the driver behind crashes, sound loss and screen glitchesWhy robots.txt is not a noindex substitute
robots.txt controls crawling, not guaranteed index removal. Google explicitly does not support a noindex rule in robots.txt. A line such as Disallow: /private-page/ can stop a compliant crawler from retrieving the page, but it does not reliably tell the engine to delete an already known URL from search results.
This distinction matters in two common situations:
- A URL is already known. Search engines can continue to show a URL, often without a description, when they cannot crawl it to see a removal directive.
- You need controlled removal. Use a page meta tag or
X-Robots-Tag, keep the URL fetchable, and allow time for recrawling.
You may still use robots.txt for crawl-management purposes, such as reducing requests to an area that does not need crawling. Do not rely on it as the mechanism for keeping a URL out of the index.
Search exclusion versus actual privacy
Choose the control based on what must be prevented:
| Goal | Recommended control | Can a visitor request the URL? | Can a crawler read the directive? | Strength |
|---|---|---|---|---|
| Discourage indexing of every WordPress URL | Settings > Reading > Discourage search engines from indexing this site | Yes | Yes, when the page is crawlable and uses wp_head |
Search exclusion request |
| Keep one HTML URL out of search | <meta name='robots' content='noindex'> |
Yes | Yes, if not blocked | Search exclusion for that URL |
| Keep a PDF, image, feed, or other non-HTML resource out of search | X-Robots-Tag: noindex |
Yes unless separately restricted | Yes, if the response is fetchable | Search exclusion for that resource |
| Prevent the public from viewing content | Login, password protection, application authorization, or server-level access control | No, after authentication is enforced | No, unless credentials are provided | Access restriction |
| Reduce crawling of a path | robots.txt rules |
Usually yes | No directive is guaranteed to be seen | Crawl control, not reliable removal |
A private staging site normally needs authentication first. You can also leave a noindex directive in place as a second line of defense, but it cannot compensate for an openly accessible site.
How to verify the setting and find conflicts
Verification should check the response that a crawler and an ordinary visitor actually receive, not just the WordPress editor screen.
- Inspect page source. Open a representative front-end URL in a browser, choose the command to view the rendered source, and search for
noindex. Confirm that the robots tag is inside the head and that its value matches your intention. - Run the WordPress diagnostic. Go to Tools > Site Health. WordPress exposes a search-engine-visibility test that reports when search engines are being discouraged.
- Use Google Search Console. Enter the exact URL in URL Inspection, or review the Page Indexing report. Look for whether Google detected
noindex, encountered a robots.txt block, or found a different indexing problem. - Check the HTTP response when needed. For a resource controlled by a header, inspect response headers and confirm that
X-Robots-Tag: noindexsurvives redirects, the web server, a CDN, and caching. - Retest after clearing caches. Purge WordPress page caches, reverse-proxy caches, and CDN objects that may contain the pre-change head. Test in a private browser window or with a fresh request.
If Search Console says “blocked by robots.txt”
Remove the conflicting Disallow rule for the URL or directory that must be deindexed. Keep the page available to crawlers so they can read noindex. After Google recrawls, URL Inspection should show the directive rather than only the crawl block.
If the source shows the wrong directive
Look for multiple systems that can write robots instructions:
- The WordPress Reading setting may be enabled while an SEO plugin outputs a different robots value.
- A theme or custom snippet may add a second robots tag or omit
wp_head(). - A server or CDN may add an
X-Robots-Tagheader that conflicts with the HTML tag. - A cache may serve an older version after the setting changed.
Resolve the conflict at its source, purge caches, and verify one fresh response. Do not assume that the dashboard checkbox wins when another layer is sending a contradictory instruction.
A safe staging-to-public workflow
For a new site, combine indexing control with an access plan rather than relying on one switch.
- During development, require authentication. Put the staging site behind a login, password, VPN, or server-level allowlist so unapproved people cannot retrieve it.
- Enable the WordPress discouragement setting. This prevents accidental indexing if the staging host becomes reachable, but it is not the access barrier.
- Test templates and generated resources. Check the home page, posts, pages, category and tag archives, feeds, attachments, PDFs, and any custom endpoints that matter to your project.
- Before launch, remove the access restriction deliberately. Confirm that the public domain, canonical URLs, redirects, and HTTPS configuration are correct.
- Turn off site-wide discouragement only when the site should be discoverable. Then inspect source and Site Health again. If only selected URLs should remain excluded, replace the global setting with targeted meta or header directives.
- Monitor Search Console after launch. URL Inspection can confirm that important pages are eligible for indexing and that intentionally excluded pages still carry
noindex.
Keep separate staging and production environments when possible. Accidentally copying a production database or configuration can expose private content; accidentally leaving the production checkbox enabled can suppress an otherwise public launch.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Common mistakes and their fixes
Assuming the checkbox makes the site invisible
Symptom: Someone can still open the site or a direct URL.
Fix: Add authentication or password protection. Keep noindex for search exclusion, but treat access control as a separate requirement.
Adding noindex to robots.txt
Symptom: The file contains a line intended to remove a URL, but Google reports that the URL is blocked or still known.
Fix: Remove the robots.txt block for that URL and publish an HTML meta directive or HTTP X-Robots-Tag.
Blocking a URL before Google can see noindex
Symptom: Search Console reports both a robots.txt block and a desired noindex outcome.
Fix: Make the URL crawlable, retain noindex, and request or wait for a fresh crawl.
The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Changing the setting but seeing old source
Symptom: The dashboard shows the new choice, but page source still contains the previous robots value.
Fix: Clear every caching layer, check the response from the public hostname, and inspect theme, plugin, and server output for overrides.
Forgetting non-HTML files
Symptom: A PDF or feed remains searchable even though HTML pages carry noindex.
Fix: Add X-Robots-Tag: noindex to that resource’s HTTP response and verify the header directly.
Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Repair Windows errors before they cause bigger problemsFix Now →Leaving global discouragement enabled after launch
Symptom: Newly published pages fail to appear in search.
Fix: Revisit Settings > Reading, disable the global option when appropriate, save, clear caches, and verify that production pages no longer emit the unwanted directive. Preserve targeted exclusions separately.
Practical decision guide
- Choose the WordPress Reading checkbox when the whole installation should be discouraged from indexing.
- Choose a page-level meta tag when specific HTML URLs should be excluded but the rest of the site should remain eligible.
- Choose
X-Robots-Tagwhen the target is a non-HTML resource or when headers are easier to control than template markup. - Choose login, password, or server authorization when the requirement is “nobody without permission can fetch this”.
- Use robots.txt for crawl management, never as your sole deindexing mechanism.
The key test is simple: can the crawler fetch the URL and see an explicit noindex instruction? If yes, the search engine has the information needed to exclude it. If the content must not be fetched by the public at all, add authentication instead of relying on indexing signals.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
