To edit WordPress’ robots.txt, use the robots editor in an SEO plugin such as Rank Math, All in One SEO or Yoast, or download the existing file, edit it as UTF-8 plain text and upload it to the site root. Before changing anything, save a backup.
Antihost offers managed WordPress hosting with free migration, one-click staging and a 14-day free trial.
A useful WordPress configuration blocks the administrative area, explicitly leaves /wp-admin/admin-ajax.php reachable, and names your XML sitemap. Use a noindex robots meta tag or an X-Robots-Tag header—not robots.txt—when the goal is to keep a URL out of search results. After editing, test the proposed rules in Google Search Console and then copy the verified version into the live file.
As an Amazon Associate I earn from qualifying purchases.
What robots.txt does in WordPress
robots.txt is a publicly readable, plain-text file at the root of a host. For example, a site at https://www.example.com normally exposes it at https://www.example.com/robots.txt. Crawlers request that file before visiting URLs and use its rules to decide which paths they may crawl.
The file is a crawl directive, not an access-control system. It does not password-protect a folder, remove a URL from a search index, or erase a result that is already known to a search engine. A URL can remain indexed even when crawling is disallowed, because a crawler may know the address from links or other sources without being allowed to fetch the page.
#1 Best Overall
Fields Google reads
User-agentidentifies the crawler to which the following rules apply.Disallownames a path that the selected crawler should not fetch.Allowmakes an exception to a broader disallow rule.Sitemapsupplies the absolute address of an XML sitemap.
Google supports those fields. Its crawler does not support crawl-delay, so adding that directive will not slow Googlebot. Field names are not case-sensitive, but path matching is case-sensitive: /Private/ and /private/ are different paths on a case-sensitive server. When rules conflict, Google follows the least restrictive applicable rule. Keep each rule deliberate and test the result rather than assuming that the order of lines alone resolves a conflict.
Decide what you are trying to achieve
Write the desired outcome before changing a rule. If the outcome is to reduce needless crawling of an area, a Disallow may be appropriate. If the outcome is to prevent a page from appearing in search, use a noindex mechanism that crawlers can actually read.
Recommended Free Tools
Reduce crawling
Administrative screens, internal endpoints and other low-value paths can consume crawler requests without helping users discover content. Blocking them can be sensible when you understand the path and have checked that public pages do not depend on it.
Keep a URL out of search
Add a noindex robots meta tag to the page, or send an X-Robots-Tag: noindex HTTP header for a non-HTML resource. Do not disallow that same URL in robots.txt while relying on noindex: a crawler that cannot fetch the URL cannot see the tag or header, so the URL may remain indexed.
Protect private information
Do not put confidential material in a directory and assume a disallow rule protects it. Anyone can request the robots file, and disallowed URLs can still be discovered. Use authentication, authorization and server-side controls for privacy; reserve robots.txt for crawl guidance.
Prepare safely before editing
- Open the current file. Visit the root robots URL and save a local copy. This tells you whether the site serves a physical file or a file generated by a plugin or hosting layer.
- Record the current sitemap address. Copy the exact absolute URL used by your site’s XML sitemap. A robots file can list more than one sitemap, but every address should be valid and publicly fetchable.
- Save a backup. For a plugin editor, copy the existing text into a dated note. For FTP or a host file manager, download the file before replacing it.
- Choose a plain-text editor. Notepad, TextEdit in plain-text mode or another code editor is suitable. A word processor can insert formatting characters that make the file invalid or unpredictable.
- Check which system owns the output. If an SEO plugin generates robots.txt dynamically, editing a separate physical file may have no visible effect, or a later plugin change may replace it. Use the editor that controls the version delivered at the root URL.
Choose an editing method
For most WordPress sites, a plugin editor is the least technical route. Manual editing is useful when you need version-controlled text, do not run an SEO plugin, or need to inspect exactly what is uploaded. Both methods require a backup and a live-file check.
Free tools Windows power users keep installed
One-click scans. No signup required.
| Method | Best when | Advantages | Risks and maintenance |
|---|---|---|---|
| SEO-plugin editor | The site already uses an SEO plugin and administrators prefer the WordPress dashboard. | No FTP access; the plugin writes or serves the file; changes are easy to review in the dashboard. | Menu labels vary by plugin version. Another plugin, a host cache or a migration can alter the generated output. |
| Manual FTP or host file manager | You control the server files or need a plugin-independent text file. | Direct control, a simple backup-and-restore path, and no dependency on a dashboard feature. | You can edit the wrong directory, upload malformed text or have a generator overwrite the file later. |
Edit robots.txt with an SEO plugin
Dashboard names change between releases. The paths below match the interfaces available in the October 14, 2026 snapshot; if your labels differ, look for a feature named Robots.txt, Robots editor or Custom robots.txt.
Rank Math
- Sign in to WordPress.
- Open Rank Math → General Settings → Edit robots.txt.
- Copy the existing contents into a backup before changing them.
- Edit only the rules you understand, then save the settings.
- Open the public
/robots.txtURL in a private browser window to verify that the delivered text changed.
All in One SEO
- Open All in One SEO → Tools → Robots.txt.
- Turn on Enable Custom Robots.txt.
- Review the generated or existing directives and make your changes.
- Save, then fetch the root robots URL and confirm that the custom file is being served.
Yoast
- Open the Yoast tools area that contains the robots.txt box.
- Review the pre-filled default settings.
- Add or adjust directives in the box, retaining the sitemap line and any required exception.
- Save and verify the public file.
Dedicated robots plugins
Plugins whose names include Virtual Robots.txt, WordPress Robots.txt Optimization or Robots.txt Editor can provide a focused editor. Treat them like any other plugin: check who owns the generated file, keep a copy of the previous text, review update notes and test after activation or deactivation. Installing a second robots editor while an SEO plugin already controls the file can create uncertainty about which output is live.
Edit the file manually with FTP or a host file manager
Use this method when you have server access and want the file itself to be the source of truth.
- Locate the document root. In FTP or the hosting file manager, open the directory that contains the WordPress installation and the public index file. The robots file belongs in that host root, not inside
wp-admin,wp-contentor a theme directory. - Download the existing file. Save it with a date in the filename so you can restore the previous version.
- Open it as plain text. Use UTF-8 encoding. Do not use a word processor and do not save it as rich text.
- Make one controlled change at a time. Keep comments short, use one directive per line and preserve a known-good copy outside the server.
- Upload to the root. Replace the existing root file, rather than creating a second copy in a subdirectory.
- Fetch the live URL. Confirm the response contains the text you uploaded and that no plugin or host process has generated a different version.
A practical WordPress starting template
The following is a starting point, not a universal prescription. Replace the sitemap variable with your own absolute sitemap URL, remove optional blocks that do not fit your site and add rules only after checking their effect.
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-content/plugins/
Disallow: /trackback/
Disallow: /readme.html
Sitemap: https://{{your-domain}}/{{your-sitemap-path}}Why these lines are used
User-agent: *applies the group to all crawlers that honor the standard.Disallow: /wp-admin/avoids crawling the administrative directory.Allow: /wp-admin/admin-ajax.phppreserves access to the AJAX endpoint even though its parent directory is disallowed. Removing this exception can interfere with front-end features that rely on that endpoint.Disallow: /wp-content/plugins/is an optional low-value-path rule. Confirm that your public pages do not require a crawler to fetch a plugin resource before adding it.Disallow: /trackback/can reduce crawling of legacy trackback endpoints when the site does not use them.Disallow: /readme.htmlkeeps the standard WordPress readme from routine crawling. It is not a security control; remove the file or restrict access at the server if disclosure is a concern.Sitemap:tells crawlers where to find the XML sitemap. Use the exact absolute URL produced by your site, including the correct protocol and path.
Do not paste this template unchanged into every installation. A multilingual site, a subdirectory installation, a membership system or custom application may need different paths. Start with the smallest rule set that achieves your goal.
Rules that prevent common optimization mistakes
Do not use robots.txt as a noindex tool
When a URL must disappear from search, leave it crawlable long enough for the crawler to see a noindex meta tag or response header. Removing the URL from the crawl path first prevents that signal from being read.
Do not add crawl-delay for Google
Google does not support the crawl-delay field. It will not provide the throttling behavior some other crawlers may implement, so it does not belong in a Google-focused optimization plan.
Keep path spelling exact
Because path matching is case-sensitive, copy paths exactly as they appear on the server. A rule for /wp-admin/ does not automatically cover a differently cased path.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Resolve conflicts deliberately
Review every matching Allow and Disallow line. Google chooses the least restrictive applicable rule when directives conflict, so an exception can make a URL crawlable even when a broader block appears nearby.
Rank #4
Do not block assets casually
Rules aimed at stylesheets, scripts or images can affect how a crawler retrieves a page. The safe approach is to leave assets alone unless you have a specific crawl reason, have checked the resulting page and can explain why the block is needed.
Test before and after publishing
- Open Google Search Console’s robots.txt report. Use the property for the exact protocol and host you are editing.
- Load the current file in the tester. Confirm that the tool is testing the same directives visitors receive at the root URL.
- Enter representative paths. Test an administrative URL,
/wp-admin/admin-ajax.php, a normal public post, the sitemap URL and every path you intend to block. - Inspect the result for each path. A blocked result should be intentional; a public page or required endpoint should remain allowed.
- Edit and retest in the tool. The tester lets you try proposed changes without altering the live file. This simulation is not a publication step.
- Copy the verified text into the real editor. Save it in the SEO plugin or upload it to the root through FTP/file manager.
- Fetch the live file again. Re-run the same checks against the URL now served publicly. If the text differs, identify the plugin, cache or host process that is generating it.
Testing is especially important after a plugin update, a domain migration, a change from HTTP to HTTPS or a move between hosts, because the file source or root directory can change.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Troubleshoot a robots.txt problem
Your edit does not appear
First open the public /robots.txt URL rather than relying on the editor’s preview. If it is unchanged, you may have edited a physical file while an SEO plugin is generating the response, or uploaded to a non-public directory. Check the active plugin settings, the host document root and any caching layer before making another edit.
PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteA page you wanted blocked is still crawlable
Check spelling and capitalization, confirm that the path begins where you think it does, and inspect every matching Allow rule. Because Google resolves conflicts using the least restrictive applicable rule, an exception may be overriding your block.
A public feature stops working
Look for a rule that blocks a required endpoint. The WordPress starter pattern keeps /wp-admin/admin-ajax.php allowed for this reason. Remove or narrow a broader rule, save, and test the affected page again.
A URL remains in search after being disallowed
This is expected behavior when the URL is known but cannot be fetched. Remove the disallow temporarily or permanently as appropriate, implement a noindex meta tag or X-Robots-Tag, and allow the crawler to retrieve the signal.
The sitemap is not discovered
Check that the Sitemap line contains an absolute, correctly spelled URL and that the sitemap itself is publicly reachable. A sitemap directive does not override a disallow rule applied to the sitemap path, so keep that path accessible.
The file contains strange characters or parser errors
Reopen the backup in a plain-text editor, save as UTF-8 and remove rich-text formatting, smart quotes and hidden markup. Upload the cleaned file and verify the live response.
Maintain the file as your site changes
- Keep a dated backup and a short change note for every edit.
- Recheck the live file after SEO-plugin updates, theme changes, host migrations and domain moves.
- Review blocked paths whenever you add a membership area, API endpoint, JavaScript-driven feature or new sitemap.
- Retest both blocked and allowed examples in Search Console after each change.
- Use page-level noindex controls for indexing decisions instead of expanding robots blocks.
- Remove obsolete directives rather than allowing the file to grow into an unreviewed list of historical paths.
A small, understandable file is easier to audit than a long collection of copied rules. The best optimization is the one that blocks only paths you have identified as unnecessary while keeping public content, required endpoints and the sitemap available to crawlers.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
