To ask Yandex’s main indexing crawler to stop crawling your site while leaving other crawler rules intact, add a separate YandexBot group to the root robots.txt file:
User-agent: YandexBot
Disallow: /Keep the site’s existing User-agent: * group and any other named groups in place. Use YandexBot, not the broader Yandex token: Yandex documents the former for its main indexing bot and the latter for all Yandex bots. This method is a crawler request, not a security barrier. Clients that must be unable to fetch content need application-layer access controls instead.
As an Amazon Associate I earn from qualifying purchases.
What this rule blocks—and what it leaves alone
A robots.txt rule is matched by a crawler’s declared user-agent group. The following group asks YandexBot, Yandex’s main indexing robot, not to crawl any path:
User-agent: YandexBot
Disallow: /The slash means every URL path covered by that site’s robots file. A crawler that identifies itself under another user-agent is governed by its own matching group, usually the site’s wildcard group. This is why a YandexBot-specific group is preferable when the objective is to change YandexBot’s behavior without rewriting the policy for Google, Bing, or other crawlers.
#1 Best Overall
- [Ultra-Lightweight Ear Buds Designed for Small Ears] Each earbud weighs only 3.7g and features a compact, ergonomic in-ear design made especially for small ears. Secure, low-profile, and comfortable for workouts, all-day wear or overnight sleeping.The ultra-slim profile sits flush in your ear with no hard edges poking or rubbing, so you can even sleep on your side without discomfort.
- [Immersive Stereo Sound with TOZO OrigX Technology] TOZO OrigX tuning delivers clear vocals, balanced mids, and natural stereo sound for music, podcasts, and videos.
- [Long Battery Life for Daily Use] Get up to 7 hours of playtime on a single charge, with up to 32 hours total using the charging case—ideal for workdays, commuting, and extended listening sessions.
- [Bluetooth 5.3 & Stable Connection] Bluetooth 5.3 provides fast pairing, stable wireless performance, and reduced dropouts as you move around home or office.
- [Deep Bass with Clear Vocals] High-performance drivers produce punchy bass while keeping vocals clean and detailed for everyday listening.
Do not replace the entire file with this two-line example unless you have deliberately reviewed and rebuilt the policy. Existing directives may control important crawl areas, named bots, or assets that other services need.
Choose the correct Yandex token
Yandex distinguishes between the main indexing bot and the broader family of Yandex robots.
| Token | Documented scope | Use when |
|---|---|---|
YandexBot |
Yandex’s main indexing bot | You want to target the main indexing crawler specifically. |
Yandex |
All Yandex bots, according to Yandex’s user-agent-group behavior | You intentionally want a broader Yandex-wide rule. |
* |
General crawler group | You are setting defaults for crawlers that do not receive a more specific group. |
If the goal is only to block the principal indexing bot, do not use User-agent: Yandex. Yandex states that when a Yandex group is present, the wildcard group is ignored for Yandex. That broader token can therefore change how several Yandex services interpret the file, rather than affecting only YandexBot.
Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Before editing: identify the outcome you actually need
Stop a compliant crawler from requesting pages
Use the YandexBot group when you want to communicate a crawl preference. Compliant crawlers are expected to honor the directive, and other crawlers continue to follow their own applicable groups.
Prevent anyone from fetching content
Do not rely on robots.txt for confidentiality, licensing restrictions, private files, or attack mitigation. RFC 9309, the Internet Engineering Task Force’s September 2022 Robots Exclusion Protocol specification, states: “These rules are not a form of access authorization.” A client can ignore the file or request a URL directly. Use authentication, authorization, server rules, an edge control, or another application-layer mechanism appropriate to the site.
Remove a URL from search results
Crawl blocking and search-result removal are different operations. Yandex notes that pages restricted in robots.txt may still participate in Yandex Search. If removal from results is the objective, use Yandex’s separate noindex or removal guidance rather than assuming Disallow guarantees de-indexing.
Step-by-step: add a YandexBot-only group
- Open the site’s existing robots file. The file must be named
robots.txtand served from the site root, such ashttps://example.com/robots.txt. Replace the example host with your own domain; do not place the file in a subdirectory. - Read the complete file before changing it. Locate every
User-agentgroup, including wildcard, Yandex-specific, and other named groups. Note existingAllowandDisallowdirectives and any comments explaining intentional exceptions. - Add a dedicated group. Place this block in the file, with the exact user-agent spelling:
User-agent: YandexBot
Disallow: /- Keep general rules in their own group. A typical combined file might look like this:
User-agent: YandexBot
Disallow: /
User-agent: *
# Keep the site's existing general crawler directives here.The comment is illustrative only. Retain the site’s real wildcard directives instead of deleting them or replacing them with a placeholder.
- Check for an existing Yandex group. If the file already contains
User-agent: Yandex, understand that it is broader thanYandexBot. Decide whether that existing policy is intentional before adding another rule. Do not leave contradictory blocks simply because they are in different parts of the file; review the whole file and consolidate only when you understand the desired behavior. - Save and publish the file at the root. The production URL must return the current file to indexing bots with an HTTP 200 response. A file that is missing, placed elsewhere, or unavailable does not implement the intended restriction.
- Test representative URLs. Check the site root, a normal article, a path that the wildcard group already handles specially, and any URL where an exception matters. Validation should confirm both the YandexBot result and that general crawler rules were not accidentally removed.
How group selection affects other crawlers
Specific groups and general groups serve different purposes. The YandexBot block is a named instruction for that bot. The wildcard block remains the place for site-wide defaults. Keeping them separate makes the policy auditable and prevents a Yandex-specific change from silently becoming a global change.
Rank #2
- 【IP68 Waterproof Multifunctinal Smart Watch】Torjalph fitness activity tracker smart watch is IP68 waterproof technology ,you can wear it when swimming,handwashing,it's rain-proof which means that you perform multiple exercise modes outside without worrying about sweat or rain corroding your fitness smart watch,and this watch equipped with a 230 mAh large capacity battery magnetic charging ,and this sports activity tracker watch can be used 5-7 days If fully charged.
- 【Larger 1.83'' Full Screen Watch & Multi-dial Selection】1.83 inch full HD color touch screen smart watch for men / women provides excellent 240*284 resolution image quality ,This Bluetooth smart watch offers a better touch experience and remarkable immersion and ,more than 200 watch faces for you to choose from App"Da Fit ", you can change the dial style according to your mood to wear keeping the world on your wrist
- 【Accurate Activity Tracker Smart Watch 】sports watch activity tracker uses sweat and dust resistant breathable silicone strap, our Fitness Tracker Smart Watch for Android iOS Phones supports 12 sports modes,walking,running,bicycle,swimming,badminton,basketball,football,rope skipping. This bluetooth calls smart watch for men women is lightweight and stylish, bringing more convenient and relaxing sports experience while also providing more protection.
- 【Your 24 Hour Personal Daily Health Monitor】Fitness watch for women men full touch screen IP67 waterproof bluetooth smartwatch for Android iOS Phones with heart rate blood pressure monitor multifunction Andriod smart watch with steps counter sleep monitor sports fitness watch, monitoring blood oxygen saturation allows you to understand your physical condition in real time to track fitness progress ,to know your health status in time.
- 【More Functions Bluetooth Call Smart Watch】This fitness activity smart watch full screen for iPhone and Android phones includes a bunch of new features including:The bluetooth smart watch can hang up the phone, receive and display WhatsAPP /Text /Facebook /Twitter /Messenger reminder.Support sedentary reminder, alarm clock, stopwatch, timer,weather information,music control, remote camera control compatible with Android 5.0 and above or iOS 9.0 and above smartphones.
For example, suppose the existing file contains:
User-agent: *
Disallow: /admin/
Disallow: /tmp/
User-agent: ExampleCrawler
Allow: /public-feed/
Disallow: /Adding the YandexBot block above those groups should not remove the administrator and temporary-directory restrictions for other crawlers, nor the dedicated policy for ExampleCrawler. The exact behavior of every existing group still depends on that crawler’s implementation, so preserve deliberate rules and verify after publication.
Yandex supports both Allow and Disallow. If your existing policy contains exceptions, do not assume that adding a root-level Disallow: / to YandexBot preserves an exception for that bot; review the complete YandexBot group and test the URLs that matter.
File placement, response, and availability requirements
Yandex’s robots guidance identifies three operational requirements that are easy to miss:
The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →- The file is named exactly
robots.txt. - It is available at the site root, not in a content directory.
- Indexing bots can retrieve it successfully with HTTP 200.
Check the public production endpoint, not only a local copy or staging server. Confirm that redirects, access rules, maintenance responses, or a CDN configuration are not preventing indexing bots from receiving the file. Yandex says that when the requirements are not met, the site may be considered open for indexing, so a syntactically correct rule is not enough if the file cannot be retrieved as intended.
Validate the result in Yandex Webmaster
After deployment, use Yandex Webmaster’s Robots.txt analysis tool. Yandex documents two useful checks:
- Syntax analysis: confirms whether the file can be parsed and exposes formatting or directive problems.
- URL allowance checking: tests how the published rules apply to a selected URL and user-agent.
Run the URL check as YandexBot for at least one URL you intend to block. Also test a representative URL under the wildcard policy so that an edit did not remove or alter a general rule. If the tool displays a result different from your expectation, compare the tested file with the exact production response and inspect all groups for a broader Yandex token or duplicate directives.
Do not assume every Yandex robot is covered
A YandexBot-only rule establishes a restriction for the main indexing bot. It does not prove that every Yandex service robot will be blocked. Yandex documents specialized robots with separate names and notes that some may ignore generic User-agent: * restrictions. Where a particular Yandex robot must be restricted, identify that robot and provide a directive for its exact user-agent according to Yandex’s documentation.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
This distinction matters for sites that see image, video, media, diagnostic, or other specialized requests in their logs. Do not infer the crawler’s scope from the word “Yandex” in a user-agent string. First determine which robot is making the request, then decide whether the policy should cover that specific service.
Rank #3
- [Low Noise High Fidelity] New bee guitar cord with 22AWG oxygen-free-copper center conductors provides enhanced signal transfer, the PVC insulation material and the high-density stranded copper double shield provide noise rejection for quiet operation
- [Ideal for Choice] New bee 1/4 straight and right angle metal connectors, ensures a secure connection in any instrument, pedal, or amp jack. L shape great fro on stage guitar stands
- [Wide Compatible] Suitable for mixer, power amplifier, microphone, effector, decoder, equalizer, electoronic organ, electric guitar, electric bass and other electronic equipments. Widely used in the installation of smart home, karaoke speaker, professional stage audio, etc
- [Easy Disassemble] Making incidental repairs quick and easy! Very nice, flexible cable
- [Dont Hesite to Order] 24 hours customer services and professional technology team are standing by
How to check that a robot belongs to Yandex
A user-agent header in a server log is only a claim. Any client can send a string that looks like YandexBot, so it is not proof of origin.
Yandex provides an IP-address verification method for checking whether a request belongs to one of its robots. Use that method when investigating requests or considering server-side enforcement. Yandex also warns that its bot IP addresses change frequently; a static allowlist copied from an old log can become inaccurate.
Verification is especially important if you move beyond advisory robots rules and create an application, server, or edge rule. Confirm the identity using Yandex’s current verification procedure before permitting or denying traffic based on an address.
Robots.txt versus enforced blocking
| Approach | Primary purpose | Who is affected | Is it enforced? |
|---|---|---|---|
YandexBot-specific robots.txt group |
Request that the main Yandex indexing crawler not crawl the site | Compliant clients identifying as YandexBot | No; it is an advisory protocol. |
| Application-layer authentication or authorization | Require permission before content is returned | Clients that fail the site’s access policy | Yes, when correctly implemented. |
| Server or edge security rule | Reject or challenge requests before the application serves content | Traffic matching the configured conditions | Yes, subject to configuration and identity accuracy. |
| Noindex or a search-removal process | Control search-result visibility | Search indexing and result display | Separate from crawl guidance; follow the search service’s documented process. |
Use the first approach when the requirement is crawl guidance. Use an enforced control when the requirement is that an untrusted client must not receive the response. Use a search-removal or noindex approach when the requirement concerns results rather than requests. Combining these goals without distinguishing them is a common cause of ineffective configurations.
Safe editing and change review
Make the smallest change that satisfies the goal: add one named group and leave unrelated directives untouched. Copy the production file before editing so you can compare the published version with the prior policy. Check for repeated User-agent labels, groups separated by unrelated directives, and comments that explain exceptions. A duplicate or broad group can make a seemingly narrow edit behave differently from the intended design.
Review the change with two questions: which requests should YandexBot stop making, and which requests must remain governed by the existing wildcard or named groups? If the answer to the second question is unclear, pause and map the current policy before publishing. This is particularly important on sites with separate rules for feeds, static assets, APIs, or administrative paths.
After publication, test both a blocked URL and a URL that should remain available to other crawlers. A successful syntax check does not prove that the correct group was selected, that the production server returned the edited file, or that a specialized Yandex robot is covered.
What a root-level block means for URLs
Disallow: / is intentionally broad: it asks YandexBot not to crawl every path represented by that robots file, including pages, files, feeds, and other URL paths. It does not selectively remove only newly published content, and it does not create a password-protected area. If only a subset should be excluded, use a narrower path rule and test its boundary cases in Yandex Webmaster.
Rank #4
- Digital Family Calendar: BSIMB digital calendar wall touch screen puts an end to your chaotic life. Designed for busy families, the digital calendar keeps you organised and in step with your family. Assign different colours and tasks to each family member to make family scheduling easier and more intuitive.
- Easy Setup and Auto-Sync: Plug in, connect to WLAN and auto-syn your electric calendar in minutes. Digital wall calendar compatible with Google calendar, iCloud calendar, Outlook calendar, Cozi calendar and Yahoo calendar. Add events directly on the device or through the ecalendar app. Electronic calendar stays current whether you're at work or traveling—no manual updates needed.
- All-In-One Home Assistant: Smart calendar builds habits and encourage independence with personalised chores. Display daily meal plan to reduce the daily ‘What's for dinner?’ stress. To-do lists keep track of everything you want. Turning chores into valuable achievements with stars and rewards, making life easier and more enjoyable.
- Manage Anytime Anywhere: BSIMB electronic calendar for family stay connected with the powerful ecalendar app. Access and manage your digital wall calendar anytime, anywhere with free features like stars & rewards, photo screensaver, chore charts, meals plans, to-do lists etc. Electronic planner paid features like Magic Import tailor advanced tools for users with special needs.
- Stay in Touch with Loved Ones: Share photos or videos from your phone to electronic wall calendar with app, instantly update your life moment, and keep in touch with your family, especially those who are far away; you can also surprise your smart calendar digital wall by sending him a wish card; it's also great for keeping tech-illiterate elders in touch with the youngsters.
Likewise, a robots policy is read per host. A file at one hostname does not automatically define rules for another hostname, protocol, or separately served host. Verify the exact production URL that receives requests from the crawler, especially when a site uses multiple domains, subdomains, or redirects.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Troubleshooting common failures
YandexBot still appears in logs
First, check timing: a robots change does not retroactively cancel requests already queued or cached by a crawler. Confirm that the live root file returns the new group with HTTP 200, then run the Yandex Webmaster URL check. Finally, verify that the requests really belong to Yandex using Yandex’s IP-check method rather than relying only on the user-agent string.
Other crawlers stopped unexpectedly
Compare the current file with the previous version. Look for an accidentally deleted wildcard group, a newly added broad User-agent: Yandex group, or edits that moved existing directives into the wrong group. Restore intentional named-bot and wildcard rules, then retest representative URLs for each important crawler policy.
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Fix the driver behind crashes, sound loss and screen glitches3Repair Windows errors before they cause bigger problemsThe rule validates but pages remain in search
A valid Disallow tells a compliant crawler not to fetch a path; it does not guarantee that the URL disappears from Yandex Search. Follow Yandex’s separate noindex or removal guidance for search-result control.
A suspected Yandex service bot continues to crawl
Identify the exact robot rather than assuming it is YandexBot. Yandex says some specialized robots may not follow generic wildcard restrictions. If that robot should be restricted, add a rule for its documented user-agent and validate the resulting behavior.
The file is not being honored at all
Check the URL, filename, root placement, HTTP status, and content returned to external clients. A local file, a staging response, a blocked request, or a non-200 response does not establish that indexing bots can read the policy.
A server rule blocks the wrong clients
Do not build an enforcement rule from a user-agent string alone. Confirm the request with Yandex’s current IP-address verification method, remember that bot addresses change frequently, and review the rule whenever the site’s access-control configuration changes. A false positive can deny legitimate users or other crawlers, while a stale address list can fail to stop the intended traffic.
Recommended Free Tools
Practical configuration checklist
- Use
User-agent: YandexBotwhen the target is only the main Yandex indexing bot. - Use
Disallow: /to request that bot avoid every path. - Preserve the existing
User-agent: *group and intentional named-bot groups. - Review the entire file for an existing broad
User-agent: Yandexgroup. - Publish the file as
/robots.txtat the site root. - Confirm that the production response is available with HTTP 200.
- Run Yandex Webmaster’s syntax analysis and URL allowance check.
- Verify suspicious requests with Yandex’s current IP-address method; do not trust a user-agent string alone.
- Use application-layer access controls when content must be protected from non-compliant clients.
- Use noindex or removal guidance when the goal is search-result removal rather than crawl guidance.
Frequently Asked Questions
Will this block every Yandex crawler?
No. User-agent: YandexBot targets Yandex’s main indexing bot. Yandex documents specialized robots with separate names, and some may not follow generic wildcard restrictions. Name a specific robot if it also needs a restriction.
Why not use User-agent: Yandex?
Yandex uses that broader token for all Yandex bots and says a Yandex group causes the wildcard group to be ignored for Yandex. Use it only when a site-wide Yandex rule is intentional.
Does Disallow: / delete pages from Yandex Search?
No. It asks a compliant crawler not to fetch the paths. Yandex notes that restricted pages may still participate in search. Use separate noindex or removal guidance for result visibility.
Can robots.txt protect private files?
No. RFC 9309 says Robots Exclusion Protocol rules are not access authorization. Use authentication, authorization, server controls, or another application-layer mechanism when clients must be prevented from receiving content.
Free tools Windows power users keep installed
One-click scans. No signup required.
Where must the file be located?
Serve a file named robots.txt from the site root, such as https://example.com/robots.txt. Yandex also requires that indexing bots can retrieve it successfully with HTTP 200.
How do I test whether the rule works?
Use Yandex Webmaster’s Robots.txt analysis tool for syntax analysis and its URL allowance check. Test a blocked URL as YandexBot and test representative URLs governed by the wildcard group.
Why does YandexBot still appear after I publish the rule?
Requests may have been queued before the change. Confirm the live file, its HTTP 200 response, and the Webmaster result. Then verify that the request is genuinely from Yandex using Yandex’s IP-address checking method.
Can I block Yandex at the server instead?
You can use an application, server, or edge control when enforced blocking is required, but verify identity carefully. A user-agent string is only a claim, and Yandex says its bot IP addresses change frequently.
Windows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallCrashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteThe Bottom Line
Add User-agent: YandexBot with Disallow: / to the root robots.txt file, while preserving the site’s wildcard and other named groups. Validate the production file in Yandex Webmaster, and remember that robots.txt guides compliant crawlers; it does not provide access control or guarantee removal from search results.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
