Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Fix the driver behind crashes, sound loss and screen glitches3Clear out junk files and repair common Windows errorsIf a link preview has no image, first check the exact URL in the page’s og:image metadata, then inspect robots.txt on the host that serves that image. A rule on the page’s host may not apply to an image hosted on a CDN or another subdomain. Test with the crawler you are investigating: a Google or Bing test does not establish whether a social platform’s fetcher can retrieve the image.
1. Find the exact Open Graph image URL
Inspect the page HTML and identify the effective og:image value. The Open Graph protocol reference describes this metadata. If the page declares more than one og:image, determine which value the preview consumer receives before changing crawler rules; precedence can depend on the platform.
As an Amazon Associate I earn from qualifying purchases.
Follow redirects and record the final URL, including its scheme, host, any nonstandard port, path, and query string. The image’s final serving authority—not simply the page URL—is the target for the robots check.
Recommended Free Tools
2. Check robots.txt for the image’s serving host
Open /robots.txt on the same scheme and host as the final image URL. For example, if the image is served from https://cdn.example.com/assets/share.png, inspect https://cdn.example.com/robots.txt, not only the page host’s robots file. Protocol, hostname, and port are relevant to robots scope; see RFC 9309 and Google’s robots.txt documentation.
If the image redirects across hosts, check the relevant robots file for the final destination too. A CDN, image subdomain, and origin can have separate configurations, so check each authority the request traverses where applicable.
3. Match the rule to the crawler and image path
Find the user-agent group for the crawler you intend to allow, then evaluate the exact image path against its Allow and Disallow rules. Google documents that it selects the most specific matching user-agent group. An exception for one named crawler should not be assumed to cover a different crawler.
Do not stop at finding a broad Disallow: /. Confirm which group applies and whether a more specific rule permits or blocks the image path. Robots rules are crawler requests, not access controls: they do not make an otherwise public image private.
4. Use a crawler-specific tester
Use a diagnostic tool for the crawler whose access you are checking, and submit the image URL rather than only the HTML page URL.
- Google: Google Search Console’s robots.txt report can help test whether Google is blocked and identify the robots file affecting a URL. See Google’s report documentation.
- Bing: Bing Webmaster Tools’ robots.txt tester accepts a URL and a selected crawler, including Bingbot. See Bing’s tester documentation.
These checks apply to the search crawler selected in the tool. A successful Google or Bing test does not prove that a social platform’s preview fetcher can access the image. The exact current user agent and preview-fetch behavior for a particular social platform should be checked in that platform’s current official documentation or tool; do not infer it from a search-crawler result.
5. Change a blocking rule and retest
If the applicable group disallows the image path, update the robots configuration on the image-serving host (or ask the provider that manages it) to permit the intended crawler to fetch that path. Prefer a narrow rule that opens only what is needed rather than permitting access to unrelated directories. Then rerun the same crawler-specific test against the final image URL.
Rank #4
Conforming crawlers that successfully download a robots file are expected to follow its parseable rules under RFC 9309. Google also explains how robots.txt controls which resources Google crawlers may access in its robots.txt guide. Confirm the change using the relevant tester and, where available, your own access logs.
The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →6. If robots.txt allows the image, check the request itself
A robots pass answers only whether that crawler is blocked by the checked robots rules. Separately verify that an unauthenticated request to the final image URL succeeds, redirects resolve, and the response is actually an image—not a login screen or an HTML error page. The sources cited here do not establish current platform-specific requirements for status codes, image formats, size limits, caching, or firewall behavior, so verify those details with the platform rather than assuming a robots pass guarantees a preview.
Best Value
Keep crawling separate from indexing
Robots.txt controls crawling; it is not a reliable way to keep a resource out of search results. Google notes that a crawler blocked from a resource cannot see a noindex directive on that resource. If your goal is to prevent an otherwise accessible resource from being indexed, the crawler must be able to fetch and see the indexing directive. See Google’s explanation of robots.txt and indexing.
Common causes and fixes
| Symptom | Likely issue | What to check |
|---|---|---|
| The page URL passes, but the preview image is missing | The image is on a different host or path governed by different rules | Test the final og:image URL and inspect that host’s robots.txt. |
| One crawler test passes and another fails | The crawlers match different user-agent groups | Inspect the applicable group and retest using the crawler in question. |
| The robots test passes, but the image still does not appear | The issue may be the request response, redirect destination, or platform-specific behavior | Request the final URL without authentication; verify redirects and that the response is an image. |
| A robots edit appears to have no effect | The rule may be on the wrong host, group, or path—or the provider controls the relevant file | Recheck the final URL’s authority and matched group; confirm the deployed file and logs with the site or CDN operator. |
Or skip the browser setup
For checking what an image URL returns, ScreenshotNeo provides a one-request screenshot API. This captures the URL you give it; it does not replace a crawler-specific robots tester or prove that a social platform can fetch the image.
ScreenshotNeo accepts cookie and consent banners and removes more than 60 known consent platforms, newsletter popups, and chat widgets before capture; each step can be turned off. Bot checks, blank pages, failed loads, timeouts, and cache hits are not billed, and responses identify the page verdict and billing status. Its MCP server lets AI agents use screenshot tools. The free plan includes 1,000 screenshots a month with no card; paid plans start at $5 for 3,000.
Example cURL request (see the ScreenshotNeo API documentation):
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://example.com/image.png -o shot.webpSign up for 1,000 free screenshots a month with no card.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
