A Next.js portfolio usually drops out of Google or ChatGPT for one of a small set of reasons: a crawler is blocked, the page tells search engines not to index it, the portfolio content is missing from the HTML a crawler receives, the canonical or metadata points somewhere else, or the sitemap is missing or wrong. The fastest way to find which one applies is to test the live production URL in the order below.
This is a diagnostic guide built on official documentation from Google Search Central, Next.js and OpenAI. It is not a reported case study. It does not show that any particular change caused a portfolio to appear or reappear, and none of the checks below guarantees a place in Google results or a citation in ChatGPT.
As an Amazon Associate I earn from qualifying purchases.
Start with what the public URL returns
Every other check depends on the response a public crawler gets from your production address. Begin there.
Recommended Free Tools
- Fetch the exact address you want indexed, following redirects, and read the final status:
curl -sIL https://www.example.com/. The result you want is a final 200 response at the address you intend as canonical. A redirect chain, a 404, a 5xx error, or a redirect to a different host means crawlers may not be receiving the page you think they are. - Do not use
localhostor a preview deployment as proof. They show what you see in your own environment, not necessarily what a public crawler receives. - Note the status before going further. Google documents that HTTP status codes affect crawling and rendering, and that a page returning a non-200 status can have rendering skipped. A non-200 response alone can explain an absent page, and no metadata change will fix it while the page does not return 200. (Google Search Central, JavaScript SEO basics)
Map each symptom to one layer
Google processes a JavaScript page in distinct stages: crawling, rendering and indexing. Googlebot first checks whether it is allowed to crawl the page, and blocked pages or resources are not rendered. Each failure below sits at a different layer, so each needs a different fix.
#1 Best Overall
| Layer | What blocks it | Where to look in a Next.js App Router project | What you will observe |
|---|---|---|---|
| Crawl permission | robots.txt disallows the portfolio path or a resource the page needs to render | app/robots.ts or app/robots.txt, then the deployed /robots.txt |
A Disallow rule that matches / or a required stylesheet or script path |
| HTTP response | Non-200 status, redirect chain, redirect to another host, error page | Hosting dashboard logs, curl -sIL |
A final status other than 200, or a final URL that differs from the one you intend |
| Index directive | noindex in robots meta or in the X-Robots-Tag header |
The robots field of the Metadata API, or headers set in next.config or middleware |
A robots meta tag or X-Robots-Tag header containing noindex |
| Rendered content | Portfolio copy appears only after client-side JavaScript runs | Components that fetch data in the browser versus on the server | Your name and project titles appear in the browser but not in the raw HTML response |
| Canonical and metadata | Canonical points to another URL, or titles and descriptions are duplicated across pages | alternates.canonical and title in metadata exports |
A canonical URL that differs from the address you are testing |
| Sitemap | Sitemap not generated, unreachable, or listing URLs on the wrong host | app/sitemap.ts and the sitemap line in robots output |
A 404 at /sitemap.xml, or URLs that do not match the canonical host |
| ChatGPT Search access | Rules that block OAI-SearchBot | The rules for OAI-SearchBot in robots output |
A disallow rule that applies to OAI-SearchBot |
Check crawl access first
Read the deployed robots.txt
In the Next.js App Router, a robots file can be generated from app/robots.ts. It can set allow and disallow rules per user agent and declare a sitemap URL.
import type { MetadataRoute } from 'next'
export default function robots(): MetadataRoute.Robots {
return {
rules: [
{ userAgent: '*', allow: '/' },
{ userAgent: 'OAI-SearchBot', allow: '/' },
],
sitemap: 'https://www.example.com/sitemap.xml',
}
}
After deployment, run curl -s https://www.example.com/robots.txt and confirm three things: the portfolio path is not disallowed, the stylesheet and script paths the page needs are not disallowed, and the sitemap line points to the host you use publicly. A staging rule such as a site-wide disallow that was carried into production is a common cause of a blocked crawler, and it will appear in this output. The Next.js robots reference is at Metadata Files: robots.txt.
Test each crawler’s access separately
Googlebot and OAI-SearchBot have separate policies, so a rule can block one and allow the other. You can see what your server returns to a given user-agent string with:
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Repair Windows errors before they cause bigger problems3Fix the driver behind crashes, sound loss and screen glitchescurl -s -A Googlebot -o /dev/null -w '%{http_code}' https://www.example.com/
curl -s -A OAI-SearchBot -o /dev/null -w '%{http_code}' https://www.example.com/
This shows only how your server responds to that string. It does not prove what the real crawler receives, because a user-agent header can be set by anyone. For Google’s own view of a URL, use the URL Inspection tool in Google Search Console. For OpenAI’s crawler, check your host or server logs for requests identifying OAI-SearchBot.
Rank #2
Look for noindex in HTML and headers
A noindex directive can appear in two places. Check both.
- The HTML head. Search the page source for a robots meta tag. The command below prints any such tag, escaped here for display:
curl -s https://www.example.com/ | grep -io '<meta[^>]*robots[^>]*>' - Response headers. Check for an
X-Robots-Tagheader:curl -sI https://www.example.com/ | grep -i x-robots-tag
In the Next.js Metadata API, the robots field controls the robots meta output. Setting index: false on a page produces a noindex directive, so check every layout and page that exports metadata, including inherited values from a parent layout. Google’s specification for robots meta tags and X-Robots-Tag is at Robots Meta Tags Specifications.
Robots.txt and noindex interact in a way that often confuses people. Google states that robots meta and X-Robots-Tag rules can only be read when crawlers can access the page, and a robots.txt disallow can stop a crawler from seeing a noindex directive at all. If your goal is to remove a page from results, you must allow crawling so the noindex can be read. Blocking the page in robots.txt alone does not do that.
Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Repair Windows errors before they cause bigger problemsFix Now →Compare the raw HTML with the rendered page
Google can render JavaScript, but it queues pages for rendering, and rendering may be delayed. Google’s guidance also says that server-side or prerendered HTML helps users and crawlers alike, and that some bots cannot run JavaScript at all. That is a robustness argument, not proof that a client-rendered page cannot be indexed. Still, if the portfolio’s meaningful content is absent from the raw response, you are relying on a rendering step you cannot see.
Rank #3
- Save the raw response with
curl -s https://www.example.com/ > raw.htmland search the file for your name, your headline and one project title. - In a browser, open DevTools and compare the Elements panel with the page source (View Page Source). If the words appear only in Elements, JavaScript is inserting them after load.
- Check that navigation uses crawlable links, not click handlers that only change the page state.
- If the content is missing from raw HTML, move the data fetching into a Server Component or generate the page at build time, then repeat step 1.
The Google reference for this stage is Understand JavaScript SEO Basics.
Review metadata and canonical signals
Next.js lets you set metadata per route. Give each page its own title and description, and declare the canonical URL explicitly:
import type { Metadata } from 'next'
export const metadata: Metadata = {
title: 'Jane Example | Frontend Engineer',
description: 'Selected Next.js and React projects by Jane Example.',
alternates: {
canonical: 'https://www.example.com/',
},
}
Check that the canonical in the raw HTML matches the canonical in the rendered DOM, and that it matches the host, protocol and trailing-slash form you use in redirects and in the sitemap. Google treats canonical markup as a signal, not a command, so a mismatch gives search engines a conflicting instruction to resolve rather than a guaranteed outcome either way. The Next.js reference is Getting Started: Metadata and OG images, and the Next.js SEO tag guide is at SEO: Special Meta Tags for Search Engines.
Confirm the sitemap lists the URLs you want found
Next.js supports a sitemap convention through app/sitemap.ts. A minimal version looks like this:
import type { MetadataRoute } from 'next'
export default function sitemap(): MetadataRoute.Sitemap {
return [
{ url: 'https://www.example.com/', lastModified: new Date() },
{ url: 'https://www.example.com/projects', lastModified: new Date() },
]
}
Run curl -s https://www.example.com/sitemap.xml and confirm it returns XML, that every listed URL returns a 200 on the same host, and that the robots.txt sitemap line points to it. A sitemap helps discovery. It does not guarantee indexing, so treat a correct sitemap as one check passed, not as the fix.
Google and ChatGPT are separate discovery paths
Passing Google’s checks does not establish anything about ChatGPT, and allowing OAI-SearchBot does not establish that ChatGPT will cite your page. OpenAI’s publisher FAQ states the one condition it names:
“For your site content to be included in summaries and snippets in ChatGPT, make sure you aren’t blocking OAI-SearchBot.”
Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
Source: OpenAI Help Center, Publishers and Developers – FAQ (displayed as updated in the month before it was checked for this guide, October 2026). The sentence describes eligibility. It is not a promise of appearance or ranking.
| Question | Google Search | ChatGPT Search |
|---|---|---|
| Governing guidance | Google Search Central documentation on JavaScript SEO and robots meta tags | OpenAI Help Center publisher FAQ |
| Crawler to keep unblocked | Googlebot | OAI-SearchBot |
| Verification tool | Search Console URL Inspection | Not stated in the OpenAI FAQ cited here |
| What the official source promises | Describes how pages are crawled, rendered and indexed; does not promise indexing | Content can be included in summaries and snippets if OAI-SearchBot is not blocked; no guarantee of inclusion or citation |
Record evidence before and after every change
A fix is only established by a before-and-after record. Keep one dated log per URL.
- Before changing anything, record the date, the final URL and status, the robots.txt result for each user agent, any noindex signal, the canonical, and whether the URL appears in the sitemap.
- Change one layer at a time, so you know which change affected which result.
- Redeploy, then repeat the
curlchecks against the production URL. Note the deployment time. - Inspect the URL in Search Console and record the status it shows and the date you checked.
- For ChatGPT, record whether requests identifying OAI-SearchBot appear in your server or host logs, and log that separately from any appearance in ChatGPT answers.
The official pages cited in this guide do not give a fixed timeline for recrawling or reindexing, so do not expect a result on a schedule. Record what you observe and when.
Troubleshooting branches
- Final status is not 200. Fix routing, redirects or hosting first. Metadata and sitemap changes cannot compensate for a page that does not return 200.
- robots.txt disallows the page or a required asset. Correct the rule, redeploy, and re-run the robots check.
- A noindex directive is present. Remove it if you want the page indexed. If you want it removed from results, keep crawling allowed so the directive can be read.
- Content appears only after JavaScript runs. Move data fetching to the server or generate the page at build time, then repeat the raw HTML check.
- The canonical points elsewhere. Set it to the intended public URL in every layer that outputs it.
- The sitemap is missing or lists the wrong host. Fix the generated file and the sitemap line in robots.txt together.
- Every check passes and the page still does not appear. The checks in this guide cover technical access and signals. They cannot confirm that a search engine will rank or cite a page. Keep the dated log and re-check after each deployment.
What the sources do not establish
The official sources cited here describe how crawling, rendering, indexing and OAI-SearchBot access work. They do not include a statistic measuring how many Next.js portfolios are indexed, and they do not quantify the effect of any single fix. Treat any percentage or timeline you see elsewhere on this topic as unverified unless it names its study, method and date.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
