The short answer
Audit the response your website actually serves. A useful service page needs a working URL, readable business information, a clear indexing policy, and links that lead to it. Test representative deep pages as well as the homepage, then separate technical eligibility from actual search inclusion.
Choose pages that represent the customer journey
A homepage-only check can miss the page that actually explains the service. Start with the homepage, one priority service page, the pricing or quotation page, the contact page, and a recently published article. Add a deliberately nonexistent URL to check how the site handles mistakes.
Write down the expected result before testing. The service page should return the service description; the contact page should explain a working contact method; the missing URL should return a genuine not-found response. This turns an audit into a comparison against a clear expectation.
Profound's Pages announcement connects page-level content, crawler activity, and citations in one view. The useful lesson for this checklist is to examine the actual page behind an observation. A broad visibility number cannot tell you whether one important URL is serving an empty response.
Inspect the response before the visual design
Ask your developer to save the HTTP status, relevant headers, and initial HTML for each selected URL. You do not need to interpret every line of code. Search the saved response for the page's service name, a distinctive paragraph, its main heading, and the contact link.
Compare that response with the page a person sees. If the browser shows detailed services while the first response contains only a loading indicator, document the gap. Rendering capabilities vary between consumers. Serving essential information directly makes the site easier to inspect and reduces dependence on a successful chain of scripts.
Also inspect the response body when the status is 200. A security challenge, generic error, or homepage fallback is not the intended service page. Conversely, a visually attractive error page should not pretend to be a successful resource. Keep a screenshot and the response evidence together so the finding is reproducible.
Check discovery and indexing controls together
Review the public robots policy, page-level indexing directives, canonical URL, sitemap, and incoming internal links as a set. They answer different questions and should express a consistent publishing decision.
Google's robots.txt documentation distinguishes control of crawling from removal from search results. Do not treat a robots rule as a reliable privacy boundary. Public content and private systems need different access arrangements.
| Check | Question | Evidence to keep |
|---|---|---|
| HTTP response | Does the final URL return the intended page? | Status and saved response |
| Canonical | Does the page identify the intended public URL? | The canonical from the HTML head |
| Indexing policy | Is a publishing or preview directive active? | Meta tags and response headers |
| Sitemap | Is this a current, indexable destination? | Matching sitemap entry |
| Internal links | Can someone reach the page from a relevant page? | Source URL and real link target |
A sitemap should not be the only route to useful content. Link a service article from its hub and connect it to the service it explains. Use descriptive link text so people can decide whether following the link will help.
Check what happens at the network edge
A page can work locally while a CDN or firewall changes the production response. Test the live domain and a deep URL from a normal unauthenticated request. Record redirects and any challenge page. Ask the person managing the edge configuration to inspect the corresponding event rather than disabling protection broadly.
Document the intended policy for search bots, training crawlers, and user-triggered retrieval separately. A synthetic request using a bot's User-Agent can reveal a rule, but it is not proof that a verified bot will receive the same treatment. For that, correlate verified traffic or use the search platform's inspection workflow.
When an audit tool receives a 403, retain the timestamp, URL, request conditions, and response information. The failure could involve a policy, a network path, or the tool itself. Avoid labelling the whole site inaccessible from a single unexplained result.
A real diagnosis: why our local sitemap returned 404
On September 28, 2026, FoundUp's local /sitemap.xml returned 404 while the production file returned 200 and listed nine pages. The cause was a build-mode difference: the local preview deliberately omitted discovery files and added noindex, while the production build generated them.
The useful repair was to select the intended build, not create a second hand-maintained URL list. The local SEO inspection build can generate a registry of 25 pages: nine core pages, ten articles and six comparison guides. It must be running to serve those pages. Do not infer production publication or search indexing from the build registry. That local count does not mean all 25 were deployed or indexed at the time of the observation.
| Evidence | Observed before the change | What it established |
|---|---|---|
| Local HTTP response | 404 at /sitemap.xml | The running local server did not serve the artifact |
| Live HTTP response | 200 with nine canonical URLs | The production file existed |
| Build configuration | Preview mode skipped sitemap generation | The difference was intentional output selection |
| Release registry | 25 intended canonical pages | The local release contained more pages than production |
For your site, compare the live file, local output directory and build command. Also inspect response headers: a sitemap can exist while a preview-wide noindex header remains active. A file's presence, a page's indexability and confirmed search indexing are three different checks.
Use a normal GET request when inspecting the body:
curl -sS -D - https://getfoundup.ai/robots.txt
curl -sS -D - https://getfoundup.ai/sitemap.xml
curl -sS -D - http://localhost:8787/sitemap.xmlThese commands inspect public responses. They do not impersonate a verified crawler or query a search engine's index.
Historical check: September 29, 2026, 04:58 UTC
At 04:58 UTC on September 29, 2026, a fresh production check found nine working core pages and nine sitemap entries. All ten article routes and six comparison routes returned 404. The local source generated 25 pages in an in-memory check, while the local preview server was not running. These are different release states, not proof that search engines have indexed the drafts.
| Check | Observed response | What the observation establishes |
|---|---|---|
| https://getfoundup.ai/sitemap.xml | 200; nine canonical entries | The production sitemap covered the nine core pages at that time. |
| Ten /blog/ article routes and six /alternatives/ detail routes | 404 | Those detail routes were not available in that release. |
| http://getfoundup.ai/ | 308 to HTTPS | The root protocol redirect worked; this does not verify every path or query. |
| Production core pages | 200; visible content and JSON-LD | Structured data existed. Its fields still required a consistency review. |
| Local preview at that check | No listener on port 8787 | Generated source output was not the same as an available preview server. |
The later production Pricing and FAQ checks at 05:57 UTC showed new page layouts. That is another reason to retain the time of a review and recheck before publishing a local copy. These observations describe the reviewed release; they do not claim a search engine fetched it.
Turn findings into small, testable repairs
Give each issue a URL, observed behavior, expected behavior, owner, and retest method. “Improve SEO” is too broad. “The pricing page returns a challenge to the test request; inspect the matching edge event and verify the intended public response” is something a developer can investigate.
After a repair, repeat the same request and check another representative page for unintended effects. For routing changes, also retest legacy URLs and the missing-page response. For content changes, inspect the initial HTML and check that the visible facts agree with metadata and structured data.
Finish with two lists: technical checks that passed and external outcomes that remain unproven. A clean response does not establish indexing, rankings, citations, or real-user speed. Pair this audit with a maintenance process and keep the publishing format simple using the HTML and Markdown guide.
Post-launch verification: September 29, 2026
At 07:42 UTC on September 29, 2026, all 25 canonical pages returned HTTP 200. The sitemap included all ten article URLs and six comparison URLs. The full-text reference contained 25 page sections, and the RSS feed contained ten entries. These checks confirm publication and access; they do not establish search indexing or AI citations.
| Check | Earlier observation | 07:42 UTC verification | Still to establish |
|---|---|---|---|
| Article and comparison routes | Sixteen routes returned 404 at 04:58 UTC. | All sixteen returned 200; all 25 canonical pages were accessible. | Actual search indexing and AI use. |
| Sitemap | Nine canonical URLs at 04:58 UTC. | 25 canonical URLs, including all articles and comparison details. | Processing by the target search engines. |
| Public reference files | Earlier observations included an outdated form description. | All five discovery resources returned 200; the full reference had 25 main sections and RSS had ten entries. | Continued agreement after future edits. |
| Page content | Local drafts and public pages had different release states. | All 25 normalized main texts matched the reviewed local build. | New checks when another version is published. |
The earlier failures remain part of the record. The later check closes the publication issue at that time, without claiming that an independent search or AI platform had refreshed its copy. Read the dated change record and the reference-file check for the related follow-up.
Questions & answers
Does a 200 response mean a page is ready for search?
No. A challenge screen, empty application shell, or soft error page can also return 200. Check that the response contains the intended business information and that indexing controls are appropriate.
Should every crawler receive the same permission?
That is a business policy decision. Search retrieval, model training, and user-triggered requests have different purposes. Document which uses you permit before changing crawler rules.
Can changing a User-Agent prove that a real bot has access?
No. It is a useful diagnostic only. A CDN may handle verified crawler traffic differently. Use verified request logs and the platform's inspection tools for stronger evidence.
Sources and further reading
Sources checked . Platform behavior may change. Workflows are FoundUp's editorial recommendations. Fictional examples and observed research are identified separately.