The short answer

Audit the response your website actually serves. A useful service page needs a working URL, readable business information, a clear indexing policy, and links that lead to it. Test representative deep pages as well as the homepage, then separate technical eligibility from actual search inclusion.

Choose pages that represent the customer journey

A homepage-only check can miss the page that actually explains the service. Start with the homepage, one priority service page, the pricing or quotation page, the contact page, and a recently published article. Add a deliberately nonexistent URL to check how the site handles mistakes.

Write down the expected result before testing. The service page should return the service description; the contact page should explain a working contact method; the missing URL should return a genuine not-found response. This turns an audit into a comparison against a clear expectation.

Profound's Pages announcement connects page-level content, crawler activity, and citations in one view. The useful lesson for this checklist is to examine the actual page behind an observation. A broad visibility number cannot tell you whether one important URL is serving an empty response.

Inspect the response before the visual design

Ask your developer to save the HTTP status, relevant headers, and initial HTML for each selected URL. You do not need to interpret every line of code. Search the saved response for the page's service name, a distinctive paragraph, its main heading, and the contact link.

Compare that response with the page a person sees. If the browser shows detailed services while the first response contains only a loading indicator, document the gap. Rendering capabilities vary between consumers. Serving essential information directly makes the site easier to inspect and reduces dependence on a successful chain of scripts.

Also inspect the response body when the status is 200. A security challenge, generic error, or homepage fallback is not the intended service page. Conversely, a visually attractive error page should not pretend to be a successful resource. Keep a screenshot and the response evidence together so the finding is reproducible.

Check discovery and indexing controls together

Review the public robots policy, page-level indexing directives, canonical URL, sitemap, and incoming internal links as a set. They answer different questions and should express a consistent publishing decision.

Google's robots.txt documentation distinguishes control of crawling from removal from search results. Do not treat a robots rule as a reliable privacy boundary. Public content and private systems need different access arrangements.

Questions to ask for each intended public page
CheckQuestionEvidence to keep
HTTP responseDoes the final URL return the intended page?Status and saved response
CanonicalDoes the page identify the intended public URL?The canonical from the HTML head
Indexing policyIs a publishing or preview directive active?Meta tags and response headers
SitemapIs this a current, indexable destination?Matching sitemap entry
Internal linksCan someone reach the page from a relevant page?Source URL and real link target

A sitemap should not be the only route to useful content. Link a service article from its hub and connect it to the service it explains. Use descriptive link text so people can decide whether following the link will help.

Check what happens at the network edge

A page can work locally while a CDN or firewall changes the production response. Test the live domain and a deep URL from a normal unauthenticated request. Record redirects and any challenge page. Ask the person managing the edge configuration to inspect the corresponding event rather than disabling protection broadly.

Document the intended policy for search bots, training crawlers, and user-triggered retrieval separately. A synthetic request using a bot's User-Agent can reveal a rule, but it is not proof that a verified bot will receive the same treatment. For that, correlate verified traffic or use the search platform's inspection workflow.

When an audit tool receives a 403, retain the timestamp, URL, request conditions, and response information. The failure could involve a policy, a network path, or the tool itself. Avoid labelling the whole site inaccessible from a single unexplained result.

A real diagnosis: why our local sitemap returned 404

On September 28, 2026, FoundUp's local /sitemap.xml returned 404 while the production file returned 200 and listed nine pages. The cause was a build-mode difference: the local preview deliberately omitted discovery files and added noindex, while the production build generated them.

The useful repair was to select the intended build, not create a second hand-maintained URL list. The local SEO inspection build can generate a registry of 25 pages: nine core pages, ten articles and six comparison guides. It must be running to serve those pages. Do not infer production publication or search indexing from the build registry. That local count does not mean all 25 were deployed or indexed at the time of the observation.

The checks that isolated the missing file
EvidenceObserved before the changeWhat it established
Local HTTP response404 at /sitemap.xmlThe running local server did not serve the artifact
Live HTTP response200 with nine canonical URLsThe production file existed
Build configurationPreview mode skipped sitemap generationThe difference was intentional output selection
Release registry25 intended canonical pagesThe local release contained more pages than production

For your site, compare the live file, local output directory and build command. Also inspect response headers: a sitemap can exist while a preview-wide noindex header remains active. A file's presence, a page's indexability and confirmed search indexing are three different checks.

Use a normal GET request when inspecting the body:

curl -sS -D - https://getfoundup.ai/robots.txt
curl -sS -D - https://getfoundup.ai/sitemap.xml
curl -sS -D - http://localhost:8787/sitemap.xml

These commands inspect public responses. They do not impersonate a verified crawler or query a search engine's index.

Historical check: September 29, 2026, 04:58 UTC

At 04:58 UTC on September 29, 2026, a fresh production check found nine working core pages and nine sitemap entries. All ten article routes and six comparison routes returned 404. The local source generated 25 pages in an in-memory check, while the local preview server was not running. These are different release states, not proof that search engines have indexed the drafts.

Selected results from the September 29 production check
CheckObserved responseWhat the observation establishes
https://getfoundup.ai/sitemap.xml200; nine canonical entriesThe production sitemap covered the nine core pages at that time.
Ten /blog/ article routes and six /alternatives/ detail routes404Those detail routes were not available in that release.
http://getfoundup.ai/308 to HTTPSThe root protocol redirect worked; this does not verify every path or query.
Production core pages200; visible content and JSON-LDStructured data existed. Its fields still required a consistency review.
Local preview at that checkNo listener on port 8787Generated source output was not the same as an available preview server.

The later production Pricing and FAQ checks at 05:57 UTC showed new page layouts. That is another reason to retain the time of a review and recheck before publishing a local copy. These observations describe the reviewed release; they do not claim a search engine fetched it.

Turn findings into small, testable repairs

Give each issue a URL, observed behavior, expected behavior, owner, and retest method. “Improve SEO” is too broad. “The pricing page returns a challenge to the test request; inspect the matching edge event and verify the intended public response” is something a developer can investigate.

After a repair, repeat the same request and check another representative page for unintended effects. For routing changes, also retest legacy URLs and the missing-page response. For content changes, inspect the initial HTML and check that the visible facts agree with metadata and structured data.

Finish with two lists: technical checks that passed and external outcomes that remain unproven. A clean response does not establish indexing, rankings, citations, or real-user speed. Pair this audit with a maintenance process and keep the publishing format simple using the HTML and Markdown guide.

Post-launch verification: September 29, 2026

At 07:42 UTC on September 29, 2026, all 25 canonical pages returned HTTP 200. The sitemap included all ten article URLs and six comparison URLs. The full-text reference contained 25 page sections, and the RSS feed contained ten entries. These checks confirm publication and access; they do not establish search indexing or AI citations.

Historical findings and the later public release
CheckEarlier observation07:42 UTC verificationStill to establish
Article and comparison routesSixteen routes returned 404 at 04:58 UTC.All sixteen returned 200; all 25 canonical pages were accessible.Actual search indexing and AI use.
SitemapNine canonical URLs at 04:58 UTC.25 canonical URLs, including all articles and comparison details.Processing by the target search engines.
Public reference filesEarlier observations included an outdated form description.All five discovery resources returned 200; the full reference had 25 main sections and RSS had ten entries.Continued agreement after future edits.
Page contentLocal drafts and public pages had different release states.All 25 normalized main texts matched the reviewed local build.New checks when another version is published.

The earlier failures remain part of the record. The later check closes the publication issue at that time, without claiming that an independent search or AI platform had refreshed its copy. Read the dated change record and the reference-file check for the related follow-up.

Questions & answers

Does a 200 response mean a page is ready for search?

No. A challenge screen, empty application shell, or soft error page can also return 200. Check that the response contains the intended business information and that indexing controls are appropriate.

Should every crawler receive the same permission?

That is a business policy decision. Search retrieval, model training, and user-triggered requests have different purposes. Document which uses you permit before changing crawler rules.

Can changing a User-Agent prove that a real bot has access?

No. It is a useful diagnostic only. A CDN may handle verified crawler traffic differently. Use verified request logs and the platform's inspection tools for stronger evidence.

Sources and further reading

Sources checked . Platform behavior may change. Workflows are FoundUp's editorial recommendations. Fictional examples and observed research are identified separately.