Fix crawlability, indexation, and site architecture
Find and sequence the technical problems that stop search engines and answer engines from understanding the site.
Make sure Google can crawl, render, and index revenue pages, then test access separately for any other automated crawler you intentionally support.
You will have a prioritized engineering queue, an indexation policy, and a clean path from the homepage to every revenue page.
Publishing more content on a site with duplicate URLs, weak internal links, or accidental noindex rules compounds waste. Technical SEO is not a giant checklist; it is the removal of specific obstacles between discovery and a useful indexed page.
IF YOU SKIP IT / A robots.txt rule, CDN response, rendering failure, or indexation decision can prevent the intended system from using a page. Each system needs its own verified test.
Crawler controls are product-specific
Keep search inclusion, user-requested visits, training controls, rendering, and indexing in separate rows. A successful fetch for one user agent says nothing about another product.
- Crawl
- A bot visiting your pages by following links, the way Google and AI engines discover what exists on your site.
- Index
- Google's stored copy of your page. Crawled does not mean indexed; a page can be visited and still left out of results.
- Canonical
- A tag that tells engines which URL is the official version when several similar ones exist. Wrong canonicals quietly delete pages from search.
- Noindex
- A tag that tells engines to keep a page out of results. Useful on purpose, catastrophic when a template applies it to revenue pages by accident.
- Orphan page
- A page with no internal links pointing to it. It has a weaker discovery and context path, although sitemaps, external links, and other signals may still expose it.
- CDN / WAF
- The security layer in front of your site (Cloudflare, Akamai, Vercel). It can return a 403 to an automated crawler even when robots.txt allows that user agent.
Each phase feeds the next. Don't skip ahead, the artifact at the end is only trustworthy if every phase ran.
- 01Configure
Crawl as desktop, mobile, and rendered JavaScript where the framework requires it.
- 02Reconcile
Compare crawlable, indexable, canonical, sitemap, analytics, backlink, and GSC URL sets.
- 03Sample
Inspect representative URLs from every revenue and content template manually.
- 04Group
Consolidate issue rows into template-level root causes.
- 05Prioritize
Use severity, revenue exposure, URL count, confidence, and effort instead of crawler defaults.
- 06Specify
Write the expected rule, affected templates, acceptance test, and rollback risk for engineering.
- 07Ship
Release critical fixes in controlled batches.
- 08Prove
Recrawl, inspect live HTML, and monitor indexation and traffic until the expected state is stable.
- ✓You now have
Technical fix queue
Create the crawl and index inventory
Compare what exists, what can be crawled, and what Google actually indexed.
Picture a library where a third of the books never made it into the catalog. The books exist. They sit on real shelves. And no visitor will ever find one, because the catalog is the only thing anyone searches. Your site works the same way: pages that exist, pages a crawler can reach, and pages Google chose to index are three different lists, and on most SaaS sites they disagree wildly.
The gaps between those lists are your entire technical roadmap, so this lesson builds the honest map first. Run the crawl, pull the Search Console export, put them side by side, and give every URL one of five verbs: Keep, Improve, Merge, Redirect, Remove. If you cannot pick a verb, you do not understand the page yet, and that is worth knowing too.
- 01
Crawl the site and export status code, canonical, robots directive, title, H1, depth, and indexability.
- 02
Export indexed and excluded URLs from Google Search Console.
- 03
Label each URL Keep, Improve, Merge, Redirect, or Remove based on buyer value and uniqueness.
- Download Screaming Frog (free up to 500 URLs). Set Configuration > Spider > Rendering to JavaScript, enter your domain, crawl, then Export the Internal All report.
- In Google Search Console: Indexing > Pages. Note the indexed count, then open each 'Why pages aren't indexed' reason and use Export on the affected URLs.
- Combine both exports in one sheet keyed by URL (a VLOOKUP or XLOOKUP on the URL column) and add a Label column with a dropdown: Keep, Improve, Merge, Redirect, Remove.
Every URL on the site has exactly one label, and you can state the number of indexable revenue pages without opening a tool.
- WATCH OUT
Crawl the real production domain, logged out, not a staging URL. Staging often has different robots rules and will lie to you.
- NO BUDGET?
Over 500 URLs and no budget? Crawl the sections that matter first: product, pricing, comparisons, integrations. The blog can wait a week.
- PRO TIP
No Search Console access yet? Verify the domain today (Settings > Ownership verification, DNS record). Every later module leans on this account.
OPTIONAL / AI CO-PILOTRun this lesson with AI, the prompt, sources and expected output+
Choose the business priority, interpret exceptions, protect confidential data, challenge weak evidence, and approve the final decision.
Clean, classify, compare, calculate, and draft rows from supplied evidence. AI may surface patterns; it does not own strategy.
Screaming Frog / Sitebulb, plus index evidence sample from GSC URL Inspection.
| Crawler | User agent | Rendering | Scope | Max URLs | Data connected | Run owner |
|---|---|---|---|---|---|---|
| Screaming Frog | Googlebot smartphone | JavaScript | Production host | 20,000 | GSC + GA4 | Technical SEO |
CLAUDE / CHATGPT PROMPTAnalyze the evidence without outsourcing the decision+
Remove personal or confidential data. Attach the named exports, explain every column, and tell the model when the dataset was collected.
You are assisting a human B2B SaaS SEO and AEO operator with Module 02: Fix crawlability, indexation, and site architecture. Current lesson: Create the crawl and index inventory Objective: Compare what exists, what can be crawled, and what Google actually indexed. Required artifact: Technical fix queue BUSINESS CONTEXT I WILL PROVIDE - Product, category, target market, pricing model, sales motion, and geography - The priority customer segment and the commercial outcome for this 90-day cycle - Internal HTML + issues exports exported from Screaming Frog / Sitebulb - Index evidence sample exported from GSC URL Inspection - Definitions for any internal fields, stages, scores, and abbreviations TASK 1. Crawl the site and export status code, canonical, robots directive, title, H1, depth, and indexability. 2. Export indexed and excluded URLs from Google Search Console. 3. Label each URL Keep, Improve, Merge, Redirect, or Remove based on buyer value and uniqueness. REQUIRED OUTPUT Return a table using these exact columns: Crawler | User agent | Rendering | Scope | Max URLs | Data connected | Run owner. For every recommendation, cite the source row, URL, call note, or data point that supports it. Add a confidence column in your analysis: High, Medium, or Low. List missing evidence separately instead of guessing. Finish with a section named HUMAN DECISIONS REQUIRED. RULES - Do not invent search volume, revenue, customer statements, product capabilities, or competitor facts. - Do not treat correlation as causation. - Preserve contradictory evidence and explain why it conflicts. - Do not make the final priority or publishing decision. Prepare the evidence for a human owner. - Use this example only as a format reference, not as evidence: A parameter URL that duplicates a feature page is “Canonicalize or block”; an old campaign page with backlinks is “Redirect.”
Every URL template has an index rule. A named human owner must verify this before the lesson is complete.
A parameter URL that duplicates a feature page is “Canonicalize or block”; an old campaign page with backlinks is “Redirect.”
Write an indexation policy
Templates should have a default rule, so the same problem does not return with every release.
Here is why technical problems come back three months after you fix them: nobody wrote down the rule. The fix was a repair, and repairs wear off. The engineer shipping the next template makes a fresh judgment call, and the same noindex mistake returns wearing a new URL.
Cities solved this with building codes. You are going to solve it with one boring document: every URL template, its crawl rule, its index rule, its canonical target, and an owner. Write it once and every future release inherits the right answer without a meeting.
/product/*/integrations/*/blog/*/tags/*/search?q=*?utm=*Blocks the fetch. The URL can still appear in results, and the noindex tag inside the page is never read.
Removes the page from the index, but only if the crawler is still allowed to fetch it and see the tag.
Cancels itself out. Pick one mechanism per template, and write down which file or tag enforces it.
- 01
List every URL template: product, use case, integration, blog, tag, search, parameter, and utility.
- 02
Choose index/follow, noindex/follow, canonical target, or disallow only after checking whether crawling is needed.
- 03
Document the owner and test for each rule before shipping.
- Open a one-page doc. List every URL pattern your site generates: /product/*, /blog/*, /tags/*, /integrations/*, ?utm= parameters, search result pages.
- For each pattern write four cells: crawl rule, index rule, canonical target, owner. Where the rule is enforced matters: meta robots tag in the template, robots.txt, or the canonical link element.
- Before shipping any rule, test one URL in Search Console's URL Inspection tool to confirm Google sees what you intended.
Every template has a written rule with an owner, and an engineer could apply it to a brand-new page type without asking you.
- WATCH OUT
robots.txt Disallow does not remove pages from the index; it only blocks crawling, and blocked pages can still appear in results. Use noindex for exclusion, and never both at once.
- PRO TIP
Put the policy doc in the engineering wiki, not the marketing drive. The people who ship templates are the ones who need to trip over it.
OPTIONAL / AI CO-PILOTRun this lesson with AI, the prompt, sources and expected output+
Choose the business priority, interpret exceptions, protect confidential data, challenge weak evidence, and approve the final decision.
Clean, classify, compare, calculate, and draft rows from supplied evidence. AI may surface patterns; it does not own strategy.
GSC URL Inspection, plus revenue url overlays from Ahrefs / Semrush.
| URL template | Crawl rule | Index rule | Canonical | Sitemap | Minimum value | Owner |
|---|---|---|---|---|---|---|
| /integrations/* | Allow | Conditional | Self | Qualified only | Setup + unique workflow | Platform |
CLAUDE / CHATGPT PROMPTAnalyze the evidence without outsourcing the decision+
Remove personal or confidential data. Attach the named exports, explain every column, and tell the model when the dataset was collected.
You are assisting a human B2B SaaS SEO and AEO operator with Module 02: Fix crawlability, indexation, and site architecture. Current lesson: Write an indexation policy Objective: Templates should have a default rule, so the same problem does not return with every release. Required artifact: Technical fix queue BUSINESS CONTEXT I WILL PROVIDE - Product, category, target market, pricing model, sales motion, and geography - The priority customer segment and the commercial outcome for this 90-day cycle - Index evidence sample exported from GSC URL Inspection - Revenue URL overlays exported from Ahrefs / Semrush - Definitions for any internal fields, stages, scores, and abbreviations TASK 1. List every URL template: product, use case, integration, blog, tag, search, parameter, and utility. 2. Choose index/follow, noindex/follow, canonical target, or disallow only after checking whether crawling is needed. 3. Document the owner and test for each rule before shipping. REQUIRED OUTPUT Return a table using these exact columns: URL template | Crawl rule | Index rule | Canonical | Sitemap | Minimum value | Owner. For every recommendation, cite the source row, URL, call note, or data point that supports it. Add a confidence column in your analysis: High, Medium, or Low. List missing evidence separately instead of guessing. Finish with a section named HUMAN DECISIONS REQUIRED. RULES - Do not invent search volume, revenue, customer statements, product capabilities, or competitor facts. - Do not treat correlation as causation. - Preserve contradictory evidence and explain why it conflicts. - Do not make the final priority or publishing decision. Prepare the evidence for a human owner. - Use this example only as a format reference, not as evidence: Integration pages index only when they contain a real workflow, setup detail, and unique value; empty directories remain noindex.
Critical and high issues have owners. A named human owner must verify this before the lesson is complete.
Integration pages index only when they contain a real workflow, setup detail, and unique value; empty directories remain noindex.
Flatten the revenue architecture
Important pages should not be orphaned or buried five clicks deep.
A store that keeps its best product in the basement, behind an unmarked door, still technically stocks it. That is what a money page five clicks deep is. Your buyer will not dig for it and neither will a crawler.
Internal links are the cheapest lever you fully control. No outreach, no budget, no waiting on anyone. You are simply deciding that your strongest pages will point at the pages that close deals, with link text that says where the door leads. An afternoon of linking often outperforms a month of new content.
- 01
Draw the path from homepage to category, use-case, comparison, and integration pages.
- 02
Classify the link text already pointing at each revenue page as descriptive, brand only, vague, or empty, excluding navigation and footer links.
- 03
Add contextual links from high-authority pages using language that explains the destination.
- 04
Keep priority revenue pages within three meaningful clicks where the site structure allows it.
- In Screaming Frog after a crawl: the Crawl Depth column shows clicks from the start page, and the Inlinks count shows how many pages link to each URL. Sort revenue URLs by both.
- Find your strongest pages in Search Console (Links report > Top linked pages). Those are the pages whose body copy should link to money pages.
- Add the links inside the content in your CMS, with anchor text that says what the destination is, then recrawl to confirm the depth dropped.
- Before you add anything, read the links you already have. In Screaming Frog, select a revenue URL and open the Inlinks pane at the bottom: the Anchor Text column lists every internal link pointing at it. Bucket each one as descriptive, brand only, vague ("learn more", "get a demo"), or empty, and exclude navigation and footer rows before you calculate a share.
Every priority revenue page is within three meaningful clicks of the homepage and has at least two contextual links from high-authority pages.
- PRO TIP
The highest-leverage single link is usually homepage body copy to your primary use-case page. Most SaaS homepages link only to /features and /pricing.
- WATCH OUT
Footer and nav links are weak votes. A link inside a paragraph, surrounded by relevant words, is the vote that counts.
- PRO TIP
Run the same anchor count on the page’s children before you conclude anything. If the children come back half descriptive and the parent close to zero, the site already knows how to write a link and simply never did it for the page that closes deals. That is a habit, and habits are cheap to change.
OPTIONAL / AI CO-PILOTRun this lesson with AI, the prompt, sources and expected output+
Choose the business priority, interpret exceptions, protect confidential data, challenge weak evidence, and approve the final decision.
Clean, classify, compare, calculate, and draft rows from supplied evidence. AI may surface patterns; it does not own strategy.
Ahrefs / Semrush, plus prioritized audit queue from PageOptimized.
| Revenue URL | Click depth | Internal links | Orphan | Source hub | Required anchor | Action |
|---|---|---|---|---|---|---|
| /pricing/ | 4 | 7 | No | /product/ | reporting software pricing | Add nav + contextual links |
CLAUDE / CHATGPT PROMPTAnalyze the evidence without outsourcing the decision+
Remove personal or confidential data. Attach the named exports, explain every column, and tell the model when the dataset was collected.
You are assisting a human B2B SaaS SEO and AEO operator with Module 02: Fix crawlability, indexation, and site architecture. Current lesson: Flatten the revenue architecture Objective: Important pages should not be orphaned or buried five clicks deep. Required artifact: Technical fix queue BUSINESS CONTEXT I WILL PROVIDE - Product, category, target market, pricing model, sales motion, and geography - The priority customer segment and the commercial outcome for this 90-day cycle - Revenue URL overlays exported from Ahrefs / Semrush - Prioritized audit queue exported from PageOptimized - Definitions for any internal fields, stages, scores, and abbreviations TASK 1. Draw the path from homepage to category, use-case, comparison, and integration pages. 2. Classify the link text already pointing at each revenue page as descriptive, brand only, vague, or empty, excluding navigation and footer links. 3. Add contextual links from high-authority pages using language that explains the destination. 4. Keep priority revenue pages within three meaningful clicks where the site structure allows it. REQUIRED OUTPUT Return a table using these exact columns: Revenue URL | Click depth | Internal links | Orphan | Source hub | Required anchor | Action. For every recommendation, cite the source row, URL, call note, or data point that supports it. Add a confidence column in your analysis: High, Medium, or Low. List missing evidence separately instead of guessing. Finish with a section named HUMAN DECISIONS REQUIRED. RULES - Do not invent search volume, revenue, customer statements, product capabilities, or competitor facts. - Do not treat correlation as causation. - Preserve contradictory evidence and explain why it conflicts. - Do not make the final priority or publishing decision. Prepare the evidence for a human owner. - Use this example only as a format reference, not as evidence: A reporting guide links to “automated client reporting software,” then the product page links to relevant integrations and comparisons.
Revenue pages are within a clear link path. A named human owner must verify this before the lesson is complete.
A reporting guide links to “automated client reporting software,” then the product page links to relevant integrations and comparisons.
Prioritize fixes by impact
Do not let minor metadata warnings outrank blocked pages or broken templates.
An emergency room does not treat patients in the order they arrived. It treats by severity, which is why the audit tool's list of 400 issues, sorted top to bottom, is a trap. A missing meta description is a paper cut. Six hundred integration pages canonicalized to the homepage is a house fire.
Sort by blast radius: does it block discovery entirely, does it break a whole template, or is it polish. Fix critical template problems first, then do the step almost everyone skips. Recrawl. A fix that shipped but was never verified is a rumor.
- 01
Set severity: Critical blocks discovery/indexing; High affects a template or revenue group; Medium reduces clarity; Low is polish.
- 02
Add affected URL count, traffic/pipeline value, effort, owner, and verification test.
- 03
Ship critical template fixes first, then validate in a second crawl and URL Inspection.
- In the combined sheet, add Severity, Affected URLs, Owner, and Acceptance test columns. Sort by severity, then by URL count.
- File the top items as engineering tickets, one per template problem, each with its acceptance test written in the ticket ('all 47 /compare/ URLs return self-canonical').
- After the deploy, recrawl the affected section and re-check Search Console. A fix is not done until the second crawl says so.
The queue is sorted by severity times scope, every critical item has an engineering owner, and a second crawl confirms the first fixes.
- PRO TIP
Bundle by template, not by URL. One ticket that fixes a canonical rule repairs 600 pages; 600 tickets fix nothing because nobody starts.
- WATCH OUT
Resist fixing Low items because they are easy. Every hour on title-tag polish while revenue pages sit noindexed is an hour billed to nothing.
OPTIONAL / AI CO-PILOTRun this lesson with AI, the prompt, sources and expected output+
Choose the business priority, interpret exceptions, protect confidential data, challenge weak evidence, and approve the final decision.
Clean, classify, compare, calculate, and draft rows from supplied evidence. AI may surface patterns; it does not own strategy.
PageOptimized, plus internal html + issues exports from Screaming Frog / Sitebulb.
| Template | Root cause | Affected URLs | Severity | Revenue risk | Owner | Acceptance test |
|---|---|---|---|---|---|---|
| /compare/* | Missing from sitemap | 30 | High | High | Platform | Included and indexable |
CLAUDE / CHATGPT PROMPTAnalyze the evidence without outsourcing the decision+
Remove personal or confidential data. Attach the named exports, explain every column, and tell the model when the dataset was collected.
You are assisting a human B2B SaaS SEO and AEO operator with Module 02: Fix crawlability, indexation, and site architecture. Current lesson: Prioritize fixes by impact Objective: Do not let minor metadata warnings outrank blocked pages or broken templates. Required artifact: Technical fix queue BUSINESS CONTEXT I WILL PROVIDE - Product, category, target market, pricing model, sales motion, and geography - The priority customer segment and the commercial outcome for this 90-day cycle - Prioritized audit queue exported from PageOptimized - Internal HTML + issues exports exported from Screaming Frog / Sitebulb - Definitions for any internal fields, stages, scores, and abbreviations TASK 1. Set severity: Critical blocks discovery/indexing; High affects a template or revenue group; Medium reduces clarity; Low is polish. 2. Add affected URL count, traffic/pipeline value, effort, owner, and verification test. 3. Ship critical template fixes first, then validate in a second crawl and URL Inspection. REQUIRED OUTPUT Return a table using these exact columns: Template | Root cause | Affected URLs | Severity | Revenue risk | Owner | Acceptance test. For every recommendation, cite the source row, URL, call note, or data point that supports it. Add a confidence column in your analysis: High, Medium, or Low. List missing evidence separately instead of guessing. Finish with a section named HUMAN DECISIONS REQUIRED. RULES - Do not invent search volume, revenue, customer statements, product capabilities, or competitor facts. - Do not treat correlation as causation. - Preserve contradictory evidence and explain why it conflicts. - Do not make the final priority or publishing decision. Prepare the evidence for a human owner. - Use this example only as a format reference, not as evidence: Canonicalizing 600 valid integration pages to the homepage is Critical; a title that is three characters long is Low.
A recrawl proves the shipped fixes. A named human owner must verify this before the lesson is complete.
Canonicalizing 600 valid integration pages to the homepage is Critical; a title that is three characters long is Low.
Separate crawler purpose and access
Search discovery, user-requested retrieval, model training, rendering, and indexing use different controls. Audit each system against the outcome you actually want.
Your robots.txt can lay out a welcome mat while the CDN holds the door shut. Bot-fight mode, WAF rules, and rate limits quietly turn away GPTBot, PerplexityBot, and ClaudeBot, and the rejection never shows up in any dashboard you normally check. Half the sites we audit are doing this without knowing it. The only witness is the server log.
One more trap while you are in there: AI Overviews rides on Googlebot, the same crawler as regular search, so blocking Google-Extended does not remove you from it and allowing it does not put you in. Check each bot by name, from the logs. Then load your pricing page with JavaScript switched off. If the table vanishes, assume the models have never seen your prices.
This whole lesson is about an hour of work. It decides whether the next six months of content spend is visible at all.
ChatGPT search discovery
The file you check. It says welcome.
Bot-fight mode. Visible only in server logs.
What the model saw: nothing.
- 01
Create a control matrix before changing robots.txt: Googlebot for Google Search discovery, rendering, indexing, and AI feature eligibility; OAI-SearchBot for ChatGPT search inclusion; GPTBot for OpenAI training preference; ChatGPT-User for user-initiated page visits; Google-Extended for Gemini training and grounding outside Google Search.
- 02
Check robots.txt, CDN/WAF responses, and verified server-log requests for the automated crawlers you intend to allow. Treat ChatGPT-User separately because user-initiated requests are not automatic crawling and robots.txt may not apply.
- 03
Use Search Console URL Inspection or Rich Results Test to verify Googlebot rendering. For other bots, confirm that the HTTP response contains the essential commercial text; server rendering is a resilience choice, not a universal ranking requirement.
- Create columns for product purpose, user agent or robots token, desired allow/disallow policy, verified IP range, robots result, HTTP status, rendering method, and verification date. Do not put ChatGPT-User in the automatic crawler rows.
- Review 48 hours of CDN or server logs and verify claimed bots against published IP ranges before changing WAF rules. Test representative commercial URLs, not just robots.txt.
- Verify Googlebot with Search Console URL Inspection or Rich Results Test. Separately inspect the initial HTML and a JavaScript-disabled render to understand resilience for fetchers that may not execute your client code.
The crawler control matrix separates product purpose, robots control, rendering test, and desired policy; verified requests receive the intended response and Googlebot rendering is confirmed in a Google testing tool.
- WATCH OUT
User-agent strings can be spoofed. Verify OpenAI crawler traffic against its published IP ranges before drawing conclusions or changing WAF policy.
- NO BUDGET?
No log access at all? Cloudflare's free-plan bot analytics still shows verified bot traffic by name, and a one-line email to your host ('please send 48h of access logs') usually works.
- PRO TIP
Re-run this check after every CDN or security change. One well-meaning WAF rule during a traffic spike has undone months of AEO work before.
OPTIONAL / AI CO-PILOTRun this lesson with AI, the prompt, sources and expected output+
Choose the business priority, interpret exceptions, protect confidential data, challenge weak evidence, and approve the final decision.
Clean, classify, compare, calculate, and draft rows from supplied evidence. AI may surface patterns; it does not own strategy.
Screaming Frog / Sitebulb, plus index evidence sample from GSC URL Inspection.
| User agent | Robots.txt rule | Observed status | Blocking layer | JS-only content | Fix | Verified |
|---|---|---|---|---|---|---|
| PerplexityBot | Allow | 403 | CDN bot-fight mode | Pricing table | Allowlist + SSR pricing | Pending recrawl |
CLAUDE / CHATGPT PROMPTAnalyze the evidence without outsourcing the decision+
Remove personal or confidential data. Attach the named exports, explain every column, and tell the model when the dataset was collected.
You are assisting a human B2B SaaS SEO and AEO operator with Module 02: Fix crawlability, indexation, and site architecture. Current lesson: Separate crawler purpose and access Objective: Search discovery, user-requested retrieval, model training, rendering, and indexing use different controls. Audit each system against the outcome you actually want. Required artifact: Technical fix queue BUSINESS CONTEXT I WILL PROVIDE - Product, category, target market, pricing model, sales motion, and geography - The priority customer segment and the commercial outcome for this 90-day cycle - Internal HTML + issues exports exported from Screaming Frog / Sitebulb - Index evidence sample exported from GSC URL Inspection - Definitions for any internal fields, stages, scores, and abbreviations TASK 1. Create a control matrix before changing robots.txt: Googlebot for Google Search discovery, rendering, indexing, and AI feature eligibility; OAI-SearchBot for ChatGPT search inclusion; GPTBot for OpenAI training preference; ChatGPT-User for user-initiated page visits; Google-Extended for Gemini training and grounding outside Google Search. 2. Check robots.txt, CDN/WAF responses, and verified server-log requests for the automated crawlers you intend to allow. Treat ChatGPT-User separately because user-initiated requests are not automatic crawling and robots.txt may not apply. 3. Use Search Console URL Inspection or Rich Results Test to verify Googlebot rendering. For other bots, confirm that the HTTP response contains the essential commercial text; server rendering is a resilience choice, not a universal ranking requirement. REQUIRED OUTPUT Return a table using these exact columns: User agent | Robots.txt rule | Observed status | Blocking layer | JS-only content | Fix | Verified. For every recommendation, cite the source row, URL, call note, or data point that supports it. Add a confidence column in your analysis: High, Medium, or Low. List missing evidence separately instead of guessing. Finish with a section named HUMAN DECISIONS REQUIRED. RULES - Do not invent search volume, revenue, customer statements, product capabilities, or competitor facts. - Do not treat correlation as causation. - Preserve contradictory evidence and explain why it conflicts. - Do not make the final priority or publishing decision. Prepare the evidence for a human owner. - Use this example only as a format reference, not as evidence: A company may allow OAI-SearchBot for ChatGPT search, disallow GPTBot for training, allow Googlebot for Search and AI features, and disallow Google-Extended without changing Google Search inclusion.
A recrawl proves the shipped fixes. A named human owner must verify this before the lesson is complete.
A company may allow OAI-SearchBot for ChatGPT search, disallow GPTBot for training, allow Googlebot for Search and AI features, and disallow Google-Extended without changing Google Search inclusion.
The screen below is what Google Search Console's indexing report looks like when you run this module. Read it left to right: how many pages Google knows about, how many it chose to index, and, the part everyone skips, the named reasons for every exclusion. Your fix queue comes from those reasons, not from a generic site score.
The workspace below is a worked example of a crawl triage for a fictional 1,248-URL SaaS site. Every number is demo data, but the shape is exactly what your own crawl export should turn into: issues grouped by template, scoped by URL count, and ranked by revenue risk, not by whatever the audit tool put at the top.
Crawl and indexation command center
Turn crawler exports into a prioritized engineering queue tied to revenue templates and verification tests.
Affected URLs by issue family
Count scope, then split by template and commercial importance.
Check each item only after the artifact meets the standard. Progress saves on this device; use the academy-home backup to move it elsewhere.