Fix crawlability, indexation, and site architecture
Find and sequence the technical problems that stop search engines and answer engines from understanding the site.
You will have a prioritized engineering queue, an indexation policy, and a clean path from the homepage to every revenue page.
Crawl and indexation command center
Turn crawler exports into a prioritized engineering queue tied to revenue templates and verification tests.
Affected URLs by issue family
Count scope, then split by template and commercial importance.
Publishing more content on a site with duplicate URLs, weak internal links, or accidental noindex rules compounds waste. Technical SEO is not a giant checklist; it is the removal of specific obstacles between discovery and a useful indexed page.
Create the crawl and index inventory
Compare what exists, what can be crawled, and what Google actually indexed.
- 01
Crawl the site and export status code, canonical, robots directive, title, H1, depth, and indexability.
- 02
Export indexed and excluded URLs from Google Search Console.
- 03
Label each URL Keep, Improve, Merge, Redirect, or Remove based on buyer value and uniqueness.
- 04
Configure: Crawl as desktop, mobile, and rendered JavaScript where the framework requires it.
- 05
Prioritize: Use severity, revenue exposure, URL count, confidence, and effort instead of crawler defaults.
Choose the business priority, interpret exceptions, protect confidential data, challenge weak evidence, and approve the final decision.
Clean, classify, compare, calculate, and draft rows from supplied evidence. AI may surface patterns; it does not own strategy.
Screaming Frog / Sitebulb, plus index evidence sample from GSC URL Inspection.
| Crawler | User agent | Rendering | Scope | Max URLs | Data connected | Run owner |
|---|---|---|---|---|---|---|
| Screaming Frog | Googlebot smartphone | JavaScript | Production host | 20,000 | GSC + GA4 | Technical SEO |
CLAUDE / CHATGPT PROMPTAnalyze the evidence without outsourcing the decision+
Remove personal or confidential data. Attach the named exports, explain every column, and tell the model when the dataset was collected.
You are assisting a human B2B SaaS SEO and AEO operator with Module 02: Fix crawlability, indexation, and site architecture. Current lesson: Create the crawl and index inventory Objective: Compare what exists, what can be crawled, and what Google actually indexed. Required artifact: Technical fix queue BUSINESS CONTEXT I WILL PROVIDE - Product, category, target market, pricing model, sales motion, and geography - The priority customer segment and the commercial outcome for this 90-day cycle - Internal HTML + issues exports exported from Screaming Frog / Sitebulb - Index evidence sample exported from GSC URL Inspection - Definitions for any internal fields, stages, scores, and abbreviations TASK 1. Crawl the site and export status code, canonical, robots directive, title, H1, depth, and indexability. 2. Export indexed and excluded URLs from Google Search Console. 3. Label each URL Keep, Improve, Merge, Redirect, or Remove based on buyer value and uniqueness. 4. Configure: Crawl as desktop, mobile, and rendered JavaScript where the framework requires it. 5. Prioritize: Use severity, revenue exposure, URL count, confidence, and effort instead of crawler defaults. REQUIRED OUTPUT Return a table using these exact columns: Crawler | User agent | Rendering | Scope | Max URLs | Data connected | Run owner. For every recommendation, cite the source row, URL, call note, or data point that supports it. Add a confidence column in your analysis: High, Medium, or Low. List missing evidence separately instead of guessing. Finish with a section named HUMAN DECISIONS REQUIRED. RULES - Do not invent search volume, revenue, customer statements, product capabilities, or competitor facts. - Do not treat correlation as causation. - Preserve contradictory evidence and explain why it conflicts. - Do not make the final priority or publishing decision. Prepare the evidence for a human owner. - Use this example only as a format reference, not as evidence: A parameter URL that duplicates a feature page is “Canonicalize or block”; an old campaign page with backlinks is “Redirect.”
Every URL template has an index rule. A named human owner must verify this before the lesson is complete.
A parameter URL that duplicates a feature page is “Canonicalize or block”; an old campaign page with backlinks is “Redirect.”
Write an indexation policy
Templates should have a default rule, so the same problem does not return with every release.
- 01
List every URL template: product, use case, integration, blog, tag, search, parameter, and utility.
- 02
Choose index/follow, noindex/follow, canonical target, or disallow only after checking whether crawling is needed.
- 03
Document the owner and test for each rule before shipping.
- 04
Reconcile: Compare crawlable, indexable, canonical, sitemap, analytics, backlink, and GSC URL sets.
Choose the business priority, interpret exceptions, protect confidential data, challenge weak evidence, and approve the final decision.
Clean, classify, compare, calculate, and draft rows from supplied evidence. AI may surface patterns; it does not own strategy.
GSC URL Inspection, plus revenue url overlays from Ahrefs / Semrush.
| URL template | Crawl rule | Index rule | Canonical | Sitemap | Minimum value | Owner |
|---|---|---|---|---|---|---|
| /integrations/* | Allow | Conditional | Self | Qualified only | Setup + unique workflow | Platform |
CLAUDE / CHATGPT PROMPTAnalyze the evidence without outsourcing the decision+
Remove personal or confidential data. Attach the named exports, explain every column, and tell the model when the dataset was collected.
You are assisting a human B2B SaaS SEO and AEO operator with Module 02: Fix crawlability, indexation, and site architecture. Current lesson: Write an indexation policy Objective: Templates should have a default rule, so the same problem does not return with every release. Required artifact: Technical fix queue BUSINESS CONTEXT I WILL PROVIDE - Product, category, target market, pricing model, sales motion, and geography - The priority customer segment and the commercial outcome for this 90-day cycle - Index evidence sample exported from GSC URL Inspection - Revenue URL overlays exported from Ahrefs / Semrush - Definitions for any internal fields, stages, scores, and abbreviations TASK 1. List every URL template: product, use case, integration, blog, tag, search, parameter, and utility. 2. Choose index/follow, noindex/follow, canonical target, or disallow only after checking whether crawling is needed. 3. Document the owner and test for each rule before shipping. 4. Reconcile: Compare crawlable, indexable, canonical, sitemap, analytics, backlink, and GSC URL sets. REQUIRED OUTPUT Return a table using these exact columns: URL template | Crawl rule | Index rule | Canonical | Sitemap | Minimum value | Owner. For every recommendation, cite the source row, URL, call note, or data point that supports it. Add a confidence column in your analysis: High, Medium, or Low. List missing evidence separately instead of guessing. Finish with a section named HUMAN DECISIONS REQUIRED. RULES - Do not invent search volume, revenue, customer statements, product capabilities, or competitor facts. - Do not treat correlation as causation. - Preserve contradictory evidence and explain why it conflicts. - Do not make the final priority or publishing decision. Prepare the evidence for a human owner. - Use this example only as a format reference, not as evidence: Integration pages index only when they contain a real workflow, setup detail, and unique value; empty directories remain noindex.
Critical and high issues have owners. A named human owner must verify this before the lesson is complete.
Integration pages index only when they contain a real workflow, setup detail, and unique value; empty directories remain noindex.
Flatten the revenue architecture
Important pages should not be orphaned or buried five clicks deep.
- 01
Draw the path from homepage to category, use-case, comparison, and integration pages.
- 02
Add contextual links from high-authority pages using language that explains the destination.
- 03
Keep priority revenue pages within three meaningful clicks where the site structure allows it.
- 04
Sample: Inspect representative URLs from every revenue and content template manually.
- 05
Specify: Write the expected rule, affected templates, acceptance test, and rollback risk for engineering.
- 06
Ship: Release critical fixes in controlled batches.
Choose the business priority, interpret exceptions, protect confidential data, challenge weak evidence, and approve the final decision.
Clean, classify, compare, calculate, and draft rows from supplied evidence. AI may surface patterns; it does not own strategy.
Ahrefs / Semrush, plus prioritized audit queue from PageOptimized.
| Revenue URL | Click depth | Internal links | Orphan | Source hub | Required anchor | Action |
|---|---|---|---|---|---|---|
| /pricing/ | 4 | 7 | No | /product/ | reporting software pricing | Add nav + contextual links |
CLAUDE / CHATGPT PROMPTAnalyze the evidence without outsourcing the decision+
Remove personal or confidential data. Attach the named exports, explain every column, and tell the model when the dataset was collected.
You are assisting a human B2B SaaS SEO and AEO operator with Module 02: Fix crawlability, indexation, and site architecture. Current lesson: Flatten the revenue architecture Objective: Important pages should not be orphaned or buried five clicks deep. Required artifact: Technical fix queue BUSINESS CONTEXT I WILL PROVIDE - Product, category, target market, pricing model, sales motion, and geography - The priority customer segment and the commercial outcome for this 90-day cycle - Revenue URL overlays exported from Ahrefs / Semrush - Prioritized audit queue exported from PageOptimized - Definitions for any internal fields, stages, scores, and abbreviations TASK 1. Draw the path from homepage to category, use-case, comparison, and integration pages. 2. Add contextual links from high-authority pages using language that explains the destination. 3. Keep priority revenue pages within three meaningful clicks where the site structure allows it. 4. Sample: Inspect representative URLs from every revenue and content template manually. 5. Specify: Write the expected rule, affected templates, acceptance test, and rollback risk for engineering. 6. Ship: Release critical fixes in controlled batches. REQUIRED OUTPUT Return a table using these exact columns: Revenue URL | Click depth | Internal links | Orphan | Source hub | Required anchor | Action. For every recommendation, cite the source row, URL, call note, or data point that supports it. Add a confidence column in your analysis: High, Medium, or Low. List missing evidence separately instead of guessing. Finish with a section named HUMAN DECISIONS REQUIRED. RULES - Do not invent search volume, revenue, customer statements, product capabilities, or competitor facts. - Do not treat correlation as causation. - Preserve contradictory evidence and explain why it conflicts. - Do not make the final priority or publishing decision. Prepare the evidence for a human owner. - Use this example only as a format reference, not as evidence: A reporting guide links to “automated client reporting software,” then the product page links to relevant integrations and comparisons.
Revenue pages are within a clear link path. A named human owner must verify this before the lesson is complete.
A reporting guide links to “automated client reporting software,” then the product page links to relevant integrations and comparisons.
Prioritize fixes by impact
Do not let minor metadata warnings outrank blocked pages or broken templates.
- 01
Set severity: Critical blocks discovery/indexing; High affects a template or revenue group; Medium reduces clarity; Low is polish.
- 02
Add affected URL count, traffic/pipeline value, effort, owner, and verification test.
- 03
Ship critical template fixes first, then validate in a second crawl and URL Inspection.
- 04
Group: Consolidate issue rows into template-level root causes.
- 05
Prove: Recrawl, inspect live HTML, and monitor indexation and traffic until the expected state is stable.
Choose the business priority, interpret exceptions, protect confidential data, challenge weak evidence, and approve the final decision.
Clean, classify, compare, calculate, and draft rows from supplied evidence. AI may surface patterns; it does not own strategy.
PageOptimized, plus internal html + issues exports from Screaming Frog / Sitebulb.
| Template | Root cause | Affected URLs | Severity | Revenue risk | Owner | Acceptance test |
|---|---|---|---|---|---|---|
| /compare/* | Missing from sitemap | 30 | High | High | Platform | Included and indexable |
CLAUDE / CHATGPT PROMPTAnalyze the evidence without outsourcing the decision+
Remove personal or confidential data. Attach the named exports, explain every column, and tell the model when the dataset was collected.
You are assisting a human B2B SaaS SEO and AEO operator with Module 02: Fix crawlability, indexation, and site architecture. Current lesson: Prioritize fixes by impact Objective: Do not let minor metadata warnings outrank blocked pages or broken templates. Required artifact: Technical fix queue BUSINESS CONTEXT I WILL PROVIDE - Product, category, target market, pricing model, sales motion, and geography - The priority customer segment and the commercial outcome for this 90-day cycle - Prioritized audit queue exported from PageOptimized - Internal HTML + issues exports exported from Screaming Frog / Sitebulb - Definitions for any internal fields, stages, scores, and abbreviations TASK 1. Set severity: Critical blocks discovery/indexing; High affects a template or revenue group; Medium reduces clarity; Low is polish. 2. Add affected URL count, traffic/pipeline value, effort, owner, and verification test. 3. Ship critical template fixes first, then validate in a second crawl and URL Inspection. 4. Group: Consolidate issue rows into template-level root causes. 5. Prove: Recrawl, inspect live HTML, and monitor indexation and traffic until the expected state is stable. REQUIRED OUTPUT Return a table using these exact columns: Template | Root cause | Affected URLs | Severity | Revenue risk | Owner | Acceptance test. For every recommendation, cite the source row, URL, call note, or data point that supports it. Add a confidence column in your analysis: High, Medium, or Low. List missing evidence separately instead of guessing. Finish with a section named HUMAN DECISIONS REQUIRED. RULES - Do not invent search volume, revenue, customer statements, product capabilities, or competitor facts. - Do not treat correlation as causation. - Preserve contradictory evidence and explain why it conflicts. - Do not make the final priority or publishing decision. Prepare the evidence for a human owner. - Use this example only as a format reference, not as evidence: Canonicalizing 600 valid integration pages to the homepage is Critical; a title that is three characters long is Low.
A recrawl proves the shipped fixes. A named human owner must verify this before the lesson is complete.
Canonicalizing 600 valid integration pages to the homepage is Critical; a title that is three characters long is Low.
Check each item only after the artifact meets the standard.