Technical SEO, crawling and indexing
Technical SEO covers the infrastructure that lets search engines crawl, render and index a site reliably: correct HTTP status codes and redirects, a fast and mobile-friendly experience, HTTPS, JavaScript that renders the important content, sensible URL structures, international targeting and valid structured data.
Which status codes matter?
| Code | Meaning for Google |
|---|---|
| 200 | Content can be considered for indexing |
| 301 / 308 | Permanent redirect: a strong signal that the target is canonical[1] |
| 302 / 307 | Temporary redirect: a weak canonical signal[1] |
| 404 / 410 | Page not found or gone; drops out of the index over time[2] |
| 429, 5xx | Google temporarily slows crawling; prolonged errors can lead to removal[2] |
What is page experience?
Google’s page experience guidance covers Core Web Vitals, HTTPS, mobile usability and avoiding intrusive interstitials.[3] Core Web Vitals measure loading (Largest Contentful Paint, good at 2.5 seconds or less), interactivity (Interaction to Next Paint, good at 200 milliseconds or less) and visual stability (Cumulative Layout Shift, good at 0.1 or less).[4] Officially documented Google says good Core Web Vitals scores alone do not guarantee top rankings.[3]
How does JavaScript affect SEO?
Google processes JavaScript in three phases (crawling, rendering and indexing) and can render most modern pages, but rendering can be delayed and some patterns fail.[5] Officially documented Structured data can be generated with JavaScript and Google can read it after rendering,[6] but many other crawlers do not run JavaScript, so server-rendered HTML and JSON-LD are the safer choice. Practitioner practice
What else belongs in a technical audit?
- HTTPS everywhere, with HTTP and
wwwvariants redirecting to one host. - Mobile-first parity: the same content and structured data on mobile and desktop.[7]
- Readable URL structure.[8]
hreflangfor language or regional versions.[9]- Valid structured data, tested with the Rich Results Test and the Schema Markup Validator.[10][11]
- A WebSite
nameon the homepage so Google can show your site name.[12]
Why does technical SEO matter for entities and AEO?
Your structured data and facts are only useful if crawlers can fetch and parse them. Answer engines add their own crawlers; see entities in AI search.
Crawling
Crawling is the process in which search engine bots such as Googlebot discover URLs and download their content. Pages are found mainly through links and sitemaps; robots.txt controls which paths crawlers may request. If an important page cannot be crawled, it cannot be indexed, and nothing on it (including entity markup) will count.
How do crawlers find pages?
Google finds most new pages by following links from pages it already knows, and from sitemaps site owners submit.[13][14] Google can generally only follow links that are <a> elements with an href attribute.[15] Officially documented
What is Googlebot?
Googlebot is the name of Google’s web crawler, which has smartphone and desktop variants.[16] Because of mobile-first indexing, Google uses the mobile version of a site’s content, crawled with the smartphone agent, for indexing and ranking.[7] Officially documented
What does robots.txt do?
A robots.txt file tells crawlers which URLs they may request. The format is standardized as RFC 9309.[17] Google warns that robots.txt is used mainly to avoid overloading a site and “is not a mechanism for keeping a web page out of Google”; use noindex or password protection instead.[18] Officially documented
User-agent: * Disallow: /cart/ Sitemap: https://www.example.com/sitemap.xml
AI crawlers use their own user agents, and robots.txt rules apply per user agent, so check each crawler’s documentation for the name to target.
What is crawl budget?
Crawl budget is the time and resources Google spends crawling a site. Google’s guide is intended mainly for large sites (1 million+ unique pages changing about weekly) or sites with 10,000+ pages changing daily.[19] Officially documented Most small sites do not need to worry about it.
How do you fix crawl problems?
- Link every important page from at least one other crawlable page.
- Keep an XML sitemap of canonical URLs only.[14]
- Check that robots.txt does not block CSS, JavaScript or key sections.
- Return correct status codes; persistent 5xx errors slow Google’s crawling.[2]
- Test a URL with Search Console’s URL Inspection tool.[20]
Next step: indexing.
Indexing
Indexing is the stage where a search engine analyses a crawled page and decides whether and how to store it. Google groups duplicate pages, picks one canonical URL to show, and may skip pages it considers low value. You influence this with redirects, rel="canonical", sitemaps and noindex, and you check it in Search Console’s Page indexing report.
What happens during indexing?
Google analyses the text, images and video on a page, works out whether it duplicates another page, and stores information about the canonical version in its index.[13] Officially documented Google states that it does not guarantee to crawl, index or serve any page.[13]
What is a canonical URL?
The canonical is the URL Google treats as the representative of a set of duplicate pages. Google lists the methods in order of strength: redirects are “a strong signal”, rel="canonical" link annotations are a strong signal, and sitemap inclusion is “a weak signal”.[21] Officially documented
<link rel="canonical" href="https://www.example.com/services/roof-repair">
How do you keep a page out of the index?
Add <meta name="robots" content="noindex"> or an X-Robots-Tag header. For the rule to work, the page must not be blocked by robots.txt, otherwise the crawler never sees it; Google also does not support noindex inside robots.txt.[22][23] Officially documented
Why is a page crawled but not indexed?
| Cause | Fix |
|---|---|
| Duplicate of another URL | Consolidate with a redirect or canonical |
| Thin or unhelpful content | Improve or merge the page ([24]) |
| Accidental noindex | Remove the rule; request reindexing |
| Soft 404 or error status | Return 200 for real content, 404/410 for removed pages ([2]) |
| Content only visible after JavaScript fails | Server-render key content ([5]) |
Search Console’s Page indexing report shows which URLs are indexed and the reason others are not.[25] Officially documented
Why does indexing matter for entities?
Your entity home, About page, author profiles and contact page should all be indexed, canonical and consistent, because these are the pages that state your core facts. The same pages are what answer engines retrieve.
Frequently asked questions
Is a 301 or 308 redirect better?
Google treats both as permanent redirects. Use whichever your server supports; 308 preserves the request method.
Do Core Web Vitals decide rankings?
They are part of page experience, but Google says good Core Web Vitals scores do not guarantee top rankings and that it seeks to show the most relevant content.
Can Google read JSON-LD added with JavaScript?
Yes, after rendering, according to Google. Other crawlers may not render JavaScript, so server-side output is safer.
Does blocking a page in robots.txt remove it from Google?
No. Google says robots.txt is not a mechanism for keeping a page out of Google. A blocked URL can still be indexed from links. Use noindex, and do not block the page, so Google can see the rule.
Do I need an XML sitemap?
Not always, but Google recommends one for large sites, new sites with few links and sites with rich media. Small, well-linked sites can be crawled without one.
What is crawl budget?
The amount of crawling Google does on a site. It matters mainly for very large or rapidly changing sites.
Is rel=canonical a directive?
Google treats it as a strong signal, not an absolute command. Redirects are a stronger signal.
How do I check if a page is indexed?
Use the URL Inspection tool or the Page indexing report in Google Search Console.
Should I noindex thin pages?
Improving or merging them is usually better. Use noindex for pages that should exist for users but not appear in search, such as internal search results.
See also
References
Pages accessed September 29, 2026 unless a date is given. See all sources and our editorial policy.
- ^ "Redirects and Google Search". Google Search Central.
- ^ "How HTTP status codes, and network and DNS errors affect Google Search". Google Search Central.
- ^ "Understanding page experience in Google Search results". Google Search Central.
- ^ "Web Vitals". web.dev (Google Chrome team).
- ^ "Understand JavaScript SEO basics". Google Search Central.
- ^ "Generate structured data with JavaScript". Google Search Central.
- ^ "Mobile site and mobile-first indexing best practices". Google Search Central.
- ^ "URL structure best practices for Google Search". Google Search Central.
- ^ "Tell Google about localized versions of your page". Google Search Central.
- ^ "Rich Results Test". Google Search.
- ^ "Schema Markup Validator". Schema.org.
- ^ "Provide a site name to Google Search". Google Search Central.
- ^ "In-depth guide to how Google Search works". Google Search Central.
- ^ "Build and submit a sitemap". Google Search Central.
- ^ "Link best practices for Google". Google Search Central.
- ^ "What is Googlebot". Google Search Central.
- ^ "RFC 9309: Robots Exclusion Protocol". IETF (RFC Editor). Published September 2022.
- ^ "Introduction to robots.txt". Google Search Central.
- ^ "Crawl budget management for large sites". Google Search Central.
- ^ "URL Inspection tool". Search Console Help.
- ^ "How to specify a canonical URL with rel="canonical" and other methods". Google Search Central.
- ^ "Block Search indexing with noindex". Google Search Central.
- ^ "Robots meta tag, data-nosnippet, and X-Robots-Tag specifications". Google Search Central.
- ^ "Creating helpful, reliable, people-first content". Google Search Central.
- ^ "Page indexing report". Search Console Help.