How Google Crawls and Indexes Websites: The Technical Process Explained

Stage One: Discovery and Crawling

Google’s in-depth guide to how Search works starts with the problem of discovery: “There isn’t a central registry of all web pages, so Google must constantly look for new and updated pages”. URLs are found three ways: pages Google has visited before, links from known pages to new ones, and sitemaps you submit.

Discovery is not a promise of a visit. Google says it “may visit (or ‘crawl’) the page”, that Googlebot uses “an algorithmic process to determine which sites to crawl, how often, and how many pages to fetch from each site”, and that it “doesn’t crawl all the pages it discovered.” Some URLs are disallowed by robots.txt, and some can’t be reached without logging in.

Crawl speed responds to your server. Google says its crawlers “try not to crawl the site too fast to avoid overloading it”, based on the site’s responses, and that “HTTP 500 errors mean ‘slow down'”. The crawl budget guide calls this ceiling the crawl capacity limit, “also known as hostload”: it rises when the site responds consistently and falls when the site slows down or “responds with server errors (5xx HTTP status codes) or rate-limiting signals (such as HTTP 429)”.

Stage Two: Rendering

Google renders pages before it indexes them. The Search guide says that during the crawl, Google “renders the page and runs any JavaScript it finds using a recent version of Chrome”. Google’s JavaScript SEO basics describes the queue:

• “Googlebot queues all pages with a 200 HTTP status code for rendering, unless a robots meta tag or header tells Google not to index the page.”
• “The page may stay on this queue for a few seconds, but it can take longer than that.”
• After rendering, “Googlebot parses the rendered HTML for links again and queues the URLs it finds for crawling. Google also uses the rendered HTML to index the page.”

So JavaScript content can be indexed, but it depends on rendering succeeding. The same documentation notes that “Google Search won’t render JavaScript from blocked files or on blocked pages”, so scripts and API endpoints the page needs must not be disallowed in robots.txt. Content that only appears after a click or a login, or a script that fails in Google’s renderer, leaves the rendered page without it. Serving important content and links in the initial HTML, through server-side rendering or static generation, removes the dependency. The details are in JavaScript SEO.

Stage Three: Indexing and Canonical Selection

Indexing is where Google decides what the page is about and whether to store it. Google says this stage “includes processing and analyzing the textual content and key content tags and attributes, such as <title> elements and alt attributes, images, videos, and more”.

Canonicalization happens here too. Google groups pages with similar content and selects “the one that’s most representative of the group” as the canonical, “the page that may be shown in search results.” The others are alternates that may be shown in specific contexts. If your preferred URL isn’t the one Google picks, canonical tags and consistent internal linking are the signals to fix.

Google lists some common indexing issues: “The quality of the content on page is low”, “Robots meta rules disallow indexing”, and “The design of the website might make indexing difficult”.

Reading the Difference in Search Console

The Page indexing report separates the stages, and two of its statuses are easy to confuse:

The first is a crawling problem; the second is an indexing decision. Treating one like the other wastes the fix.

Two other reports complete the picture:

• The robots.txt report “shows which robots.txt files Google found for the top 20 hosts on your site, the last time they were crawled, and any warnings or errors encountered”, and lets you request a recrawl of a robots.txt file in an emergency. It replaced the old robots.txt Tester; test individual URLs with URL Inspection or a robots.txt parser.
• The Crawl Stats report shows total crawl requests, download size, response time and host status, including whether Google “encountered at least one significant crawl availability issue in the last 90 days”.

For the raw record of every Googlebot request, server logs go further; see log file analysis.

Crawl Budget: Who Needs to Think About It

Google’s crawl budget guide is written for a specific audience, and its own size bands are a good test of whether you belong in it:

• “Large sites (1 million+ unique pages) with content that changes moderately often (once a week)”
• “Medium or larger sites (10,000+ unique pages) with very rapidly changing content (daily)”
• “Sites with a large portion of their total URLs classified by Search Console as Discovered – currently not indexed”

Google adds: “The numbers given here are a rough estimate to help you classify your site. These are not exact thresholds.” For a site outside those bands, look at quality, duplication and technical causes before blaming crawl budget.

For sites inside them, the work is removing waste rather than asking for more crawling: consolidate duplicate URLs, control faceted navigation, fix redirect chains, and keep important pages linked and in the sitemap. Two of Google’s own warnings are worth repeating: “Don’t use noindex, as Google will still request, but then drop the page”, and “Don’t use robots.txt to temporarily reallocate crawl budget for other pages”.

Crawl and Index Blockers, and How Google Treats Them

• robots.txt rules. They block crawling, not indexing. Google’s robots.txt introduction says a disallowed URL can still be indexed “if it is linked from other places on the web”, with the URL and anchor text appearing in results. To keep a page out of results, use noindex on a crawlable page or password protection. Syntax and pitfalls are in robots.txt syntax and common mistakes.
• Server errors. Persistent 5xx and 429 responses lower the crawl capacity limit.
• Redirect chains. Google’s HTTP status code documentation says “By default, Google’s crawlers follow up to 10 redirect hops”, and that content from the redirecting URL is ignored in favor of the final target. Chains below that limit still waste requests and add delay, so point every redirect at its final URL; see redirect chains and loops.
• Leftover noindex. A staging setting or plugin option can keep live pages out of the index. URL Inspection shows the indexing status Google saw.

Getting New Content Found Quickly

For new or updated articles, the levers are ordinary: link them from pages Google already crawls and indexes, such as the homepage and section pages, which matches Google’s own description of discovery through links from known pages; keep lastmod in the sitemap accurate and change it only when the content changes, as covered in XML sitemaps; and use URL Inspection’s indexing request for a small number of priority URLs.

Google’s Indexing API is not one of those levers. Google’s Indexing API documentation says it “can only be used to crawl pages with either JobPosting or BroadcastEvent embedded in a VideoObject”, so it applies to job postings and livestream pages, not articles or product pages; job posting markup is covered in JobPosting schema.

When a page isn’t showing up, work through the stages in order: was it discovered, was it crawled, did it render, was it indexed, and which URL did Google pick as canonical. The Search Console status tells you which stage to fix.