All guides

Question

Crawl Analysis Tools: Identifying Technical Issues

3 min read

Short answer

A crawl analysis tool requests your pages the way a search engine bot does, follows links from a starting point, and records what it finds for every URL, so technical problems show up by template instead of one page at a time. The tools split into desktop crawlers that run on your computer, cloud crawlers that run on the vendor's servers, and audit modules inside SEO suites. Choose by site size, how often you need to crawl, who needs the data, and whether you want server logs alongside the crawl. Whichever you use, a crawl shows what a bot could reach and see; whether Google indexed it is a question for Search Console.

Sources

  1. 1developers.google.com

Ask next

What a Crawl Finds

Issue class What the crawl shows
Response codes 4xx and 5xx URLs and the pages linking to them
Redirects Chains, loops, redirects to errors
Canonicals and directives Canonicals to redirected or non-indexable URLs, <!–INLINECODE0–> on pages that should rank
Rendering Content or links that appear after JavaScript runs
Internal links and depth Pages far from the start URL or with few internal links
On-page elements Missing or duplicate titles and headings by template

Each class has its own guide: redirect chains and loops, canonical tags, JavaScript SEO and internal linking strategy.

Redirect chains are worth fixing because each hop is another request for users and crawlers; Google’s crawlers “follow up to 10 redirect hops” by default, per its HTTP status code documentation, and a chain that’s longer, or ends in an error, can leave the final URL unreached.

Three Kinds of Tool

Desktop crawler Cloud crawler Suite audit module
Runs on Your computer The vendor's servers The vendor's servers
Control Full control of settings and data Settings through the platform Fewer settings
Large sites Limited by your machine Built for scale Varies by plan
Scheduling Needs your machine running Built in Built in
Sharing Export files Shared projects Shared projects
Logs Separate tools Part of some platforms Check the suite

Examples, described by their makers:

  • Screaming Frog SEO Spider: a desktop crawler with a free version limited to 500 URLs and a paid license; see Screaming Frog SEO Spider.
  • Sitebulb: its site offers both a desktop install “for local crawling” and browser access “for scalable cloud crawling”.
  • Lumar: the platform formerly called Deepcrawl, offering to “Crawl & analyze your website, at any scale.”
  • Oncrawl: a cloud platform that “collects, cross-references, and analyzes massive volumes of crawl, log, and performance data”.
  • Botify: an enterprise platform covering crawl and log data.
  • Suite audits: Site Audit in Ahrefs and Semrush, and Site Crawl in Moz Pro.

Choosing a Tool

Question Points toward
Is the site too large for one machine's memory? Cloud crawler
Do you need scheduled crawls without a machine left running? Cloud crawler or suite audit
Do you need full control of rendering, extraction and headers? Desktop crawler
Must crawl data stay on your own systems? Desktop crawler
Do several people need the same project? Cloud crawler or suite audit
Do you want server logs next to the crawl? Cloud platform with log analysis
Do you already pay for a suite? Start with its audit module

Test before committing: crawl the same section with two tools and compare URL counts, status codes and rendered content. Check differences against the settings (rendering, robots.txt, user-agent, scope) before concluding which tool is right.

Crawls and Logs

A crawl shows what a bot could find by following links. Server logs show which URLs Googlebot actually requested, how often and with what response. Together they answer questions neither answers alone, such as important pages Googlebot hasn’t requested in the log period, or URLs it requests that your crawl never reached. The method is in log file analysis.

Running Crawls That Lead to Fixes

  1. Set the scope and settings to match how Google sees the site: rendering on if content depends on JavaScript, robots.txt respected, the smartphone user-agent where the tool offers it.
  2. Add sitemap and Search Console URLs so pages without internal links are included.
  3. Group issues by template and count affected URLs.
  4. Prioritize indexable pages with traffic, then pages that should rank and can’t.
  5. Recrawl the affected URLs after the fix, and confirm in Search Console’s URL Inspection and Page indexing report.

Count a finding as fixed only when a recrawl no longer shows it and Search Console’s URL Inspection shows that Google has seen the changed page.

Link profile

Links to 9 guides

Linked from 8 guides

Cites 1 source

  1. HTTP status code documentation developers.google.com