Definition:
Screaming Frog SEO Spider is a desktop application that crawls websites and collects technical information to support SEO audits. It allows users to review links, server responses, titles, meta descriptions, headings, images and directives that affect crawling and indexing.
It is commonly known simply as Screaming Frog, although that is also the name of the company that develops it. The tool organises crawl data to locate issues and analyse a website’s structure.
Table of contents
Origins of Screaming Frog
Screaming Frog is a British company founded in 2010 by Dan Sharp, Graeme Radford and Jason Lock. Development of SEO Spider began that year as a tool for collecting technical data and supporting optimisation recommendations.
The company’s name has an unusual origin: its founders explain that it was inspired by a frog that defended itself against two cats in Dan Sharp’s garden.
How SEO Spider works
SEO Spider is installed on a computer and sends requests to a website’s pages. Starting from an initial address, it discovers links and visits the URLs permitted by the crawl configuration. It can also analyse a list of addresses supplied by the user.
During this process, it records server responses and extracts information from pages. Results are displayed in tables that allow users to filter issues, check which pages link to a URL and export the data.
When a website generates content through JavaScript, the tool can use an integrated browser to render pages before analysing them. Choosing between examining the received HTML and the rendered content affects what it can detect.
The scope depends on the configuration: exclusions, limits, authentication or access restrictions can leave parts of the site outside the analysis.
What Screaming Frog is used for
In an SEO audit, crawling makes it possible to review numerous elements together. Its main uses include the following:
- Finding broken links and errors: locating URLs with 4XX or 5XX responses and the pages that link to them.
- Reviewing redirects: identifying destinations, chains and loops, particularly during a migration.
- Analysing titles and meta descriptions: finding missing or repeated elements and reviewing their content.
- Checking directives: examining
robots.txt,noindextags, canonical URLs andhreflangannotations. - Studying internal linking: understanding how pages connect and how many links the crawler needs to follow to reach them.
- Reviewing content and resources: detecting duplication, examining headings and images, and extracting specific data from the HTML.
It can also generate XML sitemaps and compare crawls to check what has changed. The availability of certain features depends on the licence and configuration used.
Integrations with other data sources
Screaming Frog can incorporate information from Google Search Console, Google Analytics and PageSpeed Insights through its integrations.
This makes it possible to relate a URL’s technical data to its search clicks, traffic or particular performance indicators. Connections require the appropriate permissions and do not replace the configuration of those tools.
Combining sources also helps locate pages that have not been discovered by following internal links. Their absence from a crawl does not, on its own, prove that they are orphan pages: exclusions or limitations of the analysis must first be ruled out.
Free version and paid licence
SEO Spider offers a free version with a URL limit per crawl and a paid licence that provides additional features and removes that licence limit.
The practical capacity to analyse a site also depends on the computer’s memory, storage and resources. The size of the website and the information to be collected determine which configuration is suitable.
How to interpret its results
A detected issue does not always mean there is an error that needs fixing. A redirect may be intentional, and a page with noindex may be correctly excluded from search engines. Priority depends on each URL’s purpose and the site’s goals.
Similarly, Screaming Frog classifying a page as indexable does not confirm that Google has indexed it. The tool evaluates the signals it finds; its status in the search engine should be checked against Search Console.
Crawling provides a technical basis for diagnosis, but it does not automatically modify the website or determine on its own which changes will improve its rankings.