The Traffalyzer crawler

If a visitor called Traffalyzer shows up in your server log, that was our crawler. This page explains who it is, when it comes, and how to keep it off individual pages.

How it identifies itself

The crawler sends this user agent with every request:

Traffalyzer/1.0 SEO Analyzer (+https://traffalyzer.com/bot)

For the preliminary analysis of websites that do not have a Traffalyzer account yet, there is a second identifier. It requests only a handful of pages and waits at least two seconds between requests:

Traffalyzer-Prospect/1.0 (+https://traffalyzer.com/bot)

When it visits a website

The crawler does not roam the web continuously. It visits a website when someone starts a scan for it: the owner in the dashboard, a visitor via the free check on traffalyzer.com, or the regular scan of a registered domain. It fetches up to five pages at a time, follows internal links and the sitemap, and checks at most 2,000 pages per scan.

Excluding pages via robots.txt

The crawler reads the domain's robots.txt and skips everything that is blocked for it there. The name Traffalyzer is enough as the identifier in robots.txt. This excludes two sections from the scan only, without changing anything for Google:

User-agent: Traffalyzer
Disallow: /intern/
Disallow: /warenkorb

Rules for all bots (User-agent: *) apply as well. Whatever you block there, we do not look at.
The crawler always checks the home page, even with a blanket Disallow: /. Otherwise the result would be an empty scan that helps nobody.
A change takes effect from the next scan. You can start one in the dashboard at any time.

Questions or trouble

If the crawler does something it should not, write to info@traffalyzer.com with the address in question. We will look into it right away.