You Can’t Compare Backlink Counts in SEO Tools: Here’s Why

If you just looked at dashboard numbers for links and RDs in different tools you might see completely different things.

For example, here’s what we count in Ahrefs:

• Live links
• Live RDs
• 6 months of data

In Semrush, here’s what they count:

• Live + dead links
• Live + dead RDs
• 6 months of data + a bit more*

*By a bit more, what I mean is that their data goes back 6 months and to the start of the previous month. So, for instance, if it’s the 15th of the month, they would actually have about 6.5 months of data instead of 6 months of data. If it’s the last week of the month, they may have close to 7 months of data instead of 6.

This may not seem like a lot, but it can increase the numbers shown by a lot, especially when you’re still counting dead links and dead RDs.

I don’t think SEOs want to see a number that includes dead links. I don’t see a good reason to count them, either, other than to have bigger and potentially misleading numbers.

I only say this because I’ve called Semrush out on making this type of biased comparison before on Twitter, but I stopped arguing when I realized that they really didn’t want the comparison to be fair; they just wanted to win the comparison.

There are some ways you can compare the data to get somewhat similar time periods and only look at active links.

If you filter the Semrush backlinks report for “Active” links, you’ll have a somewhat more accurate number to compare against the Ahrefs dashboard number.

Alternatively, if you use the “Show history: Last 6 months” option in the Ahrefs backlink report, this would include lost links and be a fairer comparison to Semrush’s dashboard number.

Here’s an example of how to get more similar data:

• Semrush Dashboard: 5.1K = Ahrefs (6-month **** comparison): 5.6K
• Semrush All Links: 5.1K = Ahrefs (6-month **** comparison): 5.6K
• Semrush Active Links: 2.9K = Ahrefs Dashboard: 3.5K = Ahrefs (no **** comparison): 3.5K

What you should not compare is Semrush Dashboard and Ahrefs Dashboard numbers. The number in Semrush (5.1K) includes dead links. The number in Ahrefs (3.5K) doesn’t; it’s only live links!

Note that the time periods may not be exactly the same as mentioned before because of the extra days in the Semrush data. You could look at what day their data stops and select that exact day in the Ahrefs data to get an even more accurate, but still not quite accurate comparison.

I don’t think the comparison works at all with larger domains because of an issue in Semrush. Here’s what I saw for semrush.com:

• Semrush Dashboard: 48.7M = Ahrefs (6 month **** comparison): 24.7M
• Semrush All Links: 48.7M = Ahrefs (6 month **** comparison): 24.7M
• Semrush Active Links: 1.8M = Ahrefs Dashboard: 15.9M = Ahrefs (no **** comparison): 15.9M

So that’s 1.8M active links in Semrush vs 15.9M active in Ahrefs. But as I said, I don’t think this is a fair comparison. Semrush seems to have an issue with larger sites. There is a warning in Semrush that says, “Due to the size of the analyzed domain, only the most relevant links will be shown.” It’s possible they’re not showing all the links, but this is suspicious because they will show the total for all links which is a larger number, and I can filter those in other ways.

I can also sort normally by the oldest last seen **** and see all the links, but when I do last seen + active, I see only 608K links. I can’t get more than 50k rows in their system to investigate this further, but something is fishy here.

The above comparison wouldn’t be enough to make an accurate comparison. There are still a number of differences and problems that make any sort of comparison troublesome.

This tweet is as relevant as the day I wrote it:

To recap some of the main points, here are some things we do:

• We store some links inserted with JavaScript, no one else does this. We render ~250M pages a day.
• We have a canonicalization system in place that others may not, which means we shouldn’t count as many duplicates as others do.
• Our crawler tries to be intelligent about what to prioritize for crawling to avoid spam and things like infinite crawl paths.
• We count one link per page, others may count multiple links per page.

These differences make a fair link comparison nearly impossible to do.

The easiest way to see the biggest discrepancies in link totals is to go to the Referring Domains reports in the tools and sort by the number of links. You can use the dropdowns to see what kinds of issues each index may have with overcounting some links. In many cases, you’re likely to see millions of links from the same site for some of the reasons mentioned above.

For example, when I looked in Semrush I found blogspot links that they claimed to have recently checked, but these are showing 404 when I visit them. Semrush still counts them for some reason. I saw this issue on multiple domains I checked. This is one of those pages:

Seeing the dead link above counted in the total made me want to check how many dead links were in each index. I ran crawls on the list of the most recent live links in each tool to see how many were actually still live.

For Semrush, 49.6% of the links they said were live were actually dead. Some churn is expected as the web changes, but half the links in 6 months indicates that a lot of these may be on the spammier part of the web that isn’t as stable or they’re not re-crawling the links often. For some context, the same number for Ahrefs came back as 17.2% dead.

Ahrefs recently added a filter for “Best links” which you can configure to filter out noise. For instance, if you want to remove all blogspot.com blogs from the report, you can add a filter for it.

This means you’ll only see links you consider important in the reports. This can also be applied to the main dashboard numbers and charts now. If the filter is active, people will see different numbers depending on their settings.

You would think this is straightforward, but it’s not.

There are a lot of different things you’d have to solve for here:

• The extra days in Semrush’s data that you’ll have to remove or add to the Ahrefs number.
• Remember that Semrush also includes dead RDs in their dashboard numbers. So you need to filter their RD report to just “Active” to get the live ones.
• Remember that half the links in the test of Semrush live data were actually dead, so I would suspect that a number of the RDs are actually lost as well. You could possibly look for domains with low link counts and just crawl the listed links from those to remove most of the dead ones.
• After all that, you’re still going to need to strip the domains down to the root domain only to account for the differences in what each tool may be counting as a domain.

Ahrefs currently shows 206.3M RDs in our database and Semrush shows 1.6B. Domains are being counted in extremely different ways between the tools.

Part of our process is dropping spam domains, and we also treat some subdomains as different domains. We come up close to the numbers from other 3rd party studies for the number of active websites and domains, whereas Semrush seems to come in closer to the total number of websites (including inactive ones).

We’re going to simplify our methodology soon so that one domain is actually just one domain. This is going to make our RD numbers go down, but be more accurate to what people actually consider a domain. It’s also going to make for an even bigger disparity in the numbers between the tools.

Ahrefs crawls 7B+ pages every day. Semrush claims they crawl 25B pages per day. This would be ~3.5x what Ahrefs crawls per day. The problem is that I can’t find any evidence that they crawl that fast.

We saw that around half the links that Semrush had marked as active were actually dead compared to about 17% in Ahrefs, which indicated to me that they may not re-crawl links as often. That and the freshness test both pointed to them crawling slower. I decided to look into it.

Logs of my sites

I checked the logs of some of my sites and sites I have access to, and I didn’t see anything to support the claim that Semrush crawls faster. If you have access to logs of your own site, you should be able to check which bots are crawling the fastest.

I scraped and parsed ~80,000 log file summaries that contained 1 month of data each and were generated in the last couple of years. This sample contained over 9k websites in total.

I did not see evidence of Semrush crawling many times faster than Ahrefs for these sites, as they claim they do. The only bot that was crawling much faster than Ahrefsbot in this dataset was Googlebot. Even other search engines were behind our crawl rate.

That’s just data from a small-ish number of sites compared to the scale of the web. What about for a larger chunk of the web?

Some of the SEO tools share the number of pages they crawl on their websites. The only one in the chart below that doesn’t have a publicly published crawl rate is AhrefsSiteAudit bot, but I asked our team to pull the info for this. Let me put the rankings in perspective with actual and claimed crawl rates.

The math isn’t mathing. How can Semrush claim they’re crawling multiple times as fast as these others, but their ranking is lower? Cloudflare doesn’t cover the entire web, but it’s a large chunk of the web and a more than representative sample size.

When they originally made this 25B claim, I believe they were closer to 90th on Cloudflare Radar, near the bottom of the list at the time. Semrush hasn’t updated this number since then, and I recall a period of time where they were in the 60s-70s on Cloudflare Radar as well. They do seem to be getting faster, but their claimed numbers still don’t add up.

I don’t hear SEOs raving about Moz or Sistrix having the best link data, but they are 21st and 36th on the list respectively. Both are higher than Semrush.

Semrush may be conflating the term pages with links, which is actually mentioned in some of their documentation. I don’t want to link to it, but you can find it with this quote: “Daily, our bot crawls over 25 billion links”. But links are not the same thing as pages and there can be hundreds of links on a single page.

It’s also possible they’re crawling a portion of the web that’s just more spammy and isn’t reflected in the data from either of the sources I looked at. Some of the numbers indicate this may be the case.

Y’all shouldn’t trust studies done by a specific vendor when it compares them to others, even this one. I try to be as fair as I can be and follow the data, but since I work at Ahrefs you can hardly consider me unbiased. Go look at the data yourselves and run your own tests.

Don’t blindly trust the numbers on the dashboards or the general numbers because they may represent completely different things. While there’s no perfect way to compare the data between different tools, you can run many of the checks I showed to try to compare similar things and clean up the data. If something looks off, ask the tool vendors for an explanation.

If there ever comes a time when we stop winning on things like tech and crawl speed, go ahead and switch to another tool and stop paying us. But until that time, I’d be highly skeptical of any claims by other tools.