A 5-step approach to optimizing websites with millions of pages

You’ve been tasked to optimize a site that is so massive it makes the Great Wall of China look like a picket fence. And it’s your sole responsibility. You need to make a big SEO impact with fewer resources. What do you do?

In this article, I’ll share five approaches you can focus on when you’re optimizing massive websites. 

By the end, you’ll feel confident knowing that you have a battle plan to conquer that great wall of webpages that lie ahead:

• Sharpen your ******, then go to battle.
• Focus on global changes.
• Improve the top-performing pages.
• Focus on pages that target head terms.
• Clean up the content.

1. Sharpen your ******, then go to battle

It’s useful to have a certain mindset about your SEO program before you start. 

That said, two key steps will help you create an SEO game plan: 

• Site analysis.
• Competitive analysis. 

Let’s look at those in more detail.

Site analysis

In the battlefield of the search engine results pages, your website is the ******. 

So your first mission: Sharpen that ****** by making the site the best it can be so that you’re not your own worst enemy. 

A site analysis will uncover the things that are hindering your rankings. 

You can do this in a couple of ways:

• Run some SEO diagnostic tools.
• Get an SEO audit.

Diagnostic tools

The average SEO tool is meant to repair your site, not rank it. In other words, it will not tell you how to rank but what you need to fix based on best practices.

Most businesses use diagnostic tools to do a self-audit. While it can uncover some useful information, there are a couple of limitations:

• The recommendations are based on best practices and are surface-level suggestions. They don’t take into account the unique needs of the business. They do not know your competitors. They do not know the nuances of your particular industry niche. 
• Some people will not know how to discern the things that need to be done and the things that can be left alone, nor will they know how to prioritize the recommendations based on the unique situations of their websites.

For example, let’s say you’re seeing a drop in organic search traffic. The diagnostic tool will not be able to understand what factors contribute to that – is it due to an algorithm update? Did your site recently migrate? Is your content bad?  

So, it is the diagnostic tool operator’s job (a.k.a. you) to be able to take the data and apply wisdom to ensure you are focusing on the most impactful things.  

Some tools you might consider for diagnosing your site include:

• Nibbler
• SEOptimer
• UpCity’s SEO Report Card
• WebPageTest.org
• Semrush
• SEOToolSet (Disclaimer: This is my company’s software)

SEO audits

Nothing beats the strategies you can uncover from a professional-level SEO audit that marries diagnostic tools with expert wisdom from an SEO vendor. 

In addition to running the best tools, an SEO expert conducting a site audit can:

• Get to know your business, its site, past SEO decisions and your goals, and then take all of that into consideration when diagnosing your site and creating a strategy.
• Uncover the real reasons you are not performing well in the search results using expert, technical analyses. 
• Prioritize the recommendations so that you know exactly which steps to take in which order to make the strongest impact.

Competitive analysis

Once you’ve sharpened your tool, it’s time to go to the battlefield in the SERPs, where your second mission is to beat the competition (not the algorithm). 

Remember, you cannot attempt to beat an infinitely large algorithm. 

By focusing on what your competition is doing, and trying to do as well as them or better, you set the stage for a more manageable battle.  

Here again, you will rely on tools to help you uncover your competition’s strategy. 

You are looking to find things like who your true competitors are, and recommendations on optimizing your webpages based on the top-ranked pages in the SERPs.

Some tools you might consider for competitive research include:

• Semrush’s Traffic Analytic and Organic Research tool 
• Moz’s Domain Authority Checker
• Ahrefs Content Gap and Site Explorer tools
• SEOToolSet’s Single Page Analyzer, Research Summary report, Multi Page Analyzer, Links Report and Site Checker
• Bruce Clay SEO WP plugin (Disclaimer: This is also my company’s tool)

By understanding your site and your competition, you can enter the battlefield with a clearly defined plan. This makes managing a large site’s SEO less overwhelming.

2. Focus on global changes

When you have a massive website, every change needs to count. That means looking at the things that can elevate the entire site or key pages across the site. 

One of the obvious wins here is improving your website performance (and your content, but more on that later). 

You can tackle things like:

• Website speed
• Core web vitals
• Mobile-friendliness  
• Code optimization 

Website speed

Website speed can impact many things: The user experience, your website rankings and conversions … so this is something to take seriously. 

Google conducted some research on speed and its impact on users back in 2017 (highlighted below), and while the data is a bit old, the message still holds.

As you look at your site, you may find that some of your pages may load slower than others. If that happens on your most important pages, you want to start there. 

And you may have groups of pages with the same template where making a few changes can improve speed across all those types of pages. 

Some things you may need to address are:

• Compressing the information between your web server and search engine browser.
• Minifying JavaScript, and cleaning up and externalizing CSS code.
• Choosing the best file formats for images.

Some helpful tools and resources for this include:

(Note that, technically, the old page speed algorithm has been replaced by core web vitals, which I’ll cover next.)

Core web vitals

Core web vitals focus on factors that create a good user experience on your website, this includes:

• Largest contentful paint: Having an optimal webpage load time.
• First input delay: Ensuring users can easily interact with the page quickly. 
• Cumulative layout shift: Making sure the elements on a webpage are stable.

Google says that “if at least 75 percent of page views to a site meet the ‘good’ threshold, the site is classified as having ‘good’ performance for that metric.”

This most likely only matters to the top-performing pages on the site, though, not the entire site, as pointed out in this Google SEO office hours: 

However, Google has noted that some sites may not need to dedicate too many resources to core web vitals, or obsess about the scores:

IMO most people don’t need to do anything different. This is not a change in how ranking works, it’s more about how we’ve been thinking about these elements. We’ve seen people hyper-focus on these numbers, that’s not a good use of time & energy. Think holistically instead.

— John Mueller (official) · #StaplerLife (@JohnMu) April 22, 2023

Several tools offer lab and field data that can help you understand core web vitals:

• PageSpeed Insights
• Search Console (Core Web Vitals report)
• Web Vitals Extension 
• Chrome UX Report 
• Lighthouse 
• Chrome User Experience Report
• Firebase Performance Monitoring
• Chrome DevTools 
• WebPageTest

So look at the most critical pages on the site, and run some tests to see what can be improved.

Mobile-friendliness

Ensuring your site is mobile-friendly is a no-brainer in today’s mobile-centric world. Of course, there’s also Google’s mobile-first indexing.

There is more to consider than you may think in order to have a mobile-friendly site. In general, you will want to think about:

• The mobile website configuration (for example, is it responsive?).
• How you prioritize the content for mobile users and make the content mobile friendly. 
• Mobile-friendly design.

Tools that can help you along the way include:

Code optimization

Because the code on your website can impact a search engine’s ability to crawl your website, you can consider optimizing that code. 

Again, you can start with certain page templates or key page types and go from there.

Checking out W3.org’s developer tools can be helpful. 

But a warning for beginners: You can spend a lot of time and resources trying to chase perfect code and the effort doesn’t always match the outcome. 

So it’s wise to know how to prioritize the recommendations, and you may need expert help from a developer who also understands SEO or vice versa.

---

---

3. Improve the top-performing pages 

Another area to laser focus your SEO efforts when you are contending with a large website is the top-performing pages. 

Typically, a handful of webpages drive the majority of organic traffic to a website. Look at those first.

This is consistent with an Ahrefs study which found that out of billions of webpages, only about 4% of them received any meaningful traffic at all.

In other words: If you have to choose between optimizing a product page that drives a lot of traffic to your site versus a lesser-known product page featuring a left-handed socket wrench with a light on it, choose the top performer.

You can easily track down your top-performing pages using Google Analytics, and you can also use information from Search Console.

In terms of how you improve these pages, again, you’ll want to focus on page analysis and competitor analysis. 

Run some diagnostic tools like those linked earlier to get a sense of where those pages are, then compare that to the top-ranked pages in the SERPs for the queries you are targeting.

As part of this, you may need to strengthen the content, so look at your content with a critical eye, too.

4. Focus on pages that target head terms

In my experience, maybe 15% of a website is targeting its “head” terms – the broader, more generic keywords/queries that people use when searching for what you have to offer. 

Ultimately, you can drive a ton of traffic if you can rank for these more generic terms – and they can be million-dollar keywords. 

So if I were to focus my efforts on a large site, I would want to optimize the pages that target head terms (the landing pages), and then secondly, optimize all the subpages that support them. 

This typically means you need solid information architecture – how you organize your content. 

If you already have well-organized content that supports your head terms, then you’d focus on optimizing those pages using some of the tools and strategies I’ve already mentioned. 

If you don’t, however, you may need to do some reorganization of the content, links and navigation on your site. This is where SEO siloing comes in handy.

SEO siloing is a form of website content organization and internal linking practices. One of the goals of this strategy is to maximize your site’s relevance for head terms.

Not only that, but SEO siloing can help by:

• Enabling your site to show expertise and authority.
• Ranking pages better in the SERPs.
• Helping users navigate your website with ease.

Just to give a quick example: Say you own a power tools business that sells cordless power tools, electric power tools and gas-powered tools.

You’d start by organizing the website content (and site navigation) into categories supporting the three product lines you sell. 

Your SEO program aligns with this because these categories will be the head terms you are after, too.

Practically, each product line will have its own section on the site with a landing page and supporting pages. 

The goal is to have well-organized landing pages and supporting pages with helpful, expert content that aligns with how people search and learn about the things you have to offer. 

Links between the landing pages and their own subpages clarify the relationships for search engine spiders crawling the website.

You can apply siloing to all types of sites, including ecommerce, B2B and informational sites.

Of course, this is a highly simplified explanation, and the details of how to make SEO siloing work well are vast.

Can this be a massive project? Yes. But it depends on the state of the site.

For some, they may have to do a larger reorganization of the content to achieve their goals, while others may need less effort with perhaps some new internal linking.

How big of a job this is depends on things like:

• Your ability to make changes to the site (roadblocks could be buy-in or CMS issues).
• How organized or disorganized the site’s content currently is.
• If you need to create new content to fill in the gaps or significantly update content.

It can be a challenge but well worth it in most cases – it’s one of the ways we get our clients to rank for highly competitive one-word terms. 

For our final step in optimizing massive websites, let’s look at how you can elevate the quality of a website from an SEO perspective by focusing on the content.

5. Clean up the content

Last but not least, having excellent content is a priority. And this can be a huge challenge for large sites.

When you have massive amounts of content, you will need to divide and conquer. 

I’ve said previously that website publishers should spend half the time creating new content, and half the time updating old content. 

(And by the way, if you plan on creating new AI content, see my article on ChatGPT and SEO content.)

Large websites need a system for:

• Evaluating the content on the site, starting with key pages, but also looking at those pages that haven’t been touched in years. 
• Updating old pages that still offer value but need new information.
• Updating meta tags so that you have unique, original content on every key page. 
• Handling the irrelevant content that exists on the site (301 redirects, removing the page, etc.).

Google’s helpful content system looks at the site as a whole, not just at a page level. 

So if you have some great content, and then a whole bunch of “search engine first” content, it can negatively impact the ability of the site to do well in the search results. 

As you are evaluating your content, keep some important guidelines in mind from Google:

Then you will go through all the steps of optimizing those pages after the content is of the highest quality.

Making SEO for large websites more manageable

Large sites have their own unique issues. As you stand before the towering fortress that is your website, the divide-and-conquer attitude will help you stay sane. 

While the task may seem daunting, you can remind yourself that SEO is an ongoing job – so play the long game.