Yahoo Shuffle: Case Study in SERP Fluctuation

I rarely use Yahoo Search, but once in a while a friend will post something on Facebook and I innocently click through not realizing I’m entering the world of Yahoo News and Search – one of my least favorite online experiences.  Today I noticed a question about a possible discrepancy in Yahoo search and so I attempted to simulate and verify the rank accuracy and experienced the Yahoo Shuffle.  

What is the Yahoo Shuffle?  

It’s difficult to accurately define without having access to the actual search algorithm, so I’ll take you through the symptoms. It’s sort of like a ****** game, and you can play too… simply enter a search phrase in Yahoo and click the Search button to spin the roulette wheel for search results.  Now open a different browser and conduct the same search in Yahoo.  If I were a gambler, I think it would be a pretty safe bet you won’t find matching results. Join me on a journey through Yahoo search…

Yahoo Search Test #1: Chrome

This search for keyword pizza ny was conducted in the US without personalization (notice that I am not signed in to a Yahoo account) or localization settings by using Google Chrome in incognito mode.  

Here we see 2 ads (+ 3 in the sidebar) followed by a map and 5 local results before the top 10 organic search results display. Nothing unusual, yet…

This Yahoo search resulted in:

Ads: 2 + 3 in the sidebar and a few at the bottom that I didn’t screen capture.

Local Results: Map and 5-pack

Top 10 Organic results in position:

• 1 nynypizzeria.com
• 2 timeout.com
• 3 en.wikipedia.org
• 4 firstpizza.com
• 5 gonyc.about.com
• 6 gayot.com
• 7 pizzerianewyork.com
• 8 yelp.com
• 9 yelp.com
• 10 blogs.villagevoice.com

Yahoo Search Test #2: Chrome with Results Shuffle

Less than 1 minute later, I closed that browser, cleared history just in case Chrome was harboring any data, and opened a fresh Chrome browser in incognito mode performing the exact same search, but with completely different results.  The line between ads and organic results was barely noticeable, so I’ve circled it in yellow for you.

This Yahoo Search resulted in:

Ads: 4 + 2 in sidebar + 4 more at the bottom

Local Results: 0

Top 10 Organic results in position:

• 1 timeout.com
• 2 villiagevoice.com
• 3 yelp.com
• 4 ny.eater.com
• 5 bestnypizza.com
• 6 thrillist.com
• 7 newyork.com
• 8 gothamist.com
• 9 tripadvisor.com
• 10 gayot.com

Total Results: 29,700,000 (didn’t catch that on the first test, but stay with me because this changes as we continue our test)

Cached Pages

You may have noticed in some of the above SERP screenshots that many pages in Yahoo Search are labeled as “cached”.  While crawling sites, search engines save snapshots of pages. Search results can contain links to the cached page in case the live page is unavailable at the time of your search. When you view a cached page, you’ll see it as it looked when the search engine last indexed it, with search terms highlighted.

If your site content changes frequently, you may not want Yahoo serving their cached version, learn how to use the NOARCHIVE metatag to keep your pages from being cached in Yahoo Search.

Yahoo Search Test #3: Firefox with more Results Shuffle

To make this a little more interesting, I quickly opened a Firefox browser in Private Window mode and, as with the Chrome test, without being signed in (i.e., no localization or personalization is evident).  One might expect that since this test was conducted within minutes of the first that we should find pretty similar results, but that’s not the case. Here we see 3 ads (+3 in the sidebar) followed by a map, 5 local results, top 10 organic results that are different from the other results that Yahoo served us in Chrome and 3 ads. 

Yahoo Search test #3 resulted in:

Ads: 3 + 3 in sidebar + 3 more below

Local Results: Map and 5-pack

Top 10 Organic results in position:

• 1 blogs.villagevoice.com
• 2 nynypizzeria.com
• 3 yelp.com
• 4 firstpizza.com
• 5 pizzerianewyork.com
• 6 gayot.com
• 7 gonyc.about.com
• 8 timeout.com
• 9 en.wikipedia.org
• 10 yelp.com

Total Results: 6,780,000 What happened to the 29,700,000 results they displayed a few minutes ago in Chrome?

We also see – only in Firefox and Internet Explorer, but not in Chrome – the Powered by BingTM logo at the bottom of the search results page, so let’s take a quick look at Bing results for this same keyword.

Bing Test: Firefox

Our Bing test demonstrates that Yahoo may be “Powered by Bing” but they are not serving the same results. As with the previous tests, this was conducted within a 10 minute window of the original search and with browser localization and personalization settings disabled.

Interestingly, the search conducted in Bing using Firefox resulted in:

Ads: 0

Local Results: 0

Top 10 Organic results in position:

• 1 firstpizza.com
• 2 timeout.com
• 3 pizzerianewyork.com
• 4 gonyc.about.com
• 5 yelp.com
• 6 pizza.dominos.com
• 7 en.wikipedia.org
• 8 yelp.com
• 9 slice.seriouseats.com
• 10 newyorknewyork.com

Total Results of 6,190,000, so if Yahoo is Powered by Bing, then where did the 6,780,000 and 29,700,000 results they displayed less than 10 minutes ago in Chrome and Firefox come from?

Bing Results in Internet Explorer Vary from Bing Results in Firefox

Let’s be fair and test Bing in it’s native Internet Explorer environment. Again we see different results between browsers in the same search engine. For those who like Bing and/or Internet Explorer, this does present a better user experience.  The local results display in a carousel across the top of the page, but there are only 8 organic results on page 1!  And there are 14 organic results on page 2, along with 5 ads.

Bing Organic results in Internet Explorer on Page 1 = 8, Page 2 = 14.

Bing Search Results:

Ads: 1 + 1 in sidebar and 1 at bottom

Local Results: Map and Carousel (quantity depends on screen width and number of possible results)

Top 10 Organic results in position: 

• 1 nynypizzeria.com
• 2 yelp.com
• 3 gayot.com
• 4 firstpizza.com
• 5 pizzerianewyork.com
• 6 gonyc.about.com
• 7 en.wikipedia.org
• 8 timeout.com
• 9 Page 2 en.wikipedia.org
• 10 Page 2 yelp.com

Total Results of 6,810,000 vs 6,190,000 listed in Bing search on Firefox.

Scoreboard

Keeping track of those results can be dizzying, so here’s a scoreboard that provides a high level overview of the variance in Yahoo Search, as well as Bing.

Notice that nynypizzeria.com ranks in the #1 search position twice and scores the #2 rank position once, yet it’s completely missing from the top 10 in search test #2 using Yahoo in Chrome and search test #4 using Bing in Firefox – and these search fluctuation tests were conducted within a 10 minute time frame from the same computer, with the same settings. For the sake of everyone who invests a lot of time and energy doing SEO, we hope Yahoo’s shuffle ends soon.

About The Author

Laura is Rank Ranger’s Digital Marketing Manager, Editor-in-Chief and Social Media maven. Writing and marketing are Laura’s passions.

She also enjoys working with Rank Ranger’s developers on ideas for new digital marketing reports and features that will make her job easier.