Google Search Problems Sometimes Caused By Experiments That Conflict

In the last Search Off The Record podcast, Gary Illyes from the Google Search team said that sometimes issues can arise within the Google Search results due to two experiments that conflict with each other. He said at the 19:41 mark, “Very often there are experiments that need to be rolled back because two experiments might interact very badly with each other.”

Here is the video embed where he said this:

Lizzi Sassman from Google asked Gary Illyes:

Okay, and then if somebody launched something or pushed out some change and then that could potentially be the most recent thing that happened.

Gary Illyes responded:

Yeah. Very often there are experiments that need to be rolled back because two experiments might interact very badly with each other. Like, some alert is triggering that something is going wrong. And then you root that issue back to faulty experiment or a bad interaction between experiments. I mean, sorry, that’s not us. And then they would roll that back or ramp down the experiment.

There’s also like data pushes that need to happen every now and then. For example, if you think about, this is a wrong example, but the doodles that we have every now and then, they are part of data pushes. And then, if something goes wrong with the data push, like let’s say we lose, on the wire, 100 bytes, and then the doodle is like in the middle, there’s a line, a white line, that would be noticed by our monitoring systems that something looks really wrong on the search result page. And then someone would go there, would take a look, it’s like, “Oh my God, what’s up with that white line?” And then they are like, “Okay, so when did we push this doodle?” And then they would identify a time, what time is it, it was 2:33 PM push, and then they would roll back to the push before that.

Thanks to Glenn Gabe who posted this on X who wrote, “Sometimes problems in the SERPs (or maybe what we see as changes) are due to multiple experiments running that don’t work well with each other. When Google sees that, it might need to roll back those experiments (or ramp them down). Also, Gary explains that faulty data pushes could cause issues too. Google could roll back to the data push before that. Just another set of variables to consider when you see weird changes in the SERPs.”

He then goes into how one time cosmic rays caused issue for Google Search.

So it is not only Google algorithm updates that cause issues in the search results.

Forum discussion at X.