How Google lost part of the search index!

Most of the time, Google’s search engine runs properly but in April 2019 many different issues appeared at about the same time.

“Our teams work hard to prevent technical issues that could affect our users who are searching the web, or webmasters whose sites we index and serve to users. Similarly, the underlying systems that we use to power the search engine also run as intended most of the time. When small disruptions happen, they are largely not visible to anyone except our teams who ensure that our products are up and running. However, like all complex systems, sometimes larger outages can occur, which may lead to disruptions for both users and website creators.”

“In the last few months, such a situation occurred with our indexing systems, which had a ripple effect on some other parts of our infrastructure. While we worked as quickly as possible to remedy the situation, we apologize for the disruption, as our goal is to continuously provide high-quality products to our users and to the web ecosystem.”

“Since then, we took a closer, careful look into the situation. In the process, we learned a few lessons that we’d like to share with you today. In this blog post, we will go into more details about what happened, clarify how we plan to communicate better if such things happen in the future, and remind website owners of the channels they can use to communicate with us.”

So, what happened a few months ago?

In April, Google had several issues related to its index. The Search index is the database that holds the hundreds of billions of web pages that Google crawls on the web and that they think could answer some of Google users’ queries. When a user enters a query in the Google search engine, Google’s ranking algorithms sort through those pages in our Search index to find the most relevant, useful results in a fraction of a second.

To start it off, Google temporarily lost part of the Search index.

As Google pushed some planned changes to the Search index, on April 5th 2019 parts of the deployment system broke, on a Friday no-less! More specifically: as they were updating the index over some of Google data centers, a small number of documents ended up being dropped from the index accidentally.

Google fixed the issue and all data centers were fully back to a complete version of the index on April 11th.

As a consequence of the Search index having the issues described above, Search Console started to also show inconsistencies.

Around the same time as the main indexing bug explained above, Google also had brief problems gathering fresh Google News content. Additionally, while rendering pages, certain URLs started to redirect Googlebot to other unrelated pages. These issues were entirely unrelated to the indexing bug, and were quickly resolved.

On May 22nd Google experienced another issue. Here’s what happened: while processing certain URLs, Google’s duplicate management system ran out of memory after a planned infrastructure upgrade, which caused all incoming URLs to stop processing.

Here is more detailed information on what happened.