Insights into Google’s Mobile-First Indexing

Google’s New Life as a Minimalist: Part One

Google’s transition to Mobile-First Indexing takes many of its cues from the popular Minimalist movement. In this two-part series by Cindy Krum, we will look at changes in Google’s search engine indexing through the lens of this creative framework. With a better understanding of this new processes, you should be able to modify your own strategy to improve search rankings for your website.

What is Minimalism?

The premise of the modern-day Minimalism movement is similar to the original Minimalism movement in art and architecture of the 1960’s & 1970’s. Rather than visual art, however, modern Minimalism focuses on the practice of reducing material clutter to make life easier and more efficient. Accordingly, the limiting of possessions is emotionally freeing, financially liberating, and environmentally responsible. Minimalists suggest that one of the benefits of having less is that you have more time and attention to focus on the things that you really love.
This practice may sound easy to some, but it can be a very intense process that involves answering tough questions, retraining your brain, and maintaining a new level of discipline and focus over time.

Why Google Needed to Embrace Minimalism

Google’s new Mobile-First Indexing has a lot in common with the Minimalism trend that has become popular with people young and old who are looking to simplify their life.  
Mobile-First Indexing has less to do with mobile devices and more to do with ‘mobile’ content, that is, content that’s not burdened with heavy design layers, or ornate and excessive device-specific baggage. In other words, it’s content that is easy to lift and move around to many different devices.
Maintaining Google’s current huge index of the web presents a significant challenge. Following their mission to “organize the world’s information and make it universally accessible and useful”, Google has been a pack-rat for way too long, hoarding everything on the Internet that they can get their crawlers on, and rarely purging what is old, low-quality, or no longer useful.
Therefore, one of the goals of Mobile-First Indexing is to reduce the burden of their index by making it more efficient. Switching to a new index will hopefully allow them to limit the resource-intensive need to constantly crawl and re-crawl the web. The change to Mobile-First Indexing may allow them to focus their energy on making their search engine faster and more efficient. In turn, gains in search engine efficiency will allow them to improve search results without all the waste—a Minimalist’s dream!
My supposition is that Google will soon be jumping on the Minimalist bandwagon, perhaps going ‘full-Minimalist’ by finally launching Mobile-First Indexing in 2018.

Can a Search Engine Scale Back?

From what we can tell, Google has been planning this transition to Mobile-First Indexing for at least three years, but the problems that made it necessary started even further back than that.
In terms of content, Google has always pushed webmasters to create good content. However, it can be difficult for a search engine to determine what good content actually looks like. Historically, both good and bad content could rank in searches, assuming it had the right signals.
Even if Google could easily tell which content was good and which content was not, they kept all the content in the index—just in case.

Below we’ll look at some of the ways Google has started cleaning up their index—much like a person might go through all their old shoes in the back of the closet.

Google as an HTML Hoarder

Google has long rewarded older content. Because of this, it makes sense for webmasters to create lots of content, and never archive it. The outdated material might help a site rank for a relevant keyword, or it could have links that boosted another important page on the site.
By encouraging websites to retain outdated content, Google created a scenario where, from a Minimalist’s perspective, its “closets” were overflowing with low-quality stuff that was rarely used. Google was wasting time and energy maintaining and ranking URLs that received minimal search traffic, including caching HTML pages with expired promotions, old events, items that are no longer available for sale, and old news articles.

Challenges with JavaScript

For years, Google avoided crawling JavaScript because they claimed it was a security risk. Crawling JavaScript was also cumbersome and difficult.
As JavaScript became a more integral part of the web, this deficiency became more pronounced and something had to be done. Google began providing workarounds, including

• Guidelines for AJAX crawling and use of HTML snapshots
• Feedback on how to get single-page apps indexed with Phantom JS and XML sitemaps
• Instructions about how to index infinite scroll content with HTML5 and push-state

For a while, these seemed like viable solutions to the problem. After a couple years, however, it became obvious that these short-term solutions were not sustainable.
The first meaningful steps toward understanding the impact of JavaScript on the search index came with Google’s Mobile-Friendliness update, which allowed Google to crawl the CSS and JavaScript files on websites. Ostensibly, the update helped Google get a better understanding of how the websites worked on mobile. More pragmatically, though, it helped Google learn how much they had been missing with their previously limited focus on JavaScript, and determine what would be necessary to achieve more meaningful crawling and indexing.

Indexing Native Apps

Native apps also began to grow in popularity, but there was no clear-cut way to index them either. Google initially provided methods for linking apps to web-versions of content through on-page markup, but that failed. Their next attempt was to use an API, which ultimately only worked for Android apps and could not be maintained for iOS apps. They tried providing methods of linking domains with web-app manifest documents that helped identify content in the apps, but this process was cumbersome and often relied on forcing web URLs on content that otherwise didn’t need them.
In the end, Google was hit with a grim reality: lots of stuff that they wanted to return in search results didn’t have unique URLs that they could use in their index. The structure of the index needed a fundamental modification. The mess was big, and it was getting bigger at a rapid rate. Google needed a new, more efficient way to process all of their data.

Planning for the Purge

When following Minimalist principles, the magic is supposed to happen when you eliminate the things in your life that you don’t actually need. The problem is, it can be hard to know what to keep and what to get rid of. For Google, this was a constant battle.
Google wanted to solve problems concerning the quality, speed, and organization of their index, as well as indexing JavaScript and native apps, which didn’t have URLs. Google also needed a way to make a fair comparison of all the different types of content that they wanted to index so that they could determine the best order for the rankings.
Like a good Minimalist, Google started with the easy stuff—low-quality content and duplicate content. They already had the Penguin and Panda algorithm updates in place, and they continued tweaking these over time. These updates allowed them to improve the ordering of content, and focus on the best. They left the lower-quality content in the index but lowered its ranking. (For more information on these algorithms, Moz offers a great breakdown.)

Entity Search Leads to Knowledge Graph

The Google Hummingbird update laid the foundation for the real long-term plan: ‘entity search.’ This new search mechanism would allow Google to re-organize their index based on a system in which URLs were unnecessary. They also acquired a user-generated entity engine called Freebase and began encouraging webmasters to use Schema.org to mark up their content for semantic understanding.
The benefit of entity search was that it provided a means of organizing the index that did not rely on URLs as unique identifiers. Instead, it indexed pieces of content in a hierarchy, in relation to other pieces of content. It also allowed for different kinds of assets–like apps, facts, concepts, social profiles, and other information–to be associated with entity concepts.

The plan would be to reconfigure the index around the concept of entities, which would fuel a deeper, semantic understanding of the information that Google was trying to organize, and it would begin to break-down their reliance on URLs for indexing. Information could be linked to entities and concepts semantically, rather than just programmatically.
And thus, Google’s Knowledge Graph was born.

The Importance of Engagement

Google aimed to rely on relationships with web content to determine when apps should rank. At MobileMoxie, we believe that Google has cleverly planned to use people’s activity in apps to generate Private Indexes on individual user’s phones, then aggregate information about large groups of users’ engagement, to determine if a piece of content should be added to the Public Index. This would not work for incredibly time-sensitive information like news, for which Google is likely planning to use Accelerated Mobile Pages (AMP) (see Part Two). For apps, though, we think Google planned on leveraging user-engagement as one of the main drivers for Android app crawling and indexing, which is very similar to what Apple uses to determine when to promote public content to their Public Index.
In the long run, the idea behind Mobile-First Indexing based on entities and semantics was that it would eliminate the inefficiency of having to optimize content for indexing on three different platforms—web, iOS and Android.

Be sure to check out our next article in this series on Thursday, when we explore how Google slimmed down its massive index to create a leaner and more efficient search engine.

Cindy Krum is the founder of MobileMoxie, one of the leading mobile marketing companies. The company’s clients include MTV, Party Gaming and a number of Fortune 50 companies. Cindy is also a regular speaker at major conferences around the world.
PERSONAL NOTE: Anyone who knows me may know that I appreciate things that are neat and tidy when possible, but I, myself am not a Minimalist. I struggle with the concepts of Minimalism and in fact I struggle with any new and trendy way of thinking or being. My sentiments on the topic fall closely in-line with Brett and Kate McKay’s thoughts, written up here. Minimalism has the potential to be great for some people, especially in moderation, but am skeptical that it will serve all those who are aiming for a permanent transformation in #TinyHouses and #VanLife well, especially in the long term. But for a search engine like Google, I think it should be just dandy.