Beyond SEO: Googlebot Optimization
You know all about search engine optimization — the importance of a well-structured site, relevant keywords, appropriate tagging, technical standards, and lots and lots of content. But chances are you don’t think a lot about Googlebot optimization.
Googlebot optimization isn’t the same thing as search engine optimization, because it goes a level deeper. Search engine optimization is focused more upon the process of optimizing for user’s queries. Googlebot optimization is focused upon how Google’s crawler accesses your site.
There’s a lot of overlap, of course. However, I want to make this important distinction, because there are foundational ways in which it can affect your site. A site’s crawlability is the important first step to ensuring its searchability.
What is Googlebot?
Googlebot is Google’s search bot that crawls the web and creates an index. It’s also known
as a spider. The bot crawls every page it’s allowed access to, and adds it to the index where it can be accessed and returned by users’ search queries.
The whole idea of how Googlebot crawls your site is crucial to understanding Googlebot optimization. Here are the basics:
1. Googlebot spends more time crawling sites with significant pagerank
The amount of time that Googlebot gives to your site is called “crawl budget.” The greater a page’s authority, the more crawl budget it receives.
2. Googlebot is always crawling your site.
Google’s Googlebot article says this: “Googlebot shouldn’t access your site more than once every few seconds on average.” In other words, your site is always being crawled, provided your site is accurately accepting crawlers.
There’s a lot of discussion in the SEO world about “crawl rate” and how to get Google to recrawl your site for optimal ranking. There is a terminology misunderstanding here, because Google’s “crawl rate” refers to the speed of Googlebot’s requests, not the recurrence of its site crawl. You can alter the crawl rate within Webmaster Tools (gear icon → Site Settings → Crawl rate). Googlebot consistently crawls your site, and the more freshness, backlinks, social mentions, etc., the more likely it is that your site will appear in search results.
It’s important to note that Googlebot does not crawl every page on your site all the time. This is a good place to point out the importance of consistent content marketing — fresh, consistent content always gains the crawler’s attention, and improves the likelihood of top ranked pages.
3. Googlebot first accesses a site’s robots.txt to find out the rules for crawling the site.
Any pages that are disallowed will not be crawled or indexed.
4. Googlebot uses the sitemap.xml to discover any and all areas of the site to be crawled and indexed.
Because of the variation in how sites are built and organized, the crawler may not automatically crawl every page or section. Dynamic content, low-ranked pages, or vast content archives with little internal linking could benefit from an accurately-constructed Sitemap. Sitemaps are also beneficial for advising Google about the metadata behind categories like video, images, mobile, and news.
Six Principles for a Googlebot Optimized Site
1. Don’t get too fancy.
2. Do the right thing with your robots.txt.
3. Create fresh content.
4. Optimize infinite scrolling pages.
5. Use internal linking
6. Create a sitemap.xml
Analyzing Googlebot’s Performance on Your Site
The great thing about Googlebot optimization is that you don’t have to play guesswork to see how your site is performing with the crawler. Google Webmaster Tools provides helpful information on the main features.
I want to advise you that this is a limited set of data, but it will alert you to any major problems or trends with your site’s crawl performance:
Log in to Webmaster Tools, and go to “Crawl” to check these diagnostics.
Crawl Errors
You can find out if your site is experiencing any problems with crawl status. As Googlebot routinely crawls the web, your site will either subject itself to crawling with no issues, or it will throw up some red flags, such as pages that the bot expected to be there based on the last index. Checking out crawl errors is your first step for Googlebot optimization.
Some sites have crawl errors, but the errors are so few or insignificant that they don’t automatically affect traffic or ranking. Over time however, such errors are usually correlated with traffic decline.
Crawl Stats
Google tells you how many pages and kbs the bot analyzes per day. A proactive content marketing campaign that regularly pushes fresh content will provide a positive upward momentum for these stats.
Fetch as Google
The “Fetch as Google” feature allows you to look at your site or individual pages the way that Google would.
Blocked URLs
If you want to check and see if your robots.txt is working, then “Blocked URLs” will tell you what you need to know.
Sitemaps
Use the sitemap feature if you want to add a sitemap, test a sitemap, or discover what kind of content is being indexed in your sitemap.
URL Parameters
Depending on the amount of duplicate content caused by dynamic URLs, you may have some issues on the URL parameter indices. The URL Parameters section allows you to configure the way that Google crawls and indexes your site with URL parameters. By default, all pages are crawled according to the way that Googlebot decide.
Conclusion
If you want to really streamline and improve your site’s performance and SEO, then you ought to be giving some time and effort to Googlebot optimization. Some webmasters do not realize the amount of traffic that they are neglecting, simply because they have not given due attention to Googlebot optimization.
In order to be indexed and returned in search engine results, a site must be crawled. Unless the site is accurately crawled, it simply will not be indexed or returned. Start with this article, optimize your site for Googlebot, and see how it changes your traffic for the better.
Comments
Post a Comment