How to get Google to crawl your site faster

A question that often pops up for sites that are chasing “hot topics” is: how to get Google to crawl our site faster? This could be either because you have a big site and want to see changes faster or because you’re optimizing a news site. There are a few things you can do to improve your crawl speed. We’ll cover them in this post.

There are a couple of possible reasons why Google is slow when spidering your site. The first might seem obvious: if Google doesn’t find enough (quality) links pointing to your site, it doesn’t think your site is very important. The other reasons are technical: it has too much to crawl on your site, your site is too slow, or it’s encountering too many errors.

Your site doesn’t have enough authority

When your site doesn’t have enough quality inbound links, Google will not crawl your site very quickly. You can imagine it doesn’t necessarily want to spend much time on a site that it doesn’t deem important. When your site is brand new, this will be the case.

So, when you need more crawl action on your site, start doing some link building. We’ve written about that topic before, I’d suggest reading these three posts as I won’t go into it any further here:

  1. Link building from a holistic SEO perspective
  2. Link building, what not to do
  3. 6 steps to a successful link building strategy

If the above three posts are not enough for you, there’s more in our link building archive.

Technical reasons for slow crawling

The technical reasons for Google to crawl your site slowly can be divided into three groups: your site is too slow, you have too many errors, or you have too many URLs.

Your server is slow

The main reason why we see Google crawling sites slowly, is when the site itself is actually slow. Google actually indicates this to you when this is the case, on the “crawl rate” page of Google Search Console. If this is the case, Google might show you a “faster” option:

Crawl faster

Our advice: set it to “Faster”, but don’t stop with that: make your server respond faster, either by upgrading your hosting, or improving your site’s caching. Chances are that when Google is suffering long load times, your users are as well.

Optimize your site for Google News! Create XML News Sitemaps, editors’ picks RSS feeds and more with Yoast News SEO! »

News SEO for WordPress pluginBuy now » Info

Too many errors on your site

If you have a lot of errors on your site for Google, Google will start crawling slowly too. To speed up the crawl process, fix those errors. Simply 301 redirect those erroring pages to proper URLs on your site.

If you don’t know where to find those errors: use the Google Search Console integration in Yoast SEO. Or log into Google Search Console. If you have access to your site’s access logs, you can also look at those, preferably with a tool like Screaming Frog’s Log file analyzer.

To prevent your site from being crawled slowly, it’s important that you regularly look at your site’s errors and fix them. We have a more extensive article on fixing 404 errors.

When your site is down too much, Google will slow down incredibly hard. If this happens because of your hosting, switch hosting. Make sure to check on your site’s uptime with a tool like Pingdom.

Too many URLs

If you simply have too many URLs on your site, Google might crawl a lot but it will never be enough. This can happen because of faceted search navigation for instance, or another system on your site that simply generates too many URLs. To figure out whether this is the case for you, it’s always wise to regularly crawl your own site. You can either do that manually with Screaming Frog’s SEO spider, or with a tool like OnPage.org.

Tips to increase your crawl speed

A couple of simple tips to increase your site’s crawl speed:

  • Using the methods above, find and fix all the errors.
  • Make sure your site is fast.
  • Add an XML sitemap to your site and submit it to the search engines.
  • If all of that fails to improve your crawl speed, start link building!

Specifically for news sites, if you want to be indexed faster, you should also look at our News SEO plugin. When you’re in (or want to be in) Google News, it can help make sure you meet all the technical requirements.

Read more: ‘Site speed: tools and suggestions’ »

Advertisements

Author: Search Engine Optimisation Company

Our team have been offering SEO expert services to clients since the early days of the internet. Whilst the tactics have evolved over the years, our overall goal by no means has and thats making sure our users web pages to rank on page 1 for appropriate keywords at the same time only using honest and long term strategies. Plan to Dominate the Search Engines? While many agencies are convinced that they have to resort to black hat SEO tactics to get results, we can get the 1st page rankings for all of our customers despite the fact that constantly guaranteeing we stick to strategies that are not frowned upon by the the search engines. What happens when you turn to black hat strategies? Without a doubt, you will get some early success, however the internet search engines will quickly discover what you are doing and you can face severe penalties which includes having your website banned with the google search results altogether. We work within guidelines of leading search engines such as Google, Yahoo and Bing! so you never need to be concerned about receiving any penalties as a result of the work we carry out for you.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s