Are robots taking over? Should you worry???

Google has been sending out some notices lately about robot.txt files. Google is advising any website’s with older robots.txt files that block CSS or JavaScript to update their files.

We’ve already had a few clients inquire about this. Mainly, these are clients with older websites that haven’t been overhauled in some time.

You see, since Google announced that they would be penalizing sites in mobile searches which were not optimized for mobile, the robots.txt file has taken on a new importance.

What is a robot.txt file???

This is a file added to the directory of your site which instructs the search engine bots as to what they should and shouldn’t ignore. Of course, no one tells Google what to do. These are really just suggestions.

That said, it’s always a good idea to follow Google’s lead on things.

In this case, Google is saying to allow their bots to crawl your JavaScript and CSS files. Why? Well, one big reason is that these files are a good (and quick) indicator as to whether or not your site is mobile optimized.

In the past, the common practice was to block the crawlers from access to these types of files in an effort to keep certain parts of code private.

Welcome to the world of technical SEO.

Yes, content is king, and context is the alpha and the omega when it comes to SEO. But, the technical makeup of your site still matters. I would argue not as much as content and context. But, every little bit helps.

And the good news… this is a very quick and easy change to make. Contact your webmaster and ask them to make this change. But, please note, if you have not yet optimized your site for mobile devices, you might want to tackle that problem first.

For more on robots, here is great overview from SEO Master/Guru/Ninja, Rand Fishkin, in a recent Whiteboard Friday video…