Are robots taking over? Should you worry???
We’ve already had a few clients inquire about this. Mainly, these are clients with older websites that haven’t been overhauled in some time.
You see, since Google announced that they would be penalizing sites in mobile searches which were not optimized for mobile, the robots.txt file has taken on a new importance.
What is a robot.txt file???
This is a file added to the directory of your site which instructs the search engine bots as to what they should and shouldn’t ignore. Of course, no one tells Google what to do. These are really just suggestions.
That said, it’s always a good idea to follow Google’s lead on things.
In the past, the common practice was to block the crawlers from access to these types of files in an effort to keep certain parts of code private.
Welcome to the world of technical SEO.
Yes, content is king, and context is the alpha and the omega when it comes to SEO. But, the technical makeup of your site still matters. I would argue not as much as content and context. But, every little bit helps.
And the good news… this is a very quick and easy change to make. Contact your webmaster and ask them to make this change. But, please note, if you have not yet optimized your site for mobile devices, you might want to tackle that problem first.
For more on robots, here is great overview from SEO Master/Guru/Ninja, Rand Fishkin, in a recent Whiteboard Friday video…