Importance of the Robot File
So you have a website and it’s getting crawled by the search engines, great! But what about files you don’t want them to crawl? That’s where a robots file comes into play. Every site owner should have a robot text file, it’s a simple .txt file that you place in your root directory online with some commands for the robot crawlers. You can set ‘allow’ and ‘disallow’ commands for your folders.
If you have an images folder and don’t want all the contents of the folder (graphic files) crawled and listed into google images search, then you can prevent this by adding a ‘disallow’ command to your robots file. Also, consider disallowing crawls to private folders, or folders for internal company use, or password protected folders.
———————————————————————————————————-
dBaseMedia.com | Search Engine Optimization | Lead Generation | Online Advertising
If you enjoyed this post, please consider to leave a comment or subscribe to the feed and get future articles delivered to your feed reader.
