Do you have a robots file?
The robots.txt file is a useful file because it keeps search engines from indexing certain pages you do not want spidered. Most site owners ask, Why would you not want a page indexed/crawled? Well, If you have an images folder and don’t want all the contents of the folder (graphic files) crawled and listed into google images search, then you can prevent this by adding a ‘disallow’ command to your robots file for images folder.
You should also, consider disallowing crawls to private folders, or files that are under development. If you want to exclude all the search engine spiders from a certain folder (like your stats folder or your images folder), you would write something like this in the robots.txt file:
User-agent: *
Disallow: /images/
Disallow: /stats/
———————————————————————————————————-
dBaseMedia.com | Search Engine Optimization | Lead Generation | Online Advertising
If you enjoyed this post, please consider to leave a comment or subscribe to the feed and get future articles delivered to your feed reader.
