Allow User Agents

Following the proper syntax and the correct arrangement of robots.txt commands is a must. First, let a potential crawler know whether it has been invited to spider the website with the following code you see below.

Remember, as a webmaster or owner of your website, you may choose to allow only a few of the many good user-agents out there as you wish. But as someone who may want to have a successfully optimized website, you might as well permit all ( * ) user-agents to spider your site.

That is the best way to open yourself up to any opportunity of being found by a large volume of traffic waiting to receive news that your online business do exists and ready to provide a number of great products and/or services..

Robots.txt User-agents Directive