Ever Wondered Why To Implement Robot.txt File

Have you ever wondered how to check your robot.txt file that weather it’s implemented on your website or not. So to navigate that it’s very simple just open your web browser and enter www.yourdomain.com/robots.txt to view the contents of your robots txt file.

You must be wondering why to implement robot.txt file on website the answer is simple
If your website doesn't have a robots.txt file then search engines will automatically index everything they can find on your site.

There are two important official commands for the robots.txt file: User-agent and Disallow. Do not use more commands than these.

• Don't change the order of the commands. Start with the user-agent line and then add the disallow commands:

User-agent: *
Disallow: /cgi-bin/

• Don't use more than one directory in a Disallow line. "Disallow: /support /cgi-bin/ /images/" does not work. Use an extra

Disallow line for every directory:

User-agent: *
Disallow: /support
Disallow: /cgi-bin/
Disallow: /images/