Robots.txt (What It Is, How It Works, AND How To Make One) With YOAST

Not many people take the time to use a robots.txt file for their website. For search engine spiders that use the robots.txt to see what directories to search through, the robots.txt file can be very helpful in keeping the spiders indexing your actual pages and not other information, such as looking through your stats!

The robots.txt file is useful in keeping your spiders from accessing parts folders and files in your hosting directory that are totally unrelated to your actual web site content. You can choose to have the spiders kept out of areas that contain programming that search engines cannot parse properly, and to keep them out of the web stats portion of your site.

Many search engines cannot view dynamically generated content properly, mainly created by programming languages, such as PHP or ASP. If you have an online store programmed in your hosting account, and it is in a seperate directory, you would be wise to block out the spiders from this directory so it only finds relevant information.

The robots.txt file should be placed in the directory where your main files for your hosting are located. So you would be advised to create a blank text file, and save it as robots.txt, and then upload it to your web hosting to the same directory your index.htm file is located. But Yoast SEO to the rescue again.

The awesome people at Yoast SEO included a free tool to help you edit robots.txt file without having to worry about the technical issues of accessing your server.

In order to access it you need to go to the “Tools” tab inside of the Yoast “SEO” options in your dashboard.

Once you have done that just click on “File Editor”.

This is where you can make any desired changes to your robots.txt file. If you do not see one, you can follow the on screen prompts to create one.

If you need more information on your robots.txt file, I’d recommend taking a look at the Yoast SEO Example.