Best Example Of Robots.txt File For Indexing Your Site

How To Create Robots.txt File?

A Robots.txt play a big role in your site. While your site on WordPress or any other CMS. Most people use WordPress, in WordPress if you are not finding your robots.txt file then simply go in your CPanel and write robot in your search area and click on go. You will get the location and file. It helps to block and open search engine bots to index your site. If your site is not configured in good manners then it will stop search engine bots to crawl your site. So when you are changing in your robots.txt or making your robots.txt so give full concentration on it. Try to make your robots.txt file better for crawling your site regularly. Update content regularly on your site. It will also major step for boosting your site.

SEO contain the millions element and one is the big element is robots.txt. Here I am sharing two examples which will allow search engine bots to crawl your site. If you experience your site is not indexing in Google and Bing then read this article, Is My Website Regularly Indexed By Bing And Google?

1st Example Of  Best Robots.txt File

Most people use this file in which you are giving permission to search engine bots to visit your site and take anything which you like.

 [box type=”shadow” align=”” class=”” width=””]

User-agent: *


2nd Example Of Best Robots.txt File

If you want to make advance level file then it will be best.

Best Example Of Robots.txt File For Indexing Your Site On Any Search Engine
Best Example Of Robots.txt File For Indexing Your Site On Any Search Engine


If you want to discourage search engine then simply insert these 2 lines in your robots.txt file. Your site will not index or crawl by google. If you like this please comment and share with others. Submit your site to high PR directories because it will also helpful for your site. DoFollow backlinks also play a major role in your site indexing.

Show More

Muhammad Awais Azeem

"Global Blurb since 2015" M. Awais Azeem is the CEO and founder of Global Blurb. The blog designed to share ideas about different topics. My profession is Accountant and I have completed CA Inter from Institute of Chartered Accountants of Pakistan. Me and my time present to sort out any issue regarding SEO, Blogging, Content Marketing and WordPress. Hire Us and Goodbye difficulty.

Related Articles

Want to contribute