Robots.txt is a text file that we place on our website to inform the search robots not to visit or index certain pages on our site.

There are usually placed in the robots.txt main directory because when search engines do Crawling on our site. For example

In this tutorial, I will give some list of robots.txt which is used on the famous CMS such as Joomla, Wordpress, Magento, and Drupal.

Wordpress Robots.txt

Magento Robots.txt

Drupal Robots.txt

Joomla Robots.txt

Hopefully useful
Source :

Just Comment. Not Spam!