Robots.txt For Some CMS

Robots.txt is a text file that we place on our website to inform the search robots not to visit or index certain pages on our site.

There are usually placed in the robots.txt main directory because when search engines do Crawling on our site. For example http://facebook.com/robots.txt

In this tutorial, I will give some list of robots.txt which is used on the famous CMS such as Joomla, Wordpress, Magento, and Drupal.

Wordpress Robots.txt

User-agent: * Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/plugins Disallow: /wp-content/cache Disallow: /wp-content/themes Disallow: /trackback Disallow: /tag Disallow: /author Disallow: /wget/ Disallow: /httpd/ Disallow: /cgi-bin Disallow: /images/ Disallow: /search Disallow: /feed Disallow: /feed/ Disallow: /trackback/ Disallow: /rss Disallow: /comments/feed Disallow: /feed/$ Disallow: /*/feed/$ Disallow: /*/feed/rss/$ Disallow: /*/trackback/$

Magento Robots.txt

User-agent: * Disallow: /*? Disallow: /*.js$ Disallow: /*.css$ Disallow: /checkout/ Disallow: /catalogsearch/ Disallow: /app/ Disallow: /downloader/ Disallow: /images/ Disallow: /js/ Disallow: /lib/ Disallow: /media/ Disallow: /*.php$ Disallow: /pkginfo/ Disallow: /report/ Disallow: /skin/ Disallow: /var/ Disallow: /catalog/product_compare/ Disallow: /catalog/ Disallow: /customer/ Disallow: /catalogsearch/advanced/ Disallow: /wishlist/ Disallow: /404/ Disallow: /admin/ Disallow: /api/ Disallow: /install/ Disallow: /catalog/product/view/id/</code> Disallow: /customer/

Drupal Robots.txt

User-agent: * # Directories Disallow: /database/ Disallow: /includes/ Disallow: /misc/ Disallow: /modules/ Disallow: /sites/ Disallow: /themes/ Disallow: /scripts/ Disallow: /updates/ Disallow: /profiles/ # Paths (clean URLs) Disallow: /admin/ Disallow: /aggregator/ Disallow: /comment/reply/ Disallow: /contact/ Disallow: /logout/ Disallow: /node/add/ Disallow: /search/ Disallow: /user/register/ Disallow: /contact Disallow: /logout Disallow: /user/register Disallow: /user/password Disallow: /user/login Disallow: /user/password/ Disallow: /print/ Disallow: /forward/ # Files Disallow: /xmlrpc.php Disallow: /cron.php Disallow: /update.php Disallow: /install.php Disallow: /INSTALL.txt Disallow: /INSTALL.mysql.txt Disallow: /INSTALL.pgsql.txt Disallow: /CHANGELOG.txt Disallow: /MAINTAINERS.txt Disallow: /LICENSE.txt Disallow: /UPGRADE.txt # Block user tracker pages Allow: /project/track Disallow: /*/track$ Disallow: /*/track?page= If you are not using static urls: Disallow: /?q=admin/ Disallow: /?q=aggregator/ Disallow: /?q=comment/reply/ Disallow: /?q=contact/ Disallow: /?q=logout/ Disallow: /?q=node/add/ Disallow: /?q=search/ Disallow: /?q=user/password/ Disallow: /?q=user/register/ Disallow: /?q=user/login/ Disallow: /user/login/

Joomla Robots.txt

User-agent: * Disallow: /administrator/ Disallow: /cache/ Disallow: /components/ Disallow: /editor/ Disallow: /help/ Disallow: /includes/ Disallow: /language/ Disallow: /mambots/ Disallow: /media/ Disallow: /modules/ Disallow: /templates/ Disallow: /installation/ Disallow: /libraries/ Disallow: /tmp/ Disallow: /xmlrpc/ Disallow: /admin Disallow: /administrator Disallow:/admin/ Disallow: /admin.html Disallow:/admin.php

Hopefully useful
Source : http://cruzenaldo.com/robot-txt-untuk-beberapa-cms/

CMS

By Blogger Biasa - July 12, 2010 - Add Comment