robots.txt generator online free

Search Engine Optimization

Robots.txt Generator


Default - All Robots are:  
    
Crawl-Delay:
    
Sitemap: (leave blank if you don't have) 
     
Search Robots: Google
  Google Image
  Google Mobile
  MSN Search
  Yahoo
  Yahoo MM
  Yahoo Blogs
  Ask/Teoma
  GigaBlast
  DMOZ Checker
  Nutch
  Alexa/Wayback
  Baidu
  Naver
  MSN PicSearch
   
Restricted Directories: The path is relative to root and must contain a trailing slash "/"
 
 
 
 
 
 
   



Now, Create 'robots.txt' file at your root directory. Copy above text and paste into the text file.


About Robots.txt Generator

Free online robot.txt generator - robots.txt maker - robot txt file creator

In web world robots exclusion protocol (REP) plays a major role in crawling of your website and it's web pages. Robots.txt file gives instructions to all search engine robots on which web pages / folder should be crawl or which pages / directories should be ignored.

Where to place robots.txt

First please make sure the file name as a robots with an extension .txt (robots.txt)is correct

By default all search engine bots search for robots.txt file in the highest level directory i.e root directory of your website (i.e www.yourdomainname.com/robots.txt). You need to transfer the text file to the root directory of your domain through any FTP client (File Transfer Protocol Client Software).

Note : Do you know that robots.txt files are very important and a single text file that could slow down or destroy your website complete ? so be caution while your are creating the file.

Online robots.txt file maker Key Features and Benefits

  • Robots.txt file can be uploaded to your root directory of web server through any FTP Client software or C-panel.
  • The file give instructions to all web bots about their website, so that which of your website's pages or directories should and should not show up in searches results.
  • Will give you correct direction when you decide to add new instruction, file, or file path to either the new or existing robots.txt file.
  • When using our robot.txt builder tool, you can customise as to which search engines you want to include in your robots file.
  • After creating your robots.txt file, upload it to the / (root) , all search engines will be pointed to as which web page's, folder or directories of your site should or should not be shown in searches result.