How to use the robots.txt method and use detailed configuration files

file on the robots.txt function: User-agent the value of the name used to describe the search engine Robot, must have at least one User-agent record; if User-agent >

access to the wp-content directory

robots.txt file is very simple, usually starts with User-agent, containing one or more lines of Disallow or Allow records; in the robots.txt file you can use the "#" notes, the use of specific methods and PHP; in addition, the robots.txt file support *|$|? Wildcard etc.. A simple example of how to write robots.txt file #? User-agent: * # to all search engines Robot open Disallow: /wp-content/ # banned any content

how to configure the robots.txt file? Robots.txt configuration file is very simple, right click the desktop -> -> text document, named robots.txt, note that this document is ending in ".Txt". Write the rules in the robots.txt file, using FTP software to upload to the root directory of your web site can be. To check the robots.txt file configuration is correct, just enter the URL in the browser: 贵族宝贝 your domain name /robots.txt, to see whether the normal access to. Robots.txt files written in

robots.txt is a simple.Txt at the end of the text file, search engine Robot (also called search engine robots) first file to access the web program to crawl. What is the role of the robots.txt file? Robots.txt files, and each big search engine friendly dialogue, to guide the search engine you recommend the robot crawl the web to avoid some meaningless or useless ", such as website backstage, membership interactive function, it also saves the server cyber source to a certain extent. In addition, the significance of robots.txt files of Shanghai dragon is also very important, can be good to avoid repetition, similar pages, and some keywords weight loss; the robots.txt file is written, is one of the lessons of each Er will do the Shanghai dragon.

 

robots.txt documents filed this concept, many owners also may very strange: what is the robots.txt file robots.txt file? What effect is there? How to configure the robots.txt file? How to correctly use the robots.txt file? Below, these problems are analyzed, so that you understand the robots.txt file. What is the robots.txt file

? The

Leave a Reply

Your email address will not be published. Required fields are marked *