When it comes to SEO, most people understand that a website must have content, “search engine friendly” site architecture / HTML, and metadata (title tag and meta description).
Another meta element, if implemented properly, could stumble the site’s robots.txt Yes. I am reminded of this recently in the review of a large company has to build a mobile version of its website in a subdirectory spent a considerable amount of money sites. That’s fine, but in their robots.txt file is not allowed in a statement means that the site does not come into contact with the search engines (Disallow: / mobile /)
Let us recall how to properly implement the robots.txt to prevent search ranking problems and damage to your business, and how to properly prohibit search engines to crawl.

What is the robots.txt file?

Simply put, if you go domain.com/robots.txt, you should see the site’s directory, the site owner is to ask the search engine “Skip” (or “forbidden”) list. However, if you edit the robots.txt file is not careful, you can put your robots.txt file information might actually hurt your business.

There are tons of robots in the robots.txt file on the Web page of the offer, including the function does not allow correct use of information, and block “bad robot” from index your site.

The general rule of thumb is to make sure that your robots.txt file exists in the domain name (for example, domain.com / robots.txt) roots. To exclude all robots from indexing a portion of your site, your robots.txt file looks like this:

User Agent:
* Disallow: / cgi-bin directory /
Prohibition: / tmp directory /
Disallow: / junk /

All the above syntax tells the robot does not index / cgi-bin directory / in the / tmp /, and in your website / trash / directory.

Other real-life example of wrong robots.txt
In the past, I looked up the website, there are several high-quality content and a good amount of back links. However, the site had almost does not appear in the search engine results page (SERP in).
what happened? fine? Oh no. The site owner is not allowed to include a “/.” They tell the search engine robots any part of the site is not crawled.

In another case, SEO company edited robots.txt file, site prohibit all parts of the index is complete, the owner of the site SEO company stopped paying.

I remember a review of the company’s website, and noted that this was part of several catalogs of their site’s robots.txt file is prohibited. The company should have set up any old legacy pages 301 to pass the value of a permanent redirect from the old page a new page on the site, rather than ban the search engine index. Thus, the loss of all values.

robots.txt The Dos and Don’ts

There are many good reasons to index certain directories from a website and allow other search engine optimization purposes, to prevent search engines. Let’s look at some examples.

Here is what you should do robots.txt:

  • See all in your web directory. Most likely, there are directories that you want to say banned from the index search engines, including such as / cgi-bin directory /, / wp-admin /, / car /, / scripts /, and may contain other sensitive directory data.
  • For site indexing certain directories, may include duplicate content to stop search engines. For example, some sites have pages and articles so that visitors can easily print out “printed.” You should only allow search engines to index a version of your content.
  • Make sure that did not stop the search engines to index your content from the main site.
  • Looking for some of the files on your site, you may want to prevent the index from search engines, such as certain scripts or files may contain e-mail addresses, telephone numbers or other sensitive data.

Here is what you should not use robots.txt how to do:

  • Do not use comments in your robots.txt file.
  • robots.txt file is not listed in all of the files. Listing files you do not want to let people find files they find.
  • There is no one in the robots.txt file “/ Allow” command, so there is no need to add it to your robots.txt file.

By taking your site’s robots.txt file good appearance, and ensure the correct syntax settings, you’ll avoid the problem of search engine rankings. By disabling the search engine index duplicate content on your site, you may have to overcome may hurt your search engine rankings duplicate content issues.

One final point: If you do not know if you can do it correctly, please use the search engine optimization specialist.