Robots Exclusion Standard
|
Controlling search engine crawlers with Robots.txt and meta tags
|
|
GETTING STARTED
|
Robots.txt and Robots Meta TagsThe robots exclusion standard, also known as robots exclusion protocol lets you control what pages can be crawled and indexed by search engines. There are two main ways for you to use make use of this standard, one is through the robots.txt and the other is through robots meta tags.
Using Robots.txtThe robots.txt file is a page published on your site that gives search engine bots crawling directions. This file is often used to protect private pages from getting crawled by search engines. Your robots.txt file will automatically be created when you publish your site. To customize it and exclude content from search engine crawling go to the Pages tab > Page you want to edit > SEO Settings > Hide this page from search engines and it will add the page to the robots.txt file and it will not be crawled by search engines. You can see your robots.txt file by adding robots.txt to your domain. For example www.yourdomain.com/robots.txt
Robots Meta TagsRobots meta tags also let you control how search engines access your site content. While robots.txt controls the actual ability to crawl your page, the robots meta tag tells search engines whether to index your site after crawling. This is an important distinction because if another site links to a page blocked by robots.txt it could still get indexed. You can use robots meta tags as a way to make sure that whenever a search bot tries to crawl your page, even from a link, it will be told not to index the page.
Let's take a look at the two most important robots meta tags and how to add them to your site in Weebly. No Index Meta Tag This tag tells search engines not to index the page it's on. This is the best way to keep your page out of search results. Adding in Weebly is easy:
No Follow Meta Tag This tag tells search engines not to follow any links that you've placed on the page. You can add it using the same steps as the no index tag but with slightly different html.
|