Skip to content Skip to sidebar Skip to footer

Blogger: How to Set up Robot.txt on Blogger

One of the efforts to make a blog more SEO Friendly is by setting the search preferences in the blogger settings. Robots.txt is used to control the crawl engine, Googlebot, to crawl all pages on the site. This setting is done in such a way as to choose which pages are allowed to be accessed and displayed by search engines. In the end, pages that have been indexed by search engines do not have a negative impact on our blog.

What is Robots.txt?
Is a protocol for notifying about page access on a blog or website. Also known as exclusion protocol robots, web spiders, robot crawlers, or web wanderers. These robots are installed in the header to allow or limit crawling access in the way search engines work. The goal is the visibility of the page on search engines such as google, bing, MSN, or other search engines.

Pages that Blogspot doesn't display

For the reason that the following URL backlinks are examples that do not have 'strength' in terms of SEO Friendly. The URL of a certain label page, for example, is only to make it easier to browse categories or labeling in blogger but does not affect the number of quality links in Blogspot when associated with search engines. For that, some bloggers don't want to appear in search results. The following pages are commonly isolated in search engines, namely:
  1. Usually, the search label page is not displayed for SEO Friendly reasons. https://www.withoutimage.xyz/search/label/SEO The page with the URL is not displayed because it is not SEO Friendly.
  2. Archive URL page https://www.withoutimage.xyz/2018/7 - The archive page is also not SEO friendly and instead causes problems in webmaster tools.
  3. Search page https://www.withoutimage.xyz/search?cof=FORID%3A10&ie=ISO-8859-1&q=seo - This page will also be bad because it doesn't count as a quality link on the blog.
  4. Privacy blog page
  • Disclaimer
  • Term of service
  • Term of Use
Usually, the above page is not included in search results because it concerns the privacy of the blog.

Can Robots.txt manage it?

It's possible because robots.txt was designed for that purpose too. In addition to using robots.txt settings or settings for robots, Googlebot can use meta tags. For example, a blog that has been registered on the google search engine but doesn't want to be included in the search, the meta tag code is:
<meta content='noindex, nofollow name='robots'/>
<meta content='noindex' name='googlebot-image'/>
The explanation is that robots should not crawl and browse all blog content (first code). As for the second code, so that all images on the blog are not included in Google search results.

How to Setting Blogger Robots.txt?
For the trick of setting it in the settings menu then search preferences and then select custom robots.txt as follows:
User-agent: Mediapartners-Google ← robots tag Google

Disallow: /search ← non-crawled pages

Allow: / ← crawlable pages

User-agent: * ← agent robots

Disallow: /search ← non-crawled pages

Allow: / ← crawlable pages 
Sitemap: https://www.withoutimage.xyz.com/sitemap.xml ← Sitemap Standard Blogger
For how to set blogger robots.txt See → the settings are like our blog https://www.withoutimage.xyz.com/robots.txt.

How do I know the robots' visit?
If you have registered your blog with Google Webmaster Tools now Search Console, there is a menu to find out the robot visit reports or when was the last time the robots visited to crawl our site. That's a little article about the robots.txt setting trick on blogger and how to set the custom robot.txt to make it more SEO Friendly. Have questions about custom robots.txt on the blogger and its settings? please submit in the comments column in polite, concise, and easy to understand language. For the next article, namely how to set a special header tag for the Blogspot robot header.