Robots.txt is one of the special files found in the root directory of any web resource. It contains certain directives regarding access to certain sections of the portal. Robots.txt is also commonly referred to as the “search bot exceptions policy”.
In this file, the webmaster is given the opportunity to register page addresses and specify other data that should not be indexed by a search robot. So, for example, the file makes it possible to set the access settings from mobile gadgets or personal computers.
Besides, robots.txt plays one of the key roles in website promotion in search engines. In other words, this file is part of any website's SEO optimization. Using the functional features of the file allows the professionals to close for indexing those pages that he does not consider necessary to index. In addition, it is in robots that the address of the map and the addresses of the website mirrors are usually set.
But another situation is possible when the webmaster is faced with the fact that some of the pages of the resource are in the status "blocked by robots.txt". Usually, this is due to the fact that when writing the file, errors were initially made. Very often, young professionals are inattentive to writing commands.
For example, if you prohibit indexing a website in robots.txt, then the search engine can perceive this in its own way and respond with the error "Blocked in robots.txt" or "url is denied by robots.txt". In addition, if you just want to block a specific page, you can use alternative methods, for example, html-code tags, which will allow you not to index pages, not follow the specified links, etc.
You need to be very careful with robots.txt, you can close an important section or the entire website from indexing. Sonar.Network will show all pages indicating whether they are closed or open for indexing.