{"id":21463,"date":"2020-01-29T15:15:20","date_gmt":"2020-01-29T15:15:20","guid":{"rendered":"https:\/\/www.arimetrics.com\/glosario-digital\/robots-txt"},"modified":"2026-06-26T19:30:55","modified_gmt":"2026-06-26T19:30:55","slug":"robots-txt","status":"publish","type":"encyclopedia","link":"https:\/\/www.arimetrics.com\/en\/digital-glossary\/robots-txt","title":{"rendered":"Robots.txt"},"content":{"rendered":"<p><img decoding=\"async\" class=\"boxpad alignright wp-image-14858 size-full\" src=\"https:\/\/www.arimetrics.com\/wp-content\/uploads\/2020\/01\/robots_txt.png\" alt=\"Robots.txt\" width=\"300\" height=\"300\" srcset=\"https:\/\/www.arimetrics.com\/wp-content\/uploads\/2020\/01\/robots_txt.png 300w, https:\/\/www.arimetrics.com\/wp-content\/uploads\/2020\/01\/robots_txt-150x150.png 150w\" sizes=\"(max-width: 300px) 100vw, 300px\" \/><\/p>\n<p><strong>Definition<\/strong>:<\/p>\n<p><em><strong>Robots.txt<\/strong> <\/em>is the common name for a text file that is loaded into the root directory of a Web site and linked in the <a href=\"https:\/\/www.arimetrics.com\/en\/digital-glossary\/html5\">HTML<\/a> code of the Web page. The <strong>robots.txt<\/strong> file is used to provide instructions on the website for web robots and spiders. The authors of the web pages can use robots.txt so that the robots that participate in the tracking cooperate and do not have access to the entire site or parts of a website that they want to keep private.<\/p>\n\n<h2>Functionality of the robots.txt file<\/h2>\n<p>Robots.txt allows webmasters to tell search engines which parts of the site should be excluded from crawling and has great relevance to <a href=\"https:\/\/www.arimetrics.com\/en\/seo-agency-spain\">technical SEO<\/a> in terms of site indexing. Although it is not a method of protecting sensitive content, as it does not prevent direct access to <a href=\"https:\/\/www.arimetrics.com\/en\/digital-glossary\/url\">URLs<\/a>, it is an effective way to manage crawling and indexing of unwanted content. Search engines generally adhere to the guidelines set out in it, although they are not obliged to do so.<\/p>\n<h2>Importance and limitations of robots.txt<\/h2>\n<p>When assessing the importance of this file it is important to consider the following:<\/p>\n<ul>\n<li><strong>Crawl control<\/strong>: The robots.txt helps direct traffic from search bots, optimizing server bandwidth usage and improving crawl efficiency.<\/li>\n<li><strong>It is not a security tool<\/strong>: It should not be used to hide sensitive information, as it does not prevent direct access to URLs. For content protection, appropriate authentication and authorization methods should be used.<\/li>\n<li><strong>Voluntary guidelines<\/strong>: Although major search engines, such as Google and <a href=\"https:\/\/www.arimetrics.com\/en\/digital-glossary\/bing\">Bing<\/a>, respect robots.txt instructions, there is no guarantee that all bots do, especially malicious bots.<\/li>\n<\/ul>\n<h2>Location of robots.txt<\/h2>\n<p>The location of robots.txt is very important. It must be in the main directory because otherwise search engines will not be able to find it. If the file is not found in this location, search engines will assume it does not exist and will proceed to crawl and index all site content. Examples of robots.txt usage<\/p>\n<p><strong>Specific Folder Tracking Lock:<\/strong><\/p>\n<pre><code class=\"language-plaintext\">User-agent: *\r\nDisallow: \/privado\/\r\nDisallow: \/configuracion\/<\/code><\/pre>\n<p><strong>Enable Site Wide Tracking:<\/strong><\/p>\n<pre><code class=\"language-plaintext\">User-agent: *\r\nDisallow:<\/code><\/pre>\n<p><strong>Locking a Specific File<\/strong><\/p>\n<pre><code class=\"language-plaintext\">User-agent: *\r\nDisallow: \/archivo-secreto.html<\/code><\/pre>\n<h2>Frequently asked questions about Robots.txt<\/h2>\n<div class=\"geo-faq-block\">\n<details class=\"geo-faq-item\">\n<summary>What is Robots.txt?<\/summary>\n<p>Definition : Robots.txt is the common name for a text file that is loaded into the root directory of a Web site and linked in the HTML code of the Web page. The robots.txt file is used to provide instructions on the website for web robots and spiders. In the Arimetrics glossary it is placed in a digital marketing context to clarify its role, uses and practical implications.<\/p>\n<\/details>\n<details class=\"geo-faq-item\">\n<summary>What is Robots.txt used for in digital marketing?<\/summary>\n<p>It is used to better analyse an action, tool, channel or behaviour related to acquisition, measurement, communication, sales or user experience. Its value depends on applying it to a concrete decision.<\/p>\n<\/details>\n<details class=\"geo-faq-item\">\n<summary>How is Robots.txt related to a digital strategy?<\/summary>\n<p>It is related to digital strategy when it affects objectives, data, content, technology, campaigns or conversion processes. That is why it should be reviewed together with the business context, not as an isolated term.<\/p>\n<\/details>\n<details class=\"geo-faq-item\">\n<summary>What should be considered when working with Robots.txt?<\/summary>\n<p>It is advisable to review its definition, context, associated metrics, limitations and possible risks. It is also useful to validate whether the concept has a real impact on performance, user experience or decision-making.<\/p>\n<\/details>\n<\/div>\n<p><script type=\"application\/ld+json\">\n{\n  \"@context\": \"https:\/\/schema.org\",\n  \"@graph\": [\n    {\n      \"@type\": \"DefinedTerm\",\n      \"@id\": \"https:\/\/www.arimetrics.com\/en\/digital-glossary\/robots-txt#definedterm\",\n      \"name\": \"Robots.txt\",\n      \"description\": \"Definition of Robots.txt in the Arimetrics Digital Glossary.\",\n      \"inDefinedTermSet\": {\n        \"@type\": \"DefinedTermSet\",\n        \"name\": \"Arimetrics Digital Glossary\",\n        \"url\": \"https:\/\/www.arimetrics.com\/en\/digital-glossary\"\n      }\n    },\n    {\n      \"@type\": \"FAQPage\",\n      \"@id\": \"https:\/\/www.arimetrics.com\/en\/digital-glossary\/robots-txt#faq\",\n      \"mainEntity\": [\n        {\n          \"@type\": \"Question\",\n          \"name\": \"What is Robots.txt?\",\n          \"acceptedAnswer\": {\n            \"@type\": \"Answer\",\n            \"text\": \"Definition : Robots.txt is the common name for a text file that is loaded into the root directory of a Web site and linked in the HTML code of the Web page. The robots.txt file is used to provide instructions on the website for web robots and spiders. In the Arimetrics glossary it is placed in a digital marketing context to clarify its role, uses and practical implications.\"\n          }\n        },\n        {\n          \"@type\": \"Question\",\n          \"name\": \"What is Robots.txt used for in digital marketing?\",\n          \"acceptedAnswer\": {\n            \"@type\": \"Answer\",\n            \"text\": \"It is used to better analyse an action, tool, channel or behaviour related to acquisition, measurement, communication, sales or user experience. Its value depends on applying it to a concrete decision.\"\n          }\n        },\n        {\n          \"@type\": \"Question\",\n          \"name\": \"How is Robots.txt related to a digital strategy?\",\n          \"acceptedAnswer\": {\n            \"@type\": \"Answer\",\n            \"text\": \"It is related to digital strategy when it affects objectives, data, content, technology, campaigns or conversion processes. That is why it should be reviewed together with the business context, not as an isolated term.\"\n          }\n        },\n        {\n          \"@type\": \"Question\",\n          \"name\": \"What should be considered when working with Robots.txt?\",\n          \"acceptedAnswer\": {\n            \"@type\": \"Answer\",\n            \"text\": \"It is advisable to review its definition, context, associated metrics, limitations and possible risks. It is also useful to validate whether the concept has a real impact on performance, user experience or decision-making.\"\n          }\n        }\n      ]\n    }\n  ]\n}\n<\/script><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Definition: Robots.txt is the common name for a text file that is loaded into the root directory of a Web site and linked in the HTML code of the Web page. The robots.txt file is used to provide instructions on the website for web robots and spiders. The authors of the web pages can use [&hellip;]<\/p>\n","protected":false},"author":7,"featured_media":0,"template":"","encyclopedia-tag":[1004],"class_list":["post-21463","encyclopedia","type-encyclopedia","status-publish","hentry","encyclopedia-tag-indexacion-seo"],"_links":{"self":[{"href":"https:\/\/www.arimetrics.com\/en\/wp-json\/wp\/v2\/encyclopedia\/21463","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.arimetrics.com\/en\/wp-json\/wp\/v2\/encyclopedia"}],"about":[{"href":"https:\/\/www.arimetrics.com\/en\/wp-json\/wp\/v2\/types\/encyclopedia"}],"author":[{"embeddable":true,"href":"https:\/\/www.arimetrics.com\/en\/wp-json\/wp\/v2\/users\/7"}],"wp:attachment":[{"href":"https:\/\/www.arimetrics.com\/en\/wp-json\/wp\/v2\/media?parent=21463"}],"wp:term":[{"taxonomy":"encyclopedia-tag","embeddable":true,"href":"https:\/\/www.arimetrics.com\/en\/wp-json\/wp\/v2\/encyclopedia-tag?post=21463"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}