{"id":21463,"date":"2020-01-29T15:15:20","date_gmt":"2020-01-29T15:15:20","guid":{"rendered":"https:\/\/www.arimetrics.com\/glosario-digital\/robots-txt"},"modified":"2026-05-11T23:01:12","modified_gmt":"2026-05-11T23:01:12","slug":"robots-txt","status":"publish","type":"encyclopedia","link":"https:\/\/www.arimetrics.com\/en\/digital-glossary\/robots-txt","title":{"rendered":"Robots.txt"},"content":{"rendered":"<p><img decoding=\"async\" class=\"boxpad alignright wp-image-14858 size-full\" src=\"https:\/\/www.arimetrics.com\/wp-content\/uploads\/2020\/01\/robots_txt.png\" alt=\"Robots.txt\" width=\"300\" height=\"300\" srcset=\"https:\/\/www.arimetrics.com\/wp-content\/uploads\/2020\/01\/robots_txt.png 300w, https:\/\/www.arimetrics.com\/wp-content\/uploads\/2020\/01\/robots_txt-150x150.png 150w\" sizes=\"(max-width: 300px) 100vw, 300px\" \/><\/p>\n<p><strong>Definition<\/strong>:<\/p>\n<p><em><strong>Robots.txt<\/strong> <\/em>is the common name for a text file that is loaded into the root directory of a Web site and linked in the <a href=\"https:\/\/www.arimetrics.com\/en\/digital-glossary\/html5\">HTML<\/a> code of the Web page. The <strong>robots.txt<\/strong> file is used to provide instructions on the website for web robots and spiders. The authors of the web pages can use robots.txt so that the robots that participate in the tracking cooperate and do not have access to the entire site or parts of a website that they want to keep private.<\/p>\n\n<h2>Functionality of the robots.txt file<\/h2>\n<p>Robots.txt allows webmasters to tell search engines which parts of the site should be excluded from crawling and has great relevance to <a href=\"https:\/\/www.arimetrics.com\/en\/seo-agency-spain\">technical SEO<\/a> in terms of site indexing. Although it is not a method of protecting sensitive content, as it does not prevent direct access to <a href=\"https:\/\/www.arimetrics.com\/en\/digital-glossary\/url\">URLs<\/a>, it is an effective way to manage crawling and indexing of unwanted content. Search engines generally adhere to the guidelines set out in it, although they are not obliged to do so.<\/p>\n<h2>Importance and limitations of robots.txt<\/h2>\n<p>When assessing the importance of this file it is important to consider the following:<\/p>\n<ul>\n<li><strong>Crawl control<\/strong>: The robots.txt helps direct traffic from search bots, optimizing server bandwidth usage and improving crawl efficiency.<\/li>\n<li><strong>It is not a security tool<\/strong>: It should not be used to hide sensitive information, as it does not prevent direct access to URLs. For content protection, appropriate authentication and authorization methods should be used.<\/li>\n<li><strong>Voluntary guidelines<\/strong>: Although major search engines, such as Google and <a href=\"https:\/\/www.arimetrics.com\/en\/digital-glossary\/bing\">Bing<\/a>, respect robots.txt instructions, there is no guarantee that all bots do, especially malicious bots.<\/li>\n<\/ul>\n<h2>Location of robots.txt<\/h2>\n<p>The location of robots.txt is very important. It must be in the main directory because otherwise search engines will not be able to find it. If the file is not found in this location, search engines will assume it does not exist and will proceed to crawl and index all site content. Examples of robots.txt usage<\/p>\n<p><strong>Specific Folder Tracking Lock:<\/strong><\/p>\n<pre><code class=\"language-plaintext\">User-agent: *\r\nDisallow: \/privado\/\r\nDisallow: \/configuracion\/<\/code><\/pre>\n<p><strong>Enable Site Wide Tracking:<\/strong><\/p>\n<pre><code class=\"language-plaintext\">User-agent: *\r\nDisallow:<\/code><\/pre>\n<p><strong>Locking a Specific File<\/strong><\/p>\n<pre><code class=\"language-plaintext\">User-agent: *\r\nDisallow: \/archivo-secreto.html<\/code><\/pre>\n<h2>Frequently asked questions about Robots.txt<\/h2>\n<div class=\"geo-faq-block\">\n<details class=\"geo-faq-item\">\n<summary>What does Robots.txt mean in digital marketing?<\/summary>\n<p>Robots.txt refers to the concept described in this glossary entry: Definition : Robots.txt is the common name for a text file that is loaded into the root directory of a Web site and linked in the HTML code of the Web page. The robots.txt file is used to provide instructions on the website for web robots and spiders. It gives teams a shared vocabulary for analysing digital projects.<\/p>\n<\/details>\n<details class=\"geo-faq-item\">\n<summary>When should teams pay attention to Robots.txt?<\/summary>\n<p>Teams should review Robots.txt when it affects acquisition, measurement, user experience, content, automation or campaign performance. The important step is to connect the definition with a real decision.<\/p>\n<\/details>\n<details class=\"geo-faq-item\">\n<summary>How is Robots.txt used in a digital strategy?<\/summary>\n<p>Robots.txt is used by translating the concept into practical checks: where it appears in the funnel, which data or channel is involved and whether it needs optimisation, monitoring or documentation.<\/p>\n<\/details>\n<details class=\"geo-faq-item\">\n<summary>What is a common mistake when interpreting Robots.txt?<\/summary>\n<p>A common mistake is using Robots.txt too broadly. It is better to verify the context, the tool or the metric involved before making strategic or technical conclusions.<\/p>\n<\/details>\n<\/div>\n<p><script type=\"application\/ld+json\">\n{\n  \"@context\": \"https:\/\/schema.org\",\n  \"@graph\": [\n    {\n      \"@type\": \"DefinedTerm\",\n      \"@id\": \"https:\/\/www.arimetrics.com\/en\/digital-glossary\/robots-txt#definedterm\",\n      \"name\": \"Robots.txt\",\n      \"description\": \"Definition of Robots.txt in the Arimetrics Digital Glossary.\",\n      \"inDefinedTermSet\": {\n        \"@type\": \"DefinedTermSet\",\n        \"name\": \"Arimetrics Digital Glossary\",\n        \"url\": \"https:\/\/www.arimetrics.com\/en\/digital-glossary\"\n      }\n    },\n    {\n      \"@type\": \"FAQPage\",\n      \"@id\": \"https:\/\/www.arimetrics.com\/en\/digital-glossary\/robots-txt#faq\",\n      \"mainEntity\": [\n        {\n          \"@type\": \"Question\",\n          \"name\": \"What does Robots.txt mean in digital marketing?\",\n          \"acceptedAnswer\": {\n            \"@type\": \"Answer\",\n            \"text\": \"Robots.txt refers to the concept described in this glossary entry: Definition : Robots.txt is the common name for a text file that is loaded into the root directory of a Web site and linked in the HTML code of the Web page. The robots.txt file is used to provide instructions on the website for web robots and spiders. It gives teams a shared vocabulary for analysing digital projects.\"\n          }\n        },\n        {\n          \"@type\": \"Question\",\n          \"name\": \"When should teams pay attention to Robots.txt?\",\n          \"acceptedAnswer\": {\n            \"@type\": \"Answer\",\n            \"text\": \"Teams should review Robots.txt when it affects acquisition, measurement, user experience, content, automation or campaign performance. The important step is to connect the definition with a real decision.\"\n          }\n        },\n        {\n          \"@type\": \"Question\",\n          \"name\": \"How is Robots.txt used in a digital strategy?\",\n          \"acceptedAnswer\": {\n            \"@type\": \"Answer\",\n            \"text\": \"Robots.txt is used by translating the concept into practical checks: where it appears in the funnel, which data or channel is involved and whether it needs optimisation, monitoring or documentation.\"\n          }\n        },\n        {\n          \"@type\": \"Question\",\n          \"name\": \"What is a common mistake when interpreting Robots.txt?\",\n          \"acceptedAnswer\": {\n            \"@type\": \"Answer\",\n            \"text\": \"A common mistake is using Robots.txt too broadly. It is better to verify the context, the tool or the metric involved before making strategic or technical conclusions.\"\n          }\n        }\n      ]\n    }\n  ]\n}\n<\/script><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Definition: Robots.txt is the common name for a text file that is loaded into the root directory of a Web site and linked in the HTML code of the Web page. The robots.txt file is used to provide instructions on the website for web robots and spiders. The authors of the web pages can use [&hellip;]<\/p>\n","protected":false},"author":7,"featured_media":0,"template":"","encyclopedia-tag":[1004],"class_list":["post-21463","encyclopedia","type-encyclopedia","status-publish","hentry","encyclopedia-tag-indexacion-seo"],"_links":{"self":[{"href":"https:\/\/www.arimetrics.com\/en\/wp-json\/wp\/v2\/encyclopedia\/21463","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.arimetrics.com\/en\/wp-json\/wp\/v2\/encyclopedia"}],"about":[{"href":"https:\/\/www.arimetrics.com\/en\/wp-json\/wp\/v2\/types\/encyclopedia"}],"author":[{"embeddable":true,"href":"https:\/\/www.arimetrics.com\/en\/wp-json\/wp\/v2\/users\/7"}],"wp:attachment":[{"href":"https:\/\/www.arimetrics.com\/en\/wp-json\/wp\/v2\/media?parent=21463"}],"wp:term":[{"taxonomy":"encyclopedia-tag","embeddable":true,"href":"https:\/\/www.arimetrics.com\/en\/wp-json\/wp\/v2\/encyclopedia-tag?post=21463"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}