A robots.txt file instructs search engine bots to inspect and index certain pages and not to inspect or index others. It’s a simple text file placed within a website’s root directory.
In any robots.txt file, there are three options (known as elements): User-agent, Disallow, and Allow. These are messages to Google’s bots, such as Googlebot or Googlebot-Image, as to which pages and files they are permitted to access and which they are not. To target a specific Google bot, the robot.txt file should be written:
- User-agent [name of the bot the message is intended for]
- Disallow: [the URL you want to block]
- Allow: [the URL path within a blocked parent path you want to unblock]
