How To Block AI Bots (ChatGPT, Google, Anthropic & Perplexity)

If you prefer to limit or block AI bots like ChatGPT, Google’s AI, Anthropic, and Perplexity from crawling your site, the easiest way to do this is to update your robots.txt file.

This guide provides step-by-step instructions to prevent these AI bots from accessing your content, helping you maintain control over your web presence and protect your data.

Blocking ChatGPT (OpenAI)

OpenAI has two separate user agents (also known as bots or web crawlers):

  1. GPTBot:
    • Purpose: Used for web crawling.
    • Usage: Crawls web pages to gather data that can be used to improve future AI models. It avoids sites requiring paywall access, are known to primarily aggregate personally identifiable information (PII), or have text that violates OpenAI’s policies.
  2. ChatGPT-User:
    • Purpose: Used by plugins in ChatGPT.
    • Usage: Takes direct actions on behalf of ChatGPT users, answering live queries by accessing specific web content. It does not automatically crawl the web.

To block both of these AI crawlers, add these lines to your robots.txt file:

User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User 
Disallow: /

This will prevent ChatGPT from crawling your site or from users accessing your website directly using a ChatGPT plugin.

Blocking Anthropic

To block Anthropic’s AI bot crawler, add this line to your robots.txt file:

User-agent: anthropic-ai
Disallow: /

Blocking Google’s AI Bot

To block Google’s AI bot crawler, Gemini (formerly known as Bard), add this line to your robots.txt file:

User-agent: Google-Extended
Disallow: /

Blocking Perplexity AI

To block Perplexity’s AI bot crawler, add this line to your robots.txt file:

User-agent: PerplexityBot
Disallow: /

Important Notes

  • While these companies report that they respect robots.txt files, many users have reported that these AI companies are ignoring them.
  • These may not block plugins or extensions to these models that can crawl URLs. For example, there are popular GPT plugins for OpenAI’s ChatGPT that allow you to directly crawl a URL. These plugins have their own User-agent, which isn’t necessarily covered by the rules above.

Legal Genie Team

...