Your robots.txt file — the small text file that tells automated visitors what they may and may not access — currently contains rules that tell AI assistant crawlers to stay away. These are the crawlers behind tools like ChatGPT, Claude, Perplexity and Google's AI features. When they are blocked, they cannot read your pages, so your site won't be quoted or recommended when someone asks one of these tools about your topic.
Why it matters
More people now ask an AI assistant a question instead of searching directly. If those assistants can't read your pages, your business simply won't appear in their answers, even when you're the best match. Allowing them is how you stay visible in that growing channel — though blocking them remains a legitimate choice if you'd rather your content not be used this way.
How to fix it
Your robots.txt lives at the root of your domain (e.g. https://example.com/robots.txt). You are looking for lines that name AI crawlers under a Disallow rule. The main AI crawler names are:
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
PerplexityBot
Google-Extended
To allow them, remove any block like this:
User-agent: GPTBot
Disallow: /
Or replace it with an explicit allow:
User-agent: GPTBot
Allow: /
WordPress
Many SEO plugins (Yoast, Rank Math, All in One SEO) manage robots.txt for you. Go to the plugin's tools/file editor section and edit the robots.txt content there. If no plugin manages it, you can edit the physical file via your hosting file manager or FTP at the site root.
Nginx
robots.txt is usually a static file in your web root (often /var/www/html/). Edit it directly:
nano /var/www/html/robots.txt
Save the file — no server reload is needed for static files.
Apache
Same as Nginx: edit the static robots.txt in your document root (often /var/www/html/ or public_html/):
nano /var/www/html/robots.txt
Cloudflare
Cloudflare has a bot-management setting that can block AI crawlers independently of robots.txt. In the dashboard go to Security → Bots and check whether Block AI bots / AI Scrapers and Crawlers is enabled. Turn it off to allow access.
How to check it worked
Open https://yourdomain.com/robots.txt in your browser and confirm the AI crawler names no longer appear under a Disallow: / rule.