How ChatGPT, Gemini and Copilot choose their sources
OpenAI has three bots, and each one is set separately in the robots.txt. OAI-SearchBot is the one that makes websites appear in ChatGPT search: if you block it, your website won’t appear in search answers, although it can still show up as a navigation link. GPTBot is the one that trains the models, and blocking it doesn’t take you out of search. ChatGPT-User is the one that visits a page when a person asks for it in a conversation; OpenAI says that, because a person is asking, the robots.txt may not apply. All three have their IP lists published.
Google’s AI Overviews and AI Mode come from the search engine’s index, the one built by Googlebot: to appear in them, a page has to be indexed and eligible to show with a snippet. Since 31 August 2026, Search Console also has a control to appear or not in the search engine’s generative AI features, switched on by default. Google-Extended is something else: it decides whether the content can be used to train the Gemini models and to ground the answers of the Gemini app and Vertex AI, and Google says it affects neither inclusion nor ranking in Google Search.
At Microsoft, Bing, Copilot and the answers that use the Bing API share the same crawling and index, Bingbot’s. A “noarchive” meta tag takes the content out of Copilot’s answers, and “nocache” limits it to the address, the title and the snippet.
The practical conclusion: if Google and Bing haven’t indexed you, the assistants are unlikely to cite you.
What we saw on our own websites
We looked at the server access logs (Amplify) of seven of our own websites, from 18 August to 18 October 2026: sanfeliusoftware.com, joansanfeliu.com, formacio.joansanfeliu.com, social.joansanfeliu.com, cv.joansanfeliu.com, taula.ai and getguestproof.com. That’s 62 days of data, with no cookies or scripts, and every bot was verified by checking whether its IP is on the list its company publishes.
Few visits, but highly qualified ones. People arriving from an assistant: 8 in total, 6 from ChatGPT, 1 from Perplexity and 1 from Gemini. By website: 4 to taula.ai, 2 to joansanfeliu.com, 1 to getguestproof.com and 1 to sanfeliusoftware.com.
The one from Gemini was on 18 October 2026 at 13:41, on the sanfeliusoftware.com home page. Four minutes later, a request came in for a quote for custom software. They didn’t tell us how they had found us, but the timing matches. Matching doesn’t mean it was the cause: that’s why we now ask everyone who writes to us.
taula.ai, which has pricing pages, city pages and “alternative to” pages for other software, is the website that gets the most visits from ChatGPT: one went straight to the pricing of the booking software. On top of that, a person from the United States wrote to tell us they had found taula.ai on ChatGPT.
On sanfeliusoftware.com, on the other hand, OAI-SearchBot, ChatGPT’s search bot, made 45 requests in the last 30 days (from 19 September to 18 October 2026). The 42 that came from verified IPs all went to the robots.txt: it didn’t read a single content page, even though the robots.txt allows it. In the same period, GPTBot (the training one) made 338 requests, and Googlebot, 967.
Many “bots” aren’t bots. The table shows the requests that claim to be a bot but come from an IP that isn’t on its company’s official list, out of the requests that could be checked:
| Bot | Fake | Checked | % |
|---|---|---|---|
| PerplexityBot | 1,200 | 1,260 | 95% |
| ChatGPT-User | 1,150 | 1,421 | 81% |
| OAI-SearchBot | 714 | 1,800 | 40% |
| GPTBot | 749 | 2,604 | 29% |
| ClaudeBot | 230 | 1,624 | 14% |
| Googlebot | 486 | 3,659 | 13% |
| Bingbot | 59 | 1,138 | 5% |
The seven websites, from 18 August to 18 October 2026.
Many of the fake ones aren’t looking for pages, but for secret files. Across the seven websites there were 56,923 requests looking for files such as .env, .git, keys or credentials, and 4,471 posed as a well-known AI or search engine bot. None of them found anything exposed.
“Direct” traffic hides visits
Of the sessions on sanfeliusoftware.com in the last 30 days, 86% arrive as “direct”, with no source. That’s where visits that don’t say where they come from end up, as often happens with apps. So the visits you see from an assistant are a minimum: there may be more hidden in “direct”.
What helps, in order
- Let the bots that matter in through your robots.txt. Watch out for the trap: a named group (for example, “User-agent: GPTBot”) replaces the “*” group for that bot rather than adding to it; if you don’t repeat the “*” rules there, that bot doesn’t follow them. And blocking GPTBot only affects training: it doesn’t take you out of ChatGPT search, which depends on OAI-SearchBot.
- Check your CDN, firewall or hosting. Some block AI bots without the owner knowing. Cloudflare blocked them by default on new domains from July 2025, and since 15 September 2026, on new domains, it blocks training and agent bots on pages with ads and lets search bots through. Vercel, Fastly and Akamai also have options to block them.
- Have your content in the HTML, not only in JavaScript. Some bots read the HTML as it arrives, without running JavaScript: if the text is drawn by the browser, they don’t see it.
- Make pages that answer a specific question with data: prices, comparisons, alternatives and location. That’s what the pages we’ve seen getting visits from ChatGPT have in common.
- Keep your company details the same everywhere: your Google Business Profile, directories and LinkedIn, with the same name, the same address and the same phone number.
- Be indexed: Google Search Console and Bing Webmaster Tools, with the sitemap submitted, and IndexNow to tell Bing and the other search engines that accept it about changes (Google isn’t one of them). In Search Console, also check that your website has the generative AI features switched on.
What you don’t need
- llms.txt. We have one because it costs nothing, but Google’s 2026 guide on generative AI features says no special file is needed and that Google Search doesn’t use it. No major assistant has confirmed that it reads it: neither OpenAI, nor Anthropic, nor Perplexity says so on the pages where they explain their bots.
- Content-Signal. It’s the robots.txt line that says whether you allow your content to be used for search, for answering or for training. It’s a stated preference, not a lock: Cloudflare itself, which proposed it, says it doesn’t technically prevent anything.
- Paid “GEO tricks” and pages mass-produced with AI. According to the same Google guide, optimising for the search engine’s AI features is still SEO, and no special markup is needed.
How to measure it
- Look at sources such as chatgpt.com, gemini.google.com and perplexity.ai. In our logs, visits from ChatGPT usually carry utm_source=chatgpt.com in the address.
- Many visits that come from an app don’t say where they come from and show up as “direct”.
- Ask everyone who writes to you how they found you. It’s the only way to know for sure.
- Verify bots by IP, not by name: in our logs, 95% of the checked requests claiming to be PerplexityBot were fake.
- Look at the server logs: bots don’t run the analytics JavaScript, so your usual analytics doesn’t see them.
Check your website
We’ve built a free tool that tells you whether AI assistants can read your website: the robots.txt for thirteen bots, the home page as a bot that doesn’t run JavaScript sees it, the sitemap and the CDN. It doesn’t tell you whether ChatGPT or Gemini recommend it, because that depends on many more things and changes every day.