Reddit and cloaking robots.txt – analysis and tests

A reminder: what I found in 2024

I first came across Reddit’s robots.txt cloaking back in August 2024 (PL), where a standard request to reddit.com/robots.txt returned:

User-agent: *
Disallow: /

Google’s Rich Results Test tool from the same period showed a file that allowed indexing, with dozens of detailed rules for various sections of the site.

# Our robots.txt is for search engines80legs
User-agent: 008
Disallow: /
80legs' new crawler
User-agent: voltron
Disallow: /
User-Agent: bender
Disallow: /my_shiny_metal_ass
User-Agent: Gort
Disallow: /earth
User-agent: MJ12bot
Disallow: /
User-agent: PiplBot
Disallow: /
User-Agent: *
Disallow: /.json
Disallow: /.json-compact
Disallow: /.json-html
Disallow: /.xml
Disallow: /.rss
Allow: /discover.rss
Allow: /r/.rss
Disallow: /r//search.rss
Disallow: /r//comments/.rss
Disallow: /r//config/.rss
Disallow: /r//wiki/.rss
Disallow: /.i
Disallow: /.embed
Disallow: //comments/?sort=
Disallow: /comment/
Allow: /r//comments///de/
Allow: /r//comments///es/
Allow: /r//comments///fr/
Allow: /r//comments///pt/
Allow: /r//comments///it/
Allow: /r//comments///video/
Disallow: /r//comments////*
Disallow: /r//submit$
Disallow: /r//submit/$
Disallow: /message/compose*
Disallow: /api
Disallow: /post
Disallow: /submit
Disallow: /goto
Disallow: /before=
Disallow: /user/after=
Disallow: /u/after=
Disallow: /domain/t=
Allow: /login
Disallow: /remove_email/t2_
Disallow: /r//user/
Disallow: /gold?
Disallow: /search
Disallow: //search?
Disallow: //search/?
Disallow: //search$
Disallow: //search/$
Disallow: /search.compact$
Disallow: //search.compact$
Allow: /r//comments//search/$
Allow: /r//comments//search$
Disallow: /static/button/button1.js
Disallow: /static/button/button1.html
Disallow: /static/button/button2.html
Disallow: /static/button/button3.html
Disallow: /subreddits/*
Disallow: /buttonlite.js
Disallow: /timings/perf
Disallow: /counters/client-screenview
Disallow: /?feed=
Disallow: /svc/shreddit/
Disallow: /svc/sh/
Disallow: /svc/web/*
Disallow: /svc/events/preload-header
Disallow: /graphql
Disallow: /errors$
Disallow: /live/*
Disallow: /mediaembed/*
Disallow: /media
Disallow: /gallery/*
Disallow: /poll/*
Allow: /
Allow: /sitemaps/.xml
Allow: /posts/
Disallow: /answers/*
Allow: /answers$
Allow: /answers/$
User-agent: AdsBot-Google
Disallow: /search*
Disallow: //search?
Disallow: //search/?
Disallow: //search$
Disallow: //search/$
Disallow: /search.compact$
Disallow: /*/search.compact$

Reddit’s robots.txt file differs only slightly from the 2024 version.

As it turns out, the following lines have disappeared:

disallow: /login
disallow: /search
disallow: /search$
disallow: /search?q=
disallow: /search?title=

New directives have also been added:

User-agent: AdsBot-Google
Disallow: /search*
Disallow: /*/search?
Disallow: /*/search/?
Disallow: /*/search$
Disallow: /*/search/$
Disallow: /search.compact$
Disallow: /*/search.compact$

Back in 2024, I said Reddit was engaging in cloaking. Two years later, it turns out this is a multi-layered mechanism.

4 versions of 1 file

I checked directly via curl how robots.txt looks depending on the User-Agent header.

1. Block everyone, i.e. the default robots.txt
2. A dedicated robots.txt for archive.org, allowing archiving of ONLY Reddit’s homepage thanks to the Allow: /$ directive
3. Social media platforms get just enough to generate a link preview
4. A whitelist of parties with full access to the Reddit website

Default variant – blocks everyone:

curl -A "Claudebot" https://www.reddit.com/robots.txt
curl -A "Seekio.pl" https://www.reddit.com/robots.txt

Both commands return an identical file, which I also confirmed directly in the browser:

# Welcome to Reddit's robots.txt
# Reddit believes in an open internet, but not the misuse of public content.
# See https://support.reddithelp.com/hc/en-us/articles/26410290525844-Public-Content-Policy
# See https://www.reddit.com/r/reddit4researchers/ for details on how Reddit continues to support research and non-commercial use.

User-agent: *
Disallow: /

This is the version that every random bot and every random script sees – zero access to anything. But…

Variant for Internet Archive – homepage only:

curl -A "archive.org_bot" https://www.reddit.com/robots.txt

This already shows something different:

# Allowed for archiving
# =====================
User-Agent: *
Allow: /$
Disallow: /

This is a significant change over time. In an update to my previous article, I wrote that since December 2024, Reddit had completely blocked archive.org. Checking archived versions from early 2025 shows just a plain Disallow: /, with no exception at all. The current, partial version that grants access to the homepage alone (Allow: /$ – an exact match for the root URL and nothing beyond it) is a separate, later change.

The Wayback Machine Internet Archive has not respected robots.txt since 2017

Back in April 2017, Internet Archive published a post¹ explicitly stating that it was moving away from honoring robots.txt files when deciding what to archive. They started with U.S. government and military domains, but announced plans to do this „more broadly.”

Interestingly, social media platforms are served a different robots.txt.

Variant for link preview crawlers (Meta, X):

curl -A "facebookexternalhit" https://www.reddit.com/robots.txt
curl -A "Twitterbot" https://www.reddit.com/robots.txt

Testing curl with the Meta and X.com User-Agents shows just enough to generate a link preview when sharing a post, but it’s still far from full access.

# Allowed for social sharing
# ==========================
User-Agent: *
Disallow: /
Allow: /r/*
Allow: /oembed*
Allow: /avatar/*
Allow: /user/*
Allow: /u/*
Allow: /tldr/*
Allow: /best/*
Allow: /register/
Allow: /login/
Allow: /media
Allow: /c/*
Allow: /$

And of course, the classic – a whitelist with full access.

Full variant (whitelist) – accessible to Google and OpenAI

The very same file that in 2024 was only visible to Google’s cache, I can now reproduce in 2 independent ways:

1. Google Rich Results Test
2. a direct query to ChatGPT via prompt

„give me the contents of reddit.com’s robots.txt file„

I repeated the ChatGPT test both while logged into an account and in incognito mode while logged out. The result was identical in both cases!

Both methods return an identical, fully fleshed-out file. This means that the whitelist version is actively maintained and developed, and visible only to select parties. Let me just remind you that Reddit’s data-licensing revenue in Q2 2026 grew by 24% to $43 million, and at the top of the list of buyers sit OpenAI and Google²

2 layers of cloaking

Now I get to the most important discovery of this analysis. I tried spoofing the user-agent and got some interesting information:

curl -A "Googlebot" https://www.reddit.com/robots.txt
curl -A "GPTBot" https://www.reddit.com/robots.txt
curl -A "bingbot" https://www.reddit.com/robots.txt

All 3 requests returned an identical block page:

<h1>whoa there, pardner!</h1>
<p>Your request has been blocked due to a network policy.</p>

Interestingly, the block pages differed in one detail – the ticket code at the bottom of the page:

This difference suggests that requests aren’t being rejected at one single, shared location. It’s possible that a spoofed Googlebot is blocked earlier – at the CDN edge level, before it even reaches the application layer that generates the ticket – while GPTBot and bingbot get a bit further and are only rejected there.

These tests show that Reddit employs 2 independent layers of cloaking:

1. Network layer (WAF) – for the most „attractive” identifiers to spoof (Googlebot, GPTBot, bingbot), the User-Agent header alone isn’t enough. The system likely also verifies IP or rDNS.
2. File content layer (classic cloaking) – for the remaining identifiers (Claude, archive.org, facebookexternalhit, Twitterbot, any random string), the User-Agent header alone is enough to get 1 of the previously described file versions. There’s no network-level verification here at all.

Reddit does not trust the mere User-Agent declaration in the case of bots that are most valuable to it and, at the same time, most profitable to impersonate – but for everyone else, standard content-level cloaking is entirely sufficient. In my opinion, this discovery is quite interesting.

Bing – a block confirmed via an independent method and measured in the data

In 2024, I wrote that Bing’s mobile-friendly test tool showed an error when trying to analyze Reddit. I checked this again:

curl -A "bingbot" https://www.reddit.com/robots.txt

I tried multiple times to get information from Bing’s mobile-friendly test tool, but the only thing I got was a short message:

We couldn't retrieve information about this page. Please check the URL and try again.

For comparison, the same tool analyzes other websites without any issues. The problem only appears when trying to check Reddit’s robots.txt.

The site: operator to the rescue!

This block can be measured numerically by comparing the site:reddit.com operator’s results in Google and Bing.

While Google’s index grew by nearly 80%, Bing’s index has essentially stood still. This is exactly what you’d expect when 1 crawler has full, uninterrupted access while the other is being blocked.

Common Crawl – a consistent block, month after month

I also checked how Common Crawl is doing – an open project that crawls the internet, and whose data is one of the primary training sources for many LLMs. It turns out Reddit has been consistently blocking CCbot the entire time, with no exceptions.

Interestingly, despite the total block, Common Crawl’s data still shows a small number of pages scanned on reddit.com. Depending on the month, this ranges from 306 to 871 pages. However, these aren’t pages actually fetched in defiance of the block, but rather just URLs discovered by the crawler through links from other, non-blocked pages – without the content at those addresses actually being retrieved.

This is consistent with the overall picture: access to Reddit’s training datasets is a commodity, and Common Crawl, as a free, publicly available resource, isn’t party to any licensing agreement.

Perplexity – partial access to metadata, not content

I separately tested Perplexity with the same prompt asking for the contents of robots.txt:

give me the contents of the robots.txt file at https://reddit.com/robots.txt

Perplexity told me it couldn’t retrieve the file because Reddit’s server rejected the request. Makes sense, given the „disallow” – but… I then asked it to summarize a specific, fresh thread from a few hours earlier:

find the thread „Popular high-end outdoor clothing company has horrible AI copy” on reddit.com and give me a summary.

The response was partially accurate: Perplexity correctly identified the subreddit (r/SEO), the post’s flair, and the author’s username. At the same time, it openly admitted that it didn’t have access to any specific quotes or to the brand name mentioned in the post – even though the name appears explicitly in the thread’s body, not just in the title.

Perplexity had access to the post’s metadata – title, author, subreddit, and timestamp – but not to the full content or the comments.

I checked the same Reddit thread in Google and Bing:

• Google – indexed
• Bing – not indexed

So the only explanation I have is that Perplexity used what it found in Google’s SERP. This is further supported by the fact that its reasoning process explicitly shows the use of the site operator:

"Popular high-end outdoor clothing company has horrible AI copy" site:reddit.com

Summary: cloaking as a tool for commercial segmentation

Putting all the tests together gives a consistent picture:

What in 2024 looked like one file for Google and another for everyone else, turns out – two years later – to be Reddit serving different robots.txt files depending on who is actually paying, or holds a licensing agreement, on the other end of the connection.

1. Archive.org has been ignoring robots.txt since 2017 – https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ ↩︎
2. Reddit revenue grew 24% to $43 million, led by OpenAI and Google – https://investor.redditinc.com/news-events/news-releases/news-details/2026/Reddit-Reports-Second-Quarter-2026-Results/default.aspx ↩︎