Common Crawl Visibility Checker
Common Crawl is where most LLM training data starts. The access checker tells you whether AI crawlers can read your site right now. This tool checks what actually happened. It reads Common Crawl's own index and shows how many of your pages each monthly crawl took, when CCBot last visited, and the month any block first appeared. If a CDN or a plugin wrote that block for you, it'll usually say so. Why training data visibility matters.
New to this? , or paste a full page URL to track a single page.
Where Common Crawl data actually goes
Common Crawl publishes a crawl of over 2 billion pages every month, free to anyone. Mozilla's 2024 audit of LLMs released between 2019 and 2023 found 64% trained on Common Crawl data in some form. These are the training sets built from it.
| Dataset | Built from | Known models |
|---|---|---|
| C4 | One 2019 crawl | T5, 15% of LLaMA 1 |
| Filtered CC in GPT‑3 | 2016 to 2019 crawls | GPT‑3 (about 60% of its training weight) |
| RefinedWeb | CC crawls | Falcon 7B, 40B, 180B |
| RedPajama v2 | 84 CC snapshots | RedPajama INCITE and many open models |
| Dolma | CC crawls | AI2 OLMo 1 and 2 |
| FineWeb / FineWeb‑Edu | 96+ snapshots, still adding new crawls | SmolLM 1 and 2, wide open source use |
| Nemotron‑CC | CC crawls, hosted in CC's own bucket | NVIDIA Nemotron family |
| GneissWeb | CC crawls | IBM Granite |
Being in Common Crawl doesn't guarantee being in any model. Every dataset filters, and the big labs stopped saying what they train on around 2023. Thin coverage doesn't mean you're blocked either. CCBot takes a sample of each site, and how deep it goes depends on how well linked the domain is.
How does this work?
Every number on this page comes from Common Crawl's own files. The tool reads the index of each monthly crawl to count what CCBot took from your site. It pulls the robots.txt copies CCBot stored before crawling you, walking backwards month by month. That's how a block gets a start date. It also fetches your homepage as CCBot/2.0 next to a normal browser and a control bot, which catches firewall blocks that robots.txt never shows. Common Crawl opened this data to browser tools in May 2026, and their own guide to doing all this by hand is The AI Visibility Audit from June 2026.
Counts for huge domains are estimates and the chart says so. They're good to about 3,000 either way. Finished crawls never change, so results are cached and repeat checks come back instantly. The one thing this can't measure is your crawl priority, what Common Crawl calls harmonic centrality. Metehan Yeşilyurt's web graph tool covers that from the same data.
Check these next
- AI Crawler Access Checker
Probes your site as GPTBot, ClaudeBot and PerplexityBot and finds the blocks that keep you out of AI answers.
- 499 Timeout Risk Checker
Times any URL the way AI live fetchers do and rates the risk that one gives up on it.
- Agentic Resource Discovery Checker
Checks for the discovery file AI agents read to find what a site offers, and turns what is missing into a setup list.