$ ccbot --history yourdomain.com
Common Crawl Visibility Checker
Common Crawl is where most LLM training data starts. The access checker tells you whether AI crawlers can read your site right now. This tool checks what actually happened. It reads Common Crawl's own index and shows how many of your pages each monthly crawl took, when CCBot last visited, and the month any block first appeared. If a CDN or a plugin wrote that block for you, it'll usually say so. Why training data visibility matters.
New to this? , or paste a full page URL to track a single page.
Where Common Crawl data actually goes
Common Crawl publishes a crawl of over 2 billion pages every month, free to anyone. Mozilla's 2024 audit of LLMs released between 2019 and 2023 found 64% trained on Common Crawl data in some form. These are the training sets built from it.
Being in Common Crawl doesn't guarantee being in any model. Every dataset filters, and the big labs stopped saying what they train on around 2023. Thin coverage doesn't mean you're blocked either. CCBot takes a sample of each site, and how deep it goes depends on how well linked the domain is.
How does this work?
Every number on this page comes from Common Crawl's own files. The tool reads the index of each monthly crawl to count what CCBot took from your site. It pulls the robots.txt copies CCBot stored before crawling you, walking backwards month by month. That's how a block gets a start date. It also fetches your homepage as CCBot/2.0 next to a normal browser and a control bot, which catches firewall blocks that robots.txt never shows. Common Crawl opened this data to browser tools in May 2026, and their own guide to doing all this by hand is The AI Visibility Audit from June 2026.
Counts for huge domains are estimates and the chart says so. They're good to about 3,000 either way. Finished crawls never change, so results are cached and repeat checks come back instantly. The one thing this can't measure is your crawl priority, what Common Crawl calls harmonic centrality. Metehan Yeşilyurt's web graph tool covers that from the same data.
Pair it with the AI Crawler Access Checker for GPTBot, ClaudeBot and PerplexityBot, the 499 Timeout Risk Checker for speed, or the Agentic Resource Discovery Checker for what AI agents can find on your site.