By Javier Surasky
.png)
The open memory that fed generative AI
Common
Crawl becomes part of the AI power map as an open web-data infrastructure that,
without producing models of its own or offering commercial artificial
intelligence services, became one of the most important sources for training
generative AI systems.
Its power
lies in its ability to transform fragments of the public web into a reusable
archive on a planetary scale: Common Crawl collects, organizes, and publishes
web data that is later downloaded, filtered, cleaned, and recombined by
universities, laboratories, technology companies, and model developers.
The
organization, which describes itself as a nonprofit created in 2007 and
dedicated to providing web archives and datasets to the public free of charge,
reports that it maintains data collected since 2008, with a database of more
than 10 petabytes, publishing crawls roughly once a month with more than two
billion pages per crawl.
Seen as
part of the AI ecosystem, it reveals that much of generative AI rests on
algorithmic innovation, the availability of GPUs, and also access to a “prior
memory of the web” that ultimately became a material condition of possibility
for the development of models such as ChatGPT, Gemini, Claude, and Llama.
It was
precisely the release of GPT-3 that brought Common Crawl into public view; its
training data included a filtered version of Common Crawl with 410 billion
tokens, which accounted for 60% of the training mix (Brown et al.).
The paper
by Brown et al. drew attention to the dataset and to its place in the
development of large language models, and a Mozilla Foundation report argues
that Common Crawl “laid the infrastructural foundation” for the rise of
generative AI, after showing that at least 64% of 47 large language models
published between 2019 and 2023 were trained on filtered versions of its data.
This
position allows Common Crawl to be defined as an actor of indirect
infrastructural power: it does not decide which models are created or what
training practices are used, but it provides a layer of massive data
availability without which many actors would not be able to access web data at
a comparable scale.
Its power,
then, lies in reducing barriers to entry for some actors while, at the same
time, feeding the concentration of value in others.
The paradox
is clear: Common Crawl democratizes access to web data, but those data can end
up incorporated into closed models, proprietary products, and commercial
services controlled by companies that concentrate compute, capital, technical
talent, distribution channels, and legal capacity: an open infrastructure
feeding a closed market.
This
tension is part of a broader debate over digital commons, where open access to
shared resources is understood to expand research, innovation, and
accountability, but can also facilitate new forms of extraction on a massive
scale when actors with unequal capacities use a commons to capture private
benefits, something AI has amplified.
Common
Crawl states that its goal is to democratize data so that research and analysis
are not available only to large companies, and that it offers a copy of the
Internet at no cost to researchers, companies, and individuals—data that have
been used, for example, to improve translation software, predict trends, and
track the spread of diseases.
That
narrative of openness is powerful: without Common Crawl, much academic,
independent, or small-scale research on issues such as the web, language,
security, or disinformation, among others, would depend more heavily on private
or inaccessible databases, which in turn would affect transparency.
The issue,
therefore, is not to take a position for or against Common Crawl—which would
also make no sense and have no practical effect—but rather to ask under what
conditions an open web-data infrastructure such as Common Crawl can be
maintained when its downstream uses are integrated into industries of extremely
high economic value, strong technical opacity, and growing social impact.
The
controversies over copyright, text and data mining, consent, exclusion, and
compensation are another facet: in a submission prepared in response to a
United Kingdom consultation on copyright and AI, Common Crawl argued that the
“right to read” should include the “right to mine,” and that the mere reading,
listening to, or computational analysis of materials should not be equated with
a public communication of protected works. It also stated that it respects
robots.txt, does not circumvent paywalls, does not log in to private sites, and
crawls at a measured pace to avoid overloading servers.
On the
other side, publishers, media outlets, and rights holders challenge the idea
that the public availability of content amounts to authorization for its
extraction, storage, technical redistribution, or incorporation into AI
training pipelines: in 2024, Danish media grouped through the Danish Rights
Alliance demanded that Common Crawl stop copying their content and remove
articles already included in previous datasets. Wired reported that Common
Crawl agreed to comply with the request and that its executive director said
the organization was not equipped to litigate against large media companies.
The
controversy deepened with journalistic investigations that accused Common Crawl
of facilitating AI companies’ access to protected or paywalled articles: The
Atlantic argued in 2025 that Common Crawl had opened a pathway for AI
companies to train models on media articles without consent, a claim that
Common Crawl publicly denied.
This
disagreement reveals a central dimension of digital power, expressed in the
transformation of entire ecosystems of cultural, journalistic, and scientific production into computational raw material.
Its
location on AWS, Amazon’s cloud, adds another layer to the power map, since
Common Crawl reports that its data are hosted there through the AWS Open Data
Sponsorship Program, reinforcing the paradoxical dimension of the case.
In
conclusion, Common Crawl is one of the most important and least visible actors
in the AI power map because it organizes a prior condition of machine learning:
the massive availability of web data. Its public narrative rests on openness,
research, innovation, and the democratization of access, but its critics point
to extraction, opacity, insufficient consent, and harm to economic models based
on content production.
The basis
of Common Crawl’s power in the AI ecosystem lies in those ambivalences.
Basic facts
- Common Crawl Foundation is a 501(c)(3) nonprofit organization founded in 2007 by Gil Elbaz, its current Chairman of the Board. The position of Executive Director of the organization has been held since 2023 by Rich Skrenta.
- Its main funding comes from the Elbaz Family Foundation, while the hosting of its data is covered by Amazon Web Services through the AWS Open Data Sponsorship Program. In recent years, it has also received donations from organizations, including some AI companies, to support the operating and maintenance costs of its public archive.
- It began collecting web data in 2008 and maintains an open repository of web data exceeding 10 petabytes.
- Its crawls are published approximately once a month and usually include more than two billion pages. Its crawler identifies itself as CCBot, uses robots.txt directives, and publishes data in formats such as WARC, WAT, and WET.
- Its main strength is offering open access to massive-scale web data, reducing barriers for research, natural language processing, web analysis, cybersecurity, digital preservation, and model development.
- Its main vulnerability lies in disputes over copyright, consent, opt-out, traceability, unequal representation of the web, possible inclusion of protected content, and dependence on private cloud infrastructures to host and distribute its archives.
- The central conflict posed by Common Crawl is how to preserve the public value of an open memory of the web without turning it into a channel of unlimited extraction for commercial AI systems.
References
Baack, S.
(2024). Training data for the price of a sandwich: Common Crawl’s impact on
generative AI. Mozilla Foundation. https://assets.mofoprod.net/network/documents/Common_Crawl_Mozilla_Foundation_2024.pdf
Baack, S.
(2024). A critical analysis of the largest source for generative AI training
data: Common Crawl. In Proceedings of the 2024 ACM Conference on Fairness,
Accountability, and Transparency. Association for Computing Machinery. https://doi.org/10.1145/3630106.3659033
Brown, T.
B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan,
A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger,
G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C.,
Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J.,
Berner, C., McCandlish, S., Radford, A., Sutskever, I., & Amodei, D.
(2020). Language models are few-shot learners. Advances in Neural
Information Processing Systems, 33, 1877–1901. https://doi.org/10.48550/arXiv.2005.14165
Common
Crawl Foundation. (n.d.). About. Common Crawl. https://commoncrawl.org/about
Common
Crawl Foundation. (n.d.). Frequently asked questions. Common Crawl. https://commoncrawl.org/faq
Common
Crawl Foundation. (2025a, March 3). Submission to the UK’s Copyright and AI
Consultation. Common Crawl. https://commoncrawl.org/blog/submission-to-the-uks-copyright-and-ai-consultation
Common
Crawl Foundation. (2025b, September 17). Common Crawl Foundation Opt-Out
Registry. Common Crawl. https://commoncrawl.org/blog/common-crawl-foundation-opt-out-registry
Common
Crawl Foundation. (2025c, November 6). Setting the record straight: Common
Crawl’s commitment to transparency, fair use, and the public good. Common
Crawl. https://commoncrawl.org/blog/setting-the-record-straight-common-crawls-commitment-to-transparency-fair-use-and-the-public-good
Meaker, M.
(2024, June 13). Publishers target Common Crawl in fight over AI training data.
Wired. https://www.wired.com/story/the-fight-against-ai-comes-to-a-foundational-data-set/
Reisner, A.
(2025, November 4). The company quietly funneling paywalled articles to AI
developers. The Atlantic. https://www.theatlantic.com/technology/2025/11/common-crawl-ai-training-data/684567/