Data Provenance Initiative

Multi-institution audit of the domains behind C4, RefinedWeb and Dolma, with longitudinal robots.txt and terms-of-service collection. The one non-commercial measurement of AI crawler restriction.

Coverage

Training corpora and web data consent

Website

www.dataprovenance.org

Published method

Open the original document