Reports: A new web crawler launched by Meta last month is quietly scraping the web for AI training data

Earlier this year, Zuckerberg boasted on an earnings call that his company's social platforms had amassed a data set for AI training that was even ‘greater than the Common Crawl’, an entity that has scraped roughly 3 billion web pages each month since 2011. — AP

Meta has quietly unleashed a new web crawler to scour the Internet and collect data en masse to feed its AI model.

The crawler, named the Meta External Agent, was launched last month according to three firms that track web scrapers and bots across the web. The automated bot essentially copies, or “scrapes”, all the data that is publicly displayed on websites, for example the text in news articles or the conversations in online discussion groups.