01/24/2024

Most Top News Sites Block AI Bots

Meanwhile, right-wing media welcome them

As media companies haggle licensing deals with artificial intelligence (AI) powerhouses like OpenAI that are hungry for training data, they’re also throwing up a digital blockade. New data shows that more than 88 percent of top-ranked news outlets in the U.S. now block web crawlers used by AI companies to collect training data for chatbots and other AI projects. One sector of the news business is a glaring outlier, though: Right-wing media lags far behind their liberal counterparts when it comes to bot-blocking.

Data collected in mid-January on about 40 top news sites by Ontario-based AI detection startup Originality AI shows that almost all of them block AI web crawlers, including newspapers like The New York Times, The Washington Post and The Guardian, general-interest magazines like The Atlantic,and special-interest sites like Bleacher Report. OpenAI’s GPTBot is the most widely-blocked crawler. But none of the top right-wing news outlets surveyed, including Fox News, The Daily Caller and Breitbart, block any of the most prominent AI web scrapers, which also include Google’s AI data collection bot. Pundit Bari Weiss’ new website The Free Press also does not block AI scraping bots.

Most of the right-wing sites did not respond to requests for comment on their AI crawler strategy, but researchers contacted by WIRED had a few different guesses to explain the discrepancy. The most intriguing: Could this be a strategy to combat perceived political bias? "AI models reflect the biases of their training data," said Originality AI founder and CEO Jon Gillham. "If the entire left-leaning side is blocking, you could say, come on over here and eat up all of our right-leaning content."

Please select this link to read the complete article from WIRED.