Major News Organizations Are Blocking OpenAI's Crawler

According to Richard Fletcher, director of research at the Reuters Institute, the websites of traditional print publications (like The New York Times and Der Spiegel) were more likely to block AI crawlers than TV and radio stations or digital news sites, with about 57% of them doing so.

News sites were less likely to block Google's AI crawler than OpenAI's, with less than a quarter doing so, but “almost every site (97%) that decided to block Google's AI crawler also blocked OpenAI's crawler.”

The percentage of top online news sites blocking OpenAI ranges from 79% in the US to just 20% in Mexico and Poland. Meanwhile, the percentage blocking AI crawlers for Google ranges from 60% in Germany to 7% in Poland and Spain.

In every country except Germany, top news sites blocked OpenAI’s crawler more than Google’s. Furthermore, almost every site that blocked Google AI also blocked OpenAI (97%).

This could be because ChatGPT is more prominent and widely used than Gemini (Bard's new name), or it could be because the OpenAI crawler was released first.

But it's also possible that news organizations are more cautious about blocking Google out of concern that it could affect their priority recommendations in the search engine's results.

Mai Anh (according to Reuters)