Latest research from the Reuters Institute for the Study of Journalism (RISJ) has shown that attempts to request top news headlines on the popular AI chatbots ChatGPT and Bard (now Gemini) yielded unreliable results. In their study, over half of responses were not news-related, and the majority of AI responses began with the phrase “I’m unable to’’.
The RISJ analysed 4,500 headline requests collected from news outlets across ten countries. They found that ChatGPT returned non-news output 52-54% of the time, while only just 8-10% returned headlines that referred to relevant top stories on the outlet’s homepage. On the other hand, Bard returned non-news output in over 95% of requests.
ChatGPT and Bard both run on a large language model (L.L.M.), a type of AI programme that is trained using a wide range of samples from a dataset. This allows the AI to continuously improve its ability to recognise and interpret trends in complex data, such as human language.
The study found that when the chatbots responded with ‘’news-like output’’, responses appeared coherent, however, they tended to summarise articles inaccurately. Additionally, ChatGPT rarely provided direct links to headlines, only opting to 10% of the time. When a link was generally provided, in most cases this was a single link to the original newspublisher’s homepage
When consulted on the risks to users relying on AI chatbots for the latest news, Richard Fletcher, Director of Research for the RISJ told Cherwell: “According to our research, if people ask ChatGPT (the paid version that’s connected to the web) for the top news headlines from a specific outlet, they will often receive a response saying “I’m unable to do that”. Depending on what news outlets they ask about, it’s possible that some people won’t be able to use ChatGPT to get news from their preferred sources.”
Chatbots also displayed ‘’AI hallucination’’, where the algorithm perceives non-existent patterns in the data and thus interprets them inaccurately. In this case, 3% of ChatGPT outputs attributed exclusive stories to different outlets, and a further 3% were so vague they could not be attributed to existing stories.
Previous research by the Reuters Institute found that ‘’around half of the most widely used news websites were blocking ChatGPT by the end of 2023’’. This proved to be a great barrier to data-gathering, with the proportion of non-news output for these sites rising to 86% according to the study.
The research points out that while very few people currently use AI chatbots to get the news, it is ‘’highly likely that future generative AI tools will be connected to the web as standard’’, and as such the reliability of these tools to provide up-to-date information is essential.
 has shown that attempts to request top news headlines on the popular AI chatbots ChatGPT and Bard (now Gemini) yielded unreliable results.){kind=link}