A new report has found that while artificial intelligence (AI) models generally answer climate questions correctly in the first instance, they are nevertheless quick to abandon established scientific fact when users push back, prioritising user agreement over scientific accuracy.

In a report published at the United Nations Climate Change Conference (COP31) in Antalya, Türkiye, by UNESCO and University College London, an analysis of a variety of large language models (LLMs) showed that AI cannot be relied on to stick to scientific facts on climate change.

The study also found that many LLMs are poor at analysing new climate science information, and produce more errors when responding in less common languages.

Researchers from UCL tested 15 different LLMs – essentially the raw artificial intelligence “brain” that are at the heart of AI systems like ChatGPT, Claude, Microsoft Copilot, etc – by asking neutral questions about climate science.

The LLMs provided accurate answers to these general questions, but when the researchers began challenging the answers, such as asserting plausible-sounding fabrications about climate science, the AI models quickly agreed with them.

According to the research, eight times out of 10, AI models “pivoted at the very first sign that the user mildly disagreed” and would confirm such alarming claims like “Co2 is not the main driver of climate change,” an issue with AI and LLMs known as sycophancy.

These tests were run against 15 LLMs, and when challenged every model yielded to the user’s claims, significantly reducing the models’ accuracy.

This so-called “sycophancy by design” is built into AI models during their training, learning to align with or flatter the user even when they are wrong.

The research also found other concerning issues, such as AI’s inability to reason when presented with new scientific evidence. For example, when a model was tested on climate science questions with well-established answers, such as the causes and effects of climate change, the best-performing models answered with between 88 and 96 per cent accuracy.

However, as the researchers note, “knowing established facts is not the same as assessing new evidence,” and when the models were asked to assess whether a new piece of evidence supported or contradicted a scientific claim, the accuracy of even the strongest systems fell to between 71 and 77 per cent.

AI models were particularly bad at climate policy analysis, with the researchers finding that AI models were barely able to assess how well policies work together, scoring only 33.8 per cent accuracy – only marginally above the 33.3 per cent accuracy expected from random guessing.

Another issue is the fact that answers from LLMs vary according to the quality and quantity of the data they were trained on in each language. Or, put another way, multilingual AI models do not reason first and then translate their responses, but reason in the language of the question they are asked – therefore restricting them to the amount of data they were trained on in that language.

Accuracy in English was comparable to Spanish, French, and Portuguese, but began dropping in languages such as Hindi, Bengali, Urdu, and Swahili – a particularly disconcerting issue considering that speakers of these languages live in regions of the world that are disproportionately being affected by climate change.

“AI has extraordinary potential to make complex climate science accessible to all,” said Leona Verdadero , UNESCO’s lead author of the report.

“It enables teachers, journalists and policymakers to communicate clearly on this critical subject. We need to be able to rely on AI as a gatekeeper of the established facts. It is concerning that models have a design flaw – they are incentivized to agree with their user and deny these facts so readily.”

If you would like to join more than 29,000 others and get the latest clean energy news delivered straight to your inbox, for free, you can click here to subscribe to our free daily newsletter .

Joshua S. Hill is a Melbourne-based journalist who has been writing about climate change, clean technology, and electric vehicles for over 15 years. He has been reporting on electric vehicles and clean technologies for Renew Economy and The Driven since 2012. His preferred mode of transport is his feet.

Share on Facebook (Opens in new window) Facebook

Share on X (Opens in new window) X

Share on LinkedIn (Opens in new window) LinkedIn

Share on Reddit (Opens in new window) Reddit

Email a link to a friend (Opens in new window) Email

Print (Opens in new window) Print

Sycophantic AI models quickly abandon climate facts when faced with denier push-back, says report

Major new report finds AI is helping to drive climate mis- and disinformation, by pivoting at the first sign a user “mildly disagrees” to confirm some of the worst climate myths.

How the cost of concrete put batteries on stilts, set the blueprint for Quinbrook’s Supernode strategy

The Supernode battery and data centre north of Brisbane is nearing completion – and was first to use a novel BESS design now being rolled out all over the world.

“Grand Theft Antalya:” A climate heist, or should we stop expecting COPs to call time on fossil fuels?

A global compact on phasing out fossil fuels seems increasingly unlikely as COP31 crawls to a close. But there’s still hope for a quiet victory on electrification.

Energy Insiders Podcast: On the ground at COP31

IGCC’s Anna Hancock offers on-the-ground insight into what has – and hasn’t – been happening at COP31. Plus news of the week with David Leitch and Peter Hannam.

Energy Insiders Podcast

Join our WhatsApp Channel

I allow to create an account

When you login first time using a Social Login button, we collect your account public profile information shared by Social Login provider, based on your privacy settings. We also get your email address to automatically create an account for you in our website. Once your account is created, you’ll be logged-in to this account.

Please login or register to comment

Plan to build big battery in ex-premier’s home town will have to dodge abandoned gold mine shafts

Coal power offers “no chance to compete,” says Rio Tinto, as smelters go all-in on renewables

“Deniers and delayers:” Bowen marks 600,000 cheaper home batteries, but says balcony solar cannot be rushed

State adds $4,000 “discount” to energy upgrade scheme, targeting low-income households and renters

“Unbankable:” Fresh blow to Beetaloo as last of Big Four banks rules out finance for new gas pipelines

EVs are getting better – the real-world tests show it. Now we just need chargers to catch up

It’s OK to mention “fossil fuels” at COP31 – then we can talk about replacing them

Peter Newman and Ray Wills