ChatGPT ‘wrong on more than half of Google Analytics 4 FAQs’

AI powered-Bot was wrong on 52% of its answers, suggesting it's unlikely to change the face of the digital marketing industry any time soon.

Since exploding onto the scene in November 2022, ChatGPT has been one of the most hotly debated new tech trends, predicted to change the world of work. 

However, the bot is a long way from revolutionising the digital marketing landscape, after it answered only three in 10 (29%) FAQs about Google Analytics 4 (GA4) correctly, according to new research. 

With GA4 migration on the cards for many clients before the end of Universal Analytics in July, consultants from MarTech consultancy fifty-five tested ChatGPT’s capability. 

In a study of 42 FAQs about GA4, covering access to raw data, active users and behaviour modelling, ChatGPT was incorrect in more than half (52%) of its answers. 

On one in five (19%) questions, ChatGPT was ‘semi-correct’. In some cases, the bot misinterpreted the question and provided a factually correct answer to an entirely different question, showing issues with its comprehension where a human would have easily understood what was being asked. 

In others, it provided a small amount of misinformation in an otherwise correct response, which is less likely to be recognised as being incorrect and could lead to significant confusion. 

What happened when fifty-five tried replacing their consultants with ChatGPT? 

  • In the study of 42 FAQs about GA4, ChatGPT provided only 12 correct responses
  • One in five (19%) responses provided misleading answers
  • The bot currently uses data from 2021, which means that it is often out of date

“Remember that ChatGPT is a chatbot, not an oracle”

Sam Burr, consultant at fifty-five, said: “The potential of generative AI, like ChatGPT, is clear for all to see – and marketers will have to adapt to it in future, but for now they should think twice before using it to inform activity. It can point you in the right direction most of the time, but its responses should be followed up with additional research through alternative sources – it is important to remember that it is a chatbot, not an oracle.”

All AI is limited by its training data, and fifty-five’s study points to the age and quality of the data it collects as a reason that ChatGPT falls short on GA4. Currently, the bot uses data from 2021, which means that it is often out of date given how quickly and how frequently updates and bug fixes are made. 

At the same time, the bot makes no judgments about the quality of the data it uses. It does not rank data based on domain authority, which means data does not [always?] come from trusted sources, and are more likely to be incorrect.  

Given the importance of analytics for helping businesses to make marketing decisions and evaluating success, the research shows that, currently at least, generative AI is no replacement for marketing experts, and is unlikely to be for a long time, if ever.   

“It might be able to improve productivity, spark creativity or kick off a conversation, but, until this tech is more developed, it certainly wouldn’t pass probation at fifty-five,”  Burr concluded. 

Data source: