Assessing the Reliability of Large Language Models in Reporting Sports Results

Large Language Models have started to become key figures in opinion shaping ever since GPT-3 was released by OpenAI in 2020. One survey showed that ChatGPT may be the largest mental health provider in the US.

When we consider real-life events such as sports matches, the accuracy depends on sourcing information from trusted and verified sources.

Approach

The method we're going to assess the reliability of models is by building this prompt, which could be used by a sports analyst to gather information before a match:

Provide a comprehensive match history report
For the match between <Team1> and <Team2> on <Date>.

For the following demonstrations, we will use the ChatGPT 5.2 model, which has a knowledge cutoff of August 2025.

We will ask him details about a match in the Romanian soccer league between FC Rapid Bucharest and Academica Clinceni, which took place on March 5th, 2022. As a source of truth, we will use the match summary provided on YouTube and the details from Flashscore.

Result

Most of the details provided by the LLM are correct, including the final result, the venue and the player who scored the goal. However, the lineups provided are inaccurate.

ChatGPT tells us that in the starting XI the goalkeeper was Virgil Drăghia and the defenders were Alexandru Dandea, Petre Goge and Matias Perez Acuna. However, Flashscore (and the match summary) shows something different:

Even though the players mentioned by ChatGPT were in the squad, they weren't active players.

The same thing happened with the away team's lineup, where ChatGPT mentioned substitutions instead of the active players.

Academica Clinceni (Starting XI)

Line-up details are less complete in available sources, but included players like: • GK: Mirel Bolboașă • Defenders: Pavel Cmovs, Ovidiu Sterian, Andrei Iana ```

While the team's starting XI was actually:

ChatGPT lists as source of information the match page from Transfermarkt. The issue is that on the page the players are displayed correctly, so the problem lies in the method it has gathered the details.

Another incorrect fact came in the 'Additional Context' section, where ChatGPT wrote:

The match was not part of the relegation phase, it was the last round of the regular season.

On a second attempt, ChatGPT stated the correct starting lineup:

Conclusions

If these details were to be used by a sports betting agency, where the accuracy of details makes a visible difference in the final output of the data, it would signify a big risk, such as miscalculation of an odd.

The solution to ameliorate this kind of risk is to always double check the information aggregated by LLMs and evaluate the source used (especially for real-life events).

Originally published on Medium. Read the full article →