Reference Accuracy in Large Language Model Chatbots: A Metric for Inherent Misinformation?

Introduction

Increasingly, chatbots based on large language models (LLMs) are being tested in various domains for their ability to provide accurate and reliable information [1]. However, it appears that the field of otorhinolaryngology, which deals with disorders of the ear, nose, and throat (ENT), presents a unique challenge to these chatbots due to the specialized and often complex nature of its scientific literature [2]. One method to evaluate the accuracy of a chatbot is to examine the references to scientific papers it provides in response to a user query. This tests the chatbot’s ability to access and retrieve the relevant literature, uncovers its ability to discern the most credible sources, and reveals how it deals with underlying randomness in its algorithms.

The debate on the accuracy and reliability of chatbots based on LLMs is ongoing. However, the performance of chatbots can be more easily verified and quantified if they are asked to provide references rather than open-ended responses, since the latter are more subjective and harder to validate. Typically, LLMs are trained on extensive datasets, and it is therefore likely that highly cited knowledge will be more accessible and more readily retrieved. A complication, however, is that the references provided by chatbots tend to exhibit a peculiar problem – the fabrication of references [3–5]. The underlying problem is subtle but important, and it relates to the stochastic nature of LLMs. The references cited above provide a fuller account.

Two of the most well-known and most used chatbots based on LLMs are OpenAI’s ChatGPT and Google’s Gemini [6]. In terms of the former, 2 recent studies have specifically evaluated ChatGPT’s ability to provide references in the field of otorhinolaryngology [7,8], with 1 study finding that ChatGPT can achieve up to 87% accuracy in delivering appropriate references [8]. However, both cited studies also highlight significant errors, which raises questions about the consistency and reliability of the information chatbots provide. It is well-documented that ChatGPT-4 offers improved performance over its predecessor, ChatGPT-3.5, but the performance of other models, such as Gemini, remains largely unexplored. The motivation for the present study is to explore this inherent “misinformation” aspect more thoroughly.

The present study stems from our interest in chatbot technology, the rapid advancements that have been made in the area, and the need to verify their reliability in such a specialized field as otorhinolaryngology.

The importance of the accuracy of scientific references was emphasized recently in a report on chronic childhood diseases released by the Make America Healthy Again (MAHA) Commission [9]. White House officials appeared to downplay major citation errors in the report, even though its list of references seemed to contain multiple AI-generated fabrications. Reference accuracy is a uniquely objective measure for assessing LLM chatbot reliability, in contrast to the subjective quality of content. Moreover, accurate references are critical because clinical decision-making and academic integrity both rely on trustworthy and verifiable sources. A study last year using earlier versions of chatbots, specifically ChatGPT-3.5 and Bard (the precursor to Gemini), revealed that they had severe limitations, presumably intrinsic, in their ability to provide accurate references [10]. Are the latest versions any better? We explore this question here, noting that earlier work did not involve the detailed categorization of errors or their consistency – both being critical aspects of valid scientific work.

The aim of this study was to evaluate the accuracy of references provided by 3 versions of ChatGPT and 3 versions of Gemini when used in the field of otorhinolaryngology. We assessed 2024 and 2025 model releases and tested whether enabling the newly introduced web-search option improved outcomes. By systematically comparing models, we sought to determine which model currently offers the best reliability in providing verifiable scientific references, and whether progress had been made in this aspect.

Material and Methods

SELECTION OF CHATBOTS:

Two chatbots based on LLMs were tested: ChatGPT (Open AI, USA) and Gemini (Google, USA), each in 3 configurations. For ChatGPT, we used versions 4.0, 4.1, and 4.1, the latter with web browsing enabled. For Gemini, we used version 1.5 Pro, version 2.5 Pro, and version 2.5 Pro with web search enabled.

RESEARCH BASIS:

We based our research on certain scientific articles that were published with the intention of providing basic guidelines on particular topics in otorhinolaryngology. That is, these selected articles each carried the word “guideline” in their titles. Our guiding assumption was that titles intended to provide guidelines would be widely covered in the training space used for chatbots, since they are commonly published not only as scientific articles but also in books and on websites. We therefore expected that they would be widely accessible to chatbots.

ETHICAL CONSIDERATIONS:

The Ethics Committee of the Institute of Physiology and Pathology of Hearing concluded that it did not have to make a decision on the study because it did not constitute a study involving human participants. No sensitive or personal data were processed, which was the basis for this decision.

DATA COLLECTION PROCEDURE:

The topics were selected as follows: (1) The Web of Science was searched for papers which had “guideline” in the title, limiting the search to the otorhinolaryngology category of Web of Science. (2) Repeating topics were removed (eg, papers which had the same title but with “update” added). (3) Papers with at least 100 citations were selected. (4) This resulted in 25 papers which formed the basis of a list of topics (Table 1).

The prompt for the chatbots was as follows: “Please provide references to scientific papers on guidelines for [‘topic’]. Only the bibliographic data of the papers is required.”

The prompts were entered separately, and the chatbots were reset to a new conversation after each question. Responses for each chatbot were collected over 3 consecutive days.

Table 2 shows the testing schedule for each chatbot version.

ASSESSMENT OF REFERENCES:

The references found by the chatbots were verified with PubMed, Web of Science, and Google Scholar. Each reference was assigned to one of the following categories: correct (all information accurate); accurate, but DOI not given; accurate, but wrong DOI number; partially accurate, missing some information (but no false information); partially accurate, with some false information; and totally false.

In some of their responses, both tested chatbots added links to certain webpages. This aspect was not analyzed, since the questions explicitly asked for references, and any additional information provided was ignored.

Details of the study design and the procedures are summarized in Figure 1.

STATISTICAL ANALYSIS:

All analyses were made in Matlab (version 2023b, MathWorks, Natick, MA, USA). Fleiss Kappa was used to evaluate consistency [11]. The values of Kappa can be interpreted as <0.0, poor; 0.01–0.2, slight; 0.21–0.4, fair; 0.41–0.6, moderate; 0.61–0.8, substantial; and 0.81–1.0, almost perfect agreement [12]. For binomially distributed data (correct/incorrect responses), differences among chatbots were assessed using the Cochran Q test. Pairwise comparisons of accuracy between chatbots were conducted with the McNemar test, while pairwise assessments of consistency were made with the Fisher exact test. For further analysis, a Friedman test was used to assess general differences, and a Wilcoxon test was used for pairwise comparisons. In all analyses, a 95% confidence level (P<0.05) was considered statistically significant. The false discovery rate method of Benjamini and Hochberg [13] was used to adjust for multiple comparisons.

Results

Table 3 shows the accuracy of references returned in answer to the queries put to each chatbot. Accurate references were deemed to be those that had all the correct information (except perhaps a missing DOI). Since the questions were framed in terms of “guidelines”, it might be expected that the references suggested by the chatbot would be likely to include highly cited ones (ie, those from Table 1). In fact, across different sessions and versions, the fraction of responses containing these core references, and that were correct, ranged from just 16% to 72% for ChatGPT and from 20% to 48% for Gemini. The percent of core references that were always included in all 3 sessions (ie, the same reference was found by the chatbot at each session) ranged from 8% to 20% for ChatGPT and between 4% and 16% for Gemini. Consistency analysis (Fleiss Kappa) showed only slight/fair agreement across ChatGPT and Gemini, with values between 0.01 and 0.32 depending on version. These results are shown as a bar chart in Figure 2, which shows percent accuracy pooled across all sessions for the tested chatbots. A Cochrane Q test revealed a significant difference between chatbots (Q=30.49, P<0.001). Pairwise comparisons by a McNemar test on the data revealed that ChatGPT4.1 with web search provided significantly higher accuracy than all the other chatbots; nevertheless, it still reached only 58.7%.

Investigating further, we also looked at the percentage of errors in each response in terms of the visibility of each paper, defined as the total number of citations of the paper on which the question was based. The figures were averaged across the 3 sessions, and the results are shown in Figure 3. In general, there was no significant relationship between visibility and correctness, except for the oldest version of ChatGPT (ie, ChatGPT-4.0), which showed a significant negative trend. That is, a lower percentage of errors was associated with a higher number of citations.

When the chatbots gave responses to each of the 25 questions, which were framed in terms of the “topics” listed in Table 1, they typically provided multiple references, including not just the core reference that served as the basis for the question but also others. That is, there were often multiple papers relating to any given published guideline. This aspect was explored further, and all queries and responses were documented. In total, across all the queries asked, the chatbots found 1947 references. All these references were analyzed for correctness, and Table 4 shows the response accuracy across the 3 sessions for ChatGPT and Gemini. By accuracy, we refer only to the correctness of the reference(s) provided. The table shows the numbers and percent accuracy; it also divides inaccurate references into subgroups showing the nature of the errors. For ChatGPT the number of accurate references varied from 17% to 57% across the 3 sessions of the different versions, while for Gemini it varied from 10% to 42%.

The results from Table 4 are plotted as bar charts in Figure 4, which divides the errors of different chatbots into types (with results from the 3 sessions pooled). When inaccuracies in the references were examined, several types of errors emerged, and these are color-coded in Figure 4. First, there were errors only in the DOI number, which happened more often with ChatGPT. The change was often minor, such as just in the last digit, but it meant that a completely different paper was being referenced. Next, some references were partially correct, with only some missing information, but more significant were those with additional false information (for example, with added incorrect authors). When examined more closely, it seems these names appeared on the same page, and were often authors of other cited works. Sometimes there were correct names but the order of the authors was wrong, and occasionally, instead of the authors’ names, the organization to which they belong was given. Finally, there were totally false references (“hallucinations”) that occurred for 1% to 22% of those given by ChatGPT, and 0% to 26% for Gemini.

Statistical differences based on the data from Table 4 are shown in Figure 5, along with average accuracy across chatbots. A Friedman test showed that there were significant differences between chatbots (χ2=63.6, P<0.001). Pairwise comparisons by a Wilcoxon test (Figure 5B) showed several significant differences. In brief, the versions with web search enabled were better than the others, with ChatGPT4.1 being best and Gemini 2.5 being second best.

Discussion

Our study highlights substantial variability in the accuracy and consistency of references generated by leading LLM-based chatbots when they were asked questions based on broad topics in otorhinolaryngology. Disappointingly, the results demonstrate that the accuracy of references provided by the best available models of ChatGPT and Gemini remains very poor. While ChatGPT-4.1 with web search enabled demonstrated significantly higher accuracy than other models, its best performance was still below 60%, and core references were retrieved inconsistently. Previous studies have shown that the less advanced, free versions of chatbots have lower capabilities and apparently perform worse [7,8,14,15], but our findings indicate that even premium models with web access fail to achieve high and reliable bibliographic accuracy. Reported accuracy in analyses by others has been found to range widely, from below 0% (sic) to over 70% [4,14,16–18], which confirms that the problem of low bibliographic reliability is consistent across disciplines. Chatbots apparently need some in-built mechanism to stop them falsifying information in areas in which they are poorly trained. We would say that it is better for a user to be told “I don’t know” than to be misled.

Values of Fleiss Kappa indicated that there was only slight to fair agreement between chatbot responses, meaning that reproducibility across sessions was low. Correct references were only given sporadically, and the overall performance was further compromised by low consistency. The frequent occurrence of errors – including incorrect or missing DOI numbers, author mismatches, and entirely fabricated references – underscores the risk in clinical and academic settings of relying on AI tools for retrieving bibliographic data.

In general, higher citation counts for source papers led to fewer reference errors, but this effect diminished when web search was enabled, implying that access to live data does not fully resolve issues in reference accuracy. To the best of our knowledge, the relationship between citation counts and accuracy has not yet been directly studied, illustrating the novelty of our work. However, other authors have shown that reference accuracy can vary depending on the query, keywords, or subject area, indirectly supporting our observation that better access to training data tends to improve a chatbot’s performance [16,19]. The diversity and frequency of error types we saw, especially in ChatGPT, clearly demonstrate limitations in how LLMs construct bibliographic information, often blending correct elements with fabricated ones. These findings raise a critical question: If chatbots perform so poorly when providing information that can be readily verified, how reliable are they in other situations in which verification is more challenging?

Our work not only classified responses as correct or incorrect but also checked what was wrong and what was not. Our findings of omissions, incorrect authorship, inaccuracies in publication data, and fabricated references are consistent with findings of previous reports, suggesting that these problems are inherent features of current LLM-generated references [4,17,18,20–22]. In our material, DOI-related problems were particularly prominent. While some previous studies did not specifically analyze DOI numbers [7,8], other studies confirm our observations that DOIs are often missing, incorrect, or entirely fabricated [16,19,22,23]. Although the difference in a DOI number is often small, like a change in the last digit, the result is actually serious because the wrong DOI refers to a different paper. The mistakes might arise because the DOIs are totally fabricated by the chatbot, or perhaps the number was found on the same page and referred to titles of related papers (eg, as in a table of contents).

Within the field of otolaryngology, our results were poorer than those reported by Frosolini et al or Lechien et al [7,8]. One possible explanation is that the papers we selected as the basis for testing had fewer citations, which likely reduced their representation in the training data and contributed to more errors, as illustrated in Figure 2. Looking beyond otolaryngology, however, published findings show a wide range of outcomes. Some studies have documented poorer performance than we obtained, finding high proportions of fabricated or non-existent references [4,15]. Others have reported broadly similar results, finding moderate levels of accuracy [17,18,22]. A few analyses, however, have shown distinctly higher accuracy, finding more than 75% of references were correct [14]. Such variation across studies probably reflects differences in methodology and scope: accuracy has been shown to vary depending on the clinical topic [16] and across academic disciplines, with performance in the humanities being notably weaker than in the natural sciences [19].

Another factor is that the present study confirms the considerable variability in the results that chatbots provide [24]. The responses given by both ChatGPT and Gemini varied from day to day across the 3 sessions. ChatGPT appears to have improved in this respect, as has been noted by some earlier studies [8,25], but given the large variability we saw, this cannot be confirmed. We observed a general improvement in the higher (and newer) version of chatbots, but this was not consistent (Figures 2, 5). Our review of available studies suggests that very poor performance is often associated with use of older models [4,15], although this pattern is not universal, as some studies have reported that some early versions had higher accuracy [14,26].

The practical implications of this study extend well beyond scientific references in otorhinolaryngology. The consistently poor performance of both advanced AI models suggests that even cutting-edge chatbots remain unreliable for providing factual information in specialized fields. Similar concerns have been raised in other domains, where chatbots have been shown to generate fabricated or misleading references despite appearing reliable [4,27]. This has broader implications for professionals across disciplines who might use these tools for decision support. In fact, recent studies suggest that reliability varies between topics and across academic domains [16,19]. The pattern of hallucination and fabrication revealed in this study – whereby models confidently present false information rather than acknowledging knowledge gaps – raises significant concerns for any context requiring accuracy: from healthcare to legal, educational, and business settings. The correlation between error rates and citation counts further suggests that AI systems may be fundamentally less reliable when dealing with niche knowledge than with mainstream topics.

Beyond academic settings, our findings have major implications for addressing how AI systems spread misinformation. Research has demonstrated that LLM-generated misinformation can be harder to detect than human-created content, potentially causing more harm [28]. In the medical field, AI hallucinations pose particularly serious risks as wrong information can directly injure health or even be life-threatening [29,30].

Organizations and individuals should therefore be on their guard and implement rigorous verification protocols when attempting to make use of AI outputs. The spread of misinformation by chatbots needs to be minimized, particularly when specialized information is involved, and users should consider these technologies as supplementary tools rather than authoritative sources. The notable variability in responses across testing sessions in the present study is another indicator that consistency is problematic.

This study has some limitations. First, we tested only a limited number of chatbot versions and did not include all LLMs. Second, the analysis was restricted to topics in otorhinolaryngology, which of course may not reflect performance in other medical specialties. Finally, because only a limited number of highly cited topics were included, the results are not generalizable.

Conclusions

This study has demonstrated that, based on the search criteria we devised, both ChatGPT and Gemini are unreliable, making them unsuitable for retrieving references from the literature in otolaryngology. ChatGPT did perform appreciably better than Gemini, but not to a satisfactory degree. Whereas enabling web search does improve performance in some respects, significant errors and inconsistencies persist, posing a risk to clinical decision-making and academic integrity. Before these tools can be trusted in specialized healthcare and research environments, they need to be objectively and systematically validated. AI-generated references cannot be trusted. More generally, this finding casts serious doubt on the correctness of information provided by chatbots. A good place to start for evaluating the level of misinformation generated by LLM chatbots may be to test their reference lists, since at least the lists can be rigorously verified against agreed databases.

More generally, our findings indicate that, at present, chatbots are unreliable and cannot replace traditional citation databases in medicine, although they may serve a supportive role provided their outputs are carefully verified.