Can ChatGPT Provide Patient-Friendly and Reliable Information on Cervical Cancer Screening? A Study of ChatGPT-Generated Information in Polish

Introduction

THE AIM OF THE STUDY:

The aim of the study was to assess the quality of ChatGPT-generated medical information on cervical cancer screening (MICC_GPT), provided in Polish language. We performed an objective assessment of the readability of MICC_GPT and a subjective assessment of the MICC_GPT content, following the RACCA model criteria of relevance, accuracy, completeness, clarity, coherence, and appropriateness. The ChatGPT-4omni Plus chatbot model (model GPT-4o) was used.

Material and Methods

STUDY DESIGN:

The study was conducted at Warsaw Medical University, Poland, between 2 September 2024, when the pilot phase began, and 5 November 2024, when the evaluation of the content generated by ChatGPT was completed.

THEORETICAL FRAMEWORK:

The study followed Sallam et al’s guidelines for good practice in healthcare education research and practice using content generated by artificial intelligence algorithms, including LLMs [13]. As suggested by Sallam et al, there are 9 aspects to consider when designing a study and describing its results: (1) the design of the LLM model used to generate the content, (2) the methods of evaluation of the output data (objective vs subjective assessment), (3) the exact time and date of the LLM model testing and output generation, (4) the transparency of the input data, (5) the scope of the input data, (6) the randomness of the input data, (7) individual factors affecting the consistency of evaluation of input and output data, (8) the number of queries performed, and (9) prompt design [13].

FORMULATION OF CERVICAL CANCER SCREENING QUESTIONS:

The authors analyzed recent international scientific publications on women’s knowledge of CC screening [14–18], identified 5 key thematic areas, and created an initial database of 76 questions on CC screening: (1) general information about the test (16 items), (2) preparation for the test (15 items), (3) specimen collection for testing (16 items), (4) interpretation of the test results (16 items), (5) management of an abnormal test result (13 items) (Raw Data 1 Selection of Questions for the Study on Cervical Cancer Screening).

PILOT STUDY:

A pilot study was conducted on 2 September 2024 using the ChatGPT-4omni chatbot (GPT-4o model). From all the CC screening raw questions (no specific prompts) to be used in the main part of the study, one of the co-authors (JGM) selected 6 items: 3 on general information about the test and 3 on test preparation.

The questions were formulated with the simple text-based instruction method, which is most commonly used in conversations with LLMs by people unfamiliar with advanced prompt engineering techniques [19,20].

A total of 30 ChatGPT-generated responses were analyzed for this pilot study. Five responses were generated for each of the 6 questions, with the response generated in a new ChatGPT conversation window each time to avoid model memory and in-context learning effects. A preliminary assessment of the quality of the ChatGPT-generated responses and medical information on CC screening was performed by the first co-author of the publication (AM).

The pilot study aimed to identify potential issues related to the formulation of questions or the quality of responses that might require modification before the main study started. The preliminary quality assessment of the ChatGPT-generated medical information also aimed to determine whether the responses were: (1) consistent with medical knowledge, (2) clear and understandable, (3) complete, and (4) useful for patient education. It would also identify satisfactory and unsatisfactory responses before the actual study began. Analysis of the pilot-generated content did not reveal any need for question modification.

EXPERT REVIEW OF THE USEFULNESS OF CC SCREENING QUESTIONS:

Seven out of 76 questions were rejected as unsuitable following a preliminary assessment of their usefulness by an expert midwife (MS), a professor with 14 years’ experience and substantial academic achievements in midwifery.

Next, 20 licensed expert midwives further assessed the relevance of the questions. The sampling was purposive, as the aim was to recruit experienced midwives recognized as experts in performing cervical cytology. The selection criteria included: (1) professional experience in conducting cervical screening, (2) completion of specialized training in cervical cytology, and (3) active involvement in clinical practice related to cervical cancer prevention. Midwives who met these criteria were identified through professional networks and invited to participate in the study. The selection process was designed to ensure that the participants had a high level of expertise and practical competence in cervical cytology.

On 4 October 2024, the experts anonymously rated the usefulness of the remaining questions by completing a review form available online. The rating was done on a 5-point scale: 1 – definitely not useful, 2 – rather not useful, 3 – no opinion, 4 – rather useful, 5 – definitely useful. Each expert reviewer evaluated their own separate set of ChatGPT-generated answers. No single expert assessed all responses included in the study. This approach was chosen to distribute the workload and ensure an efficient evaluation process.

Items receiving at least 50% of the ‘definitely useful’ responses or at least 50% combined ‘rather useful’ and ‘definitely useful’ responses, were included in the subsequent stage of the study. Twenty questions did not meet the expert criterion for inclusion, and 49 items were finally included in the main part of the study.

The list of all 76 questions, together with information on the exclusion of particular items at each stage of the survey is presented in Raw Data 1 Selection of Questions for the Study on Cervical Cancer Screening.

THE OROCESS OF CONTENT GENERATION BY CHATGPT:

The answers to the 49 questions ultimately included in the study were generated by ChatGPT between 7 October and 11 October 2024. The study used the ChatGPT-4omni Plus chatbot model (model GPT-4o), with no personalization, and with ChatGPT learning and customization disabled. The responses were generated by the same researcher (JGM) as in the pilot study. The same computer and IP number were used.

The answers to each of the 49 questions were generated 4 times, starting with Question 1. This resulted in a total of 196 answers. To avoid learning effects of the ChatGPT model, each answer was generated in a new conversation window. When ChatGPT generated 2 answers to choose from simultaneously, both answers were stored in the database. An Excel spreadsheet (Raw Data 2) was used to record all questions and answers. The data were then cleaned by removing special characters and formatting marks (### and **) used to tag some of the words selected by ChatGPT.

An analysis of all 196 ChatGPT-generated responses confirmed that none of the items contained misleading, harmful, or potentially dangerous medical information.

READABILITY OF CHATGPT-GENERATED RESPONSES TO CC SCREENING QUESTIONS:

Analysis of the readability of the MICC_GPT was carried out using software available at https://www.jasnopis.pl/, ona scale from 1 (an easy-to-understand text, for those with basic education) to 7 (very difficult-to-understand text, for those with a doctoral degree or expert knowledge of the field). Jasnopis.pl is an online software tool used in our study to objectively assess the readability level of the ChatGPT-generated texts in Polish. It analyzes text complexity and assigns a readability score on a scale from 1 (very easy to understand, equivalent to elementary school level) to 7 (very difficult to understand, requiring approximately 15 years of formal education – equivalent to a bachelor’s or engineering degree). Jasnopis was used solely for this purpose, providing an objective measure of how understandable the generated content is for people of different education levels. The FOG index indices were also analyzed: headwords (words whose standardized/dictionary entry forms have 4 syllables or more), run-on words (words whose running forms, as they appear in the text, have 4 syllables or more), and low-frequency headwords (difficult words those whose keyword forms have 4 syllables or more).

CONTENT ANALYSIS OF THE QUALITY OF CHATGPT-GENERATED RESPONSES TO CC SCREENING QUESTIONS:

For the content quality analysis of the MICC_GPT, 4 further independent expert midwives were invited as expert reviewers (ERs). They had not been involved in the initial survey item selection.

EXPERT REVIEWER CHARACTERISTICS:

All ERs have a university degree in midwifery, a master’s degree in midwifery, are professionally active, and have a licence to practise midwifery in Poland. The detailed characteristics of ERs are presented in Table 1.

The ERs were unaware that the content was generated by ChatGPT. They used the RACCCA framework to evaluate various aspects of information quality: information reliability theories, information quality research in information systems, text readability and comprehensibility evaluation, text linguistic quality evaluation, context analysis, and style evaluation. There are a number of criteria within this framework: relevance refers to whether the answer directly addresses the query; accuracy is a criterion of precision and consistency with scientific data; completeness indicates whether the answer is comprehensive and provides all essential information; clarity indicates whether the answer is clear, unambiguous, logical and well-structured; coherence analyzes whether the answer is free from contradictions, is logically structured, thematically consistent, and flows smoothly; and appropriateness indicates whether the answer is appropriate in terms of the context, form, and intended audience (https://www.linkedin.com/pulse/frameworks-effective-prompting-using-raccca-clear-models-wadsworth-sjpke/).

A 5-point Likert scale was used for all criteria, with 1 representing complete failure to meet the criterion and 5 signifying full compliance. Standardization of scoring was ensured by providing all reviewers with detailed written instructions and a clear description of each evaluation criterion.

ChatGPT generated answers to 49 questions. Each answer was generated 4 times, resulting in a total of 196 responses (49 questions ×4 answers). This created 4 unique sets of answers, 1 for each expert reviewer. Each expert received their own unique set of 49 answers for evaluation. Each ER obtained a link to their own evaluation form and rated 49 separate MICC_GPTs. The evaluation of the MICC-GPTs took place between 27 October and 5 November 2024.

STATISTICAL ANALYSIS:

The initial analysis involved the extraction of 17 linguistic features of text readability. To ensure data consistency, the numerical values of each feature were standardized. The analysis of the effect of these features on text difficulty was carried out in 2 steps.

In the first step, machine learning techniques were applied using Microsoft Azure cloud computing to identify the main predictors of text difficulty. The most relevant features were selected using the Voting Ensemble algorithm. In the second step, an ordinal regression model was built based on the selected predictors, which allowed quantitative assessment of the impact of each predictor on readability.

The quality and reliability of the Voting Ensemble model was assessed through cross-validation and the calculation of Nagelkerke prediction error rates (RMSE, MAE) and pseudo-R2. The significance of the predictors was assessed using a matrix of regression coefficients and a weight distribution analysis with box plots. For the ordinal regression model, the odds ratio (OR) and 95% confidence interval (CI) were calculated for each predictor.

STATISTICA™ 13.3 (Tibco Software) and Microsoft Azure software were used for the calculations. The level of statistical significance was set at α=0.05.

Results

Objective Analysis

OBJECTIVE ANALYSIS:

None of the ChatGPT-generated responses included in our study contained any potentially harmful information, such as recommendations against cervical cancer screening or advice that could negatively impact patient safety.

The average MICC_GPT difficulty level was 5.314, and full comprehension of the MICC_GPT required the target audience to have participated in formal education for at least 15 years (eg, to have completed a bachelor’s or engineering degree).

The FOG index for technical terms was 12.07 on average, indicating texts of intermediate difficulty. In contrast, the FOG index for low-frequency complex headwords was higher (13), suggesting challenges for elementary-educated audience when reading specialized medical content (Table 2).

The vast majority of MICC_GPTs (82%) required the audience to have a university degree to fully understand the text (Table 3).

INFLUENCE OF SELECTED LINGUISTIC FEATURES ON READABILITY:

The results of the analysis using the Voting Ensemble technique indicate a high explanatory power of the model, with an R² value of 0.695, which means that 69.5% of the variance in text difficulty was explained by the included predictors. Prediction error rates, such as RMSE (0.429) and MAE (0.332), showed low deviations of predicted difficulty levels from the observed values. Among the predictors, ‘Percentage of verbs’ stands out as the only one with a negative effect on the level of difficulty, while ‘Number of difficult words,’ ‘Percentage of nouns,’ and ‘Average word length [syllables]’ had a significant positive effect on the level of difficulty.

The ordinal regression model showed a good quality of fit (R2N=0.625). The overall test of the model (χ2=268, df=6, P<0.001) confirmed its significance and adequacy. The regression analysis revealed the following significant predictors of text difficulty: the percentage of verbs (negative effect, OR=0.662), the number of difficult words (OR=1.073), the percentage of nouns (OR=1.337), the average word length expressed in syllables (OR=17886), the percentage of difficult words (OR=1.248), and the FOG index for complex headwords (OR=1.753). All these variables had statistically significant impact on text difficulty (Table 4).

CONTENT ANALYSIS OF CHAT-GPT GENERATED RESPONSES TO CC SCREENING QUESTIONS: In the opinion of the ERs, most MICC_GPTs met (31%) or fully met (40%) the assessed criteria of relevance, accuracy, completeness, clarity, coherence, and appropriateness (Table 5).

Individual reviewers differed in their assessment of the MICC_GPTs: Expert Reviewer 2 rated the largest number of MICC_GPTs (38%) as failing to meet the criteria. As each expert reviewer assessed a different set of 49 responses, it was not possible to statistically analyze inter-rater reliability.

RELEVANCE:

An assessment of the relevance of the MICC_GPTs showed that the responses met or fully met the criterion. More than 77% of the responses received the highest rating (4 or 5 on a 5-point scale), suggesting that ChatGPT generally provides relevant information. Discrepancies were observed between the rating of Expert Reviewer 2 and the other expert reviewers (Raw Data 3).

ACCURACY:

The assessment of the accuracy of the MICC_GPTs was also largely positive. As many as 71% of the responses were rated as meeting or fully meeting the accuracy criterion, indicating a high level of accuracy and adherence to scientific fact. Significant differences were noted between the rating of Expert Reviewer 2 and the other expert reviewers, particularly for responses rated as failing the criterion (Raw Data 3).

COMPLETENESS:

The analysis of the completeness of the MICC_GPTs showed a high degree of compliance with expert expectations for most of the responses. In total, 75% of the responses were rated as either meeting (4) or fully meeting (5) the completeness criterion, indicating the ability of the model to provide comprehensive and sufficient information. Expert Reviewer 2 rated a significant percentage of responses as not fully informative, which may indicate areas for further improvement (Raw Data 3).

CLARITY:

The results for the clarity of the ChatGPT-generated responses showed mostly positive ratings. Approximately 72% of the responses were found to meet (4) or fully meet (5) the clarity criterion. This shows that the information was generally clear and understandable to the audience. However, as with the other criteria, there were differences in the ratings between Expert Reviewer 2 and the other expert reviewers (Raw Data 3).

COHERENCE:

The evaluation of the coherence of the ChatGPT-generated content showed most responses to be logical and direct. Approximately 70% of the responses were rated as either meeting (4) or fully meeting (5) the coherence criterion. The results indicate the model’s ability to generate responses that are clear and free of inconsistencies. However, as with the other criteria, discrepancies were observed in the ratings of particular expert reviewers (Raw Data 3).

APPROPRIATENESS:

The assessment of the appropriateness of the ChatGPT-generated responses identified more than 68% of the responses as meeting (4) or fully meeting (5) the criterion, indicating the model’s ability to generate content tailored to the context and expectations of the audience. However, a significant proportion of responses were rated as not meeting (2) or completely failing (1) the criterion (Raw Data 3).

Discussion

LIMITATIONS:

Our results are not without limitations. The expert reviewers’ assessment of MICC-GPT quality was subjective in nature, and their professional experience may have had a significant influence on their judgement. In addition, as each reviewer assessed a different set of 49 MICC-GPTs, it was not possible to measure inter-rater reliability.

FURTHER DIRECTIONS FOR ON-GOING RESEARCH:

Further research should focus on improving communication between end-users and ChatGPT, particularly in terms of tailoring responses to patients’ varying levels of education and medical knowledge. Key areas for analysis include communication effectiveness and evaluation of query formulation techniques (simple text-based instructions) for users unfamiliar with advanced prompt engineering, language simplification and analysis of the impact of simplifying medical terminology on comprehension of responses and patient health decision-making, personalization of content, and analysis of ChatGPT’s ability to tailor responses to individual needs such as age, medical knowledge, or communication preferences.

Given the critical role of the human papillomavirus (HPV) vaccine in cervical cancer prevention and its frequent targeting in anti-vaccination discourse, future research should include comparative analyses of different large language models (LLMs), such as ChatGPT, Gemini, or Claude, in providing information on this topic. Such studies would allow for a comprehensive evaluation of the accuracy, completeness, and potential biases in the information these models generate, particularly concerning vaccine safety and efficacy. Understanding how various LLMs address controversial topics, including misinformation commonly found in anti-vaccination narratives, is essential for assessing their suitability as tools for public health education. Further exploration in this area could support efforts to enhance the reliability of AI-generated content and promote informed decision-making among patients and healthcare consumers.

Conclusions

PRACTICAL IMPLICATIONS:

The findings of this study highlight the potential for responsible integration of ChatGPT into midwifery education, patient counseling, and public health campaigns focused on cervical cancer screening. In midwifery education, ChatGPT can serve as a supplementary tool to enhance students’ understanding of screening guidelines and patient communication strategies by providing accessible explanations and generating patient-friendly educational materials. For patient counseling, ChatGPT may assist healthcare professionals in preparing clear and consistent information about the importance, process, and outcomes of cervical cancer screening, supporting informed decision-making. In public screening campaigns, ChatGPT can be utilized to draft educational content, such as FAQs or social media posts, tailored to different literacy levels, thus promoting awareness and improving participation rates. To ensure safety and accuracy, all ChatGPT-generated materials should be reviewed and validated by qualified healthcare professionals before dissemination.