Abstract

Aim

Artificial intelligence (AI) applications are widely used to identify solutions to patients' problems. This study aims to evaluate the scientific validity of information that patients can access about human papillomavirus (HPV) -related topics using Chat-Generative Pre-Trained Transformer (ChatGPT).

Methods

This study was conducted between July 1 and August 1, 2025. A physician developed a structured set of HPV-related questions. The responses generated by ChatGPT were independently evaluated by three clinicians with clinical experience in HPV management. Each response was rated using a five-point Likert scale based on accuracy and clinical relevance. Inter-rater reliability among reviewers was assessed using Cohen's kappa statistic.

Results

The mean scores given by the reviewers for evaluating the accuracy of ChatGPT's answers to HPV-related questions were 4.9±0.3, 4.75±0.44, and 4.75±0.55, respectively. The percentages of correct scores assigned to ChatGPT by the reviewers were 90%, 75%, and 80%, respectively. The approximately equal percentages of correct and incorrect scores were 0, 0, and 5, respectively. The percentages of nearly correct scores were 10, 25, and 15, respectively.

Conclusion

Chat-Generative Pre-Trained Transformer 4.0 demonstrated high efficacy in providing general information regarding HPV, with an 81.6% accuracy rate and a 90% near-accuracy rate. Incorporating AI tools into the facilitation of patient access to information could enhance learning processes. However, it is essential that these tools be continuously refined and utilized to complement rather than substitute for the critical judgment of medical professionals and patients.

Introduction

Human papillomavirus (HPV) infection is one of the most widespread sexually transmitted diseases in the world, affecting both men and women (1). Human papillomavirus is transmitted through skin-to-skin contact (2). Low-risk HPV can cause genital warts, which can negatively affect patients’ social, physical, and sexual lives. High-risk types of HPV can cause cancers of the anus, vagina, vulva, penis, mouth, lungs, and throat (3, 4). Infections with HPV can also lead to adverse reproductive outcomes, including reduced sperm quality and decreased sperm concentration and motility (5, 6).

Large language models (LLMs) are natural language processing (NLP) models that use deep learning algorithms to process and generate text in a manner similar to human language. Chat-Generative Pre-Trained Transformer (ChatGPT) is an NLP model developed by OpenAI that was introduced at the end of 2022 (7). Chat-Generative Pre-Trained Transformer can generate highly accurate and appropriate responses for patient education purposes. Its alignment with professional medical guidelines demonstrates its high potential for patient education (8, 9). Additionally, ChatGPT can be used by physicians to facilitate research innovation and comprehensive health management, as well as for diagnostic reasoning in some diseases, particularly in areas where rapid information retrieval and analysis are crucial for patient care (10, 11).

In the present study, we hypothesized that clinicians with HPV-related clinical experience could evaluate ChatGPT’s information about HPV and thereby determine the accuracy of the information that patients can access through ChatGPT. Patients are using chatbots that use artificial intelligence (AI) more and more to obtain health information, but it’s still not clear how reliable the information they provide about HPV-related topics is. Consequently, assessing the precision of AI-generated information is crucial for ascertaining the safe application of such tools in patient education.

Materials and Methods

Compliance with Ethical Standards

The study did not require approval by the institutional review board or ethics committee because no patient data was used. Informed consent was not specifically obtained from respondents. However, it was indicated in their survey responses. We adhered to the Strengthening the Reporting of Observational Studies in Epidemiology guidelines for reporting (12).

Study Design

This observational study was conducted from 1 July to 1 August 2025. A physician generated a set of questions. (M.Y) (Table 1). All questions were chosen subjectively to represent each physician’s area of expertise, with the aim of providing a robust and balanced overview of the relevant topics. The physician was requested to provide questions with clearly defined, evidence-based answers based on guidelines from the European Association of Urology, the American Urological Association, and the European Academy of Dermatology and Venereology. The physician developed binary (yes/no), descriptive, or multiple-correct-answer questions, all of which had similar difficulty ratings. A Likert scale can be useful for assessing the accuracy, completeness, and reliability of knowledge (13). The physician who created the questions intended to use a predefined Likert scale to assess their accuracy.

The accuracy scale was a 5-point Likert scale, with the following options: (1: completely incorrect; 2: more incorrect than correct; 3: approximately equally correct and incorrect; 4: nearly correct; and 5: completely correct.)

To assess the consistency of all questions related to HPV, a chatbot version 4.0 was used. To ensure consistency, one investigator (H.C.A.) entered all questions into the chatbot, prompting it with the phrase “Please ensure that information is medically accurate and based on current best practices and guidelines for urology, dermatology, obstetrics, and gynecology” and using unconditional prompts for each new chat. Three specialists (two urologists and one dermatologist) who specialize in the medical or surgical treatment of HPV warts in outpatient clinics were invited to assess the answers (Table 1). To avoid potential bias, respondents were instructed not to use the chatbot to screen the questions themselves. Artificial intelligence-generated answers were reviewed by three clinicians using a five-point Likert scale. The study process is summarized in Figure 1.

Statistical Analysis

Statistical analyses were performed using SPSS version 22.0 (IBM Inc., Armonk, NY, USA). Descriptive statistics were used to summarize the data. Continuous variables were expressed as mean ± standard deviation. Categorical variables were described in terms of frequency and percentage. Cohen’s kappa is used to assess inter-rater reliability (IRR). Cohen’s kappa correlation values were described as poor agreement: 0.00, slight agreement: 0.00-0.20, fair agreement: 0.21-0.40, moderate agreement: 0.41-0.60, substantial agreement: 0.61-0.80, and almost perfect agreement: 0.81-1.00 (14). A p<0.05 was considered significant.

Results

70% of the respondents confirmed that the answers to questions 1, 2, 4-10, 13, 16-18, and 20 were entirely correct. One reviewer confirmed that the answers to questions 3, 14, 15, and 19 were entirely correct, while the other two reviewers confirmed that they were nearly correct. Two reviewers confirmed that the answer to question number 12 was entirely correct. One reviewer confirmed that it was nearly correct. Three reviewers assessed the answer to question 11 as follows: one considered it entirely correct, one considered it nearly correct, and one considered it approximately equally correct and incorrect.

The mean scores given by the reviewers for assessing the accuracy of ChatGPT’s answers to HPV-related questions were 4.9±0.3, 4.75±0.44, and 4.75±0.55 respectively. The reviewers’ evaluations are presented in Table 1.

The correct score percentages given by the reviewers to ChatGPT were 90, 75, and 80, respectively. The approximately equal correct and incorrect score percentages were 0, 0, and 5, respectively. The nearly correct score percentages were 10, 25, and 15, respectively. IRR values between reviewer 1 vs. 2, reviewer 1 vs. 3, and reviewer 2 vs. 3 were 0.500 (p=0.01), -0.132 (p=0.473), and 0.448 (p=0.021), respectively. Although no statistically significant agreement was found between the first and third reviewers regarding the accuracy of ChatGPT’s responses, statistically significant agreement was observed among the other reviewers. The reviewers’ IRR values are shown in Table 2.

Discussion

Large language models can be utilized by humans to identify health issues and direct the treatment process. Chat-Generative Pre-Trained Transformer, developed by OpenAI, is one of the LLMs used for consulting on health issues. This study was designed to determine how accurately ChatGPT could answer questions posed by patients with HPV attending the outpatient clinic and to inform patients about the HPV treatment process.

According to (Surveillance, Epidemiology, and End Results) statistics, the incidence of anal carcinoma increased by an average of 2.2% per year between 2013 and 2022. During this period, it accounted for 0.5% of all new cancer diagnoses in the United States (15). Anal cancer incidence is rising, and predominantly HPV type 16 causes a high-grade squamous intraepithelial lesion (16). High-grade squamous intraepithelial lesion is the precursor lesion of anal squamous cell carcinoma (SCC). It is caused by the uncontrolled growth of squamous epithelial cells in the perianal area or the anal canal transformation zone and is a direct result of an HPV infection. These premalignant lesions may develop into anal SCC if treatment is not received (17). Approximately 88% of anal cancer cases test positive for HPV DNA, indicating a strong association between HPV and anal cancer, second only to cervical cancer. Consequently, HPV has a significant impact on the development of anal cancer (18). Although ChatGPT stated that HPV-16 had a 70% likelihood of causing anal cancer, its report that this strain is the most frequently found in anal cytology was accurate. The use of ChatGPT for medical research regarding the ratio of anal cancer cases associated with HPV 16 is not substantiated by empirical data.

A self-sampling strategy combining HPV detection in urine samples with accessible polymerase chain reaction (PCR) tools was developed as an alternative to cervical swab-based HPV screening to improve participation rates. The PCR kit can detect 14 types of HPV, including HPV-52, HPV-16, and HPV-18, in cervical and urine samples. Urine samples show promise in terms of their accuracy for HPV detection, which could increase cervical cancer screening (19). Our study revealed that ChatGPT correctly identified certain facts relating to cervical cancer screening tests, advanced urine sample options, and the PCR requirement.

The presence of HPV in sperm is associated with male infertility, as indicated by an elevated risk of oligozoospermia and asthenospermia (20). HPV in women has potentially been caused by cervical or tubal factor infertility. However, a scoping review concluded that any studies investigating HPV infection in relation to female fertility had not been conducted (21). Chat-Generative Pre-Trained Transformer has conducted a thorough review of the extant literature on HPV-related infertility in women and men, offering a comprehensive interpretation of the subject. Furthermore, it provides more comprehensive answers to the question by offering information on fertilization, implantation, the probability of pregnancy, and the transmission risk to the partner or the fetus.

Head and neck SCCs (HNSCCs) emerge from the mucosal epithelium of the oral cavity, larynx, and pharynx. The primary risk factor for HNSCCs of the larynx and oral cavity is smoking. Oropharyngeal tumors are increasingly being associated with a history of infection with carcinogenic strains of the HPV, particularly HPV-16. To a lesser extent, this association has also been observed with HPV-18 and other strains (22). The overall prevalence of oropharyngeal HPV in healthy adults in the United States of America and in Europe was reported to be between 3.6% and 6.8% in females and between 6.6% and 15.0% in males (23). The highest oral HPV prevalence was described in South America and the lowest was described in Asia, at 12.4% and 2.6%, respectively (24). Chat-Generative Pre-Trained Transformer has stated that the prevalence of oral HPV is around 7%, oral HPV-16 around 1%, and HPV-related infections around 1% per year. This information does not align with the existing literature on the subject. Furthermore, ChatGPT has stated that HPV-related infections may be temporary and may clear naturally within six to twelve months. This information conflicts with what is published in the literature. (25).

Men who have sex with men, HIV‐positive status, sexual history (number of lifetime sex partners, number of recent oral or anal sex partners, age of sexual debut), and smoking are associated with adult HPV infection (26). However, ChatGPT noted that lack of male circumcision, other sexually transmitted infections (STIs), and long-term oral contraceptive use are risk factors. The use of oral contraceptives showed an independent association with HPV16-18 infection rates (27). Nevertheless, no clear association exists between HPV infection and circumcision. Additionally, a history of STIs can affect penile, cervical, or vaginal infections.

HPV is transmitted through sexual contact; from mother to fetus; through skin contact (e.g., via the hands or contact with underwear or other inanimate objects); and by high-temperature evaporation treatment. Visible warts can be treated with physical therapy or surgery. The recurrence rate of subclinical infections caused by warts up to 1 cm can be reduced by applying laser therapy, cryotherapy, topical imiquimod, and photodynamic therapy (25). Chat-Generative Pre-Trained Transformer stated that autoinoculation, which is considered a form of skin-to-skin transmission, is a distinct cause of transmission. The information provided by ChatGPT is accurate but insufficient. Furthermore, it noted that indirect contact was very rare and clinically insignificant. Additionally, ChatGPT noted that there is no treatment for asymptomatic HPV infections and that vaccination and surveillance are sufficient. These recommendations do not align with the existing literature on the subject.

Vaccination is the most practical and cost-effective method for avoiding HPV-related health issues. HPV vaccination has been shown to prevent more than 90% of HPV-related cancers (28). In a different study, HPV vaccination was found to be associated with a reduced incidence of several types of cancer among females aged between 9 and 26 years (29). Chat-Generative Pre-Trained Transformer recommended vaccination by age and risk group; this recommendation aligns with the literature on the subject. Furthermore, it emphasized the significance of HPV vaccination and the adoption of safe sex practices, including limiting the number of sexual partners, using condoms, and enhancing immune function through smoking cessation.

The clearance time for HPV has been referred to as being less than 4 weeks (±4 weeks) in the short term and 12 months (±6 months) in the long term (25). Albero et al. (30) concluded that the period during which the HPV virus was cleared ranged from 1.3 to 42.1 months. According to ChatGPT, the median clearance time for HPV is between 6 and 18 months. HPV infections usually clear spontaneously within one to two years in immunocompetent individuals. It stated that persistence may last 12-24 months or longer, particularly for high-risk types.

Study Limitations

This study was conducted with only the ChatGPT 4.0 model, and other AI-based models were not included in the evaluation. The accuracy of the responses was assessed by a limited number of physicians, thereby restricting the diversity of perspectives. Only three reviewers, physicians from overlapping specialties, were included. The inclusion of a larger group of experts could have increased the reliability of the evaluations. Future studies involving multiple AI models and a broader range of physicians may strengthen the generalizability of the findings, particularly by including diverse specialties and practice settings to ensure a more comprehensive evaluation of the AI’s effectiveness across different patient populations. Despite these limitations, the study will enable patients to obtain preliminary information based on highly accurate data evaluated by reviewers specified by ChatGPT during the interval between seeking medical attention and visiting a physician.

Conclusion

Chat-Generative Pre-Trained Transformer 4.0 has demonstrated high efficacy in providing general information about HPV, with an 81.6% accuracy rate and a 90% near-accuracy rate. Despite the evident potential demonstrated by ChatGPT, these findings should be regarded as preliminary indications of its promise, particularly in research related to human health. Furthermore, these results should not be construed as validation of ChatGPT’s clinical adequacy. Incorporating AI tools into the facilitation of patient access to information could enhance learning processes. However, it is essential that these tools be continuously refined and utilized to complement, rather than substitute, the critical judgment of medical professionals and patients.