With the rise of chatbots and AI in healthcare, many people use these tools for medical advice. However, there are concerns about the accuracy of the information provided by these chatbots. To address these concerns, a team of doctors reviewed the top 9 health answers provided by ChatGPT.
Experiment Methodology
In April 2023, medical researchers used the Ahrefs program to analyse over 100 Google health search terms. The study excluded COVID-19-related keywords to avoid monotony.
After identifying the most popular searches, they were sent to ChatGPT. The first AI-generated message, following the query, was accepted as an answer. No further questions were asked.
Then, medical experts evaluated the accuracy of the information provided and considered whether the answers could be used for first-hand knowledge. They also noted potential pitfalls or areas where additional information may be necessary.
Assessing ChatGPT’s Credibility in Health-Related Queries
The team of doctors, led by Dr. Edna Skopljak and Dr. Donika Vata, found that while the chatbots provided some accurate information, there were also many inaccuracies and gaps in knowledge.
For example, the chatbot offered incorrect information about the availability of the vaccine to prevent RSV infection. It also failed to tailor answers to the personal health characteristics of the user, which may play a crucial role in understanding a specific medical condition.
The findings of this review have important implications for using chatbots and AI in healthcare. These tools can be helpful, but it is vital to be aware of their limitations and to seek medical advice from a licensed healthcare professional when necessary.
Can You Trust Chat GPT’s Health Answers?
For a conclusive evaluation, the team consulted with Dr. J. Wes Ulm – a US-based physician, medical researcher, and bioinformatics expert with MD and PhD degrees from Harvard Medical School and MIT.
As someone who has worked in medical text generation, Dr. Wes says it exemplifies many of the pros and cons of artificial intelligence.
So what are the cons? Experts name the same issues that hobble AI text generators everywhere.
Even the best AIs, such as Chat GPT, aren’t great at describing and communicating the quality and nuance of their knowledge. And they don’t have the intuition to tailor articles to the questions readers may have.
“The AI-generated text is correct, concise, and complete in what it conveys. Still, there’s a lack of context and connection to what’s most relevant to know,” says Dr. Wes.
He adds that AI is just the result of learned patterns in corpora (the plural form of canon, a body of text). They are fed and fine-tuned into software algorithms designed for a specific task, producing a short capsule summary of health topics.
“In layman’s terms, it explains a well-known and well-traveled medical topic in consumer health with a controlled vocabulary. For this, it scans tons of solid, succinct articles online. So the AI wouldn’t even need to train on a large corpus. It could just recapitulate a few of the best pieces already out there,” concludes Dr. Wes.
In other words, AI-generated text gives you a quick, effortless, and mostly accurate understanding of your health questions. However, don’t expect it to be comprehensive or detailed.
Remember that AI answers should not replace medical advice from healthcare providers. For any health-related concerns, consult your doctor instead of self-diagnosing online.
This way, you can protect yourself from risks and receive accurate and timely advice.
