In February of 2023, ChatGPT was found to score at or around the approximately 60 percent passing threshold for the United States Medical Licensing Exam (USMLE). Now, it has been revealed it may even have a better bedside manner than medical professionals
This according to a study reported by The National Post on Friday.
The research team gave a real doctor and the AI chatbot the same situation: someone who may have swallowed a toothpick.
The human doctor had the following to say: “While many people have swallowed toothpicks without issue, if two to six hours have passed, chances are it’s already in the intestines, and therefore not easily retrievable. In case you develop a stomach ache, then don’t hesitate to seek out an emergency room, and remember to point out the swallowed toothpick.”
The AI chatbot however took the time to respond much more empathetically.
AI-generated empathy
“It’s natural to be concerned if you have ingested a foreign object,” the chatbot responded. While it’s possible for a toothpick to lodge in the throat or puncture the digestive track, causing serious injury, it’s unlikely to happen with a “dull, cooked toothpick that is only two cm long,” it noted.
“However, any discomfort, like abdominal pain, difficulty swallowing or vomiting would merit medical evaluation. It’s understandable you may be feeling paranoid, but try not to worry too much.”
The researchers put several questions to the test using a panel of licensed health-care professionals to compare the replies offered by real doctors and the AI chatbot.
The researchers found that the panel preferred the chatbot’s responses to nearly 200 inquiries 79 percent of the time. In fact, ChatGPT’s responses were four times more likely to be ranked good, or very good quality, and 10 times more likely to be empathetic, or very empathetic.
Now the researchers are hoping to use the AI system to help them deal with their overload of work.
“That’s what we came to the table with: Can we use an AI assistant, using ChatGPT as a case study, to help answer patient messages — not only to help improve the workflow for providers, but to also improve the quality of the responses that patients receive,” told the National Post Dr. John Ayers, the study’s lead author and vice chief of innovation in the UC San Diego School of Medicine division of infectious disease and global public health.
The study was published in JAMA Internal Medicine.
Study abstract:
Importance: The rapid expansion of virtual health care has caused a surge in patient messages concomitant with more work and burnout among health care professionals. Artificial intelligence (AI) assistants could potentially aid in creating answers to patient questions by drafting responses that could be reviewed by clinicians.
Objective: To evaluate the ability of an AI chatbot assistant (ChatGPT), released in November 2022, to provide quality and empathetic responses to patient questions.
