Pharmacy researchers from Dutch medical centres have found that artificial intelligence (AI) software ChatGPT outperformed pharmacists when tested about clinical pharmacy, according to a study published in the September issue of the Journal Of Clinical Pharmacology.
ChatGPT achieved “accurate responses” for 79% of the 264 clinical pharmacy questions with which it was presented - by contrast, the “overall score” for accuracy of pharmacists using the app in 2022 was 66%, it found.
“ChatGPT demonstrated a higher accuracy and reproducibility to factual knowledge questions related to clinical pharmacy practice than pharmacists,” the researchers concluded.
Read more: Community pharmacists are the ethical face within the AI tsunami
The team, led by Dr Merel van Nuland of the Tergooi Medical Center’s department of clinical pharmacy, drew the questions from a Dutch application that is designed to “maintain a basic knowledge level for clinical pharmacists”, the paper said.
The AI’s answers were then scored for their “accuracy, concordance, quality of the substantiation and reproducibility”, it added.
“Concordance” was evaluated by whether or not the responses to the question were contradictory, with ChatGPT achieving a 95% success, according to the study.
Two independent pharmacists graded the AI software’s quality of substantiation on a four-point scale and its answers were “deemed good or excellent” 73% of the time, it said.
Read more: Use of ChatGPT in revalidation could trigger FtP case, GPhC warns
The researchers found that ChatGPT’s reproducibility - how consistently it would answer questions at different times - was “consistently high, both within and between days (>92%), as well as across different users”.
The study suggested that ChatGPT “could serve as a valuable resource to pharmacists”.
In May last year, the pharmacy regulator announced that pharmacists undertaking revalidation with the help of AI could face sanctions under a fitness-to-practise (FtP) procedure.
