0:02
Hello everyone. This is Bella Meta from RheumNow, and I'm at the ACR Convergence and I want to talk to you about a very interesting AI plenary session.
0:14
The abstract number is 0869 and this is sort of asking the question about can AI help rheumatology.
0:23
So we all know and all of us are guilty of asking questions to AI.
0:29
This group did it in a very standardized way.
0:34
When I say AI, they asked 60 clinical scenarios in rheumatoid arthritis, lupus, gout, vasculitis, and osteoporosis.
0:42
So five big diseases, 60 questions, and asked these pretty complex questions to AI — or the platforms — five different platforms: Open Evidence, ChatGPT, Gemini, the usual — and sort of compared it with, or validated it with, experts.
1:05
The key take-home is that Open Evidence was correct 91.2% of the time and the hallucination rate was 3.8% — pretty low but still there.
1:14
The next was ChatGPT, which the concordance rate was around 89.4% with the hallucination rate much higher, about 6.2%.
1:25
So basically every platform was around 90% good, which is great, but in rheumatology we don't want it just to be 91% good — what happens to the other 10%?
1:38
What we can take away from this is that we usually want to take everything that we get from AI, think about it before putting it into practice.
1:47
But one key caveat here I'd say is that when these things hallucinate, we don't know if they hallucinate or not.
2:01
Since this was structurally done, I think that a good 90% even to start with is great.
2:08
Also, there was no knowledge base particularly that these large language models were hooked to, such as the ACR guidelines etc. So once we have
2:20
RAGs — or Retrieval Augmented Generation — these are newer ways to make sure that the hallucinations are lower, some of these hallucinations around 3
2:29
or 6% that I'm talking about will also get lower.
2:32
So it's getting better. I don't think AI will replace rheumatology yet, but it definitely augments our decision making, and this would really help us make things faster.
2:45
With that, signing off — this is Bella Meta. We need to talk.
If you are a health practitioner, you may Login/Register to comment.
Due to the nature of these comment forums, only health practitioners are allowed to comment at this time.