Skip to main content

AI in Diagnostic Support

Dr. Antoni Chan and Dr. Johannes Knitza | Abstract 2959 | ACR 2026

Transcription

I'm Anthony Chan reporting for RheumNow here at ACR26 in Orlando. There's been a lot of abstracts on the area of artificial intelligence here at this conference. But today I want to discuss with you a study which is actually a randomized control trial in the use of AI, particularly an LLM, to help with diagnosis. And this comes from abstract 2950, the ALLIANCE trial. And with me here today is Johannes Nidza from the University of Marburg, who actually was the lead author for this study. So Johannes, welcome. Thank you so much for being here.

So tell us about this study that was conducted, the ALLIANCE trial. Yeah, so I think many physicians are increasingly using LLMs for clinical decision support, especially for diagnostic support, and we found there is no randomized evidence, and that's basically why we did it. So we had 82 physicians from seven hospitals from Germany and Norway, and first basically they had to solve three difficult vignettes and think of a diagnosis and also rate their diagnostic confidence without any tools, and then they were randomized to basically no support in the control group but no LLMs, and in the intervention group with a medically certified LLM.

And then what we found — the primary outcome was the percentage of correct top diagnosis. So there was a similar improvement. So the primary outcome was negative, but the case processing time was actually less than half in the language model-based group. So it was a lot faster with a similar amount of improvement. And what we also found is that the model itself was actually better in terms of the correct top diagnosis than rheumatologists numerically, and significantly if we look at the non-rheumatologists. So we had rheumatologists and non-rheumatologists. So I think especially if you're sort of not the expert in the field, this can help you a lot, and what we saw quite fast — so in just some seconds. But what we also observed was quite a substantial AI over-reliance. So physicians just accepting wrong suggestions, and also a substantial proportion of overconfidence by the model itself but also by the physicians, and using this tool actually the level of overconfidence increased. So you feel safer but still you're wrong. So you have to take care.

So that's very interesting. So basically there is more speed or acceleration but not necessarily accuracy. Yes, yes, yes. And this is obviously a randomized control trial which is a big advantage, because we are not just doing set exam questions — these are actually people doing case vignettes, you know, cases. So how do you think this might be taken forward?

I think this was a nice first step, as you say, like it was a randomized control trial, but what we probably really need is now real-world evidence, because physicians are increasingly using this in clinical routine, and obviously these vignettes are nice but clinical routine is a lot more complex. So that's what I'm looking forward to.

So we might see some efficiency by saving people's time as a result of doing this. On the other side, there's always the concern about deskilling, where if you rely too much on these tools you might not be able to skill up some of the people who are in training. What's the thought on that? Yeah, never scale or even de-skill — I mean, that's what we observed, this sort of automation bias and people really blindly accepting wrong suggestions. So that's definitely a major threat of using these tools, and I think we really need to be aware of this and be very careful and really question all the suggestions.

Yeah, I think the cases that were studied were actually quite complex as well, so you really put this system to the test. Absolutely, yeah. So that's a very nice study. Do you have any key takeaways to tell our listeners about this study, the ALLIANCE trial? Yeah, I think LLMs are really promising and I think we should use them to our advantage, but we really have to be careful when implementing this in clinical routine and we need more evidence.

Thank you very much. That was a very nice study. So that's a study that looked at the use of LLMs in terms of diagnosis. I'm Anthony Chan reporting here for RheumNow at ACR26 in Orlando.

ADD THE FIRST COMMENT

If you are a health practitioner, you may to comment.

Due to the nature of these comment forums, only health practitioners are allowed to comment at this time.

×