AI Case Simulation: Virtual Patients, Rubrics and Human Approval

Taking a complete patient history, setting the right priority in triage or breaking bad news to a family: these are skills built by doing, not by reading. You can't "practice" at a real patient's bedside. Simulation labs, standardized patient actors and instructor hours aren't sufficient in every institution. And regular attendance at simulation sessions is often unrealistic for a nurse working shifts.

AI-assisted case simulation is one of the methods that could fill this gap: the learner works by exchanging messages with a "virtual patient" playing its role on a screen or phone, and their performance is then assessed against predefined criteria. In this article, we look at how the method works, rubric-based assessment, why only synthetic cases should be used and, most importantly, its limits.

What is case simulation, and what does a virtual patient do?

Case simulation is a training method in which the learner faces a realistic clinical situation, makes decisions and then receives feedback. In the literature, the term "virtual patient" describes interactive scenarios in which this encounter takes place on a computer.

How it differs from classic methods

• Manikins and skills labs: Indispensable for hands-on skills, but they require a room, equipment and an instructor.
• Standardized patients: A trained actor plays the patient. Very valuable in communication training, but the planning burden means it can only be done occasionally.
• Written cases: Easy to distribute but not interactive; the learner reads the information without having to ask the right question.

An AI virtual patient aims to bring the accessibility of a written case closer to the interactivity of a standardized patient. If the learner doesn't ask the right question, the information doesn't come; if the patient is anxious, they may give evasive or repetitive answers.

How does it work?

A typical flow has four steps:

1. Case card: A clinical expert or training lead prepares a fictional card containing the patient's age, complaint, history, findings, information to "share only if asked" and the learning objectives.
2. Role-playing model: A language model plays the patient while staying within this card. It is framed by rules so that it doesn't invent findings that aren't on the card.
3. Encounter: The learner takes a history, makes a triage decision, checks a medication order or explains things to the patient and their relatives.
4. Assessment: After the conversation ends, a second model scores the transcript against a predefined rubric and shows the reasoning for each score.

Separating the role-playing model from the assessing model matters. If the system playing the patient also grades the encounter, the independence of the assessment is weakened.

What does the evidence say?

The virtual patient method isn't new; the generation before language models worked with pre-written branching scenarios. A 2019 systematic review and meta-analysis that pooled research from this generation (Kononowicz et al., Journal of Medical Internet Research) examined 51 randomized controlled trials with a total of 4,696 participants. According to the authors, virtual patient simulations can improve skills more effectively than traditional education, and knowledge at least as effectively. The same study also stresses that the quality of evidence was generally low and that there was marked variation between studies.

Two caveats are needed here. First, these results cannot be directly generalized to virtual patients based on language models; the effect of simulations built with generative AI needs to be measured separately. Second, in its January 2024 ethics and governance guidance on large multi-modal models, the World Health Organization (WHO) lists medical and nursing education, including simulated patient encounters, as one of five broad areas of use for these models. In short, the field is promising, but it calls for a "measure, validate, then scale" approach.

Which scenarios is it suited to?

The criteria in the table are only examples; each institution should build its rubric around its own protocols and training objectives. A virtual patient doesn't replace skills that require a body and a setting, such as hands-on procedures, physical examination or team coordination; it complements them.

Rubric-based assessment: why and how?

A rubric is a scoring table that defines in advance the criteria and levels against which a performance will be assessed. In AI assessment, the rubric serves two purposes: it makes scoring consistent across learners, and it gives the model a concrete basis for answering "Why did you give this score?"

Features of a good rubric

• Observable criteria: Behavior visible in the transcript, such as "Reflected the patient's concern in their own words" instead of "Showed empathy".
• Separating critical errors: Gaps that directly affect patient safety, such as not asking about allergies, should not be buried in the total score but flagged separately.
• Level definitions: Clear levels for each criterion, such as "not done / partly / fully".
• Mandatory justification: The model should justify each score with a quote from the conversation.

The feedback loop

The instructive part of a simulation often comes after the score. When the learner reads the reasoned feedback and tries the same or a similar case again, learning is reinforced. The nurse educator's review of results at department level reveals which criterion shows a collective gap; for example, it may become clear that allergy questions are often skipped in medication safety cases on a particular ward.

The synthetic case principle

Real patient data should not be used in case simulation. There are three main reasons:

1. Privacy: A case believed to be anonymous can become identifiable when a rare diagnosis, age and date information are combined.
2. It isn't necessary: The clinical picture needed for a training objective can be built in full on a fictional card.
3. Control: A synthetic card is designed around the learning objective; which information emerges with which question is planned deliberately.

When preparing a synthetic case card, the institution's current protocols should be the basis, and the card should not be released for use until a competent expert in the relevant field approves it. When a protocol changes, the card must be updated too; otherwise, the simulation reinforces outdated information.

Limits and risks

The WHO guidance draws attention to the fact that large language models can produce false, inaccurate, biased or incomplete statements, and to automation bias, which leads users to overlook these errors. In a simulation setting, these risks look like this:

• Drifting from the script: The virtual patient may "invent" a finding that isn't on the card. A rule framework and regular spot checks are needed.
• Incorrect scoring: The assessing model may misread a criterion. Learners need to be able to appeal, and appeals need to be reviewed by a person.
• False confidence: A high simulation score is not a certificate of clinical competence. Results should be used alongside, not instead of, observation-based competency assessment.
• Scope: A virtual patient is a training tool; it should not be used to make diagnostic or treatment decisions for a real patient.

Where does human approval come in?

Two of the six principles in WHO's 2021 guidance on the ethics of AI for health, protecting human autonomy and ensuring responsibility and accountability, also guide training applications. When launching a case simulation program in your institution, you can use the following checklist:

• The case card was approved by a clinical expert in the relevant field before release.
• The rubric was prepared together with the training unit and the department head; critical errors are defined separately.
• In the first rounds, model scores were compared with an instructor's scores to check consistency.
• Randomly selected conversations are reviewed by a person at regular intervals.
• A way for learners to appeal a score has been defined and communicated.
• When a protocol changes, the related case cards and rubrics are updated.
• Who can see simulation results has been set out in writing.

These steps aren't meant to slow the method down but to make it reliable. AI makes a large number of encounters possible; the expert remains responsible for the accuracy of the training.

Sources

• Kononowicz AA et al. Virtual Patient Simulations in Health Professions Education: Systematic Review and Meta-Analysis by the Digital Health Education Collaboration. Journal of Medical Internet Research, 2019.
• World Health Organization. WHO releases AI ethics and governance guidance for large multi-modal models (January 18, 2024).
• World Health Organization. Ethics and governance of artificial intelligence for health: Guidance on large multi-modal models.
• World Health Organization. WHO issues first global report on Artificial Intelligence (AI) in health and six guiding principles for its design and use (June 28, 2021).

Case simulation with ATP MedAI

In ATP MedAI's case simulation module, the virtual patient role-plays from a synthetic case card, and history-taking, triage, medication safety, patient communication and difficult conversation scenarios can be practiced. When the conversation ends, a second model scores it against a rubric and gives reasoned feedback. No patient data is needed for the simulation. AI-generated content doesn't go live without expert approval; results feed into the competency map by person and department, and only authorized people see individual results. We can explore together in a demo how the method could be set up in your institution.

ATP MedAI Editorial Team
This article was prepared for general information. If you spot an error or omission, let us know and we'll correct it.