Evaluation results
How Aurifex performs in testing, and how to read it
Not randomized and not independently validated. These results come from de-identified historical admissions, expert-labeled cases, blinded review and workflow testing on simulated cases.
Testing results
Internal evaluation on de-identified historical admissions held out from training.
85%+
Discharge dates within one day
On 25,000 held-out historical admissions
0.91
Rehab-candidate flag: AUROC
88% recall and 80% precision at the operating threshold
85%
Drafted consults rated acceptable
As written or with minor edits, in blinded review
45→10min
Minutes per IRF referral (simulated cases)
If a candidate isn't flagged, nothing changes: your team still refers through its usual process, and Aurifex never removes a patient from a list. What if a candidate isn't flagged?
Case-manager minutes per IRF referral, by hand → with Aurifex (simulated cases)
35 min lessIRF packets only, timed on simulated cases, including the human review.
How it was tested, and limits
| Data | 25,000 de-identified historical admissions held out from training, so the models are tested on admissions they never saw. |
|---|---|
| Labels | Rehab-candidate flags are compared with 2,000 expert-labeled admissions (3 PM&R physicians and 2 case managers). 400 drafted consults were rated in a blinded review by 6 clinicians. |
| Baseline | On the same held-out admissions, clinician-entered expected discharge dates were within one day 58% of the time, against 85%+ for the model. |
| Workflow | Case-manager referral prep time was measured in workflow testing on simulated cases. Speed was measured in sandbox replay of historical admission events. |
| Limits | Sample sizes, raters and 95% confidence intervals are in Every number, defined. Historical data can't show how people will use the flags and drafts, which is what a pilot measures. Not measured here: nurse time, discharges before noon and ED boarding hours; a pilot baselines these on your units. |
What we don't know yet, and why results will vary
- Outcomes on your units. Length of stay, excess days and IRF acceptance, against your own baseline. Units with more excess days have more room to improve.
- Missed rehab candidates in practice. Recall is measured on historical data; a pilot checks it against patients who went to rehab without a flag.
- Readmissions and emergency department returns. Tracked in a pilot as a balancing measure.
- Use, edit rates and staff time in practice. Results depend on how often teams use the flags and drafts, including drafts approved with no edits and time saved beyond referral prep.
- Patient mix, payer mix and post-acute network. Units with more stroke, orthopedic and neurological patients have more rehab candidates; prior authorization adds days that faster referrals can only partly recover; bed availability at nearby IRFs and SNFs sets a floor on waiting time.
What a pilot will measure
Measured on your units, before and after go-live.
The five measures and how each is defined
| Measure | How it's defined | When |
|---|---|---|
| Discharges before noon | Inpatient discharges with a discharge time before 12:00, as a share of all inpatient discharges on the unit | Baselined at kickoff, reported at the end of the pilot |
| Emergency department boarding | Hours admitted patients wait in the emergency department for an inpatient bed | Baselined at kickoff, reported at the end of the pilot |
| Length-of-stay index | Observed stay divided by expected stay (GMLOS) for the case mix | Baselined at kickoff, reported at the end of the pilot |
| Beds likely to open today | The unit-level forecast of beds opening by 5 p.m., compared with actual discharges | Reported at the end of the pilot |
| 30-day readmissions and emergency department returns | Returns within 30 days of discharge, for patients discharged from the pilot units. Stop rule: no increase beyond a margin set in the pilot plan, or the feature pauses | Reported before any decision to expand |
Every number, defined
Every model and system figure: what was counted, where it comes from and how to read it.
All figures and definitions
| Number | What was counted | Where it comes from | How to read it |
|---|---|---|---|
| 85%+ within one day (85.3%, 95% CI 84.9–85.7%) | The share of 25,000 held-out admissions where the model's expected discharge date, from day 2 of the stay, was within one day of the actual discharge date: 85.3% (95% CI 84.9–85.7%). | Model dates compared with actual discharge dates on de-identified historical admissions held out from training | Close enough to plan the post-acute bed, the authorization and the equipment early. Measured on held-out historical admissions. |
| 58% clinician baseline | The share of those admissions where the clinician-entered expected discharge date was within one day of the actual discharge date. | The same held-out admissions where a clinician-entered expected discharge date was recorded (n = 14,200) | The comparison point for the model's 85%+. |
| 0.6-day mean error | The average number of days between the model's expected discharge date and the actual discharge date. | The same held-out historical admissions | The within-one-day share shows how often the date is close; the mean error shows how far off it is on average. |
| 95% overall agreement | Of the 2,000 expert-labeled admissions, the share where the Possible IRF flag (flag or no flag) matched the expert label. | Expert-labeled reference set of 2,000 admissions, labeled independently by 3 PM&R physicians and 2 case managers; disagreements adjudicated (Cohen's kappa 0.82) | An overall agreement measure, counting both flagged and unflagged cases. Recall and precision for the flag are shown separately. |
| AUROC 0.91 (95% CI 0.89–0.93) · 88% recall · 80% precision | How well the rehab-candidate flag separates likely candidates from other patients (AUROC), and, at the threshold used in the worklist, the share of candidates it catches (recall, 88%) and the share of raised flags that are candidates (precision, 80%). | The expert-labeled reference set: 2,000 admissions, 240 of them rehab candidates (12%) | About 5% of discharges in the historical data went to inpatient rehab, so the flag raises about 5 to 6 flags per 100 admissions; about 1 in 5 flags is not a candidate. |
| 82% next place of care | The share of held-out admissions where the model's top predicted next place of care matched where the patient actually went. | Held-out historical admissions | Used to plan the right referral early, not to decide it. |
| 85% rated acceptable (95% CI 81–88%) | The share of reviewed draft consults rated acceptable as written or with minor edits, in a blinded review. | Blinded review of 400 drafts by 6 clinicians (3 PM&R physicians, 3 case managers), two reviewers per draft (kappa 0.78) | A reviewer would send it after small edits at most. In use, a person approves every draft. |
| 1.5% hallucination rate (95% CI 1.2–1.8%) | The share of reviewed draft statements that their cited source did not support: 1.5% (95% CI 1.2–1.8%). | The 6,000 statements in the 400 blindly reviewed drafts, checked against their cited source | Every statement must cite the note it came from, and an automated check removes unsourced ones before review. A person reads every draft. |
| 98.5% supported by their cited source | The share of drafted statements that carry a link to their source (100%), and the share that passed the automated check that the source supports the statement (98.5%). | Automated source check on drafted statements | A statement without a source is removed before a person sees the draft. |
| 45 → 10 minutes | Typical time to prepare one inpatient rehab (IRF) referral packet by hand, compared with reviewing and approving the cited draft, in workflow testing on simulated cases. Includes the human review. | Median across 60 simulated cases, timed with 6 case managers | The time saved by reviewing a cited draft instead of writing one from scratch. Measured in workflow testing on simulated cases. |
| < 5 min to the worklist · < 30 s per draft | Time from a replayed admission event to an updated worklist (under 5 minutes) and time to generate a referral draft (under 30 seconds). | Sandbox replay of historical admission events | Fast enough that the flag and draft are ready before the case manager's next review. |
| 1,200-case suite · 40/40 prompt-injection tests | Cases every model or prompt change must pass before it is released (1,200), and adversarial prompt-injection tests passed (40 of 40: direct and indirect injection through note text, instruction override and data-exfiltration attempts). | The automated evaluation gate | No change is released unless it passes the whole suite. |
| ≤ 3-point subgroup accuracy gap | The largest gap in within-one-day accuracy between age, sex and race groups; the smallest group reported has 900 admissions. | Subgroup checks in the evaluation suite | Checked on every change, not once. |
| 25,000 held-out admissions | Admissions held out from training and used only to test the models. | De-identified historical admissions | Held out so the models are tested on admissions they never saw. |
Estimate your hospital
The calculator opens with example values so you can see how it works. Replace them with your own; nothing you type is sent anywhere.
See Aurifex on your own discharge workflow
We'll walk through your worklist, a drafted referral and how a pilot would be measured on your units.