Back to blog

Why does FDA approval still leave medical AI unsafe?

Thắng Đoàn
Thắng Đoàn

Your AI device got cleared, and you think the regulatory risk is behind you. It is not. With roughly 295 AI devices authorized in 2025 alone, per MedCity News, the hardest problem now begins after approval, when the model meets patients the validation never saw.

Approval is getting faster, not stricter

The FDA has authorized roughly 1,450 AI enabled devices to date, about 295 in 2025 alone versus about 6 in 2015, per MedCity News. The agency is also moving older AI tools into easier pathways. Citeline reports the FDA finalized Class II status for four devices, including Hologic's AI Pap slide system, letting similar future devices clear through the lighter 510(k) process. Faster clearance means more AI in clinics sooner, with less evidence per device.

  • Class II moves mean future AI tools skip the heavier De Novo process.
  • Volume is exploding: 6 clearances in 2015, 295 in 2025.

Models drift after approval

Performance can change after release, often unexpectedly, even when developers never update the model. Performance can change after release, often unexpectedly, even when developers never update the model, MedCity News explains. Clinical AI meets varied patients, sites, and practices, so it operates in conditions different from its validation. MedCity News argues the industry's focus now needs to shift to what happens after release. Pre-market review, however careful, cannot anticipate every population the system will face.

  • Devices routinely face conditions unlike their validation sets.
  • Post-market performance, not clearance, is where attention must move.

Generative and autonomous AI break review

GenAI devices accept open-ended inputs and generate variable outputs, per an FDA discussion paper summarized by JDSupra. They can also evolve over time and sometimes act autonomously. That behavior does not fit a review built for fixed software. The FDA is separately seeking comment on autonomous and remote robotic devices, HealthInfoSec reports, where cyberattacks, AI malfunctions, and communication failures could harm patients before clinicians intervene. The regulator itself admits its framework does not yet fit.

Phone AI sits outside the review

Aijourn warns that consumer phone AI may reinterpret regulated device alerts outside any FDA review. The article cites Apple's redesigned Health app and Google's Gemini coaching in Fitbit as examples reshaping the space. A regulated diabetes app's alert may now pass through an unregulated AI layer. Safety depends on layers the agency does not oversee.

What this does not solve

The FDA papers are discussion drafts, not rules, so nothing binding follows yet. They do not address liability when an autonomous device harms a patient. Nor do they settle who owns the gap between a regulated app and unregulated phone software summarizing its alerts. Vendors get no safe harbor for model updates from commenting early.

If your cleared AI model drifted next quarter, how would you know before a clinician did?
Which layers of your own cleared device stack, from model to phone OS, sit outside FDA review, and who owns that risk?

List every input your cleared device receives post-deployment and flag the ones your validation set never contained.

Sources

Citeline: FDA Finalizes Class II Status For AI Tools

JDSupra: FDA Seeks Input on GenAI Device Framework

HealthInfoSec: FDA Flags Cyber, AI Risks in Medical Robots

MedCity News: The Big Challenge Comes After Approval

Aijourn: Phone AI Should Not Rewrite Device Alerts

Share:

Recommended for you

Enjoyed this article?

Subscribe for new articles. No spam. Unsubscribe anytime.

By subscribing you agree to receive the newsletter. No spam, and you can unsubscribe anytime.