AI agents in healthcare: where they assist the care team, and how to build one safely

What an AI agent for healthcare actually does

An AI agent for healthcare is a software system that uses a language model plus tools, meaning electronic-health-record access through FHIR and retrieval over approved clinical content, to carry out multi-step administrative and documentation workflows under clinician review. It drafts notes, assembles prior authorizations, and answers patient questions. It assists the care team and surfaces sourced suggestions; it does not autonomously decide care.

That boundary is the whole design, and it is grounded in evidence, not caution for its own sake. A 2025 study in Communications Medicine, part of the Nature Portfolio, gave six leading language models 300 physician-designed vignettes that each contained one fabricated lab value, sign, or disease, and recorded hallucination rates ranging from about 50% to 82%; a mitigation prompt halved the mean rate but did not remove it.⁴ A companion review reached the same conclusion: retrieval grounding and chain-of-thought reduce medical hallucinations measurably but do not eliminate them, so human oversight stays essential and foundation models should not operate autonomously in clinical settings.⁵

So the practical shape of a healthcare agent is an edit-and-sign tool; it drafts and proposes, and it does not auto-execute. The clinician reviews the basis for every suggestion, and the agent is built to abstain, cite its source, and route anything out of scope to a person. This page supports the broader AI agents guide; here the focus is the healthcare-specific framing of what is safe to automate and what is not.

AI agents in healthcare: where adoption and spend are going

Adoption of AI agents in healthcare is accelerating, and spending is concentrating on the administrative and documentation workflows agents suit best. Healthcare is one of the leading sectors for agentic AI: McKinsey reports that gen-AI implementation in healthcare rose from 25% in 2023 to about 50% by the end of 2025, and Menlo Ventures put 2025 healthcare AI spending near $1.4 billion, roughly 3x the prior year.

The detail underneath those headlines explains why agents fit. McKinsey names healthcare among the three sectors where the use of AI agents is most widely reported, though it cautions that scaling is still early: across all organizations 23% report scaling agentic AI somewhere, but in any single business function no more than about 10% do.¹ Menlo Ventures, surveying more than 700 executives, found health systems deploying AI at roughly 2.2x the rate of the broader economy and adoption of domain-specific tools reaching 22% of healthcare organizations in 2025.² The pull is structural: US healthcare administrative spending runs near $740 billion a year, and physicians spend about one hour on documentation for every five hours of patient care, which is the burnout driver ambient agents target.²

The chart below breaks the 2025 spend down by category. Read it as a map of where agent-shaped workflows are getting funded first.

Use cases that show value first

The AI agent for healthcare use cases that show value first are ambient clinical documentation, prior authorization and claims drafting, scheduling, patient intake and triage routing, care coordination, and patient question answering. Administrative and documentation workloads tend to return value fastest because they are multi-step, high-volume, and reviewable, and because a clinician or staff member signs off on the output.

Documentation is the most mature category. Kaiser Permanente rolled out an ambient scribe across 40 hospitals and more than 600 offices, and Intermountain Health reported about a 27% reduction in time-in-notes per appointment among heavy users of an ambient documentation tool, while a Mass General Brigham study saw a 21.2% reduction in burnout prevalence at 84 days.³ The table below maps the recurring patterns; each links back to work we deliver through AI agent development.

How to build an AI agent for healthcare on FHIR

You build an AI agent for healthcare by treating data access as the foundation, integrating the electronic health record through HL7 FHIR, grounding the language model in approved clinical content with retrieval, wrapping it in guardrails and PHI controls, and designing clinician review in as a core feature. An agent that reads and writes patient data reliably is a different engineering problem from a chatbot, so the data layer and the review step come first.

Work through it as a layered build.

1. FHIR as the data layer. Use HL7 FHIR resources for patient data, SMART on FHIR for authentication and launch, and CDS Hooks to invoke the agent at workflow points such as chart open or order entry. Treat every read and write as deterministic and audited.⁶
2. Retrieval over approved clinical content. Ground the model in evidence-based guidelines, formularies, payer policies, and the organization's own protocols so answers are sourced and current instead of drawn from model memory. Source attribution is repeatedly cited as a primary risk-reduction technique.⁶
3. Guardrails. Add input and output filtering, scope limits so the agent declines out-of-scope clinical questions, abstention behavior under uncertainty, citation-required answers, and PHI handling controls.
4. Clinician review as a feature. Design for edit-and-sign so a person approves every output. Clinician approval is both the safety mechanism and, as Section five explains, part of the regulatory pathway.
5. Audit logging, bias testing, and monitoring. Log every action, test for bias and equity gaps, and monitor behavior in production well beyond the acceptance milestone.

As a planning range, not a fixed quote, a narrow single-workflow pilot with data access ready tends to run about 8 to 16 weeks, while a production-grade build with EHR write-back, compliance validation, and multi-site rollout runs 6 to 12 months or more. Building that data layer, the retrieval pipeline, and the review and audit controls is what our AI agent development team does, and the regulated-data and workflow side is where it meets our healthcare software engineering work.

HIPAA, safety, and FDA limits

A healthcare agent can be HIPAA compliant if it is built correctly, which means a signed Business Associate Agreement with every provider that processes protected health information, encryption in transit and at rest, audit logging, access controls, and a policy that your data is not used for training. Compliance comes from the build and the contracts; the model alone does not deliver it. On top of that sit two limits: language models still hallucinate, so the agent assists and a clinician decides, and the FDA regulates software that crosses from decision support into autonomous diagnosis or treatment.

Take HIPAA first. Any model or cloud provider that processes PHI on your behalf is a HIPAA business associate, so you need a signed Business Associate Agreement before any PHI is sent. HIPAA-eligible enterprise options that offer such agreements include AWS Bedrock, Azure OpenAI Service, OpenAI Enterprise, and Google Gemini Enterprise; only the enterprise tiers qualify, and consumer endpoints do not. Required safeguards include TLS 1.2 or higher in transit, AES-256 at rest, audit logging, access controls, and a no-training policy; de-identifying PHI is a useful control but is not strictly required once a valid agreement is in place.⁷

The FDA line is the one to state precisely. Under the Clinical Decision Support exemption, some CDS software is excluded from the medical-device definition if it meets four criteria, the key one being that it lets a healthcare professional independently review the basis for a recommendation rather than driving the decision on its own; software that makes a specific diagnosis or treatment recommendation a clinician cannot independently verify is a regulated device.⁸ The FDA's list of AI-enabled medical devices passed 1,000 entries and exceeded 1,400 by the end of 2025, most cleared through the 510(k) pathway.⁹ The honest framing for a build is that an assistive, transparent, human-reviewed agent is often designed to sit inside the CDS exemption, and the moment it makes autonomous clinical decisions, regulatory obligations escalate. We do not describe a Resourcifi-built agent as FDA cleared unless a specific product actually is.

Kanika Mathur

Head of Service Delivery, Resourcifi

Kanika Mathur is Head of Service Delivery at Resourcifi, where her engineering pods build assistive AI agents for regulated workflows and wire them to FHIR data layers, retrieval over approved clinical content, and clinician review steps. She has scoped the HIPAA, audit-logging, and human-in-the-loop controls that decide whether a healthcare agent is safe to put in front of a care team, which is the lens this guide is written from.

Resourcifi on LinkedIn →

Keep reading

Related guides worth your time

← →

Agents & RAG Agentic RAG: When to Use It and How to Build It Agentic RAG explained: how it differs from naive and advanced RAG, the key patterns like corrective RAG and self-RAG, the... Read guide → Agents & RAG AI Agent for Fintech: Risk, Compliance, Ops, Customer AI agents in finance: fraud, AML, KYC and servicing use cases, how to build with money-movement guardrails and human appr... Read guide → Agents & RAG AI Agent for HR: Recruiting, Onboarding, People Ops AI agents for HR: screening, employee Q and A and onboarding use cases, how to build them, and the bias, EEOC and Local L... Read guide → Agents & RAG AI Agent for Legal: Intake, Discovery, Contracts, Research AI for legal research: real use cases, how accurate the tools are, the documented sanctions risk, and why attorney verifi... Read guide → Agents & RAG AI Agent for SaaS: How to Embed Autonomous Agents in Your Product AI agents' disruptive impact on the SaaS industry in 2025: Gartner sees agentic AI at 30% of app-software revenue by 2035... Read guide → Agents & RAG AI Agent for Sales: Pipeline & Outreach Automation AI agents for sales: lead qualification, outreach and CRM use cases, how to build with guardrails on autonomous outreach... Read guide → Strategy, architecture & ops AI Architecture Patterns Agentic design patterns explained: reflection, tool use, planning, and multi-agent collaboration, with a framework to pic... Read guide → Strategy, architecture & ops AI Architecture Patterns for SaaS: A Technical Guide Generative AI architecture for SaaS: layered design, multi-tenant isolation, LLM gateway, RAG, and security. Built by Res... Read guide → Building AI AI Copilots for SaaS: Build vs Buy Guide AI copilot vs AI agent for SaaS: a copilot assists, an agent acts. How an in-app copilot works, the RAG and multi-tenant... Read guide →