UPMC’s Carmody on Turning Off AI That Falls Short


Chris Carmody, Senior Vice President and CIO, UPMC

Chris Carmody, Senior Vice President and CIO, UPMC

UPMC CIO Chris Carmody explains how token costs, governance reviews and a head-to-head ambient AI pilot shape the health system’s AI investment decisions. Watch below or on YouTube.


UPMC expects each AI solution and agent to earn back its cost, and the health system is prepared to turn off the ones that do not. Chris Carmody, senior vice president and CIO at UPMC, said on a recent episode of the healthsystemCIO Show that token costs are a particular concern because AI is expensive. He had been CIO for about a month when he spoke Sept. 30. Before that, he spent 12 years as the health system’s CTO. UPMC also closed its acquisition of the four-hospital Trinity Health System in Ohio on Oct. 1. Two days later, it brought its second wave of hospitals onto a single Epic instance.

Key Takeaways

  • UPMC weighs AI solutions and agents against their expected return, with token costs in view, and is prepared to turn off or replace those that fall short.
  • Workloads move to the cloud only when the ROI is better, the controls are in place and the business case holds.
  • An AI policy and a review group evaluate what the health system decides to adopt, including technology and security, and quality checks watch for drift and bias.
  • UPMC chose its ambient AI vendor after piloting Abridge and Nuance DAX with 1,000 physicians each, although Abridge was founded out of UPMC.
  • Leaders are being pushed to replace yes-or-no targets with measurable outcomes, backed by frequent measurement to catch problems early.

UPMC Weighs Token Costs Against AI ROI

Token costs get close attention at UPMC. In particular, Carmody said the health system focuses on them as it explores AI solutions and builds agents to make its processes more efficient. Microsoft also provides the foundation for that work. Staff and teams use the models built into Copilot, and the next step is an agentic layer that automates processes. “We want to make sure it actually delivers on the ROI, because AI is expensive,” Carmody said. “We have to be measuring the right way, so that if it doesn’t make sense, we turn it off, we pivot, we change and look for a different solution.”

A similar ROI test has guided UPMC’s infrastructure for years. The health system has used the cloud for more than 15 years under an approach Carmody calls being cloud smart. In other words, a workload goes to the cloud only when the ROI is better and the controls support it. About 10 years ago, UPMC also partnered with a company to build a colocation data center in the Pittsburgh area. More recently, it placed racks of storage, networking and compute at its hospitals, an arrangement it calls a data center in a box. Carmody expects more AI to run at the edge, on laptops, phones and workstations on wheels, so compute has to be able to move.

Houston Methodist applies a similar standard to large projects. For example, Sarah Pletcher, MD, chief digital health officer at Houston Methodist, told healthsystemCIO in September that a large technology project needs some hard ROI. Compelling success stories alone, she said, will not sustain it.

Governance Sets the Pace of Adoption

Models now change so quickly that UPMC relies on governance to decide what to adopt. Carmody pointed out that 2025 belonged to OpenAI and that Anthropic now draws the attention. Yet no one knows which developer will lead in 2027, he added. As a result, the health system adopted a policy and formed a group that reviews what UPMC decides to adopt. That review covers the technology itself and its security considerations.

Those reviews have already shaped a platform choice. UPMC works with Microsoft Copilot in part because Work IQ, part of Copilot, keeps that work inside the organization’s own controlled environment. The aim is to keep staff from going to ChatGPT or other public sites to ask questions. The team also runs quality checks on the models it uses, because drift and bias are known risks. At the same time, UPMC is raising AI literacy across the organization. Staff should understand where the tools make sense and stop treating them as a black box, he said.

That diligence takes real work, and Carmody said it also means giving up the race to adopt first. “We don’t necessarily want to be first in adopting everything,” he said. “We want to be smart in adopting it and apply it in the best way possible.”

A Homegrown Vendor Still Had to Win a Bake-Off

UPMC applied the same discipline to ambient documentation, even though one of the leading vendors grew out of UPMC itself. Abridge started at UPMC Enterprises, the health system’s innovation arm, and its co-founder and CEO, Shiv Rao, MD, still practices cardiology at UPMC. Even so, when UPMC was running multiple EHRs, it piloted Abridge with 1,000 physicians and Nuance DAX with another 1,000. Abridge came out ahead, and UPMC moved forward with it.

That said, Carmody said UPMC remains open to alternatives now that Epic offers its own ambient AI. Ambient documentation is also spreading to nursing, an area where UPMC expects to evaluate different solutions. Consequently, he said the health system could stay with Abridge, move to Epic or consider a newcomer that leapfrogs both. healthsystemCIO reported the bake-off on Oct. 27 in a feature on ambient AI license use. Here, Carmody explains why the decision stays open. “We don’t want to be locked in with any technology that becomes technical debt five or 10 years from now,” he said.

Such evaluations are collaborative, he said. Specifically, clinical leaders, nursing and operations evaluate the options with IT, and the choice has to tie back to organizational goals. A physician who declares a product the best solution ever, for instance, still has to see it checked against what Epic, Abridge, Microsoft and others offer. UPMC’s preference for platforms has its own history. The health system once ran nine EHRs under a best-of-breed approach. During COVID, changing regulations forced quick adjustments across nine EHRs, and that helped push the organization toward one platform.

Measurable Goals Replace Yes-or-No Targets

As CIO, Carmody is pushing his leadership team to set measurable goals. He has an undergraduate degree in accounting and started at UPMC as an IT auditor. He is now asking his leaders to replace soft or yes-or-no goals with outcomes tied to the organization’s mission. In addition, he described a 30-minute management team meeting and said those discussions should center on the top three to five priorities. It can’t be a list of 27 things, he said.

Frequent measurement is the point of that focus. Carmody said he does not want a systemic IT issue to run for six months before anyone looks at the numbers. He also wants risks addressed before they become problems. Meanwhile, the team carries about 160 projects beyond the Epic work. He therefore relies on his managers, and on trends in the data, to show where progress has stalled.

He also reads a flat trend as a warning. “If we’re flat, if we’re complacent, that’s when we should be uncomfortable,” Carmody said. “We should always be looking for ways to improve and get better.”

Success, for Carmody, shows up in what clinicians and patients stop noticing. “When technology fades to the background, and that experience, that encounter, is the best it can be, then we did our job,” he said.

Get the next one in your inbox.

New interviews and webinars with health system IT, security and informatics leaders.

More coverage: AI in Health Systems →

Related Articles

ShareShare

We will be happy to hear your thoughts

Leave a reply

Som2ny Network
Logo
Register New Account
Compare items
  • Total (0)
Compare
0
Shopping cart