Designing AI around the Workflow for Better Insights and Trust

(US and Canada) Description: A typical approach to applying AI to a business problem is getting existing data and modeling the AI for insightful predictions. If the AI predictions have yet to win the users’ trust, especially when existing data alone cannot get AI to the threshold of being insightful, designing AI to collaborate with the users may be a solution.

Many problems today can use the help of Artificial Intelligence (“AI”), but can’t because of a variety of non-technical reasons. These problems typically trigger the question of “whose responsibility” as the AI inevitably would fail in some corner cases. Who’s responsible when the AI misdiagnosed, or steered the self-driving car into a divider? Distrust of AI alone prompts the affected parties to push back, as evident in a Boston University survey: patients believe that their medical needs are unique and cannot be adequately addressed by “algorithms”. We simply cannot let AI make decisions alone, as the stakes are high if it fails.

Human-AI Collaboration

The paper by Jessy Lin laid out 3 different approaches to allow us to still leverage AI in such problems. We would build the AI around the human workflow of interest such that the human worker either directs the AI, takes insights from the AI, or participates with the AI in the process of completing the task. Allen Hullinger, a long-time practitioner of the Lean methodology at Sutter Health, favorably compared these approaches to Jidoka of the Toyota Production System, which centers around automation whereby the human-machine interactions enable continual improvements on the machine, as well as corresponding human resource development.

Let’s look at an example use case in Healthcare. Sepsis is a potentially life-threatening condition caused by the body's response to an infection. 1 in 3 patients who died in a hospital has sepsis, and approximately 1.7 million adults develop sepsis in the U.S. each year according to the CDC. AI researchers and Healthcare IT companies have invested in using AI for sepsis detection over the years. Naturally many chose to use the Electronic Medical Records (EMR) systems that the clinicians spend significant time interacting with as the vehicle to deliver AI sepsis predictions, in the form of alerts. The trouble is when the AI’s sensitivity (true positive rate) and specificity (true negative rate) are not good enough, clinicians are getting too many false alarms. They become alert-fatigue and start to ignore the alerts.  

The consequences of ignoring alerts can be even more dire. An experienced clinician can relatively quickly conclude whether sepsis is a risk from their knowledge about the patient, especially the patient they have been working with since the onset of an episode. Alerting them of sepsis risk does not add any new insight. Worse yet, low sensitivity and/or specificity means the alerts can be quite frequently wrong. When a true-positive alerts the clinicians, and they are not aware of what actually happened, the AI will still fail to save lives because the insightful alert would have been ignored like the rest of the alerts.

What if instead of having AI autonomously make predictions based on patient attributes (including vitals and lab values), we have the clinicians “collaborate” with the AI in the process?  

Designing AI around a Workflow

This Beth Israel Deaconess research paper found that clinicians’ descriptions of an infection or infection symptoms, if incorporated into the AI model, can significantly improve sepsis predictions (from AUC of 0.67 to 0.86 for the test dataset in their research). They used the free-text notes from the care-delivery workflow as an additional input. In their findings, terms indicative of an infection like “cellulitis”, “sore throat”, and “abscess” are quite predictive. However, these free-text notes also come with their typical issues such as misspelled words and abbreviations. In addition, the clinicians who jotted down the notes at the point of care might not have used the descriptions the AI has previously seen, or might not have described an infection during the encounter.  

Imagine an alternate workflow where the AI is also present in the care-delivery workflow above.  The AI can take the shape that is the least intrusive, be it an agent listening in, a smart prompt (e.g. based on the patient’s symptoms and medical record) during note entry, or something else. Either way, it anticipates sepsis, and tries to gather additional predictive features. This approach can be generalized beyond sepsis to other critical conditions. AI-clinician collaboration allows the clinician to convey more relevant information to the AI, and the AI to subtly accumulate the data it needs to arrive at its final conclusion. Over time, the AI improves as it uncovers new predictive features, and the clinicians may also start to trust the AI more via this closer “working relationship” .

It is always crucial to consider the workflow when we introduce a change (AI in this case). An important observation here is that we are designing AI around the workflow. Implementing workflow awareness allows AI to collaborate with the clinicians. This is yet another example of the phenomenon I described as “AI encapsulates the workflow”. This collaboration is also a good use case that supports the general industry point of view that AI cannot replace doctors but will certainly make doctors better.

Cheong has been a hands-on leader in web, data and healthcare IT for two decades. His early work on web interactivity at UCSF Health resulted in patented technologies licensed to leading firms including Microsoft, Oracle and Adobe.

Before AI became trendy, he led teams at IBM in development and deployment of software systems for data mining and predictive analytics in e-commerce and enterprise knowledge management. He had been in director positions in charge of operating a targeted advertising system, and building and marketing a broadband TV delivery system.

As a senior consultant at IBM, he led teams in designing and implementing IT projects for clients in healthcare and financial industries. He has also engaged in business and innovation strategy consulting with Konica Minolta, and Zuckerberg San Francisco General Hospital.

Most recently, as the CTO of LucidAct, he brought together a team to bridge the gap between AI and the frontline workers with a SaaS system that enables smart collaborations and continuous learning toward organizational goals. The system is currently serving tens of thousands of patients at multiple healthcare organizations.

In his work, Cheong has seen how agility, AI, and automation come together to advance healthcare workflows, which he simply calls “Workflow AI”. This experience, and his years with UCSF Health, IBM, and the startup ecosystem give him a unique Point of View of Healthcare IT from the perspectives of Providers, Big Techs, and Startups, which he is eager to share.

To connect with Cheong on LinkedIn: https://www.linkedin.com/in/cheong/