AI risk in healthcare: what every clinician and business leader should know

As healthcare organizations increasingly adopt artificial intelligence to drive clinical insights, operational efficiency, surgical procedures and patient care, it is essential to recognize that AI introduces new dimensions of risk and security that go beyond traditional IT concerns.

The following are key considerations and safeguards that every clinician and business leader should understand because they represent real risk to data integrity and patient safety.

AI systems are vulnerable to attack

Targeted attacks can manipulate input data to trick AI models into making incorrect decisions that could lead to misdiagnosis, inappropriate treatment plans or worse. Data poisoning corrupts the data used to train AI models, biasing outcomes and undermining trust in AI-enabled systems. More sophisticated attacks can bypass controls and extract sensitive patient healthcare data, posing serious privacy and compliance risk and exposing providers to consequences.

Unlike traditional healthcare systems, AI models and AI-enabled systems can be extremely complex and difficult to understand and employ, especially for non-technical people making critical business decisions. The complexity and rapid adoption of AI is a combination that can significantly increase risk.

Validation, auditability, and trust

For clinicians and business leaders, it is critical that AI systems are architected with security and controls to minimize those risks. AI-driven decisions that impact patient care must be reviewed, validated and audited. That means creating mechanisms to monitor, interpret and if necessary, challenge the outputs and recommendations of AI systems. This is necessary to ensure all decisions related to patient care and business operations are clear, valid and trustworthy.

The double-edged sword

AI itself is a double-edged sword. It can enhance our business and clinical effectiveness and efficiency as well as enhance our cybersecurity operations capabilities. While this represents the "good" side of AI, bad actors are also leveraging AI to automate and upscale their attack capabilities. This includes sophisticated phishing campaigns that are perfectly crafted and difficult to detect, and the ability to automate and adapt advanced attacks without human intervention.

At the same time, they are using AI to analyze systems for vulnerabilities around the clock. Defenders must be constantly evolving our security posture to anticipate and counter these AI-driven threats. This means the effective use of good AI to counter bad AI. This is not optional.

Drift, bias, and continuous monitoring

AI models and the data they rely on are not static. Over time we see drift, bias or even deliberate manipulation that can degrade model performance, data integrity and safety. Continuous monitoring of both inputs, data validation and AI outputs is essential to detect anomalies early and maintain the integrity of clinical and operational decisions.

A rapidly evolving regulatory landscape

The regulatory landscape for AI in healthcare is rapidly evolving, with an emphasis on transparency, accuracy and accountability. Compliance is not a technical issue — it is a business issue that requires collaboration across the business functions to ensure AI systems meet legal and ethical standards, are closely monitored and validated to ensure patient care and data integrity are not impacted or compromised. This is necessary to protect the organization.

The human imperative

Despite the power and promise of AI and the impact it is making in reshaping the world and how we do everything, human oversight remains an operational imperative. Clinicians and business professionals must remain engaged in reviewing AI-driven outputs and recommendations, especially in high-stakes scenarios involving patient care. This "people-in-the-loop" approach will help identify issues that may be overlooked by automated systems, thus ensuring patient safety and business resilience.

By proactively addressing these AI-specific risks and implementing robust controls and safeguards, we can effectively harness the power and benefit of AI in healthcare, while protecting patients, data and organizational reputation. Collaboration across clinical, business, technology, security and compliance teams is essential to building and maintaining a resilient, effective and trustworthy healthcare environment.