Most organisations implement AI without considering incident response, leaving them poorly prepared when AI failures and attacks occur. In this post, we explore the reasons behind this and provide you with actionable insights based on lessons learned from developing AI incident response plans.
Understanding AI’s unique challenges
AI presents distinct challenges compared to traditional software.
It’s highly complex, with millions or billions of rules and parameters considering multiple inputs, making it challenging to identify when an external attacker manipulates it. For example, imagine an AI system for fraud detection in financial transactions. It analyses vast data and applies intricate rules to identify suspicious patterns. However, due to its complexity, detecting if a malicious actor intentionally manipulates the AI system by feeding it misleading data or exploiting its vulnerabilities becomes challenging.
Also, most AI systems provide probabilistic outcomes, meaning their decisions and predictions are not always certain but have a certain degree of uncertainty. Let’s take an AI-powered medical diagnosis system as an example. It may analyse various symptoms and medical data to provide a probability-based diagnosis. For instance, it might indicate a patient has a 70% chance of having a particular disease based on the available information. This probabilistic nature of AI systems requires meticulous testing and the establishment of acceptable thresholds for failure. It means that organisations must determine the level of confidence they need in the AI system’s accuracy and set appropriate boundaries for potential errors or misclassifications.
Moreover, AI systems experience drift towards failure as their understanding becomes less valid over time due to the dynamic nature of the real world. To illustrate this, consider an AI system designed to predict stock market trends. It learns from historical data to make predictions, but as market conditions change, the AI system’s understanding of the market may become outdated, leading to less accurate predictions over time. This phenomenon is known as “concept drift” or “model decay.”
Leveraging similarities with traditional software
While AI possesses unique characteristics, it’s vital not to reinvent the wheel. Organisations can build upon existing documentation, testing, management, and monitoring practices for traditional software. They can leverage their established processes by incorporating AI incident response plans into existing frameworks and avoid reinventing the wheel.
Insights from responding to AI incidents
To effectively respond to incidents involving AI, it is necessary to employ a combination of practices such as model risk management, computer incident response, and information security best practices.
Let’s break them down and provide some examples to make it easier to understand.
Model risk management
Model risk management (MRM) practices form a solid foundation for mitigating risks associated with AI.
MRM refers to the practices and processes implemented to identify, assess, and mitigate risks associated with using models, such as AI algorithms, in decision-making. For instance, in AI-powered credit scoring systems, MRM practices involve rigorous testing, documentation, and ongoing monitoring to ensure accurate and fair decision-making.
However, it’s worth noting that MRM practices alone do not explicitly cover AI security or incident response, which brings us to the next point.
Remember your current incident response plan
Traditional incident response guidance is a starting point for addressing various security incidents involving AI. For example, when an AI system used in autonomous vehicles experiences a malfunction or produces unexpected behaviour, following established incident response procedures can help identify the root cause, mitigate risks, and restore normal operations. However, organisations must go beyond traditional guidance and consider AI-specific attack vectors to avoid blind spots.
AI-specific attack vectors refer to the unique ways malicious actors can target or manipulate AI systems. For instance, an attacker might introduce subtle alterations in training data to deceive an AI-powered facial recognition system, allowing unauthorised access to secure facilities. Organisations can safeguard their AI systems by recognising these vulnerabilities and incorporating countermeasures into incident response plans.
By combining these practices, addressing AI-specific risks, and tailoring incident response plans accordingly, organisations can enhance their ability to detect, respond to, and recover from AI technology incidents.
Culture, change management and communication
Culture, change management, and communication are crucial for AI incident response because they:
- Drive a cultural shift: Embracing AI incident response requires a positive and accountable organisational culture.
- Enable effective change management: Smooth integration of AI incident response practices requires preparing employees, providing training, and addressing concerns.
- Facilitate communication and awareness: Clear communication ensures stakeholders understand incident response protocols, roles, and reporting mechanisms.
- Support timely response: Open dialogue and communication channels help identify and address incidents promptly, minimising their impact.
- Foster continuous improvement: Promoting a learning environment allows organisations to gather feedback, share lessons learned, and enhance incident response capabilities over time.
You need an AI incident response plan
Deploying AI without an incident response plan is a recipe for disaster. So, treat AI as a critical activity and proactively plan for AI failures.
Actions to take next
- Know how to respond to AI incidents by asking us to workshop the topic with your team.
- Plan for AI incidents effectively by asking us to draft or review your AI incident response plan.
- Respond to AI incidents promptly by reaching out to us for incident response coaching.
- Recover from AI incidents safely by asking us for a framework for incident recovery.
