Let’s explore a new AI security risk: membership inference attacks.

What is it?

It’s an AI attack during which an attacker tries to determine if you have used a particular person’s personal information to train a machine learning model. The attacker’s goal is to access the person’s personal information.

As a side note, it’s different from model inversion attacks.

How does a membership inference attack happen?

In this attack, the attacker first trains a separate machine learning model, known as a membership inference model, on the output of the target model.

When we talk about the output of a model, it’s kind of like the answer that the computer gives you after it has learned something. It’s like when you ask a question, and someone gives you an answer; the output is like the computer’s answer. For example, if we teach the computer to recognise pictures of toes, the output might be whether the picture has a toe in it or not. So basically, the output is the computer’s response after it has learned something.

The attacker trains the model to predict whether a particular data point (personal information) was part of the training dataset for the target model (your model).

An example

Suppose you train a model to predict whether a customer will likely default on a loan based on their credit history. An attacker who does not have access to the training data for the model could use a membership inference attack to determine whether the person’s credit history was part of the training dataset. If the attack succeeds, the attacker can determine whether the person is likely to default on a loan, which is valuable information they could sell or use to harm or blackmail the person.

How to prevent these attacks

These attacks can be prevented through various security techniques such as differential privacy, adversarial training, and regularisation:

  1. Differential privacy. It’s a technique you can use to protect personal information by adding noise to it. The idea is to make it hard for an attacker to determine whether you included a data subject’s personal information in a dataset. An expert would carefully calibrate the noise to preserve the overall accuracy of the dataset while still providing a high level of privacy protection.
  2. Adversarial training. This security technique involves training a model on both normal and adversarial examples. Adversarial examples are like tricky puzzles that can fool computers. For instance, you can draw a picture of a cat and show it to AI, and they can tell you it’s a cat. Well, with adversarial examples, someone draws a picture that looks like a cat to you and me, but to a computer, it might look like a picture of a dog or something else entirely. So even though it seems like a cat to us, the computer gets confused and thinks it’s something else. So you specifically craft adversarial examples to fool the model and use them to identify its vulnerabilities. Training on normal and adversarial examples makes the model more robust and can better defend against attacks.
  3. Regularisation. It’s a technique to prevent overfitting in machine learning models. Overfitting occurs when a model becomes too complex and starts to fit the noise in the data rather than the underlying patterns. This can result in the model performing very well on the data it was trained on but poorly on new data it hasn’t seen before. Regularisation works by adding a penalty term to the loss function that encourages the model to be more straightforward and generalisable. This means that the model will fit the training data and perform well on new data. To use an analogy, think of overfitting as memorising answers to a test. If you memorise the answers to a specific test, you may do well on that test but not on similar tests. However, if you understand the concepts and principles, you can perform well on various tests. Regularisation is like understanding the concepts and principles of a subject. It helps the model learn the underlying patterns and concepts in the data rather than just memorising specific examples.

Actions you can take next

  • Manage the data protection risks of your AI projects by asking us to join our data protection programme.
  • Understand how AI systems impact your data protection compliance efforts by asking us to draft a legal opinion.
  • Explore how your organisation could comply by consulting with us regarding artificial intelligence and data protection laws.
  • Take steps to disclose your use of AI systems by engaging us to help you draft a new or updated privacy policy.