Differentially Private Models Can Hurt Fairness

Chart of differentially private models losing accuracy on underrepresented groups

Differentially private models limit how much a trained system can reveal about any single person’s data. That protection has a price that falls unevenly. Experiments on handwritten digits show accuracy drops first and hardest for rare, underrepresented groups.

The result matters for anyone training on sensitive data, from music tastes to medical records.

How a model leaks its training data

Take a model trained on people’s favorite bands. Someone who probes it the right way might recover one person’s listening history. Credit card numbers have been pulled out of language models, and real faces have been reconstructed from image models.

A toy case shows the mechanism. Two models separate red and blue dots using datasets that differ in one way: a single isolated point is red in one and blue in the other. After training, the two decision boundaries differ near that point. Anyone with access to the model could guess the point’s color. If the color stood for something like a voting record, that guess is a privacy breach.

Clipping and noise

Gradient descent trains models through many small steps. Differential privacy adds two operations to each step: clip the gradient so no single example can pull too hard, and add random noise. The more the possible models for the red and blue outlier overlap, the more plausible deniability that person has.

Done correctly, the guarantee is that a model trained on a dataset is essentially indistinguishable from one trained on the same data with a single point changed. In low dimensions only a few points are outliers. In higher dimensions many more are, and more information can be extracted.

What it does to MNIST digits

In earlier research, models with increasing privacy were trained on MNIST digits, and each training digit was ranked by the highest privacy level at which it was still classified correctly. Odd examples fell first, such as a “3” that looks like a “2” or a “9”. Clean, typical digits survived even at high privacy. In some circumstances, privacy can even help generalization by limiting the pull of spurious examples.

The cost shows up in accuracy. With 7,500 training points, reaching 90% accuracy on MNIST needs an extremely low privacy level, and more privacy lowers accuracy quickly. With 60,000 points, 90% accuracy is reachable at a stronger privacy level than almost all real deployments use.

Who pays the price

Classes differ in difficulty. Recognizing an “8” with high confidence takes more data or less privacy than recognizing a “0”. If one class has fewer examples, the problem gets worse, and predicting an uncommon event with a private model can take an enormous amount of data.

Outliers also cluster within classes. A model can score well on a class overall and badly on a small group inside it, like left-slanted “1”s. Nothing makes those digits inherently harder. There are just few of them, so learning them without leaking information is difficult.

The pattern is not limited to digits. Stronger privacy has disproportionately reduced accuracy on underrepresented subgroups in image and language models. In a medical model, adding privacy cut the influence of Black patients’ data while raising the influence of white patients’ data.

Weakening privacy does not fix this. Data from minority groups is the most exposed to leaks.

Why rebalancing doesn’t help

Rebalancing without new data fails both ways. Upsampling the smaller class reduces privacy, and downsampling the larger class removes data and lowers accuracy. A test with 90-degree-rotated digits as a stand-in for a small subgroup made the point. Models without any rotated digits never beat random guessing on them. With 5% rotated digits, only models with plenty of data and low privacy classified them well. At 10% to 20% or more, higher-privacy models could do well on both kinds of digits with the same amount of data.

Extra data has diminishing returns, and the small group, often near the edge of being learned, loses accuracy faster when effective data shrinks.

Open questions and next steps

Several options might soften the tradeoff.

  • When public unlabeled data exists, Private Aggregation of Teacher Ensembles can replace clipping and noise and has a smaller disparate effect on accuracy.
  • Architectures designed for privacy and better dataset cleaning are active research areas.
  • Collecting more data from underrepresented groups remains the clearest way out.

There are further tradeoffs among accuracy, privacy and fairness. Even a private model with small accuracy gaps can lose privacy or accuracy when fairness metrics are enforced. Protecting individual data points also falls short for users who contribute many points each.

The issue reaches beyond machine learning. The 2020 US Census was the first to use differential privacy, and allocating resources from a private dataset can affect groups unevenly, including how congressional districts are drawn.

Source: pair.withgoogle.com

Written by

Saksham Bhargava

Saksham Bhargava runs Alteroid and has been writing about consumer technology for over five years, covering iPhone, Android, Windows, AI tools and gaming. With an MCA and six years of hands-on web development experience, he tests the features he writes about on his own devices and focuses on clear, practical guides that solve real problems. Away from the screen, he enjoys travelling to places most tourists skip.