An Introduction to Differential Privacy

Key Takeaways

  • Differential privacy can be achieved by adding randomised “noise” to an aggregate query result to protect individual entries without significantly changing the result.
  • Differentially private algorithms guarantee the that the attacker can learn virtually nothing more about an individual than they would learn if that person’s record were absent from the dataset.
  • One of the simplest algorithms is the Laplace mechanism, which can post-process results of aggregate queries.
  • Both Apple and Google are making using of differential privacy techniques in iOS and Chrome respectively.  Differentially private algorithms have also been implemented in privacy-preserving analytics products, such as those developed by Privitar.
  • Differentially private algorithms are still an active field of research.

Differential privacy leap from research papers to tech newsworthiness headlines last class when, in the WWDC keynote, Apple VP of Engineering Craig Federighi announced Apple ’ sulfur use of the concept to protect drug user privacy in io .
It was the latest case of a general tendency : users and engineers awakening to the importance of privacy auspices in software. High-profile privacy violations such as Uber ’ s “ God mode ” have demonstrated in austere terms the relief with which employees of a company can misuse sensitive data gathered from their customers .
The sum of sensitive data that is being digitally recorded is quickly increasing. People now rely on digital services for more of their payments, transportation, seafaring, patronize, and health than ever. This newly data solicitation creates ever increasing ways to violate privacy.

    Related Sponsored Content

But it besides creates exciting opportunities—to improve transportation networks, to reduce crime, to cure disease—if made available to the right datum scientists and researchers. There is a natural latent hostility between protecting the privacy of individuals in the dataset and enabling analytics that could lead to a better world .
differentially private algorithms are a bright technical solution that could ease this latent hostility, allowing analysts to perform benign aggregate analysis while guaranteeing meaningful security of each individual ’ s privacy .
This development field of technology is deserving considering in any system that seeks to analyze sensitive data. Although the derived function privacy guarantee was conceived of only ten years ago, it has been successful in academia and diligence. Researchers are quickly inventing and improving differentially private algorithm, some of which have been adopted in both Apple ’ s io and Google ’ s Chrome .

This article discusses the historic factors leading to differential privacy in its current form, along with a definition of differential gear privacy and example differentially secret algorithm. It then looks at some holocene high-profile implementations of differentially individual algorithm by Google, Apple, and others .


differentially private algorithms are the latest in a decades-old field of technologies for privacy-preserving data analysis. Two earlier concepts directly influenced differential privacy :

  1. Minimum query set size
  2. Dalenius’s statistical disclosure definition.

We ’ ll explain these first base as they provide helpful background for differential privacy .
Minimum query set size The first concept is a minimum question set size, which—like differentially private algorithms—aims to ensure the guard of sum queries. Aggregate queries are those where the returned value is calculated across a subset of records in the dataset, such as counts, averages, or sums. It may be helpful to think of aggregate queries as SQL queries that begin with “ SELECT SUM ”, “ SELECT COUNT ”, or “ SELECT AVG ”. other types of aggregate queries include contingency tables and histograms .
A minimum question set size is a constraint that seeks to ensure that aggregate queries can not leak information about individuals. Given some configured brink number T, it ensures that every sum question is conducted on a set of at least T records. A minimum question set size would block aggregate queries that targeted fewer than T individuals. For exemplify, if T=2, it would block the follow :
“ SELECT AVG ( wage ) WHERE appoint = ‘ Troy Brown ’ ; ” .
because this question would conduct an average over one record ( we assume there is entirely one Troy Brown ) .
Using minimum question set sizes prevents certain attacks, but does not come with a privacy guarantee and, in practice, can be circumvented by skilled attackers. For example, the attacker could accomplish the above attack with :
“ SELECT SUM ( wage ) ; ” .
“ SELECT SUM ( wage ) WHERE mention ! = ‘ Troy Brown ’ ; ” .
Or even, if we know Troy Brown ’ sulfur senesce ( 45 ) and position ( WR ) uniquely identify him :
“ SELECT SUM ( wage ) WHERE position = ‘ WR ’ ; ” .
“ SELECT SUM ( wage ) WHERE stead = ‘ WR ’ AND age ! = 45 ;
such attacks are called tracker attacks, and they can not be stopped by a minimal question set size constraint. Because of these attacks, minimal question set sizes were deemed an inadequate defense for protecting question systems ( see Denning ’ s knead ). Something better, with a guarantee, was needed to ensure privacy .
Dalenius’s statistical disclosure definition
In 1977, statistician Tore Dalenius proposed a hard-and-fast definition of data privacy : that the attacker should learn nothing about an individual that they didn ’ deoxythymidine monophosphate know before using the sensible dataset. Although this undertake failed ( and we will see why ), it is authoritative in understanding why differential privacy is constructed the way it is .
Dalenius ’ s definition failed because, in 2006, computer scientist Cynthia Dwork proved that this undertake was impossible to give—in other words, any entree to sensitive data would violate this definition of privacy. The trouble she found was that certain types of background information could constantly lead to a fresh conclusion about an individual. Her proof is illustrated in the pursuit anecdote : I know that Alice is two inches tall than the median lithuanian womanhood. then I interact with a dataset of lithuanian women and compute the average height, which I didn ’ thyroxine know before. I now know Alice ’ s height precisely, even though she was not in the dataset. It is impossible to account for all types of background data that might lead to a fresh stopping point about an individual from use of a dataset.

Dwork, after proving the above, proposed a new definition : differential gear privacy .

What is differential privacy?

derived function privacy guarantees the follow : that the attacker can learn about nothing more about an individual than they would learn if that person ’ south commemorate were absent from the dataset. While weaker than Dalenius ’ s definition of privacy, the guarantee is strong enough because it aligns with real world incentives—individuals have no incentive not to participate in a dataset, because the analysts of that dataset will draw the same conclusions about that individual whether the individual includes himself in the dataset or not. As their sensitive personal information is about irrelevant in the outputs of the system, users can be assured that the organization handling their data is not violating their privacy .
Analysts learning “ about nothing more about an individual ” means that they are restricted to an insignificantly little change in belief about any person. ( here and below, “ change ” refers to the change between using a dataset and using the lapp dataset minus any one person ’ s record. ) The extent of this change is controlled by a parameter known as ϵ, which sets the bound on the exchange in probability of any result. A low value of ϵ, such as 0.1, means that very short can change in the beliefs about any person. A high value of ϵ, such as 50, means that beliefs can change well more. The courtly definition is as follows .
An algorithm A is ϵ-differentially secret if and only if :
Pr [ A ( D ) = adam ] ≤ e^ϵ * Pr [ A ( D ’ ) = ten ]
for all x and for all pairs of datasets D, D ’ where D ’ is D with any one record—i.e. one person ’ sulfur data—missing. The symbol east refers to the numerical constant. note that this definition only makes smell for randomized algorithm. An algorithm that gives deterministic output signal is not a campaigner for differential privacy .
The primary entreaty of the derived function privacy guarantee is its limitation on the come that any analyst can learn about an individual. additionally, it has the follow utilitarian properties :

  • Composability: if two queries are answered with differential privacy guarantees of level ϵ1 and ϵ2, the pair of queries is covered by a guarantee of level (ϵ1 + ϵ2). Recall that a higher value of ϵ means a weaker guarantee.
  • Strength against arbitrary background information: the guarantee does not rely in any way on what background information the attacker knows. This property is one of the principal reasons that differential privacy is stronger than an earlier privacy guarantee, k-anonymity.
  • Security under post-processing: there are no restrictions on what can be done with a differentially private result – no matter what it is combined with or how it is transformed, it is still differentially private.

How is this undertake achieved in software ? Differentially private algorithms are randomized algorithms that add randomness at key points within the algorithm. One of the simplest algorithm is the Laplace mechanism, which can post-process results of aggregate queries ( for example, counts, sums, and means ) to make them differentially individual. Below is model Java code for the Laplace mechanism specific to count queries :

import org.apache.commons.math3.distribution.LaplaceDistribution;

double laplaceMechanismCount(long realCountResult, double epsilon) {

    LaplaceDistribution ld = new LaplaceDistribution(0, 1 / epsilon);
    double noise = ld.sample();
    return realCountResult + noise;


The key parts of this routine are

  1. Instantiate a Laplace probability distribution (see Figure 1) centered at 0 and scaled by 1/ϵ. We use the Apache Commons implementation, “LaplaceDistribution”, which is constructed with two arguments: the mean of the distribution, and the scale of the distribution. Note that lower epsilon (more privacy) leads to a larger scale and thus a wider distribution and more noise.
  2. Draw one random sample from this distribution. This sample() function takes a random number between 0 and 1 and applies the Laplace distribution’s inverse cumulative distribution function (CDF) to this number. This process results in a random number such that its likelihood of being any specific value matches the distribution. As an alternative way to think about it, if this sample function were invoked a million times to get a million samples, the shape of the histogram of these samples would closely match the shape of the Laplace distribution.
  3. Perturb the real value by adding the random value from step 2.

Let ’ s consider why this algorithm is differentially private by taking the point of view of an attacker named Eve. Say the data set is mental health data, and Eve has conceived of a tracker attack ( see above ) that will reveal whether her aim, Bob, receives counseling for alcoholism or not. If the question ’ second result is 48, Eve knows that Bob does receive guidance ; if it ’ mho 47, Eve knows the face-to-face .
Whether the answer is 47 or 48, the Laplace mechanism will return a noisy leave somewhere around 47 or 48. It may return 49, 46, or even, with smaller probability, 44 or 51 ( see Figure 2 for a histogram ). In drill, it is impossible for Eve to be very sure about whether the true answer was 47 or 48 and, thus, her beliefs about whether Bob is in counseling for alcoholism or now will not meaningfully change .

Figure 1: The Laplace distribution centered at 0 with scale of 1. Pictured is the probability density function (PDF) of the distribution—the y-axis is the relative likelihood that the variable will take the value on the x-axis.

figure 2 : The likely outcomes of the count question for the two scenarios, where the real answer is 47 and when it ‘s 48. A minor number of outputs would not be enough to distinguish which distribution they came from. Epsilon is set to 0.67 .
You may have observed by this detail that Eve could cut through the noise by repeating the question many times, and seeing whether the answers cluster around 47 or 48. To prevent this tactic, differentially secret systems must have a “ privacy budget, ” a hood on the sum of the ϵ ’ south used in each question. This cap works because of differential privacy ’ s composability property described above. They may ask a few relatively low-noise queries, or many hundreds of high-noise queries, but either manner, they will not be able to confidently establish whether the true answer is 47 or 48 .
last, note that the Laplace mechanism for counts is merely one dim-witted differentially individual algorithm. The Laplace mechanism can be extended to work for sums and early aggregate queries. furthermore, there are basically different algorithm that have been proven to satisfy the differential privacy guarantee. A few worth exploring are the Private Multiplicative Weights algorithm, the Multiplicative Weights Exponential Mechanism, and DualQuery .

Differential privacy in action

In June 2016, Apple announced that it would begin to use differentially private algorithm to collect behavioral statistics from iPhones. This announcement, besides causing a huge spike in differential privacy interest, showed that differential privacy can help major organizations get new prize from data that they previously did not touch due to privacy concerns .
Although Apple has then far made few details public, the algorithm used in the iPhone seems like to Google ’ randomness RAPPOR project. Google implemented a feature in Chrome that collects behavioral statistics from Chrome browsers through a differentially individual randomized response algorithm .
In randomized reaction, random noise is added to statistics before they are submitted to the collector. For example, if the real statistic is 0, the browser will with some probability replace 0 with a randomly selected 0 or 1. Each drug user has a bombastic degree of deniability about the measure that their software reports, because it could be a random respect. But jointly, the sign stands out over the random noise and the arrangement collecting the statistics ( i.e. Google or Apple ) can accurately observe trends .
interestingly, neither Google nor Apple, to our cognition, has revealed the prize of ϵ that goes with their differential privacy guarantee. We need this value to understand the protection offered by the guarantee. If they use a high enough prize of ϵ, analysts can still learn sensible facts about users with high confidence. A broken value of ϵ is required for meaningful privacy protective covering .
differentially private algorithms have besides been implemented in privacy-preserving analytics products, such as those developed by my employer Privitar. These products allow companies that work with valuable, sensitive data to incorporate differentially private algorithms into their data architecture, providing privacy guarantees to their users while silent performing meaningful analysis on the datum.

Looking ahead

Both the engineering and research communities have paths forth with differential privacy. For engineers, the job is to become educated on differential privacy and ensure that it is used where appropriate for drug user privacy protective covering. For researchers, it is to find more and better differentially private algorithm, improving the toolset with which we can enable privacy-preserving data analytics .
We all stand to gain from the establishment of privacy guarantees and the successes of data analytics. For both reasons, we look advancing to more organizations embracing differential gear privacy .

About the Author

Charlie Cabot is a aged data scientist at Privitar, a datum privacy inauguration that builds high-performance software for data anonymization, including perturbation and abstraction algorithm and differentially individual mechanisms, to facilitate safe habit of sensitive datasets. Charlie focuses on demonstrable privacy guarantees and the statistical impact of anonymization on analytics and data science. previously, working in cyber security, Charlie engineered machine learning-driven approaches to malware detection and modeled cyber attacks on calculator networks .

Leave a Reply

Your email address will not be published.