the promise of differential privacy n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
The Promise of Differential Privacy PowerPoint Presentation
Download Presentation
The Promise of Differential Privacy

Loading in 2 Seconds...

play fullscreen
1 / 72

The Promise of Differential Privacy - PowerPoint PPT Presentation


  • 144 Views
  • Uploaded on

The Promise of Differential Privacy. Cynthia Dwork, Microsoft Research . TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: A A A A A A A A. NOT A History Lesson. Developments presented out of historical order; key results omitted. NOT Encyclopedic.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'The Promise of Differential Privacy' - yama


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
the promise of differential privacy

The Promise of Differential Privacy

Cynthia Dwork, Microsoft Research

TexPoint fonts used in EMF.

Read the TexPoint manual before you delete this box.: AAAAAAAA

not a history lesson

NOT A History Lesson

Developments presented out of historical order; key results omitted

not encyclopedic

NOT Encyclopedic

Whole sub-areas omitted

outline
Outline
  • Part 1: Basics
    • Smoking causes cancer
    • Definition
    • Laplace mechanism
    • Simple composition
    • Histogram example
    • Advanced composition
  • Part 2: Many Queries
    • Sparse Vector
    • Multiplicative Weights
    • Boosting for queries
  • Part 3: Techniques
    • Exponential mechanism and application
    • Subsample-and-Aggregate
    • Propose-Test-Release
    • Application of S&A and PTR combined
  • Future Directions
basics

Basics

Model, definition, one mechanism, two examples, composition theorem

model for this tutorial

?

Model for This Tutorial
  • Database is a collection of rows
    • One per person in the database
  • Adversary/User and curator computationally unbounded
  • All users are part of one giant adversary
    • “Curator against the world”

C

databases that teach
Databases that Teach
  • Database teaches that smoking causes cancer.
    • Smoker S’s insurance premiums rise.
    • This is true even if S not in database!
  • Learning that smoking causes cancer is the whole point.
    • Smoker S enrolls in a smoking cessation program.
  • Differential privacy: limit harms to the teachings, not participation
    • The outcome of any analysis is essentially equally likely, independent of whether any individual joins, or refrains from joining, the dataset.
    • Automatically immune to linkage attacks
differential privacy d mcsherry nissim smith 06

Pr [response]

Z

Z

Z

Bad Responses:

Differential Privacy [D., McSherry, Nissim, Smith 06]

M gives (ε,0) -differential privacy if for all adjacent x and x’, and all C µ range(M): Pr[ M (x) 2 C] ≤e Pr[ M (x’) 2 C]

Neutralizes all linkage attacks.

Composes unconditionally and automatically: Σiε i

ratio bounded

d differential privacy

Pr [response]

Z

Z

Z

Bad Responses:

(, d) - Differential Privacy

M gives (ε,d) -differential privacy if for all adjacent x and x’, and all C µ range(M ): Pr[ M (D) 2 C] ≤e Pr[ M (D’) 2 C] + d

Neutralizes all linkage attacks.

Composes unconditionally and automatically: (Σii ,Σidi )

ratio bounded

This talk: negligible

slide10

Range

Equivalently,

Useful Lemma [D., Rothblum, Vadhan’10]:

Privacy loss bounded by expected loss bounded by 2

“Privacy Loss”

sensitivity of a function
Sensitivity of a Function

Adjacent databases differ in at most one row.

Counting queries have sensitivity 1.

Sensitivity captures how much one person’s data can affect output

  • f = maxadjacentx,x’ |f(x) – f(x’)|
laplace distribution lap b
Laplace Distribution Lap(b)

p(z) = exp(-|z|/b)/2b

variance = 2b2

¾ = √2 b

Increasing b flattens curve

calibrate noise to sensitivity
Calibrate Noise to Sensitivity

f = maxadjx,x’ |f(x) – f(x’)|

Theorem [DMNS06]: On query f, to achieve -differential privacy, use

scaled symmetric noise [Lap(b)] with b = f/.

0

-4b

-3b

-2b

-b

b

2b

3b

4b

5b

Noise depends on f and , not on the database

Smaller sensitivity (f) means less distortion

example counting queries
Example: Counting Queries
  • How many people in the database satisfy property ?
    • Sensitivity = 1
    • Sufficient to add noise
  • What about multiple counting queries?
    • It depends.
vector valued queries
Vector-Valued Queries

f = maxadj x, x’||f(x) – f(x’)||1

Theorem [DMNS06]: On query f, to achieve -differential privacy, use

scaled symmetric noise [Lap(f/)]d.

0

-4b

-3b

-2b

-b

b

2b

3b

4b

5b

Noise depends on f and , not on the database

Smaller sensitivity (f) means less distortion

example histograms
Example: Histograms

f = maxadj x, x’ ||f(x) – f(x’)||1

Theorem: To achieve -differential privacy, use

scaled symmetric noise [Lap(f/)]d.

why does it work

Pr[ M(f, x – Me) = t]

=exp(-(||t- f-||-||t- f+||)/R)≤ exp(f/R)

Pr[ M (f, x+ Me) = t]

Why Does it Work ?

f = maxx, Me||f(x+Me) – f(x-Me)||1

Theorem: To achieve -differential privacy, add

scaled symmetric noise [Lap(f/)].

0

-4b

-3b

-2b

-b

b

2b

3b

4b

5b

simple composition
“Simple” Composition
  • k-fold composition of (,±)-differentially private mechanisms is (k, k±)-differentially private.
composition d rothblum vadhan 10
Composition [D., Rothblum,Vadhan’10]
  • Qualitively: Formalize Composition
    • Multiple, adaptively and adversarially generated databases and mechanisms
    • What is Bob’s lifetime exposure risk?
      • Eg, for a 1-dp lifetime in 10,000 ²-dp or (,±)-dp databases
      • What should be the value of ²?
  • Quantitatively
    • : the -fold composition of -dp mechanisms is -dp
    • rather than
adversary s goal guess b
Adversary’s Goal: Guess b

Choose b ² {0,1}

Adversary

(x1,0, x1,1), M1

(x2,0, x2,1), M2

(xk,0, xk,1), Mk

M2(x2,b)

Mk(xk,b)

M1(x1,b)

b = 0 is real world

b=1 is world in which

Bob’s data replaced

with junk

flavor of privacy proof
Flavor of Privacy Proof
  • Recall “Useful Lemma”:
    • Privacy loss bounded by expected loss bounded by
  • Model cumulative privacy loss as a Martingale [Dinur,D.,Nissim’03]
    • Bound on max loss ()
    • Bound on expected loss (22)
    • PrM1,…,Mk[ | iloss from Mi| > z A + kB] < exp(-z2/2)

A

B

extension to d dp mechanisms
Extension to (,d)-dp mechanisms
  • Reduce to previous case via “dense model theorem” [MPRV09]

(, d)-dp

Y

z

(, d)-dp

d- close

(,0)-dp

Y’

composition theorem
Composition Theorem
      • : the -fold composition of -dp mechanisms is -dp
  • What is Bob’s lifetime exposure risk?
    • Eg, 10,000 -dp or -dpdatabases, for lifetime cost of -dp
    • What should be the value of ?
    • 1/801
    • OMG, that is small! Can we do better?
  • Can answer low-sensitivity queries with distortion o)
    • Tight [Dinur-Nissim’03 &ff.]
  • Can answer low-sensitivity queries with distortion o(n)
    • Tight? No. And Yes.
outline1
Outline
  • Part 1: Basics
    • Smoking causes cancer
    • Definition
    • Laplace mechanism
    • Simple composition
    • Histogram example
    • Advanced composition
  • Part 2: Many Queries
    • Sparse Vector
    • Multiplicative Weights
    • Boosting for queries
  • Part 3: Techniques
    • Exponential mechanism and application
    • Subsample-and-Aggregate
    • Propose-Test-Release
    • Application of S&A and PTR combined
  • Future Directions
many queries

Many Queries

Sparse Vector; Private Multiplicative Weights, Boosting for Queries

slide26

Caveat: Omitting polylog(various things, some of them big) terms

Error

[Hardt-Rothblum]

Runtime Exp(|U|)

sparse vector
Sparse Vector
  • Database size
  • # Queries , eg, super-polynomial in
  • # “Significant” Queries
    • For now: Counting queries only
    • Significant: count exceeds publicly known threshold
  • Goal: Find, and optionally release, counts for significant queries, paying only for significant queries

insignificant

insig

insignificant

insig

insig

algorithm and privacy analysis
Algorithm and Privacy Analysis

[Hardt-Rothblum]

Algorithm:

When given query :

  • If : [insignificant]
    • Output
  • Otherwise [significant]
    • Output

Caution: Conditional branch leaks private information!

Need noisy threshold

  • First attempt: It’s obvious, right?
    • Number of significant queries invocations of Laplace mechanism
    • Can choose so as to get error
slide29

Algorithm and Privacy Analysis

Algorithm:

When given query :

  • If : [insignificant]
    • Output
  • Otherwise [significant]
    • Output

Caution: Conditional branch leaks private information!

  • Intuition: counts far below T leak nothing
    • Only charge for noisy counts in this range:
slide30

Let

  • denote adjacent databases
  • denote distribution on transcripts on input
  • denote distribution on transcripts on input
  • Sample
  • Consider
  • Show

Fact: (3) implies -differential privacy

slide31

Write as

“privacy loss in round ”

Define borderline event on noise

as “a potential query release on ”

Analyze privacy loss inside and outside of

borderline event case
Borderline eventCase

Release condition:

Borderline event

Definition of

Mass to the left of = Mass to the right of

  • Properties
  • Conditioned on round t is a release with prob
  • Conditioned on we have
  • Conditioned on we have
borderline event case1
Borderline eventCase

Release condition:

Borderline event

Definition of

Mass to the left of = Mass to the right of

  • Properties
  • Conditioned on round t is a release with prob
  • Conditioned on we have
  • Conditioned on we have

Think about x’ s.t.

borderline event case2
Borderline eventCase

Release condition:

Borderline event

  • Properties
  • Conditioned on round t is a release with prob
  • Conditioned on we have
  • (vacuous: Conditioned on we have )
slide35

Properties

  • Conditioned on round t is a release with prob
  • Conditioned on we have
  • Conditioned on we have

By (2,3),UL,+Lemma, )

By (1), E[#borderline rounds] = #releases

wrapping up sparse vector analysis
Wrapping Up: Sparse Vector Analysis
  • Probability of (significantly) exceeding expected number of borderline events is negligible (Chernoff)
  • Assuming not exceeded: Use Azuma to argue that whp actual total loss does not significantly exceed expected total loss
  • Utility: With probability at least all errors are bounded by .
  • Choose

Expected total privacy loss

slide37

Private Multiplicative Weights [Hardt-Rothblum’10]

  • Theorem (Main). There is an -differentially private mechanism answering linear online queries over a universe U and database of size in
    • time per query
    • error .

Represent database as (normalized) histogram on U

multiplicative weights
MultiplicativeWeights

Recipe (Delicious privacy-preserving mechanism):

Maintain public histogram (with uniform)

For each

  • Receive query
  • Output if it’s already accurate answer
  • Otherwise, output
    • and “improve” histogram

How to improve ?

slide39

Estimate

Input

Query

1

. . .

0

1

2

3

N

4

5

Before update

Suppose

slide40

Estimate

Input

Query

1

. . .

0

1

2

3

N

4

5

x 1.3

x 1.3

x 0.7

x 0.7

x 1.3

x 0.7

After update

slide41

Algorithm:

  • Input histogram with
  • Maintain histogram with being uniform
  • Parameters
  • When given query :
    • If : [insignificant]
      • Output
    • Otherwise [significant; update]
      • Output
        • where )
      • Renormalize
analysis
Analysis
  • Utility Analysis
    • Few update rounds
    • Allows us to choose
    • Potential argument [Littlestone-Warmuth’94]
    • Uses linearity
  • Privacy Analysis
    • Same as in Sparse Vector!
slide43

Caveat: Omitting polylog(various things, some of them big) terms

Error

[Hardt-Rothblum]

Runtime Exp(|U|)

boosting schapire 1989
Boosting[Schapire, 1989]
  • General method for improving accuracy of any given learning algorithm
  • Example: Learning to recognize spam e-mail
    • “Base learner” receives labeled examples, outputs heuristic
    • Run many times; combine the resulting heuristics
slide45

S: Labeled examples from D

Base Learner

Does well on ½ + ´ ofD

A

A1, A2, …

Combine A1, A2, …

Update D

Terminate?

slide46

S: Labeled examples from D

Base learner only sees samples, not all of D

Base Learner

Does well on ½ + ´ ofD

A

A1, A2, …

Combine A1, A2, …

Update D

Terminate?

How?

boosting for queries
Boosting for Queries?
  • Goal: Given database x and a set Q of low-sensitivity queries, produce an object O such that 8 q 2Q : can extract from O an approximation of q(x).
  • Assume existence of (²0, ±0)-dp Base Learner producing an object O that does well on more than half of D
    • Pr q» D [ |q(O) – q(DB)| < ¸ ] > (1/2 + ´)
slide48

Initially:

D uniform on Q

S: Labeled examples from D

Base Learner

Does well on ½ + ´ ofD

A

A1, A2, …

Combine A1, A2, …

Update D

slide49

(truth)

1

. . .

0

1

2

3

|Q |

4

5

Before update

slide50

(truth)

1

. . .

0

1

2

3

|Q |

4

5

x 0.7

x 1.3

x 0.7

x 0.7

x 1.3

x 0.7

After update

increased where disparity is large, decreased elsewhere

slide51

Initially:

D uniform on Q

S: Labeled examples from D

Base Learner

Does well on ½ + ´ ofD

A

Privacy?

Terminate?

A1, A2, …

median

Combine A1, A2, …

Update D

-1/+1

renormalize

Individual can affect

many queries at once!

privacy is problematic
Privacy is Problematic
  • In boosting for queries, an individual can affect the quality of q(At) simultaneously for many q
  • As time progresses, distributions on neighboring databases could evolve completely differently, yielding very different distributions (and hypotheses At)
    • Must keep secret!
    • Ameliorated by sampling – outputs don’t reflect “too much” of the distribution
    • Still problematic: one individual can affect quality of all sampled queries
privacy

Error x

Error x’

Privacy?

Error ofSton q

λ

“Good enough”

Queries q∈Q

privacy1
Privacy???

Weightof q

λ

by x

by x’

Queries q∈Q

private boosting for queries variant of adaboost
Private Boosting for Queries [Variant of AdaBoost]
  • Initial distribution D is uniform on queries in Q
  • S is always a set of k elements drawn from Qk
  • Combiner is median [viz. Freund92]
  • Attenuated Re-Weighting
    • If very well approximated by At, decrease weight by factor of e (“-1”)
    • If very poorly approximated by At, increase weight by factor of e (“+1”)
    • In between, scale with distance of midpoint (down or up):

2 ( |q(DB) – q(At)| - (¸ + ¹/2) ) / ¹ (sensitivity: 2½/¹)

 + 

Error increasing

(log |Q |3/2½ √k) / ²´4

private boosting for queries variant of adaboost1
Private Boosting for Queries [Variant of AdaBoost]
  • Initial distribution D is uniform on queries in Q
  • S is always a set of k elements drawn from Qk
  • Combiner is median [viz. Freund92]
  • Attenuated Re-Weighting
    • If very well approximated by At, decrease weight by factor of e (“-1”)
    • If very poorly approximated by At, increase weight by factor of e (“+1”)
    • In between, scale with distance of midpoint (down or up):

2 ( |q(DB) – q(At)| - (¸ + ¹/2) ) / ¹ (sensitivity: 2½/¹)

Reweighting similar under x and x’

Need lots of samples to detect difference

Adversary never gets hands on lots of samples

 + 

Error increasing

(log |Q |3/2½ √k) / ²´4

privacy2
Privacy???

Pr of q

λ

by x

by x’

Queries q ∈ Q

agnostic as to type signature of q
Agnostic as to Type Signature of Q
  • Base Generator for Counting Queries[D.-Naor-Reingold-Rothblum-Vadhan’09]
    • Use noise for all queries
      • An -dp process for collecting a set S of responses
    • Fit an n log|U|/log|Q| bit database to the set S; poly(|U|)
    • -dp
  • Using a Base Generator for Arbitrary Real-Valued Queries
    • Use noise for all queries
      • An -dp process for collecting a set S of responses
    • Fit an n-element database; exponential in |U|
    • -dp
analyzing privacy loss
Analyzing Privacy Loss
  • Know that “the epsilons and deltas add up”
    • T invocations of Base Generator (base , ±base)
    • Tk samples from distributions (sample, ±sample)
      • k each from D1, … , DT
      • Fair because samples in iteration t are mutually independent and distribution being sampled depends only on A1, …, A t-1 (public)
  • Improve on (T base + Tksample , T±base+Tk±sample)-dpvia composition theorem
slide60

Caveat: Omitting polylog(various things, some of them big) terms

Error

[Hardt-Rothblum]

Runtime Exp(|U|)

non trivial accuracy with dp
Non-Trivial Accuracy with -DP

Stateless Mechanism

Statefull Mechanism

Barrier at

[D.,Naor,Vadhan]

non trivial accuracy with dp1
Non-Trivial Accuracy with -DP

Independent Mechanisms

Statefull Mechanism

Barrier at

[D.,Naor,Vadhan]

non trivial accuracy with dp2
Non-Trivial Accuracy with -DP

Independent Mechanisms

Statefull Mechanism

Moral: To handle many databases must relax adversary assumptions or introduce coordination

outline2
Outline
  • Part 1: Basics
    • Smoking causes cancer
    • Definition
    • Laplace mechanism
    • Simple composition
    • Histogram example
    • Advanced composition
  • Part 2: Many Queries
    • Sparse Vector
    • Multiplicative Weights
    • Boosting for queries
  • Part 3: Techniques
    • Exponential mechanism and application
    • Subsample-and-Aggregate
    • Propose-Test-Release
    • Application of S&A and PTR combined
  • Future Directions
discrete valued functions
Discrete-Valued Functions
    • Strings, experts, small databases, …
    • Each has a utility for , denoted
  • Exponential Mechanism [McSherry-Talwar’07]

Output with probability

exponential mechanism applied
Exponential Mechanism Applied

Many (fractional) counting queries[Blum, Ligett, Roth’08]:

Given -row database , set of properties, produce a synthetic database giving good approx to “What fraction of rows of satisfy property ?” .

  • is set of all databases of size
  • |

-62/4589

-1/3

-7/286

-1/100000

-1/310

slide67

What happened in 2009?

Error

[Hardt-Rothblum]

Runtime Exp(|U|)

high unknown sensitivity functions
High/Unknown Sensitivity Functions
  • Subsample-and-Aggregate [Nissim, Raskhodnikova, Smith’07]
functions expected to behave well
Functions “Expected” to Behave Well
  • Propose-Test-Release [D.-Lei’09]
    • Privacy-preserving test for “goodness” of data set
      • Eg, low local sensitivity [Nissim-Raskhodnikova-Smith07]

Big gap

……

Robust statistics theory:

Lack of density at median is the only thing that can go wrong

PTR: Dptest for low sensitivity median (equivalently, for high density)

if good, then release median with low noise

else output (or use a sophisticated dp median algorithm)

application feature selection
Application: Feature Selection

If “far” from collection with no large majority value, then output most common value.

Else quit.

future directions
Future Directions
  • Realistic Adversaries(?)
    • Related: better understanding of the guarantee
      • What does it mean to fail to have -dp?
      • Large values of can make sense!
  • Coordination among curators?
  • Efficiency
    • Time complexity: Connection to Tracing Traitors
    • Sample complexity / database size
  • Is there an alternative to dp?
    • Axiomatic approach [Kifer-Lin’10]
  • Focus on a specific application
    • Collaborative effort with domain experts
thank you
Thank You!

0

-4R

-3R

-2R

-R

R

2R

3R

4R

5R