Informational complexity notion of reduction for concept classes
1 / 14

Informational Complexity Notion of Reduction for Concept Classes - PowerPoint PPT Presentation

  • Uploaded on

Informational Complexity Notion of Reduction for Concept Classes. Shai Ben-David Cornell University, and Technion Joint work with Ami Litman Technion. Measures of the Informational Complexity of a class. The VC-dimension of the class.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about ' Informational Complexity Notion of Reduction for Concept Classes ' - dinesh

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Informational complexity notion of reduction for concept classes

Informational Complexity Notion of Reduction for Concept Classes

Shai Ben-David

Cornell University, and Technion

Joint work with

Ami Litman


Measures of the informational complexity of a class

Measures of the Informational Complexity of a class

The VC-dimension of the class.

The sample complexity for learning the class from random examples.

The optimal mistake bound for learning the class online (or the query complexity of learning this class using membership and equivalence queries).

The size of the minimal compression scheme for the class.

Outline of the talk

Outline of the talk

Defining our reductions, and the induced notion of

complete concept classes.

Introducing a specific family of classes that contains many

natural concept classes.

Prove that the class of half-spaces is complete w.r.t. that family.

Demonstrate some non-reducability results.

Corollaries concerning the existence of compression schemes.

Defining reductions

Defining Reductions

We consider pair of sets (X,Y) where X is a domain and Y is a

set of concepts. A concept class is arelations R overXxY

(so eachyeY can be viewed as the subset{x: (x,y)eR}ofX ).

An embedding of C=(X,Y,R) into C’=(X’,Y’,R’) is a pair of


r:X X’, t:Y Y’, so that (x,y)eR iff(r (x), t (y))eR’ .

Creduces toC’, denoted C C,’ if such an embedding exits.

Relationship to info complexity

Relationship to Info Complexity

If C C’then, for each of the complexity parameters mentioned

above, C’ is at least as complex asC.

E.g., if C C,’ then, for every e and d, the sample complexity

of (e, d) learning C is at most that needed for learning C’.

(This is in the agnostic prediction model)

Immediate observations

Immediate observations

If we take into account the computational complexity

of the embedding functions, then we can also bound

the computational complexity of learning C by that of

learning C’

For every k, the class of all binary functions on a k-size

domain is minimal w.r.t. the family of all classes having


Universal classes

Universal Classes

We say that a concept class C is universal for a family

of classesFif every member of F reduces to C .

Universal classes play a role analogous to that of, say,

NP-hard decision problems – they are as complex as any

member of the family F

Some important classes

Some important classes

For an integer k, let HSk denote the class of half

spaces over Rk. That is HSk=(Rk, Rk+1, H) where

((x1,….xk),(a1,…ak+1))eH iff Saixi +ak+10

Let PHSk denote the class of positive half spaces,

that is, half spaces in which a1=1.

Finally, let HSk0denote the class of homogenous half

spaces (I.e., those having ak+1=0), and PHSk0

the class of poditive and homogenous half spaces.

Half spaces and completeness

Half Spaces and Completeness

The first family of classes that comes to mind is

the family VCn- the family of all concept classes

having VC-dimensions n.

Theorem: For anyn>2, no classHSk is universal forVCn

(This holds even if we consider only finite classes)

Dudley classes 1

Dudley Classes (1)

Next, we define a rich subfamily of VCn

for which classes of half spaces are universal.

LetF be a family of real valued functions over some

domain set X. For any function g , let h be any real

valued functionover X and definea concept class

DF,h = (X, F, RF,h ) where RF,h = {(x,f) : f(x)+h(x)0}.

(Note that all the PPD’s defined by Adam yesterday were of

this form)

Dudley classes 2

Dudley Classes (2)

Classes of the form DF,h = (X, F, RF,h ) are called

Dudley Classesif the family of functions F is a vector

space over the reals (with respect to point-wise addition

and scalar multiplication).

Examples of Dudley classes:

HSk , PHSk ,HSk0 ,PHSk0 , and the class of all balls

in any Euclidean space Rk

Dudley s theorem

Dudley’s Theorem

Theorem: If the a family of functionsF is a vector space,

then, for everyh, the VC dimension ofDF,h

equals the (linear) dimension of the vector spaceF .

Corollary: Easy calculations for the VC dimension of

the classes HSk , PHSk ,HSk0 ,PHSk0 , k-dimensional balls.

A completeness theorem

A Completeness Theorem

Theorem: For every k, PHSk+10isuniversal,

(and therefore, complete) for the family of all

k -dimensional Dudley classes.


Let f1 , …fk be a basis for the vector space F, define

r:X Rk+1, t:F Rk+1, be

r(x) = (f1(x), …. fk(x), h(x)) and for f= Saifi

t(f)=(a1, …ak, 1, 0)



k-size compression schemes for any k-dimensional

Dudley class.

Learning algorithms for all Dudley classes.

An easy proof to Dudley’s theorem.

(show that for anyk–dimensionalF, the classHSk0

is embeddable into DF,h ,for h=0)