Loading in 2 Seconds...

Informational Complexity Notion of Reduction for Concept Classes

Loading in 2 Seconds...

- 119 Views
- Uploaded on

Download Presentation
## PowerPoint Slideshow about ' Informational Complexity Notion of Reduction for Concept Classes ' - dinesh

**An Image/Link below is provided (as is) to download presentation**

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

### Informational Complexity Notion of Reduction for Concept Classes

### Measures of the Informational Complexity of a class

### Outline of the talk

### Defining Reductions

### Relationship to Info Complexity

### Immediate observations

### Universal Classes

### Some important classes

### Half Spaces and Completeness

### Dudley Classes (1)

### Dudley Classes (2)

### Dudley’s Theorem

### A Completeness Theorem

### Corollaries

Shai Ben-David

Cornell University, and Technion

Joint work with

Ami Litman

Technion

The VC-dimension of the class.

The sample complexity for learning the class from random examples.

The optimal mistake bound for learning the class online (or the query complexity of learning this class using membership and equivalence queries).

The size of the minimal compression scheme for the class.

Defining our reductions, and the induced notion of

complete concept classes.

Introducing a specific family of classes that contains many

natural concept classes.

Prove that the class of half-spaces is complete w.r.t. that family.

Demonstrate some non-reducability results.

Corollaries concerning the existence of compression schemes.

We consider pair of sets (X,Y) where X is a domain and Y is a

set of concepts. A concept class is arelations R overXxY

(so eachyeY can be viewed as the subset{x: (x,y)eR}ofX ).

An embedding of C=(X,Y,R) into C’=(X’,Y’,R’) is a pair of

functions

r:X X’, t:Y Y’, so that (x,y)eR iff(r (x), t (y))eR’ .

Creduces toC’, denoted C C,’ if such an embedding exits.

If C C’then, for each of the complexity parameters mentioned

above, C’ is at least as complex asC.

E.g., if C C,’ then, for every e and d, the sample complexity

of (e, d) learning C is at most that needed for learning C’.

(This is in the agnostic prediction model)

If we take into account the computational complexity

of the embedding functions, then we can also bound

the computational complexity of learning C by that of

learning C’

For every k, the class of all binary functions on a k-size

domain is minimal w.r.t. the family of all classes having

VC-dimensionk.

We say that a concept class C is universal for a family

of classesFif every member of F reduces to C .

Universal classes play a role analogous to that of, say,

NP-hard decision problems – they are as complex as any

member of the family F

For an integer k, let HSk denote the class of half

spaces over Rk. That is HSk=(Rk, Rk+1, H) where

((x1,….xk),(a1,…ak+1))eH iff Saixi +ak+10

Let PHSk denote the class of positive half spaces,

that is, half spaces in which a1=1.

Finally, let HSk0denote the class of homogenous half

spaces (I.e., those having ak+1=0), and PHSk0

the class of poditive and homogenous half spaces.

The first family of classes that comes to mind is

the family VCn- the family of all concept classes

having VC-dimensions n.

Theorem: For anyn>2, no classHSk is universal forVCn

(This holds even if we consider only finite classes)

Next, we define a rich subfamily of VCn

for which classes of half spaces are universal.

LetF be a family of real valued functions over some

domain set X. For any function g , let h be any real

valued functionover X and definea concept class

DF,h = (X, F, RF,h ) where RF,h = {(x,f) : f(x)+h(x)0}.

(Note that all the PPD’s defined by Adam yesterday were of

this form)

Classes of the form DF,h = (X, F, RF,h ) are called

Dudley Classesif the family of functions F is a vector

space over the reals (with respect to point-wise addition

and scalar multiplication).

Examples of Dudley classes:

HSk , PHSk ,HSk0 ,PHSk0 , and the class of all balls

in any Euclidean space Rk

Theorem: If the a family of functionsF is a vector space,

then, for everyh, the VC dimension ofDF,h

equals the (linear) dimension of the vector spaceF .

Corollary: Easy calculations for the VC dimension of

the classes HSk , PHSk ,HSk0 ,PHSk0 , k-dimensional balls.

Theorem: For every k, PHSk+10isuniversal,

(and therefore, complete) for the family of all

k -dimensional Dudley classes.

Proof:

Let f1 , …fk be a basis for the vector space F, define

r:X Rk+1, t:F Rk+1, be

r(x) = (f1(x), …. fk(x), h(x)) and for f= Saifi

t(f)=(a1, …ak, 1, 0)

k-size compression schemes for any k-dimensional

Dudley class.

Learning algorithms for all Dudley classes.

An easy proof to Dudley’s theorem.

(show that for anyk–dimensionalF, the classHSk0

is embeddable into DF,h ,for h=0)

Download Presentation

Connecting to Server..