Learning with Local and Global Consistency

1. Learning with Local and Global Consistency By Dengyong Zhou, Olivier Bousquet, Tomas Navin Lal, Jason Weston and Bernhard Schölkopf at NIPS 2003 Presented by Qiuhua Liu Duke University Machine Learning Group March 23, 2007

2. Outline • The consistency assumption for semi-supervised learning: Why unlabeled data could help the classification? • The Consistency algorithm: a very simple algorithm based on the above assumption. • The Relation to Xuejun’s Label Iteration algorithm • Experimental results • Conclusions

3. Semi-supervised Learning Problem • We all know that Semi-supervised Learning is very important, but why could we do that? • The key to semi-supervised learning problems is the prior assumption of consistency: (1)Local Consistency: nearby points are likely to have the same label; (2)Global Consistency: Points on the same structure (cluster or manifold) are likely to have the same label;

4. Local and Global Consistency • The key to the consistency algorithm is to let every point iteratively spread its label information to its neighbors until a global stable state is achieved.

5. Some Terms • x, data point set: • L, Label set: • F, a classification on x: • Y, initial classification on x, which is a special case of F with:

6. The Consistency Algorithm 1.Construct the affinity matrix W defined by a Gaussian kernel: 2. Normalize W symmetrically by 3. Iterate until converge. 4. Let denote the limit of the sequence {F(t)}. The classification results is : where D is a diagonal matrix with

7. The Consistency Algorithm (Cont.) • The first two steps are the same as Spectral Clustering. • The third step: • First term: each point receive information from its neighbors. • Second term: retains the initial information. • From the iteration equation, it is very easy to show that: So we could compute F* directly without iterations.

8. The convergence process • The initial label information are diffused along the two moons.

9. Another view of the algorithm from regularization framework • It could be easily shown that iteration result F* is equivalent to minimize the following cost function: With • The first term is the Smoothing Constraint: nearby points are likely to have the same label; • The second term is the Fitting Constraint: the classification results does not change too much from the initial assignment.

10. Some variations of the consistency algorithm Let: Variation (1): Variation (2): Variation (3) is Xuejun’s label iteration algorithm, where Go is our graph:

11. Experiment Results (b)Text classification: topics including autos, motorcycles, baseball and hockey from the 20-newsgroups (a)Digit recognition: digit 1-4 from the USPS data set

12. Conclusions • The key to semi-supervised learning problem is the consistency assumption. • The consistency algorithm proposed was demonstrated effective on the data set considered.