Unsupervised Learning by Convex and Conic Coding

1. Unsupervised Learning by Convex and Conic Coding D. D. Lee and H. S. Seung NIPS’1997

2. Introduction • Learning algorithms based on convex and conic encoders are introduced. • Less constrained than VQ but more constrained than PCA. • VQ • Encode each input as the index of the closest prototype. • Capture nonlinear structure • Highly localized • PCA • Encode as the coefficient of a linear superposition of a set of basis vectors. • Distributed representation • Can only model linear structures.

3. Can produce sparse distributed representation. • Learning algorithms can be understood as approximate matrix factorization.

4. Affine, Convex, Conic, and Point Coding • Definition • Given a set of basis vectors , the linear combination is called • Affine hull • Convex hull • Conic hull

5. Goal of encoding • Find the nearest point to the input in the respective hulls. • Minimize the reconstruction error • Encoding of convex, conic encoders • Sparse encoding • Contain coefficients that vanish, due to the nonnegativity constraints in the optimization.

6. Learning • Objective Function • X : nm matrix of training set • n : dimension, m: number of data • W : nr matrix (basis vectors) • V : rm matrix (encodings) • Description • Approximate factorization of the data matrix X into a matrix W of basis vectors and a matrix V of code vectors.

7. Constraints • If the input vectors in X have been scaled to the range [0, 1], the constraints on the optimizations are given by • The nonnegativity constraints prevent cancellations from occurring in the linear combinations.

8. Example: modeling handwritten digits • Experimental Setup • Affine, Convex, Conic, and VQ learning to the USPS database. • Handwritten digits segmented from actual zip codes. • 7291 training and 2007 test images were normalized to a 16 16 grid with pixel intensities in the range [0, 1]. • Training examples were segregated by digit class. • Separate basis vectors were trained for each of the classes using four encodings.

10. VQ • k-means algorithm was used • Restarted with various initial conditions and the best solution was chosen. • Affine • Determine the affine space that best models the input data. • No obvious interpretation. • Convex • Finds the r basis vectors whose convex hull best fits the input data. • Alternate between projected gradient steps of W and V. • The basis vectors are interpretable as templates and are less blurred than those found in VQ. • Eliminate many invariant transformations, because they would violate the nonnegativity constraints.

11. Conic • Finds basis vectors whose conic hull best models the input images. • Representation allows combinations of basis vectors. • The basis vectors found are features rather than templates.

12. Classification • Separately reconstructing the test images with different digit model. • Associate the image with the model having the smallest reconstruction error.

13. Results • With r=25 patterns per digit class • Convex : error rate = 113/2007 = 5.6% • With r=100 patterns, 89/2007 = 4.4% • Conic: 138/2007 = 6.8% • With r > 50, worse performance as the feature shrink to small spots. • Non-trivial correlations still remain in the and also need to be taken into account.

14. Discussion • Convex coding is similar to other locally linear models. • Conic coding is similar to the noisy OR and harmonium models • Conic uses continuous variables rather than binary variables. • Makes the encoding computationally tractable and allows for interpolation between basis vectors. • Convex and Conic coding is can be viewed as probabilistic latent variable models. • No explicit model P(va) for the hidden variables was used. • Limit the quality of the Conic models • Building hierarchical representations is needed.