1 / 59

On Natural Scenes Analysis, Sparsity and Coding Efficiency

On Natural Scenes Analysis, Sparsity and Coding Efficiency. Vivienne Ming. Mind, Brain & Computation Stanford University. Redwood Center for Theoretical Neuroscience University of California, Berkeley. Adapted by J. McClelland for PDP class, March 1, 2013. Two Proposals.

ronda
Download Presentation

On Natural Scenes Analysis, Sparsity and Coding Efficiency

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. On Natural Scenes Analysis, Sparsity and Coding Efficiency Vivienne Ming Mind, Brain & Computation Stanford University Redwood Center for Theoretical Neuroscience University of California, Berkeley Adapted by J. McClelland for PDP class, March 1, 2013

  2. Two Proposals • Natural Scene Analysis • Neural/cognitive computation can only be fully understood in “naturalistic” contexts • Efficient (Sparse) Coding Theory • Neural computation should follow information theoretic principles Natural Scenes Analysis

  3. Classical Physiology Natural Scenes Analysis

  4. Classical Physiology + Natural Scenes Analysis

  5. Classical Physiology + Natural Scenes Analysis

  6. Reverse Correlation Jones and Palmer (1987) Natural Scenes Analysis

  7. Limits of Classical Physiology • Assumes units (neurons) are linear • so known nonlinearities are "added on" to the models • Contrast sensitivity • “Non-classical receptive fields” • Two-tone inhibition • ETC. • Assumes that units operate independently • activity of one cell doesn't depend on the activity of others • i.e., characterizing cell-by-cell equivalent to characterizing the whole population • of evolution and development, drifting gratings and white noise are very "unnatural“ • Is it possible that our sensory systems are functionally adapted to the statistics of “natural” (evolutionarily relevant) signals? • Would this adaptation affect our characterization of cells? Natural Scenes Analysis

  8. Response to Natural Movie Classical Receptive Field Response Response in “Context” Natural Scenes Analysis

  9. Limits of Classical Physiology • Assumes units (neurons) are linear • so known nonlinearities are "added on" to the models • Contrast sensitivity • “Non-classical receptive fields” • Two-tone inhibition • ETC. • Assumes that units operate independently • activity of one cell doesn't depend on the activity of others • i.e., characterizing cell-by-cell equivalent to characterizing the whole population • Finally, in terms of evolution and development, drifting gratings and white noise seem very "unnatural“ • Is it possible that our sensory systems are functionally adapted to the statistics of “natural” (evolutionarily relevant) signals? • Would this adaptation affect our characterization of cells? • How can we test this? Natural Scenes Analysis

  10. Efficient Coding Theory Barlow (1961); Attneave (1954) • Natural images are redundant • Statistical dependencies amongst pixel values in space and time • An efficient visual system should reduce redundancy • Removing statistical dependencies Natural Scenes Analysis

  11. Information TheoryShannon (1949) Optimally efficient codes reflect the statistics of target signals Natural Scenes Analysis

  12. Natural Scenes Analysis:First-Order Statistics Naïve Models Natural Scenes Analysis

  13. Natural Scenes Analysis:First-Order Statistics Intensity Histogram Histogram Equalization Natural Scenes Analysis

  14. Natural Scenes Analysis:Second-Order Statistics Natural Scenes Analysis

  15. Natural Scenes Analysis:Second-Order Statistics Natural Scenes Analysis

  16. Natural Scenes Analysis:Second-Order Statistics Natural Scenes Analysis

  17. Spatial Correlations Compare intensity at this pixel To the intensity at this neighbor Natural Scenes Analysis

  18. Spatial Correlations Natural Scenes Analysis

  19. The Ubiquitous . Flat (White) Power Spectrum Natural Scenes Analysis

  20. Example: synthetic 1/f signals Natural Scenes Analysis

  21. Natural Scenes Analysis:Principal Components Analysis PCA Rotation Whitening Information theory says this is an ideal code. No redundancy Natural Scenes Analysis

  22. PCA vs. Center Surround Natural Scenes Analysis

  23. Natural Scenes Analysis:Higher-Order Statistics PCA Rotation Whitening Principle dimensions of variation don’t align with data’sintrinsic structure Natural Scenes Analysis

  24. Natural Scenes Analysis:Higher-Order Statistics Need a more powerful learning algorithm Independent Component Analysis (ICA) Natural Scenes Analysis

  25. Which are the independent components in the scene below? Natural Scenes Analysis

  26. = + +_______ Natural Scenes Analysis

  27. The Model Information Theory demands sparseness x= s+ n • Overcomplete: #(s) >> #(x) • Factorial: p(s) = i p(si) • Sparse: p(si) = exp(g(si)) • Where g(.) is some non-Gaussian distribution • e.g., Laplacian: g(s) = −|s| • e.g., Cauchy: g(s) = −log(2 + s2) • The noise is assumed to be additive Gaussian • n ~ N(0, 2I) • Goal: find dictionary of functions, ,such that coefficients, s,are as sparse and statistically independent as possible Natural Scenes Analysis

  28. Learning • log likelihood L() = <log p(x|)> • Learning rule: • Basically the delta rule: D= (x− s)sT • Impose constraint to encourage the variances of each sto be approximately equal to prevent trivial solutions • Usually whiten the inputs before learning • Forces network to find structure beyond second-order • Increases stability Natural Scenes Analysis

  29. Sparsity Natural Scenes Analysis

  30. ? Natural Scenes Analysis

  31. Efficient Auditory CodingSmith & Lewicki (2006) • Extend Olshausen (2002) to deal with time-varying signals • e.g., sounds or movies • Train the network on “Natural” sounds • Environmental Transients • Environmental Ambients • Animal Vocalizations Natural Scenes Analysis

  32. Natural Scenes Analysis

  33. Cat ANF Revcor Filters Natural Scenes Analysis

  34. Efficient Kernels Natural Scenes Analysis

  35. Population Coding Natural Scenes Analysis

  36. Population Coding Natural Scenes Analysis

  37. Population Coding Natural Scenes Analysis

  38. Population Coding Natural Scenes Analysis

  39. Population Coding Natural Scenes Analysis

  40. Speech Natural Scenes Analysis

  41. Speech Natural Scenes Analysis

  42. Speech Natural Scenes Analysis

  43. Empirical Weliky, Fiser, Hunt & Wagner (2003) Vinje & Gallant (2002) DeWeese, Wehr & Zador (2003) Laurent (2002) Theunissen (2003) Theoretical Field (1987) van Hateren (1992) Simoncelli & Olshausen (2001) Olshausen & Field (1996) Bell & Sejnowski (1997) Hyvarinen & Hoyer (2000) Smith & Lewicki (2006) Doi & Lewicki (2006) Efficient Coding Literature Natural Scenes Analysis

  44. Hierarchical Structure? • Can we identify interesting structure in the world by looking at higher order statistics of the activations of the linear features discovered by the first-order model? • Karklin and Lewicki (2005) looked for patterns at the level of the variances of the linear features. • Karklin and Lewicki (2009) looked for patterns at the level of the covariances of the linear features. Natural Scenes Analysis

  45. Looking at Hierarchical Structure Natural Scenes Analysis

  46. Looking at Hierarchical Structure Natural Scenes Analysis

  47. Looking at Hierarchical Structure Natural Scenes Analysis

  48. Looking at Hierarchical Structure Natural Scenes Analysis

  49. Looking at Hierarchical Structure Natural Scenes Analysis

  50. Generalizing the standard ICA model Natural Scenes Analysis

More Related