1 / 18

Visual decoding… Can you see what I see?

Visual decoding… Can you see what I see?. Sophia Yang 4-22-2010. Visual pathway . retina o ptic chiasm t halamus, lateral geniculate nucleus (LGN) o ccipital cortex, primary visual cortex (V1) v entral and dorsal streams, downstream processing

kevyn
Download Presentation

Visual decoding… Can you see what I see?

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Visual decoding…Can you see what I see? Sophia Yang 4-22-2010

  2. Visual pathway • retina • optic chiasm • thalamus, lateral geniculate nucleus (LGN) • occipital cortex, primary visual cortex (V1) • ventral and dorsal streams, downstream processing **retinotopic mapping in lower visual areas

  3. Retinotopy in V1 Visual fields mapped retinotopically to V1 • visual fields

  4. Retinotopic mapping in cat V1 Visual stimulus Corresponding V1 activity

  5. Decoding with fMRI • functional magnetic resonance imaging (fMRI) methods • analyze neural activity (BOLD signals) in response to stimuli • traditional univariate analyses • Single voxels… ROIs involvement • multivariate analyses, machine learning approaches • classification, reconstruction, decoding!

  6. Reconstructing images • Miyawaki et al. (2008)

  7. Overview of visual decoding projects • classify, identify, reconstruct images • establish a systematic mapping between visual stimuli and brain activity • activity to stimulus • evaluate mapping • stimulus to activity • inversion

  8. Goal: reconstruct visual contrast image (10x10 square grey-scale patches) Input - fMRI signals from all V1 and V2 voxels (algorithm later selects relevant voxels and weights) while viewing a contrast image Output - reconstructed image Approach: • Multivoxel patterns of fMRI signals in low visual areas • Multiscale visual representation

  9. Presentation of image stimuli Random image session: 6s random contrast image, followed by 6s rest Figure image session: 12s geometric or alphabet shape contrast image, followed by 12s rest

  10. The reconstruction algorithm r = given fMRI signals I(x|r) = the reconstruction image = local image basis x = spatial position in the image Cm(r) = predicted contrast of local image basis = combination coefficient of local image basis

  11. Visual image reconstruction

  12. Training local decoders • Local decoders defined to predict the mean contrast of each local image basis • Individually trained with fMRI data and the corresponding class labels representing the mean contrast values • Each local decoder consists of a multi-class classifier (classify fMRI data samples into the classes defined by the mean contrast values) • Predetermined relevant voxels selection and individual voxel weights (“sparse logistic regression,” Yamashita et al., 2008) • Lineardiscriminant function for contrast class k: • rd = fMRI signal of voxeld • wkd= weight parameter for voxeld and contrast class k • wk0 = bias • Probability that r belongs to contrast class k: • Softmax function • K = number of contrast classes • Predicted contrast class for mth local image basis Cm(r) chosen as contrast class with highest probability

  13. Reconstruction results

  14. Reconstruction results

  15. Image identification using reconstructed images

  16. reconstruction demonstration Video: http://www.popsci.com/science/article/2010-01/mind-readers http://www.youtube.com/watch?v=h1Gu1YSoDaY

  17. Applications of brain decoding? • Neural prosthetics, BMI, HCI • Control prosthetic device using neural activity in motor cortex • Reconstruct contents of visual imagery, dreams • Can you see what I see? • Reconstruction of the subjective contents of human perception • Ethical issues?

More Related