multispectral imaging and unmixing n.
Skip this Video
Loading SlideShow in 5 Seconds..
Multispectral Imaging and Unmixing PowerPoint Presentation
Download Presentation
Multispectral Imaging and Unmixing

Loading in 2 Seconds...

play fullscreen
1 / 58

Multispectral Imaging and Unmixing - PowerPoint PPT Presentation

  • Uploaded on

Multispectral Imaging and Unmixing. Jürgen Glatz Chair for Biological Imaging Munich, 06/06/12. Intraoperative Fluorescence Imaging. Fluorescence Channel. Color Channel . Outline. Multispectral Imaging Unmixing Methods Exercise: Implementation.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Multispectral Imaging and Unmixing' - alain

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
multispectral imaging and unmixing

Multispectral Imaging and Unmixing

Jürgen Glatz

Chair for Biological Imaging

Munich, 06/06/12

intraoperative fluorescence imaging
Intraoperative Fluorescence Imaging





  • Multispectral Imaging
  • Unmixing Methods
  • Exercise: Implementation
multispectral imaging
Multispectral Imaging
  • Multispectral Imaging
  • Unmixing Methods
  • Exercise: Implementation
multispectral imaging1
Multispectral Imaging


Spectral Resolution

Sensitivity Range

Spatial Resolution





Spectral Resolution has practically not improved since first camera

Spatial Resolution and Magnificationare significantly improved

color vision
Color Vision

Anyone feeling hungry?

Monochrome image of an apple tree

Color image of an apple tree

  • Color vision helps to distinguish and identify objects against their background (here: fruit and foliage)
  • Color vision provides contrast based on optical properties
color vision1
Color Vision

Spectral sensitivity of the human eye

low light






  • Color receptors (cone cells) with different spectral sensitivity enable trichromatic vision
  • Limited spectral range and poor resolution










limited spectral range
Limited spectral range



Evening primrose

Cleopatra butterfly

  • Human eyes can only see a portion of the light spectrum (ca. 400-750nm)
  • Certain patterns are invisible to the eye
limited spectral resolution


Same colorappearance

Limited spectral resolution



  • Color vision is insufficient to distinguish between two green objects
  • Differences in the spectra reveal different chemical composition







optical spectroscopy
Optical Spectroscopy
  • Absorbance
  • Fluorescence
  • Transmittance
  • Emission
  • Spectroscopy analyzes the interaction between optical radiation and a sample (as a function of λ)
  • Provides compositional and structural information
directions of optical methods
Directions of optical Methods



Currently there are two “directions” in optical analysis of an object





Provides spatial information

Provides spectral information

Reveals morphological features

No information about structure or composition / no spectral analysis

Spectrum reveals composition and structure

No information about spatial distribution

imaging spectroscopy
Imaging Spectroscopy



Imaging Spectroscopy

Spatial dimension y

Spectral dimension λ

Spatial information

Spectral information

Spatial dimension x

Spatial dimension y

Spectral dimension λ

Spatial dimension x

Spectral Cube

Spatial and spectral information

spectral cube
Spectral Cube

Chemicalcompound A

Chemicalcompound B


Pseudo-color image representing the distribution of compounds A and B (chlorophyll and plastic)










  • Acquisition of spatially coregistered images at different wavelengths
  • The maximum number of components that can be distinguished equals the number of spectral bands
  • The accuracy of spectral unmixing increases with the number of bands
multispectral imaging modalities
Multispectral Imaging Modalities
  • Camera + Filter Wheel
  • Bayer Pattern
  • Cameras + Prism
  • Multispectral Optoacoustic Tomography
  • etc.
let s find those apples
Let’s find those apples
  • Multispectral imaging alone is only one side of the medal
  • Appropriate data analysis techniques are required to extract information from the measurements
unmixing methods
Unmixing Methods
  • Multispectral Imaging
  • Unmixing Methods
  • Exercise: Implementation
the unmixing problem
The Unmixing Problem


  • Finding the sources that constitute the measurements
  • For multispectral imaging this means separating image components of different, overlapping spectra
  • Unmixing is a general problem in (multivariate) data analysis
multifluorescence microscopy
Multifluorescence Microscopy
  • Disjoint spectra can be separated by bandpass filtering
  • Overlapping emission spectra create crosstalk



  • Autofluorescence exhibits a broadband spectrum
  • Only mixed observations of the components can be measured
  • Post-processing to unmix them
forward modeling
Forward Modeling

What constitutes a multispectral measurement at a certain point and wavelength?

Principle of superposition: Sum of individual component emission

A component‘s emission over different wavelengths λ is denoted by its spectrum, its spatial distribution is still to be defined.

setting up a simple forward problem 1
Setting up a simple forward problem (1)
  • Two fluorochromes on a homogeneous background
  • Note: We define images as row vectors of length n
  • All components are merged in the (n x k) source matrix O

n: Number of image pixelsk: Number of spectral components

setting up a simple forward problem 2
Setting up a simple forward problem (2)

Relative Absorption [%]

  • Defining the emission spectra for all components at the measurement points
  • Combining them into the (k x m) spectral matrix

Wavelength [nm]

k: Number of spectral components

m: Number of multispectral measurements

m ≥ k

setting up a simple forward problem 3
Setting up a simple forward problem (3)

Relative Absorption [%]

  • Two fluorochromes on a homogeneous background
  • Heavily overlapping spectra
  • 25 equidistant measurements under ideal conditions

Wavelength [nm]

mathematical formulation
Mathematical Formulation


Multispectral measurement matrix

(n x m)

Original component matrix

(n x k)

Spectral mixing matrix

(k x m)

Noise, artefacts, etc.

(n x m)

mathematical formulation1
Mathematical Formulation





linear regression spectral fitting
Linear Regression: Spectral Fitting
  •  Reconstructing O
  • System generally overdetermined: No direct inverse S-1
  • Generalized inverse: Moore-Penrose PseudoinvereS+
  • Spectral Fitting: Finding the components that best explain the measurements given the spectra
  • Minimizing the error:
spectral fitting
Spectral Fitting
  • Orthogonality principle: optimal estimation (in a least squares sense) is orthogonal to observation space
spectral fitting2
Spectral Fitting

Spectral Fitting

  • Given full spectral information (i.e. about all source components) the data can be unmixed
multifluorescence imaging
Multifluorescence Imaging

RGB image





Nude mice with two different species of autofluorescence and three subcutaneous fluorophore signals: FITC, TRITC and Cy3.5.

(Totally 5 signals)



spectral fitting3
Spectral Fitting

Fast, easy and computationally stable

Known order and number of unmixed components


Requires complete spectral information

Crucially depends on accuracy of spectra (systematic errors)

Suitable for detection and localization of known compositions

principal component analysis
Principal Component Analysis
  • Blind source separation (BSS) technique
  • Requires no a priori spectral information
  • Estimates both O and S from M
  • Assumption:Sources are uncorrelated, while mixed measurements are not
principal component analysis1
Principal Component Analysis
  • Unmixing by decorrelation: Orthogonal linear transformation
  • Transforms the data into a space spanned by the orthogonal PCs
  • Maximum variance along first PC, maximum remaining variance along second PC, etc.
unmixing multispectral data with pca
Unmixing multispectral data with PCA
  • 25 multispectral measurements are correlated
  • Their entire variance can (ideally) be expressed by only 3 PCs  Dimension reduction
  • Those 3 PCs are the unmixed sources
  • Note that matrix orientations may vary between different implementations
computing pca
Computing PCA

Method 1 (preferred for computational reasons)

  • Subtract mean from multispectral observations
  • Covariance Matrix:
  • DiagonalizingCM: Eigenvalue Decomposition
  • Eigenvectors of CM are the principal components, roots of the eigenvalues are the singular values
  • Projecting Monto the PCs:
computing pca with the svd
Computing PCA with the SVD

Method 2 (not suitable for implementation)

  • Subtract mean from multispectral observations
  • Singular Value Decomposition: M = UΣVT
  • Uis a (m x m) matrix of orthonormal (uncorrelated!) vectors
  • Projecting Monto those decorrelates the measurements
  • Singular values in Σdenote how much variance is explained by the respective PC
pca does more than just unmix
PCA does more than just unmix



(UT)-1 = U ≈ S


Multispectral data space

Original data space

  • Uis a (non-quantitative) approximation of the PCs spectra
  • These can be used to verify a components identity
  • Σis the singular value matrixRelatively small singular values indicate irrelevant components


principal component analysis pca
Principal Component Analysis (PCA)

Needs no a priori spectral information

Also reconstructs spectral properties

Significance measurement through singular values

Unknown order and number of components

Generally not quantitative

Crucially depends on uncorrelatedness of the sources

Suitable for many compounds and identification of unknown components

advanced blind source separation
Advanced Blind Source Separation
  • Independent Component Analysis (ICA): assumes statistically independent source components, which is a stronger condition than PCA’s orthogonality
  • Non-negative Matrix Factorization (NNMF): constraint that all elements must be positive
  • Commonly computed by iterative optimization of cost functions, gradient descent, etc.
independent component analysis
Independent Component Analysis
  • Assumes and requires independent sources:
  • Independence is stronger than uncorrelatedness
independent component analysis1
Independent Component Analysis
  • Central limit theorem: Sum of non-gaussian variables is more gaussian than the individual variables
  • Kurtosis measures non-gaussianity:
  • Maximize kurtosis to find IC
  • Reconstruction:
practical considerations
Practical Considerations
  • Noise
  • Artifacts (from reconstruction, reflections, measurement,…)
  • Systematic errors (spectra, laser tuning, illumination,…)
  • Unknown and unwanted components
exercise implementation
Exercise: Implementation
  • Multispectral Imaging
  • Unmixing Methods
  • Exercise: Implementation
forward problem mixing
Forward Problem / Mixing
  • Define at least 3 non-constant images representing the original components
  • Plot them and store them in the matrix O
  • Define an emission spectrum for every component at an appropriate number of measurment points
  • Plot them and store them in the matrix S
  • Calculate the measurement matrix as M = OS (and save everything)
forward problem mixing1
Forward Problem / Mixing

Relative Absorption [%]

Wavelength [nm]



forward problem mixing2
Forward Problem / Mixing

Useful MatLab functions

  • Change matrices into vectors: y=reshape(X,…) or y=X(:)
  • Plot image from a matrix: imagesc(X) or imshow(X)
spectral fitting4
Spectral Fitting
  • Create an m-file and write a function that
  • Has M and S as input variables
  • Calculates the pseudoinverse S+
  • Returns the unmixing Rpinv
  • Test it on your data
spectral fitting5
Spectral Fitting

Useful MatLab functions

  • Functions: function [out] = name([input])
  • Regular matrix inverse: y = inv(x)
principal component analysis2
Principal Component Analysis
  • Create an m-file and write a function that
  • Has M as an input variable
  • Subtracts the mean from the measurements in M
  • Computes the covariance matrix CM
  • Performs an eigenvalue decomposition on CM
  • Sorts the eigenvalues (and corresponding vectors) by size
  • Projects M onto the eigenvectors
  • Returns the projected unmixing, the principal components and their loadings
principal component analysis3
Principal Component Analysis

Useful MatLab functions

  • Mean: y = mean(x)
  • Eigenvalue Decomposition: [e_vec e_val] = eig(X)
testing your code
Testing your code
  • Try fitting and PCA on your mixed data
  • Try adding different types and amounts of noise to M(e.g. using imnoise)
  • Simulate systematic errors in your spectra (noise, changing values, offset,…)
independent component analysis voluntary
Independent Component Analysis (voluntary)
  • You can download the FastICAMatLabcode from
  • Type doc fastica for function description
  • Use the fasticafunction to unmix your simulated data
  • Compare the result to PCA. What are advantages and disadvantages of ICA?
recommended reading
Recommended Reading
  • Shlens, J. – A Tutorial on Principal Component Analysis
  • Garini, Y., Young, I.T. and McNamara, G. – Spectral Imaging: Principles and Applications; Cytometry Part A 69A: p.735-747 (2006)
  • Stone, J.V. – A brief Introduction to ICA; Encyclopedia of Statistics in Behavioral Science, Vol. 2, p. 907-912