sensor networks rate distortion codes and spin glasses n.
Skip this Video
Loading SlideShow in 5 Seconds..
Sensor Networks, Rate Distortion Codes, and Spin Glasses PowerPoint Presentation
Download Presentation
Sensor Networks, Rate Distortion Codes, and Spin Glasses

Loading in 2 Seconds...

play fullscreen
1 / 54

Sensor Networks, Rate Distortion Codes, and Spin Glasses - PowerPoint PPT Presentation

  • Uploaded on

Sensor Networks, Rate Distortion Codes, and Spin Glasses. NTT Communication Science Laboratories Tatsuto Murayama In collaboration with Peter Davis March 7 th , 2008 at the Chinese Academy of Sciences. Problem Statement. Sensor Networks. Sensor

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

Sensor Networks, Rate Distortion Codes, and Spin Glasses

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
sensor networks rate distortion codes and spin glasses

Sensor Networks, Rate Distortion Codes, and Spin Glasses

NTT Communication Science Laboratories

Tatsuto Murayama

In collaboration with Peter Davis

March 7th, 2008 at the Chinese Academy of Sciences

sensor networks
Sensor Networks


Sensors transmit their noisy observations independently.



Computer estimates the quantity of interest from sensor information.




Network has a limited bandwidth constraint.

a pessimistic forecast
A Pessimistic Forecast

《Supply Side Economics》Semiconductors are going to be very small and also cheap, so they’d like to sell them a lot!



Large-scale information integration


IC tags…

Central Unit

Network Capacity is limited

Target Source

Information loss via sensing

Information loss via communications


High Noise RegionNetwork is going to be large and dense!

Finite Network CapacityEfficient use of the given bandwidth is required!

Need a new information integration theory!

what to look for
What to look for?
  • Given a combined data rate, we examine the optimal aggregation level for sensor networks.

Saturate Strategy (SS)

Transmit as much sensor information as possiblewithout data compression.

Which strategy is outperforming?

A small quantity of high quality statistics

A large quantity of low quality statistics

Large System Strategy (LSS)

Transmit the overwhelming majority of compressed sensor information.

what to evaluate
What to Evaluate?
  • It is natural to introduce the following indicator function in decibel manner.
  • Which Strategy is Outperforming to the Other?
  • The large system strategy is outperforming when the indicator function is negative.
  • The saturate strategy is outperforming when the indicator function is positive.
  • The zero level corresponds to the strategic transition point if available.
what to expect
What to Expect?
  • Conjecture on the existence of the strategic transition point.

Strategic Transition Point.

  • Some Evidences
  • At the low noise level, the indicator function should diverge to infinity.
  • At the high noise level, the indicator function should converge to zero.
sensing model
Sensing Model
  • Target Information is a Bernoulli(1/2) Source.
  • Environmental Noise is modeled by the Binary Symmetric Channel.



  • Binary Symmetric Channel (BSC)
  • The input alphabet is `flipped’ with a given probability.
communication model
Communication Model
  • To satisfy the bandwidth constraints, each sensor encodes its observation independently.



  • Nature of Bandwidth-Given Communication
  • If the bandwidth is bigger than the entropy rate, revertible coding can be possible.
  • If the bandwidth is smaller than the entropy rate, only non-revertible coding can be possible.
estimation model
Estimation Model
  • Collective estimation is done by applying the majority vote algorithm to the reproductions.


In case of the `Ising’ alphabet

  • Majority Vote
  • Estimation is calculated from the reproductions by sequentially applying the following algorithm.
system model
System Model

Assume purely random Source is observed

Sensing Model

Encoding Model

Independent decoding process is forced

Bitwise majority vote is concerned

Estimation Model

case of saturate strategy
Case of Saturate Strategy




2 messages saturate network.


Cost of comm.= # of sensors ( bits of info.)

Moderate aggregation levels are possible.

case of large system strategy
Case of Large System Strategy

Still 2 messages saturate network.





Cost of comm.= # of sensors data rate

We can make system as large as we want!

rate distortion tradeoff
Rate Distortion Tradeoff
  • Variety of communication reduces to a simple rate distortion tradeoff.

Black Box

  • Rate Distortion Tradeoff
  • Each observation bit is flipped with the same probability.
effective distortion
Effective Distortion
  • Under the stochastic description of the tradeoff, we introduce the effective distortion as follows.
  • Then, our sensing and communications tasks reduces to a channel.
  • The Channel Model
  • The channel is labeled by effective distortion.
formula for finite sensors
Formula for Finite Sensors
  • Finite-scale Sensor Networks
  • Given the number of sensors, we get
  • with
  • where
a glimpse at statistics
A Glimpse at Statistics
  • In the large system limit, binomial distribution converges to normal distribution.
changing variables
Changing Variables
  • By the change of variables

we have the following result.

formula for infinite sensors
Formula for Infinite Sensors
  • Infinite-scale Sensor Networks
  • Given only the noise and bandwidth, we get
  • with
  • where we naturally expect that
lossy data compression
Lossy Data Compression
  • There exists tradeoff between compression rate and the resulting quality of reproduction.




What is the best bound for the lossy compression?

rate distortion theory
Rate Distortion Theory
  • Theory for compression beyond entropy rate.






Best bound is the rate distortion function.

can the ceo be informed
Can the CEO be informed?
  • Rate Distortion Function gives the best bound.
  • Large System Strategy by optimal codes

Leading Contribution

Taylor Expansion

Non-trivial regions are feasible

The CEO can be informed!

Does LSS have any advantage over SS?

indicator function
Indicator Function
  • In what condition the large system strategy outperforms the saturate strategy?
  • Saturate Strategy is used as the `reference’ in the decibel measure.



Which is outperforming?

LSS is outperforming when measure is negative.

SS is outperforming when measure is positive.

theoretical system gain
Theoretical System Gain
  • In the noisy environment, LSS is superior to SS!

Existence of comparative advantage gives a strong motivation for making large systems.

definition of vq
Definition of VQ
  • Any information bit belongs to the Voronoi region, and is replaced by its representative bit.
  • Index map specifies the representative bits.
  • Voronoi region is labeled by an index.
gauge of representative bit
Gauge of Representative Bit
  • Information is first divided into Voronoi regions, and then representative gauge is chosen.
isolated free energy
Isolated Free Energy
  • Free energy can be decoupled.
  • Hamming Distortion can be derived.

Exact Solution

Cost Function


Random Walk Statistics

Isolated Model Reduces to Random Walk Statistics.

bit error probability
Bit Error Probability
  • Substitute exact solution into general formula.
  • Theoretical Performance
large system gain
Large System Gain
  • Bit error probability in decibel measure

Large system strategy is not so outperforming

rate distortion theory1
Rate Distortion Theory
  • N bit sequence is encoded into M bit codeword.
  • M bit codeword is decoded to reproduce N bit sequence, but not perfectly.
  • Tradeoff relation between the rate R=M/N and the Hamming distortion D.
    • Rate distortion function for random sequences
sparse matrix coding
Sparse Matrix Coding
  • Find a codeword sequence that satisfies:

where the fidelity criterion:

    • Boolean matrix A is characterized by K ones per row and C per column; an LDPC matrix.
    • Bit wise reproduction errors are considered; the Hamming distortion measure D is selected.
example 4 bit sequence
Example: 4 bit sequence
  • Set an LDPC matrix.
  • Given a sequence:
  • Find a codeword:
  • Reproduce the original sequence.
design principle
Design Principle
  • Algebraic constraints are represented in a graph.
  • Probabilistic constraint is considered as a prior.

Microscopic consistency might induce the macroscopic order of the frustrated system.

low resource computation



Low-resource Computation
  • Introduce the mean field to avoid complex tasks.
  • Eliminate many candidates of the solution by dynamical techniques.
tap approach
TAP Approach
  • A codeword bit is calculated by its marginal.
  • Marginal probability is evaluated by heuristics.
empirical performance
Empirical Performance
  • Message passing algorithm works very well.
example of saturate strategy
Example of Saturate Strategy
  • Six sensors transmit their original datawords.

5.4k bps

BER 20.0%


32.4k bps


BER 20.0%

BER 9%


example of large system strategy
Example of Large System Strategy
  • Nine sensors transmit their codewords.

5.4k bps

BER 20.0%







32.4k bps

BER 24.7%

BER 5%


frustrated free energy
Frustrated Free Energy
  • Free energy cannot be decoupled.
  • General formula for Hamming Distortion


Cost Function


Replica Method

Saddle Point of Free Energy

Frustrated model reduces to spin glass statistics.

bit error probability1
Bit Error Probability
  • Substitute replica solution into general formula.
  • Theoretical Performance

Scaling Evaluation

for Replica Solution

characteristic constant
Characteristic Constant
  • Constant:
  • Saddle Point Equations
    • Variance of order parameter:
    • Non-negative entropy condition:
  • Measure:
large system gain k 2
Large System Gain: K=2
  • Bit error probability in decibel measure

Similar to the case of optimal random coding.

large system gain k
Large System Gain: K→∞
  • Bit error probability in decibel measure

Coincides with optimal random coding.

concluding remarks
Concluding Remarks
  • We consider the problem of distributed sensing in a noisy environment.
  • Limited bandwidth constraint induces tradeoff between reducing errors due to environmental noise and increasing errors due to lossy coding as number of sensors increases.
  • Analysis shows threshold behavior for optimal number of sensors.
  • TM and M. Okada: `Rate Distortion Function in the Spin Glass State: A Toy Model’, Advances in Neural Information Processing Systems 15, 423-430, MIT Press (2003).
  • Available at
  • TM and P. Davis: `Rate Distortion Codes in Sensor Networks: A System-level Analysis’, Advances in Neural Information Processing Systems 18, 931-938, MIT Press (2006).
  • Available at
  • TM: `Statistical mechanics of the data compression theorem’, Journal of Physics A 35, L95-L100 (2002).
  • Available at
  • TM: `Thouless-Anderson-Palmer Approach for Lossy Compression’, Physical Review E 69, 035105(R) (2004).
  • Available at
  • TM and P. Davis: `Statistical mechanics of sensing and communications: Insights and techniques’, Journal of Physics: Conference Series 95, 012010 (2007).
  • Available at
  • For more information, please google “tatsuto murayama” or “村山立人”.