1 / 24

Physics of information

Physics of information. ‘Communication in the presence of noise ’ C.E. Shannon, Proc. Inst. Radio Eng. (1949) ‘Some informational aspects of visual perception’ , F. Attneave, Psych. Rev. (1954). Ori Katz ori.katz@weizmann.ac.il. Talk overview. Information capacity of a physical channel

audi
Download Presentation

Physics of information

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Physics of information ‘Communication in the presence of noise’ C.E. Shannon, Proc. Inst. Radio Eng. (1949) ‘Some informational aspects of visual perception’, F. Attneave, Psych. Rev. (1954) Ori Katz ori.katz@weizmann.ac.il

  2. Talk overview Information capacity of a physical channel Redundancy, entropy and compression Connection to biological systems Emphasis concepts,intuitions, and examples

  3. A little background An extension of “A mathematical theory of communications”, (1948). The basis for information theory field (first use in print of ‘bit’) Shannon worked for Bell-labs at the time. His Ph.D thesis: “An algebra for theoretical genetics”, was never published Built the first juggling machine (‘W.C.Fields’), and a mechanical-mouse with learning capabilities (‘Theseus’) ‘Theseus’ ‘W.C. Fields’

  4. A general communication system Continuous function s(t)+n(t) s(t) – pressure amplitude Continuous function s(t) ‘message’ ‘message’ Physical Channel (bandwidth W) Receiver Decoder Encoder Transmitter Information destination Information source Added noise • Shannon’s route for this abstract problem: • Encoder codes each message  continuous waveform s(t) • Sampling theorem: s(t) represented by finite number of samples • Geometric representation: samples  a point in Euclidean space. • Analyze the addition of noise (physical channel) •  a limit on reliable transmission rate

  5. Vn=[s(Δt), s(2 Δt),…] The (Nyquist/Shannon) sampling theorem Transmitted waveform = a continuous function in time s(t), bandwidth (W) limited by the physical channel: S(f>W)=0 sample its values at discrete times Δt=1/fs: (fs = sampling frequency) s(t) can be representedexactly by the discrete samples Vnas long as: fs  2W (Nyquist sampling rate) Result: waveform of duration T, is represented by 2WT numbers = a vector in 2WT-dimensions space: V=[s(1/2W), s(2/2W),… , s(2WT/2W)] Fourier (freq.) domain: S(f>W)=0

  6. Audible human-ear frequency range: 20Hz - 20KHz • The Nyquist rate is therefore: 2 x 20KHz = 40KHz • CDsampling rate = 44.1KHz, fulfilling Nyquist rate. An example for Nyquist rate – a music CD Anecdotes: Exact rate was inherited from late 70’s magnetic-tape storage conversion devices. Long debate between Philips (44,056 samples/sec) and Sony (44,100 samples/sec)...

  7. Each continuous signal s(t) of duration T and bandwidth W, mapped to •  a point in 2WT-dimension space (coordinates = sampled amplitudes): • V = [x1,x2,…, x2WT] = [s(1/2W), …, s(2WT/2W)] • In our example: • A 1 hour CD recording a single point in a space having: • 44,100 x 60sec x 60min = 158.8x106 dimensions (!!) • The norm (distance2) in this space is measures signal power / total energy  An Euclidean space metric The geometric representation

  8. N P P • Example in a 3-dimensional space (first 3 samples in the CD): • V = [x1,x2,…, x2WT] = [s(Δt), s(2Δt), …, s(T)] Addition of noise in the channel x3 “mapping” x1 x2 • Addition of white Gaussian (thermal) noise with an average powerN smears each point into a sphere cloud with a radii N: • For large T, noise power  N (statistical average) •  Received point, located on sphere shell: distance = noise  N •  “clouded” sphere of uncertainty becomes rigid VS+N = [s(Δt)+n(Δt), s(2Δt)+n(2Δt), …, s(T)+n(T)]

  9. Reliable transmission: receiver must distinguish between any two different messages, under the given noise conditions The number of distinguishable messages x3 N P P x1 x2 • Max number of distinguishable messages (M)  the ‘sphere-packing’ problem in 2TW dimensions: • Longer mapped message, ‘rigid’-er spheres •  probability to err is as small as one wants (reliable transmission)

  10. Channel bandwidth Signal to Noise Ratio (SNR) The channel capacity Number of distinguishable messages (coded as signals of length T): Number of different distinguishable bits: The reliably transmittablebit-rate (bits per unit time): (in bits/second) The celebrated ‘channel capacity theorem’ by Shannon. - Also proved that C can be reached

  11. P{s=v} • With no signal, the receiver measures a fluctuating noise • In our example: pressure fluctuations of air molecules impinging on the microphone (thermal energy): Gaussian white noise = Thermal noise? • The statistics of thermal noise is Gaussian: P{s(t)=v}  exp(-(m/2KT)v2) • The power spectral-density is constant: (power-spectrum |S(f)|2=const) “white” “pink/brown”

  12. (in bits/second) Some examples for physical channels Channel capacity limit: 1) Speech (e.g. this lecture): W=20KHz, P/N=~1 - 100  C  20,000bps – 130,000bps Actual bit-rate = ~ (2 words/sec) x (5 letters/word) x (5 bits/letter) = 50 bps 2) Visual sensory channel: (Images/sec) x  (receptors/image)  x (Two eyes) Bandwidth (W) =      ~25     x    ~50x106       x     ~2  = ~2.55x109 Hz P/N > 256  C  2.5x109 x log2(256) = ~20x109 bps A two-hour movie:  2hours x 60min x 60 sec x 20Gbps = 1.4x1014bits = ~15,000 Gbytes (DVD = 4.7Gbyte) We’re not using the channel capacity  redundant information Simplify processing by compressing signal Extracting only the essential information (what is essential…?!)

  13. Original sample: 44.1Ks/s x 16bit/s = 705Kbps (CD quality) Redundant information demonstration (using Matlab)

  14. With only 4bit per sample 44.1Ks/s x 4bit/s = 176.4Kbps

  15. With only 3bit per sample 44.1Ks/s x 3bit/s = 132.3Kbps

  16. With only 2bit per sample 44.1Ks/s x 2bit/s = 88.2Kbps

  17. With only 1bit per sample (!) 44.1Ks/s x 1bit/s = 44.1Kbps Sounds not-too-good, but the essence is there… Main reason: not all of ‘phase-space’ is accessible by mouth/ear Another example: (smart) high-compression mp3 algorithm: @16Kbps

  18. Visual redundancy / compression • edges • short-range similarities • patterns • repetitions • symmetries • repetitions • etc, etc…. What information is essential?? (evolution…?) (2008) 400x600 704Kbyte .bmp 30.6Kbyte .jpg 10.9Kbyte .jpg 8Kbyte .jpg 6.3Kbyte .jpg 5Kbyte .jpg 4Kbyte .jpg Images: Redundancies in Attneave’s paper  image compression formats “a bottle” on “a table” (1954) 80x50 pixels • Movies: the same + consecutive images are similar… • Text: future ‘language’ lesson (Lilach & David)

  19. How much can we compress? Source ‘Entropy’ • How many bits are needed to code a message? • Intuitively: #bits = log2M (M - possible messages) • Regularities/Lawfulness  smaller M • some messages more probable  can do better than log2M • Can code a message with: (without loss of information) • Intuition: • Can use shorter bit-strings for probable messages.

  20. lossless-compression example (entropy code) • Example: M=4 possible messages (e.g. tones): • ‘A’ (94%), ‘B’ (2%), ‘C’ (2%), ‘D’ (2%) • 1) Without compression: 2 bits/message: • ‘A’00, ‘B’01, ‘C’10, ‘D’11. • 2) A better code: • ‘A’0, ‘B’10 , ‘C’110, ‘D’111 • <bits/message> = 0.94x1 + 0.02x2 + 2x (0.02x3) = 1.1 bits/msg

  21. Why entropy? • Any regularity  probable patterns  lower entropy • (redundant information) • Theonly measure that fulfills 4 ‘physical’ requirements: • H=0 if P(Mi)=1. • A message with P(Mi)=0 does not contribute • Maximum entropy for equally distributed messages • Addition of two independent messages-spaces: • Hx+y = Hx+Hy

  22. The speech Vocoder (VOice-CODer) Model the vocal-tract with a small number of parameters. Lawfulness of speech subspace only  fails for musical input Used by Skype / Google-talk / GSM (~8-15KBps) The ancestor of modern speech CODECs (COder-DECoders): The ‘Human organ’

  23. Link to biological systems • Information is conveyed via. a physical channel: • Cell to cell , DNA to cell, Cell to its descendant , Neurons/nerve system • The physical channel: • concentrations of molecules (mRNA, ions….) as a function of space and time. • Bandwidth limit: • parameters cannot change at an infinite rate (diffusion, chemical reaction timescales…) • Signal to noise: • Thermal fluctuations, environment • Major difference:not 100% reliable transmission •  Model: an overlap of non-rigid uncertainty clouds. • Use channel-capacity theorem at your own risk...

  24. Summary • Physical channel Capacity theorem • SNR, bandwidth • Geometrical representation • Entropy as a measure of redundancy • Link to biological systems

More Related