Download Presentation
## Entropy and Information Theory

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -

**Entropy and Information Theory**Aida Austin 4/24/2009**Overview**• What is information theory? • Random variables and entropy • Entropy in information theory • Applications • Compression • Data Transmission**Information Theory**• Developed in 1948 by Claude • E. Shannon at Bell Laboratories • Introduced in “A Mathematical • Theory of Communication'' • Goal: Efficient transmission of • information over a noisy • network • Defines fundamental limits on • compression needed for reliable • data communication Claude E. Shannon**Random Variables**• A random variable is a function. • Assigns numerical values to all possible • outcomes (events) • Example: A fair coin is tossed. Let X • represent a random variable. • Possible outcomes:**Entropy in Information Theory**• Entropy is the measure of the average information • content missing from a set of data when the value • of the random variable is not known. • Helps determine the average number of bits • needed for storage or communication of a signal. • As the number of possible outcomes for a random • variable increases, entropy increases. • As entropy increases, information decreases • Example: MP3 sampled at 128 kbps has higher • entropy rate than 320 kbps MP3**Applications**• Data Compression • MP3 (lossy) • JPEG (lossy) • ZIP (lossless) • Cryptography • Encryption • Decryption • Signal Transmission Across a Network • Email • Text Message • Cell phone**Data Compression**• “Shrinks” the size of a signal/file/etc. to • reduce cost of storage and transmission • Smaller data size reduces the possible • outcomes of the associated random • variables, thus decreasing the entropy of • the data. • Entropy - minimum number of bits needed • to encode with a lossless compression. • Lossless (no data lost) if the rate of • compression = entropy rate**Signal/Data Transmission**• Channel coding reduces bit error and bit loss • due to noise in a network. • As entropy increases, the likelihood of valuable • information transmitted decreases. • Example: Consider a signal composed of • random variables. • We may know the probability of certain values • being transmitted, but we do not know the exact • values will be received unless the transmission • rate = entropy rate.**Resources**http://www.gap-system.org/~history/PictDisplay/Shannon.html http://en.wikipedia.org/wiki/Information_entropy http://en.wikipedia.org/wiki/Bit_rate http://lcni.uoregon.edu/~mark/Stat_mech/thermodynamic_entropy_and_information.html http://www.data-compression.com/theory.html