What’s A Bit?

1. COMP 40: Machine Structure and Assembly Language Programming (Spring 2014) What’s A Bit? Noah Mendelsohn Tufts UniversityEmail: noah@cs.tufts.edu Web: http://www.cs.tufts.edu/~noah

2. Topics • What is information? • What’s a bit? • How do computer memories store information?

3. Shannon&Information Theory

4. Claude Shannon 1948: Claude Shannon publishes: A mathematical theory of communication* * http://www.alcatel-lucent.com/bstj/vol27-1948/articles/bstj27-3-379.pdf Photo by Tekniska Museet

5. Questions • We can weigh things that have mass • We can determine the volume of solid object or a liquid • We can measure the height of the walls in this room • Can we measure information? • Can we distinguish more information from less? • What units could we use?

6. Intuition • There is more information in the library of congress than there is in a single word of text… • …but how can we prove that rigorously?

7. Crucial insight • Whenever two parties communicate: • We can view the communication as answering one or more questions • Example: you and I are deciding whether to have dinner. We agree in advancethat I am going to phone you and give you the shortest possible message to convey the answer. I will say “yes” if we’re having dinner and “no” if not. • Harder example: we also need to decide whether we’re going to the movies. This time, we agree in advancethat I will say “yes, yes” for movie and dinner, “yes, no” for dinner only, “no, yes” for movie only, and “no, no” for staying home. This is profound… …communication is answering questions!

8. Crucial insight • Whenever two parties communicate: • We can view the communication as answering one or more questions • Example: you and I are deciding whether to have dinner. We agree in advance that I am going to phone you and give you the shortest possible message to convey the answer. I will say “yes” if we’re having dinner and “no” if not. • Harder example: we also need to decide whether we’re going to the movies. This time, we agree in advancethat I will say “yes, yes” for movie and dinner, “yes, no” for dinner only, “no, yes” for movie only, and “no, no” for staying home. This is profound… …communication is choosing among possibilites!

9. Measuring information • Just saying “yes” or “no” isn’t enough… • …we have to agree on what the choices are • The more choices we have to make, the more “yes” or “no” answers we’ll have to communicate Shannon’s paper introduced the term bit! (atributing it to co-worker John Tukey) We define the ability to convey a single yes/no answer as a bit We define the amount of information as the number of yes/no questions to be answered

10. How many bits for Beethoven’s 9th Symphony? • If you and I agree in advance that we are choosing between only two recordings that we both have, then: • We can choose between them with 1 bit! • If we agree in advance only that it is some digital sound recording, then: • We need enough bits so that you can choose the intended sound wave from all possible such 70+ minute recordings* * By the way, the compact disc format was chosen to have enough bits to encode Beethoven’s 9th , at 44KHz x 16bits/sample x 2 channels estimated at 74 minutes. Approximately 5,872,025,600 bits

11. Things to notice • We always have to agree in advance what the possible choices are: • Whether we’re having dinner or not • Which of N sound wave forms I want you to reproduce • We always have to agree on which answers (bit values) corresponds to which choices • We can use any labels we like for the bit values, e.g. • [yes] will mean Beethoven • [yesyes] will mean dinner and movie • Or… • [true]will mean Beethoven • [true false] will mean dinner and no movie

12. What if we want to encode numbers? • We always have to agree in advance what the possible choices are: • Whether we’re having dinner or not • Which of N sound wave forms I want you to reproduce • Which of N numbers I’ve stored in a computers’s memory • Question: What are good labels for encoding numbers?

13. Let’s try some labels for encoding numbers

14. Let’s try some labels for encoding numbers Any two labels will do… …but do you notice a pattern?

15. Let’s try some labels for encoding numbers Hey, that’s the binary representation of the number!

16. Encoding numbers in a computer memory • How many bits do I need if we need to encode which of 8 values are in the memory? • 1 bit: 0 or 1 [two choices] • 2 bits: 00, 01, 10,11 [four choices] • 3 bits: 000, 001, 010, 011, 100, 101, 110, 111 [8 choices] • Number_of_choices = 2N-bits N-Bits = log2 (number_of_choices)

17. Encoding numbers in a computer memory • How many bits do I need if we need to encode which of 8 values are in the memory? • 1 bit: 0 or 1 [two choices] • 2 bits: 00, 01, 10,11 [four choices] • 3 bits: 000, 001, 010, 011, 100, 101, 110, 111 [8 choices] • Number_of_choices = 2N-bits • As we said, those are binary numbers! 1 x 22 + 0 x 21 + 1 x 20 • So…that’s why we label the states zero and one, because we can play this game to assign bit patterns to binary encodings of numbers N-Bits = log2 (number_of_choices) = 5 Our hardware has instructions to do very efficient arithmetic on these binary representations of numbers

18. Note: we will discuss negative numbers, numbers with fractions, very large and very small numbers, and arithmetic on all of these, at a later time

19. Software structures model real world objects and concepts • Number • Students • Bank statements • Photographic images • Sound recordings • Etc. These things aren’t bits!! They don’t live in computers, but…

20. Software structures model real world objects and concepts • Numbers • Students • Bank statements • Photographic images • Sound recordings • Etc. We build data structures that model them ...we agree which bit patterns represent them

21. What we’ve learned so far… • Bits encode yes/no choices • To communicate, we agree in advance on which bit patterns represent which choices • More information: more choices…which means more bits! • We can store any information in a computer memory as long as we agree on which bit patterns represent which choice • If we label the bit states 0 and 1, then binary numbers are an obvious representation for the integers • We choose other encodings for characters (e.g. ASCII), photos (pixel on/off), music (digitized wave amplitude)

22. How Do We Build Bits into Computers?

23. Building a bit in hardware • We need hardware that can be in a choice of two states • Computer main memory history: • 1940’s: spots on a TV tube; sound pressure waves in a mercury delay line; vacuum tubes • 1950’s: rotating magnetic drum; vacuum tubes • 1950s – 1970s: tiny magnetizeable iron donuts (core memory) • 1970s – present: charges on a capacitor driving a transistor • Computer bulk storage • Magnetizeable tape • Magnetizeable disk • Transistors holding charge or solid state magnetic devices These vary in cost/size/speed – all encode bits

24. Technology for Storing Bits Thyratrons&Vacuum Tubes* Relay

25. Technology for Storing Bits Magnetic Tape Transistors Core Memory Punch Cards Integrated circuit USB Key Limited Integration:

26. Binary Numbers

27. Learn your binary numbers 220 ~= 1M 230~= 1B 232~= 4B 264 = HUGE

28. Another way to think about it 1011 = 11 (decimal) 16 8 0

29. Another way to think about it 1011 16 8 0

30. Another way to think about binary numbers 1011 16 8 0

31. Another way to think about binary numbers 1011 16 8 0

32. Another way to think about binary numbers 1011 16 8 0 The binary representation encodes a binary search for the number!

33. 0 1 1 0 0 1 1 0 1 1 1 0 0 1 1 0 0 1 1 0 0 1 1 0 1 1 1 0 0 1 1 0 0 1 1 0 0 1 1 0 1 1 1 0 0 1 1 0 0 1 1 0 0 1 1 0 1 1 1 0 0 1 1 0 32 bits

34. The logical structure of computer memory Can we get a C pointer to a bit in memory? NO! 0 1 1 0 0 1 1 0 1 1 1 0 0 1 1 0 0 1 1 0 0 1 1 0 1 1 1 0 0 1 1 0 0 1 1 0 0 1 1 0 1 1 1 0 0 1 1 0 0 1 1 0 0 1 1 0 1 1 1 0 0 1 1 0 0 1 2 3 4 5 6 7 Addr: Pointers (on most modern machines) are to whole bytes

35. Why byte addressing? • Can address more memory with smaller pointers • Not too big, not too small  • 256 values: about right for holding the characters Western cultures need (ASCII) – one character fits in one byte • 8 is a power of 2 … we’ll see advantages later • Unfortunately: we need multiple byte representations for non-alphabetic languages (Chinese, Japanese, Arabic, etc.) – we deal with that in software What’s the largest integer we can store in a byte?

36. Computers can work efficiently with bigger words Sizes vary with machine architecture: these are for AMD 64 Byte Byte Byte Byte Byte Byte Byte Byte BYTE (8) SHORT (16) INT (32) LONG (64) POINTER (64) • C has types for these • The hardware has instructions to directly manipulate these • The memory system moves these efficiently (in parallel)

37. Review

38. Review • Bits encode choices • We can thus choose a representation for information in bits • We can interpret the same bit values in different ways (number 66 or ASCII letter C) • If we call the bit states 0 & 1: then we easily get binarynumbers • We know how to implement bit stores in hardware and to compute with them • We generally address bytes not bits • We often use words of types like integer…these are useful, and the machine handles them efficiently

39. Abstractions -- again

40. Abstractions Are Layered at Every Level in our Systems • “Real world” concepts modeled in Hanson ADTs & C Types • Hanson ADTs implemented in C Types • Soon: bits & bytes used encode machine instructions • Words, Bytes and bits used to implement C types • Bytes grouped in hardware to make words (int, long, etc.) • True/false bits grouped to make bytes • Information modeled as true/false bits • True/false bits encoded in charges on transistors

41. An Aside on Information Theory

42. We’ve over-simplified the story a little • What we’ve said about bits and choices is true • However: • Many encodings are wasteful…I.e. values of the bits are somewhat predictable • Example: for each day of the year: [1=There was a hurricane, 0=No hurricane]…we know most bits will be zero • Can you find a better encoding? • To really measure information: we use the smallest possible encoding • Also:Shannon didn’t just measure information…he predicted how reliably you could send it through a noisy transmission line Still: what we’ve studied here is a great start on thinking about bits and information, which are the foundations for modern digital computing.