Review: Sending Models and Data Given the Models

1. Review: Sending Models and Data Given the Models • Models/Explanation/Theories are combination of random variables. • Need to send random variables and their parameters • Need to send structure between random variables (next lecture) • Then send data given the model • Best model minimizes length in bits of sending model and then data given model. CSI 661 - Uncertainty in A.I. Lecture 24

2. Review: Discrete Random Variable Encoding - 1 • Non-combinatorial approach • Create a Huffman code-book. One code word for each individual state/source code symbol • Assign code word lengths according to –log(P(X=i) • Number of bits to send model/dictionary (code-book) • Number of bits to send data given model is N.H[X] CSI 661 - Uncertainty in A.I. Lecture 24

3. Review: Discrete Random Variable Encoding - 2 • Combintorial Approach • Create a Huffman code-book. One code-word for each possible combintorial sequence the events • log(S) bits to send one of S equally likely events, • Number of bits to send model • Number of bits to send data CSI 661 - Uncertainty in A.I. Lecture 24

4. Continuous Random Variables • Number of “events” for continuous data is infinte. But data is actually xAOM/2 • Num. bits to specify data given model log(AOM.f(x|)) as f(.) is a pdf, d = m / v • But the parameters of the model are also continuous values CSI 661 - Uncertainty in A.I. Lecture 24

5. Discretizing the Parameter Space • Need to discretize parameter space into cells • If cell mid-point is the parameter to encode/decode data, coding scheme is highly dependent on apriori specified trivial coding details. • Instead measure expected (average) message length using all the parameter values within the cell AOPV AOPV CSI 661 - Uncertainty in A.I. Lecture 24

6. Original MML Formulation • Message length calculation for normal dist. • Differentiating expression with respect to unknowns seperately gives CSI 661 - Uncertainty in A.I. Lecture 24

7. A Better MML Formulation • Mess Len to specify model : -log(h()) . ??? • E[Mess Len] to specify data given model • Total Mess Len CSI 661 - Uncertainty in A.I. Lecture 24