1 / 18

Estimating Maximal Information Amount under Interval Uncertainty

Gang Xiang Philips Electronics Healthcare Informatics Business Group gxiang@acm.org. Estimating Maximal Information Amount under Interval Uncertainty. Measure of Uncertainty. Uncertainty: we know n possible states. Measure of uncertainty:

dolph
Download Presentation

Estimating Maximal Information Amount under Interval Uncertainty

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Gang Xiang Philips Electronics Healthcare Informatics Business Group gxiang@acm.org Estimating Maximal Information Amount under Interval Uncertainty

  2. Measure of Uncertainty • Uncertainty: we know n possible states. • Measure of uncertainty: - A natural measure of uncertainty is the average number of binary (“yes” – “no”) questions we need to find the actual state. -In information theory, the average number of binary questions is called the amount of information. • Simple situation: - With K binary questions, we have 2K possible combinations of K answers. - So, for n=2K, we need K=log2n questions.

  3. Entropy for a Probability Distribution • Situation: - We know n states. - We knowthe probability p1, …, pnof the states. - What is the average number of binary questions to determine the state? • Answer: We need questions. (Shannon’s Entropy) • Exmaple: When p1=, …,= pn= 1/n, we need S=log2n questions (as before).

  4. Entropy of the Empirical Distribution • Probabilities can be estimated as frequencies. • Given:n values x1, …, xn where xiis the observed number of cases with state i. • Estimate: • Entropy:

  5. Formulation of the Problem • In practice, we sometimes only know bounds on xi: • We want to know largest possible number of questions (maximal information amount), i.e.

  6. Towards an O(n2) Time Algorithm • We have - If then , i.e., xi -> xi - h (h>0) increases S. - If then , i.e., xi -> xi + h (h>0) increases S. • So, considering each - If then to maximize S. (Selection of other value of xi leads to the contradiction that xi - h can obtain a greater S.) - If then to maximize S. (Selection of other value of xi leads to the contradiction that xi + h can obtain a greater S.) - If then to maximize S. (Selection of other value of xi leads to the contradiction that xi - h or xi + h can obtain a greater S.)

  7. Towards an O(n2) Time Algorithm A • This means, to maximize S, the choice of xi should make log2xi as close as possible to log2 xi

  8. Resulted O(n2) Time Algorithm • Step1: Sort 2n endpoints of into x(1) ≤ x(2) ,…, x(2n-1) ≤x(2n). Also, denote x(0)= 0, x(2n+1)=+∞. • Step 2: For each of 2n+1 intervals [x(k), x(k+1)], we assume that log2x(k) ≤ A≤ log2x(k+1). Then we can choose xi(the resulted tuple denoted as Xk) based on this assumption. Verify assumption by computing A (denoted as Ak), and check if log2x(k) ≤ Ak≤ log2x(k+1). If assumption is verified, continue to compute S (denoted as Sk) as a candidate of Smax. • Step 3: Pick up the largest of computed candidates as Smax.

  9. O(n2) Time Complexity • Step 1 (sorting) requires O(n·log(n)) time. • In Step 2, for each of the 2n+1 iterations: Choosing values of xirequires O(n) time. Computing Akrequires O(n) time. Computing Skrequires O(n) time. There, each iteration requires O(n) + O(n) + O(n) = O(n) time. Total 2n+1 iterations require O(n2) time. • Step 3 (finding maximum) requires O(n) time. • The algorithm totally requires O(n·log(n)) + O(n2) + O(n) = O(n2) time. • Can we speed it up?

  10. Towards an O(n·log(n)) Time Algorithm • Idea to speed up: we do not actually require O(n) time to obtain Xk, Ak and Sk for every k. • Once Xk, Ak and Sk are obtained, choosing Xk+1, computing Ak+1 and Sk+1 only require O(1) time, since Xk and Xk+1 only differ in the choice of one xi: • Therefore, all Xk, Ak and Sk can be obtained in O(n) time. (O(n) time to obtain X0, A0 and S0, and 2n·O(1) time to obtain others.) • The improved algorithm totally requires O(n·log(n)) (sorting) + O(n) (computing Smax’s candidates) + O(n) (finding maximum) = O(n·log(n)) time.

  11. Towards a Linear Time Algorithm • First idea to speed up: we do not actually require to obtain Xk, Akand Sk for every k. • We proved that: with the increase of k, first Ak>log2 x(k+1) , thenlog2 x(k) ≤ Ak≤ log2 x(k+1)for just one case, then Ak< log2 x(k). • Therefore, we can use binary search to find the maximizing k. We only need to obtain O(log(n))Xk ’s and Ak ’s, and compute Sk only once. • However, since we need O(n·log(n)) time for sorting, this speed-up does not reduce the total time complexity of O(n·log(n)).

  12. Towards a Linear Time Algorithm • Second approach to speed up: we do not actually need to sort all 2n endpoints of • To obtain Xk andAk,we only require to know the values ofx(k) and x(k+1), then classifyxi’s into “lower endpoint group”, “upper endpoint group” and “inner point group”. • Classification ofxiis actually the classification of the two endpoints of the interval whether they are both to the left of[x(k),x(k+1)], or they are both to the right of[x(k),x(k+1)], or one of them to the left and the other to the right.

  13. Towards a Linear Time Algorithm • In each iteration of binary search, we have three sets of endpoints. Left-side Endpoints Undecided Endpoints Right-side Endpoints

  14. Towards a Linear Time Algorithm • In each iteration of binary search. Find the median (attempted x(k)) and next greater endpoint (attempted x(k+1)) of undecided group, and use this median to divide undecided group. Left-side Endpoints Right-side Endpoints

  15. Towards a Linear Time Algorithm • In each iteration of binary search. Compute Ak and compare it with log2 x(k) and ln x(k+1). Assume these 2n-k endpoints are all right-side endpoints. Assume these k+1 endpoints are all left-side endpoints. Left-side Endpoints Right-side Endpoints

  16. Towards a Linear Time Algorithm • In each iteration of binary search. Update the sets of endpoints base on the result of comparison. For example, if Ak>log2 x(k+1) Left-side Endpoints Undecided Endpoints Right-side Endpoints

  17. Towards a Linear Time Algorithm • In the first iteration, the sets of left-side endpoints and right-side endpoints are empty. Akis computed directly. Time complexity of the iteration is O(2n+1). • In each other iteration of binary search. Akis computed based on the value of Ak from previous iteration. Time complexity of the iteration is only O(U) where U is the size of undecided group. • Totally, O(2n+1) + O((2n+1)/2) + O((2n+1)/4)+… (finding the maximizing k) + O(n) (computing Skas Smax) = O(2n+1) + O(n) = O(n)

  18. O(n·log(n)) Time Algorithm VS Linear Time Algorithm • The constant in the corresponding asymptotic. - 2 forthe O(n·log(n)) time algorithm. (Sorting 2n endpoints) - 2 * 20 = 40 for the lineartime algorithm. (Finding median in the 2n endpoints) • The linear time algorithm is better when log2(n)>20, i.e., n>106 • In practice, the O(n·log(n)) time algorithm is better.

More Related