- 111 Views
- Uploaded on
- Presentation posted in: General

ENGG2012B Lecture 19 Probability mass function and central limit theorem

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

ENGG2012BLecture 19Probability mass function and central limit theorem

Kenneth Shum

ENGG2012B

Domain

Range

y = f(w)

w

y

x

y = f(x)

s

t = f(s)

t

ENGG2012B

- f(x) = x(x-1)(x+1)

Range

Matlab commands for the graph

>> x = -2:0.01:2;

>> plot(x, x.*(x-1).*(x+1))

Domain

ENGG2012B

Range

- A linear transformation given by T(x,y) = (x+0.25y, y)

x x+ 0.25 y

y y

Domain

Linear transformation maps a parallelogram to a parallelogram

Eigenvectors are vectors whose direction does not change after transformation, T(v) = v .

ENGG2012B

T

Linear time-invariant system

s(t)

T( s(t) )

Domain

Range

(an RLC circuit for example)

Periodic signal

Periodic signal

Sinusoidal signals are eigen-functions.

Input is sinusoidal Output is sinusoidal

ENGG2012B

- A random variable X() is a function from a sample space to the real number line.

Pr(E) = probability of an event E

The real number line

E

X()

ENGG2012B

Real-life example: lucky rainbow

- Pick a random point in a rectangular board
- You win
- $2 if the point is inside the triangle.
- $1 if the point is insider the circle.
- nothing otherwise.

- Let X(p) be the winningsas a function of the random point p.
- X(p) = 2 if p is inside the triangle.
- X(p) = 1 if p is inside thecircle
- X(p) = 0 if p is outside thetriangle and the circle.

ENGG2012B

- Motivation: Sometimes, the underlying sample space is very complicated.
- We may try to forget the sample space and work with the probability mass function (pmf),
f(i) = Pr(X() = i).

- If we want to emphasize that it is the pmf of random variable X(), we can write fX(i).

f(i) = Pr( )

X()

i

ENGG2012B

- In the remainder of this lecture, we assume that the range of the random variables are non-negative integers.
- This kind of random variables are called discrete random variables.
- A discrete random variable may assume infinitely many values.

ENGG2012B

- Random experiment: toss n fair coins.
- The sample space contains 2n outcomes, and the outcomes are equally likely.
- An outcome is a string of H and T of length n.
- Let X() be the number of heads in .
- Let Y() be the number of tails in .
- As functions, X() is not equal to Y(). For example:
- X(HHHTHHT) = 5.
- Y(HHHTHHT) = 2. // same input, different outputs

- But X() and Y() have the same probability mass function.
- For i=0,1,…,n,

ENGG2012B

- Two random variables X and Y, whose underlying sample spaces are not necessarily the same, are said to be identically distributed if
fX(i) = fY(i) for all i.

- The example in the previous slide is an example of identically distributed random variables.
- By looking at the pmf’s of two identically distributed random variables, we cannot tell whether the sample spaces behind them are the same or not.

ENGG2012B

2

1

Toss two fair coins.Let Y() = 1 if both are heads, and

let Y() = 2 otherwise.

Pick a random point in a circle of radius 2

Let X() = 1 if the distance from to the centre is less than 1, and X() = 2 if the distance from to the centre is larger than or equal to 1.

P(X() = 1) = P(Y() = 1) = 0.25

P(X() = 2) = P(Y() = 2) = 0.75

ENGG2012B

- Two discrete random variables X() and Y() defined on the same sample space are said to be statisticallyindependent if
Pr(X() = i and Y() = j) = fX(i) fY(j) for all i and j.

- Example: Throw an isocahedral die and a dodecahedral die at the same time. The values of the two dice are independent (but not identically distributed).

ENGG2012B

- In each of the four examples, we pick a point randomly in the area.

X and Y are independent and i.d.

X and Y are i.d. but not independent

X=1, Y=1

X=1, Y=2

X=1, Y=3

X=2, Y=2

X=3, Y=1

X=2, Y=2

X=2, Y=1

X=1, Y=1

X=1, Y=2

X=1, Y=3

X=1, Y=1

X=1, Y=2

X=2, Y=2

X=2, Y=1

X=2, Y=3

X=2, Y=1

X=2, Y=2

X and Y are independent but not i.d.

X and Y are not i.d. and not independent .

ENGG2012B

ENGG2012B

- Toss a fair coin repeatedly until we get a head.
- I give you
- $0 if we have a head in the first toss
- $1 if we have a head in the second toss
- $2 if we have a head in the third toss
- …

- In other words, the amount of money I give you is equal to the number of tails.

ENGG2012B

- Let X be the amount I give you at the end of the game.
- The pmf of X is
for i =0,1,2,3,4,…

H

T

$0

H

T

$1

H

T

$2

H

T

$3

ENGG2012B

for i=0,1,2,3…

ENGG2012B

- This is not a free game.
- One need to pay c dollars to play this coin-tossing game.
- Suppose that I am the host of the game and I play this game with 1000 people. Each of them pay $c. What should be the value of c if I do not want to lose money?

ENGG2012B

- Most computer language comes with a pseudo-random number generator.
- The number returned can be regarded as uniformly distributed between 0 and 1.
- Construct another random variable Y, which is identically distributed to X, so that Y can be generated by computer easily.
- Then we can collect statistics from r.v. Y and get some intuition.

ENGG2012B

- log2(2) = 1, log2(4) = 2, log2(8) = 3, …
- -log2(1/2) = 1, -log2(1/4) = 2, -log2(1/8) = 3, …

ENGG2012B

- Let Y = floor(-log2( U ))
- U is drawn uniformly between 0 and 1.
- floor(x) is the largest integer less than or equal to x.

ENGG2012B

>> Y = floor(-log2(rand)) % generate one sample of Y

>> floor(-log2(rand(1,1000))) % generate one thousand % independent samples

>> hist(ans, 0:16) %plot the histogram of the samples

>> sum(floor(-log2(rand(1,1000))) ) % Total amount I paid to % 1000 people

ENGG2012B

- For a random variable with pmf fX(i), the average value we have after running the random experiment n times is approximately
- This motivates the definition of the expected value of a random variable with pmf fX(i):
- The expected value is often called the mean.

ENGG2012B

- For any constant c,
E[c X] = c E[X]

- For any two r.v. X and Y, which are not necessarily independent and not necessarily identically distributed, we have
E[X+Y] = E[X]+E[Y]

ENGG2012B

- The variance of a random variable measures how far the distribution is spread out.
- The variance is usually denoted by 2.
- The square root of the variance is called the standard deviation.

Larger variance

Smaller variance

ENGG2012B

- For a discrete random variable X with mean , the variance of X is defined as the expected deviation from the mean:
- Useful trick in computing variance

E[X2] is calledthe second momentof X

ENGG2012B

- For any constant c,
Var(c X) = c2 Var(X)

- For any two independent r.v. X and Y, we have
Var(X+Y) = Var(X) + Var(Y)

(When X and Y are not independent, this may or may not be true.)

ENGG2012B

ENGG2012B

- Compare the games on the left and right.

+

H

H

T

H

T

T

$0

$0

$0

H

T

H

T

H

T

$2

$1

$1

H

T

H

H

T

T

$4

$2

$2

H

T

H

H

T

T

$6

$3

$3

ENGG2012B

- Let X1 and X2 be independent and identically distributed random variables, with pmf
for i = 0,1,2,3,…

- What is the distribution of X1 + X2?

ENGG2012B

1/2

1/4

1/8

1/64

1/32

1/16

X2=5

X2=0

X2=4

X2=1

X2=2

X2=3

X1=0

1/2

X1=1

1/4

1/8

X1=2

1/16

X1=3

X1=4

1/32

X1=5

1/64

We have use the assumption that X1 and X2 be independentwhen we fill in the probabilities in the table.

ENGG2012B

1/2

1/4

1/8

1/64

1/32

1/16

X2=5

X2=0

X2=4

X2=1

X2=2

X2=3

X1=0

1/2

X1=1

1/4

1/8

X1=2

1/16

X1=3

X1=4

1/32

X1=5

1/64

Pr(X1 + X2 =0) = 1/4

Pr(X1 + X2 =3) = 4/32

Pr(X1 + X2 =1) = 2/8

Pr(X1 + X2 =4) = 5/64

Pr(X1 + X2 =2) = 3/16

In general, Pr(X1 + X2 =i) = (i+1) / 2i+2

ENGG2012B

For each non-negative integer i,

// enumerate all possibilities

// statistically independent

// substitute the values for each r.v.

ENGG2012B

for i=0,1,2,3…

ENGG2012B

- The calculation of the pmf of the sum of two independent random variable is essentially a convolution operation.
- Regard the pmf of a r. v. as a sequence of real numbers.
Pmf of X1 : (0.5, 0.25, 0.125, 0.0625, …)

Pmf of X2 : (0.5, 0.25, 0.125, 0.0625, …)

(0.5, 0.25, 0.125, 0.0625, …)

Pr(X1 + X2 =0)

(…, 0.0625, 0.125, 0.25, 0.5)

0.25

ENGG2012B

(0.5, 0.25, 0.125, 0.0625, …)

1/8+1/8=0.25

(0.5, 0.25, 0.125, 0.0625, …)

1/16+1/16+1/16

= 0.1875

(0.5, 0.25, 0.125, 0.0625, …)

Pr(X1 + X2 =1)

Pr(X1 + X2 =2)

Pr(X1 + X2 =3)

(…, 0.0625, 0.125, 0.25, 0.5)

(…, 0.0625, 0.125, 0.25, 0.5)

(…, 0.0625, 0.125, 0.25, 0.5)

1/32+1/32+1/32+1/32

= 0.125

ENGG2012B

>> v = 2.^-(1:20) % the pmf of X, truncated after 20

>> f = conv(v,v) % take the convolution of v with itself

>> stem(0: (length(f)-1),f) % stem plot

ENGG2012B

ENGG2012B

- A pmf can be represented by power series.
- Suppose that the pmf of random variable X is f(i),for i=0,1,2,3,…
- Define a power series
- The coefficients are the values of function f.
- The exponents of z are the values of X.
- The power series is sometime called the generating function of f.

ENGG2012B

- For the pmf f(i) = 1/2i+1 for i=0,1,2,3,… the power series representation is a geometric series:

ENGG2012B

- When evaluated at z=1, we get g(1)=1.
- If we differentiate g(z) once and evaluate at z=1, we get the mean.

ENGG2012B

- If we differentiate twice and evaluate at z=1, we get the variance

ENGG2012B

- For the pmf f(i) = 1/2i+1 for i=0,1,2,3,…the power series is g(z) = 1/(2-z).

ENGG2012B

- Regard the pmf of a r. v. as a power series.
- In the example of coin-tossing game,
Pmf of X1 : g(z) = 0.5+ 0.25z + 0.125z2 + 0.0625z3

Pmf of X2 : h(z) = 0.5+ 0.25z + 0.125z2 + 0.0625z3

Pmf of X1 + X2 : g(z)h(z) = 0.25 + 0.25z + 0.1875z2 + 0.125z3 +…

ENGG2012B

- Convolution is just the product of two power series.
- This is the same as the maxim“Convolution in time domain is equivalent to multiplication in frequency domain”

For example:

0.5+ 0.25z +0.125z2 + 0.0625z3 + …)

1/16z2+1/16z2+1/16z2=0.1875z2

Pr(X1 + X2 =2)coeff. of z2

(…, 0.0625z3+ 0.125z2+ 0.25z + 0.5)

ENGG2012B

- Compare the mean and variance of the two games.

+

H

H

T

H

T

T

$0

$0

$0

H

T

H

T

H

T

$2

$1

$1

H

T

H

H

T

T

$4

$2

$2

H

T

H

H

T

T

$6

$3

$3

ENGG2012B

- Suppose X, X1 and X2 are independent and identically distributed r.v. with pmf f(i) = 1/2i+1 for i=0,1,2,3,…
- E[2X] = 2 E[X] =2,
Var(2X) = 4Var(X) = 8.

- E[X1 + X2] = E[X1] + E[X2] = 2,
Var(X1 + X2) = Var(X1) + Var(X2) = 2+2=4.

ENGG2012B

ENGG2012B

>> v = 2.^-(1:20) % the pmf of X , truncated after 20

>> f = conv(v,v) % take the convolution of v with itself

>> f = conv(f,v) % take the convolution with v again

>> stem(0: 30, f(1:31) ) % plot the first 31 numbers

ENGG2012B

>> v = 2.^-(1:20) % the pmf of X , truncated after 20

>> f = [1]; % Initialize f to the “impulse response”

>> for j=1:50 f = conv(f,v); end % take the convolution of with v 50 times

>> stem(0: 100,f(1:101)) % plot the first 101 numbers

ENGG2012B

>> hold on

>> stem(0: 100,f(1:101)) % plot the first 101 numbers

>> x = 0:0.1:100;

>> plot(x , sqrt(2*pi*100)^-1 * exp(-((x-50)).^2/(2*100)), 'r')

% plot the Gaussian distribution with the same mean and variance

>> hold off

Mean = 501Var = 502

ENGG2012B

- A.k.a. the normal distribution.

ENGG2012B

- Start with an arbitrarily given probability mass function f, with
- mean = and
- Variance = 2.

- Generate n independent r.v.’s Z1, Z2, …, Zn, according to f.
- The sum Z1+ Z2+ …+ Zn is another r.v.
- The pmf of the sum Z1+ Z2+ …+ Zn is well approximated by the Gaussian distribution with mean n and variance n 2 when n is large .
- rule of thumb: n>30.

ENGG2012B

>>v = [0.1 0.4 0.2 0.3]; % an arbitrary pmf , P(X=0)=v(1), P(X=1)=v(2), P(X=2)=v(3) ...

>> n = 20; % we will plot the pmf of n independent r. v.

>> mu = sum(v.*(0:(length(v))-1)); % mean

>> variance = sum(v.*(0:length(v)-1).^2) - mu^2; % variance

>> f=[1]; % initialize f to the ``impulse response''

>> for i = 1:n

>> f = conv(f,v) ; % take convolution with v n times

>> end

>> clf % clear figure

>> hold on

>> stem(0:(length(f)-1),f) % plot the pmf of the sum of n independent random variables

>> x = 0:0.01:length(f); % Plot the Gaussian approximation

>> plot(x , sqrt(2*pi*n*variance)^-1 * exp(-((x-n*mu)).^2/(2*n*variance)), 'r')

>> hold off

ENGG2012B

ENGG2012B

- Toss a fair coin repeatedly until we get a head.
- I give you
- $1 if we have a head in the first toss
- $2 if we have a head in the second toss
- $4 if we have a head in the third toss
- $8 if we have a head in the third toss

- The amount of money I give you is equal to 2^(the number of coin tosses).

ENGG2012B

- Let X be the amount I give you at the end of the game.
- The expectation of X is

H

T

$1

H

T

$2

H

T

$4

H

T

If the cost is $1, are you willing to enter the game?

$8

How about $100?

How about $10?

ENGG2012B