Entropy

Cairo University Institute of Statistical Studies & Research Department of Mathematical Statistics M.Sc Proposal Tests Based on Sampling Entropy By Mohamed Soliman Abdallah Supervised By Prof.Samir Kamel Ashour Dr.Esam Aly Amin 2008

Contents •

What is the sampling Entropy?

•

Measures related to Sampling Entropy.

•

Principle of Maximum Entropy.

•

Using Sampling Entropy for Goodness of fit.

What is the sampling Entropy? Definition In the information theory, the entropy is a Greek notation meaning transformation that is a measure or an index of the uncertainty associated with a random variable.

Suppose we have two classes with four students, the probability that each student can pass the exam as follows: Class A : .7,.1,.1,.1 Class B:.25,.25,.25,.25 It is obvious Class A has more information than Class B.

Shannon (1948) demonstrated many nice properties about the entropy measure to be called a measure of information: 2. The quantity H(X) reaches a minimum, equal to zero, when one of the events is a certainty.

1. If some events have zero probability, they can just as well be left out of the entropy

when

we

evaluate

the

uncertainty. 2. Entropy information must be symmetric that doesn’t depend on the order of the probabilities. 3. Entropy information should be maximal if all the outcomes are equally likely.

If X has discrete distribution, its entropy can be n

S ( X ) = −∑ p ( xi ) log b p ( xi ) i =1

• The base typically be the nature base in social science and refers to the bit for communication field. • Shannon(1948) claimed the following formula can satisfy the previous properties. • For Class A

S A ( x ) =.94

• For Class B

S B ( x) = 1.38

En tr op y is con cave S(p)

0

½

1

p

Joint Entropy Definition Joint Entropy is a measurement concerned with uncertainty of the two variable takes the following formula:

n

S ( X , Y ) = −∑ p( xi , yi ) ln p ( xi , yi ) i =1

If the two variables are independent the Joint Entropy will be : S ( X , Y ) = S ( x) + S ( y )

Mutual Information Definition Mutual Information measures the information that X and Y share, in other words how much knowing one of these variables reduces our uncertainty about the other, takes the following formula: n

p ( xi , yi ) I ( X , Y ) = −∑ p( xi , yi ) ln p ( x i ) p ( yi ) i =1

If the two variables are independent I(x,y)=0

Conditional Entropy Definition Conditional Entropy S(X / Y) is a measure of what Y doesn’t say about X, meaning how much information in X doesn’t in Y, takes the following formula: S ( X / Y ) = S ( X , Y ) − S (Y )

If the two variables are independent: S(X /Y ) = S(X )

We can express the previous measures in a Venn diagram as following:

Note:

• all previous measures can be extended to continuous distribution. • Esteban (1995) mentioned there are 22 measures of entropy in the literature rather than Shannon entropy . • Entropy can be considered in this regard as similar to the variance of a random variable whose values are real numbers Frenken (2003) .

Principle of Maximum Entropy Principle of Maximum Entropy (POME) is a relative new estimation in statistical analysis singh(1986). According to jaynes(1957), the idea we search for estimates for the parameters guarantee minimally biased distribution of X subject to a given prior information .

Example Suppose there is a restaurant has three meals {C,D,E} with {1 $,2 $,3 $} respectively, if we have information that the customer can spend in average 1.5 $ of the meal. what is the different probabilities that the customer will demand each meal ?

According to Paul(2003)using Lagrange Multiplier Method as follows:

n

n

n

i =1

i =1

i =1

L = −∑ p ( xi ) ln p ( xi ) − (λ − 1)(∑ p( xi ) − 1) − µ (∑ xi p( xi ) − 1.2)

dL = − ln p ( xi ) − λ − µ j xi = 0 dp( xi ) p

( x)

m i

e

i = 1..3

− λ − µ ⋅ xi

i = 1..3

With little algebra the different probabilities can be obtained as follows: p ( D) .15 p ( E) .02 p ( C)

.83

Gelder (2000) mentioned singh(1985) was the leader for estimating continuous distribution’s parameter by POME, Evans(1978)proposed an approach for determining constraints, singh(1998) and others conclude by Monto Carlo simulation : • POME offers an alternative method for estimating

parameters among all the methods of estimation . • POME yielded the least parameter bias for all

sample sizes.

1.There is a high relation between POME & Maximum Likelihood Estimation as follows: L( x) = Πf ( x;θ ) n ln L( x) n∑ ln f ( x;θ ) = = nE (ln f ( x;θ )) = − nS ( X ) n n

Note : there is a high argument in the literature about the relation between Bayesian estimation and POME Jaynes(1976,1982).

Goodness of fit based on maximum entropy

Vasiek’s test (1976)

Test Stengos and Ximing (2007)

1.

Vasicek (1976), he proposed an entropy based test for the composite hypothesis of normality based on Monte Carlo simulation .

2.

Dudewicz and van der Meulen (1981) extended Vasicek (1976) test and proposed test for uniformity.

3.

Mudholkar and Lin ( 1984), Taufer (2002) proposed a test based on entropy for testing exponentiality .

4.

Gokhale(1983) using maximum entropy for testing Gamma distribution , Beta distribution and Double Exponential distribution.

• Stengos and Ximing (2007) drive a general test based on maximum entropy density for normality using Lagrange Multiplier, Wald test and the conventional Likelihood Ratio, they proved their test has asymptotic χ with five degrees of freedom. 2