Page - 295 - in Differential Geometrical Theory of Statistics

Entropy2016,18, 442 solutions.AsshowninFigure1, theupperenvelopeofGaussiandensitiescorrespondsto the lower envelopeofparabolas.Wehave Ci,j(a,b)=Mi(a,b) ( logw′j− logσ′j− 1 2 log(2π)− 1 2(σ′j)2 ( (μ′j−μi)2+σ2i )) + wiσi 2 √ 2π(σ′j)2 ⎡⎣(a+μi−2μ′j)e−(a−μi) 2 2σ2i −(b+μi−2μ′j)e −(b−μi)2 2σ2i ⎤⎦ , (22) Mi(a,b)=−wi2 ( erf ( b−μi√ 2σi ) −erf ( a−μi√ 2σi )) . (23) 2.3.4. TheCaseofGammaDistributions Forsimplicity,weonlyconsidergammadistributionswith theshapeparameterk>0fixedand the scaleλ> 0varying. Thedensity isdefinedon (0,∞) as p(x;k,λ) = x k−1e− x λ λkΓ(k) ,whereΓ(·) is the gammafunction. ItsCDFisΦ(x;k,λ)=γ(k,x/λ)/Γ(k),whereγ(·, ·) is the lower incompletegamma function. Twoweightedgammadensitiesw1p(x;k,λ1)andw2p(x;k,λ2) (withλ1 =λ2) intersectata uniquepoint x = ( log w1 λk1 − logw2 λk2 )/( 1 λ1 − 1 λ2 ) (24) ifx >0;otherwise theydonot intersect. Fromstraightforwardderivations, Ci,j(a,b)= log w′j (λ′j)kΓ(k) Mi(a,b)+wi ∫ b a xk−1e− x λi λkiΓ(k) ( x λ′j −(k−1) logx ) dx, (25) Mi(a,b)=− wiΓ(k) ( γ ( k, b λi ) −γ ( k, a λi )) . (26) Similar tothecaseofRayleighmixtures, the last terminEquation(25)reliesonnumerical integration. 3.Upper-BoundingtheDifferentialEntropyofaMixture First, considerafiniteparametricmixturemodelm(x)=∑ki=1wip(x;θi). Usingthechainruleof theentropy,weendupwith thewell-knownlemma: Lemma1. The entropy of a d-variatemixture is upper bounded by the sumof the entropy of itsmarginal mixtures: H(m)≤∑di=1H(mi),wheremi is the1Dmarginalmixturewith respect tovariable xi. Since the1DmarginalsofamultivariateGMMareunivariateGMMs,wethusgeta looseupper bound. Ageneric sample-basedprobabilistic bound is reported for the entropies of distributions withgivensupport [31]: Themethodbuildsprobabilisticupperandlowerpiecewisely linearCDFs basedonani.i.d. finitesamplesetof sizenandagivendeviationprobability threshold. It thenbuilds algorithmicallybetweenthose twoboundsthemaximumentropydistribution[31]withaso-called string-tighteningalgorithm. Instead,weproceedas follows: Considerfinitemixtures of componentdistributionsdefined on the full supportRd thathavefinite componentmeansandvariances (likeexponential families). Thenweshalluse the fact that themaximumentropydistributionwithprescribedmeanandvariance isaGaussiandistribution,andconclude theupperboundbypluggingthemixturemeanandvariance in thedifferential entropy formulaof theGaussiandistribution. Ingeneral, themaximumentropy withmomentconstraintsyieldsasasolutionanexponential family. 295