Page - 287 - in Differential Geometrical Theory of Statistics

entropy Article GuaranteedBoundsonInformation-Theoretic MeasuresofUnivariateMixturesUsingPiecewise Log-Sum-ExpInequalities FrankNielsen1,2,*andKeSun3 1 ComputerScienceDepartmentLIX,ÉcolePolytechnique,91128PalaiseauCedex,France 2 SonyComputerScienceLaboratories Inc,Tokyo141-0022, Japan 3 KingAbdullahUniversityofScienceandTechnology,Thuwal23955,SaudiArabia; sunk.edu@gmail.com * Correspondence: Frank.Nielsen@acm.org;Tel.: +33-1-7757-8070 AcademicEditor:AntonioM.Scarfone Received: 20October2016;Accepted: 5December2016;Published: 9December2016 Abstract:Information-theoreticmeasures,suchastheentropy,thecross-entropyandtheKullback–Leibler divergence between two mixture models, are core primitives in many signal processing tasks. Since theKullback–Leiblerdivergenceofmixturesprovablydoesnotadmitaclosed-formformula, it is inpracticeeitherestimatedusingcostlyMonteCarlostochastic integration, approximatedor boundedusingvarious techniques.Wepresenta fastandgenericmethodthatbuildsalgorithmically closed-formlowerandupperboundsontheentropy, thecross-entropy, theKullback–Leiblerand theα-divergencesofmixtures.Weillustrate theversatilemethodbyreportingourexperiments for approximatingtheKullback–Leiblerandtheα-divergencesbetweenunivariateexponentialmixtures, Gaussianmixtures,RayleighmixturesandGammamixtures. Keywords: informationgeometry;mixturemodels;α-divergences; log-sum-expbounds 1. Introduction Mixturemodelsare commonlyused insignalprocessing. A typical scenario is tousemixture models [1–3] to smoothly model histograms. For example, Gaussian Mixture Models (GMMs) can be used to convert grey-valued images into binary images by building a GMM fitting the image intensity histogram and then choosing the binarization threshold as the average of the Gaussian means [1]. Similarly, Rayleigh Mixture Models (RMMs) are often used in ultrasound imagery[2] tomodelhistograms,andperformsegmentationbyclassification.Whenusingmixtures, a fundamentalprimitive is todefineaproperstatisticaldistancebetweenthem.TheKullback–Leibler (KL)divergence [4], alsocalledrelativeentropyor informationdiscrimination, is themostcommonly used distance. Hence the main target of this paper is to faithfully measure the KL divergence. Letm(x) = ∑ki=1wipi(x) andm ′(x) = ∑k ′ i=1w ′ ip ′ i(x) be twofinite statistical densitymixtures of k andk′ components, respectively.Notice that theCumulativeDensityFunction(CDF)ofamixture is like itsdensityalsoaconvexcombinationsof thecomponentCDFs.However,beware thatamixture is notasumofrandomvariables (RVs). Indeed, sumsofRVshaveconvolutionaldensities. Instatistics, themixturecomponents pi(x)areoftenparametric: pi(x)= p(x;θi),whereθi isavectorofparameters. Forexample,amixtureofGaussians(MoGalsousedasashortcutinsteadofGMM)haseachcomponent distributionparameterizedby itsmeanμi anditscovariancematrixΣi (so that theparametervector is θi=(μi,Σi)). LetX={x∈R : p(x;θ)>0}bethesupportof thecomponentdistributions.Denote byH×(m,m′) =− ∫ Xm(x) logm ′(x)dx the cross-entropy [4] between twocontinuousmixtures of Entropy2016,18, 442 287 www.mdpi.com/journal/entropy