Page - 275 - in Differential Geometrical Theory of Statistics

Entropy2016,18, 407 3.GeneralizationofRényiDivergence In this section, weprovide a generalization of Rényi divergence, which is given in terms of aϕ-function. Thisgeneralizationalsodependsonaparameterα∈ [−1,1]; forα=±1, it isdefined as a limit. Supposing that theunderlying ϕ-function is continuouslydifferentiable,we show that this limitexistsandresults in theϕ-divergence [12]. Inwhat follows,allprobabilitydistributionsare assumedtohavepositivedensity. Inotherwords, theybelongto thecollection Pμ= { p∈L0 : ∫ T pdμ=1and p>0 } , where L0 is the space of all real-valued, measurable functions on T, with equality μ-a.e. (μ-almosteverywhere). TheRényidivergenceoforderα∈ (−1,1)betweentwoprobabilitydistributions pandq inPμ is definedas D(α)R (p‖ q)= 4 α2−1 log (∫ T p 1−α 2 q 1+α 2 dμ ) . (4) Forα=±1, theRényidivergence isdefinedbytakinga limit: D(−1)R (p‖ q)= limα↓−1D (α) R (p‖ q), (5) D(1)R (p‖ q)= limα↑1D (α) R (p‖ q). (6) Under some conditions, the limits in (5) and (6) are finite-valued, and converge to the Kullback–Leiblerdivergence. Inotherwords, D(−1)R (p‖ q)=D(1)R (q‖ p)=DKL(p‖ q)<∞, whereDKL(p‖ q)denotes theKullback–Leiblerdivergencebetween pandq,which isgivenby DKL(p‖ q)= ∫ T p log (p q ) dμ. TheseconditionsarestatedinProposition1,givenintheendof thissection, for thecase involving thegeneralizedRényidivergence. TheRényidivergence in its standardformisgivenby D(α)(p‖ q)= 1 1−α log (∫ T pαq1−αdμ ) , forα∈ (0,1). (7) Expression(4) is relatedto this formby D(α)R (p‖ q)= 2 1−αD ((1−α)/2)(p‖ q). Beyondthechangeofvariables,whichresults inα rangingin [−1,1], expressions (4)and(7)differ bythe factor2/(1−α).Weoptedto insert the term2/(1−α) so that somekindofsymmetrycouldbe maintainedwhenthe limitsα↓−1andα↑1areconsidered. Inaddition, thegeometry inducedbythe version(4)conformswithAmari’snotation[5]. TheRényi divergenceD(α)R (· ‖ ·) canbedefined for every α ∈R. However, for α /∈ (−1,1), theexpression(4)maynotbefinite-valuedforevery pandq inPμ. Toavoidsometechnicalities,we justconsiderα∈ [−1,1]. 275