Coupled entropy

Coupled entropy is a generalized measure of uncertainty that extends the classical Boltzmann–Gibbs–Shannon (BGS) entropy to systems exhibiting non-exponential probability structures. It introduces a single nonlinear statistical coupling parameter ($\kappa$) that deforms the logarithmic and exponential functions used to quantify uncertainty while preserving the classical form in the limit $\kappa \to 0$.

Coupled entropy was developed to provide a generalization flexible enough to characterize complex systems with scale-dependent or shape-dependent behavior, while maintaining a clear mathematical and physical interpretation. The framework naturally models non-exponential distributions, including both heavy-tailed and compact-support families, through a unified scale–shape representation.

The measure can be interpreted as isolating the linear component of uncertainty within a nonlinear system, though the precise relationship between total and nonlinear uncertainty remains an active area of research. By basing the generalization on an explicit coupling parameter, the formulation provides a transparent link between entropy, distribution shape, and the underlying sources of non-linearity in complex systems

Background

Boltzmann–Gibbs–Shannon (BGS) entropy is the classical measure of uncertainty in probability theory, with Boltzmann introducing the logarithmic form for equiprobable states, Gibbs extending it to general probability distributions, and Shannon establishing its role in information theory. Because it is based on the standard logarithm, BGS entropy naturally describes systems whose probability distributions belong to the exponential family.

However, many complex systems deviate from exponential behavior due to nonlinear interactions, correlations across scales, or structural constraints such as compact support or heavy tails. Earlier generalized entropy frameworks , including the Rényi, Havrda-Charvá, Sharma-Tanej-Mital, Tsallis, and Hanel-Thurner entropies, introduced deformation parameters, but the physical interpretation of these parameters and their connection to identifiable sources of nonlinearity has remained unclear in many applications .

Coupled entropy frames the generalization on the degree of nonlinear statistical coupling ($\kappa$), which quantifies both the source of nonlinearity and the tail shape of the distribution. The coupling defines the deformation of the logarithmic and exponential functions. Two important parameters retained from the exponential family are the potential of the variable ($x^{\alpha }$), which determines the shape of the distribution near the location and the dimension ($d$) of the random variable. These primary properties are related to the entropic index by the relationship $q=1+{\frac {\alpha \kappa }{1+d\kappa }}$, which from the relationship $p_{i}^{q}$ is understood to be the number of independent random variables in the same state i. This separation makes the role of nonlinearity explicit and mathematically transparent. The location parameter ($\mu$) is unchanged, though it may not be defined by the mean statistic. And the standard deviation, ($\sigma$) is generalized to a scale parameter.

Scale invariance demonstration: σ=5 remains constant while coupling κ increases fluctuations in multiplicative noise process.

This perspective is consistent with the broader interpretation of entropy as a type of statistical average analogous to how the mean, median, and mode capture different aspects of central tendency. BGS entropy corresponds to evaluating the overall average uncertainty of a distribution. For the exponential distribution, $H(X)=1+\ln(\sigma )$, and for its generalization the Pareto Type II distribution, $H(X)=1+\ln(\sigma )+\kappa$. The linear dependence on the nonlinear source of uncertainty, $\kappa$, dominates, motivating the need for complementary measures.

Derived below, the coupled entropy for the Pareto Type II or coupled exponential distribution is $H(X)=1+\ln _{\frac {\kappa }{1+\kappa }}\sigma$; that is, the coupled entropy is 1 plus a generalized logarithm of the scale. This removes the linear dependency on the nonlinear coupling, thereby providing a modified measure of the uncertainty due to the scale. While not exact, an approximate relationship is that the BGS entropy is approximately $\kappa$ plus the coupled entropy. So, the coupling is approximately the nonlinear uncertainty, and the coupled entropy is approximately the linear uncertainty.

Special Cases

Coupled entropy reduces to several well-known entropy forms:

• Shannon entropy for discrete probability distributions when $\kappa \to 0$,
• Boltzmann–Gibbs entropy for physical systems of independent, extensive components.

Mathematical formulation

The informational scale $\sigma$ is invariant to distribution shape in coupled exponentials (right), unlike in $q$-exponentials (left), where scale depends on the shape parameter.

This section summarizes the mathematical framework presented in Nelson (2025).

Coupled logarithm and exponential

The coupled entropy framework is built on deformed exponential and logarithm functions parameterized by the coupling constant ($\kappa$):

$\exp _{\kappa }(x){\stackrel {\mathrm {def} }{=}}(1+\kappa x)_{+}^{1/\kappa },$

$\ln _{\kappa }(x){\stackrel {\mathrm {def} }{=}}{\frac {1}{\kappa }}\left(x^{\kappa }-1\right),$

where $(a)_{+}=\max(0,a)$. These functions reduce to the standard exponential and natural logarithm as $\kappa \to 0$, recovering the classical Boltzmann–Gibbs–Shannon entropy framework.

For stretched distributions where the argument is $x^{\alpha }$, the framework extends to:

${\frac {1}{\alpha }}\ln _{\kappa }(x^{\alpha })=\ln _{\alpha \kappa }(x)={\frac {1}{\alpha \kappa }}\left(x^{\alpha \kappa }-1\right),$

where $\alpha$ represents the stretching exponent that influences the local distribution shape.

Discrete form

For a one dimensional probability distribution $\mathbf {p} =\{p_{1},\ldots ,p_{N}\}$, the coupled entropy is defined as:

$H_{\kappa }(\mathbf {p} ;\alpha ,d=1,\gamma )=\left[\sum _{i=1}^{N}p/_{i}^{\frac {1+(1+\alpha )\kappa }{1+\kappa }}\ln _{\alpha \kappa }\left(p_{i}^{-{\frac {1}{1+\kappa }}}\right)\right]^{\frac {1}{\gamma }}$ where the independent equals (or escort) distribution is denoted by:

$p/_{i}^{\frac {1+(1+\alpha )\kappa }{1+\kappa }}={\frac {p_{i}^{\frac {1+(1+\alpha )\kappa }{1+\kappa }}}{\sum _{j=1}^{N}p_{j}^{\frac {1+(1+\alpha )\kappa }{1+\kappa }}}}$.

Continuous form

For a continuous $d-$dimensional probability density function $f({\mathbf {x}})$, the coupled entropy using the Lebesque-Stieltjes:

$H_{\kappa }({\mathbf {X}};\alpha ,d,\gamma )=\left[\int _{\mathcal {X}}f/^{\frac {1+(d+\alpha )\kappa }{1+d\kappa }}({\mathbf {x}})\ln _{\alpha \kappa }\left(f({\mathbf {x}})^{-{\frac {1}{1+d\kappa }}}\right)dF({\mathbf {x}})\right]^{\frac {1}{\gamma }}$ where the escort density is:

$f/^{\frac {1+(d+\alpha )\kappa }{1+d\kappa }}({\mathbf {x}})={\frac {f(x)^{\frac {1+(d+\alpha )\kappa }{1+d\kappa }}}{\int _{\mathcal {X}}f({\mathbf {x}})^{\frac {1+(d+\alpha )\kappa }{1+d\kappa }}dF({\mathbf {x}})}}$

Parameter roles

$\kappa$: nonlinear statistical coupling parameter quantifying both the source of nonlinearity and the tail shape

$\alpha$: stretching exponent influencing distribution shape near the location

$d$: system dimension

$\gamma$: non-trace power factor for composability and extensivity

Relationship to other entropy measures

Figure 3. Comparison of generalized entropies shows coupled entropy (green) balances between Tsallis (blue, too cold) and normalized Tsallis (red, too hot).

Coupled entropy can be compared with several established generalized entropy measures, particularly in terms of how each framework incorporates deformation parameters, nonlinearity, or deviations from classical Boltzmann–Gibbs statistics. It provides a unifying perspective for examining systems with heavy tails, nonlinear interactions, or scale–dependent effects.

Comparison with Tsallis entropy

Although the coupled and Tsallis entropies are related via the independent equals probability $p^{q}$, where $q=1+{\frac {\alpha \kappa }{1+d\kappa }}$, the Tsallis entropy has a different normalization, which impacts its ability to be a measure of uncertainty. The Tsallis entropy converges to 1, that is, it is "too cold", for the coupled exponential distribution as the coupling goes to infinity. The normalized Tsallis entropy was an attempt to improve the structure of the Tsallis entropy but is unstable and goes to infinity as the coupling goes to infinity for the coupled exponential distribution. These properties are shown in Figure 3.

Shannon entropy

In the limit $\kappa \to 0$, the coupled entropy converges to the Boltzmann–Gibbs–Shannon (BGS) entropy for any value of the stretching exponent $\alpha$. That is, the coupling term not the stretching exponent determines departures from classical extensive statistics. Explicitly, in this limit:

$S_{\kappa \to 0}(\mathbf {p} )\to -\sum _{i}p_{i}\ln p_{i},$

recovering conventional Shannon entropy.

Non-exponential distributions

Coupled entropy is naturally associated with non-exponential probability distributions, a class that includes both heavy-tailed and compact-support families. Examples include:

• Student’s t distributions,
• Pareto-type power laws,
• stretched exponential distributions,
• coupled-exponential distributions,
• compact-support models used in machine learning for pruning or sparsity.

This viewpoint broadens the traditional distinction between “exponential” and “heavy-tailed” systems by recognizing that many models of interest in complex systems exhibit nonlinear scaling behavior, whether through long tails or finite support.

One-Dimensional Coupled Exponential Family

The one-dimensional coupled exponential family (CEF) generalizes the classical exponential family to account for heavy-tailed and compact-support distributions via a nonlinear coupling parameter $\kappa$.

For a random variable $X$, the one-dimensional coupled exponential PDF is defined as:

$f_{\kappa }(x;\theta ,\alpha )={\frac {h(x)}{Z_{\kappa }(\theta ,\alpha )}}\exp _{\alpha \kappa }^{-(1+\kappa )}{\big [}\eta (\theta )\cdot T(x){\big ]},$

where $\alpha$ is the exponent of the variable function, $h(x)$ is the base measure, and $Z_{\kappa }$ is the generalized partition function.

Coupled Stretched Exponential Distribution

When $h(x)=1$ in the one-dimensional coupled exponential family, the resulting distribution is called the coupled stretched exponential. The one dimensional PDF and survival function are:

$p_{\kappa }(x)={\frac {1}{\sigma }}\left(1+\kappa \left({\frac {x-\mu }{\sigma }}\right)^{\alpha }\right)^{-{\frac {1+\kappa }{\alpha \kappa }}},$

$S_{\kappa }(x)=\left(1+\kappa \left({\frac {x-\mu }{\sigma }}\right)^{\alpha }\right)^{-{\frac {1}{\alpha \kappa }}}.$

Parameters:

• $\mu$: location
• $\sigma$: scale
• $\kappa$: nonlinear coupling parameter
• $\alpha$: stretch exponent

Special Cases

• $\alpha =1$: Coupled Exponential Distribution (generalized Pareto)

• $\alpha =2$: Coupled Gaussian Distribution (Student’s t)

Parameter Interpretation

• $\kappa$ controls tail heaviness (exponential, heavy-tailed, or compact-support).
• $\alpha$ controls the stretch or curvature near the location.

Dynamical Origin

Coupled stretched exponentials arise as stationary solutions of stochastic differential equations involving both additive and multiplicative noise, such as

$dX_{t}=f(X_{t})\,dt+A\,dW_{t}^{(a)}+M\,g(X_{t})\circ dW_{t}^{(m)}.$

Under appropriate conditions on $g(x)$, the stationary solution takes the form

$p(x)\propto \exp _{\kappa }^{-(1+\kappa )/2}\!\left({\frac {g(x)^{2}}{\sigma ^{2}}}\right),$

where the coupling parameter is

$\kappa ={\frac {M^{2}}{2\tau }},$

and the scale is

$\sigma ^{2}={\frac {A^{2}}{2\tau }}.$

This provides a physical interpretation of $\kappa$ as the relative strength of multiplicative noise, and of $\alpha$ through the growth of $g(x)$.

Relation to Coupled Entropy

These distributions maximize the coupled entropy under independent-equals-moment constraints.

Historical Context and Foundational Theory

The mathematical foundation for the coupled exponential family was formalized in the paper .

In this work, the coupled-product ($\otimes _{\kappa }$) was introduced as a nonlinear generalization of multiplication, enabling the construction of multivariate coupled-exponential functions. This operation provides the algebraic basis for defining probability distributions whose components interact through nonlinear statistical coupling.

The paper also showed how the coupled-product generates multivariate generalizations of the $\kappa$-exponential and how parameters such as scale and coupling separate naturally within this framework. While the stretched-exponential case ($\alpha \neq 2$) was not derived in that paper, the coupled-product formalism laid the structural foundation for later generalizations, including the coupled stretched exponential.

Applications

Used in statistical modeling of heavy-tailed processes, turbulence, and robust machine learning.

Properties

Figure 4. Coupled entropy uniquely aligns with scale parameter σ across different couplings, unlike other generalized entropies

Fundamental properties

• Reduces to the Boltzmann-Gibbs-Shannon entropy when the coupling is zero.
• Sensitive to nonlinear coupling between microstates.
• Provides a generalized measure of uncertainty applicable to heavy-tailed systems.
• The asymptotic scaling can be adjusted to fulfill requirements such as extensivity.
• Admits generalized additivity laws via the coupled logarithm.

Scale alignment

As visualized, coupled entropy uniquely maintains alignment with the distribution's scale parameter σ across different coupling values, unlike other entropy measures which deviate from the characteristic scale.

Applications

Coupled entropy has been applied in:

• Statistical mechanics of complex and non-extensive systems,
• Information theory with nonlinear dependencies,
• Thermodynamics of coupled or interacting subsystems,
• Modeling heavy-tailed noise and uncertainty,
• Signal processing and robust inference,
• Complex networks and interacting particle systems.

Recent or advanced applications

Recent studies have applied the coupled entropy framework to advanced statistical and computational methods. For instance, Nelson, Oliveira, and Al‑Najafi (2025) utilized the curved geometry of coupled free energy in variational inference, demonstrating how coupled exponential distributions can enhance inference for heavy-tailed and complex systems.

See also

• Shannon entropy
• Tsallis entropy
• Rényi entropy
• statistical mechanics
• Heavy-tailed distribution
• Generalized mean