LogSumExp

 ( Logarithms ) 

The LogSumExp (LSE) (also called RealSoftMax or multivariable softplus) function is a smooth maximum – a smooth function approximation to the maximum function, mainly used by machine learning algorithms. It is defined as the logarithm of the sum of the exponentials of the arguments:

$$\backslash mathrm\{LSE\}(x\_1,\; \backslash dots,\; x\_n)\; =\; \backslash log\backslash left(\; \backslash exp(x\_1)\; +\; \backslash cdots\; +\; \backslash exp(x\_n)\; \backslash right).$$

---

Properties The LogSumExp function domain is $\backslash R^n$, the real coordinate space, and its codomain is $\backslash R$, the real line. It is an approximation to the maximum $\backslash max\_i\; x\_i$ with the following bounds $$\backslash max\{\backslash \{x\_1,\; \backslash dots,\; x\_n\backslash \}\}\; \backslash leq\; \backslash mathrm\{LSE\}(x\_1,\; \backslash dots,\; x\_n)\; \backslash leq\; \backslash max\{\backslash \{x\_1,\; \backslash dots,\; x\_n\backslash \}\}\; +\; \backslash log(n).$$ The first inequality is strict unless $n\; =\; 1$. The second inequality is strict unless all arguments are equal. (Proof: Let $m\; =\; \backslash max\_i\; x\_i$. Then $\backslash exp(m)\; \backslash leq\; \backslash sum\_\{i=1\}^n\; \backslash exp(x\_i)\; \backslash leq\; n\; \backslash exp(m)$. Applying the logarithm to the inequality gives the result.)

In addition, we can scale the function to make the bounds tighter. Consider the function $\backslash frac\; 1\; t\; \backslash mathrm\{LSE\}(tx\_1,\; \backslash dots,\; tx\_n)$. Then (Proof: Replace each $x\_i$ with $tx\_i$ for some $t>0$ in the inequalities above, to give and, since $t>0$ finally, dividing by $t$ gives the result.)

Also, if we multiply by a negative number instead, we of course find a comparison to the $\backslash min$ function:

The LogSumExp function is convex function, and is strictly increasing everywhere in its domain. It is not strictly convex, since it is affine function (linear plus a constant) on the diagonal and parallel lines:

$\backslash mathrm\{LSE\}(x\_1\; +\; c,\; \backslash dots,\; x\_n\; +\; c)\; =\backslash mathrm\{LSE\}(x\_1,\; \backslash dots,\; x\_n)\; +\; c.$

Other than this direction, it is strictly convex (the Hessian matrix has rank ), so for example restricting to a hyperplane that is transverse to the diagonal results in a strictly convex function. See $\backslash mathrm\{LSE\}\_0^+$, below.

Writing $\backslash mathbf\{x\}\; =\; (x\_1,\; \backslash dots,\; x\_n),$ the partial derivatives are: $$\backslash frac\{\backslash partial\}\{\backslash partial\; x\_i\}\{\backslash mathrm\{LSE\}(\backslash mathbf\{x\})\}\; =\; \backslash frac\{\backslash exp\; x\_i\}\{\backslash sum\_j\; \backslash exp\; \{x\_j\}\},$$ which means the gradient of LogSumExp is the softmax function.

The convex conjugate of LogSumExp is the negative entropy.

---

log-sum-exp trick for log-domain calculations The LSE function is often encountered when the usual arithmetic computations are performed on a logarithmic scale, as in log probability.

Similar to multiplication operations in linear-scale becoming simple additions in log-scale, an addition operation in linear-scale becomes the LSE in log-scale:

$$\backslash mathrm\{LSE\}(\backslash log(x\_1),\; ...,\; \backslash log(x\_n))\; =\; \backslash log(x\_1\; +\; \backslash dots\; +\; x\_n)$$ A common purpose of using log-domain computations is to increase accuracy and avoid underflow and overflow problems when very small or very large numbers are represented directly (i.e. in a linear domain) using limited-precision floating point numbers.

Unfortunately, the use of LSE directly in this case can again cause overflow/underflow problems. Therefore, the following equivalent must be used instead (especially when the accuracy of the above 'max' approximation is not sufficient).

$$\backslash mathrm\{LSE\}(x\_1,\; \backslash dots,\; x\_n)\; =\; x^*\; +\; \backslash log\backslash left(\; \backslash exp(x\_1-x^*)+\; \backslash cdots\; +\; \backslash exp(x\_n-x^*)\; \backslash right)$$ where $x^*\; =\; \backslash max\{\backslash \{x\_1,\; \backslash dots,\; x\_n\backslash \}\}$

Many math libraries such as IT++ provide a default routine of LSE and use this formula internally.

---

A strictly convex log-sum-exp type function LSE is convex but not strictly convex. We can define a strictly convex log-sum-exp type function by adding an extra argument set to zero:

$$\backslash mathrm\{LSE\}\_0^+(x\_1,...,x\_n)\; =\; \backslash mathrm\{LSE\}(0,x\_1,...,x\_n)$$ This function is a proper Bregman generator (strictly convex and differentiable). It is encountered in machine learning, for example, as the cumulant of the multinomial/binomial family.

In tropical analysis, this is the sum in the log semiring.

---

See also

• Logarithmic mean
• Log semiring
• Smooth maximum
• Softmax function

Post Comment