tanh Software Implementation

The hyperbolic tangent function, commonly referred to as tanh, is a mathematical function that maps real numbers to the range (−1,1). Defined as:

$\tanh(x)={\frac {e^{x}-e^{-x}}{e^{x}+e^{-x}}}$

it is a smooth, continuous, and symmetric function centered at the origin. The tanh function is widely used in mathematics, physics, and engineering, particularly in areas involving hyperbolic geometry and signal processing.

In machine learning and deep learning, tanh is a popular activation function because it introduces non-linearity and maps inputs to a centered range around zero, helping to balance gradients during optimization. Its output approaches 1 for large positive inputs and −1 for large negative inputs, with steep transitions around x=0.

Simple Approximation with Exponential Math Function

Id	Formula	Highest ULP Error	ULP <= 2.0	NaN
exp_v1	${\frac {e^{x}-e^{-x}}{e^{x}+e^{-x}}}$	x < 0: 8.55638e+08 at -2.98023e-08 with 0x0p+0 instead of -0x1p-25 x > 0: 8.55638e+08 at 2.98023e-08 with 0x0p+0 instead of 0x1p-25	-88.7246 < x < -7.14197 7.14197 < x < 88.7246	-inf < x < -88.7246 88.7246 < x < inf
exp_v2	${\frac {e^{2x}-1}{e^{2x}+1}}$	x < 0: 8.47249e+08 at -1.49012e-08 with 0x0p+0 instead of -0x1p-26 x > 0: 8.55638e+08 at 2.98023e-08 with 0x0p+0 instead of 0x1.fffffep-26	-inf < x < -0.547821 0.254822 < x < 8.31787 8.38379 < x < 44.3623	44.3623 < x < inf
exp_v3	$1-{\frac {2}{e^{2x}+1}}$	x < 0: 8.59832e+08 at -4.47035e-08 with 0x0p+0 instead of -0x1.8p-25 x > 0: 8.68221e+08 at 8.9407e-08 with 0x0p+0 instead of 0x1.7ffffcp-24	-inf < x < -6.26233 0.346382 < x < inf	-
expm1	${\frac {expm1(2x)}{expm1(2x)+2}}$	x < 0: 2.49585 at -3.95627 with -0x1.ffa00ap-1 instead of -0x1.ffa00ep-1 x > 0: 2.41657 at 0.0155837 with 0x1.fe9b66p-7 instead of 0x1.fe9b62p-7	-inf < x < -5.57227 -0.42981 < x < 0.00777948 0.125351 < x < 44.3623	44.3623 < x < inf

The first equation is the definition of the hyperbolic tangent function. This equation requires two exponentials with different values. Since exponentials are expensive to compute, we can use the second equation to reduce the number of exponentials to one.

The second equation has the drawback that both the numerator and the denominator can become infinity for large values of x which results in NaN. That happens for $|x|>\log(FLT\_MAX)/2\approx 44$.

The third equation solves this problem but has a slightly higher error for $x\approx 2^{-25}$. In the following graph you can see the error chart for equation 3. The error charts for the other two equations are very similar. All three equations have an error of up to 16 mio ULPs for $x<2^{-5}$ which means that up to 24 bits could be wrong.

Localized Approximation Functions

Id	Formula	ULP <= 2.0	NAN
signum	$\operatorname {sgn} {x}$	-inf < x < -8.31763 8.31763 < x < inf	-
poly1	$x$	-0.000704229 < x < 0.000704229	-
poly3	$x-{\frac {1}{3}}x^{3}$	-0.0291781 < x < 0.0291781	-
lampert7	${\frac {x\cdot (135135+x^{2}\cdot (17325+x^{2}\cdot (378+x^{2})))}{135135+x^{2}\cdot (62370+x^{2}\cdot (3150+28\cdot x^{2}))}}$	-0.000947416 < x < 0.000947207	-inf < x < -1.51629e+06 1.51629e+06 < x < inf
pade	$x+x\cdot {\frac {x^{2}\cdot \left(p_{1}\cdot x^{2}+p_{0}\right)}{\left(x^{2}+q_{1}\right)\cdot x^{2}+q_{0}}}$ ${\begin{aligned}p_{0}&=-0.2059432032\\p_{1}&=-0.0009577527\\q_{0}&=0.6178299136\\q_{1}&=0.25\end{aligned}}$	-0.0016973 < x < 0.0016973	-inf < x < -6.14896e+18 6.14896e+18 < x < inf

External Links

• https://yaikhom.com/2020-04-28-localised-approximation-of-hyperbolic-tangents.html#mjx-eqn-eqn7-th%20degree%20lambert%20approximant
• https://github.com/llvm/llvm-project/blob/main/libc/src/math/generic/tanhf.cpp