tanh Software Implementation

The hyperbolic tangent function, commonly referred to as tanh, is a mathematical function that maps real numbers to the range $(-1,1)$ . It is defined as:

\tanh(x)={\frac {e^{x}-e^{-x}}{e^{x}+e^{-x}}}

From a software implementation perspective, tanh is a numerically challenging function due to the use of exponential functions, which can easily overflow for large inputs and suffer from catastrophic cancellation for very small values of $x$ . These issues are particularly relevant in single-precision floating-point (float32) implementations, where both accuracy and performance are critical.

In machine learning and deep learning, tanh is widely used as an activation function because it introduces non-linearity and produces outputs centered around zero, which can improve gradient-based optimization. As a result, tanh is often evaluated billions of times during training and inference, making efficient and numerically stable implementations essential.

This article focuses on practical software implementations of tanh, analyzing different computational formulations with respect to floating-point accuracy, ULP error, overflow behavior, and NaN generation. The goal is to provide a foundation for designing robust tanh implementations suitable for performance-critical environments such as numerical libraries, embedded systems, and AI workloads.

Problem Overview

The hyperbolic tangent function poses several challenges when implemented in finite-precision floating-point arithmetic. While the mathematical definition is simple, direct translations into software often suffer from numerical instability and performance issues.

A primary challenge is the use of exponential functions. For large input values, exp(x) and exp(-x) can overflow in single-precision floating-point arithmetic, leading to infinities and, in some formulations, NaN results. Avoiding overflow therefore requires careful reformulation or explicit range handling.

For very small input values, catastrophic cancellation becomes the dominant issue. Expressions such as $e^{x}-e^{-x}$ involve the subtraction of nearly equal numbers, causing a severe loss of significant bits and large ULP errors. In these regions, naive implementations may return zero or highly inaccurate results even though $\tanh(x)\approx x$ .

In addition to accuracy concerns, performance is a critical factor. In many applications, especially machine learning and numerical simulations, tanh is evaluated extremely frequently. This makes it necessary to balance numerical robustness with the cost of transcendental function calls, branch complexity, and instruction-level efficiency.

Direct Exponential Formulations


Id	Formula	Highest ULP Error	ULP <= 2.0	NaN
exp_v1	${\frac {e^{x}-e^{-x}}{e^{x}+e^{-x}}}$	x < 0: 8.55638e+08 at -2.98023e-08 with 0x0p+0 instead of -0x1p-25 x > 0: 8.55638e+08 at 2.98023e-08 with 0x0p+0 instead of 0x1p-25	-88.7246 < x < -7.14197 7.14197 < x < 88.7246	-inf < x < -88.7246 88.7246 < x < inf
exp_v2	${\frac {e^{2x}-1}{e^{2x}+1}}$	x < 0: 8.47249e+08 at -1.49012e-08 with 0x0p+0 instead of -0x1p-26 x > 0: 8.55638e+08 at 2.98023e-08 with 0x0p+0 instead of 0x1.fffffep-26	-inf < x < -0.547821 0.254822 < x < 8.31787 8.38379 < x < 44.3623	44.3623 < x < inf
exp_v3	$1-{\frac {2}{e^{2x}+1}}$	x < 0: 8.59832e+08 at -4.47035e-08 with 0x0p+0 instead of -0x1.8p-25 x > 0: 8.68221e+08 at 8.9407e-08 with 0x0p+0 instead of 0x1.7ffffcp-24	-inf < x < -6.26233 0.346382 < x < inf	-
expm1	${\frac {expm1(2x)}{expm1(2x)+2}}$	x < 0: 2.49585 at -3.95627 with -0x1.ffa00ap-1 instead of -0x1.ffa00ep-1 x > 0: 2.41657 at 0.0155837 with 0x1.fe9b66p-7 instead of 0x1.fe9b62p-7	-inf < x < -5.57227 -0.42981 < x < 0.00777948 0.125351 < x < 44.3623	44.3623 < x < inf

The first equation is the definition of the hyperbolic tangent function. This equation requires two exponentials with different values. Since exponentials are expensive to compute, we can use the second equation to reduce the number of exponentials to one.

The second equation has the drawback that both the numerator and the denominator can become infinity for large values of x which results in NaN. That happens for $|x|>\log(FLT\_MAX)/2\approx 44$ .

The third equation solves this problem but has a slightly higher error for $x\approx 2^{-25}$ . In the following graph you can see the error chart for equation 3. The error charts for the other two equations are very similar. All three equations have an error of up to 16 mio ULPs for $x<2^{-5}$ which means that up to 24 bits could be wrong.

exp_v2

The following error graph shows the ULP error of the exp_v2 formulation across the full float32 input range. This formulation reduces the number of exponential evaluations to one, but is particularly sensitive to catastrophic cancellation for very small input values and to overflow for large $|x|$ .

Key observation: For $|x|\ll 1$ , the ULP error grows rapidly and reaches extremely large values due to loss of significance. For sufficiently large $x$ , both numerator and denominator overflow, leading to NaN results.

expm1

This graph visualizes the ULP error of the expm1-based formulation. Using expm1 significantly improves numerical accuracy for small input values by avoiding catastrophic cancellation in the expression $e^{2x}-1$ .

Key observation: Compared to direct exponential formulations, the expm1 approach maintains low ULP error for small and medium values of $x$ . Overflow is still present for very large inputs, but the overall error behavior is substantially more stable.

Numerical Issues and Error Metrics

To evaluate and compare different software implementations of tanh, several numerical error metrics are used throughout this article. These metrics make it possible to reason about accuracy, robustness, and failure modes in floating-point arithmetic.

ULP Error

ULP (Unit in the Last Place) measures the distance between two adjacent representable floating-point numbers. The ULP error of a computed result is defined as the difference, measured in ULPs, between the computed value and the correctly rounded reference value.

An error of ULP ≤ 2 is commonly considered acceptable for single-precision transcendental functions, as it indicates that the result is very close to the correctly rounded value.

Large ULP errors typically indicate catastrophic cancellation, loss of significant bits, or severe rounding effects.

NaN and Overflow

NaN (Not a Number) results occur when undefined floating-point operations are performed, such as inf / inf. In tanh implementations, this often happens when both the numerator and denominator of a formulation overflow to infinity.

Tracking NaN generation is critical, as NaNs can silently propagate through numerical pipelines and invalidate downstream results.

Error Graphs

The error graphs shown in the following sections visualize ULP error across the full float32 input domain. They make it possible to identify:

regions with catastrophic cancellation,
input ranges that trigger overflow or NaNs,
and numerical stability differences between formulations.

These graphs are a key tool for understanding not only the maximum error, but also how errors are distributed across different input magnitudes.

Taylor Expansion (Local Approximation)

Taylor series approximations of tanh(x) expand the function around $x=0$ and therefore provide a very accurate approximation only for sufficiently small input magnitudes.

For small $|x|$ , the hyperbolic tangent behaves approximately linear ( $\tanh(x)\approx x$ ), and low-order Taylor polynomials can achieve excellent accuracy with minimal computational cost. This makes Taylor-based approximations attractive as a fast path in range-reduced implementations.

However, Taylor expansions have a strictly limited radius of convergence. Outside a narrow neighborhood around zero, the approximation error grows extremely rapidly and quickly becomes unusable. As a result, Taylor polynomials are not suitable as standalone implementations of tanh.

Key observation: The Taylor approximation is highly accurate near zero but diverges extremely fast outside its convergence region. This makes it unsuitable as a standalone tanh implementation, but very effective as a local approximation when combined with range-based dispatching.

Localized Approximation Functions

Localized approximation functions are designed to provide highly accurate and computationally efficient evaluations of tanh(x) within a restricted input range. They are typically used as building blocks in range-based dispatch strategies, where different approximations are selected depending on the magnitude of the input value.

By limiting the valid input range, these approximations can achieve very low ULP error with simple polynomial or rational expressions, avoiding expensive transcendental functions.

Id	Formula	ULP <= 2.0	NaN
signum	$\operatorname {sgn} {x}$	-inf < x < -8.31763 8.31763 < x < inf	-
poly1	$x$	-0.000704229 < x < 0.000704229	-
poly3	$x-{\frac {1}{3}}x^{3}$	-0.0291781 < x < 0.0291781	-
lambert7	${\frac {x\cdot (135135+x^{2}\cdot (17325+x^{2}\cdot (378+x^{2})))}{135135+x^{2}\cdot (62370+x^{2}\cdot (3150+28\cdot x^{2}))}}$	-0.000947416 < x < 0.000947207	-inf < x < -1.51629e+06 1.51629e+06 < x < inf
pade	$x+x\cdot {\frac {x^{2}\cdot \left(p_{1}\cdot x^{2}+p_{0}\right)}{\left(x^{2}+q_{1}\right)\cdot x^{2}+q_{0}}}$ ${\begin{aligned}p_{0}&=-0.2059432032\\p_{1}&=-0.0009577527\\q_{0}&=0.6178299136\\q_{1}&=0.25\end{aligned}}$	-0.0016973 < x < 0.0016973	-inf < x < -6.14896e+18 6.14896e+18 < x < inf

The functions listed above are not intended to be used independently over the full input domain. Instead, they are commonly combined with explicit range checks and sign handling to form a complete tanh implementation.

For example, low-order polynomials are well suited for very small $|x|$ , while rational or higher-order approximations extend the usable range before switching to exponential-based formulations or saturation.

Recommended Implementation Strategy

A robust and high-performance tanh(x) implementation typically combines multiple techniques, each optimized for a specific input range. No single formulation provides optimal accuracy and performance across the entire floating-point domain.

A common strategy is based on range-dependent dispatching:

Sign handling

Use the odd symmetry of tanh (

\tanh(-x)=-\tanh(x)

) to reduce the implementation to non-negative inputs.*Very small inputs ( $|x|\ll 1$ )

Use a low-order polynomial or Taylor approximation, exploiting

\tanh(x)\approx x

to avoid cancellation and expensive transcendental functions.

Small to medium inputs

Apply a localized polynomial or rational approximation with bounded ULP error.

Medium to large inputs

Use an expm1-based formulation to maintain numerical stability while avoiding catastrophic cancellation.

Very large inputs

Directly return ±1, as tanh(x) saturates rapidly and additional computation provides no meaningful benefit.

This hybrid approach minimizes ULP error, avoids NaN generation, and allows fine-grained control over the accuracy–performance trade-off. It is widely used in optimized math libraries and hardware-accelerated implementations.

External Links

tanh Software Implementation

Contents

Problem Overview