NeurIPS 2026

Bayesian Optimization with Fisher Information Geometry

Gradient Bounds and Trust-Region Methods

Saksham Kiroriwal · Julius Pfrommer · Jürgen Beyerer

Fraunhofer IOSB · Karlsruhe Institute of Technology

The surrogate maps the input space to the statistical manifold of predictive distributions. The acquisition function depends on the predictive parameters rather than the input directly.

Abstract

We study Bayesian optimization (BO) through the lens of information geometry. Pulling back the Fisher information metric through the surrogate posterior map yields a local sensitivity tensor on the input space, which leads to an upper bound on the gradient of reparameterizable acquisition functions (AFs). This view explains vanishing-gradient behavior in high-dimensional BO and provides a common interpretation of heuristics such as RAASP and dimension-scaled lengthscales. The upper bound can also serve as a generic recipe to understand the behavior of new acquisition functions or surrogates in high input dimensions as well. Using the bounds, we also identify necessary conditions for an input location to be an optimizer of the AF for a univariate Gaussian predictive posterior. Building on this analysis, we propose FITR, a trust-region-based BO method that replaces lengthscale-based scaling with local pullback-Fisher weights. FITR is not restricted to GP kernels with explicit lengthscales and easily generalizes to non-isotropic surrogates. FITR shows competitive performance on several benchmarks.

Manifold view of Bayesian Optimization

The surrogate induces geometry on the input space through the posterior map.

Acquisition function gradient upper bound

Using the manifold view of Bayesian optimization, we prove an upper bound for the acquisition function gradient that helps us understand the behaviour of Bayesian optimization in high dimensions.

Conventional BO

Model predictionMean and uncertainty over the inputA

MeanObservations

Acquisition functionThe utility assigned to each predictionB

Acquisition

Visualization of manifold view and upper bound

Manifold viewThe same inputs, in predictive coordinatesC

Posterior map

Gradient boundSquared gradient and its upper boundD

Animation controls

Input

Acquisition SE lengthscale 0.10

The moving point follows one input through all four views. Away from the observations, the GP returns toward its prior: distant inputs map to similar predictive states.

Notation and geometry

A common view of high-dimensional BO

Why RAASP works and √D scaling is the optimal scaling

Keep candidates near observationsFixed lengthscale; global versus local candidatesA

GlobalRAASP-style local

Background and references

Papenmeier et al. (2025) analyze why RAASP helps acquisition optimization, while Hvarfner et al. (2024) show the importance of dimension-scaled lengthscale priors. Our manifold view gives a common theoretical explanation for both.

Scale lengthscales with dimensionSame global candidates; different scaling exponentsB

Fixed lengthscale

Acquisition function optimization

Characteristics of stationary points for acquisition function optimization

Characteristics of mean and variance gradients

The orange marker represents a numerically zero acquisition gradient at stationary input location. Both plots use the same GP and acquisition function.

Mathematical explanation

The acquisition function is generally non-convex. A stationary point like saddle point, maxima or minima (local or global) have zero gradient . Using the manifold view we show novel properties for those stationary points that can be used by future optimization methods for better acquistion function optimization.

Using the induced geometry as a trust region

FITR uses the pullback tensor to enhance TuRBO.

Our manifold view extends TuRBO (Eriksson et al. (2019)) to differentiable surrogates without explicit lengthscales. FITR uses the local pullback tensor to shape the region instead of kernel lengthscales, while retaining TuRBO's size adaptation and restarts. We test it using GPs with SE and IBNN kernels.

Trust-region shapeGP with SE kernel: TuRBO versus FITR

Animation controls

Illustration

Pause and click the plot to inspect a location. The IBNN kernel has no coordinate lengthscales for a TuRBO reference box.