from _api_doc_utils import *Optimizers
Static optimization routines for Python callbacks
1 Where it fits
Group: Estimation interfaces
Optimizers is a small namespace for callback-driven numerical optimization. It is not an estimator; it exposes reusable routines for smooth objectives, nonlinear least squares, and a simple stochastic global search.
2 Solver mechanics
All methods bridge Python callbacks into Rust and normalize the result into the same dictionary, but their step construction and success rules differ.
2.1 BFGS and L-BFGS
Both call fun(theta) and grad(theta) separately, use a More-Thuente line search, and apply tolerance to gradient and cost stopping. BFGS initializes a dense inverse-Hessian approximation at identity and updates the full matrix. L-BFGS retains seven correction pairs instead. Both return Argmin’s best parameter and classify success only from its termination status; reaching the iteration budget returns a dictionary with success=False rather than raising.
2.2 Nonlinear conjugate gradient
This method uses Polak-Ribiere-plus directions and an Armijo backtracking line search with sufficient-decrease constant \(0.2\). It supports periodic and orthogonality restarts. Because the underlying solver does not use the public tolerance directly, the wrapper reevaluates the final Python gradient and upgrades the result to success when its Euclidean norm is at most tolerance.
2.3 Gauss-Newton least squares
The package owns this loop. At each iterate it calls the residual and Jacobian functions, requires Jacobian shape (n_residuals, n_parameters), forms \(g=J'r\) and \(J'J\), explicitly inverts \(J'J\), and proposes \((J'J)^{-1}g\). Gradient or step norm below tolerance stops. Otherwise a halving line search requires strict decrease in \(\|r\|^2/2\) down to scale \(10^{-8}\). A small objective change succeeds; a singular normal matrix, callback error, or malformed shape raises, while line-search or iteration exhaustion returns success=False.
2.4 Simulated annealing
The proposal modifies at least one randomly selected coordinate, adds a uniform perturbation in [-step_size, step_size], and clamps it to optional elementwise bounds. Proposal and acceptance RNGs use separate Xoshiro streams; with a seed the second stream uses seed + 1. Argmin’s Boltzmann temperature schedule runs until its stopping status or 1,000 consecutive stalls in best or accepted states. This is the only method that can use OS entropy when no seed is supplied.
3 Performance and failure behavior
Every objective, gradient, residual, or Jacobian evaluation crosses the Python-Rust boundary and copies NumPy results. BFGS stores \(O(p^2)\) curvature state, L-BFGS and nonlinear CG use \(O(p)\) state aside from callback data, and native Gauss-Newton forms and inverts a dense \(p\times p\) normal matrix. The wrappers return the best parameter, then reevaluate the public objective to populate fun; a callback that is stochastic or stateful can therefore report a value different from the one attached to the optimizer’s best state.
4 Python API
Constructor: cm.Optimizers
The methods are static and return plain dictionaries with scipy-like keys: x, fun, nit, success, message, and method. Smooth minimizers require objective and gradient callbacks; Gauss-Newton requires residual and Jacobian callbacks; simulated annealing only requires the objective.
print(inspect.signature(cm.Optimizers))()cls = cm.Optimizers
display(HTML(html_table(["Public method"], public_methods(cls))))| Public method |
|---|
minimize_bfgs(fun, x0, grad, max_iterations=100, tolerance=1e-06) |
minimize_gauss_newton_ls(residual_fn, x0, jacobian_fn, max_iterations=100, tolerance=1e-06) |
minimize_lbfgs(fun, x0, grad, max_iterations=100, tolerance=1e-06) |
minimize_nonlinear_cg(fun, x0, grad, max_iterations=100, restart_iters=10, restart_orthogonality=0.1, tolerance=1e-06) |
minimize_simulated_annealing(fun, x0, lower=None, upper=None, temp=15.0, step_size=0.1, max_iterations=5000, seed=None) |
5 Minimal example
def fun(theta):
return float(np.sum((theta - np.array([1.0, -2.0])) ** 2))
def grad(theta):
return 2.0 * (theta - np.array([1.0, -2.0]))
res = cm.Optimizers.minimize_bfgs(fun, np.zeros(2), grad, max_iterations=100)
print(res){'x': array([ 1., -2.]), 'fun': 0.0, 'nit': 1, 'success': True, 'message': 'Solver converged', 'method': 'bfgs'}6 summary() contract
The table below is generated by fitting the live class in this repository and then inspecting summary(). Shapes are shown because most values are plain NumPy arrays or scalars.
class _Dummy:
def summary(self):
return {'note': 'Optimizers has static methods, not fitted state'}
model = _Dummy()
summary = model.summary()
display(HTML(html_table(["summary() key", "shape"], summary_shape_rows(summary))))| summary() key | shape |
|---|---|
note |
() |