Logit

Binary logistic regression

from _api_doc_utils import *

1 Where it fits

Group: Regression

Logit models a binary outcome through

\[ \Pr(Y_i=1\mid X_i=x_i)=\Lambda(\alpha+x_i'\beta), \]

with optional L2 regularization controlled by alpha. predict(x) returns probabilities for label 1; predict_label(x) returns class labels.

2 Likelihood and solver

With \(\eta_i=\alpha_0+x_i'\beta\) and \(p_i=\{1+\exp(-\eta_i)\}^{-1}\), the native fit minimizes the unnormalized penalized negative log-likelihood

\[ Q(\alpha_0,\beta) = \sum_{i=1}^n \left[ \log\{1+\exp(\eta_i)\}-y_i\eta_i \right] +\frac{\lambda}{2}\|\beta\|_2^2, \]

where the constructor argument named alpha is \(\lambda\) and the intercept is unpenalized. The implementation evaluates \(\log(1+\exp\eta)\) with a branch-stable softplus and evaluates the logistic function without overflowing for large negative indices. It supplies the analytic gradient to ten-vector-memory L-BFGS with a More-Thuente line search, starts all parameters at zero, and stops only when Argmin reports solver convergence or a target cost.

Reaching max_iterations is not convergence. fit() raises ValueError and clears any previous fitted state if the budget is exhausted or optimization otherwise fails. A successful summary() records converged, iterations, termination_reason, and the final penalized objective. The linear index and probability always refer to label 1.

3 Inference

Analytic inference is available only when \(\lambda=0\). Let \(\tilde X=[\mathbf1,X]\) and \(W=\operatorname{diag}\{\hat p_i(1-\hat p_i)\}\). The returned model-based covariance is

\[ \widehat V = (\tilde X'W\tilde X+10^{-8}I)^{-1}. \]

The \(10^{-8}\) diagonal ridge is numerical stabilization, not a reported statistical penalty. The implementation has no heteroskedastic or cluster-robust logit sandwich option. Wald tests use this Fisher covariance. The pairs bootstrap refits the same fixed penalty in each sample; a resample on which the binary fit cannot be estimated aborts the bootstrap.

4 Performance and numerical behavior

Each L-BFGS objective and gradient evaluation is \(O(np)\), with solver state linear in \(p\) for a fixed memory size. Computing the Fisher covariance forms a dense \((p+1)\times(p+1)\) information matrix in \(O(np^2)\) and inverts it in \(O(p^3)\). Complete or near separation can drive coefficients to large magnitudes and make the information matrix nearly singular; the small diagonal ridge can make inversion succeed without resolving the inferential weakness. Bootstrap multiplies the full optimization cost by the number of draws, and any nonconverged replicate aborts the bootstrap.

5 Python API

Constructor: cm.Logit

Fit with integer labels using fit(x, y_int32). Labels must be exactly 0 and 1, with both represented. summary() returns intercept and coefficient estimates plus fit diagnostics. Fisher-information standard errors are available only for the unpenalized fit (alpha=0); penalized fits are explicitly prediction-only. bootstrap() resamples observations and refits the classifier.

print(inspect.signature(cm.Logit))

(alpha=0.0, max_iterations=100, gradient_tolerance=0.0001)

cls = cm.Logit
display(HTML(html_table(["Public method"], public_methods(cls))))

Public method
bootstrap(self, /, n_bootstrap, seed=None)
fit(self, /, x, y)
predict(self, /, x)
predict_label(self, /, x, cutoff=0.5)
predict_lin(self, /, x)
summary(self, /)
wald_test(self, /, r, q=None)

6 Minimal example

rng = np.random.default_rng(5)
x = rng.normal(size=(220, 3))
eta = -0.2 + x @ np.array([0.7, -0.4, 0.9])
p = 1 / (1 + np.exp(-eta))
y = rng.binomial(1, p, size=220).astype(np.int32)
model = cm.Logit(max_iterations=200)
model.fit(x, y)
fit = model.summary()
print({key: fit[key] for key in ['converged', 'iterations', 'termination_reason', 'objective']})
print(fit['coef'])
print(model.predict(x[:5]))

{'converged': True, 'iterations': 7, 'termination_reason': 'Solver converged', 'objective': 129.69250448711378}
[ 0.56568906 -0.58550069  0.85818987]
[0.5239276  0.4143503  0.68494357 0.42404209 0.2112137 ]

7 summary() contract

The table below is generated by fitting the live class in this repository and then inspecting summary(). Shapes are shown because most values are plain NumPy arrays or scalars.

rng = np.random.default_rng(105)
x = rng.normal(size=(90, 3))
p = 1 / (1 + np.exp(-(x @ np.array([0.7, -0.4, 0.9]))))
y = rng.binomial(1, p, size=90).astype(np.int32)
model = cm.Logit(max_iterations=200)
model.fit(x, y)
summary = model.summary()
display(HTML(html_table(["summary() key", "shape"], summary_shape_rows(summary))))

summary() key	shape
intercept	()
coef	(3,)
penalty	()
inference_available	()
converged	()
iterations	()
termination_reason	()
objective	()
intercept_se	()
coef_se	(3,)
vcov	(4, 4)