from _api_doc_utils import *Ridge
L2-regularized least squares with optional CV
1 Where it fits
Group: Regression
Ridge solves
\[ \min_{\alpha,\beta} \sum_i (y_i - \alpha - x_i'\beta)^2 + \lambda \|\beta\|_2^2. \]
A scalar penalty gives one ridge fit. A penalty grid with cv selects a penalty by cross-validation, stores the coefficient path, and refits on the full sample.
2 Criterion and tuning
With \(\tilde X=[\mathbf 1,X]\) and \(P_\lambda=\operatorname{diag}(0,\lambda,\ldots,\lambda)\), the fitted parameter solves
\[ \hat\theta_\lambda = \arg\min_\theta \sum_i w_i(y_i-\tilde x_i'\theta)^2+\theta'P_\lambda\theta. \]
The penalty is not divided by \(n\), so its scale depends on the sample weights and sample size. The intercept is never penalized. The implementation solves an augmented least-squares problem with \(\sqrt{\lambda}I\) rows, avoiding an explicit normal-equation inverse for the point estimate.
When given a grid, the class fits every penalty and chooses the lowest mean validation MSE. Folds are deterministic, unshuffled assignments \(i\bmod K\); weighted fits use weighted validation MSE. It then selects the corresponding full-sample path coefficient. The grid search does not standardize features, so penalty effects depend directly on column scale.
3 Inference
Let \(B=\tilde X'\tilde X+P_\lambda\) after any square-root weight transformation and define the effective degrees of freedom
\[ d_{\mathrm{eff}} = \operatorname{tr}\{\tilde X'\tilde X B^{-1}\}. \]
The model-based covariance is
\[ \widehat V_{\mathrm{vanilla}} = \hat\sigma^2B^{-1}\tilde X'\tilde X B^{-1}, \qquad \hat\sigma^2=\frac{e'e}{n-d_{\mathrm{eff}}}. \]
HC1, Newey-West, and cluster options use parameter scores \(e_i\tilde x_i'B^{-1}\) and the same finite-sample corrections as OLS, replacing \(n-p\) by \(n-d_{\mathrm{eff}}\). These are conditional linearization variances around the penalized estimator. They do not remove ridge shrinkage bias or account for selecting \(\lambda\) by cross-validation. The pairs bootstrap refits at the already selected penalty; it does not rerun the grid search.
4 Performance and numerical behavior
A single dense fit has the usual \(O(np^2+p^3)\) least-squares cost and \(O(np+p^2)\) storage. A path of \(K\) penalties repeats that work \(K\) times, and \(K\)-penalty, \(F\)-fold validation performs about \(KF\) additional fits. The implementation does not reuse decompositions across penalties. Dense covariance construction is quadratic in the number of coefficients. Ridge improves conditioning for penalized slopes, but the unpenalized intercept and badly scaled columns can still create numerical problems.
5 Python API
Constructor: cm.Ridge
The main methods mirror OLS: fit, fit_weighted, predict, summary, and bootstrap. summary() includes the selected penalty and, for grid fits, cross-validation diagnostics and coefficient paths.
print(inspect.signature(cm.Ridge))(penalty=None, cv=5)cls = cm.Ridge
display(HTML(html_table(["Public method"], public_methods(cls))))| Public method |
|---|
bootstrap(self, /, n_bootstrap, seed=None) |
fit(self, /, x, y) |
fit_weighted(self, /, x, y, sample_weight) |
predict(self, /, x) |
summary(self, /, vcov='hc1', lags=None, clusters=None) |
6 Minimal example
rng = np.random.default_rng(2)
x = rng.normal(size=(240, 5))
y = 0.3 + x @ np.array([1.0, -0.8, 0.0, 0.25, 0.1]) + rng.normal(scale=0.6, size=240)
model = cm.Ridge(penalty=np.array([0.0, 0.05, 0.2, 1.0]), cv=4)
model.fit(x, y)
print(model.summary()['penalty'])
print(model.summary()['coef'])
print(model.predict(x[:3]))1.0
[ 0.96303432 -0.80048681 -0.1007246 0.29078042 0.11061705]
[0.38294603 1.55660946 1.29480827]7 summary() contract
The table below is generated by fitting the live class in this repository and then inspecting summary(). Shapes are shown because most values are plain NumPy arrays or scalars.
rng = np.random.default_rng(102)
x = rng.normal(size=(90, 4))
y = 0.3 + x @ np.array([1, -0.5, 0.2, 0]) + rng.normal(size=90)
model = cm.Ridge(penalty=np.array([0.0, 0.1, 1.0]), cv=3)
model.fit(x, y)
summary = model.summary()
display(HTML(html_table(["summary() key", "shape"], summary_shape_rows(summary))))| summary() key | shape |
|---|---|
intercept |
() |
coef |
(4,) |
intercept_se |
() |
coef_se |
(4,) |
penalty |
() |
penalties |
(3,) |
vcov_type |
() |
best_penalty_index |
() |
cv_mse |
(3,) |
intercept_path |
(3,) |
coef_path |
(4, 3) |