Ridge

L2-regularized least squares with optional CV

from _api_doc_utils import *

1 Where it fits

Group: Regression

Ridge solves

\[ \min_{\alpha,\beta} \sum_i (y_i - \alpha - x_i'\beta)^2 + \lambda \|\beta\|_2^2. \]

A scalar penalty gives one ridge fit. A penalty grid with cv selects a penalty by cross-validation, stores the coefficient path, and refits on the full sample.

2 Criterion and tuning

With \(\tilde X=[\mathbf 1,X]\) and \(P_\lambda=\operatorname{diag}(0,\lambda,\ldots,\lambda)\), the fitted parameter solves

\[ \hat\theta_\lambda = \arg\min_\theta \sum_i w_i(y_i-\tilde x_i'\theta)^2+\theta'P_\lambda\theta. \]

The penalty is not divided by \(n\), so its scale depends on the sample weights and sample size. The intercept is never penalized. The implementation solves an augmented least-squares problem with \(\sqrt{\lambda}I\) rows, avoiding an explicit normal-equation inverse for the point estimate.

When given a grid, the class fits every penalty and chooses the lowest mean validation MSE. Folds are deterministic, unshuffled assignments \(i\bmod K\); weighted fits use weighted validation MSE. It then selects the corresponding full-sample path coefficient. The grid search does not standardize features, so penalty effects depend directly on column scale.

3 Implementation walkthrough

The point estimate is implemented as augmented least squares rather than through \((X'X+P_\lambda)^{-1}X'y\).

1. The wrapper validates the penalty array and data, prepends a column of ones, and, for a weighted fit, multiplies each design row and outcome by \(\sqrt{w_i}\). Zero weights therefore contribute zero rows to the numerical problem.
2. For \(\lambda>0\), it appends one synthetic row for each penalized coefficient. The slope block of those rows is \(\sqrt{\lambda}I\) and their synthetic outcomes are zero; the intercept column is omitted from this block. Ordinary QR least squares on the augmented system exactly minimizes the stated ridge criterion. At \(\lambda=0\), it solves the original design directly.
3. A penalty path is deliberately simple: it repeats that complete augmented-QR solve for every grid value and stores every intercept and slope vector. There is no shared eigendecomposition, warm start, or recursive update between penalties.
4. Cross-validation assigns row \(i\) to fold \(i\bmod K\). For each penalty and fold it constructs owned training and test arrays, refits from scratch, predicts the held-out rows, and records ordinary or weight-normalized MSE. The first minimum in grid order wins ties. The already-computed full-sample path column for that penalty becomes the fitted model.
5. Prediction is a dense \(X\beta\) plus intercept. The training design, response, optional weights, selected penalty, and complete path are retained because summary(), robust covariance calculations, and the pairs bootstrap need them.

This QR construction is more numerically defensible than explicitly solving the normal equations for coefficients, but the inferential code still forms dense cross-products and inverses. The deterministic interleaved folds make a run reproducible without a seed; they are inappropriate when row order itself encodes time, clusters, or another dependence structure.

4 Inference

Let \(B=\tilde X'\tilde X+P_\lambda\) after any square-root weight transformation and define the effective degrees of freedom

\[ d_{\mathrm{eff}} = \operatorname{tr}\{\tilde X'\tilde X B^{-1}\}. \]

The model-based covariance is

\[ \widehat V_{\mathrm{vanilla}} = \hat\sigma^2B^{-1}\tilde X'\tilde X B^{-1}, \qquad \hat\sigma^2=\frac{e'e}{n-d_{\mathrm{eff}}}. \]

HC1, Newey-West, and cluster options use parameter scores \(e_i\tilde x_i'B^{-1}\) and the same finite-sample corrections as OLS, replacing \(n-p\) by \(n-d_{\mathrm{eff}}\). These are conditional linearization variances around the penalized estimator. They do not remove ridge shrinkage bias or account for selecting \(\lambda\) by cross-validation. The pairs bootstrap refits at the already selected penalty; it does not rerun the grid search.

5 Performance and numerical behavior

A single dense fit has the usual \(O(np^2+p^3)\) least-squares cost and \(O(np+p^2)\) storage. A path of \(K\) penalties repeats that work \(K\) times, and \(K\)-penalty, \(F\)-fold validation performs about \(KF\) additional fits. The implementation does not reuse decompositions across penalties. Dense covariance construction is quadratic in the number of coefficients. Ridge improves conditioning for penalized slopes, but the unpenalized intercept and badly scaled columns can still create numerical problems.

6 Python API

Constructor: cm.Ridge

The main methods mirror OLS: fit, fit_weighted, predict, summary, and bootstrap. summary() includes the selected penalty and, for grid fits, cross-validation diagnostics and coefficient paths.

print(inspect.signature(cm.Ridge))

(penalty=None, cv=5)

cls = cm.Ridge
display(HTML(html_table(["Public method"], public_methods(cls))))

Public method
bootstrap(self, /, n_bootstrap, seed=None)
fit(self, /, x, y)
fit_weighted(self, /, x, y, sample_weight)
predict(self, /, x)
summary(self, /, vcov='hc1', lags=None, clusters=None)

7 Minimal example

rng = np.random.default_rng(2)
x = rng.normal(size=(240, 5))
y = 0.3 + x @ np.array([1.0, -0.8, 0.0, 0.25, 0.1]) + rng.normal(scale=0.6, size=240)
model = cm.Ridge(penalty=np.array([0.0, 0.05, 0.2, 1.0]), cv=4)
model.fit(x, y)
print(model.summary()['penalty'])
print(model.summary()['coef'])
print(model.predict(x[:3]))

1.0
[ 0.96303432 -0.80048681 -0.1007246   0.29078042  0.11061705]
[0.38294603 1.55660946 1.29480827]

8 summary() contract

The table below is generated by fitting the live class in this repository and then inspecting summary(). Shapes are shown because most values are plain NumPy arrays or scalars.

rng = np.random.default_rng(102)
x = rng.normal(size=(90, 4))
y = 0.3 + x @ np.array([1, -0.5, 0.2, 0]) + rng.normal(size=90)
model = cm.Ridge(penalty=np.array([0.0, 0.1, 1.0]), cv=3)
model.fit(x, y)
summary = model.summary()
display(HTML(html_table(["summary() key", "shape"], summary_shape_rows(summary))))

summary() key	shape
intercept	()
coef	(4,)
intercept_se	()
coef_se	(4,)
penalty	()
penalties	(3,)
vcov_type	()
best_penalty_index	()
cv_mse	(3,)
intercept_path	(3,)
coef_path	(4, 3)