from _api_doc_utils import *BaggedPolynomialRegressor
Bagged random-subspace polynomial ridge regression
1 Where it fits
Group: Regression
BaggedPolynomialRegressor is a prediction-only ensemble. Each base learner samples rows and a feature subspace, expands the selected features into monomials of total degree one through \(d\), standardizes those polynomial columns, fits ridge regression, and contributes to the averaged prediction.
For \(k\) selected features, each base learner uses
\[ \binom{k+d}{d}-1 \]
non-intercept polynomial terms. The estimator checks this count and the dense design size before allocation.
2 Base-learner objective
For learner \(b\), let \(I_b\) be its sampled rows and \(J_b\) its sampled feature subset. The implementation enumerates every monomial in \(x_{J_b}\) with total degree from one through \(d\). If \(\phi_b(x)\) is that \(q\)-term vector, it computes in-bag means \(\mu_b\) and population-standard-deviation scales \(s_b\), then fits
\[ (\hat a_b,\hat\gamma_b) = \arg\min_{a,\gamma} \sum_{i\in I_b} \left[ y_i-a-\left\{\frac{\phi_b(x_i)-\mu_b}{s_b}\right\}'\gamma \right]^2 +\lambda\|\gamma\|_2^2. \]
Zero-variance terms retain scale one. The ensemble prediction is the simple average
\[ \hat f(x)=\frac1B\sum_{b=1}^B \left[ \hat a_b+\left\{\frac{\phi_b(x)-\mu_b}{s_b}\right\}'\hat\gamma_b \right]. \]
Rows are sampled with replacement when bootstrap sampling is enabled. Otherwise the learner uses a shuffled subset without replacement. Feature subsets are always sampled without replacement.
3 Implementation walkthrough
The ensemble, polynomial expansion, standardization, ridge solve, and OOB accounting are all implemented in the package.
- Before drawing anything, the fit resolves
max_featuresandmax_samples, computes \(q=\binom{k+d}{d}-1\) with checked 128-bit arithmetic, and checks both the term limit and the proposed dense design size. It then generates the term specification once for a \(k\)-variable local coordinate system and shares that immutable list across all learners. - Terms are generated degree by degree by a recursion over nondecreasing feature indices. Repeated indices create powers and distinct indices create interactions: for two local variables and degree two, the order is \(x_0,x_1,x_0^2,x_0x_1,x_1^2\). The implementation evaluates each term by starting from ones and multiplying the referenced columns; it does not use a symbolic polynomial library.
- One seeded Rust RNG drives the entire ensemble. A bootstrap learner draws
max_samplesrow indices independently with replacement; a nonbootstrap learner shuffles all row indices, truncates, and sorts them. Features are independently shuffled, truncated without replacement, and sorted. Sorting makes stored subspaces deterministic to inspect but does not change the draw. - The learner selects its raw columns, materializes the full in-bag polynomial matrix, and standardizes each term with its in-bag population variance (denominator \(m\), not \(m-1\)). A standard deviation at or below \(10^{-12}\) is replaced by one, so a constant term becomes a centered zero column rather than producing division by zero.
- An intercept is prepended and ridge is fit with the same augmented-QR primitive as
Ridge: \(\sqrt{\lambda}I\) rows are appended for polynomial slopes, while the intercept is unpenalized. The learner stores its selected original-column indices, shared term list, in-bag means and scales, coefficient vector, and in-bag MSE. - For bootstrap fits, each completed learner immediately predicts all training rows. Its prediction is accumulated only for rows never drawn by that learner. At the end, each covered row’s accumulated prediction is divided by its OOB learner count; MSE and coverage are computed on that covered subset. Duplicate in-bag draws are tracked as one Boolean membership for this purpose.
- At prediction time every learner repeats raw-column selection and monomial construction, applies that learner’s own stored centering and scaling, and multiplies by its coefficient vector. The public result is the unweighted arithmetic mean across learners. There is no pruning or coefficient pooling across repeated subspaces.
Two design choices matter statistically. Standardizing inside each bootstrap sample makes the ridge penalty comparable across terms within a learner, but it also makes each learner’s coordinate system sample-dependent. Sharing a term specification is valid because every learner selects exactly \(k\) features and the term indices are local positions; it saves metadata, not polynomial computation.
4 Diagnostics and inference
This is a prediction estimator and has no coefficient-level analytic inference. With bootstrap row sampling, an observation’s out-of-bag prediction averages only learners for which that observation was absent from \(I_b\); the reported OOB MSE is computed over covered observations and the coverage fraction is reported separately. No OOB statistic is produced for sampling without replacement, even if a learner uses fewer than all rows. The OOB MSE is a prediction diagnostic, not a standard error or confidence interval.
5 Performance and resource limits
For \(k\) selected raw features,
\[ q=\binom{k+d}{d}-1 \]
grows combinatorially. One learner stores a dense \(m\times q\) design and costs roughly \(O(mq^2+q^3)\) to fit; prediction costs \(O(nq)\) per learner. Total work is multiplied by \(B\). OOB computation currently asks every bootstrap learner to predict all \(n\) training rows before retaining only uncovered rows.
The implementation rejects \(q>100{,}000\) and any single polynomial design above 50 million cells. These guards prevent the largest allocations but still permit expensive dense fits. Reducing selected features is usually more effective than reducing rows because \(q\) is combinatorial in feature count and degree. Learners are fit serially, and no polynomial matrix is shared across different feature subspaces.
6 Python API
Constructor: cm.BaggedPolynomialRegressor
Use BaggedPolynomialRegressor(n_estimators, degree, max_features, max_samples, bootstrap, penalty, seed), then fit(x, y), predict(x), and summary(). It does not report coefficient standard errors. With bootstrap sampling, summary() reports out-of-bag MSE and coverage.
print(inspect.signature(cm.BaggedPolynomialRegressor))(n_estimators=50, degree=2, max_features=None, max_samples=None, bootstrap=True, penalty=1.0, seed=42)display(HTML(html_table(["Public method"], public_methods(cm.BaggedPolynomialRegressor))))| Public method |
|---|
fit(self, /, x, y) |
predict(self, /, x) |
summary(self, /) |
7 Minimal example
rng = np.random.default_rng(18)
x = rng.normal(size=(320, 6))
y = 0.5 + 0.8 * x[:, 0] * x[:, 1] - 0.4 * x[:, 2] ** 2
y += rng.normal(scale=0.2, size=x.shape[0])
model = cm.BaggedPolynomialRegressor(
n_estimators=40,
degree=2,
max_features=4,
max_samples=240,
penalty=0.5,
seed=18,
)
model.fit(x, y)
print(model.predict(x[:3]))
print(model.summary()["oob_mse"])[ 0.35052417 -0.18592581 -0.03055871]
0.371999368323511778 summary() contract
summary = model.summary()
display(HTML(html_table(["summary() key", "shape"], summary_shape_rows(summary))))| summary() key | shape |
|---|---|
n_estimators |
() |
degree |
() |
max_features |
() |
max_samples |
() |
bootstrap |
() |
penalty |
() |
seed |
() |
n_features_in |
() |
n_terms |
() |
feature_indices |
(40, 4) |
term_counts |
(40,) |
train_mse |
(40,) |
oob_mse |
() |
oob_coverage |
() |
inference_available |
() |
max_features and max_samples are resolved fitted values rather than possible None constructor values. n_terms is the checked polynomial width per base learner. feature_indices records each random subspace. oob_mse is None when no observation receives an out-of-bag prediction.
9 Resource policy
Polynomial expansion is dense. The implementation rejects more than 100,000 polynomial terms per learner or more than 50 million cells in one polynomial design. Reduce degree, max_features, or max_samples when a guard triggers.