Cross-Validation Ridge Regression System

A Cross-Validation Ridge Regression System is a Cross-Validation System that implements a Ridge Regression Algorithm and a Cross-Validation Algorihtm to solve a Cross-Validation Ridge Regression Task.

• AKA: Ridge CV System, Ridge Regression CV System.
• Example(s):
  • sklearn.linear_model.RidgeCV a Scikit Learn mode implements a cross-validation ridge regression algorithm.
    • http://andrew47.github.io/scikitlearn-linear-reg.html#Cross-Validation
    • Lab 10 - Ridge Regression and the Lasso in in Python pp. 6
  • ridge.cv from the R package parcor v0.2-6
• Counter-Example(s):
  • Ridge Regression System.
  • LASSO CV System.
  • LASSO-LARS CV System.
• See: Leave-One-Out Cross-Validation, Evaluation Algorithm, Regression Analysis Task.

---

---

References

2017a

• (Scikit Learn, 2017) ⇒ "1.1.2.2. Setting the regularization parameter: generalized Cross-Validation" Retrieved: 2017-17-09
  • QUOTE: RidgeCV implements ridge regression with built-in cross-validation of the alpha parameter. The object works in the same way as GridSearchCV except that it defaults to Generalized Cross-Validation (GCV), an efficient form of leave-one-out cross-validation:

    {| class="wikitable" style="margin-left: 50px;border:1px;background:#f2f2f2"

|style="font-family:monospace; font-size:10.5pt;font-weight=bold;text-align:left;width:700px;"| >>> from sklearn.linear_model import linear_model

>>> reg = linear_model.RidgeCV(alphas=[0.1, 1.0, 10.0])

>>> reg.fit([ [0, 0], [0, 0], [1, 1] ], [0, .1, 1])

RidgeCV(alphas=[0.1, 1.0, 10.0], cv=None, fit_intercept=True, scoring=None, normalize=False)

>>> reg.alpha_

0.1 |}=== 2017b===

• (Scikit Learn, 2017) ⇒ 3.2.4.1.9. sklearn.linear_model.RidgeCV Retrieved: 2017-17-09
  • QUOTE: Ridge regression with built-in cross-validation.

    By default, it performs Generalized Cross-Validation, which is a form of efficient Leave-One-Out cross-validation(...)

2017c

• (jcrouser,2017) ⇒ Lab 10 - Ridge Regression and the Lasso in Python pp. 6
  • QUOTE: Instead of arbitrarily choosing alpha $ = 4$, it would be better to use cross-validation to choose the tuning parameter alpha. We can do this using the cross-validated ridge regression function, RidgeCV(). By default, the function performs generalized cross-validation (an efficient form of LOOCV), though this can be changed using the argument cv.

    {| class="wikitable" style="margin-left: 50px;border:1px;background:#f2f2f2"

|style="font-family:monospace; font-size:10.5pt;font-weight=bold;text-align:left;width:700px;"|

In [105]:ridgecv = RidgeCV(alphas=alphas, scoring=’mean_squared_error’, normalize=True

ridgecv.fit(X_train, y_train)

ridgecv.alpha_

Out[105]: 0.57487849769886779

|}

Therefore, we see that the value of alpha that results in the smallest cross-validation error is 0.57. What is the test MSE associated with this value of alpha?

{| class="wikitable" style="margin-left: 50px;border:1px;background:#f2f2f2"

|style="font-family:monospace; font-size:10.5pt;font-weight=bold;text-align:left;width:700px;"| In [106]ridge4 = Ridge(alpha=ridgecv.alpha_, normalize=True)

ridge4.fit(X_train, y_train)

mean_squared_error(y_test, ridge4.predict(X_test))

Out[106]: 99825.648962927298 |}

This represents a further improvement over the test MSE that we got using alpha $ = 4$

2017d

• (Kraemer, 2017) ⇒ From parcor v0.2-6 by Nicole Kraemer https://www.rdocumentation.org/packages/parcor/versions/0.2-6/topics/ridge.cv Retrieved: 2017-09-17
  • QUOTE: This function computes the optimal ridge regression model based on cross-validation.

    Keywords: multivariate

    Usage

    ridge.cv(X, y, lambda, scale = TRUE, k = 10, plot.it = FALSE)

---