Faster Hyperparameter Tuning with Scikit-Learn’s HalvingGridSearchCV | by Kyle Gilde | Jan, 2021

[ad_1]


Photo by Roberta Sorge on Unsplash

My Experiment

Upgrade Scikit-Learn

# !! pip install scikit-learn --upgrade
import sklearn
print(sklearn.__version__)
0.24.0

Loading the Dataset

import numpy as np  
import pandas as pd

DEP_VAR = 'SalePrice'
train_df = pd.read_csv('../kaggle/input/house-prices-advanced-regression-techniques/train.csv')
.set_index('Id')

y_train = train_df.pop(DEP_VAR)

Creating a Pipeline & Model

from sklearn import set_config        
from sklearn.utils import estimator_html_repr
from IPython.core.display import display, HTML

from pipeline_ames import pipe
set_config(display='diagram')
display(HTML(estimator_html_repr(pipe)))

Experimental Controls

from sklearn.metrics import mean_squared_log_error, make_scorer

np.random.seed(123) # set a global seed
pd.set_option("display.precision", 4)

root_mean_squared_log_error = lambda y_true, y_pred: np.sqrt(mean_squared_log_error(y_true, y_pred))
scorer = make_scorer(root_mean_squared_log_error, greater_is_better=False)

param_grid = {"model__max_depth": [5, 6, 7],
'model__learning_rate': [.01, 0.03, .06],
'model__subsample': [.7, .8, .9],
'model__colsample_bylevel': [.8, .9, 1]}

grid_search_params = dict(estimator=pipe,
param_grid=param_grid,
scoring=scorer,
cv=3,
n_jobs=-1,
verbose=2)

Tests

1. GridSearchCV

%%time
from sklearn.model_selection import GridSearchCV
full_results = GridSearchCV(**grid_search_params)
.fit(train_df, y_train)
pd.DataFrame(full_results.best_params_, index=[0])
.assign(RMSLE=-full_results.best_score_)
Fitting 3 folds for each of 81 candidates, totalling 243 fits
Wall time: 32min 53s
png

2. HalvingGridSearchCV with n_samples

%%time

from sklearn.experimental import enable_halving_search_cv
from sklearn.model_selection import HalvingGridSearchCV

FACTOR = 2
MAX_RESOURCE_DIVISOR = 4

n_samples = len(train_df)
halving_results_n_samples =
HalvingGridSearchCV(resource='n_samples',
min_resources=n_samples//
MAX_RESOURCE_DIVISOR,
factor=FACTOR,
**grid_search_params
)
.fit(train_df, y_train)

n_iterations: 3
n_required_iterations: 7
n_possible_iterations: 3
min_resources_: 365
max_resources_: 1460
aggressive_elimination: False
factor: 2
----------
iter: 0
n_candidates: 81
n_resources: 365
Fitting 3 folds for each of 81 candidates, totalling 243 fits
----------
iter: 1
n_candidates: 41
n_resources: 730
Fitting 3 folds for each of 41 candidates, totalling 123 fits
----------
iter: 2
n_candidates: 21
n_resources: 1460
Fitting 3 folds for each of 21 candidates, totalling 63 fits
Wall time: 34min 46s
from compare_functions import *compare_cv_best_params(full_results, *[halving_results_n_samples])
.style.applymap(lambda cell: ‘background: pink’ if cell == 9 else)

HalvingGridSearchCV with n_estimators

%%time
halving_results_n_estimators =
HalvingGridSearchCV(resource='model__n_estimators',
max_resources=1000,
min_resources=1000 // MAX_RESOURCE_DIVISOR,
factor=FACTOR,
**grid_search_params
)
.fit(train_df, y_train)
n_iterations: 3
n_required_iterations: 7
n_possible_iterations: 3
min_resources_: 250
max_resources_: 1000
aggressive_elimination: False
factor: 2
----------
iter: 0
n_candidates: 81
n_resources: 250
Fitting 3 folds for each of 81 candidates, totalling 243 fits
----------
iter: 1
n_candidates: 41
n_resources: 500
Fitting 3 folds for each of 41 candidates, totalling 123 fits
----------
iter: 2
n_candidates: 21
n_resources: 1000
Fitting 3 folds for each of 21 candidates, totalling 63 fits
Wall time: 22min 59s
compare_cv_best_params(full_results, *[halving_results_n_samples, 
halving_results_n_estimators])
.style.apply(lambda row:
row.apply(lambda col:
'background: lightgreen' if row.name == 2 else ''),
axis=1)

Conclusion

Read More …

[ad_2]


Write a comment