LightGBM’s Hyperparameter Tuning
Optimizing the performance of a LightGBM model requires careful consideration of its hyperparameters. To get the best results for a given dataset, LightGBM offers a variety of hyperparameters that can be fine-tuned. Selecting the hyperparameter settings that yield the best model performance is the aim of hyperparameter tuning; this is usually assessed using evaluation metrics such as accuracy, AUC, or log loss. For hyperparameter tuning, two popular methods are grid search and random search. Grid search involves giving the model a predetermined set of hyperparameter values, and the algorithm determines how well the model performs in every possible combination. When the search space is large, random search is more effective since it randomly samples hyperparameters from predetermined ranges.
Two popular methods for hyperparameter tuning are grid search and random search. In grid search, the model’s performance is assessed for every potential combination of hyperparameter values that you specify in advance. On the other hand, random search is more effective when the search space is large since it randomly samples hyperparameters from predetermined ranges.
Implementing Hyperparameter Tuning with LightGBM
Let’s see how to perform hyperparameter tuning with LightGBM.
Import the required libraries
Python3
import lightgbm as lgbimport numpy as npfrom sklearn.datasets import load_irisfrom sklearn.model_selection import GridSearchCVfrom sklearn.metrics import accuracy_scorefrom sklearn.model_selection import train_test_split |
This code uses GridSearchCV from scikit-learn for hyperparameter tuning and LightGBM, a gradient boosting framework. The model loads the Iris dataset, splits the data into train and test, and then uses grid search to find the optimal hyperparameters. The accuracy measure is used to assess the model’s performance.
Loading Dataset and Splitting Data
Python3
# Load the Iris datasetdata = load_iris()X = data.datay = data.target# Split the dataset into training and testing setsX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42) |
With 80% of the data utilized for training and 20% put aside for testing, this code snippet loads the Iris dataset and divides it into training and testing sets. In order to guarantee reproducibility, the random seed is fixed using the random_state parameter.
Defining Parameters
Python3
# Define a range of values for the hyperparameters to search throughparam_grid = { 'num_leaves': [5, 20, 31], 'learning_rate': [0.05, 0.1, 0.2], 'n_estimators': [50, 100, 150]} |
To find the right hyperparameter tuning, this code defines a grid of values for the hyperparameters. It allows for an exhaustive search across these combinations during hyperparameter optimization because it specifies different values for “num_leaves,” “learning_rate,” and “n_estimators.”
Model Development
Python3
# Initialize an empty dictionary to store the best hyperparameters and their valuesbest_hyperparameters = {}best_values = {}# Initialize the LightGBM classifierlgb_classifier = lgb.LGBMClassifier(objective='multiclass', num_class=3, boosting_type='gbdt')# Initialize GridSearchCV for hyperparametersgrid_search = GridSearchCV(estimator=lgb_classifier, param_grid=param_grid, scoring='accuracy', cv=5)# Fit the model to the training data to search for the best hyperparametersgrid_search.fit(X_train, y_train)# Get the best hyperparameters and their valuesbest_params = grid_search.best_params_best_hyperparameters = list(best_params.keys())best_values = list(best_params.values()) |
This method uses a GridSearchCV with a LightGBM classifier to conduct hyperparameter tuning. In order to save the optimal hyperparameters and their values, it initializes an empty dictionary called best_hyperparameters. Additionally, it initializes GridSearchCV and the LightGBM classifier by providing the estimator, the number of cross-validation folds (cv=5), the scoring metric (‘accuracy’), and the parameter grid to search across (param_grid). Next, the grid_search finds the optimal hyperparameters by fitting the model to the training set.
After the search is finished, the best hyperparameters and their matching values are found and added to best_params, a dictionary where the best values of each hyperparameter are the values and the names of the hyperparameters are the keys. Next, the best_hyperparameters and best_values lists include the optimal hyperparameters and their values, respectively.
Training the model
Python3
# Train a LightGBM model with the best hyperparametersbest_model = lgb.LGBMClassifier(**best_params)best_model.fit(X_train, y_train) |
The optimal hyperparameters found through hyperparameter tuning are used to train a LightGBM model in this code. **best_params** is passed in to initialize a new LightGBM classifier, best_model, with the optimal hyperparameters. Subsequently, it fits the best_model to the training set (X_train and y_train), enabling the training of a model with optimum parameters.
Prediction and Evaluation
Python3
# Make predictions on the test set using the best modely_pred = best_model.predict(X_test)# Calculate accuracyaccuracy = accuracy_score(y_test, y_pred)print('Best hyperparameters:', best_hyperparameters)print('Best values:', best_values)print(f'Accuracy with best hyperparameters: {accuracy:.4f}') |
Output:
Best hyperparameters: ['learning_rate', 'n_estimators', 'num_leaves']
Best values: [0.05, 50, 5]
Accuracy with best hyperparameters: 1.0000
This code makes predictions on the test set (X_test) using the best model that was trained with the optimized hyperparameters. These forecasts’ accuracy is computed and reported. It also shows the best values (best_values) and hyperparameters (best_hyperparameters) that produced the highest accuracy for the model.
Cross-validation and Hyperparameter tuning of LightGBM Model
In a variety of industries, including finance, healthcare, and marketing, machine learning models have become essential for resolving challenging real-world issues. Gradient boosting techniques have become incredibly popular among the myriad of machine learning algorithms due to their remarkable prediction performance. Due to its speed and effectiveness, LightGBM (Light Gradient Boosting Machine) is one such technique that many data scientists and machine learning practitioners now turn to first.
We will examine LightGBM in this post with an emphasis on cross-validation, hyperparameter tweaking, and the deployment of a LightGBM-based application. To clarify the ideas covered, we shall use code examples throughout the article.