When you create a machine learning model, you are choosing more than just the algorithm. You are also setting its hyperparameters. These are the high-level settings or "knobs" that you, the data scientist, configure before the training process begins.

• Model Parameters are learned from the data during training (e.g., the coefficients in a linear regression).
• Hyperparameters are set before training (e.g., max_depth in a Decision Tree, n_neighbors in kNN).

Finding the optimal set of hyperparameters can dramatically improve your model's performance.

1. GridSearchCV

This is the most straightforward tuning method. It performs an exhaustive search over a specified grid of hyperparameter values.

How it works: You define a dictionary where keys are the hyperparameters and values are the lists of settings you want to try. GridSearchCV will then train and evaluate a model for every possible combination of these settings, using cross-validation to find which combination performs best.

• Pros: Guaranteed to find the best combination within your provided grid.
• Cons: Extremely slow and computationally expensive, especially with many hyperparameters. It suffers from the "curse of dimensionality."

2. RandomizedSearchCV

This method offers a smarter and more efficient alternative to an exhaustive search.

How it works: Instead of trying every single combination, RandomizedSearchCV samples a fixed number of random combinations from the hyperparameter space. You provide it with a distribution for each hyperparameter (e.g., a range of integers or a list of options).

• Pros: Much faster than GridSearchCV. It often finds a result that is very close to the best one, as not all hyperparameters are equally important.
• Cons: It's not guaranteed to find the absolute best combination.

3. A Glimpse into Bayesian Optimization

This is a more advanced and powerful technique. Unlike Grid or Randomized Search, which are "blind," Bayesian Optimization is a guided search.

How it works: It builds a probabilistic model (called a surrogate model) of the relationship between the hyperparameters and the model's performance. It then uses this model to intelligently select the most promising set of hyperparameters to try next. It balances exploring new areas of the search space and exploiting areas it already knows are good. This often allows it to find the best settings in far fewer iterations.

Python

from sklearn.model_selection import GridSearchCV, RandomizedSearchCV
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification
from scipy.stats import randint

# Generate sample data
X, y = make_classification(n_samples=500, n_features=20, random_state=42)

# Create a model
model = RandomForestClassifier(random_state=42)

# --- GridSearchCV ---
# Define the grid of parameters to search
param_grid = {
    'n_estimators': [100, 200],
    'max_depth': [5, 10, None],
    'min_samples_leaf': [1, 2, 4]
}

# Total fits = 2 * 3 * 3 * 5 (CV folds) = 90
grid_search = GridSearchCV(estimator=model, param_grid=param_grid, cv=5, n_jobs=-1, verbose=1)
grid_search.fit(X, y)
print(f"Best parameters from GridSearch: {grid_search.best_params_}")

# --- RandomizedSearchCV ---
# Define the distributions to sample from
param_dist = {
    'n_estimators': randint(50, 250),
    'max_depth': randint(3, 15),
    'min_samples_leaf': randint(1, 5)
}
# Total fits = 10 (n_iter) * 5 (CV folds) = 50
random_search = RandomizedSearchCV(estimator=model, param_distributions=param_dist,
                                   n_iter=10, cv=5, n_jobs=-1, verbose=1, random_state=42)
random_search.fit(X, y)
print(f"\nBest parameters from RandomizedSearch: {random_search.best_params_}")