Grid Search & Random Search

Grid search and random search are both hyperparameter tuning techniques used in machine learning to find the best set of hyperparameters for a given model. By using these techniques we can improve the overall performance of the model by finding the optimal values of hyperparameters.

Random Search: Random search is a technique of searching hyperparameters by randomly sampling hyperparameters from a specified distribution.Here, we randomly pick combinations of hyperparameters to train our model. This means we're not following any specific pattern or grid when choosing these hyperparameters. For each combination of hyperparameters, a model is trained and evaluated on the validation set. The best combination of hyperparameters is the one that results in the best performance metric.

To better understand the concept consider the following example:

Example: Imagine We have a machine learning model that has two hyperparameters: the learning rate and the number of trees in a random forest. With random search, instead of trying out every possible combination (like grid search does), we randomly select different values for the learning rate and the number of trees to see which combination works best.

Advantages of Random Search:

1) It's less computationally expensive compared to grid search because it doesn't try every combination.

2) It can sometimes find better hyperparameter values because it explores a wider range of possibilities.

However, random search does not guarantee that all possible combinations of hyperparameters are explored, and thus it may not find the optimal set of hyperparameters. Nonetheless, it has been shown that random search can often outperform grid search, especially when the number of hyperparameters is large, and the search space is vast.

Grid Search: Grid search is a technique of searching hyperparameters by creating a grid of all possible combinations of hyperparameters and evaluating each combination using cross-validation. Here we create a grid of hyperparameter values and try out all possible combinations from this grid. The best combination of hyperparameters is the one that results in the best performance metric.

To understand the concept consider the following example

Example: Let's consider the same example of a machine learning model with two hyperparameters: learning rate and number of trees. In grid search, we define a grid of values for each hyperparameter (for example, [0.01, 0.1, 1.0] for learning rate and [10, 50, 100] for number of trees). Grid search then tries all combinations of these values (e.g., (0.01, 10), (0.01, 50), (0.01, 100), (0.1, 10), etc.) to find the best combination.

Advantages:

1) It guarantees finding the best hyperparameter values within the specified grid.
2) It's straightforward and easy to implement.

However, it is computationally expensive, as it needs to train and evaluate a model for each combination of hyperparameters in the grid. Additionally, it may also be limited by the resolution of the grid, as it may not explore the hyperparameters in enough detail.

Comparison:

• Random Search is like randomly exploring a forest to find the best path, while Grid Search is like systematically searching every possible path in the forest.
• Random Search is faster but might miss some optimal combinations, while Grid Search is exhaustive but can be computationally expensive.
• Random Search is useful when we have limited computational resources and want to explore a wide range of hyperparameters. Grid Search is useful when we have the computational power and want to ensure we've tried every possible combination within a specified range.