Cost Function

In machine learning, a cost function, also known as a loss function, is a mathematical function that is used to evaluate the performance of a machine learning model. It computes the difference between the predicted output of the model and the actual output (or target) for a set of training data.

The goal of the optimization algorithm is to minimize the cost function. which means finding the set of model parameters that produce the best predictions for the given inputs. The cost function provides a way to evaluate the performance of the model during training and helps the optimization process.

Example - 1

An example of a cost function is the mean squared error (MSE), which is commonly used in linear regression models. The MSE measures the average squared difference between the predicted values and the actual values of the target variable.

---

Mathematically, the MSE can be expressed as:

MSE = 1/n * Σ(yi_a - yi_p)²

where:

• n is the number of data points in the training set
• yi_a is the actual value of the target variable for the ith data point
• yi_p is the predicted value of the target variable for the ith data point

The goal of the linear regression model is to find the values of the parameters that minimize the MSE. This is typically done using an optimization algorithm such as gradient descent, which iteratively adjusts the parameter values in the direction that decreases the cost function.

---

Example - 2

Another example of cost function is the binary cross-entropy loss function, which is used in binary classification problems. In binary classification, the goal is to predict whether a given input belongs to one of two possible classes. The binary cross-entropy loss function measures the difference between the predicted probability and the actual probability of the input belonging to the positive class.

Mathematically, the binary cross-entropy loss function can be expressed as:

L(y, ŷ) = -1/n * Σ[yi*log(ŷi) + (1-yi)*log(1-ŷi)]

where:

• n is the number of data points in the training set
• yi is the actual label of the ith data point (0 or 1)
• ŷi is the predicted probability of the ith data point belonging to the positive class

The binary cross-entropy loss function penalizes the model heavily if it predicts a high probability for the wrong class. If the predicted probability is close to the actual probability, the loss function will be low. The goal of the optimization algorithm is to find the parameters that minimize the cost function.