Logistic Regression

Understanding Logistic Regression: A Beginner's Guide

Jan 29, 2025

To build out a logistic regression, there is an important mathematical expression called the sigmoid function also referred to as the Logistic function

sigmoid function looks like S-curved

Logistic Regression in Machine Learning ... — image from javapoint

\(The\hspace{0.1cm}sigmoid\hspace{0.1cm} function\hspace{0.1cm} is\hspace{0.1cm} defined\hspace{0.1cm} as: g(z) = \frac{1}{1 + e^{-z}} \hspace{0.6cm} ; \hspace{0.2cm} z = \vec{W }\cdot \vec{X} + b \)

Decision Boundary

To get a better sense of how logistic regression is computing these predictions. Let’s take a look at the Decision boundary

The decision boundary is the set of points where the model’s predicted probability is exactly 0.5.

It is determined by the logistic regression model's weights (W) and bias (b).

The shape of the decision boundary

Linear Decision Boundary: if there are two features (x1 & x2) , the decision boundary is a straight line
here, our equation would look like this

\( f_{\vec{W},b}(\vec{x}) = g(z) = g(w_1x_1 +w_2x_2 + b) \)

Non-linear decision boundary: if you use polynomial features or transformations, the decision boundary can become nonlinear (e.g., curve, etc.)

Cost Function

The cost function evaluates the model's performance by quantifying the error between predicted probabilities and actual labels.
Why is the cost function necessary?

It indicates the extent to which the model’s predictions deviate from the actual labels.
It allows for optimizing the model’s parameters (W⃗ and b) by minimizing the cost.

Logistic loss (cross-entropy Loss)

cost function for the entire dataset:

The log loss function penalizes the incorrect predictions more heavily:

If y^(i)=1 and y’^(i) is close to 0, the term log⁡(y’^(i)) becomes very large (heavily penalized).
If y(i)=0 and y^(i) is close to 1, the term log(1−y^(i)) becomes very large (heavily penalized).

This ensures that the model will incentivize to predict the probabilities close to actual labels.

Simplified loss function :

calculating the error for a single training example