Cross Entropy Loss: Formula, Working, Examples, and Implementation

Cross entropy loss is one of the most popular loss functions used in machine learning and deep learning, particularly for classification tasks. It computes the difference between the predicted probability distribution and the true distribution.

In other words, cross entropy shows how incorrect the model’s prediction is. The lower the loss value, the higher the probability the model assigns to the correct class; and vice versa.

Cross entropy loss is utilized in image recognition, natural language processing, sentiment analysis, fraud detection, recommendation systems, and many more AI use cases.

This guide will cover what cross entropy loss is, how it is computed, types of cross entropy, example implementations, and best practices.

What is Cross Entropy Loss?

Cross Entropy Loss measures how dissimilar the true class is from the predicted probability distribution for a given class classification model.

Suppose an image classification model identifies three objects:

Cat:  0.80
Dog:  0.15
Bird: 0.05

If the image actually contains a cat, the model has made a good prediction because it assigned an 80% probability to the correct class.

The overall training process typically looks like:

Input Data
↓
Neural Network
↓
Model Output
↓
Class Probabilities
↓
Cross Entropy Loss
↓
Backpropagation
↓
Update Model Weights

During training, the optimization algorithm attempts to minimize this loss.

Why is Cross Entropy Used in Machine Learning?

Classification models generally predict probabilities rather than simply outputting a category.

Consider these two predictions where Dog is the correct answer.

Prediction A:

Cat: 0.05
Dog: 0.90
Bird: 0.05

Prediction B:

Cat: 0.45
Dog: 0.10
Bird: 0.45

Prediction A is much better because the model assigns 90% probability to the correct class.

Cross entropy captures this difference. It gives Prediction A a small penalty and Prediction B a much larger penalty.

This encourages models to become not only correct but also appropriately confident about their predictions.

Cross Entropy Loss Formula

For multiclass classification, cross entropy can be expressed as:

L=−∑i=1Cyilog⁡(pi)L=-\sum_{i=1}^{C}y_i\log(p_i)

Where:

1. CC = total number of classes
2. yiy_i = actual value for class ii
3. pip_i = predicted probability for class ii
4. log⁡\log = natural logarithm

When labels use one-hot encoding, only the correct class contributes to the final loss.

The formula effectively becomes:

L=−log⁡(pcorrect)L=-\log(p_{\text{correct}})

This explains why giving the correct class a higher probability results in lower loss.

Cross Entropy Loss Example

Suppose an AI model needs to classify an image into:

1. Cat
2. Dog
3. Rabbit

The actual class is Cat.

The target representation is:

Cat    = 1
Dog    = 0
Rabbit = 0

The model predicts:

Cat    = 0.80
Dog    = 0.15
Rabbit = 0.05

The loss is:

L=−log⁡(0.80)L=-\log(0.80)

Approximately:

L=0.223L=0.223

Because the model assigns a high probability to the correct class, the loss is relatively low.

Now imagine it predicts:

Cat    = 0.05
Dog    = 0.90
Rabbit = 0.05

The loss becomes:

L=−log⁡(0.05)L=-\log(0.05)

Approximately:

L=2.996L=2.996

The incorrect and highly confident prediction therefore receives a much larger penalty.

As the correct probability approaches 1, loss approaches 0.

Types of Cross Entropy Loss

Various types of classification problems need various versions of cross entropy. There are three versions: binary cross-entropy, categorical cross-entropy, and sparse categorical cross-entropy.

L=−[ylog⁡(p)+(1−y)log⁡(1−p)]L=-[y\log(p)+(1-y)\log(1-p)]

Actual = 1
Predicted probability = 0.90

−log⁡(0.90)≈0.105-\log(0.90)\approx0.105

Cat OR Dog OR Bird

[0, 1, 0]

[0.05, 0.90, 0.05]

Cat  = [1, 0, 0]
Dog  = [0, 1, 0]
Bird = [0, 0, 1]

Cat  = 0
Dog  = 1
Bird = 2

How Softmax Works With Cross Entropy?

A neural network often produces raw numbers called logits.

For example:

[2.0, 1.0, 0.1]

These values are not probabilities.

Softmax converts the logits into probabilities:

pi=ezi∑jezjp_i=\frac{e^{z_i}}{\sum_j e^{z_j}}

Conceptually:

Logits
↓
Softmax
↓
Probabilities
↓
Cross Entropy

The resulting probabilities add up to 1, making softmax appropriate for mutually exclusive multiclass classification.

Modern machine-learning frameworks often combine softmax-related calculations and cross entropy internally for better numerical stability.

How Sigmoid Works With Binary Cross Entropy?

For binary classification, a model often produces a single logit. The sigmoid function converts it into a probability between 0 and 1.

σ(z)=11+e−z\sigma(z)=\frac{1}{1+e^{-z}}

The workflow becomes:

Model
↓
Logit
↓
Sigmoid
↓
Probability
↓
Binary Cross Entropy

Framework functions often include a joint operation that takes logits as input, which is more numerically stable than applying sigmoid and then taking the log separately.

Cross Entropy Loss in PyTorch

PyTorch provides CrossEntropyLoss for multiclass classification.

import torch
import torch.nn as nn
loss_fn = nn.CrossEntropyLoss()
logits = torch.tensor([
[2.0, 1.0, 0.1]
])
target = torch.tensor([0])
loss = loss_fn(logits, target)
print(loss.item())

An important point is that PyTorch’s standard CrossEntropyLoss expects raw logits.

Therefore, you generally should not do:

probabilities = torch.softmax(logits, dim=1)
loss = loss_fn(probabilities, target)

Instead:

loss = loss_fn(logits, target)

The loss function handles the relevant transformation internally.

For binary classification, a common option is:

loss_fn = nn.BCEWithLogitsLoss()

which combines the sigmoid-related calculation with binary cross entropy.

Cross Entropy Loss in TensorFlow/Keras

TensorFlow also provides built-in cross entropy loss functions.

For integer class labels:

import tensorflow as tf
loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(
from_logits=True
)
y_true = [0]
y_pred = [[2.0, 1.0, 0.1]]
loss = loss_fn(y_true, y_pred)
print(loss.numpy())

The setting:

from_logits=True

indicates that the model output contains raw logits.

If the model already produces probabilities through softmax, the loss configuration should be adjusted accordingly.

Cross Entropy Loss for Multilabel Classification

Multiclass and multilabel classification are different.

Consider an image containing:

Person ✓
Car    ✓
Dog    ✗
Road   ✓

Several labels can be correct simultaneously.

Softmax is generally unsuitable here because it makes the class probabilities compete and sum to 1.

Instead, the model can generate independent sigmoid probabilities:

Person → 0.95
Car    → 0.86
Dog    → 0.04
Road   → 0.91

Binary cross entropy can then calculate loss independently for each label.

Weighted Cross Entropy for Imbalanced Data

Classification datasets are not always balanced.

Consider fraud detection:

Legitimate = 99%
Fraud       = 1%

A model predicting every transaction as legitimate could achieve 99% accuracy but fail completely at detecting fraud.

Weighted cross entropy gives greater importance to selected classes.

Conceptually:

L=−wylog⁡(py)L=-w_y\log(p_y)

For example:

Legitimate weight = 1
Fraud weight      = 10

The exact weighting should depend on the dataset and the real-world cost of different errors.

Best Practices for Cross Entropy Loss

The loss must always align with the architecture and the classification goal. First, figure out what kind of problem it is: binary, mutually exclusive multi-class, or multilabel before you choose the loss.

Use built-in framework losses instead of implementing your own logarithms and probabilities because they are more numerically stable.

When dealing with imbalanced datasets, consider weighting options and assess class-level metrics. Track both the training and validation losses to detect overfitting.

Metrics like accuracy, precision, recall, F1, ROC-AUC, PR-AUC, confusion matrix, calibration, or whatever else are worth measuring.

Cross entropy must be viewed as the training goal of the model, not the only way of assessing the usefulness of the final system.

How Moon Technolabs Helps With Machine Learning Development?

Moon Technolabs helps companies develop AI and machine learning solutions for classification, prediction, computer vision, natural language processing, intelligent automation, recommendations, and other data-driven needs.

Moon Technolabs teams specialize in AI and ML development and can support the full process, from data preparation and loss function selection to model training, validation, API integration, cloud deployment, and production monitoring.

For classification tasks, beyond model performance, you must consider much more when designing the overall solution.

Conclusion

Cross entropy loss is a crucial loss function that is used to train classification models. This loss function compares true classes against estimated probabilities. A model is rewarded for assigning high probabilities to correct classes and punished for being confident in wrong predictions.

The binary cross entropy function is usually applied in cases of binary or multilabel classification, whereas categorical cross entropy is generally used in cases where mutually exclusive multiclass classification is required. In turn, sparse categorical cross-entropy lets us use integer class indices as labels.

Knowledge of how logits, sigmoid, softmax, probabilities, and cross entropy correlate with each other helps to configure classification models properly. When combined with appropriate data sets and evaluation criteria, cross entropy becomes a powerful tool to train a model.