5.7 Neural Networks and Deep Learning

Artificial Neural Networks (ANNs)

An Artificial Neural Network (ANN) is a computational model inspired by the biological neural structures of the human brain. It consists of interconnected nodes called neurons arranged in layers that work together to process data and learn patterns.

Structure of a Neural Network

A standard neural network has three types of layers:

A network with multiple hidden layers is called a deep neural network.

Key Components of a Neuron

• Weights ($w$): Determine the importance of each input signal. A higher weight means the input has more influence on the output.
• Bias ($b$): A constant added to the weighted sum, allowing the activation function to shift and better fit the data.
• Activation Function: Introduces non-linearity into the output, enabling the network to learn complex patterns. Common examples:
  • ReLU (Rectified Linear Unit): $f(x)=\max (0,x)$
  • Sigmoid: $f(x)=\frac{1}{1+e^{-x}}$
  • Softmax: used in the output layer for multi-class classification

The output of a single neuron is computed as: $\text{output}=\text{activation}\left({\sum }_i{w}_i{x}_i+b\right)$

---

Deep Learning

Deep Learning is a subset of machine learning that uses artificial neural networks with multiple hidden layers (deep architectures). These deep architectures allow the model to automatically learn hierarchical representations and complex patterns from large amounts of data — without manual feature engineering.

Why 'Deep'?

The word deep refers to the depth of the network — the number of hidden layers. A shallow network has one hidden layer; a deep network has many.

---

Training a Neural Network

Forward Propagation

Data flows from the input layer through hidden layers to the output layer, producing a prediction.

Loss Function

The loss function measures the difference between the predicted output and the actual (correct) output. The goal of training is to minimise this loss.

Backpropagation

Backpropagation is the algorithm used to train neural networks. It works by:

1. Calculating the gradient of the loss function with respect to each weight.
2. Propagating the error backwards through the network.
3. Updating weights using gradient descent to reduce the error.

During a successful training session, the error (loss) decreases as weights are optimised over multiple iterations (epochs).

---

Applications of Deep Learning

Deep Learning excels at tasks involving large, complex datasets:

---

Model Performance Metrics

To evaluate how well a machine learning or deep learning model performs, we use the following key metrics:

1. Accuracy

$\text{Accuracy}=\frac{\text{Correct Predictions}}{\text{Total Predictions}}\times 100\%$ Best used when classes are balanced.

2. Precision

$\text{Precision}=\frac{\text{True Positives}}{\text{True Positives}+\text{False Positives}}$ Answers: Of all the items I predicted as positive, how many actually were?

3. Recall (Sensitivity)

$\text{Recall}=\frac{\text{True Positives}}{\text{True Positives}+\text{False Negatives}}$ Answers: Of all the actual positives, how many did I correctly find?

4. F1-Score

$\text{F1}=2\times \frac{\text{Precision}\times \text{Recall}}{\text{Precision}+\text{Recall}}$ The harmonic mean of precision and recall. Useful when the dataset is imbalanced (unequal class sizes).

5. Loss

The value of the loss function during training. A decreasing loss over epochs indicates the model is learning.

---

Summary