Deep Learning Foundations

Understanding Neural Networks from Scratch

Biological Neurons → Artificial Neurons → Deep Neural Networks

1. What is Deep Learning?

Deep Learning is a subset of Machine Learning that uses Artificial Neural Networks with multiple hidden layers to automatically learn complex patterns from data.

Unlike traditional machine learning algorithms where humans manually define features, Deep Learning automatically discovers important features from data.

Examples of Deep Learning Applications
  • Face Recognition
  • ChatGPT and Large Language Models
  • Self Driving Cars
  • Medical Image Diagnosis
  • Voice Assistants
  • Fraud Detection
  • Recommendation Systems

2. AI vs Machine Learning vs Deep Learning

Artificial Intelligence

Making machines perform tasks that normally require human intelligence.

Machine Learning

Subset of AI where systems learn patterns from data.

Deep Learning

Subset of Machine Learning using Deep Neural Networks.

3. Biological Neuron

The human brain contains approximately 86 billion neurons. Each neuron receives signals, processes information, and transmits output signals.

Main Components

Biological neurons inspired the design of Artificial Neural Networks.

4. Artificial Neuron

An Artificial Neuron receives numerical inputs, performs mathematical operations, and generates output.

Mathematical Representation

z = (w₁x₁ + w₂x₂ + w₃x₃ + ... + wₙxₙ) + b
Where: Output:
a = Activation(z)

5. Artificial Neuron Example

Suppose:

z = (2 × 0.5) + (3 × 0.8) + 1
z = 1 + 2.4 + 1
z = 4.4
After activation function:
a = Activation(4.4)

6. Artificial Neural Network Architecture

Input Layer → Hidden Layer(s) → Output Layer

An ANN consists of interconnected neurons organized into layers.

7. Input Layer

The Input Layer receives raw features from the dataset.

Example Dataset
Age Salary Experience
25 50000 2

Number of Input Neurons = Number of Features

Input Neurons = 3

8. Hidden Layer

Hidden Layers perform feature extraction and pattern learning.

Responsibilities
More hidden layers allow learning complex representations.

9. Output Layer

Problem Type Output Neurons
Binary Classification 1
Multi-Class Classification Number of Classes
Regression 1

10. Weights and Biases

Weights represent importance of features.

Higher Weight = More Influence

Bias allows shifting the activation function.

y = wx + b

11. Why Deep Learning Works

Part 1 Summary

Next Module: Activation Functions, Gradient, Learning Rate, Forward Propagation and Loss Functions.

Part 2 : Activation Functions, Gradients & Forward Propagation

Mathematics Behind Neural Network Learning

12. Why Do We Need Activation Functions?

Suppose a neural network has multiple layers but no activation functions. Then every layer performs only linear operations.

y = mx + c

Combining multiple linear equations still produces another linear equation. Therefore the network cannot learn complex patterns.

Without activation functions, Deep Learning becomes simple Linear Regression.

13. Sigmoid Activation Function

Sigmoid converts any number into a probability between 0 and 1.

σ(x) = 1 / (1 + e⁻ˣ)
Example
x = 2 σ(2) = 1/(1+e⁻²) ≈ 0.88

The neuron predicts an 88% probability.

Range
0 to 1
Advantages
Disadvantages

14. Derivative of Sigmoid

Backpropagation requires derivatives. Derivative of Sigmoid:

σ'(x) = σ(x)(1-σ(x))
Example
σ(x)=0.99 σ'(x)=0.99(1-0.99) =0.0099

Very small gradient. After many layers:

0.009 × 0.008 × 0.007 × 0.006 ≈ 0

This causes Vanishing Gradient.

15. Tanh Activation Function

tanh(x) = (eˣ - e⁻ˣ) / (eˣ + e⁻ˣ)
Range
-1 to +1
Advantages
Disadvantages

16. ReLU Activation Function

f(x) = max(0,x)
Examples
ReLU(5)=5
ReLU(-5)=0
Advantages

17. Leaky ReLU

f(x)=x if x > 0
f(x)=0.01x if x < 0

Leaky ReLU solves the Dying ReLU problem.

18. Softmax Function

Used in Multi-Class Classification.

Softmax(xᵢ) = eˣⁱ / Σeˣʲ
Example
Class Probability
Cat 0.80
Dog 0.15
Horse 0.05

Highest probability class is selected.

19. What is Gradient?

Gradient measures how much the loss changes with respect to a parameter.

Gradient = ∂Loss / ∂Weight

Think of gradient as the slope of a hill.

20. Why Gradient is Important?

Neural Networks learn by adjusting weights. Gradient tells:

21. Loss Function

Loss measures prediction error.

Mean Squared Error
Loss = (Actual - Predicted)²
Example
Actual = 10 Predicted = 8 Loss = (10-8)² Loss = 4

22. Learning Rate

Learning Rate controls the size of weight updates.

New Weight = Old Weight - Learning Rate × Gradient
Example
Weight = 2 Gradient = 0.5 Learning Rate = 0.1
New Weight = 2 - (0.1 × 0.5) = 1.95

23. Effect of Learning Rate

Learning Rate Result
0.0001 Very Slow Learning
0.01 Good Learning
1.0 Overshooting

24. Forward Propagation

Forward Propagation is the process of sending data from Input Layer to Output Layer.

Steps
  1. Input Features
  2. Multiply by Weights
  3. Add Bias
  4. Apply Activation Function
  5. Generate Prediction
  6. Calculate Loss

25. Complete Forward Propagation Example

x = 3 w = 0.5 b = 1
z = wx + b
z = (3 × 0.5)+1
z = 2.5
Apply Sigmoid:
σ(2.5) = 1/(1+e^-2.5)
Output ≈ 0.924

Network predicts 92.4% probability.

Part 2 Summary

Next Part: Backpropagation, Chain Rule, Vanishing Gradient Mathematics, Gradient Descent, SGD, Mini Batch GD, Adam, RMSProp, AdaGrad and Training Process.

Part 3 : Backpropagation, Vanishing Gradient & Optimizers

How Neural Networks Actually Learn

26. What Happens After Forward Propagation?

During Forward Propagation, the network generates predictions. The next question is: How does the network know whether its prediction is correct or wrong?

The answer lies in:

Learning = Reducing Loss by Updating Weights

27. Loss Function Revisited

The loss function measures the difference between actual and predicted values.

Example
Actual Predicted
1 0.6
Loss = (1 - 0.6)²
Loss = 0.16

The goal of Deep Learning is:

Minimize Loss → 0

28. What is Backpropagation?

Backpropagation is the process of sending error information backward through the network.

It determines:

Forward Propagation → Prediction Backpropagation → Learning

29. Chain Rule (Heart of Backpropagation)

Backpropagation uses Calculus. Specifically:

Chain Rule
Formula
dL/dW = (dL/dY) × (dY/dZ) × (dZ/dW)
Where:

The Chain Rule allows gradients to travel backward through multiple layers.

30. Gradient Calculation Example

Suppose:

Loss = W²
Derivative:
dLoss/dW = 2W
If:
W = 5
Then:
Gradient = 10

A large gradient means the weight should change significantly.

31. Weight Update Formula

New Weight = Old Weight − Learning Rate × Gradient
Example
Weight = 5 Gradient = 10 Learning Rate = 0.01
New Weight = 5 - (0.01 × 10)
New Weight = 4.9

32. Complete Training Cycle

  1. Input Data
  2. Forward Propagation
  3. Prediction
  4. Loss Calculation
  5. Backpropagation
  6. Gradient Calculation
  7. Weight Update
  8. Repeat Until Convergence

33. Vanishing Gradient Problem

One of the biggest challenges in Deep Neural Networks.

While backpropagating through many layers, gradients become extremely small.

0.5 × 0.5 × 0.5 × 0.5 × 0.5 = 0.03125

As layers increase:

0.01 × 0.01 × 0.01 × 0.01 ≈ 0

The gradient nearly disappears.

34. Effects of Vanishing Gradient

Gradient ≈ 0 ⇒ Learning Stops

35. Solutions to Vanishing Gradient

36. Exploding Gradient Problem

Opposite of Vanishing Gradient.

Gradients become extremely large.

5 × 5 × 5 × 5 × 5 = 3125

Huge gradients cause unstable training.

37. What is an Optimizer?

An optimizer updates weights to reduce loss.

Optimizer = Brain Behind Learning

38. Gradient Descent

W = W - η∇J(W)
Where:

Goal: Find minimum loss.

39. Types of Gradient Descent

Type Data Used
Batch GD Entire Dataset
SGD One Sample
Mini Batch GD Small Batch

40. Stochastic Gradient Descent (SGD)

Updates weights after every training sample.

Advantages: Disadvantages:

41. Mini Batch Gradient Descent

Most commonly used approach.

Example:
Dataset = 10,000 samples Batch Size = 100 Iterations = 100
Benefits:

42. Momentum Optimizer

Adds previous movement information.

Velocity = βV + ηGradient
Benefits:

43. AdaGrad Optimizer

Adaptive Learning Rate Optimizer.

Benefits: Limitation: Learning rate becomes extremely small.

44. RMSProp Optimizer

Improves AdaGrad.

Cache = 0.9 × Cache + 0.1 × Gradient²
Advantages:

45. Adam Optimizer

Most widely used optimizer in Deep Learning.

Adam combines:

Adam = Momentum + RMSProp
Advantages:

46. Epoch, Batch Size & Iteration

Dataset = 1000 Samples
Batch Size = 100
Iterations = 1000/100 = 10
1 Epoch = 10 Iterations

47. Training vs Validation Loss

Training Loss Validation Loss
Learning on training data Performance on unseen data
Good Model
Overfitting

Part 3 Summary

Next Part: ANN Development using Keras, Binary Classification, Multi-Class Classification, Confusion Matrix, Precision, Recall, F1 Score, Hyperparameter Tuning, Early Stopping, Dropout and Real Projects.

Part 4 : ANN Development, Evaluation & Hyperparameter Tuning

Building Real Deep Learning Models Using TensorFlow & Keras

48. ANN Model Development Using Keras

TensorFlow Keras is one of the most widely used Deep Learning frameworks. Keras provides a high-level API for building, training and deploying neural networks.

Typical Workflow
  1. Load Dataset
  2. Preprocess Data
  3. Split Train/Test Data
  4. Create ANN Model
  5. Compile Model
  6. Train Model
  7. Evaluate Model
  8. Predict New Data

49. Required Libraries 

import numpy as np
import pandas as pd

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense

50. Data Preprocessing

Neural Networks perform better when data is normalized or standardized. 

scaler = StandardScaler()

X_train = scaler.fit_transform(X_train)

X_test = scaler.transform(X_test)
Why Scaling?

51. Creating an ANN Architecture 


model = Sequential()

model.add(Dense(
    units=16,
    activation='relu',
    input_dim=10
))

model.add(Dense(
    units=8,
    activation='relu'
))

model.add(Dense(
    units=1,
    activation='sigmoid'
))
Architecture
Input Layer (10) ↓ Hidden Layer (16) ↓ Hidden Layer (8) ↓ Output Layer (1)

52. Model Compilation 


model.compile(
optimizer='adam',
loss='binary_crossentropy',
metrics=['accuracy']
)
Components
Component Purpose
Optimizer Weight Updates
Loss Error Calculation
Metrics Performance Evaluation

53. Model Training 


history = model.fit(
X_train,
y_train,
epochs=100,
batch_size=32,
validation_split=0.2
)
Training Process

54. Binary Classification

Binary Classification contains only two classes.

Examples:
Output Layer
Dense(1, activation='sigmoid')
Loss Function
Binary Cross Entropy

55. Binary Cross Entropy Loss

Loss = -[y log(p) + (1-y) log(1-p)]
Where:

Used for Binary Classification Problems.

56. Multi-Class Classification

More than two classes.

Examples:
Output Layer
Dense(Number_of_Classes, activation='softmax')

57. Categorical Cross Entropy

Loss = -Σ y log(p)

Used in Multi-Class Classification.

Example
Cat = 0.80 Dog = 0.15 Horse = 0.05
Prediction = Cat

58. Making Predictions 


predictions =
model.predict(X_test)



predictions > 0.5

Threshold 0.5 converts probability to class labels.

59. Model Evaluation 


loss, accuracy =
model.evaluate(
X_test,
y_test
)
Accuracy = Correct Predictions / Total Predictions

60. Confusion Matrix

Predicted Positive Predicted Negative
Actual Positive TP FN
Actual Negative FP TN
Terminology

61. Accuracy

Accuracy = (TP + TN) / (TP + TN + FP + FN)
Example
TP=50 TN=40 FP=5 FN=5
Accuracy = 90%

62. Precision

Precision = TP / (TP + FP)

Out of all predicted positives, how many were actually positive?

63. Recall

Recall = TP / (TP + FN)

Out of all actual positives, how many were detected?

64. F1 Score

F1 = 2 × Precision × Recall / (Precision + Recall)

Balances Precision and Recall.

65. Classification Report 


from sklearn.metrics import classification_report

print(
classification_report(
y_test,
y_pred
)
)
Provides:

66. Hyperparameters

Hyperparameters are values chosen before training.

Hyperparameter Example
Learning Rate 0.001
Epochs 100
Batch Size 32
Optimizer Adam
Hidden Layers 2
Neurons 16,8

67. Hyperparameter Tuning

Methods

68. Dropout Layer

Dropout randomly disables neurons during training. 


from tensorflow.keras.layers import Dropout

model.add(
Dropout(0.3)
)
Benefits:

69. Early Stopping

Stops training when validation loss stops improving. 


from tensorflow.keras.callbacks import EarlyStopping

early_stop =
EarlyStopping(
patience=10
)
Benefits:

70. Activation Function Comparison

Activation Range Use Case
Sigmoid 0 to 1 Binary Classification
Tanh -1 to 1 Hidden Layers
ReLU 0 to ∞ Most Hidden Layers
Leaky ReLU -∞ to ∞ Avoid Dead Neurons
Softmax 0 to 1 Multi-Class Output

71. Frequently Asked Interview Questions

  1. What is Deep Learning?
  2. Difference between ANN and CNN?
  3. What is Backpropagation?
  4. What is Vanishing Gradient?
  5. Why ReLU is preferred?
  6. What is Adam Optimizer?
  7. Difference between Precision and Recall?
  8. What is Overfitting?
  9. What is Dropout?
  10. What is Early Stopping?

Part 4 Summary

🎯 Congratulations! You have completed the Deep Learning Foundations Module covering: 1. Deep Learning Basics 2. Activation Functions & Gradients 3. Backpropagation & Optimizers 4. ANN Development & Model Evaluation You are now ready to move to CNN, Transfer Learning, Computer Vision and Advanced Deep Learning Architectures.