🌳 Decision Trees

Machine Learning for Beginners

Learn how computers make decisions just like humans.

🌲 What is a Tree?

In nature, a tree starts from a root, grows into branches, and ends with leaves. A Decision Tree follows the same concept.

Real Tree ➜ Root ➜ Branches ➜ Leaves
Decision Tree ➜ Question ➜ Answers ➜ Prediction

What is a Decision Tree?

A Decision Tree is a Supervised Machine Learning Algorithm used for Classification and Regression tasks. It learns patterns by asking questions and making decisions.

Decision Tree = Questions + Answers + Decisions

Types of Decision Trees

Classification Tree

Pass / Fail
Spam / Not Spam
Disease / No Disease
Buy / Don't Buy

Regression Tree

Salary Prediction
House Price Prediction
Stock Price Prediction
Temperature Prediction

🍕 Indian Pizza Dataset

Goal: Predict whether a student will order pizza.

Weather	Pocket Money	Friend Available	Order Pizza
Hot	500	Yes	Yes
Hot	400	Yes	Yes
Cold	100	No	No
Rainy	200	No	No
Rainy	500	Yes	Yes
Hot	600	No	Yes

Input Features & Target

🌤 Weather
Input Feature

💰 Pocket Money
Input Feature

👬 Friend Available
Input Feature

🍕 Order Pizza
Target

🌳 How a Decision Tree Thinks

Dataset

↓

Weather ?

☀️ HOT

🍕 Order Pizza

❄️ COLD

❌ Don't Order

🌧️ RAINY

💰 Money > 300 ?

YES

🍕

❌

🚀 How Decision Trees Learn

Step 1: Load Dataset

Step 2: Identify Features and Target

Step 3: Study Every Feature

Step 4: Choose Best Feature

Step 5: Create Branches

Step 6: Ask More Questions if Needed

Step 7: Make Final Prediction

📝 Summary

Decision Trees mimic human decision making.
Classification Trees predict categories.
Regression Trees predict numbers.
Features are inputs.
Target is the output.
Decision Trees ask questions before predicting.

PART 2

🌳 Decision Tree Terminology & Structure

Understanding Root Nodes, Internal Nodes, Branches, Leaf Nodes, Pure Nodes and Impure Nodes

🤔 Why Does a Decision Tree Ask Questions?

Imagine you want to decide whether to play cricket. You don't randomly decide. You first ask questions.

🌧 Is it raining?
✔ NO → Play Cricket
❌ YES → Stay Home

Decision Trees work exactly the same way. They continuously ask questions until a final decision is reached.

📚 Decision Tree Terminology

🌱

Root Node

Starting point of the tree

🌿

Internal Node

Decision-making question

🔀

Branch

Path between nodes

🍁

Leaf Node

Final Prediction

🌳 Visual Tree Structure

This is a real Decision Tree generated by a Machine Learning algorithm.

🌱 Root Node

The Root Node is the first question asked by the Decision Tree. It is the top-most node of the tree. Every Decision Tree contains only one Root Node.

Example:

Weather = Hot ?

🌿 Internal Node

Internal Nodes are additional questions asked after the Root Node. They help separate mixed data.

Example:

Pocket Money > ₹300 ?

🔀 Branch

Branches connect nodes together. They represent answers to questions.

Question	Branches
Weather Hot?	Yes / No
Money > 300?	Yes / No

🍁 Leaf Node

Leaf Nodes contain final predictions. No further questions are asked.

🍕 Order Pizza

❌ Don't Order

✅ Pure Node

A Pure Node contains only one class.

YES
YES
YES
YES
YES

NO
NO
NO
NO
NO

Impurity = 0
Perfect Node

⚠️ Impure Node

Impure Nodes contain multiple classes.

YES
YES
NO
YES
NO

Mixed Data
Decision Tree will try to split this node.

⚖️ Pure Node vs Impure Node

Pure Node	Impure Node
Only One Class	Multiple Classes
Easy Prediction	Difficult Prediction
Impurity = 0	Impurity > 0
Preferred	Not Preferred

🏏 Cricket Match Dataset

Goal: Predict whether a cricket match will be played.

Weather	Ground Wet	Play Match
Sunny	No	Yes
Sunny	No	Yes
Rainy	Yes	No
Rainy	Yes	No
Cloudy	No	Yes

🚀 How a Tree is Built

Step 1
Load Data

Step 2
Ask Question

Step 3
Split Data

Step 4
Check Purity

Step 5
Split Again

Step 6
Leaf Node

📝 Part 2 Summary

Root Node is the first question asked.
Internal Nodes are additional questions.
Branches connect nodes.
Leaf Nodes contain final predictions.
Pure Nodes contain only one class.
Impure Nodes contain mixed classes.
Decision Trees split data to reduce impurity.

👉 Next: Part 3 – Gini Impurity, Entropy and Information Gain

PART 3A

🧮 Why Splitting Happens?

Understanding how a Decision Tree chooses the first question before learning Gini Impurity.

🎯 Our Goal

We want to predict:

🍕 Will a Student Order Pizza?

To answer this question, we have collected data.

📊 Pizza Ordering Dataset

Weather	Pocket Money	Friend Available	Order Pizza
Hot	500	Yes	Yes
Hot	400	Yes	Yes
Cold	100	No	No
Rainy	200	No	No
Rainy	500	Yes	Yes
Hot	600	No	Yes

😵 Step 1 : All Data is Mixed Together

Initially, the Decision Tree has not asked any questions. All records are inside a single node.

YES
YES
NO
NO
YES
YES

Some students ordered pizza. Some students did not order pizza.

The tree is confused.

🌳 Initial Tree

Currently all records are inside one node.

⚖️ Pure Node vs Impure Node

Pure Node

YES
YES
YES
YES

No confusion. All records belong to one class.

Impure Node

YES
YES
NO
YES
NO

Mixed classes. The tree is confused.

✂️ Step 2 : Why Does Splitting Happen?

Decision Trees hate confusion.

The goal of the tree is:

Convert Impure Nodes ➡ Into Pure Nodes

🧐 Step 3 : Which Question Should Be Asked First?

The tree can choose any feature.

🌤 Weather ?

💰 Pocket Money ?

👬 Friend Available ?

But which one is best?

The tree tests each feature one by one.

🌤 Step 4 : Testing Weather

The tree creates a temporary split.

Weather	Pizza Result
Hot	YES YES YES
Cold	NO
Rainy	YES NO

🔍 Step 5 : Analyze the Split

HOT

YES
YES
YES

Pure Node ✅

COLD

NO

Pure Node ✅

RAINY

YES
NO

Impure Node ❌

🤯 How Good Is This Split?

Weather created:

2 Pure Nodes
1 Impure Node

This seems good. But can another feature do even better?

The tree now tests Pocket Money. Then Friend Available.

📏 We Need a Score

The tree needs a numerical score to compare features.

Feature A → Good Split
Feature B → Better Split
Feature C → Best Split

How do we measure this?

Gini Impurity

In Part 3B, we will learn exactly how Gini is calculated.

📝 Part 3A Summary

All records start inside one node.
Mixed data creates confusion.
Mixed nodes are called Impure Nodes.
Decision Trees try to create Pure Nodes.
The tree tests every feature.
Each feature creates a temporary split.
The tree evaluates every split.
To compare splits, the tree uses Gini Impurity.

👉 Next: Part 3B – Gini Impurity Formula, Step-by-Step Calculations & Weighted Gini

PART 3B

🧮 Gini Impurity & Choosing the Best Split

How does a Decision Tree compare features and choose the Root Node?

🎯 Step 1 : We Need a Score

In Part 3A, we tested Weather. The split looked good.

But how can we prove it is good?

Decision Trees need a score.

This score is called:

Gini Impurity

🤔 What Does Gini Measure?

Gini measures confusion.

YES
YES
YES
YES

No confusion. Pure Node.

YES
YES
NO
NO

Confusing. Impure Node.

More confusion → Higher Gini
Less confusion → Lower Gini

📐 Gini Formula

Gini = 1 − P(Yes)² − P(No)²

P(Yes) = Probability of YES

P(No) = Probability of NO

🧪 Example 1 : Pure Node

YES
YES
YES
YES

Total Records = 4

P(Yes)=4/4=1

P(No)=0/4=0


Gini

= 1 - (1)² - (0)²

= 1 - 1 - 0

= 0

Gini = 0
Perfect Node

🧪 Example 2 : Impure Node

YES
NO

P(Yes)=1/2=0.5

P(No)=1/2=0.5


Gini

= 1 - (0.5)² - (0.5)²

= 1 - 0.25 - 0.25

= 0.5

Gini = 0.5
Very Mixed Node

🌤 Step 6 : Calculate Weather Split

HOT Node

YES
YES
YES


P(Yes)=1

P(No)=0

Gini

=1-(1²)-(0²)

=0

COLD Node


P(Yes)=0

P(No)=1

Gini

=0

RAINY Node

YES
NO


P(Yes)=0.5

P(No)=0.5

Gini

=1-(0.5²)-(0.5²)

=0.5

⚖️ Weighted Gini

Not every node contains the same number of records.

Node	Records	Gini
Hot	3	0
Cold	1	0
Rainy	2	0.5

🧮 Final Weather Gini


Weighted Gini

=

(3/6 × 0)

+

(1/6 × 0)

+

(2/6 × 0.5)

=

0.167

Weather Gini = 0.167

🏆 Comparing Features

Feature	Weighted Gini
Weather	0.167
Pocket Money	0.25
Friend Available	0.33

Lowest Gini Wins

🌳 Root Node Selection

Weather becomes the Root Node because it has the Lowest Gini.

📝 Part 3B Summary

Gini measures confusion.
Pure nodes have Gini = 0.
Mixed nodes have higher Gini.
Each split creates multiple nodes.
Each node gets its own Gini.
Weighted Gini combines all node scores.
The feature with the lowest Weighted Gini wins.
The winning feature becomes the Root Node.

👉 Next Part 4: Entropy & Information Gain