Machine Learning – Decision Tree

A Decision Tree is one of the most popular and intuitive algorithms in Machine Learning. It mimics human decision-making processes, making it simple yet powerful for both classification and regression tasks.

In this tutorial on The Coding College, we’ll explain Decision Trees, their components, and how to implement them using Python.

What Is a Decision Tree?

A Decision Tree is a tree-structured algorithm where:

• Each node represents a feature.
• Each branch represents a decision based on a feature value.
• Each leaf represents the final output (class or value).

It works by splitting the dataset into subsets based on feature values, creating a tree-like structure.

Advantages of Decision Trees

1. Easy to Interpret: Visual representation makes it intuitive.
2. Versatile: Can handle both numerical and categorical data.
3. No Scaling Required: Works well with unscaled data.

Key Components of a Decision Tree

1. Root Node: The starting point that contains the entire dataset.
2. Decision Nodes: Nodes where the dataset is split based on a feature.
3. Leaf Nodes: Nodes that provide the output (class or value).

Common Splitting Criteria

• Gini Impurity (default for classification in Scikit-Learn).
• Entropy (used in Information Gain).
• Mean Squared Error (for regression tasks).

Implementing Decision Trees in Python

Dataset Example

We’ll use the popular Iris dataset for this example.

from sklearn.datasets import load_iris
from sklearn.tree import DecisionTreeClassifier, plot_tree
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

# Load the dataset
iris = load_iris()
X = iris.data
y = iris.target

# Split the dataset
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Initialize the Decision Tree Classifier
dt_classifier = DecisionTreeClassifier(criterion="gini", max_depth=3, random_state=42)

# Train the model
dt_classifier.fit(X_train, y_train)

# Make predictions
y_pred = dt_classifier.predict(X_test)

# Evaluate the model
accuracy = accuracy_score(y_test, y_pred)
print(f"Model Accuracy: {accuracy}")

Visualizing the Decision Tree

You can visualize the tree to understand the splits and decisions:

import matplotlib.pyplot as plt

# Plot the decision tree
plt.figure(figsize=(12, 8))
plot_tree(dt_classifier, feature_names=iris.feature_names, class_names=iris.target_names, filled=True)
plt.show()

Decision Tree for Regression

Decision Trees are not just for classification; they work for regression too!

from sklearn.tree import DecisionTreeRegressor
from sklearn.metrics import mean_squared_error

# Example dataset
X = [[1], [2], [3], [4], [5]]
y = [1.2, 1.9, 3.2, 4.1, 5.3]

# Initialize the Decision Tree Regressor
dt_regressor = DecisionTreeRegressor(max_depth=2)

# Train the model
dt_regressor.fit(X, y)

# Make predictions
y_pred = dt_regressor.predict(X)

# Evaluate the model
mse = mean_squared_error(y, y_pred)
print(f"Mean Squared Error: {mse}")

Advantages and Disadvantages of Decision Trees

Advantages

• Intuitive and easy to explain.
• Handles both classification and regression tasks.
• Requires little preprocessing.

Disadvantages

• Overfitting: Deep trees can become too complex and fit noise.
• Bias: Simple splits might miss complex relationships.

Exercises

Exercise 1: Train a Decision Tree

Use the Wine dataset from Scikit-Learn to train a Decision Tree Classifier. Evaluate its accuracy and visualize the tree.

Exercise 2: Prune the Tree

Limit the depth of a Decision Tree to avoid overfitting. Compare results with and without pruning.

Exercise 3: Decision Tree Regression

Use the Boston Housing dataset to train a Decision Tree Regressor. Evaluate its performance using Mean Squared Error (MSE).

Decision Trees vs. Other Algorithms

Why Learn at The Coding College?

At The Coding College, we emphasize practical knowledge to help you understand and implement core Machine Learning algorithms like Decision Trees. Our tutorials are designed to make learning effective and enjoyable.

Conclusion

A Decision Tree is an intuitive and powerful algorithm that serves as the foundation for many advanced models like Random Forests and Gradient Boosting. By mastering this algorithm, you’ll have a versatile tool for tackling classification and regression problems.