Decission Tree With Example

Decision Tree Code and Description

** Please type code into your code window,
instead of copying and pasting
-this can help you understand the process better **

Section 1: Imports

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.tree import DecisionTreeClassifier, plot_tree, export_text
from sklearn.metrics import accuracy_score, confusion_matrix, classification_report

We import dataset tools, split helpers, Decision Tree classifier, visualization helpers, and evaluation metrics.

Section 2: Load Data

data = load_breast_cancer()
X = pd.DataFrame(data.data, columns=data.feature_names)
y = data.target

print("Shape:", X.shape)
print("Target classes:", np.unique(y))

Breast cancer dataset is a binary classification problem and a reliable starter dataset for tree modeling.

Section 3: Train-Test Split

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

Stratified split keeps class proportions stable in both train and test sets.

Section 4: Baseline Decision Tree

dt_base = DecisionTreeClassifier(random_state=42)
dt_base.fit(X_train, y_train)

y_pred_base = dt_base.predict(X_test)
print("Baseline Accuracy:", round(accuracy_score(y_test, y_pred_base), 4))
print("Baseline Confusion Matrix:\n", confusion_matrix(y_test, y_pred_base))

Baseline tree gives a quick starting point and usually overfits when depth is not controlled.

Section 5: Hyperparameter Tuning

param_grid = {
    "max_depth": [3, 4, 5, 6, None],
    "min_samples_split": [2, 5, 10],
    "min_samples_leaf": [1, 2, 4, 8],
    "criterion": ["gini", "entropy"]
}

grid = GridSearchCV(
    DecisionTreeClassifier(random_state=42),
    param_grid=param_grid,
    scoring="f1",
    cv=5,
    n_jobs=-1
)
grid.fit(X_train, y_train)

best_dt = grid.best_estimator_
y_pred_best = best_dt.predict(X_test)

print("Best Params:", grid.best_params_)
print("Tuned Accuracy:", round(accuracy_score(y_test, y_pred_best), 4))
print("Classification Report:\n", classification_report(y_test, y_pred_best))

Grid search helps choose practical settings instead of manually guessing depth and leaf-size controls.

Section 6: Cost-Complexity Pruning (ccp_alpha)

path = best_dt.cost_complexity_pruning_path(X_train, y_train)
ccp_alphas = path.ccp_alphas

alpha_scores = []
for alpha in ccp_alphas:
    dt_alpha = DecisionTreeClassifier(
        random_state=42,
        ccp_alpha=alpha,
        max_depth=grid.best_params_["max_depth"],
        min_samples_split=grid.best_params_["min_samples_split"],
        min_samples_leaf=grid.best_params_["min_samples_leaf"],
        criterion=grid.best_params_["criterion"]
    )
    dt_alpha.fit(X_train, y_train)
    alpha_scores.append((alpha, dt_alpha.score(X_test, y_test)))

best_alpha, best_alpha_score = max(alpha_scores, key=lambda x: x[1])
print("Best ccp_alpha:", best_alpha)
print("Best pruned accuracy:", round(best_alpha_score, 4))

Pruning removes weak branches and can improve generalization while making the tree easier to explain.

Section 7: Feature Importance and Rule Export

dt_final = DecisionTreeClassifier(
    random_state=42,
    ccp_alpha=best_alpha,
    max_depth=grid.best_params_["max_depth"],
    min_samples_split=grid.best_params_["min_samples_split"],
    min_samples_leaf=grid.best_params_["min_samples_leaf"],
    criterion=grid.best_params_["criterion"]
)
dt_final.fit(X_train, y_train)

importances = pd.Series(dt_final.feature_importances_, index=X.columns)
print(importances.sort_values(ascending=False).head(10))

print("\nTop rules (depth limited):")
print(export_text(dt_final, feature_names=list(X.columns), max_depth=3))

Feature importance highlights key split drivers; exported rules make model behavior human-readable.