What a Decision Tree Really Does
A Decision Tree converts a prediction problem into a sequence of simple questions. Each question splits the data into smaller groups. The model keeps splitting until each final group is reasonably pure (classification) or consistent in value (regression).
Internal Components
Root node: full dataset starts here.
Decision node: a condition like feature_j <= threshold.
Branch: outcome of a condition (true/false path).
Leaf node: final prediction (class or numeric value).
How a Split Is Chosen
The algorithm tries many candidate splits and scores each one. It picks the split that maximally reduces impurity/error in child nodes.
For classification, common split scores are Gini and Entropy. For regression, split quality is usually based on variance/MSE reduction.
Conceptual Example (Classification)
Suppose we classify loan risk.
Node 1 asks: repayment_score < 620?
If yes, many defaults may cluster there. If no, safer profiles cluster on the other side. Then each side is split again using the next best question.
Eventually one leaf may say: "High risk (default probability 0.82)", another leaf may say: "Low risk (default probability 0.08)".
Conceptual Example (Regression)
Suppose we predict house price.
Root split: area <= 1200. Left side contains smaller homes, right side larger homes. Then location and age splits refine each side.
Leaf output is typically the mean target value of that leaf’s training samples.
Why Trees Feel Intuitive
Each path from root to leaf is a readable rule. Example: if area>1200 and location_score>8 and age<10 then price_segment=high.
This readability is why trees are popular in domains needing explanation.
Where Trees Fail Without Controls
A fully grown tree can memorize training noise. This is overfitting: excellent training score, weaker unseen-data performance.
How We Control Complexity
max_depth: limits levels.
min_samples_split: stops tiny-node splitting.
min_samples_leaf: avoids fragile leaves with too few samples.
ccp_alpha: post-pruning to remove weak branches.
Internal Working Sequence
1. Start with all data at root.
2. Evaluate candidate splits.
3. Pick best split by impurity/error reduction.
4. Recurse on each child.
5. Stop based on limits/gain.
6. Output prediction at leaves.
Fundamental Misconceptions
Misconception: "More depth is always better."
Reality: More depth can hurt generalization.
Misconception: "High feature importance means causal truth."
Reality: Importance is model-specific influence, not guaranteed causality.
Mental Model to Retain
Decision Tree is a greedy rule builder. At each step it makes the best local split decision, not a perfect global plan. Good hyperparameter control and validation turn this greedy process into a reliable model.
Visual 1: Decision Tree Picture
This picture shows the internal flow: root question -> branch decisions -> leaf predictions. Each level narrows uncertainty by partitioning data into more homogeneous groups.
Visual 2: Entropy Graph
Entropy is highest when classes are mixed (around p=0.5) and lowest when a node is pure (p near 0 or 1). Tree splits aim to move child nodes toward lower entropy.
Next step: After this page, continue with DECISSION TREE - With Example for full coding flow.