Principal Component Analysis

Ready-to-run code, examples, unbiased analysis

Copy-paste starter code

This example uses the built-in breast cancer dataset from scikit-learn. The dataset has many numeric features. We will reduce them using PCA and check how much information is retained.

from sklearn.datasets import load_breast_cancer
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
import pandas as pd
import numpy as np

# Load sample data
data = load_breast_cancer()
X = data.data
feature_names = data.feature_names

# Convert to DataFrame for easier understanding
df = pd.DataFrame(X, columns=feature_names)
print("Original shape:")
print(df.shape)

# Scale the data before PCA
scaler = StandardScaler()
X_scaled = scaler.fit_transform(df)

# Apply PCA
pca = PCA()
X_pca = pca.fit_transform(X_scaled)

# Explained variance
explained_variance = pca.explained_variance_ratio_

# Show explained variance of first 10 components
for i, var in enumerate(explained_variance[:10], start=1):
    print(f"PC{i}: {var:.4f}")

print("\nTotal variance explained by first 2 components:")
print(explained_variance[:2].sum())

print("\nTotal variance explained by first 5 components:")
print(explained_variance[:5].sum())

What this code does

The dataset is first loaded into Python.Then the values are scaled using StandardScaler.

After scaling, PCA is applied. The code prints how much variance is explained by each principal component. The output helps us decide how many components may be useful. If the first few components explain a large percentage of the variance, PCA has successfully summarized the data.

PCA with only two components

The following code reduces the dataset into only two principal components.

This is useful for plotting and visualization.

from sklearn.datasets import load_breast_cancer
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
import pandas as pd

# Load data
data = load_breast_cancer()
X = data.data
y = data.target

# Scale data
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

# Reduce to 2 components
pca_2 = PCA(n_components=2)
X_pca_2 = pca_2.fit_transform(X_scaled)

# Create DataFrame
pca_df = pd.DataFrame(X_pca_2, columns=["PC1", "PC2"])
pca_df["target"] = y

print(pca_df.head())
print("\nExplained variance by 2 components:")
print(pca_2.explained_variance_ratio_)
print("\nTotal explained variance:")
print(pca_2.explained_variance_ratio_.sum())

Plot PCA output

import matplotlib.pyplot as plt
from sklearn.datasets import load_breast_cancer
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
import pandas as pd

# Load data
data = load_breast_cancer()
X = data.data
y = data.target

# Scale data
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

# PCA with 2 components
pca = PCA(n_components=2)
X_pca = pca.fit_transform(X_scaled)

# DataFrame
pca_df = pd.DataFrame(X_pca, columns=["PC1", "PC2"])
pca_df["target"] = y

# Plot
plt.figure(figsize=(8, 6))
plt.scatter(pca_df["PC1"], pca_df["PC2"], c=pca_df["target"])
plt.xlabel("Principal Component 1")
plt.ylabel("Principal Component 2")
plt.title("PCA: Data reduced to 2 components")
plt.colorbar(label="Target")
plt.show()

How to read the chart

Each point represents one record. The x-axis is the first principal component. The y-axis is the second principal component.

If the points form visible groups, PCA has helped reveal structure in the data.

If the groups overlap heavily, PCA may still have captured variance, but the first two components may not be enough to clearly separate the classes.

This is important because PCA does not know the target labels while creating components. It only looks at feature variation.

Choosing number of components

Instead of guessing the number of components, we can ask PCA to keep enough components to explain a required amount of variance.

For example, this code keeps enough components to explain 95% of the variance.

from sklearn.datasets import load_breast_cancer
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA

# Load data
data = load_breast_cancer()
X = data.data

# Scale data
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

# Keep 95% variance
pca = PCA(n_components=0.95)
X_pca = pca.fit_transform(X_scaled)

print("Original number of features:")
print(X.shape[1])
print("\nReduced number of components:")
print(X_pca.shape[1])
print("\nTotal explained variance:")
print(pca.explained_variance_ratio_.sum())

PCA before Logistic Regression

PCA is often used before another machine learning model.

Here is a simple example using PCA before Logistic Regression.

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.metrics import accuracy_score, confusion_matrix

# Load data
data = load_breast_cancer()
X = data.data
y = data.target

# Train-test split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42
)

# Pipeline: scaling + PCA + model
model = Pipeline([
    ("scaler", StandardScaler()),
    ("pca", PCA(n_components=0.95)),
    ("logistic", LogisticRegression(max_iter=5000))
])

# Train
model.fit(X_train, y_train)

# Predict
predictions = model.predict(X_test)
print("Accuracy:")
print(accuracy_score(y_test, predictions))
print("\nConfusion matrix:")
print(confusion_matrix(y_test, predictions))

# Check how many PCA components were used
pca_step = model.named_steps["pca"]
print("\nNumber of PCA components used:")
print(pca_step.n_components_)

Why Pipeline is used

The pipeline keeps the steps together.
First, the data is scaled.
Second, PCA is applied.
Third, Logistic Regression is trained.

This is a safer and cleaner method because the same transformations are applied properly during training and testing.

Try changing the same code

Change:

PCA(n_components=0.95)

to:

PCA(n_components=0.90)

or:

PCA(n_components=2)

Then compare the accuracy and confusion matrix. You may notice that reducing too much can hurt performance.
You may also notice that keeping too many components may not simplify the data enough. This is why PCA should be tested, not assumed.

PCA: Where it succeeds

1. When many numeric columns are related

PCA works well when many columns are correlated. For example, in finance data, income, spending, credit limit, and loan amount may be related.
In medical data, different measurements may move together. In manufacturing data, machine readings may be connected.

PCA can summarize these related features into fewer components.

2. When visualization is needed

PCA is very useful when we want to see high-dimensional data on a simple 2D plot. It may reveal clusters, separation, outliers, or unusual records. This makes PCA useful in the early stages of analysis.

3. When models are slow due to too many features

Some models become slower when there are many columns. PCA can reduce the number of columns before modelling. This may improve speed and reduce unnecessary complexity.

4. When multicollinearity is a problem

Multicollinearity means some input columns are strongly related to each other. This can confuse some models and make interpretation unstable. PCA transforms the original columns into new components that are not correlated with each other.
This can help certain modelling workflows.

5. When noise reduction is useful

Sometimes small variations in data are not useful. By keeping only the main components, PCA may remove weaker noise and preserve stronger patterns. This can help in image processing, sensor data, and exploratory analysis.

PCA: Where it fails or becomes risky

1. When the data is not scaled
PCA can fail badly if scaling is ignored. Columns with large numeric values may dominate the result. For example, salary may overpower age only because salary has larger numbers. This does not mean salary is more important. It may simply be larger in scale.

So scaling is usually required before PCA.

2. When business interpretation is important

PCA components are combinations of original columns. A component may contain parts of many features. This can make the result difficult to explain. If a manager asks, “Which exact factor caused this result?”, PCA may not give a simple answer.
In such cases, using original columns may be more understandable.

3. When the target depends on low-variance information

PCA keeps directions with high variance. But sometimes the most useful predictive signal may be in a low-variance feature. Since PCA does not look at the target variable, it may remove information that is important for prediction. This is one of the biggest cautions in supervised machine learning.

High variance does not always mean high business importance.

4. When relationships are highly non-linear

PCA is a linear method. It looks for straight-line directions in the data. If the real pattern is curved or complex, PCA may not capture it well.

In such cases, other methods may work better.

5. When categorical data is not handled properly

PCA is designed mainly for numeric data. If categorical columns like city, gender, department, or product type are used directly, PCA will not understand them properly. Categorical variables must be encoded carefully before PCA.

Even after encoding, interpretation may become difficult.

6. When too many components are removed

If we reduce the data too aggressively, we may lose important information. For example, reducing 50 columns to 2 components may be useful for a chart, but it may not be enough for prediction.

A 2D PCA plot is helpful for understanding, but not always enough for final modelling.

Unbiased practical summary in my view

PCA is not magic.

It is a useful mathematical technique for reducing many numeric columns into fewer new components. It succeeds when the dataset has many related numeric features and when the main patterns can be captured through linear directions.

It is especially useful for visualization, simplification, noise reduction, and preprocessing. But PCA can fail when the data is not scaled, when interpretation is more important than compression, when the target signal is hidden in low-variance features, or when the true pattern is non-linear.

A good analyst should not ask only:

Can I apply PCA?

A better question is:

Does PCA preserve the information I need for this specific purpose?

Use PCA as a tool, not as a compulsory step.

Final takeaway

Principal Component Analysis reduces many numeric features into fewer principal components.

It helps simplify data, visualize patterns, reduce noise, and support machine learning workflows. But the new components may be harder to explain than the original columns. PCA is most useful when we need structure and simplification. It is less useful when we need direct business explanation from original variables.

In simple words:

PCA helps us see the main shape of the data without carrying every small detail.