Principal Component Analysis

Machine Learning View.

In machine learning, Principal Component Analysis (PCA) is an unsupervised technique used for dimensionality reduction. It transforms many original features into fewer new features called principle components.

These new components are linear combinations of the original variables and are arranged in descending order of explained variance. The first component captures the most variation, the second captures the next most, and so on.

Why PCA is useful.
Real datasets often have many correlated columns. PCA removes redundancy and compresses the information into fewer dimensions, which helps with faster training, easier visualization, and noise reduction.

How PCA works conceptually.
PCA rotates the coordinate system. Instead of measuring data along old axes, it measures along new axes that capture maximum spread in the data. These axes are mutually orthogonal.

Centering and scaling.
PCA is sensitive to feature scale. Variables should usually be standardized before applying PCA, especially when units differ (for example age, income, and percentage).

Core idea.
PCA does not select existing columns. It creates new columns (components) that summarize the data with minimal information loss.

Explained variance ratio.
Each component has an explained variance ratio showing how much total information it captures. A cumulative ratio such as 90% or 95% is often used to decide how many components to keep.

PCA and prediction models.
PCA itself is not a prediction algorithm. It is a preprocessing step. After transforming data, the reduced components can be passed to models like logistic regression, SVM, or clustering algorithms.

Interpretability trade-off.
Components are combinations of many variables, so interpretation can be harder than using original features. PCA improves compression and numerical behavior but may reduce direct business explainability.

When to avoid PCA.
If features are already few, or direct interpretability is mandatory, PCA may not add value. Also, PCA is linear; if structure is highly non-linear, other techniques may be better.

Related Statistics page: Regression from Statistics Perspective

Important caution.
Fit PCA only on training data. Then apply the same fitted transform to validation and test data to avoid data leakage.

A simple mental picture.
Think of PCA as turning the camera angle to see the direction where data spreads most. Keep the strongest directions and drop weaker ones.

PCA is a dimensionality reduction method that builds new orthogonal components to preserve maximum variance with fewer features.

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score

data = load_breast_cancer()
X = data.data
y = data.target

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42
)

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

pca = PCA(n_components=0.95)  # keep 95% variance
X_train_pca = pca.fit_transform(X_train_scaled)
X_test_pca = pca.transform(X_test_scaled)

model = LogisticRegression(max_iter=5000)
model.fit(X_train_pca, y_train)

predictions = model.predict(X_test_pca)
print("Original feature count:", X_train.shape[1])
print("Reduced feature count:", X_train_pca.shape[1])
print("Explained variance ratio:", pca.explained_variance_ratio_)
print("Accuracy:", accuracy_score(y_test, predictions))