Linear Feature Extraction Techniques (in Machine Learning)

Feature extraction is used in dimensionality reduction to transform the data from a high-dimensional space to one with fewer dimensions. Linear feature extraction techniques are used when the relationships between the features follow a linear pattern.

Feature extraction methods are divided in non-linear or linear approaches. The 4 most common linear feature extraction techniques are:

1. Principal Component Analysis (PCA)
2. Non-negative matrix factorization (NMF)
3. Linear Discriminant Analysis (LDA)

1. Principal component analysis (PCA)

Principal component analysis, or PCA, is the main linear technique for dimension reduction.

The linear mapping of the data to a lower-dimensional space is performed in a way that maximizes the variance of the data.

PCA assumes that features with low variance are irrelevant and features with high variance are informative. PCA models are difficult to interpret.

2. Non-negative matrix factorization (NMF)

Non-negative matrix factorization, or NMF, is a dimension reduction technique that combines the product of non-negative features into a single one.

Interpretability of the NMF model

NMF will decompose documents and images into common patterns. Because of the decomposition of the documents, the NMF models are easy to interpret.

When to use NMF dimension reduction?

Use NMF on non-negative features such as word frequency array, recommender systems, purchase history on e-commerce sites and computer vision.

3. Linear Discriminant Analysis (LDA)

Linear discriminant analysis, or LDA, is a linear dimensional reduction technique used in the preprocessing for machine learning.

It is a generalization of Fisher’s linear discriminant. It works similarly to PCA but focuses on maximizing the separability within two or more known categories.

This is it, next, you should follow this tutorial to find out about common Non-Linear Feature Extraction Techniques used in Dimensionality reduction.