1.13. Feature selection

The classes in the sklearn.feature_selection module can be usedfor feature selection/dimensionality reduction on sample sets, either toimprove estimators’ accuracy scores or to boost their performance on veryhigh-dimensional datasets.

1.13.1. Removing features with low variance

VarianceThreshold is a simple baseline approach to feature selection.It removes all features whose variance doesn’t meet some threshold.By default, it removes all zero-variance features,i.e. features that have the same value in all samples.

As an example, suppose that we have a dataset with boolean features,and we want to remove all features that are either one or zero (on or off)in more than 80% of the samples.Boolean features are Bernoulli random variables,and the variance of such variables is given by

so we can select using the threshold .8 * (1 - .8):

>>>

>>> from sklearn.feature_selection import VarianceThreshold
>>> X = [[0, 0, 1], [0, 1, 0], [1, 0, 0], [0, 1, 1], [0, 1, 0], [0, 1, 1]]
>>> sel = VarianceThreshold(threshold=(.8 * (1 - .8)))
>>> sel.fit_transform(X)
array([[0, 1],
       [1, 0],
       [0, 0],
       [1, 1],
       [1, 0],
       [1, 1]])

As expected, VarianceThreshold has removed the first column,which has a probability

of containing a zero.

1.13.2. Univariate feature selection

Univariate feature selection works by selecting the best features based onunivariate statistical tests. It can be seen as a preprocessing stepto an estimator. Scikit-learn exposes feature selection routinesas objects that implement the transform method:

• SelectKBest removes all but the

  
  highest scoring features

• SelectPercentile removes all but a user-specified highest scoringpercentage of features

• using common univariate statistical tests for each feature:false positive rate SelectFpr, false discovery rateSelectFdr, or family wise error SelectFwe.

• GenericUnivariateSelect allows to perform univariate featureselection with a configurable strategy. This allows to select the bestunivariate selection strategy with hyper-parameter search estimator.

For instance, we can perform a

test to the samplesto retrieve only the two best features as follows:

>>>

>>> from sklearn.datasets import load_iris
>>> from sklearn.feature_selection import SelectKBest
>>> from sklearn.feature_selection import chi2
>>> X, y = load_iris(return_X_y=True)
>>> X.shape
(150, 4)
>>> X_new = SelectKBest(chi2, k=2).fit_transform(X, y)
>>> X_new.shape
(150, 2)

These objects take as input a scoring function that returns univariate scoresand p-values (or only scores for SelectKBest andSelectPercentile):

• For regression: f_regression, mutual_info_regression

• For classification: chi2, f_classif, mutual_info_classif

The methods based on F-test estimate the degree of linear dependency betweentwo random variables. On the other hand, mutual information methods can captureany kind of statistical dependency, but being nonparametric, they require moresamples for accurate estimation.

Feature selection with sparse data

If you use sparse data (i.e. data represented as sparse matrices),chi2, mutual_info_regression, mutual_info_classifwill deal with the data without making it dense.

Warning

Beware not to use a regression scoring function with a classificationproblem, you will get useless results.

Examples:

• Univariate Feature Selection

• Comparison of F-test and mutual information

1.13.3. Recursive feature elimination

Given an external estimator that assigns weights to features (e.g., thecoefficients of a linear model), recursive feature elimination (RFE)is to select features by recursively considering smaller and smaller sets offeatures. First, the estimator is trained on the initial set of features andthe importance of each feature is obtained either through a coef attributeor through a feature_importances attribute. Then, the least importantfeatures are pruned from current set of features.That procedure is recursivelyrepeated on the pruned set until the desired number of features to select iseventually reached.

RFECV performs RFE in a cross-validation loop to find the optimalnumber of features.

Examples:

• Recursive feature elimination: A recursive feature elimination exampleshowing the relevance of pixels in a digit classification task.

• Recursive feature elimination with cross-validation: A recursive featureelimination example with automatic tuning of the number of featuresselected with cross-validation.

1.13.4. Feature selection using SelectFromModel

SelectFromModel is a meta-transformer that can be used along with anyestimator that has a coef or feature_importances attribute after fitting.The features are considered unimportant and removed, if the correspondingcoef or feature_importances values are below the providedthreshold parameter. Apart from specifying the threshold numerically,there are built-in heuristics for finding a threshold using a string argument.Available heuristics are “mean”, “median” and float multiples of these like“0.1*mean”.

For examples on how it is to be used refer to the sections below.

Examples

• Feature selection using SelectFromModel and LassoCV: Selecting the twomost important features from the Boston dataset without knowing thethreshold beforehand.

1.13.4.1. L1-based feature selection

Linear models penalized with the L1 norm havesparse solutions: many of their estimated coefficients are zero. When the goalis to reduce the dimensionality of the data to use with another classifier,they can be used along with feature_selection.SelectFromModelto select the non-zero coefficients. In particular, sparse estimators usefulfor this purpose are the linear_model.Lasso for regression, andof linear_model.LogisticRegression and svm.LinearSVCfor classification:

>>>

>>> from sklearn.svm import LinearSVC
>>> from sklearn.datasets import load_iris
>>> from sklearn.feature_selection import SelectFromModel
>>> X, y = load_iris(return_X_y=True)
>>> X.shape
(150, 4)
>>> lsvc = LinearSVC(C=0.01, penalty="l1", dual=False).fit(X, y)
>>> model = SelectFromModel(lsvc, prefit=True)
>>> X_new = model.transform(X)
>>> X_new.shape
(150, 3)

With SVMs and logistic-regression, the parameter C controls the sparsity:the smaller C the fewer features selected. With Lasso, the higher thealpha parameter, the fewer features selected.

Examples:

• Classification of text documents using sparse features: Comparisonof different algorithms for document classification including L1-basedfeature selection.

L1-recovery and compressive sensing

For a good choice of alpha, the Lasso can fully recover theexact set of non-zero variables using only few observations, providedcertain specific conditions are met. In particular, the number ofsamples should be “sufficiently large”, or L1 models will perform atrandom, where “sufficiently large” depends on the number of non-zerocoefficients, the logarithm of the number of features, the amount ofnoise, the smallest absolute value of non-zero coefficients, and thestructure of the design matrix X. In addition, the design matrix mustdisplay certain specific properties, such as not being too correlated.

There is no general rule to select an alpha parameter for recovery ofnon-zero coefficients. It can by set by cross-validation(LassoCV or LassoLarsCV), though this may lead tounder-penalized models: including a small number of non-relevantvariables is not detrimental to prediction score. BIC(LassoLarsIC) tends, on the opposite, to set high values ofalpha.

Reference Richard G. Baraniuk “Compressive Sensing”, IEEE SignalProcessing Magazine [120] July 2007http://users.isr.ist.utl.pt/~aguiar/CS_notes.pdf

1.13.4.2. Tree-based feature selection

Tree-based estimators (see the sklearn.tree module and forestof trees in the sklearn.ensemble module) can be used to computefeature importances, which in turn can be used to discard irrelevantfeatures (when coupled with the sklearn.feature_selection.SelectFromModelmeta-transformer):

>>>

>>> from sklearn.ensemble import ExtraTreesClassifier
>>> from sklearn.datasets import load_iris
>>> from sklearn.feature_selection import SelectFromModel
>>> X, y = load_iris(return_X_y=True)
>>> X.shape
(150, 4)
>>> clf = ExtraTreesClassifier(n_estimators=50)
>>> clf = clf.fit(X, y)
>>> clf.feature_importances_  
array([ 0.04...,  0.05...,  0.4...,  0.4...])
>>> model = SelectFromModel(clf, prefit=True)
>>> X_new = model.transform(X)
>>> X_new.shape               
(150, 2)

Examples:

• Feature importances with forests of trees: example onsynthetic data showing the recovery of the actually meaningfulfeatures.

• Pixel importances with a parallel forest of trees: exampleon face recognition data.

1.13.5. Feature selection as part of a pipeline

Feature selection is usually used as a pre-processing step before doingthe actual learning. The recommended way to do this in scikit-learn isto use a sklearn.pipeline.Pipeline:

clf = Pipeline([
  ('feature_selection', SelectFromModel(LinearSVC(penalty="l1"))),
  ('classification', RandomForestClassifier())
])
clf.fit(X, y)

In this snippet we make use of a sklearn.svm.LinearSVCcoupled with sklearn.feature_selection.SelectFromModelto evaluate feature importances and select the most relevant features.Then, a sklearn.ensemble.RandomForestClassifier is trained on thetransformed output, i.e. using only relevant features. You can performsimilar operations with the other feature selection methods and alsoclassifiers that provide a way to evaluate feature importances of course.See the sklearn.pipeline.Pipeline examples for more details.