1.12. Multiclass and multilabel algorithms

Warning

All classifiers in scikit-learn do multiclass classificationout-of-the-box. You don’t need to use the sklearn.multiclass moduleunless you want to experiment with different multiclass strategies.

The sklearn.multiclass module implements meta-estimators to solvemulticlass and multilabel classification problemsby decomposing such problems into binary classification problems. multioutputregression is also supported.

  • Multiclass classification: classification task with more than two classes.Each sample can only be labelled as one class.

For example, classification using features extracted from a set of images offruit, where each image may either be of an orange, an apple, or a pear.Each image is one sample and is labelled as one of the 3 possible classes.Multiclass classification makes the assumption that each sample is assignedto one and only one label - one sample cannot, for example, be both a pearand an apple.

Valid multiclass representations fortype_of_target (y) are:

  • 1d or column vector containing more than two discrete values. Anexample of a vector y for 3 samples:

    >>>
    1. >>> import numpy as np>>> y = np.array(['apple', 'pear', 'apple'])>>> print(y)['apple' 'pear' 'apple']

  • sparse binary matrix of shape (n_samples, n_classes) with asingle element per row, where each column represents one class. Anexample of a sparse binary matrix y for 3 samples, wherethe columns, in order, are orange, apple and pear:

    >>>
    1. >>> from scipy import sparse>>> row_ind = np.array([0, 1, 2])>>> col_ind = np.array([1, 2, 1])>>> y_sparse = sparse.csr_matrix((np.ones(3), (row_ind, col_ind)))>>> print(y_sparse)  (0, 1)        1.0  (1, 2)        1.0  (2, 1)        1.0
  • Multilabel classification: classification task labelling each sample withx labels from nclasses possible classes, where x can be 0 ton_classes inclusive. This can be thought of as predicting properties of asample that are not mutually exclusive. Formally, a binary output is assignedto each class, for every sample. Positive classes are indicated with 1 andnegative classes with 0 or -1. It is thus comparable to running n_classesbinary classification tasks, for example withsklearn.multioutput.MultiOutputClassifier. This approach treatseach label independently whereas multilabel classifiers _may treat themultiple classes simultaneously, accounting for correlated behaviour amoungthem.

For example, prediction of the topics relevant to a text document or video.The document or video may be about one of ‘religion’, ‘politics’, ‘finance’or ‘education’, several of the topic classes or all of the topic classes.

Valid representation of multilabel y is either dense (or sparse)binary matrix of shape (n_samples, n_classes). Each columnrepresents a class. The 1’s in each row denote the positive classes asample has been labelled with. An example of a dense matrix y for 3samples:

>>>

  1. >>> y = np.array([[1, 0, 0, 1], [0, 0, 1, 1], [0, 0, 0, 0]])
  2. >>> print(y)
  3. [[1 0 0 1]
  4.  [0 0 1 1]
  5.  [0 0 0 0]]

An example of the same y in sparse matrix form:

>>>

  1. >>> y_sparse = sparse.csr_matrix(y)
  2. >>> print(y_sparse)
  3.   (0, 0)    1
  4.   (0, 3)    1
  5.   (1, 2)    1
  6.   (1, 3)    1
  • Multioutput regression: predicts multiple numerical properties for eachsample. Each property is a numerical variable and the number of propertiesto be predicted for each sample is greater than or equal to 2. Some estimatorsthat support multioutput regression are faster than just running n_outputestimators.

For example, prediction of both wind speed and wind direction, in degrees,using data obtained at a certain location. Each sample would be dataobtained at one location and both wind speed and directtion would beoutput for each sample.

Valid representation of multilabel y is dense matrix of shape(n_samples, n_classes) of floats. A column wise concatenation ofcontinuous variables. An example of y for 3 samples:

>>>

  1. >>> y = np.array([[31.4, 94], [40.5, 109], [25.0, 30]])
  2. >>> print(y)
  3. [[ 31.4  94. ]
  4.  [ 40.5 109. ]
  5.  [ 25.   30. ]]
  • Multioutput-multiclass classification(also known as multitask classification):classification task which labels each sample with a set of non-binaryproperties. Both the number of properties and the number ofclasses per property is greater than 2. A single estimator thushandles several joint classification tasks. This is both a generalization ofthe multilabel classification task, which only considers binaryattributes, as well as a generalization of the multiclass classificationtask, where only one property is considered.

For example, classification of the properties “type of fruit” and “colour”for a set of images of fruit. The property “type of fruit” has the possibleclasses: “apple”, “pear” and “orange”. The property “colour” has thepossible classes: “green”, “red”, “yellow” and “orange”. Each sample is animage of a fruit, a label is output for both properties and each label isone of the possible classes of the corresponding property.

Valid representation of multilabel y is dense matrix of shape(n_samples, n_classes) of floats. A column wise concatenation of 1dmulticlass variables. An example of y for 3 samples:

>>>

  1. >>> y = np.array([['apple', 'green'], ['orange', 'orange'], ['pear', 'green']])
  2. >>> print(y)
  3. [['apple' 'green']
  4.  ['orange' 'orange']
  5.  ['pear' 'green']]

Note that any classifiers handling multioutput-multiclass (also known asmultitask classification) tasks, support the multilabel classification taskas a special case. Multitask classification is similar to the multioutputclassification task with different model formulations. For more information,see the relevant estimator documentation.

All scikit-learn classifiers are capable of multiclass classification,but the meta-estimators offered by sklearn.multiclasspermit changing the way they handle more than two classesbecause this may have an effect on classifier performance(either in terms of generalization error or required computational resources).

Summary

Number oftargetsTargetcardinalityValidtype_of_target
Multiclassclassification1>2-‘multiclass’
Multilabelclassification>12 (0 or 1)-‘multilabel-indicator’
Multioutputregression>1Continuous-‘continuous-multioutput’
Multioutput-multiclassclassification>1>2-‘multiclass-multioutput’

Below is a summary of the classifiers supported by scikit-learngrouped by strategy; you don’t need the meta-estimators in this classif you’re using one of these, unless you want custom multiclass behavior:

Warning

At present, no metric in sklearn.metricssupports the multioutput-multiclass classification task.

1.12.1. Multilabel classification format

In multilabel learning, the joint set of binary classification tasks isexpressed with label binary indicator array: each sample is one row of a 2darray of shape (n_samples, n_classes) with binary values: the one, i.e. the nonzero elements, corresponds to the subset of labels. An array such asnp.array([[1, 0, 0], [0, 1, 1], [0, 0, 0]]) represents label 0 in the firstsample, labels 1 and 2 in the second sample, and no labels in the third sample.

Producing multilabel data as a list of sets of labels may be more intuitive.The MultiLabelBinarizertransformer can be used to convert between a collection of collections oflabels and the indicator format.

>>>

  1. >>> from sklearn.preprocessing import MultiLabelBinarizer
  2. >>> y = [[2, 3, 4], [2], [0, 1, 3], [0, 1, 2, 3, 4], [0, 1, 2]]
  3. >>> MultiLabelBinarizer().fit_transform(y)
  4. array([[0, 0, 1, 1, 1],
  5.        [0, 0, 1, 0, 0],
  6.        [1, 1, 0, 1, 0],
  7.        [1, 1, 1, 1, 1],
  8.        [1, 1, 1, 0, 0]])

1.12.2. One-Vs-The-Rest

This strategy, also known as one-vs-all, is implemented inOneVsRestClassifier. The strategy consists in fitting one classifierper class. For each classifier, the class is fitted against all the otherclasses. In addition to its computational efficiency (only n_classesclassifiers are needed), one advantage of this approach is itsinterpretability. Since each class is represented by one and only one classifier,it is possible to gain knowledge about the class by inspecting itscorresponding classifier. This is the most commonly used strategy and is a fairdefault choice.

1.12.2.1. Multiclass learning

Below is an example of multiclass learning using OvR:

>>>

  1. >>> from sklearn import datasets
  2. >>> from sklearn.multiclass import OneVsRestClassifier
  3. >>> from sklearn.svm import LinearSVC
  4. >>> X, y = datasets.load_iris(return_X_y=True)
  5. >>> OneVsRestClassifier(LinearSVC(random_state=0)).fit(X, y).predict(X)
  6. array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
  7.        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
  8.        0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
  9.        1, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 1, 1, 1, 1, 1, 1, 1,
  10.        1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
  11.        2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 2, 2, 2, 1, 2, 2, 2, 2,
  12.        2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2])

1.12.2.2. Multilabel learning

OneVsRestClassifier also supports multilabel classification.To use this feature, feed the classifier an indicator matrix, in which cell[i, j] indicates the presence of label j in sample i.

../_images/sphx_glr_plot_multilabel_0011.png

Examples:

1.12.3. One-Vs-One

OneVsOneClassifier constructs one classifier per pair of classes.At prediction time, the class which received the most votes is selected.In the event of a tie (among two classes with an equal number of votes), itselects the class with the highest aggregate classification confidence bysumming over the pair-wise classification confidence levels computed by theunderlying binary classifiers.

Since it requires to fit n_classes * (n_classes - 1) / 2 classifiers,this method is usually slower than one-vs-the-rest, due to itsO(n_classes^2) complexity. However, this method may be advantageous foralgorithms such as kernel algorithms which don’t scale well withn_samples. This is because each individual learning problem only involvesa small subset of the data whereas, with one-vs-the-rest, the completedataset is used n_classes times. The decision function is the resultof a monotonic transformation of the one-versus-one classification.

1.12.3.1. Multiclass learning

Below is an example of multiclass learning using OvO:

>>>

  1. >>> from sklearn import datasets
  2. >>> from sklearn.multiclass import OneVsOneClassifier
  3. >>> from sklearn.svm import LinearSVC
  4. >>> X, y = datasets.load_iris(return_X_y=True)
  5. >>> OneVsOneClassifier(LinearSVC(random_state=0)).fit(X, y).predict(X)
  6. array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
  7.        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
  8.        0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
  9.        1, 2, 1, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 1, 1, 1, 1, 1, 1, 1, 1,
  10.        1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
  11.        2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
  12.        2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2])

References:

  • “Pattern Recognition and Machine Learning. Springer”,Christopher M. Bishop, page 183, (First Edition)

1.12.4. Error-Correcting Output-Codes

Output-code based strategies are fairly different from one-vs-the-rest andone-vs-one. With these strategies, each class is represented in a Euclideanspace, where each dimension can only be 0 or 1. Another way to put it isthat each class is represented by a binary code (an array of 0 and 1). Thematrix which keeps track of the location/code of each class is called thecode book. The code size is the dimensionality of the aforementioned space.Intuitively, each class should be represented by a code as unique aspossible and a good code book should be designed to optimize classificationaccuracy. In this implementation, we simply use a randomly-generated codebook as advocated in 3 although more elaborate methods may be added in thefuture.

At fitting time, one binary classifier per bit in the code book is fitted.At prediction time, the classifiers are used to project new points in theclass space and the class closest to the points is chosen.

In OutputCodeClassifier, the code_size attribute allows the user tocontrol the number of classifiers which will be used. It is a percentage of thetotal number of classes.

A number between 0 and 1 will require fewer classifiers thanone-vs-the-rest. In theory, log2(n_classes) / n_classes is sufficient torepresent each class unambiguously. However, in practice, it may not lead togood accuracy since log2(n_classes) is much smaller than n_classes.

A number greater than 1 will require more classifiers thanone-vs-the-rest. In this case, some classifiers will in theory correct forthe mistakes made by other classifiers, hence the name “error-correcting”.In practice, however, this may not happen as classifier mistakes willtypically be correlated. The error-correcting output codes have a similareffect to bagging.

1.12.4.1. Multiclass learning

Below is an example of multiclass learning using Output-Codes:

>>>

  1. >>> from sklearn import datasets
  2. >>> from sklearn.multiclass import OutputCodeClassifier
  3. >>> from sklearn.svm import LinearSVC
  4. >>> X, y = datasets.load_iris(return_X_y=True)
  5. >>> clf = OutputCodeClassifier(LinearSVC(random_state=0),
  6. ...                            code_size=2, random_state=0)
  7. >>> clf.fit(X, y).predict(X)
  8. array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
  9.        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
  10.        0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 1, 1,
  11.        1, 2, 1, 1, 1, 1, 1, 1, 2, 1, 1, 1, 1, 1, 2, 2, 2, 1, 1, 1, 1, 1, 1,
  12.        1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
  13.        2, 2, 2, 2, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 2, 2, 2, 1, 1, 2, 2, 2,
  14.        2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2])

References:

  • “Solving multiclass learning problems via error-correcting output codes”,Dietterich T., Bakiri G.,Journal of Artificial Intelligence Research 2,1995.
  • 3
  • “The error coding method and PICTs”,James G., Hastie T.,Journal of Computational and Graphical statistics 7,1998.
  • “The Elements of Statistical Learning”,Hastie T., Tibshirani R., Friedman J., page 606 (second-edition)2008.

1.12.5. Multioutput regression

Multioutput regression support can be added to any regressor withMultiOutputRegressor. This strategy consists of fitting oneregressor per target. Since each target is represented by exactly oneregressor it is possible to gain knowledge about the target byinspecting its corresponding regressor. AsMultiOutputRegressor fits one regressor per target it can nottake advantage of correlations between targets.

Below is an example of multioutput regression:

>>>

  1. >>> from sklearn.datasets import make_regression
  2. >>> from sklearn.multioutput import MultiOutputRegressor
  3. >>> from sklearn.ensemble import GradientBoostingRegressor
  4. >>> X, y = make_regression(n_samples=10, n_targets=3, random_state=1)
  5. >>> MultiOutputRegressor(GradientBoostingRegressor(random_state=0)).fit(X, y).predict(X)
  6. array([[-154.75474165, -147.03498585,  -50.03812219],
  7.        [   7.12165031,    5.12914884,  -81.46081961],
  8.        [-187.8948621 , -100.44373091,   13.88978285],
  9.        [-141.62745778,   95.02891072, -191.48204257],
  10.        [  97.03260883,  165.34867495,  139.52003279],
  11.        [ 123.92529176,   21.25719016,   -7.84253   ],
  12.        [-122.25193977,  -85.16443186, -107.12274212],
  13.        [ -30.170388  ,  -94.80956739,   12.16979946],
  14.        [ 140.72667194,  176.50941682,  -17.50447799],
  15.        [ 149.37967282,  -81.15699552,   -5.72850319]])

1.12.6. Multioutput classification

Multioutput classification support can be added to any classifier withMultiOutputClassifier. This strategy consists of fitting oneclassifier per target. This allows multiple target variableclassifications. The purpose of this class is to extend estimatorsto be able to estimate a series of target functions (f1,f2,f3…,fn)that are trained on a single X predictor matrix to predict a seriesof responses (y1,y2,y3…,yn).

Below is an example of multioutput classification:

>>>

  1. >>> from sklearn.datasets import make_classification
  2. >>> from sklearn.multioutput import MultiOutputClassifier
  3. >>> from sklearn.ensemble import RandomForestClassifier
  4. >>> from sklearn.utils import shuffle
  5. >>> import numpy as np
  6. >>> X, y1 = make_classification(n_samples=10, n_features=100, n_informative=30, n_classes=3, random_state=1)
  7. >>> y2 = shuffle(y1, random_state=1)
  8. >>> y3 = shuffle(y1, random_state=2)
  9. >>> Y = np.vstack((y1, y2, y3)).T
  10. >>> n_samples, n_features = X.shape # 10,100
  11. >>> n_outputs = Y.shape[1] # 3
  12. >>> n_classes = 3
  13. >>> forest = RandomForestClassifier(random_state=1)
  14. >>> multi_target_forest = MultiOutputClassifier(forest, n_jobs=-1)
  15. >>> multi_target_forest.fit(X, Y).predict(X)
  16. array([[2, 2, 0],
  17.        [1, 2, 1],
  18.        [2, 1, 0],
  19.        [0, 0, 2],
  20.        [0, 2, 1],
  21.        [0, 0, 2],
  22.        [1, 1, 0],
  23.        [1, 1, 1],
  24.        [0, 0, 2],
  25.        [2, 0, 0]])

1.12.7. Classifier Chain

Classifier chains (see ClassifierChain) are a way of combining anumber of binary classifiers into a single multi-label model that is capableof exploiting correlations among targets.

For a multi-label classification problem with N classes, N binaryclassifiers are assigned an integer between 0 and N-1. These integersdefine the order of models in the chain. Each classifier is then fit on theavailable training data plus the true labels of the classes whosemodels were assigned a lower number.

When predicting, the true labels will not be available. Instead thepredictions of each model are passed on to the subsequent models in thechain to be used as features.

Clearly the order of the chain is important. The first model in the chainhas no information about the other labels while the last model in the chainhas features indicating the presence of all of the other labels. In generalone does not know the optimal ordering of the models in the chain sotypically many randomly ordered chains are fit and their predictions areaveraged together.

References:

  • Jesse Read, Bernhard Pfahringer, Geoff Holmes, Eibe Frank,
  • “Classifier Chains for Multi-label Classification”, 2009.

1.12.8. Regressor Chain

Regressor chains (see RegressorChain) is analogous toClassifierChain as a way of combining a number of regressionsinto a single multi-target model that is capable of exploitingcorrelations among targets.