3.3. Metrics and scoring: quantifying the quality of predictions

There are 3 different APIs for evaluating the quality of a model’spredictions:

• Estimator score method: Estimators have a score method providing adefault evaluation criterion for the problem they are designed to solve.This is not discussed on this page, but in each estimator’s documentation.

• Scoring parameter: Model-evaluation tools usingcross-validation (such asmodel_selection.cross_val_score andmodel_selection.GridSearchCV) rely on an internal scoring strategy.This is discussed in the section The scoring parameter: defining model evaluation rules.

• Metric functions: The metrics module implements functionsassessing prediction error for specific purposes. These metrics are detailedin sections on Classification metrics,Multilabel ranking metrics, Regression metrics andClustering metrics.

Finally, Dummy estimators are useful to get a baselinevalue of those metrics for random predictions.

See also

For “pairwise” metrics, between samples and not estimators orpredictions, see the Pairwise metrics, Affinities and Kernels section.

3.3.1. The scoring parameter: defining model evaluation rules

Model selection and evaluation using tools, such asmodel_selection.GridSearchCV andmodel_selection.cross_val_score, take a scoring parameter thatcontrols what metric they apply to the estimators evaluated.

3.3.1.1. Common cases: predefined values

For the most common use cases, you can designate a scorer object with thescoring parameter; the table below shows all possible values.All scorer objects follow the convention that higher return values are betterthan lower return values. Thus metrics which measure the distance betweenthe model and the data, like metrics.mean_squared_error, areavailable as neg_mean_squared_error which return the negated valueof the metric.

Usage examples:

>>>

>>> from sklearn import svm, datasets
>>> from sklearn.model_selection import cross_val_score
>>> X, y = datasets.load_iris(return_X_y=True)
>>> clf = svm.SVC(random_state=0)
>>> cross_val_score(clf, X, y, cv=5, scoring='recall_macro')
array([0.96..., 0.96..., 0.96..., 0.93..., 1.        ])
>>> model = svm.SVC()
>>> cross_val_score(model, X, y, cv=5, scoring='wrong_choice')
Traceback (most recent call last):
ValueError: 'wrong_choice' is not a valid scoring value. Use sorted(sklearn.metrics.SCORERS.keys()) to get valid options.

Note

The values listed by the ValueError exception correspond to the functions measuringprediction accuracy described in the following sections.The scorer objects for those functions are stored in the dictionarysklearn.metrics.SCORERS.

3.3.1.2. Defining your scoring strategy from metric functions

The module sklearn.metrics also exposes a set of simple functionsmeasuring a prediction error given ground truth and prediction:

• functions ending with _score return a value tomaximize, the higher the better.

• functions ending with _error or _loss return avalue to minimize, the lower the better. When convertinginto a scorer object using make_scorer, setthe greater_is_better parameter to False (True by default; see theparameter description below).

Metrics available for various machine learning tasks are detailed in sectionsbelow.

Many metrics are not given names to be used as scoring values,sometimes because they require additional parameters, such asfbeta_score. In such cases, you need to generate an appropriatescoring object. The simplest way to generate a callable object for scoringis by using make_scorer. That function converts metricsinto callables that can be used for model evaluation.

One typical use case is to wrap an existing metric function from the librarywith non-default values for its parameters, such as the beta parameter forthe fbeta_score function:

>>>

>>> from sklearn.metrics import fbeta_score, make_scorer
>>> ftwo_scorer = make_scorer(fbeta_score, beta=2)
>>> from sklearn.model_selection import GridSearchCV
>>> from sklearn.svm import LinearSVC
>>> grid = GridSearchCV(LinearSVC(), param_grid={'C': [1, 10]},
...                     scoring=ftwo_scorer, cv=5)

The second use case is to build a completely custom scorer objectfrom a simple python function using make_scorer, which cantake several parameters:

• the python function you want to use (my_custom_loss_funcin the example below)

• whether the python function returns a score (greater_is_better=True,the default) or a loss (greater_is_better=False). If a loss, the outputof the python function is negated by the scorer object, conforming tothe cross validation convention that scorers return higher values for better models.

• for classification metrics only: whether the python function you provided requires continuous decisioncertainties (needs_threshold=True). The default value isFalse.

• any additional parameters, such as beta or labels in f1_score.

Here is an example of building custom scorers, and of using thegreater_is_better parameter:

>>>

>>> import numpy as np
>>> def my_custom_loss_func(y_true, y_pred):
...     diff = np.abs(y_true - y_pred).max()
...     return np.log1p(diff)
...
>>> # score will negate the return value of my_custom_loss_func,
>>> # which will be np.log(2), 0.693, given the values for X
>>> # and y defined below.
>>> score = make_scorer(my_custom_loss_func, greater_is_better=False)
>>> X = [[1], [1]]
>>> y = [0, 1]
>>> from sklearn.dummy import DummyClassifier
>>> clf = DummyClassifier(strategy='most_frequent', random_state=0)
>>> clf = clf.fit(X, y)
>>> my_custom_loss_func(clf.predict(X), y)
0.69...
>>> score(clf, X, y)
-0.69...

3.3.1.3. Implementing your own scoring object

You can generate even more flexible model scorers by constructing your ownscoring object from scratch, without using the make_scorer factory.For a callable to be a scorer, it needs to meet the protocol specified bythe following two rules:

• It can be called with parameters (estimator, X, y), where estimatoris the model that should be evaluated, X is validation data, and y isthe ground truth target for X (in the supervised case) or None (in theunsupervised case).

• It returns a floating point number that quantifies theestimator prediction quality on X, with reference to y.Again, by convention higher numbers are better, so if your scorerreturns loss, that value should be negated.

Note

Using custom scorers in functions where n_jobs > 1

While defining the custom scoring function alongside the calling functionshould work out of the box with the default joblib backend (loky),importing it from another module will be a more robust approach and workindependently of the joblib backend.

For example, to use n_jobs greater than 1 in the example below,custom_scoring_function function is saved in a user-created module(custom_scorer_module.py) and imported:

>>>

>>> from custom_scorer_module import custom_scoring_function 
>>> cross_val_score(model,
...  X_train,
...  y_train,
...  scoring=make_scorer(custom_scoring_function, greater_is_better=False),
...  cv=5,
...  n_jobs=-1) 

3.3.1.4. Using multiple metric evaluation

Scikit-learn also permits evaluation of multiple metrics in GridSearchCV,RandomizedSearchCV and cross_validate.

There are two ways to specify multiple scoring metrics for the scoringparameter:

• • As an iterable of string metrics::

>>>

>>> scoring = ['accuracy', 'precision']

• • As a dict mapping the scorer name to the scoring function::

>>>

>>> from sklearn.metrics import accuracy_score
>>> from sklearn.metrics import make_scorer
>>> scoring = {'accuracy': make_scorer(accuracy_score),
...            'prec': 'precision'}

Note that the dict values can either be scorer functions or one of thepredefined metric strings.

Currently only those scorer functions that return a single score can be passedinside the dict. Scorer functions that return multiple values are notpermitted and will require a wrapper to return a single metric:

>>>

>>> from sklearn.model_selection import cross_validate
>>> from sklearn.metrics import confusion_matrix
>>> # A sample toy binary classification dataset
>>> X, y = datasets.make_classification(n_classes=2, random_state=0)
>>> svm = LinearSVC(random_state=0)
>>> def tn(y_true, y_pred): return confusion_matrix(y_true, y_pred)[0, 0]
>>> def fp(y_true, y_pred): return confusion_matrix(y_true, y_pred)[0, 1]
>>> def fn(y_true, y_pred): return confusion_matrix(y_true, y_pred)[1, 0]
>>> def tp(y_true, y_pred): return confusion_matrix(y_true, y_pred)[1, 1]
>>> scoring = {'tp': make_scorer(tp), 'tn': make_scorer(tn),
...            'fp': make_scorer(fp), 'fn': make_scorer(fn)}
>>> cv_results = cross_validate(svm.fit(X, y), X, y, cv=5, scoring=scoring)
>>> # Getting the test set true positive scores
>>> print(cv_results['test_tp'])
[10  9  8  7  8]
>>> # Getting the test set false negative scores
>>> print(cv_results['test_fn'])
[0 1 2 3 2]

3.3.2. Classification metrics

The sklearn.metrics module implements several loss, score, and utilityfunctions to measure classification performance.Some metrics might require probability estimates of the positive class,confidence values, or binary decisions values.Most implementations allow each sample to provide a weighted contributionto the overall score, through the sample_weight parameter.

Some of these are restricted to the binary classification case:

Others also work in the multiclass case:

Some also work in the multilabel case:

And some work with binary and multilabel (but not multiclass) problems:

In the following sub-sections, we will describe each of those functions,preceded by some notes on common API and metric definition.

3.3.2.1. From binary to multiclass and multilabel

Some metrics are essentially defined for binary classification tasks (e.g.f1_score, roc_auc_score). In these cases, by defaultonly the positive label is evaluated, assuming by default that the positiveclass is labelled 1 (though this may be configurable through thepos_label parameter).

In extending a binary metric to multiclass or multilabel problems, the datais treated as a collection of binary problems, one for each class.There are then a number of ways to average binary metric calculations acrossthe set of classes, each of which may be useful in some scenario.Where available, you should select among these using the average parameter.

• "macro" simply calculates the mean of the binary metrics,giving equal weight to each class. In problems where infrequent classesare nonetheless important, macro-averaging may be a means of highlightingtheir performance. On the other hand, the assumption that all classes areequally important is often untrue, such that macro-averaging willover-emphasize the typically low performance on an infrequent class.

• "weighted" accounts for class imbalance by computing the average ofbinary metrics in which each class’s score is weighted by its presence in thetrue data sample.

• "micro" gives each sample-class pair an equal contribution to the overallmetric (except as a result of sample-weight). Rather than summing themetric per class, this sums the dividends and divisors that make up theper-class metrics to calculate an overall quotient.Micro-averaging may be preferred in multilabel settings, includingmulticlass classification where a majority class is to be ignored.

• "samples" applies only to multilabel problems. It does not calculate aper-class measure, instead calculating the metric over the true and predictedclasses for each sample in the evaluation data, and returning their(sample_weight-weighted) average.

• Selecting average=None will return an array with the score for eachclass.

While multiclass data is provided to the metric, like binary targets, as anarray of class labels, multilabel data is specified as an indicator matrix,in which cell [i, j] has value 1 if sample i has label j and value0 otherwise.

3.3.2.2. Accuracy score

The accuracy_score function computes theaccuracy, either the fraction(default) or the count (normalize=False) of correct predictions.

In multilabel classification, the function returns the subset accuracy. Ifthe entire set of predicted labels for a sample strictly match with the trueset of labels, then the subset accuracy is 1.0; otherwise it is 0.0.

If

is the predicted value ofthe-th sample and is the corresponding true value,then the fraction of correct predictions over isdefined as

where

is the indicator function.

>>>

>>> import numpy as np
>>> from sklearn.metrics import accuracy_score
>>> y_pred = [0, 2, 1, 3]
>>> y_true = [0, 1, 2, 3]
>>> accuracy_score(y_true, y_pred)
0.5
>>> accuracy_score(y_true, y_pred, normalize=False)
2

In the multilabel case with binary label indicators:

>>>

>>> accuracy_score(np.array([[0, 1], [1, 1]]), np.ones((2, 2)))
0.5

Example:

• See Test with permutations the significance of a classification scorefor an example of accuracy score usage using permutations ofthe dataset.

3.3.2.3. Balanced accuracy score

The balanced_accuracy_score function computes the balanced accuracy, which avoids inflatedperformance estimates on imbalanced datasets. It is the macro-average of recallscores per class or, equivalently, raw accuracy where each sample is weightedaccording to the inverse prevalence of its true class.Thus for balanced datasets, the score is equal to accuracy.

In the binary case, balanced accuracy is equal to the arithmetic mean ofsensitivity(true positive rate) and specificity (true negativerate), or the area under the ROC curve with binary predictions rather thanscores.

If the classifier performs equally well on either class, this term reduces tothe conventional accuracy (i.e., the number of correct predictions divided bythe total number of predictions).

In contrast, if the conventional accuracy is above chance only because theclassifier takes advantage of an imbalanced test set, then the balancedaccuracy, as appropriate, will drop to

.

The score ranges from 0 to 1, or when adjusted=True is used, it rescaled tothe range

to 1, inclusive, withperformance at random scoring 0.

If

is the true value of the-th sample, andis the corresponding sample weight, then we adjust the sample weight to:

where

is the indicator function.Given predicted for sample, balanced accuracy isdefined as:

With adjusted=True, balanced accuracy reports the relative increase from

. In the binary case, this is also known asYouden’s J statistic,or informedness.

Note

The multiclass definition here seems the most reasonable extension of themetric used in binary classification, though there is no certain consensusin the literature:

• Our definition: [Mosley2013], [Kelleher2015] and [Guyon2015], where[Guyon2015] adopt the adjusted version to ensure that random predictionshave a score of

and perfect predictions have a score of..

• Class balanced accuracy as described in [Mosley2013]: the minimum between the precisionand the recall for each class is computed. Those values are then averaged over the totalnumber of classes to get the balanced accuracy.

• Balanced Accuracy as described in [Urbanowicz2015]: the average of sensitivity and specificityis computed for each class and then averaged over total number of classes.

References:

• Guyon2015(1,2)

• I. Guyon, K. Bennett, G. Cawley, H.J. Escalante, S. Escalera, T.K. Ho, N. Macià,B. Ray, M. Saeed, A.R. Statnikov, E. Viegas, Design of the 2015 ChaLearn AutoML Challenge,IJCNN 2015.

• Mosley2013(1,2)

• L. Mosley, A balanced approach to the multi-class imbalance problem,IJCV 2010.

• Kelleher2015

• John. D. Kelleher, Brian Mac Namee, Aoife D’Arcy, Fundamentals ofMachine Learning for Predictive Data Analytics: Algorithms, Worked Examples,and Case Studies,2015.

• Urbanowicz2015

• Urbanowicz R.J., Moore, J.H. ExSTraCS 2.0: description and evaluation of a scalable learningclassifier system, Evol. Intel. (2015) 8: 89.

3.3.2.4. Cohen’s kappa

The function cohen_kappa_score computes Cohen’s kappa statistic.This measure is intended to compare labelings by different human annotators,not a classifier versus a ground truth.

The kappa score (see docstring) is a number between -1 and 1.Scores above .8 are generally considered good agreement;zero or lower means no agreement (practically random labels).

Kappa scores can be computed for binary or multiclass problems,but not for multilabel problems (except by manually computing a per-label score)and not for more than two annotators.

>>>

>>> from sklearn.metrics import cohen_kappa_score
>>> y_true = [2, 0, 2, 2, 0, 1]
>>> y_pred = [0, 0, 2, 2, 0, 2]
>>> cohen_kappa_score(y_true, y_pred)
0.4285714285714286

3.3.2.5. Confusion matrix

The confusion_matrix function evaluatesclassification accuracy by computing the confusion matrixwith each row corresponding to the true class<https://en.wikipedia.org/wiki/Confusion_matrix>`_.(Wikipedia and other references may use different convention for axes.)

By definition, entry

in a confusion matrix isthe number of observations actually in group, butpredicted to be in group. Here is an example:

>>>

>>> from sklearn.metrics import confusion_matrix
>>> y_true = [2, 0, 2, 2, 0, 1]
>>> y_pred = [0, 0, 2, 2, 0, 2]
>>> confusion_matrix(y_true, y_pred)
array([[2, 0, 0],
       [0, 0, 1],
       [1, 0, 2]])

plot_confusion_matrix can be used to visually represent a confusionmatrix as shown in theConfusion matrixexample, which creates the following figure:The parameter normalize allows to report ratios instead of counts. Theconfusion matrix can be normalized in 3 different ways: 'pred', 'true',and 'all' which will divide the counts by the sum of each columns, rows, orthe entire matrix, respectively.

>>>

>>> y_true = [0, 0, 0, 1, 1, 1, 1, 1]
>>> y_pred = [0, 1, 0, 1, 0, 1, 0, 1]
>>> confusion_matrix(y_true, y_pred, normalize='all')
array([[0.25 , 0.125],
       [0.25 , 0.375]])

For binary problems, we can get counts of true negatives, false positives,false negatives and true positives as follows:

>>>

>>> y_true = [0, 0, 0, 1, 1, 1, 1, 1]
>>> y_pred = [0, 1, 0, 1, 0, 1, 0, 1]
>>> tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()
>>> tn, fp, fn, tp
(2, 1, 2, 3)

Example:

• See Confusion matrixfor an example of using a confusion matrix to evaluate classifier outputquality.

• See Recognizing hand-written digitsfor an example of using a confusion matrix to classifyhand-written digits.

• See Classification of text documents using sparse featuresfor an example of using a confusion matrix to classify textdocuments.

3.3.2.6. Classification report

The classification_report function builds a text report showing themain classification metrics. Here is a small example with custom target_namesand inferred labels:

>>>

>>> from sklearn.metrics import classification_report
>>> y_true = [0, 1, 2, 2, 0]
>>> y_pred = [0, 0, 2, 1, 0]
>>> target_names = ['class 0', 'class 1', 'class 2']
>>> print(classification_report(y_true, y_pred, target_names=target_names))
              precision    recall  f1-score   support
 
     class 0       0.67      1.00      0.80         2
     class 1       0.00      0.00      0.00         1
     class 2       1.00      0.50      0.67         2
 
    accuracy                           0.60         5
   macro avg       0.56      0.50      0.49         5
weighted avg       0.67      0.60      0.59         5

Example:

• See Recognizing hand-written digitsfor an example of classification report usage forhand-written digits.

• See Classification of text documents using sparse featuresfor an example of classification report usage for textdocuments.

• See Parameter estimation using grid search with cross-validationfor an example of classification report usage forgrid search with nested cross-validation.

3.3.2.7. Hamming loss

The hamming_loss computes the average Hamming loss or Hammingdistance between two setsof samples.

If

is the predicted value for the-th label ofa given sample, is the corresponding true value, and is the number of classes or labels, then theHamming loss between two samples is defined as:

where

is the indicator function.

>>>

>>> from sklearn.metrics import hamming_loss
>>> y_pred = [1, 2, 3, 4]
>>> y_true = [2, 2, 3, 4]
>>> hamming_loss(y_true, y_pred)
0.25

In the multilabel case with binary label indicators:

>>>

>>> hamming_loss(np.array([[0, 1], [1, 1]]), np.zeros((2, 2)))
0.75

Note

In multiclass classification, the Hamming loss corresponds to the Hammingdistance between y_true and y_pred which is similar to theZero one loss function. However, while zero-one loss penalizesprediction sets that do not strictly match true sets, the Hamming losspenalizes individual labels. Thus the Hamming loss, upper bounded by the zero-oneloss, is always between zero and one, inclusive; and predicting a proper subsetor superset of the true labels will give a Hamming loss betweenzero and one, exclusive.

3.3.2.8. Precision, recall and F-measures

Intuitively, precision is the abilityof the classifier not to label as positive a sample that is negative, andrecall is theability of the classifier to find all the positive samples.

The F-measure(

and measures) can be interpreted as a weightedharmonic mean of the precision and recall. A measure reaches its best value at 1 and its worst score at 0.With, and are equivalent, and the recall and the precision are equally important.

The precision_recall_curve computes a precision-recall curvefrom the ground truth label and a score given by the classifierby varying a decision threshold.

The average_precision_score function computes theaverage precision(AP) from prediction scores. The value is between 0 and 1 and higher is better.AP is defined as

where

and are the precision and recall at thenth threshold. With random predictions, the AP is the fraction of positivesamples.

References [Manning2008] and [Everingham2010] present alternative variants ofAP that interpolate the precision-recall curve. Currently,average_precision_score does not implement any interpolated variant.References [Davis2006] and [Flach2015] describe why a linear interpolation ofpoints on the precision-recall curve provides an overly-optimistic measure ofclassifier performance. This linear interpolation is used when computing areaunder the curve with the trapezoidal rule in auc.

Several functions allow you to analyze the precision, recall and F-measuresscore:

Note that the precision_recall_curve function is restricted to thebinary case. The average_precision_score function works only inbinary classification and multilabel indicator format. Theplot_precision_recall_curve function plots the precision recall asfollows.

Examples:

• See Classification of text documents using sparse featuresfor an example of f1_score usage to classify textdocuments.

• See Parameter estimation using grid search with cross-validationfor an example of precision_score and recall_score usageto estimate parameters using grid search with nested cross-validation.

• See Precision-Recallfor an example of precision_recall_curve usage to evaluateclassifier output quality.

References:

• Manning2008

• C.D. Manning, P. Raghavan, H. Schütze, Introduction to Information Retrieval,2008.

• Everingham2010

• M. Everingham, L. Van Gool, C.K.I. Williams, J. Winn, A. Zisserman,The Pascal Visual Object Classes (VOC) Challenge,IJCV 2010.

• Davis2006

• J. Davis, M. Goadrich, The Relationship Between Precision-Recall and ROC Curves,ICML 2006.

• Flach2015

• P.A. Flach, M. Kull, Precision-Recall-Gain Curves: PR Analysis Done Right,NIPS 2015.

3.3.2.8.1. Binary classification

In a binary classification task, the terms ‘’positive’’ and ‘’negative’’ referto the classifier’s prediction, and the terms ‘’true’’ and ‘’false’’ refer towhether that prediction corresponds to the external judgment (sometimes knownas the ‘’observation’’). Given these definitions, we can formulate thefollowing table:

In this context, we can define the notions of precision, recall and F-measure:

Here are some small examples in binary classification:

>>>

>>> from sklearn import metrics
>>> y_pred = [0, 1, 0, 0]
>>> y_true = [0, 1, 0, 1]
>>> metrics.precision_score(y_true, y_pred)
1.0
>>> metrics.recall_score(y_true, y_pred)
0.5
>>> metrics.f1_score(y_true, y_pred)
0.66...
>>> metrics.fbeta_score(y_true, y_pred, beta=0.5)
0.83...
>>> metrics.fbeta_score(y_true, y_pred, beta=1)
0.66...
>>> metrics.fbeta_score(y_true, y_pred, beta=2)
0.55...
>>> metrics.precision_recall_fscore_support(y_true, y_pred, beta=0.5)
(array([0.66..., 1.        ]), array([1. , 0.5]), array([0.71..., 0.83...]), array([2, 2]))
 
 
>>> import numpy as np
>>> from sklearn.metrics import precision_recall_curve
>>> from sklearn.metrics import average_precision_score
>>> y_true = np.array([0, 0, 1, 1])
>>> y_scores = np.array([0.1, 0.4, 0.35, 0.8])
>>> precision, recall, threshold = precision_recall_curve(y_true, y_scores)
>>> precision
array([0.66..., 0.5       , 1.        , 1.        ])
>>> recall
array([1. , 0.5, 0.5, 0. ])
>>> threshold
array([0.35, 0.4 , 0.8 ])
>>> average_precision_score(y_true, y_scores)
0.83...

3.3.2.8.2. Multiclass and multilabel classification

In multiclass and multilabel classification task, the notions of precision,recall, and F-measures can be applied to each label independently.There are a few ways to combine results across labels,specified by the average argument to theaverage_precision_score (multilabel only), f1_score,fbeta_score, precision_recall_fscore_support,precision_score and recall_score functions, as describedabove. Note that if all labels are included, “micro”-averagingin a multiclass setting will produce precision, recall and

that are all identical to accuracy. Also note that “weighted” averaging mayproduce an F-score that is not between precision and recall.

To make this more explicit, consider the following notation:

the set of predicted pairs

the set of true pairs

the set of labels

the set of samples

the subset of with sample,i.e.

the subset of with label

• similarly,

and are subsets of

for somesets and

(Conventions vary on handling; this implementation uses, and similar for.)

Then the metrics are defined as:

average	Precision	Recall	F_beta
"micro"
"samples"
"macro"
"weighted"
None

>>>

>>> from sklearn import metrics
>>> y_true = [0, 1, 2, 0, 1, 2]
>>> y_pred = [0, 2, 1, 0, 0, 1]
>>> metrics.precision_score(y_true, y_pred, average='macro')
0.22...
>>> metrics.recall_score(y_true, y_pred, average='micro')
0.33...
>>> metrics.f1_score(y_true, y_pred, average='weighted')
0.26...
>>> metrics.fbeta_score(y_true, y_pred, average='macro', beta=0.5)
0.23...
>>> metrics.precision_recall_fscore_support(y_true, y_pred, beta=0.5, average=None)
(array([0.66..., 0.        , 0.        ]), array([1., 0., 0.]), array([0.71..., 0.        , 0.        ]), array([2, 2, 2]...))

For multiclass classification with a “negative class”, it is possible to exclude some labels:

>>>

>>> metrics.recall_score(y_true, y_pred, labels=[1, 2], average='micro')
... # excluding 0, no labels were correctly recalled
0.0

Similarly, labels not present in the data sample may be accounted for in macro-averaging.

>>>

>>> metrics.precision_score(y_true, y_pred, labels=[0, 1, 2, 3], average='macro')
0.166...

3.3.2.9. Jaccard similarity coefficient score

The jaccard_score function computes the average of Jaccard similaritycoefficients, also called theJaccard index, between pairs of label sets.

The Jaccard similarity coefficient of the

-th samples,with a ground truth label set and predicted label set, is defined as

jaccard_score works like precision_recall_fscore_support as anaively set-wise measure applying natively to binary targets, and extended toapply to multilabel and multiclass through the use of average (seeabove).

In the binary case:

>>>

>>> import numpy as np
>>> from sklearn.metrics import jaccard_score
>>> y_true = np.array([[0, 1, 1],
...                    [1, 1, 0]])
>>> y_pred = np.array([[1, 1, 1],
...                    [1, 0, 0]])
>>> jaccard_score(y_true[0], y_pred[0])
0.6666...

In the multilabel case with binary label indicators:

>>>

>>> jaccard_score(y_true, y_pred, average='samples')
0.5833...
>>> jaccard_score(y_true, y_pred, average='macro')
0.6666...
>>> jaccard_score(y_true, y_pred, average=None)
array([0.5, 0.5, 1. ])

Multiclass problems are binarized and treated like the correspondingmultilabel problem:

>>>

>>> y_pred = [0, 2, 1, 2]
>>> y_true = [0, 1, 2, 2]
>>> jaccard_score(y_true, y_pred, average=None)
array([1. , 0. , 0.33...])
>>> jaccard_score(y_true, y_pred, average='macro')
0.44...
>>> jaccard_score(y_true, y_pred, average='micro')
0.33...

3.3.2.10. Hinge loss

The hinge_loss function computes the average distance betweenthe model and the data usinghinge loss, a one-sided metricthat considers only prediction errors. (Hingeloss is used in maximal margin classifiers such as support vector machines.)

If the labels are encoded with +1 and -1,

: is the truevalue, and is the predicted decisions as output bydecision_function, then the hinge loss is defined as:

If there are more than two labels, hinge_loss uses a multiclass variantdue to Crammer & Singer.Here isthe paper describing it.

If

is the predicted decision for true label and is themaximum of the predicted decisions for all other labels, where predicteddecisions are output by decision function, then multiclass hinge loss is definedby:

Here a small example demonstrating the use of the hinge_loss functionwith a svm classifier in a binary class problem:

>>>

>>> from sklearn import svm
>>> from sklearn.metrics import hinge_loss
>>> X = [[0], [1]]
>>> y = [-1, 1]
>>> est = svm.LinearSVC(random_state=0)
>>> est.fit(X, y)
LinearSVC(random_state=0)
>>> pred_decision = est.decision_function([[-2], [3], [0.5]])
>>> pred_decision
array([-2.18...,  2.36...,  0.09...])
>>> hinge_loss([-1, 1, 1], pred_decision)
0.3...

Here is an example demonstrating the use of the hinge_loss functionwith a svm classifier in a multiclass problem:

>>>

>>> X = np.array([[0], [1], [2], [3]])
>>> Y = np.array([0, 1, 2, 3])
>>> labels = np.array([0, 1, 2, 3])
>>> est = svm.LinearSVC()
>>> est.fit(X, Y)
LinearSVC()
>>> pred_decision = est.decision_function([[-1], [2], [3]])
>>> y_true = [0, 2, 3]
>>> hinge_loss(y_true, pred_decision, labels)
0.56...

3.3.2.11. Log loss

Log loss, also called logistic regression loss orcross-entropy loss, is defined on probability estimates. It iscommonly used in (multinomial) logistic regression and neural networks, as wellas in some variants of expectation-maximization, and can be used to evaluate theprobability outputs (predict_proba) of a classifier instead of itsdiscrete predictions.

For binary classification with a true label

and a probability estimate,the log loss per sample is the negative log-likelihoodof the classifier given the true label:

This extends to the multiclass case as follows.Let the true labels for a set of samplesbe encoded as a 1-of-K binary indicator matrix

,i.e., if sample has labeltaken from a set of labels.Let be a matrix of probability estimates,with.Then the log loss of the whole set is

To see how this generalizes the binary log loss given above,note that in the binary case,

and,so expanding the inner sum overgives the binary log loss.

The log_loss function computes log loss given a list of ground-truthlabels and a probability matrix, as returned by an estimator’s predict_probamethod.

>>>

>>> from sklearn.metrics import log_loss
>>> y_true = [0, 0, 1, 1]
>>> y_pred = [[.9, .1], [.8, .2], [.3, .7], [.01, .99]]
>>> log_loss(y_true, y_pred)
0.1738...

The first [.9, .1] in y_pred denotes 90% probability that the firstsample has label 0. The log loss is non-negative.

3.3.2.12. Matthews correlation coefficient

The matthews_corrcoef function computes theMatthew’s correlation coefficient (MCC)for binary classes. Quoting Wikipedia:

“The Matthews correlation coefficient is used in machine learning as ameasure of the quality of binary (two-class) classifications. It takesinto account true and false positives and negatives and is generallyregarded as a balanced measure which can be used even if the classes areof very different sizes. The MCC is in essence a correlation coefficientvalue between -1 and +1. A coefficient of +1 represents a perfectprediction, 0 an average random prediction and -1 an inverse prediction.The statistic is also known as the phi coefficient.”

In the binary (two-class) case,

,, and are respectively the number of true positives, true negatives, falsepositives and false negatives, the MCC is defined as

In the multiclass case, the Matthews correlation coefficient can be defined in terms of aconfusion_matrix

for classes. To simplify thedefinition consider the following intermediate variables:

the number of times class truly occurred,

the number of times class was predicted,

the total number of samples correctly predicted,

the total number of samples.

Then the multiclass MCC is defined as:

When there are more than two labels, the value of the MCC will no longer rangebetween -1 and +1. Instead the minimum value will be somewhere between -1 and 0depending on the number and distribution of ground true labels. The maximumvalue is always +1.

Here is a small example illustrating the usage of the matthews_corrcoeffunction:

>>>

>>> from sklearn.metrics import matthews_corrcoef
>>> y_true = [+1, +1, +1, -1]
>>> y_pred = [+1, -1, +1, +1]
>>> matthews_corrcoef(y_true, y_pred)
-0.33...

3.3.2.13. Multi-label confusion matrix

The multilabel_confusion_matrix function computes class-wise (default)or sample-wise (samplewise=True) multilabel confusion matrix to evaluatethe accuracy of a classification. multilabel_confusion_matrix also treatsmulticlass data as if it were multilabel, as this is a transformation commonlyapplied to evaluate multiclass problems with binary classification metrics(such as precision, recall, etc.).

When calculating class-wise multilabel confusion matrix

, thecount of true negatives for class is, falsenegatives is, true positives isand false positives is.

Here is an example demonstrating the use of themultilabel_confusion_matrix function withmultilabel indicator matrix input:

>>>

>>> import numpy as np
>>> from sklearn.metrics import multilabel_confusion_matrix
>>> y_true = np.array([[1, 0, 1],
...                    [0, 1, 0]])
>>> y_pred = np.array([[1, 0, 0],
...                    [0, 1, 1]])
>>> multilabel_confusion_matrix(y_true, y_pred)
array([[[1, 0],
        [0, 1]],
 
       [[1, 0],
        [0, 1]],
 
       [[0, 1],
        [1, 0]]])

Or a confusion matrix can be constructed for each sample’s labels:

>>>

>>> multilabel_confusion_matrix(y_true, y_pred, samplewise=True)
array([[[1, 0],
        [1, 1]],
<BLANKLINE>
       [[1, 1],
        [0, 1]]])

Here is an example demonstrating the use of themultilabel_confusion_matrix function withmulticlass input:

>>>

>>> y_true = ["cat", "ant", "cat", "cat", "ant", "bird"]
>>> y_pred = ["ant", "ant", "cat", "cat", "ant", "cat"]
>>> multilabel_confusion_matrix(y_true, y_pred,
...                             labels=["ant", "bird", "cat"])
array([[[3, 1],
        [0, 2]],
 
       [[5, 0],
        [1, 0]],
 
       [[2, 1],
        [1, 2]]])

Here are some examples demonstrating the use of themultilabel_confusion_matrix function to calculate recall(or sensitivity), specificity, fall out and miss rate for each class in aproblem with multilabel indicator matrix input.

Calculatingrecall(also called the true positive rate or the sensitivity) for each class:

>>>

>>> y_true = np.array([[0, 0, 1],
...                    [0, 1, 0],
...                    [1, 1, 0]])
>>> y_pred = np.array([[0, 1, 0],
...                    [0, 0, 1],
...                    [1, 1, 0]])
>>> mcm = multilabel_confusion_matrix(y_true, y_pred)
>>> tn = mcm[:, 0, 0]
>>> tp = mcm[:, 1, 1]
>>> fn = mcm[:, 1, 0]
>>> fp = mcm[:, 0, 1]
>>> tp / (tp + fn)
array([1. , 0.5, 0. ])

Calculatingspecificity(also called the true negative rate) for each class:

>>>

>>> tn / (tn + fp)
array([1. , 0. , 0.5])

Calculating fall out(also called the false positive rate) for each class:

>>>

>>> fp / (fp + tn)
array([0. , 1. , 0.5])

Calculating miss rate(also called the false negative rate) for each class:

>>>

>>> fn / (fn + tp)
array([0. , 0.5, 1. ])

3.3.2.14. Receiver operating characteristic (ROC)

The function roc_curve computes thereceiver operating characteristic curve, or ROC curve.Quoting Wikipedia :

“A receiver operating characteristic (ROC), or simply ROC curve, is agraphical plot which illustrates the performance of a binary classifiersystem as its discrimination threshold is varied. It is created by plottingthe fraction of true positives out of the positives (TPR = true positiverate) vs. the fraction of false positives out of the negatives (FPR = falsepositive rate), at various threshold settings. TPR is also known assensitivity, and FPR is one minus the specificity or true negative rate.”

This function requires the true binaryvalue and the target scores, which can either be probability estimates of thepositive class, confidence values, or binary decisions.Here is a small example of how to use the roc_curve function:

>>>

>>> import numpy as np
>>> from sklearn.metrics import roc_curve
>>> y = np.array([1, 1, 2, 2])
>>> scores = np.array([0.1, 0.4, 0.35, 0.8])
>>> fpr, tpr, thresholds = roc_curve(y, scores, pos_label=2)
>>> fpr
array([0. , 0. , 0.5, 0.5, 1. ])
>>> tpr
array([0. , 0.5, 0.5, 1. , 1. ])
>>> thresholds
array([1.8 , 0.8 , 0.4 , 0.35, 0.1 ])

This figure shows an example of such an ROC curve:The roc_auc_score function computes the area under the receiveroperating characteristic (ROC) curve, which is also denoted byAUC or AUROC. By computing thearea under the roc curve, the curve information is summarized in one number.For more information see the Wikipedia article on AUC.

>>>

>>> import numpy as np
>>> from sklearn.metrics import roc_auc_score
>>> y_true = np.array([0, 0, 1, 1])
>>> y_scores = np.array([0.1, 0.4, 0.35, 0.8])
>>> roc_auc_score(y_true, y_scores)
0.75

In multi-label classification, the roc_auc_score function isextended by averaging over the labels as above.

Compared to metrics such as the subset accuracy, the Hamming loss, or theF1 score, ROC doesn’t require optimizing a threshold for each label.

The roc_auc_score function can also be used in multi-classclassification. Two averaging strategies are currently supported: theone-vs-one algorithm computes the average of the pairwise ROC AUC scores, andthe one-vs-rest algorithm computes the average of the ROC AUC scores for eachclass against all other classes. In both cases, the predicted labels areprovided in an array with values from 0 to n_classes, and the scorescorrespond to the probability estimates that a sample belongs to a particularclass. The OvO and OvR algorithms support weighting uniformly(average='macro') and by prevalence (average='weighted').

One-vs-one Algorithm: Computes the average AUC of all possible pairwisecombinations of classes. [HT2001] defines a multiclass AUC metric weighteduniformly:

where

is the number of classes and is theAUC with class as the positive class and class as thenegative class. In general, in the multiclasscase. This algorithm is used by setting the keyword argument multiclassto 'ovo' and average to 'macro'.

The [HT2001] multiclass AUC metric can be extended to be weighted by theprevalence:

where

is the number of classes. This algorithm is used by settingthe keyword argument multiclass to 'ovo' and average to'weighted'. The 'weighted' option returns a prevalence-weighted averageas described in [FC2009].

One-vs-rest Algorithm: Computes the AUC of each class against the rest[PD2000]. The algorithm is functionally the same as the multilabel case. Toenable this algorithm set the keyword argument multiclass to 'ovr'.Like OvO, OvR supports two types of averaging: 'macro' [F2006] and'weighted' [F2001].

In applications where a high false positive rate is not tolerable the parametermax_fpr of roc_auc_score can be used to summarize the ROC curve upto the given limit.

Examples:

• See Receiver Operating Characteristic (ROC)for an example of using ROC toevaluate the quality of the output of a classifier.

• See Receiver Operating Characteristic (ROC) with cross validationfor an example of using ROC toevaluate classifier output quality, using cross-validation.

• See Species distribution modelingfor an example of using ROC tomodel species distribution.

References:

• HT2001(1,2)

• Hand, D.J. and Till, R.J., (2001). A simple generalisationof the area under the ROC curve for multiple class classification problems.Machine learning, 45(2), pp.171-186.

• FC2009

• Ferri, Cèsar & Hernandez-Orallo, Jose & Modroiu, R. (2009).An Experimental Comparison of Performance Measures for Classification.Pattern Recognition Letters. 30. 27-38.

• PD2000

• Provost, F., Domingos, P. (2000). Well-trained PETs: Improvingprobability estimation trees (Section 6.2), CeDER Working Paper #IS-00-04,Stern School of Business, New York University.

• F2006

• Fawcett, T., 2006. An introduction to ROC analysis.Pattern Recognition Letters, 27(8), pp. 861-874.

• F2001

• Fawcett, T., 2001. Using rule sets to maximizeROC performanceIn Data Mining, 2001.Proceedings IEEE International Conference, pp. 131-138.

3.3.2.15. Zero one loss

The zero_one_loss function computes the sum or the average of the 0-1classification loss (

) over. Bydefault, the function normalizes over the sample. To get the sum of the, set normalize to False.

In multilabel classification, the zero_one_loss scores a subset asone if its labels strictly match the predictions, and as a zero if thereare any errors. By default, the function returns the percentage of imperfectlypredicted subsets. To get the count of such subsets instead, setnormalize to False

If

is the predicted value ofthe-th sample and is the corresponding true value,then the 0-1 loss is defined as:

where

is the indicator function.

>>>

>>> from sklearn.metrics import zero_one_loss
>>> y_pred = [1, 2, 3, 4]
>>> y_true = [2, 2, 3, 4]
>>> zero_one_loss(y_true, y_pred)
0.25
>>> zero_one_loss(y_true, y_pred, normalize=False)
1

In the multilabel case with binary label indicators, where the first labelset [0,1] has an error:

>>>

>>> zero_one_loss(np.array([[0, 1], [1, 1]]), np.ones((2, 2)))
0.5
 
>>> zero_one_loss(np.array([[0, 1], [1, 1]]), np.ones((2, 2)),  normalize=False)
1

Example:

• See Recursive feature elimination with cross-validationfor an example of zero one loss usage to perform recursive featureelimination with cross-validation.

3.3.2.16. Brier score loss

The brier_score_loss function computes theBrier scorefor binary classes. Quoting Wikipedia:

“The Brier score is a proper score function that measures the accuracy ofprobabilistic predictions. It is applicable to tasks in which predictionsmust assign probabilities to a set of mutually exclusive discrete outcomes.”

This function returns a score of the mean square difference between the actualoutcome and the predicted probability of the possible outcome. The actualoutcome has to be 1 or 0 (true or false), while the predicted probability ofthe actual outcome can be a value between 0 and 1.

The brier score loss is also between 0 to 1 and the lower the score (the meansquare difference is smaller), the more accurate the prediction is. It can bethought of as a measure of the “calibration” of a set of probabilisticpredictions.

where :

is the total number of predictions, is thepredicted probability of the actual outcome.

Here is a small example of usage of this function::

>>>

>>> import numpy as np
>>> from sklearn.metrics import brier_score_loss
>>> y_true = np.array([0, 1, 1, 0])
>>> y_true_categorical = np.array(["spam", "ham", "ham", "spam"])
>>> y_prob = np.array([0.1, 0.9, 0.8, 0.4])
>>> y_pred = np.array([0, 1, 1, 0])
>>> brier_score_loss(y_true, y_prob)
0.055
>>> brier_score_loss(y_true, 1 - y_prob, pos_label=0)
0.055
>>> brier_score_loss(y_true_categorical, y_prob, pos_label="ham")
0.055
>>> brier_score_loss(y_true, y_prob > 0.5)
0.0

Example:

• See Probability calibration of classifiersfor an example of Brier score loss usage to perform probabilitycalibration of classifiers.

References:

• G. Brier, Verification of forecasts expressed in terms of probability,Monthly weather review 78.1 (1950)

3.3.3. Multilabel ranking metrics

In multilabel learning, each sample can have any number of ground truth labelsassociated with it. The goal is to give high scores and better rank tothe ground truth labels.

3.3.3.1. Coverage error

The coverage_error function computes the average number of labels thathave to be included in the final prediction such that all true labelsare predicted. This is useful if you want to know how many top-scored-labelsyou have to predict in average without missing any true one. The best valueof this metrics is thus the average number of true labels.

Note

Our implementation’s score is 1 greater than the one given in Tsoumakaset al., 2010. This extends it to handle the degenerate case in which aninstance has 0 true labels.

Formally, given a binary indicator matrix of the ground truth labels

and thescore associated with each label,the coverage is defined as

with

.Given the rank definition, ties in y_scores are broken by giving themaximal rank that would have been assigned to all tied values.

Here is a small example of usage of this function:

>>>

>>> import numpy as np
>>> from sklearn.metrics import coverage_error
>>> y_true = np.array([[1, 0, 0], [0, 0, 1]])
>>> y_score = np.array([[0.75, 0.5, 1], [1, 0.2, 0.1]])
>>> coverage_error(y_true, y_score)
2.5

3.3.3.2. Label ranking average precision

The label_ranking_average_precision_score functionimplements label ranking average precision (LRAP). This metric is linked tothe average_precision_score function, but is based on the notion oflabel ranking instead of precision and recall.

Label ranking average precision (LRAP) averages over the samples the answer tothe following question: for each ground truth label, what fraction ofhigher-ranked labels were true labels? This performance measure will be higherif you are able to give better rank to the labels associated with each sample.The obtained score is always strictly greater than 0, and the best value is 1.If there is exactly one relevant label per sample, label ranking averageprecision is equivalent to the meanreciprocal rank.

Formally, given a binary indicator matrix of the ground truth labels

and the score associated with each label,the average precision is defined as

where

,, computes the cardinality of the set (i.e., the number ofelements in the set), and is the “norm”(which computes the number of nonzero elements in a vector).

Here is a small example of usage of this function:

>>>

>>> import numpy as np
>>> from sklearn.metrics import label_ranking_average_precision_score
>>> y_true = np.array([[1, 0, 0], [0, 0, 1]])
>>> y_score = np.array([[0.75, 0.5, 1], [1, 0.2, 0.1]])
>>> label_ranking_average_precision_score(y_true, y_score)
0.416...

3.3.3.3. Ranking loss

The label_ranking_loss function computes the ranking loss whichaverages over the samples the number of label pairs that are incorrectlyordered, i.e. true labels have a lower score than false labels, weighted bythe inverse of the number of ordered pairs of false and true labels.The lowest achievable ranking loss is zero.

Formally, given a binary indicator matrix of the ground truth labels

and thescore associated with each label,the ranking loss is defined as

where

computes the cardinality of the set (i.e., the number ofelements in the set) and is the “norm”(which computes the number of nonzero elements in a vector).

Here is a small example of usage of this function:

>>>

>>> import numpy as np
>>> from sklearn.metrics import label_ranking_loss
>>> y_true = np.array([[1, 0, 0], [0, 0, 1]])
>>> y_score = np.array([[0.75, 0.5, 1], [1, 0.2, 0.1]])
>>> label_ranking_loss(y_true, y_score)
0.75...
>>> # With the following prediction, we have perfect and minimal loss
>>> y_score = np.array([[1.0, 0.1, 0.2], [0.1, 0.2, 0.9]])
>>> label_ranking_loss(y_true, y_score)
0.0

References:

• Tsoumakas, G., Katakis, I., & Vlahavas, I. (2010). Mining multi-label data. InData mining and knowledge discovery handbook (pp. 667-685). Springer US.

3.3.3.4. Normalized Discounted Cumulative Gain

Discounted Cumulative Gain (DCG) and Normalized Discounted Cumulative Gain(NDCG) are ranking metrics; they compare a predicted order to ground-truthscores, such as the relevance of answers to a query.

from the Wikipedia page for Discounted Cumulative Gain:

“Discounted cumulative gain (DCG) is a measure of ranking quality. Ininformation retrieval, it is often used to measure effectiveness of web searchengine algorithms or related applications. Using a graded relevance scale ofdocuments in a search-engine result set, DCG measures the usefulness, or gain,of a document based on its position in the result list. The gain is accumulatedfrom the top of the result list to the bottom, with the gain of each resultdiscounted at lower ranks”

DCG orders the true targets (e.g. relevance of query answers) in the predictedorder, then multiplies them by a logarithmic decay and sums the result. The sumcan be truncated after the first

results, in which case we call itDCG@K.NDCG, or NDCG@K is DCG divided by the DCG obtained by a perfect prediction, sothat it is always between 0 and 1. Usually, NDCG is preferred to DCG.

Compared with the ranking loss, NDCG can take into account relevance scores,rather than a ground-truth ranking. So if the ground-truth consists only of anordering, the ranking loss should be preferred; if the ground-truth consists ofactual usefulness scores (e.g. 0 for irrelevant, 1 for relevant, 2 for veryrelevant), NDCG can be used.

For one sample, given the vector of continuous ground-truth values for eachtarget

, where is the number of outputs, andthe prediction, which induces the ranking function, theDCG score is

and the NDCG score is the DCG score divided by the DCG score obtained for

.

References:

• Wikipedia entry for Discounted Cumulative Gain:https://en.wikipedia.org/wiki/Discounted_cumulative_gain

• Jarvelin, K., & Kekalainen, J. (2002).Cumulated gain-based evaluation of IR techniques. ACM Transactions onInformation Systems (TOIS), 20(4), 422-446.

• Wang, Y., Wang, L., Li, Y., He, D., Chen, W., & Liu, T. Y. (2013, May).A theoretical analysis of NDCG ranking measures. In Proceedings of the 26thAnnual Conference on Learning Theory (COLT 2013)

• McSherry, F., & Najork, M. (2008, March). Computing information retrievalperformance measures efficiently in the presence of tied scores. InEuropean conference on information retrieval (pp. 414-421). Springer,Berlin, Heidelberg.

3.3.4. Regression metrics

The sklearn.metrics module implements several loss, score, and utilityfunctions to measure regression performance. Some of those have been enhancedto handle the multioutput case: mean_squared_error,mean_absolute_error, explained_variance_score andr2_score.

These functions have an multioutput keyword argument which specifies theway the scores or losses for each individual target should be averaged. Thedefault is 'uniform_average', which specifies a uniformly weighted meanover outputs. If an ndarray of shape (n_outputs,) is passed, then itsentries are interpreted as weights and an according weighted average isreturned. If multioutput is 'raw_values' is specified, then allunaltered individual scores or losses will be returned in an array of shape(n_outputs,).

The r2_score and explained_variance_score accept an additionalvalue 'variance_weighted' for the multioutput parameter. This optionleads to a weighting of each individual score by the variance of thecorresponding target variable. This setting quantifies the globally capturedunscaled variance. If the target variables are of different scale, then thisscore puts more importance on well explaining the higher variance variables.multioutput='variance_weighted' is the default value for r2_scorefor backward compatibility. This will be changed to uniform_average in thefuture.

3.3.4.1. Explained variance score

The explained_variance_score computes the explained varianceregression score.

If

is the estimated target output, the corresponding(correct) target output, and is Variance, the square of the standard deviation,then the explained variance is estimated as follow:

The best possible score is 1.0, lower values are worse.

Here is a small example of usage of the explained_variance_scorefunction:

>>>

>>> from sklearn.metrics import explained_variance_score
>>> y_true = [3, -0.5, 2, 7]
>>> y_pred = [2.5, 0.0, 2, 8]
>>> explained_variance_score(y_true, y_pred)
0.957...
>>> y_true = [[0.5, 1], [-1, 1], [7, -6]]
>>> y_pred = [[0, 2], [-1, 2], [8, -5]]
>>> explained_variance_score(y_true, y_pred, multioutput='raw_values')
array([0.967..., 1.        ])
>>> explained_variance_score(y_true, y_pred, multioutput=[0.3, 0.7])
0.990...

3.3.4.2. Max error

The max_error function computes the maximum residual error , a metricthat captures the worst case error between the predicted value andthe true value. In a perfectly fitted single output regressionmodel, max_error would be 0 on the training set and though thiswould be highly unlikely in the real world, this metric shows theextent of error that the model had when it was fitted.

If

is the predicted value of the-th sample,and is the corresponding true value, then the max error isdefined as

Here is a small example of usage of the max_error function:

>>>

>>> from sklearn.metrics import max_error
>>> y_true = [3, 2, 7, 1]
>>> y_pred = [9, 2, 7, 1]
>>> max_error(y_true, y_pred)
6

The max_error does not support multioutput.

3.3.4.3. Mean absolute error

The mean_absolute_error function computes mean absoluteerror, a riskmetric corresponding to the expected value of the absolute error loss or

-norm loss.

If

is the predicted value of the-th sample,and is the corresponding true value, then the mean absolute error(MAE) estimated over is defined as

Here is a small example of usage of the mean_absolute_error function:

>>>

>>> from sklearn.metrics import mean_absolute_error
>>> y_true = [3, -0.5, 2, 7]
>>> y_pred = [2.5, 0.0, 2, 8]
>>> mean_absolute_error(y_true, y_pred)
0.5
>>> y_true = [[0.5, 1], [-1, 1], [7, -6]]
>>> y_pred = [[0, 2], [-1, 2], [8, -5]]
>>> mean_absolute_error(y_true, y_pred)
0.75
>>> mean_absolute_error(y_true, y_pred, multioutput='raw_values')
array([0.5, 1. ])
>>> mean_absolute_error(y_true, y_pred, multioutput=[0.3, 0.7])
0.85...

3.3.4.4. Mean squared error

The mean_squared_error function computes mean squareerror, a riskmetric corresponding to the expected value of the squared (quadratic) error orloss.

If

is the predicted value of the-th sample,and is the corresponding true value, then the mean squared error(MSE) estimated over is defined as

Here is a small example of usage of the mean_squared_errorfunction:

>>>

>>> from sklearn.metrics import mean_squared_error
>>> y_true = [3, -0.5, 2, 7]
>>> y_pred = [2.5, 0.0, 2, 8]
>>> mean_squared_error(y_true, y_pred)
0.375
>>> y_true = [[0.5, 1], [-1, 1], [7, -6]]
>>> y_pred = [[0, 2], [-1, 2], [8, -5]]
>>> mean_squared_error(y_true, y_pred)
0.7083...

Examples:

• See Gradient Boosting regressionfor an example of mean squared error usage toevaluate gradient boosting regression.

3.3.4.5. Mean squared logarithmic error

The mean_squared_log_error function computes a risk metriccorresponding to the expected value of the squared logarithmic (quadratic)error or loss.

If

is the predicted value of the-th sample,and is the corresponding true value, then the mean squaredlogarithmic error (MSLE) estimated over isdefined as

Where

means the natural logarithm of. This metricis best to use when targets having exponential growth, such as populationcounts, average sales of a commodity over a span of years etc. Note that thismetric penalizes an under-predicted estimate greater than an over-predictedestimate.

Here is a small example of usage of the mean_squared_log_errorfunction:

>>>

>>> from sklearn.metrics import mean_squared_log_error
>>> y_true = [3, 5, 2.5, 7]
>>> y_pred = [2.5, 5, 4, 8]
>>> mean_squared_log_error(y_true, y_pred)
0.039...
>>> y_true = [[0.5, 1], [1, 2], [7, 6]]
>>> y_pred = [[0.5, 2], [1, 2.5], [8, 8]]
>>> mean_squared_log_error(y_true, y_pred)
0.044...

3.3.4.6. Median absolute error

The median_absolute_error is particularly interesting because it isrobust to outliers. The loss is calculated by taking the median of all absolutedifferences between the target and the prediction.

If

is the predicted value of the-th sampleand is the corresponding true value, then the median absolute error(MedAE) estimated over is defined as

The median_absolute_error does not support multioutput.

Here is a small example of usage of the median_absolute_errorfunction:

>>>

>>> from sklearn.metrics import median_absolute_error
>>> y_true = [3, -0.5, 2, 7]
>>> y_pred = [2.5, 0.0, 2, 8]
>>> median_absolute_error(y_true, y_pred)
0.5

3.3.4.7. R² score, the coefficient of determination

The r2_score function computes the coefficient ofdetermination,usually denoted as R².

It represents the proportion of variance (of y) that has been explained by theindependent variables in the model. It provides an indication of goodness offit and therefore a measure of how well unseen samples are likely to bepredicted by the model, through the proportion of explained variance.

As such variance is dataset dependent, R² may not be meaningfully comparableacross different datasets. Best possible score is 1.0 and it can be negative(because the model can be arbitrarily worse). A constant model that alwayspredicts the expected value of y, disregarding the input features, would get aR² score of 0.0.

If

is the predicted value of the-th sampleand is the corresponding true value for total samples,the estimated R² is defined as:

where

and.

Note that r2_score calculates unadjusted R² without correcting forbias in sample variance of y.

Here is a small example of usage of the r2_score function:

>>>

>>> from sklearn.metrics import r2_score
>>> y_true = [3, -0.5, 2, 7]
>>> y_pred = [2.5, 0.0, 2, 8]
>>> r2_score(y_true, y_pred)
0.948...
>>> y_true = [[0.5, 1], [-1, 1], [7, -6]]
>>> y_pred = [[0, 2], [-1, 2], [8, -5]]
>>> r2_score(y_true, y_pred, multioutput='variance_weighted')
0.938...
>>> y_true = [[0.5, 1], [-1, 1], [7, -6]]
>>> y_pred = [[0, 2], [-1, 2], [8, -5]]
>>> r2_score(y_true, y_pred, multioutput='uniform_average')
0.936...
>>> r2_score(y_true, y_pred, multioutput='raw_values')
array([0.965..., 0.908...])
>>> r2_score(y_true, y_pred, multioutput=[0.3, 0.7])
0.925...

Example:

• See Lasso and Elastic Net for Sparse Signalsfor an example of R² score usage toevaluate Lasso and Elastic Net on sparse signals.

3.3.4.8. Mean Poisson, Gamma, and Tweedie deviances

The mean_tweedie_deviance function computes the mean Tweediedeviance errorwith a power parameter (

). This is a metric that elicitspredicted expectation values of regression targets.

Following special cases exist,

• when power=0 it is equivalent to mean_squared_error.

• when power=1 it is equivalent to mean_poisson_deviance.

• when power=2 it is equivalent to mean_gamma_deviance.

If

is the predicted value of the-th sample,and is the corresponding true value, then the mean Tweediedeviance error (D) for power, estimated overis defined as

Tweedie deviance is a homogeneous function of degree 2-power.Thus, Gamma distribution with power=2 means that simultaneously scalingy_true and y_pred has no effect on the deviance. For Poissondistribution power=1 the deviance scales linearly, and for Normaldistribution (power=0), quadratically. In general, the higherpower the less weight is given to extreme deviations between trueand predicted targets.

For instance, let’s compare the two predictions 1.0 and 100 that are both50% of their corresponding true value.

The mean squared error (power=0) is very sensitive to theprediction difference of the second point,:

>>>

>>> from sklearn.metrics import mean_tweedie_deviance
>>> mean_tweedie_deviance([1.0], [1.5], power=0)
0.25
>>> mean_tweedie_deviance([100.], [150.], power=0)
2500.0

If we increase power to 1,:

>>>

>>> mean_tweedie_deviance([1.0], [1.5], power=1)
0.18...
>>> mean_tweedie_deviance([100.], [150.], power=1)
18.9...

the difference in errors decreases. Finally, by setting, power=2:

>>>

>>> mean_tweedie_deviance([1.0], [1.5], power=2)
0.14...
>>> mean_tweedie_deviance([100.], [150.], power=2)
0.14...

we would get identical errors. The deviance when power=2 is thus onlysensitive to relative errors.

3.3.5. Clustering metrics

The sklearn.metrics module implements several loss, score, and utilityfunctions. For more information see the Clustering performance evaluationsection for instance clustering, and Biclustering evaluation forbiclustering.

3.3.6. Dummy estimators

When doing supervised learning, a simple sanity check consists of comparingone’s estimator against simple rules of thumb. DummyClassifierimplements several such simple strategies for classification:

• stratified generates random predictions by respecting the trainingset class distribution.

• most_frequent always predicts the most frequent label in the training set.

• prior always predicts the class that maximizes the class prior(like most_frequent) and predict_proba returns the class prior.

• uniform generates predictions uniformly at random.

• • constant always predicts a constant label that is provided by the user.
  • A major motivation of this method is F1-scoring, when the positive classis in the minority.

Note that with all these strategies, the predict method completely ignoresthe input data!

To illustrate DummyClassifier, first let’s create an imbalanceddataset:

>>>

>>> from sklearn.datasets import load_iris
>>> from sklearn.model_selection import train_test_split
>>> X, y = load_iris(return_X_y=True)
>>> y[y != 1] = -1
>>> X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)

Next, let’s compare the accuracy of SVC and most_frequent:

>>>

>>> from sklearn.dummy import DummyClassifier
>>> from sklearn.svm import SVC
>>> clf = SVC(kernel='linear', C=1).fit(X_train, y_train)
>>> clf.score(X_test, y_test)
0.63...
>>> clf = DummyClassifier(strategy='most_frequent', random_state=0)
>>> clf.fit(X_train, y_train)
DummyClassifier(random_state=0, strategy='most_frequent')
>>> clf.score(X_test, y_test)
0.57...

We see that SVC doesn’t do much better than a dummy classifier. Now, let’schange the kernel:

>>>

>>> clf = SVC(kernel='rbf', C=1).fit(X_train, y_train)
>>> clf.score(X_test, y_test)
0.94...

We see that the accuracy was boosted to almost 100%. A cross validationstrategy is recommended for a better estimate of the accuracy, if itis not too CPU costly. For more information see the Cross-validation: evaluating estimator performancesection. Moreover if you want to optimize over the parameter space, it is highlyrecommended to use an appropriate methodology; see the Tuning the hyper-parameters of an estimatorsection for details.

More generally, when the accuracy of a classifier is too close to random, itprobably means that something went wrong: features are not helpful, ahyperparameter is not correctly tuned, the classifier is suffering from classimbalance, etc…

DummyRegressor also implements four simple rules of thumb for regression:

• mean always predicts the mean of the training targets.

• median always predicts the median of the training targets.

• quantile always predicts a user provided quantile of the training targets.

• constant always predicts a constant value that is provided by the user.

In all these strategies, the predict method completely ignoresthe input data.