Precision, recall and F1: judging a classifier beyond accuracy

Suppose a model flags transactions as fraud, and it is "98% accurate". Is it good? You cannot tell yet, and the reason is the most important idea in evaluating classifiers: accuracy hides the kind of mistakes a model makes.

The four outcomes

A yes/no classifier can be right or wrong in two ways each, giving four outcomes. Together they form the confusion matrix.

Everything else is arithmetic on those four counts.

The metrics

• Accuracy = (TP + TN) / everything. The share of all predictions that were right.
• Precision = TP / (TP + FP). When the model says yes, how often is it right? Low precision means many false alarms.
• Recall (also called sensitivity) = TP / (TP + FN). Of all the real yeses, how many did the model find? Low recall means many misses.
• F1 = 2 × precision × recall / (precision + recall). A single number that is high only when both are high. It is the harmonic mean, which punishes a big gap between the two.

A worked example

Sixteen cases: six are really positive and ten are negative. The model finds four of the six, and wrongly flags two of the ten.

actual    = [1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
predicted = [1, 1, 1, 1, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0]

tp = sum(a == 1 and p == 1 for a, p in zip(actual, predicted))
fp = sum(a == 0 and p == 1 for a, p in zip(actual, predicted))
fn = sum(a == 1 and p == 0 for a, p in zip(actual, predicted))
tn = sum(a == 0 and p == 0 for a, p in zip(actual, predicted))

precision = tp / (tp + fp)
recall = tp / (tp + fn)
f1 = 2 * precision * recall / (precision + recall)
accuracy = (tp + tn) / len(actual)

print(f"confusion matrix: TP={tp} FP={fp} FN={fn} TN={tn}")
print(f"precision {precision:.2f}  recall {recall:.2f}  F1 {f1:.2f}  accuracy {accuracy:.2f}")

# the trap: a model that never predicts a positive
lazy = [0] * len(actual)
print("always-negative model: accuracy", sum(a == p for a, p in zip(actual, lazy)) / len(actual), "recall", 0 / sum(actual))

confusion matrix: TP=4 FP=2 FN=2 TN=8
precision 0.67  recall 0.67  F1 0.67  accuracy 0.75
always-negative model: accuracy 0.625 recall 0.0

Precision is 4 of the 6 flagged, recall is 4 of the 6 real positives, and accuracy is 12 of 16.

The accuracy trap

The last line is the point. A model that never predicts a positive is 62.5% accurate here, because most cases are negative, and it finds nothing (recall 0). The real thing is more extreme: if 1 in 1,000 transactions is fraud, a model that says "not fraud" every time is 99.9% accurate and completely useless.

Whenever the classes are unbalanced, which is the usual case for the problems worth solving (fraud, disease, defects, churn), accuracy tells you very little. Look at precision and recall.

Choosing what to optimise

The two errors have different costs, and the right balance depends on the problem.

The threshold

Most models produce a score, and you pick a threshold above which you say "yes". Moving it trades one metric for the other: a lower threshold finds more real cases (higher recall) and raises more false alarms (lower precision). So the model's quality is better described by the precision-recall curve than by one pair of numbers, and the threshold is a business decision, not a technical afterthought.

Good practice

• Evaluate on data the model has not seen. Metrics on training data flatter the model.
• Report the confusion matrix with the metrics. The counts show how much evidence the percentages rest on.
• Compare with a trivial baseline such as always predicting the most common class. If your model barely beats it, it has learned little.
• Check the metrics per group (by region, device, customer type). A good average can hide a group the model fails.