Accuracy, precision and recall

The concept

Once a model is trained, we test it on examples it has never seen, where we already know the right answers. Then we count hits and misses.

Three common scores:

• Accuracy: out of everything, how much did it get right?
• Precision: when it raised an alarm, how often was it right?
• Recall: out of all the real problems, how many did it catch?

Walkthrough: spotting faulty phones on a factory line

A factory tests 1,000 phones. 20 are actually faulty.

Model A is lazy. It says "fine" for every phone.

• Right on 980 good phones, wrong on all 20 faulty ones.
• Accuracy: 980 / 1,000 = 98%.
• Faulty phones caught: 0. Every one ships to a customer.

98% accurate. Useless.

Model B actually tries. It flags 25 phones as faulty.

• 15 of those 25 really are faulty (correct alarms).
• 10 are fine (false alarms).
• 5 faulty phones slip through (misses).
• The other 970 good phones are correctly passed.

Score it:

• Accuracy: (15 + 970) / 1,000 = 98.5%. Barely different from lazy Model A.
• Precision: 15 / 25 = 60%. When it flags a phone, it's right 6 times out of 10.
• Recall: 15 / 20 = 75%. It catches 3 out of every 4 faulty phones.

Precision and recall reveal what accuracy hid.

The trade-off

Make the model more cautious (flag more phones) and recall goes up, but so do false alarms, so precision drops. Make it stricter and the opposite happens. You choose based on which mistake costs more:

• Missing a problem is expensive? Favor recall.
• False alarms are expensive? Favor precision.

Where you see it at work

• Healthcare: screening tests favor recall, because missing a disease is worse than an extra follow-up test.
• Banking: fraud teams balance both, since blocking a good customer's card annoys them.
• Email: spam filters favor precision, because losing a real email hurts more than seeing some spam.
• Business analysis: whenever someone quotes one accuracy number, ask "how many did it miss?"

In one sentence

Accuracy says how often a model is right overall; precision says how much to trust its alarms; recall says how many real problems it catches.