First automated evaluation. Moving from manual scoring to math-based metrics โ BLEU and ROUGE โ to compare model outputs against a reference answer.
A summarisation task on a sample AI text. All 4 models summarised the same passage. We then measured how closely each summary matched a reference summary using two automated metrics.
What BLEU and ROUGE Mean (Simply)
Metric
What It Measures
Think of It As
BLEU
How many exact word phrases from the model output match the reference
Word-for-word overlap โ like checking if the model used the same phrases
ROUGE-1
Single word overlap between output and reference
Did the model use the same words at all?
ROUGE-2
Two-word phrase overlap
Did the model use the same two-word combinations?
ROUGE-L
Longest matching sequence of words in order
Did the model follow a similar sentence structure?
Higher scores mean closer to the reference โ but closer is not always better. A model can be factually correct in completely different words and score low. These metrics measure similarity, not quality.
Results
Model
Time (s)
BLEU
ROUGE-1
ROUGE-2
ROUGE-L
deepseek-r1:7b
84.78
24.61
0.6667
0.3429
0.6111
mistral
19.11
15.75
0.5641
0.3158
0.5385
llama3.1:8b
25.50
11.05
0.5111
0.2273
0.4222
gemma2:9b
37.82
9.86
0.3953
0.1429
0.2326
Key Learnings
deepseek scored highest but was slowest. Best similarity to reference (BLEU 24.61, ROUGE-1 0.67), but took 85 seconds. There is a trade-off between quality and speed.
llama scored lower because it added a preamble. It started with "Here is a summary of the text in 2-3 sentences:" โ those words are not in the reference, which pulls the score down even though the actual summary was accurate. This reveals a limitation of these metrics.
gemma scored lowest despite a reasonable summary. It added "Let me know if you'd like me to elaborate" at the end โ again, not in the reference. The metric penalises additions, not just errors.
These metrics cannot evaluate meaning. A model could say "AI enables machines to think" and another could say "Artificial intelligence allows computers to perform intelligent tasks" โ similar meaning, low overlap score. That is why we move to semantic evaluation next.
Automated metrics remove human bias. Manual scoring depends on who is reading. BLEU and ROUGE give the same number every time for the same inputs โ fully repeatable and comparable across runs.
What This Evaluation is Called
BLEU (Bilingual Evaluation Understudy) and ROUGE (Recall-Oriented Understudy for Gisting Evaluation) are reference-based automated metrics. They come from the field of machine translation and text summarisation research. They are fast, cheap, and objective โ but they measure surface similarity, not semantic correctness.