Phase 2 โ€” Evaluation Techniques

Test 3: Automated Metrics

First automated evaluation. Moving from manual scoring to math-based metrics โ€” BLEU and ROUGE โ€” to compare model outputs against a reference answer.

๐Ÿ“ phase-2-evaluation-techniques/test3_bleu_score_evaluation.py

What We Tested

A summarisation task on a sample AI text. All 4 models summarised the same passage. We then measured how closely each summary matched a reference summary using two automated metrics.

What BLEU and ROUGE Mean (Simply)

MetricWhat It MeasuresThink of It As
BLEUHow many exact word phrases from the model output match the referenceWord-for-word overlap โ€” like checking if the model used the same phrases
ROUGE-1Single word overlap between output and referenceDid the model use the same words at all?
ROUGE-2Two-word phrase overlapDid the model use the same two-word combinations?
ROUGE-LLongest matching sequence of words in orderDid the model follow a similar sentence structure?
Higher scores mean closer to the reference โ€” but closer is not always better. A model can be factually correct in completely different words and score low. These metrics measure similarity, not quality.

Results

ModelTime (s)BLEUROUGE-1ROUGE-2ROUGE-L
deepseek-r1:7b84.7824.610.66670.34290.6111
mistral19.1115.750.56410.31580.5385
llama3.1:8b25.5011.050.51110.22730.4222
gemma2:9b37.829.860.39530.14290.2326

Key Learnings

What This Evaluation is Called

BLEU (Bilingual Evaluation Understudy) and ROUGE (Recall-Oriented Understudy for Gisting Evaluation) are reference-based automated metrics. They come from the field of machine translation and text summarisation research. They are fast, cheap, and objective โ€” but they measure surface similarity, not semantic correctness.