cd phase-1-first-experiments
# Step 1 โ run all model calls, save outputs
python3 test2_full_evaluations.py
# Step 2 โ score each output manually (1โ5)
python3 test2_score_evaluations.py
These two scripts are designed to run in sequence. The first collects outputs; the second reads that file and walks
you through scoring each one interactively.
Dependencies & Run Order
โฌ Depends on Test 1. Run test1_basic_test.py first to confirm all models respond
before committing to a long multi-run evaluation.
test2_score_evaluations.py depends on evaluation_results.json โ if that file is
missing (because test2_full_evaluations.py has not run), the scoring script will detect this and exit with a clear
error message. No silent crash.
What the Terminal Shows
test2_full_evaluations.py โ prints a coloured intro with total call count (e.g. 9 model calls
total) before starting. Each call prints a live [N/total] model run X/3 โ waiting... line that
updates in place to done (Xs) when the response arrives. You always know exactly where you are in a
long run.
test2_score_evaluations.py โ prints a coloured intro, tells you how many ratings you will be asked
for, shows a "Press Enter to start" gate so you can read before diving in, then walks you through each output with
its metrics one at a time.
What We Did
Ran three different types of tasks on all 4 models, 3 times each โ to see how consistent they are and how quality
varies across task types. Then manually scored each output on a 1โ5 scale for coherence, accuracy, and other
qualities.
Speed (Times in Seconds)
Model
Text Generation
Q&A
Summarisation
Note
deepseek-r1:7b
209โ267s
130โ233s
~200โ250s
Slowest โ reasoning focused
mistral
116โ136s
98โ117s
~100โ120s
Fastest across all tasks
llama3.1:8b
142โ179s
~150โ180s
~140โ170s
Mid-range
gemma2:9b
136โ151s
~130โ160s
~130โ150s
Mid-range
Output Quality by Task Type
Task
What We Observed
Learning
Text Generation
All 4 models wrote coherent robot-garden stories, but plots and themes differed every run.
Creative tasks produce varied outputs โ the model has freedom to imagine, so results are not consistent run
to run.
Q&A
Answers to factual questions were nearly identical across all 3 runs per model.
Factual tasks are reliable. When there is a correct answer, models converge on it consistently.
Summarisation
Accurate summaries with only small wording differences across runs. deepseek most thorough, mistral most
concise.
Condensation tasks are consistent โ models handle structured knowledge well.
Consistency Finding
Key insight: The type of task determines how consistent results are. Creative tasks vary a lot โ
that is expected and by design. Factual and structured tasks are reliable. This is important when designing your own
test cases: do not judge a creative task for inconsistency, and do not accept inconsistency in a factual task.
Key Learnings
All 4 models ran without crashing. Basic stability confirmed across task types โ a good
starting baseline.
Mistral is the fastest, deepseek the most thorough. Choose a model based on what matters more
for your use case โ speed or depth.
Manual scoring gives you a number, but numbers need context. A score of 4/5 for coherence means
nothing without knowing what 4 looks like versus 3. This is why structured criteria matter in evaluation.
Three runs reveal variance, one run hides it. If you only run a test once, you cannot tell if
the result was typical or an outlier. Run at least 3 times for anything you care about.
You ran models against real tasks, collected outputs, and scored them yourself. That manual scoring process โ where
you are the judge โ is the baseline that all automated evaluation methods try to replicate.
The key question Phase 1 leaves unanswered: can a computer do what you just did? Can it rate an
output for coherence, accuracy, or conciseness without a human typing in a number?
Phase 2 โ Eval Techniques (Tests 3โ7)
The next phase introduces automated evaluation methods that replace manual scoring:
โข Test 3 โ BLEU & ROUGE: Word-overlap scores. Compare a model output to a reference
answer mathematically.
โข Test 4 โ Deterministic Eval: Rule-based checks (must-contain, must-not-contain, regex).
โข Test 5 โ LLM-as-a-Judge: Use a second model to rate the first model's output.
โข Test 6 โ Semantic Similarity: Compare meaning, not just words.
โข Test 7 โ Groundedness Scoring: Did the answer come from the source, or was it
invented?
No dependencies between Phase 1 and Phase 2 โ you can start Test 3 independently.