Phase 1 โ€” First Experiments

Test 2: Full Evaluations

3 use cases ร— models ร— 3 runs each. Outputs saved to JSON, then manually scored on quality metrics. Two-script pipeline.

๐Ÿ“ phase-1-first-experiments/test2_full_evaluations.py  |  test2_score_evaluations.py

How to Run

cd phase-1-first-experiments

# Step 1 โ€” run all model calls, save outputs
python3 test2_full_evaluations.py

# Step 2 โ€” score each output manually (1โ€“5)
python3 test2_score_evaluations.py

These two scripts are designed to run in sequence. The first collects outputs; the second reads that file and walks you through scoring each one interactively.

Dependencies & Run Order

โฌ… Depends on Test 1. Run test1_basic_test.py first to confirm all models respond before committing to a long multi-run evaluation.

test2_score_evaluations.py depends on evaluation_results.json โ€” if that file is missing (because test2_full_evaluations.py has not run), the scoring script will detect this and exit with a clear error message. No silent crash.

What the Terminal Shows

test2_full_evaluations.py โ€” prints a coloured intro with total call count (e.g. 9 model calls total) before starting. Each call prints a live [N/total] model run X/3 โ€” waiting... line that updates in place to done (Xs) when the response arrives. You always know exactly where you are in a long run.

test2_score_evaluations.py โ€” prints a coloured intro, tells you how many ratings you will be asked for, shows a "Press Enter to start" gate so you can read before diving in, then walks you through each output with its metrics one at a time.

What We Did

Ran three different types of tasks on all 4 models, 3 times each โ€” to see how consistent they are and how quality varies across task types. Then manually scored each output on a 1โ€“5 scale for coherence, accuracy, and other qualities.

Speed (Times in Seconds)

Model Text Generation Q&A Summarisation Note
deepseek-r1:7b 209โ€“267s 130โ€“233s ~200โ€“250s Slowest โ€” reasoning focused
mistral 116โ€“136s 98โ€“117s ~100โ€“120s Fastest across all tasks
llama3.1:8b 142โ€“179s ~150โ€“180s ~140โ€“170s Mid-range
gemma2:9b 136โ€“151s ~130โ€“160s ~130โ€“150s Mid-range

Output Quality by Task Type

Task What We Observed Learning
Text Generation All 4 models wrote coherent robot-garden stories, but plots and themes differed every run. Creative tasks produce varied outputs โ€” the model has freedom to imagine, so results are not consistent run to run.
Q&A Answers to factual questions were nearly identical across all 3 runs per model. Factual tasks are reliable. When there is a correct answer, models converge on it consistently.
Summarisation Accurate summaries with only small wording differences across runs. deepseek most thorough, mistral most concise. Condensation tasks are consistent โ€” models handle structured knowledge well.

Consistency Finding

Key insight: The type of task determines how consistent results are. Creative tasks vary a lot โ€” that is expected and by design. Factual and structured tasks are reliable. This is important when designing your own test cases: do not judge a creative task for inconsistency, and do not accept inconsistency in a factual task.

Key Learnings

View Full Evaluation Results with Scores โ†’

What's Next โ€” After This Test

You have now completed Phase 1.

You ran models against real tasks, collected outputs, and scored them yourself. That manual scoring process โ€” where you are the judge โ€” is the baseline that all automated evaluation methods try to replicate.

The key question Phase 1 leaves unanswered: can a computer do what you just did? Can it rate an output for coherence, accuracy, or conciseness without a human typing in a number?
Phase 2 โ€” Eval Techniques (Tests 3โ€“7)

The next phase introduces automated evaluation methods that replace manual scoring:

 โ€ข Test 3 โ€” BLEU & ROUGE: Word-overlap scores. Compare a model output to a reference answer mathematically.
 โ€ข Test 4 โ€” Deterministic Eval: Rule-based checks (must-contain, must-not-contain, regex).
 โ€ข Test 5 โ€” LLM-as-a-Judge: Use a second model to rate the first model's output.
 โ€ข Test 6 โ€” Semantic Similarity: Compare meaning, not just words.
 โ€ข Test 7 โ€” Groundedness Scoring: Did the answer come from the source, or was it invented?

No dependencies between Phase 1 and Phase 2 โ€” you can start Test 3 independently.