Phase 1 β€” First Experiments

Test 1: Basic Test

Single prompt across all models to check basic functionality, response quality and speed. The first script to run in the whole repository.

πŸ“ phase-1-first-experiments/test1_basic_test.py

How to Run

cd phase-1-first-experiments
python3 test1_basic_test.py

The script prints a banner before starting so you know what it is doing, then queries each model one at a time with a live "waiting…" indicator. You will see output immediately as each model responds β€” you do not need to wait for all models to finish.

Dependencies & Run Order

βœ” No prior test needed. This is the first script to run.

Before running, make sure:
 β€’ ollama serve is running in a separate terminal
 β€’ All models in config.yaml β†’ phase1.MODELS are pulled (ollama pull <model>)

After this: run test2_full_evaluations.py β†’ then test2_score_evaluations.py

What the Terminal Shows

The script prints a coloured intro block explaining what it will do and which models it will query. Then for each model it prints [1/N] Querying <model> ... Sending prompt β€” waiting for response... followed by the full response and time as soon as it arrives. No silent waiting.

What We Tested

A single joke prompt sent to all models. Goal: see if models respond at all, how fast they are, and whether responses differ in style.

Prompt used: "Hello, can you tell me a short joke?"

Model Comparison

deepseek-r1:7b mistral llama3.1:8b gemma2:9b
Output Certainly! Here's a short joke for you:

Why don't scientists trust atoms? Because they make up everything!
Of course! Here's one for you:

Why don't scientists trust atoms?

Because they make up everything!
Here's one:

What do you call a fake noodle?

An impasta.

Hope that made you smile! Do you want to hear another one?
Why don't scientists trust atoms?

Because they make up everything! πŸ˜„
Time 54.13s 11.63s 19.39s 20.49s
Note Slowest β€” reasoning-focused model Fastest response Different joke β€” more creative Added emoji for casual tone

Key Learnings

What's Next β€” After This Test

Step 2: Run test2_full_evaluations.py

Test 1 only sends one prompt and prints the result. That tells you whether models work and how fast they are β€” but it says nothing about quality across different types of tasks.

Test 2 is the next script to run. It sends 3 different prompts (a story, a factual question, a summary) to all your models, 3 times each, and saves every response to a file called evaluation_results.json.

⚠ Warning: Test 2 takes a long time β€” roughly 1–4 minutes per model call. With 4 models Γ— 3 tasks Γ— 3 runs = 36 calls, expect it to run for 30–90 minutes. Start it and leave it running.
Step 3: Run test2_score_evaluations.py (after Step 2 finishes)

Once evaluation_results.json exists, this script reads it and walks you through each response one by one, asking you to rate it 1–5 on metrics like coherence, accuracy, and conciseness. No model is called β€” you are the judge. Takes about 10–15 minutes of reading and typing. Your scores are saved to scores.json.