This test introduces observability for LLM applications. You will set up Langfuse locally using Docker, instrument a banking chatbot to send every interaction as a trace, and explore what the Langfuse dashboard reveals about your model's behaviour โ latency, pass rates, and failure patterns โ without changing how the model runs.
In traditional software, observability means logs, metrics, and traces. You can tell exactly what happened, when, and why a failure occurred. LLM systems have the same need โ but the inputs and outputs are unstructured text, not typed function calls. A model can return the "right" response category and still be wrong in subtle ways.
Langfuse is the observability layer for LLM applications. Every time your bot calls a model, Langfuse records the prompt, the response, the latency, and any scores you attach. Over many runs you can see patterns that a single test run will never show: which categories fail most often, which questions are slow, whether a model change improved or degraded quality.
| File | Purpose | Notes |
|---|---|---|
test14_langfuse_evaluator.py |
The main script. Loads test cases, runs the banking bot, sends every interaction to Langfuse as a trace with spans and scores. | Config-driven via config.yaml |
banking_bot.py |
Self-contained banking chatbot. Calls Ollama directly โ no imports from other phase folders. | System prompt hardcoded inside this file |
docker-compose.yml |
Spins up Langfuse + PostgreSQL locally. One command to start. All data stays on your machine. | Uses langfuse/langfuse:2 image |
banking_test_cases.xlsx |
20 test cases across 6 categories: account_info, transactions, loans, security, fees, out_of_scope. | Generate with create_test_cases.py if missing |
create_test_cases.py |
One-time script to generate the Excel test case file. | Run once, then delete or keep |
Langfuse runs inside Docker containers. Docker is a tool that packages an application and everything it needs (databases, servers) into isolated containers that run the same way on any machine.
docker --versionIf you see a version number, Docker is installed.
.dmg file and drag Docker to Applications.docker --version docker compose versionBoth should print version numbers. If
docker compose fails, try docker-compose (older
installs use the hyphen form).
docker compose up -d, Docker will
download two images โ langfuse/langfuse:2 (the Langfuse server, ~500 MB) and
postgres:16-alpine (the database, ~100 MB). These are downloaded once and cached locally. All
subsequent starts are instant. Your data is stored in a Docker volume on your machine โ nothing is sent anywhere.
The docker-compose.yml file in this folder defines two services: the Langfuse server and a
PostgreSQL database. One command starts both.
cd phase-5-production-eval/langfuse-observability docker compose up -d
The -d flag means "detached" โ the containers run in the background. You will see Docker pulling
images on first run (this takes 1โ3 minutes depending on your internet speed). Subsequent starts take a few
seconds.
After the command completes, verify both containers are running:
docker compose ps
You should see two rows with State: running โ one for langfuse-server and one for
langfuse-db.
| Problem | Cause | Fix |
|---|---|---|
| Port 3000 already in use | Another app is using port 3000 | Change "3000:3000" to "3001:3000" in docker-compose.yml, then use
http://localhost:3001 |
| Container exits immediately | Database not ready yet | Wait 10s and run docker compose up -d again. The healthcheck retries automatically. |
| Cannot connect to localhost:3000 | Docker Desktop not running | Open Docker Desktop from Applications, wait for the whale icon to stop animating, then retry. |
| docker compose not found | Older Docker version | Try docker-compose up -d (with hyphen) |
# Stop containers (data is preserved) docker compose down # Start again docker compose up -d # Wipe all data and start fresh (destructive) docker compose down -v
llm-evaluation-test14.config.yaml under phase5_langfuse or set them as environment variables
(LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY). The config.yaml file is
committed โ if you put keys there, do not commit that change.
phase5_langfuse: BOT_MODEL: "llama3.1:8b" BOT_PROVIDER: "ollama" EXCEL_FILE: "banking_test_cases.xlsx" TEST_SHEET: "Test Cases" LANGFUSE_HOST: "http://localhost:3000" LANGFUSE_PUBLIC_KEY: "pk-lf-..." LANGFUSE_SECRET_KEY: "sk-lf-..."
export LANGFUSE_PUBLIC_KEY="pk-lf-..." export LANGFUSE_SECRET_KEY="sk-lf-..."
pip install "langfuse>=2,<3" "openai>=1.30.0,<2"
The openai package is a dependency of the Langfuse SDK โ it is needed even though we are using
Ollama. You do not need an OpenAI API key.
cd phase-5-production-eval/langfuse-observability python3 create_test_cases.py
This creates banking_test_cases.xlsx with 20 test cases across 6 categories. Run it once.
cd phase-5-production-eval/langfuse-observability python3 test14_langfuse_evaluator.py
The script asks how many cases to run โ start with 5 to verify the pipeline, then run all 20. Each case appears
in your terminal as it completes, and its trace is sent to Langfuse in real time. After the script finishes,
langfuse.flush() ensures all queued traces are delivered before the process exits.
After the script runs, open http://localhost:3000. Here is a guided tour of what to look for and why it matters.
banking-bot-tc01), when it ran, the latency, and the tags.
llm-call span shows exactly what was sent to the model and what
came back, with its own latencykeyword_pass (1 or 0), response_length_chars,
latency_secondssecurity. You now see only the 4 security-category traces.
keyword_pass = 0. You now see only the failing cases.
model: llama3.1:8b. In this test all traces use the same model, so all 20 will appear.
BOT_MODEL to
mistral and rerun, you can filter by model and compare the keyword_pass rate between the two
model versions side by side. This is A/B comparison without any extra tooling.
session_id. In this test each trace is independent (one question, one response). But in a real
chatbot a user might ask 5 follow-up questions in one session.
The key pattern in test14_langfuse_evaluator.py is three nested calls:
trace = langfuse.trace(name=..., input=question, tags=[category, model])
span = trace.span(name="llm-call", input={"question": question})
response = ask_banking_bot(question, model, ollama_url) # the actual LLM call
span.end(output={"response": response})
trace.score(name="keyword_pass", value=1.0)
trace.score(name="latency_seconds", value=2.3)
ask_banking_bot() function in
banking_bot.py is unchanged โ it just calls Ollama and returns a string. All Langfuse instrumentation
is in the evaluator script, wrapping the call. This is the correct pattern: observability is added at the
orchestration layer, not inside the model call itself.
phase5_langfuse: BOT_MODEL: "llama3.1:8b" # model to use for the banking bot BOT_PROVIDER: "ollama" # provider (ollama / openai / anthropic / gemini) EXCEL_FILE: "banking_test_cases.xlsx" TEST_SHEET: "Test Cases" LANGFUSE_HOST: "http://localhost:3000" LANGFUSE_PUBLIC_KEY: "" # from Settings โ API Keys in dashboard LANGFUSE_SECRET_KEY: "" # from Settings โ API Keys in dashboard
trace() / span()
pattern which shows exactly what is happening.Langfuse is the bridge between offline evaluation (Tests 1โ13) and production monitoring. In Tests 1โ13 you run a script, get a result, and move on. In a live system you cannot do that โ the model is serving users continuously. Langfuse lets you attach the same evaluation logic (keyword pass, scores, categories) to every live request and watch the results accumulate over time.
The mental model to take from Test 14: evaluation logic does not change โ keyword checks, categories, pass/fail rules are the same as Test 4. What changes is that Langfuse stores the results, makes them searchable, and lets you track trends instead of just running a batch and discarding the output.