Phase 5 โ€” Production Eval

Test 14: Langfuse Observability

This test introduces observability for LLM applications. You will set up Langfuse locally using Docker, instrument a banking chatbot to send every interaction as a trace, and explore what the Langfuse dashboard reveals about your model's behaviour โ€” latency, pass rates, and failure patterns โ€” without changing how the model runs.

๐Ÿ“ phase-5-production-eval/langfuse-observability/  |  test14_langfuse_evaluator.py ยท banking_bot.py ยท docker-compose.yml ยท banking_test_cases.xlsx
โš ๏ธ This test requires Docker. Langfuse runs as a local Docker container โ€” it does not call any external API. Your traces, scores, and model outputs stay entirely on your machine. If you do not have Docker installed, complete Step 1 below before anything else.

What Observability Means in LLM Systems

In traditional software, observability means logs, metrics, and traces. You can tell exactly what happened, when, and why a failure occurred. LLM systems have the same need โ€” but the inputs and outputs are unstructured text, not typed function calls. A model can return the "right" response category and still be wrong in subtle ways.

Langfuse is the observability layer for LLM applications. Every time your bot calls a model, Langfuse records the prompt, the response, the latency, and any scores you attach. Over many runs you can see patterns that a single test run will never show: which categories fail most often, which questions are slow, whether a model change improved or degraded quality.

๐Ÿ” The key mental shift. Evaluation (Tests 1โ€“13) gives you a verdict per run. Observability gives you a history. With Langfuse you are not asking "did this test pass?" โ€” you are asking "how is this system behaving over time, across categories, and across model versions?"

Core Langfuse Concepts

๐Ÿ”
Trace
One complete interaction โ€” e.g. one test case from question to final response. The top-level unit in Langfuse.
๐Ÿ“ฆ
Span
A step inside a trace โ€” e.g. the LLM call itself. Spans capture input, output, and duration of one operation.
๐Ÿท๏ธ
Score
A numeric value attached to a trace โ€” e.g. keyword_pass=1, latency=2.3s. Used for filtering and trend analysis.
๐Ÿ—‚๏ธ
Tags
Labels on a trace โ€” e.g. "security", "loans". Let you filter all traces of one category in the dashboard.
๐Ÿ“Š
Metadata
Arbitrary key-value data on a trace โ€” model name, case number, notes. Searchable in the dashboard.

What We Built

File Purpose Notes
test14_langfuse_evaluator.py The main script. Loads test cases, runs the banking bot, sends every interaction to Langfuse as a trace with spans and scores. Config-driven via config.yaml
banking_bot.py Self-contained banking chatbot. Calls Ollama directly โ€” no imports from other phase folders. System prompt hardcoded inside this file
docker-compose.yml Spins up Langfuse + PostgreSQL locally. One command to start. All data stays on your machine. Uses langfuse/langfuse:2 image
banking_test_cases.xlsx 20 test cases across 6 categories: account_info, transactions, loans, security, fees, out_of_scope. Generate with create_test_cases.py if missing
create_test_cases.py One-time script to generate the Excel test case file. Run once, then delete or keep

Step 1 โ€” Install Docker (one time only)

Langfuse runs inside Docker containers. Docker is a tool that packages an application and everything it needs (databases, servers) into isolated containers that run the same way on any machine.

โš ๏ธ If Docker is already installed, skip to Step 2. Check with:
docker --version
If you see a version number, Docker is installed.

Install Docker Desktop (Mac)

  1. Go to https://www.docker.com/products/docker-desktop/
  2. Download the installer for your chip: Apple Silicon (M1/M2/M3) or Intel.
  3. Open the downloaded .dmg file and drag Docker to Applications.
  4. Open Docker from Applications. It will ask for your password to install helper tools โ€” allow it.
  5. Wait for the Docker whale icon in your menu bar to stop animating. Docker is ready when the icon is still.
  6. Verify in terminal:
    docker --version
    docker compose version
    Both should print version numbers. If docker compose fails, try docker-compose (older installs use the hyphen form).
๐Ÿ’ก What Docker is doing here. When you run docker compose up -d, Docker will download two images โ€” langfuse/langfuse:2 (the Langfuse server, ~500 MB) and postgres:16-alpine (the database, ~100 MB). These are downloaded once and cached locally. All subsequent starts are instant. Your data is stored in a Docker volume on your machine โ€” nothing is sent anywhere.

Step 2 โ€” Start Langfuse with Docker Compose

The docker-compose.yml file in this folder defines two services: the Langfuse server and a PostgreSQL database. One command starts both.

cd phase-5-production-eval/langfuse-observability
docker compose up -d

The -d flag means "detached" โ€” the containers run in the background. You will see Docker pulling images on first run (this takes 1โ€“3 minutes depending on your internet speed). Subsequent starts take a few seconds.

After the command completes, verify both containers are running:

docker compose ps

You should see two rows with State: running โ€” one for langfuse-server and one for langfuse-db.

โœ“ Open the Langfuse dashboard: http://localhost:3000

You will see a registration screen on first launch. Create a local account โ€” email and password are stored only in your local PostgreSQL container.

Common Docker issues

Problem Cause Fix
Port 3000 already in use Another app is using port 3000 Change "3000:3000" to "3001:3000" in docker-compose.yml, then use http://localhost:3001
Container exits immediately Database not ready yet Wait 10s and run docker compose up -d again. The healthcheck retries automatically.
Cannot connect to localhost:3000 Docker Desktop not running Open Docker Desktop from Applications, wait for the whale icon to stop animating, then retry.
docker compose not found Older Docker version Try docker-compose up -d (with hyphen)

Stop and restart Langfuse

# Stop containers (data is preserved)
docker compose down

# Start again
docker compose up -d

# Wipe all data and start fresh (destructive)
docker compose down -v

Step 3 โ€” Create Your Langfuse Account and Get API Keys

  1. Open http://localhost:3000 in your browser.
  2. Click Sign Up and create a local account with any email and password. These are stored only in your local Docker database โ€” they are not sent anywhere.
  3. After signing in, click Create a new project. Name it anything, e.g. llm-evaluation-test14.
  4. In the left sidebar, go to Settings โ†’ API Keys.
  5. Click Create new API key. Langfuse generates a Public Key and a Secret Key.
  6. Copy both keys. You will need them in Step 4.
โš ๏ธ Keep your keys local. These keys only work on your local Langfuse instance โ€” they have no access to any external service. Still, treat them like any credential: do not commit them to Git. Add them to config.yaml under phase5_langfuse or set them as environment variables (LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY). The config.yaml file is committed โ€” if you put keys there, do not commit that change.

Step 4 โ€” Configure Keys and Run the Evaluator

Option A โ€” Set keys in config.yaml (easiest, do not commit)

phase5_langfuse:
  BOT_MODEL: "llama3.1:8b"
  BOT_PROVIDER: "ollama"
  EXCEL_FILE: "banking_test_cases.xlsx"
  TEST_SHEET: "Test Cases"
  LANGFUSE_HOST: "http://localhost:3000"
  LANGFUSE_PUBLIC_KEY: "pk-lf-..."
  LANGFUSE_SECRET_KEY: "sk-lf-..."

Option B โ€” Set keys as environment variables

export LANGFUSE_PUBLIC_KEY="pk-lf-..."
export LANGFUSE_SECRET_KEY="sk-lf-..."

Install the Python SDK

pip install "langfuse>=2,<3" "openai>=1.30.0,<2"

The openai package is a dependency of the Langfuse SDK โ€” it is needed even though we are using Ollama. You do not need an OpenAI API key.

Generate the test case Excel file

cd phase-5-production-eval/langfuse-observability
python3 create_test_cases.py

This creates banking_test_cases.xlsx with 20 test cases across 6 categories. Run it once.

Run the evaluator

cd phase-5-production-eval/langfuse-observability
python3 test14_langfuse_evaluator.py

The script asks how many cases to run โ€” start with 5 to verify the pipeline, then run all 20. Each case appears in your terminal as it completes, and its trace is sent to Langfuse in real time. After the script finishes, langfuse.flush() ensures all queued traces are delivered before the process exits.

Step 5 โ€” What to Explore in the Langfuse Dashboard

After the script runs, open http://localhost:3000. Here is a guided tour of what to look for and why it matters.

Dashboard Tour ยท 01
Traces list โ€” see every interaction
Click Traces in the left sidebar. You will see one row per test case. Each row shows the trace name (e.g. banking-bot-tc01), when it ran, the latency, and the tags.

What to notice: The total latency column shows how long each case took end-to-end. Look for outliers โ€” cases that took 2ร— longer than average. That is a signal worth investigating. Click any trace to open its detail view.
Dashboard Tour ยท 02
Trace detail โ€” drill into one interaction
Inside a trace you see: What to notice: Compare the span latency vs the trace latency. They should be nearly equal since the only operation is one LLM call. If they differ significantly, that is overhead worth profiling in a real production system.
Dashboard Tour ยท 03
Filter by tag โ€” isolate a category
In the Traces list, use the Tags filter dropdown and select security. You now see only the 4 security-category traces.

What to notice: Do the security cases have a lower keyword_pass rate? Security cases have stricter keyword rules (must warn about phishing, must not say "provide password"). If any fail, click into the trace to read the full response and understand why.
Dashboard Tour ยท 04
Filter by score โ€” find all keyword failures
In the Traces list, use the Scores filter and set keyword_pass = 0. You now see only the failing cases.

What to notice: Which categories fail most? Are failures clustered in one category or spread across all? This is the same question a QA engineer asks when looking at a defect category distribution โ€” the technique is identical, just applied to LLM outputs.
Dashboard Tour ยท 05
Latency comparison across categories
Sort the Traces list by latency (click the latency column header). Look at which categories produce the longest responses.

What to notice: Loan and out_of_scope questions often produce longer responses โ€” the model either explains loan eligibility in detail or adds disclaimers. Account_info questions are usually shorter and faster. This latency pattern is directly relevant in production: longer responses cost more tokens and increase response time for users.
Dashboard Tour ยท 06
Metadata search โ€” find traces by model
In the Traces list, use the search or metadata filter and look for model: llama3.1:8b. In this test all traces use the same model, so all 20 will appear.

Why this matters in production: If you later change BOT_MODEL to mistral and rerun, you can filter by model and compare the keyword_pass rate between the two model versions side by side. This is A/B comparison without any extra tooling.
Dashboard Tour ยท 07
Sessions โ€” multi-turn conversations
Langfuse supports grouping multiple traces into a Session using a session_id. In this test each trace is independent (one question, one response). But in a real chatbot a user might ask 5 follow-up questions in one session.

What to think about: How would you evaluate a multi-turn conversation? The quality of turn 3 depends on turns 1 and 2. Langfuse sessions let you see the full conversation history as one unit. Test 15 explores this directly with DeepEval's ConversationalMetric.

How the Script Instruments the Bot

The key pattern in test14_langfuse_evaluator.py is three nested calls:

trace = langfuse.trace(name=..., input=question, tags=[category, model])

span = trace.span(name="llm-call", input={"question": question})
response = ask_banking_bot(question, model, ollama_url)   # the actual LLM call
span.end(output={"response": response})

trace.score(name="keyword_pass", value=1.0)
trace.score(name="latency_seconds", value=2.3)
๐Ÿ’ก The bot does not know it is being observed. The ask_banking_bot() function in banking_bot.py is unchanged โ€” it just calls Ollama and returns a string. All Langfuse instrumentation is in the evaluator script, wrapping the call. This is the correct pattern: observability is added at the orchestration layer, not inside the model call itself.

Config Reference

phase5_langfuse:
  BOT_MODEL: "llama3.1:8b"          # model to use for the banking bot
  BOT_PROVIDER: "ollama"             # provider (ollama / openai / anthropic / gemini)
  EXCEL_FILE: "banking_test_cases.xlsx"
  TEST_SHEET: "Test Cases"
  LANGFUSE_HOST: "http://localhost:3000"
  LANGFUSE_PUBLIC_KEY: ""            # from Settings โ†’ API Keys in dashboard
  LANGFUSE_SECRET_KEY: ""            # from Settings โ†’ API Keys in dashboard

What This Test Does Not Cover

What This Sets Up

Langfuse is the bridge between offline evaluation (Tests 1โ€“13) and production monitoring. In Tests 1โ€“13 you run a script, get a result, and move on. In a live system you cannot do that โ€” the model is serving users continuously. Langfuse lets you attach the same evaluation logic (keyword pass, scores, categories) to every live request and watch the results accumulate over time.

The mental model to take from Test 14: evaluation logic does not change โ€” keyword checks, categories, pass/fail rules are the same as Test 4. What changes is that Langfuse stores the results, makes them searchable, and lets you track trends instead of just running a batch and discarding the output.