LLM regression testing in pytest

ghostrun catches LLM behavior regressions before they ship by turning real application calls into deterministic pytest evals.

The problem

LLM apps regress in ways ordinary unit tests miss. A prompt change can remove an apology, skip a refund policy, change tone, break JSON structure, or call the wrong tool while still producing plausible text.

Running those checks against live models in CI is slow, expensive, and flaky. Moving them into a separate eval platform can also drift away from the real code path users hit.

The ghostrun approach

1. Write a normal pytest test around your real app function.
2. Record the live LLM API call once with @ghostrun.record.
3. Replay that call deterministically on every later test run.
4. Assert on meaning, tone, structure, and tool calls.
5. Use ghostrun diff to classify regressions, fixes, stable checks, added checks, and removed checks.

import ghostrun
from my_app import generate_reply

@ghostrun.record(model="gpt-4o-mini")
def test_refund_reply_does_not_regress():
    reply = generate_reply("Where is my refund?")

    ghostrun.expect(reply).contains_intent("apology")
    ghostrun.expect(reply).contains_intent("refund policy")
    ghostrun.expect(reply).does_not_contain_intent("arguing")
    ghostrun.expect(reply).tone_is("empathetic")

Why this is different from generic LLM evals

Many eval tools start from a dataset, prompt matrix, trace, or dashboard. ghostrun starts from the Python code you already ship. That makes it best for app-native regression testing: CI checks that protect real user-facing behavior.

Best fit

• Testing OpenAI, Anthropic, Gemini, Bedrock, or Ollama-backed app code.
• Checking chatbot responses, RAG answers, tool calls, JSON extraction, support workflows, and prompt changes.
• Running LLM evals in pull requests without paying for the same call every time.