Pytest LLM evals

ghostrun lets Python developers write LLM evals as ordinary pytest tests, next to the code they protect.

Why pytest is the right home

Developers already trust pytest for regressions, fixtures, CI, failure output, and pull request checks. LLM evals should fit that workflow instead of forcing every app team into a separate dashboard or YAML-only eval system.

What an eval looks like

@ghostrun.record(model="gpt-4o-mini")
def test_support_reply_semantics():
    reply = generate_reply("I was charged twice")

    ghostrun.expect(reply).contains_intent("acknowledge billing issue")
    ghostrun.expect(reply).contains_intent("offer next step")
    ghostrun.expect(reply).tone_is("calm")

The first run records the real LLM call. Later runs replay the response and evaluate the assertions without repeating the provider request.

What to test

• Intent: did the reply include the required policy or instruction?
• Tone: did the answer stay empathetic, concise, professional, or safe?
• Negative behavior: did it avoid arguing, hallucinating, or leaking internal details?
• Structure: did it produce valid JSON, expected fields, or tool calls?
• Regression state: did a prompt edit make an assertion fail compared with the previous run?

When to use ghostrun

Use ghostrun when you want app-native LLM evals in Python: real code, real calls, deterministic replay, semantic assertions, and CI-friendly failures.