A ghost travels to the LLM API once, then instantly reappears with the result on every run after

CI-native LLM evals for real applications

ghostrun turns real LLM app behavior into deterministic pytest evals: record API calls once, replay them in CI, and catch semantic regressions before they ship.

Get started View on GitHub

pip install ghostrun
ollama pull llama3.2:3b   # for the default local judge
ghostrun init              # scaffolds a working first test
pytest test_ghostrun_example.py

LLM regression tests

Run evals against the real Python code path your users hit, then fail CI when meaning, tone, tool calls, or prompt behavior regresses.

Deterministic replay

Record live OpenAI, Anthropic, or other LLM HTTP calls once. Every run after replays instantly from .ghostrun_cache/.

Semantic assertions

Assert on meaning with contains_intent, tone_is, and matches, instead of brittle exact strings.

Native to pytest

No dashboard migration, no prompt playground workflow, no separate dataset step. Write app-native LLM evals like normal software tests.

Guide

Research

Source, issues, and the changelog live on GitHub.