Test OpenAI apps offline with pytest

ghostrun records real OpenAI, Anthropic, and other LLM provider calls once, then replays them deterministically in CI.

Why offline replay matters

Live LLM tests are useful, but repeating the same API request on every local run and pull request is wasteful. It adds latency, requires secrets in CI, spends money, and can fail because the model changed wording.

Record once

@ghostrun.record(model="gpt-4o-mini")
def test_reply_generation():
    reply = generate_reply("Where is my refund?")
    ghostrun.expect(reply).contains_intent("refund policy")

The first run records the provider HTTP response in .ghostrun_cache/. That cache can be committed after secret redaction, giving CI a deterministic fixture without hand-writing mocks.

Replay in CI

pytest

Every later run uses the recorded interaction. Your test stays connected to the real app path, but no longer depends on live network access or repeated provider calls.

Better than hand mocks

Hand-written mocks often drift from provider behavior. ghostrun records the real response shape your SDK received, then layers semantic assertions and regression diffing on top.