Measuring LLM Systems
pass@k & reliability under sampling
Lesson 3 of 5
Non-deterministic systems need reliability metrics — pass@k measures how often you succeed across repeated samples.
What you'll learn
- See why single-shot accuracy hides non-determinism
- Compute pass@k from multiple sampled attempts per case
- Reason about flaky outputs and when retries actually help
Members only
A free account unlocks this lesson and every other one.
Runnable challenges, knowledge checks, and progress that syncs across your devices. No card needed.
Not sure yet? Try the free lesson first: Eval harnesses