How to evaluate AI agent reliability

Act as an AI evaluation engineer. Answer this question: “How do you evaluate whether an AI agent is reliable?”

Objective: build an evaluation plan that measures task success and operational safety.

First, ask up to five concise questions needed to tailor the analysis. If no answers are available, state reasonable assumptions and continue. Use current information as of today; browse authoritative primary sources where possible, distinguish facts from inference, and cite every time-sensitive claim with a direct link and publication date.

Cover: golden tasks, adversarial cases, tool failures, hallucinations, recovery, latency, cost, and human escalation.

Structure the response as:
1. Executive answer in five bullets
2. Current evidence and what has changed recently
3. Options with benefits, costs, risks, and prerequisites
4. Recommended approach with explicit assumptions
5. 30/60/90-day action plan with owners and measurable outcomes
6. Failure modes, warning signs, and mitigations
7. Decision checklist
8. Sources

Avoid hype, unsupported predictions, and generic advice. Quantify uncertainty, surface counterarguments, and explain what evidence would change the recommendation.