public

The question that cleaned up an eval set for prompts

This started as a messy question about an eval set for prompts. The follow-up made the answer practical.

AI & Prompt Engineeringclaude-opus-4.5#ai #claude #evals #prompting

Caleb Diaz

Help me build a small test set with ordinary cases, edge cases, and clear pass criteria.

Context:

Goal: [what a good result lets me do]
Current situation: [paste notes, draft, data, or constraints]
Audience: [who will use the result]
Deadline or limit: [time, budget, tools, length]

Before answering, ask up to three questions only if the answers would materially change your recommendation. Otherwise, state your assumptions and continue.

Give me:

A practical first pass I can use now
The main trade-offs or risks
The smallest check or test to run next
What information would change your answer

If a fact is missing, label the gap instead of inventing a detail. Use direct language. Keep the result specific to the context I provide.

Claudeclaude-opus-4.5

This is usable as written. The important parts are the definition of done, the instruction to expose assumptions, and the request for a smallest test. For an eval set for prompts, I would start with these checks:

Put the source of truth in the prompt
Mark missing evidence instead of filling it in
Add a stop condition before tool use

If you share real material, remove anything private first and replace bracketed fields with concrete details.

Caleb Diaz

That is the version I kept. The three-question cap stopped the setup from turning into an interview.

3 comments

Julian Lin@julianlin·4w ago
Small note: I got a better result after pasting a bad example as well as a good one.
Mae Cole@maecole·4w ago
This would have saved me an hour last week. My prompt never said what done looked like.
Mae Singh@maesingh·2w ago
Useful distinction between having information and being able to apply it.