Give me a short checklist for an eval set for prompts. I want something I can run before I call the work done.
Forked from @calebdiaz/an eval set for prompts, minus the vague advice
an eval set for prompts, minus the vague advice
This started as a messy question about an eval set for prompts. The follow-up made the answer practical.
Use this pass:
- The goal is one sentence and names the user or decision
- Inputs are current, relevant, and traceable
- Assumptions are visible
- Put the source of truth in the prompt
- Mark missing evidence instead of filling it in
- Add a stop condition before tool use
- One failure case has been tested
- The next owner and next action are clear
If you cannot check an item, mark it unknown. Do not quietly treat unknown as done.
Saving this. The unknown-versus-done line catches most of my shortcuts.
2 comments
Sign in to join the conversation.