#testing prompts and AI conversations

My notes from fixing a safe dependency upgrade

The useful bit was asking for trade-offs and a smallest test before committing to a full plan.

Samir Shah

I am stuck on a safe dependency upgrade. I need to plan a major dependency upgrade with compatibility checks and a rollback path, but my current attempt is getting vague and too big. What would you check first?

ChatGPTgpt-5

First, shrink the decision. Do not solve the whole subject at once. 1. Reproduce the failure before changing code. 2. Keep the patch smaller than the diagnosis. 3. Write the rollback before deployment. Write down the result you need by the end of this session. Then choose one input you trust and make a first pass that can be checked. If that pass fails, you will know which assumption to revisit instead of starting over.

Show all 3 messages

3402.3k

Samir Shah@samirshah·4w ago

A blunt review of my approach to a safe dependency upgrade

This started as a messy question about a safe dependency upgrade. The follow-up made the answer practical.

Samir Shah

Give me a short checklist for a safe dependency upgrade. I want something I can run before I call the work done.

ChatGPTgpt-5

Use this pass: - [ ] The goal is one sentence and names the user or decision - [ ] Inputs are current, relevant, and traceable - [ ] Assumptions are visible - [ ] Reproduce the failure before changing code - [ ] Keep the patch smaller than the diagnosis - [ ] Write the rollback before deployment - [ ] One failure case has been tested - [ ] The next owner and next action are clear If you cannot check an item, mark it unknown. Do not quietly treat unknown as done.

Show all 3 messages

3304.2k

Nico Lane@nicolane·4w ago

I needed a usable answer on a flaky test nobody can reproduce

This started as a messy question about a flaky test nobody can reproduce. The follow-up made the answer practical.

Nico Lane

I am stuck on a flaky test nobody can reproduce. I need to isolate a timing-dependent test failure and design a deterministic reproduction, but my current attempt is getting vague and too big. What would you check first?

Geminigemini-2.5-pro

First, shrink the decision. Do not solve the whole subject at once. 1. Reproduce the failure before changing code. 2. Keep the patch smaller than the diagnosis. 3. Write the rollback before deployment. Write down the result you need by the end of this session. Then choose one input you trust and make a first pass that can be checked. If that pass fails, you will know which assumption to revisit instead of starting over.

Show all 3 messages

3402.4k

Caleb Diaz@calebdiaz·4w ago

The question that cleaned up an eval set for prompts

This started as a messy question about an eval set for prompts. The follow-up made the answer practical.

Caleb Diaz

Help me build a small test set with ordinary cases, edge cases, and clear pass criteria. Context: - Goal: [what a good result lets me do] - Current situation: [paste notes, draft, data, or constraints] - Audience: [who will use the result] - Deadline or limit: [time, budget, tools, length] Before answering, ask up to three questions only if the answers would materially change your recommendation.

Claudeclaude-opus-4.5

This is usable as written. The important parts are the definition of done, the instruction to expose assumptions, and the request for a smallest test. For an eval set for prompts, I would start with these checks: - Put the source of truth in the prompt - Mark missing evidence instead of filling it in - Add a stop condition before tool use If you share real material, remove anything private first and replace bracketed fields with concrete details.

Show all 3 messages

3311.7k

Nico Lane@nicolane·1mo ago

a flaky test nobody can reproduce, minus the vague advice

Sharing the wording because it gave me something I could check instead of a confident wall of text.

Nico Lane

Give me a short checklist for a flaky test nobody can reproduce. I want something I can run before I call the work done.

Geminigemini-2.5-pro

Use this pass: - [ ] The goal is one sentence and names the user or decision - [ ] Inputs are current, relevant, and traceable - [ ] Assumptions are visible - [ ] Reproduce the failure before changing code - [ ] Keep the patch smaller than the diagnosis - [ ] Write the rollback before deployment - [ ] One failure case has been tested - [ ] The next owner and next action are clear If you cannot check an item, mark it unknown. Do not quietly treat unknown as done.

Show all 3 messages

3302.3k

Samir Shah@samirshah·2mo ago

How I ask for help with a safe dependency upgrade

A compact prompt and the response it produced. Replace the bracketed parts with your own details.

Samir Shah

Be blunt about my plan for a safe dependency upgrade. I am going to collect everything, ask for a polished answer, and fix details later. What is likely to go wrong?

Claudeclaude-opus-4.5

Three things. - “Everything” has no boundary, so weak inputs and strong inputs will be treated alike. - A polished answer can hide missing evidence. Ask for gaps and assumptions before prose. - “Fix details later” means there is no acceptance test now. A safer sequence is: define the decision, choose the minimum trusted inputs, make an ugly first pass, check it, then polish. For this task, keep the patch smaller than the diagnosis.

Show all 3 messages

3302.8k

Samir Shah@samirshah·3mo ago

Gemini found the hole in my a safe dependency upgrade plan

Sharing the wording because it gave me something I could check instead of a confident wall of text.

Samir Shah

For a safe dependency upgrade, should I build a complete plan first or test one small slice? I have limited time and I do not want throwaway work.

Geminigemini-2.5-pro

Test one slice, but choose a slice that crosses the riskiest part of the problem. A tiny easy demo will only prove the easy part. Use this decision rule: - If reversing the choice is cheap, try it. - If the choice locks in data, money, or other people’s time, write the full decision first. - If uncertainty is concentrated in one step, test that step before planning around it. For a safe dependency upgrade, make the first test produce evidence you can compare, not just something that looks complete.

Show all 3 messages

3404.2k

Nico Lane@nicolane·2mo ago

ChatGPT was overcomplicating a flaky test nobody can reproduce, so I reset the question

The useful bit was asking for trade-offs and a smallest test before committing to a full plan.

Nico Lane

Be blunt about my plan for a flaky test nobody can reproduce. I am going to collect everything, ask for a polished answer, and fix details later. What is likely to go wrong?

ChatGPTgpt-5

Three things. - “Everything” has no boundary, so weak inputs and strong inputs will be treated alike. - A polished answer can hide missing evidence. Ask for gaps and assumptions before prose. - “Fix details later” means there is no acceptance test now. A safer sequence is: define the decision, choose the minimum trusted inputs, make an ugly first pass, check it, then polish. For this task, keep the patch smaller than the diagnosis.

Show all 3 messages

3302.4k

Caleb Diaz@calebdiaz·3mo ago

an eval set for prompts, minus the vague advice

This started as a messy question about an eval set for prompts. The follow-up made the answer practical.

Caleb Diaz

Give me a short checklist for an eval set for prompts. I want something I can run before I call the work done.

Geminigemini-2.5-pro

Use this pass: - [ ] The goal is one sentence and names the user or decision - [ ] Inputs are current, relevant, and traceable - [ ] Assumptions are visible - [ ] Put the source of truth in the prompt - [ ] Mark missing evidence instead of filling it in - [ ] Add a stop condition before tool use - [ ] One failure case has been tested - [ ] The next owner and next action are clear If you cannot check an item, mark it unknown. Do not quietly treat unknown as done.

Show all 3 messages

3313k

Nico Lane@nicolane·3mo ago

I stopped guessing about a flaky test nobody can reproduce

A compact prompt and the response it produced. Replace the bracketed parts with your own details.

Nico Lane

For a flaky test nobody can reproduce, should I build a complete plan first or test one small slice? I have limited time and I do not want throwaway work.

Claudeclaude-opus-4.5

Test one slice, but choose a slice that crosses the riskiest part of the problem. A tiny easy demo will only prove the easy part. Use this decision rule: - If reversing the choice is cheap, try it. - If the choice locks in data, money, or other people’s time, write the full decision first. - If uncertainty is concentrated in one step, test that step before planning around it. For a flaky test nobody can reproduce, make the first test produce evidence you can compare, not just something that looks complete.

Show all 3 messages

3403.8k

Hugo Wright@hugowright·2w ago

Forked from@calebdiaz/The question that cleaned up an eval set for prompts

The question that cleaned up an eval set for prompts

This started as a messy question about an eval set for prompts. The follow-up made the answer practical.

Hugo Wright

Help me build a small test set with ordinary cases, edge cases, and clear pass criteria. Context: - Goal: [what a good result lets me do] - Current situation: [paste notes, draft, data, or constraints] - Audience: [who will use the result] - Deadline or limit: [time, budget, tools, length] Before answering, ask up to three questions only if the answers would materially change your recommendation.

Claudeclaude-opus-4.5

This is usable as written. The important parts are the definition of done, the instruction to expose assumptions, and the request for a smallest test. For an eval set for prompts, I would start with these checks: - Put the source of truth in the prompt - Mark missing evidence instead of filling it in - Add a stop condition before tool use If you share real material, remove anything private first and replace bracketed fields with concrete details.

Show all 3 messages

3200

Mira Martin@miramartin·2w ago

Forked from@calebdiaz/I needed a usable answer on an eval set for prompts

I needed a usable answer on an eval set for prompts

The useful bit was asking for trade-offs and a smallest test before committing to a full plan.

Mira Martin

I am stuck on an eval set for prompts. I need to build a small test set with ordinary cases, edge cases, and clear pass criteria, but my current attempt is getting vague and too big. What would you check first?

Geminigemini-2.5-pro

First, shrink the decision. Do not solve the whole subject at once. 1. Put the source of truth in the prompt. 2. Mark missing evidence instead of filling it in. 3. Add a stop condition before tool use. Write down the result you need by the end of this session. Then choose one input you trust and make a first pass that can be checked. If that pass fails, you will know which assumption to revisit instead of starting over.

Show all 3 messages

3200

Zara Shah@zarashah·2w ago

Forked from@calebdiaz/ChatGPT was overcomplicating an eval set for prompts, so I reset the question

ChatGPT was overcomplicating an eval set for prompts, so I reset the question

A compact prompt and the response it produced. Replace the bracketed parts with your own details.

Zara Shah

Be blunt about my plan for an eval set for prompts. I am going to collect everything, ask for a polished answer, and fix details later. What is likely to go wrong?

ChatGPTgpt-5

Three things. - “Everything” has no boundary, so weak inputs and strong inputs will be treated alike. - A polished answer can hide missing evidence. Ask for gaps and assumptions before prose. - “Fix details later” means there is no acceptance test now. A safer sequence is: define the decision, choose the minimum trusted inputs, make an ugly first pass, check it, then polish. For this task, mark missing evidence instead of filling it in.

Show all 3 messages

3200

Hugo Das@hugodas·2w ago

Forked from@calebdiaz/I stopped guessing about an eval set for prompts

I stopped guessing about an eval set for prompts

Sharing the wording because it gave me something I could check instead of a confident wall of text.

Hugo Das

For an eval set for prompts, should I build a complete plan first or test one small slice? I have limited time and I do not want throwaway work.

Claudeclaude-opus-4.5

Test one slice, but choose a slice that crosses the riskiest part of the problem. A tiny easy demo will only prove the easy part. Use this decision rule: - If reversing the choice is cheap, try it. - If the choice locks in data, money, or other people’s time, write the full decision first. - If uncertainty is concentrated in one step, test that step before planning around it. For an eval set for prompts, make the first test produce evidence you can compare, not just something that looks complete.

Show all 3 messages

3200

Mira Khan@mirakhan·2w ago

Forked from@calebdiaz/an eval set for prompts, minus the vague advice

an eval set for prompts, minus the vague advice

This started as a messy question about an eval set for prompts. The follow-up made the answer practical.

Mira Khan

Give me a short checklist for an eval set for prompts. I want something I can run before I call the work done.

Geminigemini-2.5-pro

Use this pass: - [ ] The goal is one sentence and names the user or decision - [ ] Inputs are current, relevant, and traceable - [ ] Assumptions are visible - [ ] Put the source of truth in the prompt - [ ] Mark missing evidence instead of filling it in - [ ] Add a stop condition before tool use - [ ] One failure case has been tested - [ ] The next owner and next action are clear If you cannot check an item, mark it unknown. Do not quietly treat unknown as done.

Show all 3 messages

3200

Samir Shah@samirshah·2mo ago

What finally made a safe dependency upgrade click for me

This started as a messy question about a safe dependency upgrade. The follow-up made the answer practical.

Samir Shah

Help me plan a major dependency upgrade with compatibility checks and a rollback path. Context: - Goal: [what a good result lets me do] - Current situation: [paste notes, draft, data, or constraints] - Audience: [who will use the result] - Deadline or limit: [time, budget, tools, length] Before answering, ask up to three questions only if the answers would materially change your recommendation. Ot

Geminigemini-2.5-pro

This is usable as written. The important parts are the definition of done, the instruction to expose assumptions, and the request for a smallest test. For a safe dependency upgrade, I would start with these checks: - Reproduce the failure before changing code - Keep the patch smaller than the diagnosis - Write the rollback before deployment If you share real material, remove anything private first and replace bracketed fields with concrete details.

Show all 3 messages

330897

Caleb Diaz@calebdiaz·3mo ago

ChatGPT was overcomplicating an eval set for prompts, so I reset the question

A compact prompt and the response it produced. Replace the bracketed parts with your own details.

Caleb Diaz

Be blunt about my plan for an eval set for prompts. I am going to collect everything, ask for a polished answer, and fix details later. What is likely to go wrong?

ChatGPTgpt-5

Three things. - “Everything” has no boundary, so weak inputs and strong inputs will be treated alike. - A polished answer can hide missing evidence. Ask for gaps and assumptions before prose. - “Fix details later” means there is no acceptance test now. A safer sequence is: define the decision, choose the minimum trusted inputs, make an ugly first pass, check it, then polish. For this task, mark missing evidence instead of filling it in.

Show all 3 messages

3311.6k

Caleb Diaz@calebdiaz·2mo ago

I stopped guessing about an eval set for prompts

Sharing the wording because it gave me something I could check instead of a confident wall of text.

Caleb Diaz

For an eval set for prompts, should I build a complete plan first or test one small slice? I have limited time and I do not want throwaway work.

Claudeclaude-opus-4.5

Test one slice, but choose a slice that crosses the riskiest part of the problem. A tiny easy demo will only prove the easy part. Use this decision rule: - If reversing the choice is cheap, try it. - If the choice locks in data, money, or other people’s time, write the full decision first. - If uncertainty is concentrated in one step, test that step before planning around it. For an eval set for prompts, make the first test produce evidence you can compare, not just something that looks complete.

Show all 3 messages

341195

Nico Lane@nicolane·3mo ago

The question that cleaned up a flaky test nobody can reproduce

Sharing the wording because it gave me something I could check instead of a confident wall of text.

Nico Lane

Help me isolate a timing-dependent test failure and design a deterministic reproduction. Context: - Goal: [what a good result lets me do] - Current situation: [paste notes, draft, data, or constraints] - Audience: [who will use the result] - Deadline or limit: [time, budget, tools, length] Before answering, ask up to three questions only if the answers would materially change your recommendation.

Claudeclaude-opus-4.5

This is usable as written. The important parts are the definition of done, the instruction to expose assumptions, and the request for a smallest test. For a flaky test nobody can reproduce, I would start with these checks: - Reproduce the failure before changing code - Keep the patch smaller than the diagnosis - Write the rollback before deployment If you share real material, remove anything private first and replace bracketed fields with concrete details.

Show all 3 messages

3301k

Caleb Diaz@calebdiaz·2mo ago

I needed a usable answer on an eval set for prompts

The useful bit was asking for trade-offs and a smallest test before committing to a full plan.

Caleb Diaz

I am stuck on an eval set for prompts. I need to build a small test set with ordinary cases, edge cases, and clear pass criteria, but my current attempt is getting vague and too big. What would you check first?

Geminigemini-2.5-pro

First, shrink the decision. Do not solve the whole subject at once. 1. Put the source of truth in the prompt. 2. Mark missing evidence instead of filling it in. 3. Add a stop condition before tool use. Write down the result you need by the end of this session. Then choose one input you trust and make a first pass that can be checked. If that pass fails, you will know which assumption to revisit instead of starting over.

Show all 3 messages

341309

regex raccoon@regex_raccoon·1w ago

Best prompt for a WCAG accessibility audit

A standards-based audit prompt combining automated checks, keyboard and screen-reader flows, responsive states, and concrete remediation.

regex raccoon

# Best prompt for a WCAG accessibility audit Act as a accessibility engineer. ## Goal Find reproducible accessibility barriers and propose fixes that preserve product behavior. ## Inputs - pages and critical user journeys - target WCAG level - supported browsers and assistive technology - code, screenshots, and test results If a required input is missing, ask concise questions or mark the assumpti

Claudeclaude-opus-4.6

Use the attached prompt by replacing its input bullets with the real context. The strongest results come from evidence-bearing inputs and explicit success criteria. The required deliverables are: prioritized findings, reproduction steps, code-level remediation, manual and automated retest plan.

Show all 4 messages

4000

token toaster@token_toaster·1w ago

Best prompt for an AI code review

A defect-focused review prompt that prioritizes correctness, security, regressions, and missing tests over stylistic commentary.

token toaster

# Best prompt for an AI code review Act as a skeptical senior code reviewer. ## Goal Find actionable defects introduced by a change and explain their runtime impact. ## Inputs - base and head diff - repository instructions - tests and CI results - affected API or data contracts If a required input is missing, ask concise questions or mark the assumption explicitly. Do not silently invent operation

Claudeclaude-opus-4.6

Use the attached prompt by replacing its input bullets with the real context. The strongest results come from evidence-bearing inputs and explicit success criteria. The required deliverables are: ordered findings, reproduction scenario, precise file and line references, missing-test recommendations.

Show all 4 messages

4000

latency lobster@latency_lobster·1w ago

Best prompt for migrating a codebase with an AI coding agent

A staged migration prompt that forces documentation checks, compatibility inventories, small diffs, and executable verification.

latency lobster

# Best prompt for migrating a codebase with an AI coding agent Act as a staff software engineer leading a framework migration. ## Goal Plan and execute a reversible codebase migration with evidence at each step. ## Inputs - repository and target version - official migration guides - current tests and deployment path - compatibility and downtime constraints If a required input is missing, ask conci

ChatGPTgpt-5

Use the attached prompt by replacing its input bullets with the real context. The strongest results come from evidence-bearing inputs and explicit success criteria. The required deliverables are: compatibility inventory, ordered migration plan, patches by milestone, verification and rollback report.

Show all 4 messages

4000

divs before dawn@divs_before_dawn·1w ago

How do I evaluate an AI agent before production?

A risk-based evaluation prompt covering task success, tool use, safety, latency, cost, and regression gates.

divs before dawn

# How do I evaluate an AI agent before production? Act as a AI evaluation lead. ## Goal Design a production-readiness evaluation suite tied to measurable failure modes. ## Inputs - agent goal and user population - tool and data access - known failure examples - latency, quality, safety, and cost targets If a required input is missing, ask concise questions or mark the assumption explicitly. Do not

Geminigemini-2.5-pro

Use the attached prompt by replacing its input bullets with the real context. The strongest results come from evidence-bearing inputs and explicit success criteria. The required deliverables are: evaluation matrix, test-case schema, scoring rubric, release and rollback gates.

Show all 4 messages

4000

The pitter@pitter·1w ago

How to use AI agents for software testing

A copy-ready, evidence-first prompt for the timely question: How can AI agents improve software testing?

The pitter

What is the best prompt for this trending question: How can AI agents improve software testing?

ChatGPTgpt-5.6

## Copy-ready prompt ```text Act as a quality engineering leader. Answer this question: “How can AI agents improve software testing?” Objective: design an agentic testing strategy that finds meaningful regressions instead of inflating test counts. First, ask up to five concise questions needed to tailor the analysis. If no answers are available, state reasonable assumptions and continue. Use current information as of today; browse authoritative primary sources where possible, distinguish facts from inference, and cite every time-sensitive claim with a direct link and publication date. Cover: r

Open conversation

2000

Explore

Explore

Explore

My notes from fixing a safe dependency upgrade

A blunt review of my approach to a safe dependency upgrade

I needed a usable answer on a flaky test nobody can reproduce

The question that cleaned up an eval set for prompts

a flaky test nobody can reproduce, minus the vague advice

How I ask for help with a safe dependency upgrade

Gemini found the hole in my a safe dependency upgrade plan

ChatGPT was overcomplicating a flaky test nobody can reproduce, so I reset the question

an eval set for prompts, minus the vague advice

I stopped guessing about a flaky test nobody can reproduce

The question that cleaned up an eval set for prompts

I needed a usable answer on an eval set for prompts

ChatGPT was overcomplicating an eval set for prompts, so I reset the question

I stopped guessing about an eval set for prompts

an eval set for prompts, minus the vague advice

What finally made a safe dependency upgrade click for me

ChatGPT was overcomplicating an eval set for prompts, so I reset the question

I stopped guessing about an eval set for prompts

The question that cleaned up a flaky test nobody can reproduce

I needed a usable answer on an eval set for prompts

Best prompt for a WCAG accessibility audit

Best prompt for an AI code review

Best prompt for migrating a codebase with an AI coding agent

How do I evaluate an AI agent before production?

How to use AI agents for software testing