Explore

Tagged #flaky-tests

All AI & Prompt Engineering324 Engineering176 Data & Analytics89 Product4 Design28 Sales2 Marketing4 Finance24 Strategy31 Operations4 Customer Success3 People & Recruiting4 Security & Compliance8 Legal2 Other Professional1

#flaky-testsClear filters

Nico Lane@nicolane·4w ago

I needed a usable answer on a flaky test nobody can reproduce

This started as a messy question about a flaky test nobody can reproduce. The follow-up made the answer practical.

Nico Lane

I am stuck on a flaky test nobody can reproduce. I need to isolate a timing-dependent test failure and design a deterministic reproduction, but my current attempt is getting vague and too big. What would you check first?

Geminigemini-2.5-pro

First, shrink the decision. Do not solve the whole subject at once. 1. Reproduce the failure before changing code. 2. Keep the patch smaller than the diagnosis. 3. Write the rollback before deployment. Write down the result you need by the end of this session. Then choose one input you trust and make a first pass that can be checked. If that pass fails, you will know which assumption to revisit instead of starting over.

Show all 3 messages

3402.4k

Nico Lane@nicolane·1mo ago

a flaky test nobody can reproduce, minus the vague advice

Sharing the wording because it gave me something I could check instead of a confident wall of text.

Nico Lane

Give me a short checklist for a flaky test nobody can reproduce. I want something I can run before I call the work done.

Geminigemini-2.5-pro

Use this pass: - [ ] The goal is one sentence and names the user or decision - [ ] Inputs are current, relevant, and traceable - [ ] Assumptions are visible - [ ] Reproduce the failure before changing code - [ ] Keep the patch smaller than the diagnosis - [ ] Write the rollback before deployment - [ ] One failure case has been tested - [ ] The next owner and next action are clear If you cannot check an item, mark it unknown. Do not quietly treat unknown as done.

Show all 3 messages

3302.3k

Nico Lane@nicolane·2mo ago

ChatGPT was overcomplicating a flaky test nobody can reproduce, so I reset the question

The useful bit was asking for trade-offs and a smallest test before committing to a full plan.

Nico Lane

Be blunt about my plan for a flaky test nobody can reproduce. I am going to collect everything, ask for a polished answer, and fix details later. What is likely to go wrong?

ChatGPTgpt-5

Three things. - “Everything” has no boundary, so weak inputs and strong inputs will be treated alike. - A polished answer can hide missing evidence. Ask for gaps and assumptions before prose. - “Fix details later” means there is no acceptance test now. A safer sequence is: define the decision, choose the minimum trusted inputs, make an ugly first pass, check it, then polish. For this task, keep the patch smaller than the diagnosis.

Show all 3 messages

3302.4k

Nico Lane@nicolane·3mo ago

I stopped guessing about a flaky test nobody can reproduce

A compact prompt and the response it produced. Replace the bracketed parts with your own details.

Nico Lane

For a flaky test nobody can reproduce, should I build a complete plan first or test one small slice? I have limited time and I do not want throwaway work.

Claudeclaude-opus-4.5

Test one slice, but choose a slice that crosses the riskiest part of the problem. A tiny easy demo will only prove the easy part. Use this decision rule: - If reversing the choice is cheap, try it. - If the choice locks in data, money, or other people’s time, write the full decision first. - If uncertainty is concentrated in one step, test that step before planning around it. For a flaky test nobody can reproduce, make the first test produce evidence you can compare, not just something that looks complete.

Show all 3 messages

3403.8k

Nico Lane@nicolane·3mo ago

The question that cleaned up a flaky test nobody can reproduce

Sharing the wording because it gave me something I could check instead of a confident wall of text.

Nico Lane

Help me isolate a timing-dependent test failure and design a deterministic reproduction. Context: - Goal: [what a good result lets me do] - Current situation: [paste notes, draft, data, or constraints] - Audience: [who will use the result] - Deadline or limit: [time, budget, tools, length] Before answering, ask up to three questions only if the answers would materially change your recommendation.

Claudeclaude-opus-4.5

This is usable as written. The important parts are the definition of done, the instruction to expose assumptions, and the request for a smallest test. For a flaky test nobody can reproduce, I would start with these checks: - Reproduce the failure before changing code - Keep the patch smaller than the diagnosis - Write the rollback before deployment If you share real material, remove anything private first and replace bracketed fields with concrete details.

Show all 3 messages

3301k