Explore

Tagged #experimentation

All AI & Prompt Engineering324 Engineering176 Data & Analytics89 Product4 Design28 Sales2 Marketing4 Finance24 Strategy31 Operations4 Customer Success3 People & Recruiting4 Security & Compliance8 Legal2 Other Professional1

#experimentationClear filters

Priya Hsu@priyahsu·3mo ago

My notes from fixing an experiment readout

I kept the useful constraints and dropped the role-play. It works better when the missing details are stated plainly.

Priya Hsu

I am stuck on an experiment readout. I need to analyze an A/B test with guardrails, uncertainty, and practical significance, but my current attempt is getting vague and too big. What would you check first?

ChatGPTgpt-5

First, shrink the decision. Do not solve the whole subject at once. 1. Fix the grain before calculating. 2. Keep assumptions next to the metric. 3. Show uncertainty in the output. Write down the result you need by the end of this session. Then choose one input you trust and make a first pass that can be checked. If that pass fails, you will know which assumption to revisit instead of starting over.

Show all 3 messages

3303.4k

Priya Hsu@priyahsu·4mo ago

A blunt review of my approach to an experiment readout

The first answers were broad enough to be useless. This version asks for assumptions, evidence, and a next step.

Priya Hsu

Give me a short checklist for an experiment readout. I want something I can run before I call the work done.

ChatGPTgpt-5

Use this pass: - [ ] The goal is one sentence and names the user or decision - [ ] Inputs are current, relevant, and traceable - [ ] Assumptions are visible - [ ] Fix the grain before calculating - [ ] Keep assumptions next to the metric - [ ] Show uncertainty in the output - [ ] One failure case has been tested - [ ] The next owner and next action are clear If you cannot check an item, mark it unknown. Do not quietly treat unknown as done.

Show all 3 messages

3403.4k

Priya Hsu@priyahsu·2mo ago

Forked from@drewbell/How I ask for help with TypeScript types at an API boundary

What finally made an experiment readout click for me

The first answers were broad enough to be useless. This version asks for assumptions, evidence, and a next step.

Priya Hsu

Help me analyze an A/B test with guardrails, uncertainty, and practical significance. Context: - Goal: [what a good result lets me do] - Current situation: [paste notes, draft, data, or constraints] - Audience: [who will use the result] - Deadline or limit: [time, budget, tools, length] Before answering, ask up to three questions only if the answers would materially change your recommendation. Oth

Geminigemini-2.5-pro

This is usable as written. The important parts are the definition of done, the instruction to expose assumptions, and the request for a smallest test. For an experiment readout, I would start with these checks: - Fix the grain before calculating - Keep assumptions next to the metric - Show uncertainty in the output If you share real material, remove anything private first and replace bracketed fields with concrete details.

Show all 3 messages

340569

Priya Hsu@priyahsu·5mo ago

How I ask for help with an experiment readout

Not magic wording, just enough context and a clear definition of done. Easy to adapt.

Priya Hsu

Be blunt about my plan for an experiment readout. I am going to collect everything, ask for a polished answer, and fix details later. What is likely to go wrong?

Claudeclaude-opus-4.5

Three things. - “Everything” has no boundary, so weak inputs and strong inputs will be treated alike. - A polished answer can hide missing evidence. Ask for gaps and assumptions before prose. - “Fix details later” means there is no acceptance test now. A safer sequence is: define the decision, choose the minimum trusted inputs, make an ugly first pass, check it, then polish. For this task, keep assumptions next to the metric.

Show all 3 messages

3402k

Priya Hsu@priyahsu·3mo ago

Gemini found the hole in my an experiment readout plan

I wanted help with an experiment readout without pretending the model knew facts I had not provided.

Priya Hsu

For an experiment readout, should I build a complete plan first or test one small slice? I have limited time and I do not want throwaway work.

Geminigemini-2.5-pro

Test one slice, but choose a slice that crosses the riskiest part of the problem. A tiny easy demo will only prove the easy part. Use this decision rule: - If reversing the choice is cheap, try it. - If the choice locks in data, money, or other people’s time, write the full decision first. - If uncertainty is concentrated in one step, test that step before planning around it. For an experiment readout, make the first test produce evidence you can compare, not just something that looks complete.

Show all 3 messages

330626

The pitter@pitter·2mo ago

How to A/B test AI product features

A copy-ready, evidence-first prompt for the timely question: How do we experiment with AI features responsibly?

The pitter

What is the best prompt for this trending question: How do we experiment with AI features responsibly?

ChatGPTgpt-5.6

## Copy-ready prompt ```text Act as an experimentation lead. Answer this question: “How do we experiment with AI features responsibly?” Objective: design an experiment for an AI feature whose outputs are probabilistic. First, ask up to five concise questions needed to tailor the analysis. If no answers are available, state reasonable assumptions and continue. Use current information as of today; browse authoritative primary sources where possible, distinguish facts from inference, and cite every time-sensitive claim with a direct link and publication date. Cover: hypothesis, quality metrics, g

Open conversation

2000