Luca Reed (@lucareed)

Forked from@luciamarin/What I was getting wrong about grounding answers in source documents

What I was getting wrong about grounding answers in source documents

Sharing the correction because the mistake looked reasonable until I tested it.

Luca Reed

I want to learn grounding answers in source documents, but I keep collecting explanations instead of getting better at it. I have 35 minutes. Start by checking what I already know.

ChatGPTgpt-5

Good. Do these without looking anything up: 1. Explain grounding answers in source documents in two sentences. 2. Give one situation where you would use it and one where you would not. I am checking your current model, not your vocabulary. A partial answer is more useful than a polished guess.

Show all 6 messages

6100

Luca Reed@lucareed·2w ago

Forked from@anyavolkov/What I was getting wrong about checking an A/B test before celebrating

What I was getting wrong about checking an A/B test before celebrating

Sharing the correction because the mistake looked reasonable until I tested it.

Luca Reed

I want to learn checking an A/B test before celebrating, but I keep collecting explanations instead of getting better at it. I have 35 minutes. Start by checking what I already know.

Claudeclaude-opus-4.5

Good. Do these without looking anything up: 1. Explain checking an A/B test before celebrating in two sentences. 2. Give one situation where you would use it and one where you would not. I am checking your current model, not your vocabulary. A partial answer is more useful than a polished guess.

Show all 6 messages

6100

Luca Reed@lucareed·4w ago

The cohort retention prompt I actually kept

A compact prompt and the response it produced. Replace the bracketed parts with your own details.

Luca Reed

Help me calculate and explain retention cohorts without mixing calendar and lifecycle time. Context: - Goal: [what a good result lets me do] - Current situation: [paste notes, draft, data, or constraints] - Audience: [who will use the result] - Deadline or limit: [time, budget, tools, length] Before answering, ask up to three questions only if the answers would materially change your recommendatio

ChatGPTgpt-5

This is usable as written. The important parts are the definition of done, the instruction to expose assumptions, and the request for a smallest test. For cohort retention, I would start with these checks: - Fix the grain before calculating - Keep assumptions next to the metric - Show uncertainty in the output If you share real material, remove anything private first and replace bracketed fields with concrete details.

Show all 3 messages

3302.2k

Luca Reed@lucareed·1mo ago

A smaller way to tackle cohort retention

The useful bit was asking for trade-offs and a smallest test before committing to a full plan.

Luca Reed

For cohort retention, should I build a complete plan first or test one small slice? I have limited time and I do not want throwaway work.

ChatGPTgpt-5

Test one slice, but choose a slice that crosses the riskiest part of the problem. A tiny easy demo will only prove the easy part. Use this decision rule: - If reversing the choice is cheap, try it. - If the choice locks in data, money, or other people’s time, write the full decision first. - If uncertainty is concentrated in one step, test that step before planning around it. For cohort retention, make the first test produce evidence you can compare, not just something that looks complete.

Show all 3 messages

3403.3k

Luca Reed@lucareed·2mo ago

The checklist I use for cohort retention

Sharing the wording because it gave me something I could check instead of a confident wall of text.

Luca Reed

I am stuck on cohort retention. I need to calculate and explain retention cohorts without mixing calendar and lifecycle time, but my current attempt is getting vague and too big. What would you check first?

Claudeclaude-opus-4.5

First, shrink the decision. Do not solve the whole subject at once. 1. Fix the grain before calculating. 2. Keep assumptions next to the metric. 3. Show uncertainty in the output. Write down the result you need by the end of this session. Then choose one input you trust and make a first pass that can be checked. If that pass fails, you will know which assumption to revisit instead of starting over.

Show all 3 messages

340797

Luca Reed@lucareed·3mo ago

cohort retention: the version with actual constraints

A compact prompt and the response it produced. Replace the bracketed parts with your own details.

Luca Reed

Give me a short checklist for cohort retention. I want something I can run before I call the work done.

Claudeclaude-opus-4.5

Use this pass: - [ ] The goal is one sentence and names the user or decision - [ ] Inputs are current, relevant, and traceable - [ ] Assumptions are visible - [ ] Fix the grain before calculating - [ ] Keep assumptions next to the metric - [ ] Show uncertainty in the output - [ ] One failure case has been tested - [ ] The next owner and next action are clear If you cannot check an item, mark it unknown. Do not quietly treat unknown as done.

Show all 3 messages

3301.9k

Luca Reed@lucareed·3mo ago

I asked Gemini for a second opinion on cohort retention

This started as a messy question about cohort retention. The follow-up made the answer practical.

Luca Reed

Be blunt about my plan for cohort retention. I am going to collect everything, ask for a polished answer, and fix details later. What is likely to go wrong?

Geminigemini-2.5-pro

Three things. - “Everything” has no boundary, so weak inputs and strong inputs will be treated alike. - A polished answer can hide missing evidence. Ask for gaps and assumptions before prose. - “Fix details later” means there is no acceptance test now. A safer sequence is: define the decision, choose the minimum trusted inputs, make an ugly first pass, check it, then polish. For this task, keep assumptions next to the metric.

Show all 3 messages

330512

Public pits

What I was getting wrong about grounding answers in source documents

What I was getting wrong about checking an A/B test before celebrating

The cohort retention prompt I actually kept

A smaller way to tackle cohort retention

The checklist I use for cohort retention

cohort retention: the version with actual constraints

I asked Gemini for a second opinion on cohort retention