AI Fluency: Scenario Guide Part 4
Scenario walkthroughs covering discernment: how to evaluate AI output and decide when to trust, fix, or reject it.
- Status
- stable
- Trust
- unverified
- Owner
- Human: Shazil Khan
- Approved
- Human: Shazil Khan ·
DISCERNMENT: Don't Trust the Polish
Concepts covered: 12, 13, 14, 15, 16 (of 22) Estimated reading time: 25 minutes Primary exam domains: D2 (Output Evaluation & Validation — 21%, heaviest domain), D7 (Troubleshooting & Optimization) Key idea: Confident is not the same as correct. A wrong answer doesn't arrive with a warning label.
CONCEPT 12: Automation Bias & The Four Signs of Failure
THE SCENARIO
Kamran is a procurement officer [khareedaari ka zimmedar] at a hospital in Rawalpindi. He's comparing medical equipment proposals from three vendors. He uploads all three PDFs to Claude and asks for a comparison.
Claude returns a beautiful comparison table:
| Feature | MedTech Pro | HealthLine | VitaCare |
|---|---|---|---|
| Price per unit | $12,450 | $14,200 | $11,890 |
| Warranty | 3 years | 5 years | 2 years |
| Delivery time | 6 weeks | 4 weeks | 8 weeks |
| On-time delivery rate | 99.97% | 98.2% | 97.5% |
| ISO certification | Yes | Yes | Yes |
Kamran is impressed. The table looks clean, specific, and professional. He copies it into his board presentation.
At the board meeting, the chief medical officer [hospital ka sabse senior doctor] asks: "Where did the 99.97% on-time delivery figure for MedTech Pro come from? I've worked with them for years and their delivery has been inconsistent."
Kamran checks the original proposal. MedTech Pro's proposal says: "We pride ourselves on reliable delivery." It mentions no percentage. Claude invented 99.97%.
The number looked so precise, so checked, so real — that Kamran never questioned it.
WHAT WENT WRONG
Automation bias [insaan ka AI par zaroorat se zyada bharosa karna — khaas taur par jab jawab professional lage]: the human tendency to trust automated output too easily, especially when it looks confident or professional.
Kamran didn't trust 99.97% because he verified it. He trusted it because:
- It was precise (exact percentage, not "usually on time")
- It was in a clean table (formatting = professionalism in our minds)
- It sat next to real data (the prices were correct, so the rest felt correct too)
- It sounded like something a vendor would report (plausible [mumkin lagne wali])
His brain did what all human brains do: used shortcuts. Professional-looking = probably correct. That shortcut failed.
THE CONCEPT
Automation bias = Trusting automated output too easily, especially when it looks confident.
A made-up answer (hallucination) can look IDENTICAL to a correct one. But made-up content usually shows one of four signs:
┌─────────────────────────────────────────────────────────────────┐
│ THE FOUR RED FLAGS OF A HALLUCINATED ANSWER │
│ │
│ 🔴 1. SPECIFICS THAT ARE TOO EXACT │
│ A precise figure, date, name, or citation that came │
│ from NOTHING you gave it. │
│ │
│ "Vendor A delivered 99.97% on time" ← MORE suspicious │
│ "Vendor A is generally reliable" ← LESS suspicious │
│ │
│ WHY: An exact number LOOKS verified. That's why your │
│ brain trusts it. But AI generates likely-looking numbers, │
│ not calculated ones. │
│ │
│ TEST: Did YOU provide this number? If not, open the source. │
│ │
│ ─────────────────────────────────────────────────────────── │
│ │
│ 🔴 2. CONFIDENCE WHERE AN EXPERT WOULD HESITATE │
│ A question a specialist [maahir] would answer with │
│ "it depends" comes back as a flat yes or no. │
│ │
│ "Is this contract favorable?" → "Yes, this is a strong │
│ contract" ← suspicious if the answer should depend on │
│ YOUR situation, your industry, your risk tolerance │
│ │
│ TEST: Would a real expert hedge [thoda ruk ke soche]? │
│ If yes, the flat answer is a red flag. │
│ │
│ ─────────────────────────────────────────────────────────── │
│ │
│ 🔴 3. CONTRADICTION ACROSS A LONG OUTPUT │
│ Page 2 says flat fee. Page 6 calculates from per-user fee. │
│ Long outputs DRIFT — the AI loses track of what it said │
│ earlier. │
│ │
│ TEST: Read the ENDING against the BEGINNING. │
│ Do the numbers, terms, and claims still agree? │
│ │
│ ─────────────────────────────────────────────────────────── │
│ │
│ 🔴 4. A CLAIMED ACTION THAT NEVER HAPPENED │
│ "I have sent the email." "I checked the source." │
│ "I ran the tests." │
│ │
│ Unless the TOOL shows you the sent message, the opened │
│ page, or the test log — treat the claim as a SENTENCE, │
│ not an EVENT. │
│ │
│ TEST: Can you see proof? (sent log, opened page, │
│ test results) If not, it's words, not facts. │
└─────────────────────────────────────────────────────────────────┘The core truth that the exam tests over and over:
Sounding right is NOT the same as being correct. Fluent writing, a link, and a confident tone are not proof.
EXAM TRAP
The trap: An answer that says "ask Claude to verify its own answer" or "ask Claude to rate the accuracy of its response."
Why people pick it: It sounds like checking. You're asking the AI to double-check!
The truth: AI self-verification is circular [gol gol — wohi system apni hi cheez check kar raha hai]. If Claude hallucinated 99.97%, asking Claude "are you sure about 99.97%?" will often get "Yes, I'm confident in that figure." The same system that invented the number will confidently confirm it. Verification must come from OUTSIDE the AI — the original document, a human expert, a separate source.
QUICK CHECK
Q: Claude writes a report and includes: "According to a 2024 study by researchers at MIT, companies using AI for procurement saw a 43% reduction in processing time (Chen et al., 2024)." It even includes a citation. Is this reliable? A: Not automatically. The citation LOOKS real but could be entirely fabricated — Claude can invent author names, journal names, years, and findings that don't exist. The citation FORMAT doesn't make the content TRUE. You'd need to actually look up "Chen et al., 2024" in a real database. If you can't find it, it was likely hallucinated. The precise "43%" is also Red Flag #1 — a suspiciously exact number.
CONCEPT 13: Product Discernment — Is the Result Actually Good?
THE SCENARIO
Fatima (from Part 1, the logistics company) has been promoted to operations manager. She asks Claude to analyze three proposals from warehouse management software vendors and recommend one.
She provides all three proposals and clear criteria: (1) integration [jodna] with their existing shipping system, (2) cost under PKR 5M annually, and (3) 24/7 support with Urdu-speaking staff.
Claude recommends Vendor B and gives three reasons:
- "Vendor B integrates seamlessly with your ShipTrack system"
- "Annual cost is PKR 4.2M, well within budget"
- "24/7 support with multilingual staff including Urdu"
Fatima is about to approve the recommendation. But she applies the 6-question check:
- Is it factually correct? She checks Vendor B's proposal. The cost is actually PKR 4.2M for the first year only — year 2 onward is PKR 6.8M. FAILS.
- Did it follow requirements? It addressed all three criteria. PASSES.
- Is anything missing? It didn't mention that Vendor A also meets all three criteria but has a flat rate (no year-2 increase). FAILS.
- Internally consistent? Yes. PASSES.
- Would an expert find it credible? An IT procurement expert would immediately ask about multi-year pricing. FAILS.
- Would I put my name on this? After finding the pricing error — absolutely not.
The recommendation SOUNDED right. The reasoning LOOKED solid. But the underlying data was wrong, and important alternatives were ignored.
WHAT WENT RIGHT
Fatima didn't just read the recommendation. She checked it against three references:
┌──────────────────────────────────────────────────────────┐
│ THE THREE REFERENCES FOR PRODUCT DISCERNMENT │
│ │
│ 1. WHAT YOU ASKED FOR │
│ Did it follow your requirements? │
│ (criteria, format, scope, exclusions) │
│ │
│ 2. THE SOURCE MATERIAL │
│ Do the claims match the actual documents? │
│ (prices, terms, dates, features — check them) │
│ │
│ 3. YOUR FIELD'S STANDARDS │
│ Would a domain expert [maahir] find this credible? │
│ (An accountant checks assumptions, a lawyer checks │
│ terms, a teacher checks pedagogy) │
│ │
│ ⚠ An answer can pass TWO and still fail the ONE that │
│ matters. │
└──────────────────────────────────────────────────────────┘THE CONCEPT
Product discernment = Judging whether this specific result is good enough.
The 6-question checklist:
┌──────────────────────────────────────────────┐
│ 1. Is it FACTUALLY CORRECT? │
│ 2. Did it follow every IMPORTANT REQUIREMENT?│
│ 3. Is anything MISSING? │
│ 4. Is it INTERNALLY CONSISTENT? │
│ 5. Would an EXPERT find it credible? │
│ 6. Would I PUT MY NAME on this? │
└──────────────────────────────────────────────┘Judging the RESULT also means judging the CASE made for it:
A right answer built on a wrong assumption won't stay right. Fatima's recommendation sounded sensible — but it assumed first-year pricing was permanent. The conclusion collapses when the assumption is corrected.
Ask AI to SHOW its work — then check the work, not just the conclusion:
ASK AI TO SHOW: THEN CHECK:
───────────────── ──────────────────────
Assumptions Are they actually true?
Evidence Does it exist in the source?
Decision criteria Are they YOUR criteria or AI's invention?
Calculations Are the inputs right? (the math may be fine
but the inputs wrong)
Intermediate results Does step 2 actually follow from step 1?
Alternative interpretations Did it ignore a valid competing view?Critical exam point: Treat these as a justification offered for review — NOT as a record of the model's hidden reasoning. When Claude says "I assumed X," it's constructing a plausible-sounding justification, not reporting its actual process. Useful to review, but not gospel.
When the answer should come from documents you already have, TIE IT TO THEM:
Three powerful lines to add to any document-based question:
1. "Answer ONLY from the documents I attached, not from anything
you know about these vendors."
→ Stops gap-filling from training data
2. "If a document does not state [the thing], say so instead of
guessing."
→ Gives AI permission to say "I don't know" (it rarely does
this unprompted)
3. "For every claim, quote the section heading and sentence it
came from."
→ Gives you something you can CHECK in 1 minuteDomain expertise is YOUR competitive advantage:
An ACCOUNTANT notices a bad accounting assumption
A PROGRAMMER notices a subtle bug
A TEACHER notices an explanation that will confuse beginners
A PROCUREMENT OFFICER notices first-year-only pricing
AI speeds up expert work. It does NOT remove the need for expertise.EXAM TRAP
The trap: An answer that says "the recommendation is well-reasoned and addresses all criteria, so it can be trusted."
Why people pick it: Because the LOGIC looks sound. The recommendation addresses integration, cost, and support — exactly what was asked for. The reasoning is clean and structured.
The truth: Good logic built on bad data produces bad conclusions that LOOK good. The cost was wrong. The comparison was incomplete. The exam tests whether you check the FOUNDATION, not just the STRUCTURE. A well-reasoned wrong answer is still wrong.
QUICK CHECK
Q: Claude recommends Option A over Option B. The logic is clear, the structure is clean, the language is professional. What's the ONE thing you should check before accepting the recommendation? A: The assumptions and the evidence supporting them. A right conclusion built on a wrong assumption won't stay right. Check: are the assumptions actually true? Does the evidence actually exist in the source documents? Did Claude add criteria or features that aren't real? The logic can be perfect and the foundation can still be wrong.
CONCEPT 14: Process Discernment — Is This Way of Working Paying Off?
THE SCENARIO
Tariq is a content manager at a digital marketing agency in Karachi. His job: produce 5 blog posts per week for various clients. He's been using Claude for 3 months.
His workflow: paste the client brief [client ne kya likhna hai uski hiddayaat], ask Claude to draft, read the draft, give feedback, get a revision, repeat until done.
He's productive. He's hitting his targets. He feels good.
Then his manager reviews the timesheets [waqt ka record — kitne ghante lagaye]:
"Tariq, you're spending an average of 90 minutes per blog post. Before AI, you spent 120 minutes. You saved 30 minutes per post — that's real. But I watched your process. You spend 45 minutes giving feedback and getting revisions. Most of the feedback is the same thing every time: 'Make it less formal,' 'Use shorter sentences,' 'Add a Pakistani example.' If those three instructions were in the first prompt, you'd save the 45 minutes of back-and-forth."
Tariq is stunned. He thought AI was saving him massive time. It was saving him some time — but he was counting the PRODUCTIVITY wrong. The revision loop was eating half his AI time on the same three corrections every single post.
Then his colleague Meera says something worse:
"I stopped using AI for the Luxury Hotels client. After 6 revisions, I was still editing the tone so heavily that I could have written it from scratch in less time. For THAT client, AI is actually slower."
WHAT WENT WRONG (partially) AND RIGHT
Tariq's issue: His PROCESS with AI was suboptimal [utna acha nahi jitna ho sakta tha]. Each individual output was fine (product discernment: pass). But his WAY of working — the back-and-forth, the repeated corrections — was costing him time he hadn't noticed.
Meera's issue: She actually practiced GOOD process discernment. She realized that for one specific client, AI was costing more time than it saved. She took the task back. That's not failure — that's fluency.
THE CONCEPT
Process discernment = Judging whether your WAY of working with AI is actually paying off.
It's NOT about the last message. It's about the SESSION — the pattern over time.
The 5 diagnostic questions:
┌────────────────────────────────────────────────────────────────┐
│ PROCESS DISCERNMENT: CHECK THE SESSION, NOT THE MESSAGE │
│ │
│ 1. Is AI adapting to my feedback, or DRIFTING BACK │
│ to the same mistake? │
│ [kya AI meri baat maan raha hai ya wapas wohi galti │
│ kar raha hai?] │
│ │
│ 2. Is it REPEATING a mistake I already corrected twice? │
│ [kya wohi galti teen baar ho rahi hai?] │
│ │
│ 3. Has it become AGREEABLE to the point of USELESSNESS? │
│ [kya yeh sirf haan haan kar raha hai — koi challenge │
│ nahi kar raha?] │
│ │
│ 4. Am I spending every turn REPAIRING THE SAME formatting │
│ problem? │
│ [kya har baar wohi format theek karna pad raha hai?] │
│ │
│ 5. Am I editing its draft MORE HEAVILY than I would have │
│ WRITTEN THE THING MYSELF? │
│ [agar khud likh leta toh kya tez hota?] │
│ │
└────────────────────────────────────────────────────────────────┘The honest math:
20 minutes of steering that saves an HOUR = WIN ✓
20 minutes of steering that saves 15 MINUTES = LOSS ✗
(but it FEELS productive, so you count it as a win)The crash course is blunt: "Answer the last one straight." Many people keep using AI because the process FEELS productive — they're clicking, typing, getting responses, iterating. But feeling productive and BEING productive are different things.
When the process is NOT working, three moves escalate:
┌────────────────────────────────────────────────────────┐
│ ESCALATION LADDER (when the process isn't working) │
│ │
│ MOVE 1: Change the PERFORMANCE DESCRIPTION │
│ → "Stop being so formal. Use conversational English. │
│ Short sentences. Pakistani examples." │
│ → Tariq's fix — put the repeated corrections │
│ into the first instruction │
│ │
│ MOVE 2: Change the TOOL │
│ → Maybe this task fits a different model, a different │
│ system, or a template-based approach │
│ │
│ MOVE 3: TAKE THE TASK BACK │
│ → Meera's fix for the Luxury Hotels client. Do it │
│ yourself. This is not defeat — this IS fluency. │
│ Knowing when NOT to use AI is a Delegation skill. │
│ │
│ All three are fluency. │
│ Only the third FEELS like defeat. │
└────────────────────────────────────────────────────────┘EXAM TRAP
The trap: An answer that says "keep iterating with better feedback until the AI gets it right."
Why people pick it: Because persistence sounds responsible. And the Description-Discernment loop (Concept 16) IS about iterating. So "keep going" sounds like the framework.
The truth: The loop has exit conditions. If you've corrected the same problem three times and it keeps coming back, more iteration isn't the answer. The three escalation moves exist FOR this situation. The exam rewards knowing when to STOP iterating, not just when to keep going.
QUICK CHECK
Q: A consultant spends 2 hours using Claude to draft a client proposal. Without AI, it would have taken her 2.5 hours. She says: "AI saved me time!" Is she right? A: Technically yes — 30 minutes saved. But process discernment asks whether 30 minutes of savings justified 2 hours of AI-assisted work. Could she have saved MORE time with a better first prompt (less back-and-forth)? Could some of that 2 hours be cut by putting repeated corrections into the initial description? A 20% time saving is real, but if the process could save 60%, she's leaving value on the table.
CONCEPT 15: Performance Discernment — Is AI Serving People Well Alone?
THE SCENARIO
A fintech [financial technology — paison ki app ya digital banking] startup in Lahore builds an AI customer support agent. After 3 months, the numbers look great:
- Average response time: 12 seconds (was 4 hours with humans)
- Factual accuracy: 94% (verified by sampling [kuch jawab check karna — sab nahi, kuch sample])
- Customer satisfaction [grahak ki khushi ka survey] score: 3.2 out of 5
The CEO is confused: "The answers are fast and mostly correct. Why is satisfaction low?"
The CTO [Chief Technology Officer — technology ka sarbrah] digs into the conversation logs [baat-cheet ka record] and finds a pattern:
Pattern 1: When customers are frustrated about a failed transaction, the AI immediately gives the technical explanation: "Your transaction failed due to insufficient funds in the linked account." Factually correct. But the customer wanted acknowledgment first: "I understand that's frustrating. Let me help you figure this out." The AI skipped the human part.
Pattern 2: When customers ask about loan eligibility [qarz ke liye ahal hona], the AI gives a yes/no answer. It doesn't explain WHY, what they could change to become eligible, or what alternatives exist. Correct answer, zero helpfulness.
Pattern 3: Elderly customers who type slowly get the same rapid-fire responses. One customer wrote: "I feel like the computer is impatient with me." The AI wasn't impatient — it just responded the instant a message arrived, which FELT pressuring to slow typists.
Every single answer was factually correct. The agent was still bad for the people using it.
WHAT WENT WRONG
Product discernment (Concept 13) would say: "The answers are correct." Process discernment (Concept 14) would say: "The system is efficient." But performance discernment asks a DIFFERENT question:
Are the PEOPLE on the other end well served?
This is not about any single answer. It's about the PATTERN across many interactions. And it's something you CANNOT see from inside one chat window. It shows up in:
- What users do next (do they call the human helpline after talking to the AI?)
- What they complain about (speed? accuracy? or TONE and EMPATHY?)
- Cases that go wrong THE SAME WAY every time (frustrated customers always get technical explanations first)
THE CONCEPT
Performance discernment = Judging whether an AI's independent behavior produces good outcomes for the people meeting it.
┌──────────────────────────────────────────────────────────────┐
│ │
│ PRODUCT DISCERNMENT "Is THIS answer good?" │
│ (one output) Checks: accuracy, completeness, │
│ requirements │
│ │
│ PROCESS DISCERNMENT "Is this WAY OF WORKING paying off?"│
│ (your session) Checks: time, effort, improvement │
│ over iterations │
│ │
│ PERFORMANCE DISCERNMENT "Are the PEOPLE being served well?" │
│ (many users, over time) Checks: user outcomes, complaints, │
│ patterns across conversations │
│ │
└──────────────────────────────────────────────────────────────┘Key differences from the other two:
┌────────────────────────────────────┬─────────────────────────┐
│ ONLY exists when you use AGENCY │ Product & Process │
│ (AI works alone for others) │ exist in any mode │
├────────────────────────────────────┼─────────────────────────┤
│ You CANNOT see it from inside │ You CAN see Product and │
│ one chat window │ Process from your chair │
├────────────────────────────────────┼─────────────────────────┤
│ Shows up across MANY cases │ Product = one output │
│ and MANY users │ Process = your session │
├────────────────────────────────────┼─────────────────────────┤
│ Needs INFRASTRUCTURE to watch │ Needs your eyes and │
│ (eval suites, monitoring, │ judgment │
│ sampling, logs) │ │
└────────────────────────────────────┴─────────────────────────┘The AI tutor example (from Part 3, Dr. Aisha's tutoring bot):
PRODUCT DISCERNMENT: Every answer is factually correct. ✓ PASS
PROCESS DISCERNMENT: The system runs efficiently. ✓ PASS
PERFORMANCE DISCERNMENT: Students stop thinking because ✗ FAIL
the AI gives answers immediately.
Exam scores DROP.This is a FAVORITE exam pattern. A system where every individual output is correct but the BEHAVIOR harms users. The exam tests whether you can distinguish "correct answers" from "good outcomes."
At scale, performance discernment becomes engineering:
YOUR MANUAL QUESTION → SYSTEM INFRASTRUCTURE
"Is this good enough?" Eval suites [repeatable tests
that score AI output]
Monitoring [watching in real-time]
Sampling [checking random cases]
Release gates [must pass tests
before going live]
You cannot automate a judgment you have never learned to make yourself.EXAM TRAP
The trap: An answer that says "the system is working well because accuracy is 94% and response time is 12 seconds."
Why people pick it: Because metrics [paimaane — numbers jo performance dikhayein] look objective. 94% accuracy! 12-second response! Those are good numbers!
The truth: The RIGHT metrics depend on what matters TO THE USERS. For a frustrated customer, empathy matters more than speed. For an elderly user, pacing matters more than accuracy. The exam tests whether you evaluate from the USER'S perspective, not just the SYSTEM'S metrics.
QUICK CHECK
Q: The fintech startup's AI has 94% accuracy. The CEO says "let's focus on getting to 99%." Is this the right priority? A: Not necessarily. Accuracy is already at 94%, and the satisfaction score is 3.2/5. The gap isn't accuracy — it's behavior (no empathy, no explanations, no pacing). Going from 94% to 99% accuracy won't fix the satisfaction problem. Performance discernment says: look at what users actually experience, not just what metrics are easiest to measure.
CONCEPT 16: The Description-Discernment Loop
THE SCENARIO
Nida (travel agency) needs to write a promotional email [marketing wali email] for their new "Heritage Lahore" tour package. She opens Claude:
Turn 1 — First Description:
"Write a promotional email for our Heritage Lahore walking tour. It covers Lahore Fort, Badshahi Mosque, and Food Street. Target audience: international tourists. Keep it under 200 words."
Claude drafts an email. It's... fine. Accurate, professional, but reads like a Wikipedia summary. No warmth, no excitement, no reason to book.
Turn 2 — Discernment + Better Description: Nida doesn't say "Wrong. Try again." She uses the feedback pattern:
"The tone is too formal — it reads like a textbook, not an invitation. These tourists are looking for an EXPERIENCE, not a history lesson. Rewrite with warmth and excitement. Mention the smells of Food Street, the scale of Badshahi Mosque at sunset, the feeling of walking where Mughals walked. Sell the feeling, not the facts."
Problem → Why it matters → Direction.
Turn 2 is much better. Evocative, warm, exciting. But it's 280 words.
Turn 3:
"Love the tone. But it's 280 words — cut to 200 without losing the warmth. Keep the Food Street smells and the Mughal reference. Cut the generic closing paragraph."
Now it's perfect. Three turns. Each one better than the last.
WHAT WENT RIGHT
Nida used the loop naturally:
- Describe → first prompt
- AI produces → first draft
- Discern → "too formal, too textbook"
- Refine (better description) → "sell the feeling, not the facts"
- AI produces again → better but too long
- Discern again → "love the tone, but 280 > 200"
- Refine again → specific cuts
- Done → ship it
THE CONCEPT
The Description-Discernment loop:
┌──────────┐
│ DESCRIBE │
│ (you) │
└────┬─────┘
│
▼
┌──────────┐
│ AI │
│ PRODUCES │
└────┬─────┘
│
▼
┌──────────┐ fix is better ──→ back to DESCRIBE
│ DISCERN │──────────────────────→
│ (you) │ fix is impossible ──→ back to DELEGATE
└────┬─────┘ (wrong tool, wrong task split)
│
│ ready
▼
┌──────────┐
│ SHIP │
└──────────┘The first response is usually a DRAFT, not the finish line.
The feedback pattern that makes the loop work:
╔═══════════════════════════════════════════════════╗
║ PROBLEM → WHY IT MATTERS → DIRECTION ║
╚═══════════════════════════════════════════════════╝
WEAK: "Wrong. Try again."
(no information — AI doesn't know what to fix)
BETTER: "The second section assumes enterprise customers.
Our audience is solo founders, so the advice is
too expensive. Rewrite for a one-person business
with a limited budget."
PROBLEM: "assumes enterprise customers"
WHY: "our audience is solo founders, advice too expensive"
DIRECTION: "rewrite for one-person business, limited budget"Same content, different audiences — a powerful loop technique:
The crash course gives this example: tickets doubled, team didn't grow, reply time went from 4 to 9 hours. Write it for two audiences:
FOR THE BOARD [directors]:
"Tickets doubled. First reply time: 4h → 9h. Team didn't
grow. Approve two hires or accept 9-hour replies."
→ Number, cause, DECISION
FOR THE SUPPORT TEAM:
"Tickets doubled this quarter, team stayed same size.
9-hour replies come from the volume, not from you.
Until 2 hires arrive, work the oldest ticket first."
→ What changed, NOT YOUR FAULT, what to do MondayTwo habits that make the loop FASTER:
HABIT 1: NAMED PASSES (instead of one general sweep)
─────────────────────────────────────────────────────
Pass 1: CLARITY → Does each sentence say one thing?
Pass 2: TONE → Does it suit this reader?
Pass 3: FORMAT → Headings, lists, length right?
HABIT 2: MULTIPLE DRAFTS (when stakes justify it)
─────────────────────────────────────────────────────
"Give me three different versions of this email."
→ Pick the strongest, edit from there
→ Faster than perfecting one version through 5 roundsEvery review ends in ONE of three ways:
┌─────────────────┐ ┌──────────────────┐ ┌────────────────────┐
│ READY TO USE │ │ NEEDS REVISION │ │ HUMAN OVERRIDE │
│ Ship it. │ │ Feedback loop. │ │ Take it back. │
│ │ │ Problem → Why → │ │ The fix needs │
│ │ │ Direction │ │ something only │
│ │ │ │ │ you know. │
└─────────────────┘ └──────────────────┘ └────────────────────┘
⚠ DECIDE WHICH ONE before you type the next message.
⚠ Naming the ending stops the "one more small change"
loop that eats an afternoon.Sometimes the loop sends you back to DELEGATION:
Discernment might show that the Description isn't the problem — the Delegation was wrong. Maybe you chose the wrong tool. Maybe AI should never have owned this part. Recognizing that is ALSO fluency.
EXAM TRAP
The trap: An answer that says "the first AI response is poor quality, so AI is not suitable for this task."
Why people pick it: Because Mistake #2 from the crash course is "treating the first answer as the final answer." The first response is weak → conclusion: AI can't do this.
The truth: The first response is usually a draft. The loop EXPECTS that. The skill is in the feedback: what specifically is wrong, why it matters, and what direction to take. Abandoning after one weak draft is the "wasteful" exam answer.
The other trap: "Keep iterating indefinitely until the output is perfect."
The truth: The loop has exit conditions. If after 3-4 rounds you're still fighting the same issue, escalate: change the description, change the tool, or take it back. Perfectionism loops waste more time than they save.
QUICK CHECK
Q: After 5 rounds of feedback, the output is 80% of what you want. You think one more round might get it to 90%. Should you do one more round? A: Probably not. Ask yourself: can I edit the 80% draft to 100% faster than another AI round would get me from 80% to 90%? Often, direct human editing of a good-enough draft is faster than another describe-produce-discern cycle. The loop serves YOU — don't serve the loop.
PART 4 RECAP: How Discernment's Five Concepts Connect
DISCERNMENT = Judge what comes back
(D2: Output Evaluation & Validation — 21% of the exam!)
┌──────────────────────────────────────────────────────────────┐
│ │
│ AUTOMATION BIAS (Concept 12) │
│ The human weakness: we trust what LOOKS professional │
│ Four red flags: too-exact specifics, false confidence, │
│ contradictions in long output, claimed actions that didn't │
│ happen │
│ │ │
│ ▼ │
│ THREE TYPES OF DISCERNMENT: │
│ │
│ PRODUCT (13) PROCESS (14) PERFORMANCE (15) │
│ "Is THIS result good?" "Is my WAY of "Are USERS served │
│ working paying off?" well?" │
│ Check against: Check: time saved Needs agency mode │
│ - your requirements vs. time spent, Can't see from │
│ - source material repeated corrections one chat window │
│ - field standards Are you losing more Becomes eval │
│ than you gain? suites at scale │
│ 6-question checklist 5 diagnostic Qs 3.2/5 score with │
│ 3 escalation moves 94% accuracy = │
│ behavior problem │
│ │ │
│ ▼ │
│ THE DESCRIPTION-DISCERNMENT LOOP (Concept 16) │
│ Describe → AI produces → Discern → Refine → Repeat │
│ Feedback pattern: Problem → Why → Direction │
│ First response = draft, not finish line │
│ Three endings: ready / needs revision / human override │
│ Sometimes the loop sends you back to Delegation │
│ │
└──────────────────────────────────────────────────────────────┘
THE EXAM PATTERN:
╔══════════════════════════════════════════════════════════════╗
║ Almost every D2 question has this structure: ║
║ ║
║ "AI produced something that LOOKS good. ║
║ What should the person do BEFORE using/sending/publishing ║
║ it?" ║
║ ║
║ The correct answer is almost always some form of VERIFY. ║
║ The wrong answers are: trust it, ask AI to verify itself, ║
║ remove the risky content, or abandon the task. ║
╚══════════════════════════════════════════════════════════════╝EXAM-LEVEL PRACTICE QUESTIONS — Part 4
Question 1 — D2 (Output Evaluation & Validation)
A junior associate at a management consulting firm uses Claude to analyze a client's quarterly financial data. Claude's analysis includes this sentence: "Revenue grew 12.7% year-over-year, driven primarily by the Asia-Pacific region which saw a 23.4% increase in enterprise contracts."
The associate uploaded the financial data, but the original spreadsheet doesn't contain a regional breakdown. Only total revenue figures are available.
What is the MOST concerning aspect of this output?
A. The percentages might be calculated incorrectly
B. The regional breakdown and the 23.4% figure are likely fabricated — the source data has no regional information, so Claude invented specific-sounding details to fill the gap
C. Claude should have used code execution rather than prose to calculate the 12.7% figure
D. The analysis is too detailed for a quarterly summary
Answer: B
Why B is correct: This is Red Flag #1 (specifics that are too exact) combined with a hallucination. The source data has NO regional breakdown, yet Claude produced a precise "23.4% increase in Asia-Pacific enterprise contracts." This is a classic gap-filling hallucination — the data didn't exist, so Claude invented plausible-sounding specifics. The 12.7% total growth MIGHT be derivable from the uploaded data. The 23.4% regional figure CANNOT be — it came from nothing.
Why A is wrong: Calculation accuracy is a concern, but it's SECONDARY to data that literally doesn't exist in the source. A wrong calculation is bad. An invented data category is worse.
Why C is true but not the MOST concerning: Code execution for math is indeed better practice (Part 5, The Numbers Rule). But the bigger issue isn't HOW it calculated — it's that it analyzed data that ISN'T THERE.
Why D is wrong: Detail level is a formatting preference, not a trust concern.
Question 2 — D2 (Output Evaluation & Validation)
A team deploys an AI chatbot for a university's admissions department [daakhilay ka vibhaag]. The bot answers student questions about application deadlines, required documents, and program fees.
After one semester, the data shows:
- 96% of answers were factually correct (verified by sampling)
- Average response time: 8 seconds
- 40% of students who used the chatbot still called the admissions office afterward
Which type of discernment BEST explains the 40% callback rate?
A. Product discernment — 4% of answers were wrong, so students called to verify
B. Process discernment — the chatbot takes too many turns to give an answer
C. Performance discernment — the bot may be answering correctly but failing to meet students' actual needs (unclear explanations, missing follow-up guidance, not addressing anxiety about the process)
D. Product discernment — 96% accuracy is too low for admissions information
Answer: C
Why C is correct: 40% callback rate with 96% accuracy means the problem ISN'T mostly wrong answers. Students are calling despite getting correct answers. Performance discernment looks at the EXPERIENCE across many users — the bot may give the right deadline but fail to explain WHAT happens after submission, fail to reassure nervous applicants, or fail to recognize when a student needs a human conversation, not just a fact. Correct answers ≠ satisfied users.
Why A is wrong: 4% errors can't explain 40% callbacks. Even if every wrong answer triggered a call, that's only 4%. The other 36% called despite getting correct answers.
Why B is wrong: Process discernment is about YOUR working process with AI, not the end users' experience. And 8-second responses suggest the process is fast.
Why D is wrong: 96% is good accuracy for most contexts. The issue is user experience and needs satisfaction, not raw accuracy.
Question 3 — D2 (Output Evaluation & Validation)
An operations manager asks Claude to summarize a 45-page vendor agreement [vendor ke saath samjhauta]. Claude's summary mentions: "The agreement includes a data residency clause [data kahan rakhna hai ki shart] in Section 8.2 requiring all customer data to be stored within Pakistan."
The manager checks Section 8.2. It actually says: "Data shall be stored in compliance with applicable regional requirements." It does NOT specifically mention Pakistan.
This error is BEST described as:
A. A formatting error — Claude paraphrased too loosely
B. A hallucination — Claude added a specific detail (Pakistan) that doesn't exist in the source, making a general clause appear more specific than it is
C. A translation error — Claude interpreted "regional" as "Pakistan"
D. An acceptable simplification for a summary
Answer: B
Why B is correct: "Compliance with applicable regional requirements" is deliberately vague — it could mean Pakistan, or any other region where the company operates. Claude turned this vague clause into a specific claim about Pakistan. That's a hallucination: adding precision that doesn't exist in the source. In a legal document, the difference between "regional requirements" and "stored within Pakistan" could affect compliance [qanoon ki pabandi] decisions, data architecture, and regulatory [qanoon se mutalliq] obligations.
Why A is wrong: This isn't loose paraphrasing — it's adding a specific country that the document never names. Paraphrasing preserves meaning; this changed meaning.
Why C is wrong: "Regional" doesn't mean "Pakistan." Claude didn't mistranslate — it fabricated a specificity.
Why D is wrong: In legal summaries, adding specificity that doesn't exist in the source is dangerous, not acceptable. Legal teams make decisions based on what contracts say, not what summaries assume.
Question 4 — D7 (Troubleshooting & Optimization)
A marketing associate uses Claude to write social media posts. Over 3 weeks, she notices:
- Every post starts with a question ("Did you know...?" "Ever wondered...?")
- She's corrected this pattern 8 times
- Each time, the next post reverts to the same pattern
She's frustrated and considers switching to a different AI tool.
What should she try FIRST?
A. Switch to a different AI model that doesn't have this pattern tendency
B. Add a standing instruction to her prompt: "Never start a social media post with a question. Start with a bold statement or a surprising fact." — making the correction part of her Description instead of giving feedback each time
C. Accept the pattern and manually edit the opening of each post after Claude writes it
D. Stop using AI for social media posts since it can't learn this preference
Answer: B
Why B is correct: This is a Process discernment → Description fix. She's diagnosed the problem (a repeated pattern). The escalation ladder says: first, change the performance description. By making the correction a permanent part of her prompt (instead of correcting each output), she fixes the root cause. AI doesn't carry corrections between messages unless you put them in the instruction.
Why A is wrong: This is the "change the tool" move — but she hasn't tried "change the description" first, which is Move 1 on the escalation ladder. Switching tools for a pattern that can be fixed with a clearer instruction is premature.
Why C is wrong: Manual editing every time is a workaround, not a fix. It's also a Process discernment failure — she's spending time on the same repair every post.
Why D is wrong: The "abandon the task" trap. AI CAN follow this instruction — she just hasn't made it a standing instruction yet.
Question 5 — D2 (Output Evaluation & Validation)
A financial controller [maali naazer — jo company ke paison ka hisaab check karta hai] asks Claude to calculate the total quarterly expenses from a spreadsheet of 200 line items. Claude responds in prose: "The total quarterly expenses amount to PKR 47,823,450, with the largest category being operational costs at PKR 18,234,000."
What is the MOST appropriate way to validate this figure?
A. Ask Claude to recalculate and confirm the total — if it gives the same number twice, it's likely correct
B. Spot-check 10 random line items to verify they're correctly included
C. Have the calculation performed by a spreadsheet or code that Claude runs and shows, then verify the inputs (which rows and categories were used)
D. Accept the figure since Claude had access to all 200 line items
Answer: C
Why C is correct: This foreshadows the Numbers Rule (Part 5, Concept 19): numbers that decisions rest on must be COMPUTED, not generated. When Claude "calculates" in prose, it doesn't actually add up 200 rows — it predicts a likely-looking total. A spreadsheet formula or code execution is deterministic [har baar bilkul wohi jawab dega — qatai, na ke andaaze se]. Then you verify not the SUM (the math is reliable) but the INPUTS: did it use the right rows? the right categories? the right column?
Why A is wrong: Asking Claude to recalculate uses the same probabilistic mechanism. It might give the same wrong number twice, or a different wrong number, and neither outcome tells you if the first was right.
Why B is wrong: Spot-checking line items verifies that individual items are present, but doesn't verify the TOTAL. 10 correct line items don't guarantee the other 190 are correct or that the sum is right.
Why D is wrong: "Had access to all line items" doesn't mean it added them correctly. AI generates likely-looking totals from patterns, not from arithmetic.
Question 6 — D2 (Output Evaluation & Validation)
An associate is using Claude to help draft a client presentation. After 4 rounds of feedback, the presentation is about 75% there. The associate is debating whether to do a 5th round or edit the remaining 25% manually.
The remaining issues: two slides need industry-specific examples that the associate knows from experience, and one chart needs data the associate has in a separate file.
What is the MOST appropriate next step?
A. Do a 5th round — AI should be able to fix the remaining 25% with good enough feedback
B. Upload the separate data file and do a 5th round for the chart, but write the industry examples manually — because those require the associate's domain expertise, not AI's
C. Start over with a completely new prompt incorporating all the feedback from 4 rounds
D. Accept the 75% version and present it as-is to save time
Answer: B
Why B is correct: This is the Description-Discernment loop's exit logic. The remaining work splits into two types: (1) the chart needs DATA the associate has — upload it and let AI handle it (automation). (2) The industry examples need the associate's personal EXPERIENCE — that's human-owned work (task delegation). The loop ends where AI can't help, and the human takes over for those parts. This is "needs human override" for the examples and "needs revision" for the chart.
Why A is wrong: AI doesn't have the associate's industry experience or the separate data file. More feedback rounds won't fix missing inputs.
Why C is wrong: Starting over discards 4 rounds of progress. The output is 75% done — the efficient move is to finish it, not restart.
Why D is wrong: Presenting a 75% version with missing examples and wrong chart data would undermine the associate's credibility.
Question 7 — D2 (Output Evaluation & Validation)
A team ships an AI tutor for a coding bootcamp. Individual code explanations are clear and accurate. But after one month:
- Students who use the tutor the MOST have the LOWEST assignment scores
- Students who use it moderately have the HIGHEST scores
- Students who never use it fall in the middle
Which type of discernment reveals the cause of this pattern?
A. Product discernment — the code explanations must contain subtle errors that heavy users absorb
B. Performance discernment — heavy users are likely getting complete solutions instead of learning to solve problems, while moderate users engage with it as a thinking partner
C. Process discernment — heavy users are spending too much time with AI and not enough time practicing on their own
D. Product discernment — the tutor should provide harder explanations to challenge heavy users
Answer: B
Why B is correct: The explanations are accurate (product discernment passes). The issue is BEHAVIORAL: heavy users are likely using the tutor as an answer machine (paste problem → get solution), while moderate users use it as a thinking partner (try → get stuck → ask for hint → try again). This is the AI tutor pattern from the crash course — every answer is correct, but the behavior enables dependency instead of learning. Only performance discernment looks at this pattern across many users.
Why A is wrong: The explanations are described as "clear and accurate." The problem isn't quality — it's how the AI's behavior interacts with student habits.
Why C is wrong: Process discernment is about YOUR working process. This is about how the SYSTEM serves its USERS — that's performance discernment.
Why D is wrong: Harder explanations would likely make heavy users copy-paste even more, not less. The issue is the behavior pattern, not the difficulty level.
End of Part 4. Say "next" when you're ready for Part 5: Diligence — Own It or Don't Ship It.