HOW WE KNOW WHAT ISN T SO: COGNITIVE SCIENCE AND MIND TRAPS Jean-Marc Fix, FSA, MAAA UW seminar 2012
COGNITIVE SCIENCE How do we make judgments? Are there systematic issues with our decision making process? a machine for jumping to conclusions (1) Sources: How we know what isn t so, the fallibility of human reason in everyday life; Thomas Gilovich Knowing you, why cognitive science matters for actuarial science; James Guszcza, Contingences May 2012 (1) Attributed to Kahneman Contingencies May 2012 p18
LOOKING AT DATA Something out of nothing Too much from too little Seeing what we expect to see The ultimate mind trap: correlation
SOMETHING OUT OF NOTHING Recognizing random The hot streak Seeing patterns Regression to the mean Reward or punishment?
RECOGNIZING RANDOM A coin is tossed 20 times. We get twice as many of one side than the other (i.e. 13/7, 14/6 ). I say the coin was fair. I am: 1. a liar (10% that what I say is true) 2. a big liar (5% that it is true) 3. I am the new Baron Munchausen (1% that it is true)
RECOGNIZING RANDOM A coin is tossed 20 times. We get twice as many of one side than the other (i.e. 13/7, 14/6 ). I say the coin was fair. I am: a con man. The answer is about 30%.
THE HOT STREAK I toss a coin 20 times. How rare is a streak of: At least 7 times? 12% At least 10 times? 1%
REGRESSION TO THE MEAN Truth Average Skill Error Luck
EXAMPLE: SKILL ONLY 1000 perfect students answer a 20 yes/no questions. Average score will be around 100% Take top 100 students (whose average will be around 100%) and give them another similar test. Their average score will be 100% again.
EXAMPLE: LUCK/RANDOM 1000 students answer a 20 yes/no questions at random. Average score will be around 50% Take top 100 students (whose average will be around 75%) and give them another similar test also answered at random. Their average score will be closer to 50%.
EXAMPLE: REAL LIFE 1000 random students answer a 20 yes/no questions at the best of their ability. Average score will be around X% Take top 100 students. Average score is likely much above X%. Some were good and some were bad but lucky. Give them another similar test. Their average score will be closer to X%.
APPLICATION: REWARD VS. PUNISHMENT A B-student gets an A Parent congratulate Next time kid not so lucky on test gets B. Parent notices reward did not work A B-student gets a C Parent punish Next time kid not so unlucky on test gets B. Parent notices punishment worked
POST TEST ESTIMATION
THE TRUTH Person is really Sick Healthy
THE TEST Person is really Sick Healthy Positive Test is Negative
THE GOOD Person is really Sick Healthy Test is Positive Negative True Positive
THE (STILL) GOOD Person is really Sick Healthy Test is Positive Negative True Positive True Negative
THE BAD Person is really Sick Healthy Test is Positive True Positive False Positive Negative True Negative
THE UGLY Person is really Sick Healthy Test is Positive True Positive False Positive Negative False Negative True Negative
SENSITIVITY Are all the sick ones gotten by the test? Sensitivity= True Positive True Positive + False Negative Sick ones that the test missed
SENSITIVITY Sensitivity= True Positive All sick people
SPECIFICITY Are all the ones with a negative test healthy? Specificity= True Negative True Negative + False Positive Healthy ones that the test thought were sick
SPECIFICITY Specificity= True Negative All healthy people
GOOD TEST Sensitivity Specificity
A GOOD TEST Let s assume a test for a really really bad disease has those characteristics Sensitivity: 99% Specificity: 95% You underwent the test and are positive. Should you jump in front of a train?
AN EXAMPLE High sensitivity: 99% High specificity: 95% 1000 healthy 1000 sick people (pre test prevalence) Test is positive negative truly sick truly healthy 990 50 10 950
AN OTHER EXAMPLE High sensitivity: 99% High specificity: 95% 1990 healthy 10 sick people (pre test prevalence) Test is positive negative truly sick truly healthy 10 100 0 1890
POSITIVE PREDICTIVE VALUE PPV= True Positive True Positive + False Positive Example 1: PPV= 990/(990+50) = 95% Example 2: PPV= 10/(10+100) = 9% Same test but different expected prevalence
SOMETHING OUT OF NOTHING 20 4 12 28
SOMETHING OUT OF NOTHING 10 17 19 20
SOMETHING OUT OF NOTHING Everyday process Notice a pattern in data Assume pattern is now proven Scientific process Notice a pattern in data: Kramer and scurvy, Jenner and smallpox Make hypothesis Verify hypothesis on new set of data
TOO MUCH FROM TOO LITTLE Excessive impact of confirmatory evidence Hidden or absent data
CONFIRMATORY EVIDENCE Flash card has a letter on one side and a number on the other. Hypothesis to test: All vowels have an even number on the other side. 4 cards. Which two to flip first: A B 2 3
HIDDEN OR ABSENT DATA Does underwriting do a good job at selecting risk?
UNDERWRITING DATA NEEDED Long lived Short lived Decline Accept
UNDERWRITING DATA AVAILABLE Long lived Short lived Decline Accept
UNDERWRITING DATA NEEDED Long lived Short lived Decline Accept
ANALYZING DATA Seeing what we expect to see Seeing what we want to see It is a freaky coincidence world out there. Or is it?
SEEING WHAT WE EXPECT TO SEE Value of cognitive rationality Believing by story Death penalty study We do not ignore information that clashes with what we believe We just scrutinize more and find flaws
SEEING WHAT WE WANT TO SEE Paul Broca and craniology French vs. German and men vs. women brains The danger of moving end points
OUTCOME ASYMMETRIES Hedonic asymmetries One result has different emotional impact than the other Pattern asymmetries Digital clock neat pattern Definitional asymmetries always tell a facelift Base rate departure Unconventional cancer treatment P 66
CORRELATION Measure of the tendency of two sets of data to move similarly (or opposite)
CAUSATION VS. CORRELATION We see correlation We want causation Correlation is not causation Correlation may not imply causation, but it sure can help us insinuate it Vali Chandrasekaran, Bloomberg Businessweek
SPURIOUS CORRELATION Correlation: A moves more or less in synch with B Causation: A causes B Spurious correlation Coincidence A and B both depend on C: ice cream causes drowning fireman correlation BP and income
MATHEMATICAL SPURIOUS CORRELATION When two measures share a common term, they will show correlation Example: BMI and height
EASY TO SPOT? Hormone replacement therapy lowers coronary heart disease Increasing good cholesterol (HDL) will decrease the risk of heart attack Correlation existed but causation studies failed
HIDDEN VARIABLES Look at confounders Age Gender Smoking Income
RECAP Something out of nothing Recognizing random Regression to the mean Prior probability and testing Seeing pattern Too much from too little Confirmatory evidence Absent data
RECAP Seeing what we expect or want to see Believe by story Selective scrutiny Correlation Not causation Hidden variable(s) Confounders