HOW WE KNOW WHAT ISN T SO: COGNITIVE SCIENCE AND MIND TRAPS

Transcription

1 HOW WE KNOW WHAT ISN T SO: COGNITIVE SCIENCE AND MIND TRAPS Jean-Marc Fix, FSA, MAAA UW seminar 2012

2 COGNITIVE SCIENCE How do we make judgments? Are there systematic issues with our decision making process? a machine for jumping to conclusions (1) Sources: How we know what isn t so, the fallibility of human reason in everyday life; Thomas Gilovich Knowing you, why cognitive science matters for actuarial science; James Guszcza, Contingences May 2012 (1) Attributed to Kahneman Contingencies May 2012 p18

3 LOOKING AT DATA Something out of nothing Too much from too little Seeing what we expect to see The ultimate mind trap: correlation

4 SOMETHING OUT OF NOTHING Recognizing random The hot streak Seeing patterns Regression to the mean Reward or punishment?

5 RECOGNIZING RANDOM A coin is tossed 20 times. We get twice as many of one side than the other (i.e. 13/7, 14/6 ). I say the coin was fair. I am: 1. a liar (10% that what I say is true) 2. a big liar (5% that it is true) 3. I am the new Baron Munchausen (1% that it is true)

6 RECOGNIZING RANDOM A coin is tossed 20 times. We get twice as many of one side than the other (i.e. 13/7, 14/6 ). I say the coin was fair. I am: a con man. The answer is about 30%.

7 THE HOT STREAK I toss a coin 20 times. How rare is a streak of: At least 7 times? 12% At least 10 times? 1%

8 REGRESSION TO THE MEAN Truth Average Skill Error Luck

9 EXAMPLE: SKILL ONLY 1000 perfect students answer a 20 yes/no questions. Average score will be around 100% Take top 100 students (whose average will be around 100%) and give them another similar test. Their average score will be 100% again.

10 EXAMPLE: LUCK/RANDOM 1000 students answer a 20 yes/no questions at random. Average score will be around 50% Take top 100 students (whose average will be around 75%) and give them another similar test also answered at random. Their average score will be closer to 50%.

11 EXAMPLE: REAL LIFE 1000 random students answer a 20 yes/no questions at the best of their ability. Average score will be around X% Take top 100 students. Average score is likely much above X%. Some were good and some were bad but lucky. Give them another similar test. Their average score will be closer to X%.

12 APPLICATION: REWARD VS. PUNISHMENT A B-student gets an A Parent congratulate Next time kid not so lucky on test gets B. Parent notices reward did not work A B-student gets a C Parent punish Next time kid not so unlucky on test gets B. Parent notices punishment worked

13 POST TEST ESTIMATION

14 THE TRUTH Person is really Sick Healthy

15 THE TEST Person is really Sick Healthy Positive Test is Negative

16 THE GOOD Person is really Sick Healthy Test is Positive Negative True Positive

17 THE (STILL) GOOD Person is really Sick Healthy Test is Positive Negative True Positive True Negative

18 THE BAD Person is really Sick Healthy Test is Positive True Positive False Positive Negative True Negative

19 THE UGLY Person is really Sick Healthy Test is Positive True Positive False Positive Negative False Negative True Negative

20 SENSITIVITY Are all the sick ones gotten by the test? Sensitivity= True Positive True Positive + False Negative Sick ones that the test missed

21 SENSITIVITY Sensitivity= True Positive All sick people

22 SPECIFICITY Are all the ones with a negative test healthy? Specificity= True Negative True Negative + False Positive Healthy ones that the test thought were sick

23 SPECIFICITY Specificity= True Negative All healthy people

24 GOOD TEST Sensitivity Specificity

25 A GOOD TEST Let s assume a test for a really really bad disease has those characteristics Sensitivity: 99% Specificity: 95% You underwent the test and are positive. Should you jump in front of a train?

26 AN EXAMPLE High sensitivity: 99% High specificity: 95% 1000 healthy 1000 sick people (pre test prevalence) Test is positive negative truly sick truly healthy

27 AN OTHER EXAMPLE High sensitivity: 99% High specificity: 95% 1990 healthy 10 sick people (pre test prevalence) Test is positive negative truly sick truly healthy

28 POSITIVE PREDICTIVE VALUE PPV= True Positive True Positive + False Positive Example 1: PPV= 990/(990+50) = 95% Example 2: PPV= 10/(10+100) = 9% Same test but different expected prevalence

29 SOMETHING OUT OF NOTHING

30 SOMETHING OUT OF NOTHING

31 SOMETHING OUT OF NOTHING Everyday process Notice a pattern in data Assume pattern is now proven Scientific process Notice a pattern in data: Kramer and scurvy, Jenner and smallpox Make hypothesis Verify hypothesis on new set of data

32 TOO MUCH FROM TOO LITTLE Excessive impact of confirmatory evidence Hidden or absent data

33 CONFIRMATORY EVIDENCE Flash card has a letter on one side and a number on the other. Hypothesis to test: All vowels have an even number on the other side. 4 cards. Which two to flip first: A B 2 3

34 HIDDEN OR ABSENT DATA Does underwriting do a good job at selecting risk?

35 UNDERWRITING DATA NEEDED Long lived Short lived Decline Accept

36 UNDERWRITING DATA AVAILABLE Long lived Short lived Decline Accept

37 UNDERWRITING DATA NEEDED Long lived Short lived Decline Accept

38 ANALYZING DATA Seeing what we expect to see Seeing what we want to see It is a freaky coincidence world out there. Or is it?

39 SEEING WHAT WE EXPECT TO SEE Value of cognitive rationality Believing by story Death penalty study We do not ignore information that clashes with what we believe We just scrutinize more and find flaws

40 SEEING WHAT WE WANT TO SEE Paul Broca and craniology French vs. German and men vs. women brains The danger of moving end points

41 OUTCOME ASYMMETRIES Hedonic asymmetries One result has different emotional impact than the other Pattern asymmetries Digital clock neat pattern Definitional asymmetries always tell a facelift Base rate departure Unconventional cancer treatment P 66

42 CORRELATION Measure of the tendency of two sets of data to move similarly (or opposite)

43 CAUSATION VS. CORRELATION We see correlation We want causation Correlation is not causation Correlation may not imply causation, but it sure can help us insinuate it Vali Chandrasekaran, Bloomberg Businessweek

44 SPURIOUS CORRELATION Correlation: A moves more or less in synch with B Causation: A causes B Spurious correlation Coincidence A and B both depend on C: ice cream causes drowning fireman correlation BP and income

45 MATHEMATICAL SPURIOUS CORRELATION When two measures share a common term, they will show correlation Example: BMI and height

46 EASY TO SPOT? Hormone replacement therapy lowers coronary heart disease Increasing good cholesterol (HDL) will decrease the risk of heart attack Correlation existed but causation studies failed

47 HIDDEN VARIABLES Look at confounders Age Gender Smoking Income

48 RECAP Something out of nothing Recognizing random Regression to the mean Prior probability and testing Seeing pattern Too much from too little Confirmatory evidence Absent data

49 RECAP Seeing what we expect or want to see Believe by story Selective scrutiny Correlation Not causation Hidden variable(s) Confounders