Modeling television viewership

Transcription

1 Modeling television viewership The Nielsen ratings are the best known measures of viewership of television shows. These ratings form the basis for the setting of advertising rates, and are thus crucial for the success (and survival) of shows. The ratings, which are based on diaries kept by a sample of Nielsen families, are intended to measure in home viewing. This can result in biased figures for shows that are often watched in large public areas such as bars or restaurants, such as sports shows, or shows that appeal to people who watch television in groups, such as those aimed at college aged viewers. Two ratings values are typically examined: the rating, which is (an estimate of) the percentage of televisions tuned to a particular show at a particular time out of all televisions, and the share, which is the percentage of televisions tuned to a particular show at a particular time out of all televisions that are being used at that time. The latter variable corrects for the fact that more or less television in general is watched on certain nights of the week. These numbers can then be converted into estimates of the total number of viewers of the program (and thereby total number of potential customers for the advertisers!). The data analyzed here are the estimated household ratings for each of the 157 television shows for new episodes during the season (that is, not including repeat episodes shown in the regular time slot). I am indebted to Karl Rosen for sharing these data with me. For each show the network (ABC, CBS, CW, Fox, or NBC) and type (comedy, drama, news, reality/participation, or animation) are recorded. Since these data are actually a listing of all shows for , they are not a sample from that season, but rather should be viewed as a snapshot sample from a (hopefully) stable ongoing process. That is, a significant difference between two networks, for example, would hopefully say something about the season and beyond. Note that NBC s Sunday Night Football and ABC s Saturday Movie of the Week are not included, since they were the only shows of their type broadcast in prime time by the networks during the season. Side by side boxplots show that there are definitely network and type effects. The networks fall into three groups: CBS, the other major networks (NBC, ABC, and Fox), and the netlet CW, which seriously lags behind. Animation and news shows are generally lower rated, while comedies, dramas, and reality shows are similar. There are noticeably different amounts of variability in household rating across the different networks and the different types of shows. c 2017, Jeffrey S. Simonoff 1

2 Our first attempt to fit a two way ANOVA model to these data ends in failure: Minitab refuses to fit the model with the interaction, giving the message General Linear Model: HH Rating versus Network, Type The following terms cannot be estimated and were removed: Network*Type and then fits the model with only main effects. A table that cross classifies the shows by network and type reveals the problem: there are combinations that never occur (seven of them), making it impossible to fit a model with an interaction effect. c 2017, Jeffrey S. Simonoff 2

3 Rows: Network Columns: Full type Evening Comedy Drama Animation News Reality/Participatio All ABC CBS CW FOX NBC All Cell Contents: Count There are three things we might do here. One would be to fit a model with only main effects (what Minitab did automatically). I could do that, but I am interested in whether certain networks are better performers for certain types of shows, and that defines the interaction effect. A second possibility is to figure out a way to fit an interaction effect even when there are holes in the data. That can be done, in fact, but I ll postpone that to an appendix. A third possibility is to change our data a bit so that the holes aren t there any more. I ll do that here, by noting three things. First, evening animation is a big problem, since those shows only occurred on Fox. All of those shows are comedies, however, so I will just reclassify them as comedies. Second, the CW network is a problem, since it has no comedies or news shows. Third, news shows are a problem, since neither CW nor Fox has any of them. I will address by removing CW shows and news shows for now. This still leaves 85.4% (134) of the shows in the sample. Here is a two way ANOVA for the 134 comedies, dramas, and reality shows on ABC, CBS, Fox, and NBC: General Linear Model: HH Rating versus Network, Type Method Factor coding (-1, 0, +1) Factor Information Factor Type Levels Values Network Fixed 4 ABC, CBS, c 2017, Jeffrey S. Simonoff 3

4 FOX, NBC Type Fixed 3 Comedy, Drama, Reality/Participatio Analysis of Variance Source DF Adj SS Adj MS F-Value P-Value Network Type Network*Type Error Total Model Summary S R-sq R-sq(adj) R-sq(pred) % 36.53% 29.71% Coefficients Term Coef SE Coef T-Value P-Value VIF Constant Network ABC CBS FOX Type Comedy Drama Network*Type ABC Comedy ABC Drama CBS Comedy CBS Drama FOX Comedy FOX Drama Regression Equation HH Rating = Network_ABC Network_CBS Network_FOX Network_NBC Type_Comedy Type_Drama Type_Reality/Participatio Network*Type_ABC Comedy c 2017, Jeffrey S. Simonoff 4

5 -0.170Network*Type_ABC Drama Network*Type_ABC Reality/Participatio Network*Type_CBS Comedy Network*Type_CBS Drama Network*Type_CBS Reality/Participatio Network*Type_FOX Comedy Network*Type_FOX Drama Network*Type_FOX Reality/Participatio Network*Type_NBC Comedy Network*Type_NBC Drama Network*Type_NBC Reality/Participatio The interaction effect is not close to statistically significant. The two main effects are statistically significant, but we should remember than in an unbalanced design situation like this, it can happen that the presence of an insignificant interaction effect can make main effects look significant when they wouldn t be once the interaction is removed from the model. Unfortunately, a plot of residuals versus fitted values shows that we have long right-tailed residuals and nonconstant variance, which suggests modeling viewers in the logged scale. Here are side-by-side boxplots for logged viewership separated by network and type. While the general patterns are similar to before, there is some evidence that the nonconstant variance might be alleviated somewhat. c 2017, Jeffrey S. Simonoff 5

6 Here is an ANOVA with logged viewership as the response. General Linear Model: Logged rating versus Network, Type Method Factor coding (-1, 0, +1) Factor Information Factor Type Levels Values Network Fixed 4 ABC, CBS, c 2017, Jeffrey S. Simonoff 6

7 FOX, NBC Type Fixed 3 Comedy, Drama, Reality/Participatio Analysis of Variance Source DF Adj SS Adj MS F-Value P-Value Network Type Network*Type Error Total Model Summary S R-sq R-sq(adj) R-sq(pred) % 33.20% 26.03% Coefficients Term Coef SE Coef T-Value P-Value VIF Constant Network ABC CBS FOX Type Comedy Drama Network*Type ABC Comedy ABC Drama CBS Comedy CBS Drama FOX Comedy FOX Drama Regression Equation Logged rating = Network_ABC Network_CBS Network_FOX Network_NBC Type_Comedy Type_Drama Type_Reality/Participatio Network*Type_ABC c 2017, Jeffrey S. Simonoff 7

8 Comedy Network*Type_ABC Drama Network*Type_ABC Reality/Participatio Network*Type_CBS Comedy Network*Type_CBS Drama Network*Type_CBS Reality/Participatio Network*Type_FOX Comedy Network*Type_FOX Drama Network*Type_FOX Reality/Participatio Network*Type_NBC Comedy Network*Type_NBC Drama Network*Type_NBC Reality/Participatio The interaction effect is still quite insignificant, but there is still a problem, in that there are four clear outliers: These are the four lowest-rated shows of the season: two versions of Comedy Time Saturday on CBS, Q Viva on Fox, and Escape Routes on NBC. While these four shows are lowest-rated, it might not be immediately apparent why they are so distinctly outlying. The issue is that they are particularly low-rated for their own groups (CBS comedy, Fox reality, and NBC reality); if they had all been NBC comedies (the lowest-rated combination), for example, they might not have been outliers. Did these shows have an important effect? Apparently so: c 2017, Jeffrey S. Simonoff 8

9 General Linear Model: Logged rating versus Network, Type Method Factor coding (-1, 0, +1) Factor Information Factor Type Levels Values Network Fixed 4 ABC, CBS, FOX, NBC Type Fixed 3 Comedy, Drama, Reality/Participatio Analysis of Variance Source DF Adj SS Adj MS F-Value P-Value Network Type Network*Type Error Total Model Summary S R-sq R-sq(adj) R-sq(pred) % 53.56% 48.34% Coefficients Term Coef SE Coef T-Value P-Value VIF Constant Network ABC CBS FOX Type Comedy Drama Network*Type ABC Comedy ABC Drama CBS Comedy c 2017, Jeffrey S. Simonoff 9

10 CBS Drama FOX Comedy FOX Drama Regression Equation Logged rating = Network_ABC Network_CBS Network_FOX Network_NBC Type_Comedy Type_Drama Type_Reality/Participatio Network*Type_ABC Comedy Network*Type_ABC Drama Network*Type_ABC Reality/Participatio Network*Type_CBS Comedy Network*Type_CBS Drama Network*Type_CBS Reality/Participatio Network*Type_FOX Comedy Network*Type_FOX Drama Network*Type_FOX Reality/Participatio Network*Type_NBC Comedy Network*Type_NBC Drama Network*Type_NBC Reality/Participatio Means Fitted Term Mean SE Mean Network ABC CBS FOX NBC Type Comedy Drama Reality/Participatio Network*Type ABC Comedy ABC Drama ABC Reality/Participatio CBS Comedy CBS Drama CBS Reality/Participatio FOX Comedy FOX Drama FOX Reality/Participatio NBC Comedy NBC Drama c 2017, Jeffrey S. Simonoff 10

11 NBC Reality/Participatio The interaction effect is now statistically significant, so apparently the relative performance of comedies, dramas, and reality shows differs from network to network. Note that in a model that includes the interaction effect the fitted (and predicted) values correspond to the means for each network / type combination, which is the average response value for each combination. Let s look at residual plots to see if the assumptions of the regression seem reasonable now. c 2017, Jeffrey S. Simonoff 11

12 The residual plots look better than they have before. There is still a bit of a right tail, and some evidence of nonconstant variance. We can look at Levene s test to see if nonconstant variance is indicated by it. Analysis of Variance Source DF Adj SS Adj MS F-Value P-Value Network Type Network*Type Error Total Model Summary S R-sq R-sq(adj) R-sq(pred) % 18.98% 10.47% The test indicates nonconstant variance related to both network and type (and both together). Note, by the way, that if the interaction term in this test had been insignificant, we would then rerun the Levene s test with only main effects, since the presence of the interaction could obscure the potential importance of main effects in accounting for nonconstant variance. c 2017, Jeffrey S. Simonoff 12

13 How would we handle this heteroscedasticity? We would use weighted least squares, of course. I will stick with logged rating as the response variable because of the long right tails of the residuals in the original analysis, although another approach would be to do WLS for rating in the original scale. The appendix discusses a second way of getting the weights, but I will use here the same method we used for one-way ANOVA. First, we determine the weights based on the standard deviations of the residuals from the model. Test for Equal Variances: SRES3 versus Network, Type 95% Bonferroni Confidence Intervals for Standard Deviations Network Type N StDev CI ABC Comedy ( , ) ABC Drama ( , ) ABC Reality/Participatio ( , ) CBS Comedy ( , ) CBS Drama ( , ) CBS Reality/Participatio ( , ) FOX Comedy ( , ) FOX Drama ( , ) FOX Reality/Participatio ( , ) NBC Comedy ( , ) NBC Drama ( , ) NBC Reality/Participatio ( , ) The weights are the inverse of the squared entries given under StDev. We fit a WLS model based on all of the observations (including the OLS outliers), since an observation might not be an outlier any more relative to a higher estimated standard deviation. This is in fact the case here, since there are now only 3 outliers apparent from the WLS fit of the two-way ANOVA: c 2017, Jeffrey S. Simonoff 13

14 The show Q Viva is no longer an outlier, because logged ratings for Fox reality shows have larger-than-average variability (the two versions of Comedy Time Saturday and Escape Routes are still outliers). It turns out, however, that once the other three shows are omitted Q Viva shows up as a little unusual, so we ll go back to omitting all four of them: c 2017, Jeffrey S. Simonoff 14

15 General Linear Model: Logged rating versus Network, Type Method Factor coding (-1, 0, +1) Weights wt Factor Information Factor Type Levels Values Network Fixed 4 ABC, CBS, FOX, NBC Type Fixed 3 Comedy, Drama, Reality/Participatio Analysis of Variance Source DF Adj SS Adj MS F-Value P-Value Network Type Network*Type Error Total Model Summary S R-sq R-sq(adj) R-sq(pred) % 71.03% 67.86% Coefficients Term Coef SE Coef T-Value P-Value VIF Constant Network ABC CBS FOX Type Comedy Drama Network*Type ABC Comedy ABC Drama c 2017, Jeffrey S. Simonoff 15

16 CBS Comedy CBS Drama FOX Comedy FOX Drama Means Fitted Term Mean SE Mean Network ABC CBS FOX NBC Type Comedy Drama Reality/Participatio Network*Type ABC Comedy ABC Drama ABC Reality/Participatio CBS Comedy CBS Drama CBS Reality/Participatio FOX Comedy FOX Drama FOX Reality/Participatio NBC Comedy NBC Drama NBC Reality/Participatio The interaction effect is highly statistically significant. The following interaction plot summarizes the effect: c 2017, Jeffrey S. Simonoff 16

17 We see that Fox and NBC are very similar to each other, with reality shows having the highest ratings, dramas lower, and comedies lowest; Fox is a bit higher than NBC, which has the lowest ratings in all three categories. CBS and ABC have higher ratings for dramas, and lower ratings for comedies and reality shows (although the differences between types are smaller, especially for ABC), with CBS having the highest ratings for all three types of shows and ABC somewhat lower. Another way of looking at this is that the networks generally rank CBS / ABC / Fox / NBC, with the differences between the ratings of the networks being largest for comedies, smaller for dramas, and smallest for reality shows. Residual plots and diagnostics look fine (remember that the guideline for leverage values is (2.5)(12)/130 =.231, since there are = 11 predictor variables in the regression that corresponds to the two-way ANOVA fit). c 2017, Jeffrey S. Simonoff 17

18 c 2017, Jeffrey S. Simonoff 18

19 Row Show SRES4 HI4 COOK4 1 NCIS DANCING WITH THE STARS AMERICAN IDOL-WEDNESDAY DANCING W/STARS RESULTS AMERICAN IDOL-THURSDAY NCIS: LOS ANGELES BIG BANG THEORY, THE TWO AND A HALF MEN MENTALIST, THE PERSON OF INTEREST CRIMINAL MINDS VOICE UNFORGETTABLE CSI BLUE BLOODS MODERN FAMILY MIKE & MOLLY CASTLE BROKE GIRLS HAWAII FIVE X-FACTOR-THU SURVIVOR: SOUTH PACIFIC GOOD WIFE, THE X-FACTOR-WED ROB GREY S ANATOMY c 2017, Jeffrey S. Simonoff 19

20 27 CSI: MIAMI CSI: NY AMAZING RACE AMERICA S GOT TALENT-TUE SURVIVOR: ONE WORLD AMERICA S GOT TALENT-MON VOICE:RESULTS SHOW HOW I MET YOUR MOTHER BODY OF PROOF ONCE UPON A TIME RULES OF ENGAGEMENT DESPERATE HOUSEWIVES GIFTED MAN, A UNDERCOVER BOSS AMAZING RACE BACHELOR, THE LAST MAN STANDING BACHELORETTE, THE REVENGE MIDDLE, THE HOW TO BE A GENTLEMAN HARRY S LAW MISSING NYC BONES SCANDAL PRIVATE PRACTICE LAST MAN STANDING-8:30PM TOUCH SUBURGATORY LAW AND ORDER:SVU DUETS GLEE TERRA NOVA SMASH HOUSE ALCATRAZ CHARLIE S ANGELS MAN UP! GCB ROOKIE BLUE BIGGEST LOSER NEW GIRL APPRENTICE BIGGEST LOSER c 2017, Jeffrey S. Simonoff 20

21 72 HAPPY ENDINGS AMER FUNN HOME VIDEOS SO YOU THINK CN DANCE DON T TRUST THE B-APT FEAR FACTOR WHO S STILL STANDING PAN AM OFF THEIR ROCKRS CELEBRITY WIFE SWAP SHARK TANK WHO DO YOU THINK YOU ARE EXTREME MAKEOVER:HOME ED HELL S KITCHEN-MON EXTREME MAKEOVER:HM ED-9P FAMILY GUY WORK IT SIMPSONS WIPEOUT-THURS HELL S KITCHEN-MON 9P FINDER PARENTHOOD AMERICAN NINJA WARRIOR MOBBED RIVER, THE OFF THEIR ROCKRS OFFICE PRIME SUSPECT GRIMM FASHION STAR YOU DESERVE IT COUGAR TOWN WHITNEY I HATE MY TEENAGE DGHTR PLAYBOY CLUB RAISING HOPE NAPOLEON DYNAMITE AMERICAN DAD SING OFF CLEVELAND-SUN 8:30P UP ALL NIGHT COPS ARE YOU THERE CHELSEA ALLEN GREGORY FREE AGENTS PARKS AND RECREATION c 2017, Jeffrey S. Simonoff 21

22 117 AWAKE ROCK KITCHEN NIGHTMARES COMMUNITY BOB S BURGERS COPS CHUCK CLEVELAND FIRM BEST FRIENDS FOREVER FRINGE BREAKING IN BENT 9P BENT c 2017, Jeffrey S. Simonoff 22

23 Appendix: Fitting a two way ANOVA model to data where some combinations are missing How could we have fit a two way ANOVA model including an interaction effect to the full data set? The key is to fit the interaction manually using indicator or effect coding variables, and determine the appropriate partial F test by hand. So, for example, in this example, four variables are created to represent the Network main effect, three are created to represent the Type main effect (keeping the animation shows as comedies), and then 12 pairwise products are created to represent the interaction (although as we will see not all of those are used). In fact, Minitab gives us the partial F -test that we need, although it obscures this fact somewhat. Here is the fit of the two-way ANOVA based on only the main effects; we are using all of the shows other than the four identified as outliers in the earlier analysis, and are now including the CW as a network and news shows as a type: General Linear Model: Logged rating versus Network, Type Method Factor coding (-1, 0, +1) Factor Information Factor Type Levels Values Network Fixed 5 ABC, CBS, CW, FOX, NBC Type Fixed 4 Comedy, Drama, News, Reality/Participatio Analysis of Variance Source DF Adj SS Adj MS F-Value P-Value Network Type Error Lack-of-Fit Pure Error Total Model Summary c 2017, Jeffrey S. Simonoff 23

24 S R-sq R-sq(adj) R-sq(pred) % 72.34% 70.54% It turns out that what Minitab is reporting as the Lack-of-Fit test is, in fact, the test for the two-way interaction between Network and Type. As you can see, it is strongly statistically significant, with F = 2.65 and p =.007. Since the effect is statistically significant we clearly need to fit the model, so we can check assumptions (construct residual plots and diagnostics, perform a Levene s test, and so on); indeed, even if the interaction is not statistically significant, we would still want to do this to make sure that violations of assumptions haven t resulted in the test mistakenly indicating that the interaction is not needed. To create the indicator variables we need, click on Calc Make Indicator Variables. Enter the first categorical variable (say Network) under Indicator variables for:. The program will automatically provide names for the indicator variables that will be formed, but you can change those if you want. Note that an indicator variable for each of the categories will be formed, but one should be ignored. Do the same for the second categorical variable (once again ignoring one of the variables formed). Finally, use the calculator to construct the pairwise products of each indicator for rows by each indicator for columns. If you now try to fit the model using these variables using the regression program (not the General Linear Model) treating the indicators as continuous predictors, it will work, but you have to remember to not include any of the product variables that are all zeroes in your regression call (there will one of these for each of the empty cells). Here are results based on logged rating (this output is from Minitab 16, which is why it looks a little different). This is OLS output, but if nonconstant variance was indicated a WLS analysis would be conducted by constructing a weight variable in the same way as was done earlier. Regression Analysis: Logged rating versus Network_ABC, Network_CBS,... The regression equation is Logged rating = Network_ABC Network_CBS Network_CW Network_FOX Type_Comedy Type_Drama Type_News ABCComedy ABCDrama ABCNews CBSComedy CBSDrama CBSNews CWDrama FoxComedy FoxDrama Predictor Coef SE Coef T P c 2017, Jeffrey S. Simonoff 24

25 Constant Network_ABC Network_CBS Network_CW Network_FOX Type_Comedy Type_Drama Type_News ABCComedy ABCDrama ABCNews CBSComedy CBSDrama CBSNews CWDrama FoxComedy FoxDrama S = R-Sq = 77.5% R-Sq(adj) = 74.9% Analysis of Variance Source DF SS MS F P Regression Residual Error Total Here is an interaction plot: c 2017, Jeffrey S. Simonoff 25

26 In addition to the patterns we saw before, we see that CW has much lower ratings than any of the other networks, and while news shows for NBC have ratings at about the middle level for that network, they are lowest for CBS shows, and by far the lowest-rated shows for ABC. c 2017, Jeffrey S. Simonoff 26

27 Minitab commands Two way analysis of variance is conducted by clicking on Stat ANOVA General Linear Model Fit General Linear Model. Enter the target variable under Responses: and the two categorizing predicting variables under Factors:. To include the interaction effect, click on Model. Highlight the two factor variables to the left, and click on Add. This will add the variables multiplied by each other (i.e., ROW*COL) under Terms in the model:. Residual plots and storage are obtained as stated earlier. To get effect estimates for your model, click on Options and then All terms in the model in the drop-down menu next to Means:. Note that the effects for main effects are not interpretable in the presence of the interaction. To construct an interaction plot, click on Stat ANOVA Interaction plot. Enter the two predicting variables that define the interaction under Factors:, and enter the response variable next to the box labeled Responses:. Levene s test is constructed in the usual way by fitting a two-way ANOVA with the absolute standardized residuals as the response. Note that if n ij = 1 in some cell(s) the standardized residual produced by Minitab for that single observation will be set to the missing value code * because technically the standardized residual is undefined (h ii = 1 for the observation in a cell with n ij = 1, so the standardized residual is 0/0). For such an observation set the standardized residual equal to 0 and the weight equal to 1, since the observation will be fit perfectly (resulting in a zero residual) no matter what weight is used. Remember that if the interaction effect is not significant in the Levene s test ANOVA you should run it again with the interaction effect removed to see if it is related to one or the other main effect; to do so highlight the product term in the box under Terms in the model and click X. If weights for a weighted least squares fit depend on only one of the effects, they can be determined using the method described for one-way ANOVA models. If weights are needed based on two categorical variables (either if the interaction effect in the Levene s test is statistically significant or if it is not but both main effects in the Levene s test are), they can be estimated simultaneously. Click on Stat ANOVA Test for Equal Variances. Enter the residuals from the OLS fit under Response:, and the two variables under Factors:. The resultant output gives the standard deviations of the residuals separated by the levels of the variables under StDev in the portion labeled Bonferroni confidence intervals for standard deviations. The weights are one over the squared standard deviations. Note that you should not use the tests provided in the output as your test of constant variance in a two-way ANOVA, as they do not take into account the potential structure in the nonconstant variance; construct Levene s test as is described in the handout. An alternative approach to get weights is to estimate the variances in the way c 2017, Jeffrey S. Simonoff 27

28 that is discussed for a numerical predictor in the Appendix of the CAPM handout. That is, save the standardized residuals SRES from the original two way ANOVA, perform a two-way ANOVA with log(sres SRES) as the target variable, saving the fitted values; and then set the weights equal to WT = 1/exp(FITS). To construct a table tabulating counts of the observations separated by a cross-classification of predictive variables, click Stat Tables Cross Tabulation and Chi-Square. Enter the variables that define the effects in the ANOVA under Categorical variables (one under For rows and the other under For columns). To get a table of means of the response variable separated by the predicting variables, click Stat Tables Descriptive Statistics. Enter the variables that define the effects in the ANOVA under Categorical variables (one under For rows and the other under For columns), and click on Associated Variables. Enter the target variable for the ANOVA under Associated variables: and click in the box next to Means. In this situation, you might also want to obtain the estimated target variable for each combination of the two predictors. This is not the response cell mean, since the interaction effect hasn t been fit. Using a calculator, calculate the overall average of the fitted means given for one of the two effects (it doesn t matter which one). The estimated expected response for the (i, j)th combination is the ith row effect + the jth column effect the overall average. If a two-way ANOVA model is fit without an interaction term, multiple comparisons for either main effect (or both) can be obtained by highlighting and entering each term (or both) under Choose terms for comparisons as is done for one-way ANOVA models. In a model that includes an interaction, comparisons can be made between the different combinations of row and column level by entering the interaction (ROW*COL for the data analyzed in this handout) under Terms:. c 2017, Jeffrey S. Simonoff 28