Comparison of Mixed-Effects Model, Pattern-Mixture Model, and Selection Model in Estimating Treatment Effect Using PRO Data in Clinical Trials Xiaolei Zhou, 1,2 Jianmin Wang, 1 Jessica Zhang, 1 Hongtu Zhu 2 1 RTI-Health Solutions 2 Department of Biostatistics, University of North Carolina JSM, August 1, 2011 LEADING RESEARCH MEASURES THAT COUNT
Introduction Compared with clinical outcomes evaluated by physicians, patient-reported outcomes (PROs) are a relatively new addition to clinical trial endpoints What are PROs? Patient self-reported Reflect patient views on outcomes (e.g. HRQOL) Usually collected via validated questionnaires How are PROs used in clinical trials? Primary endpoints Secondary endpoints, (efficacy, safety) as part of an integrated evaluation of the treatment effect Increasingly, clinical studies have included PRO assessments starting with phase 2 trials page 2
Missing PRO Data Missing data commonly occur in PROs Reasons for missing data Administrative reasons Patient s health condition, disease- or treatmentrelated symptoms Burden of completion Once a patient has missed a PRO assessment, the retrospective collection of these data is usually impossible page 3
FDA Guidance for Industry in Using PRO in Clinical Trials for Label Claims PRO instrument development Minimize administrator burden and patient burden Protocol Continue to collect PRO data after treatment discontinuation Establish a process to obtain PROs before or shortly after patient withdrawal from treatment Have backup plans for gathering all treatment-related reasons for missing data Statistical analysis plan Describe how missing data (e.g., withdrawal, death) will be handled in the analysis State the number of nonmissing items required for the domain score to be valid The FDA does not consider any single statistical strategy to deal with missing data due to early termination of patients to be preferred over other methods page 4
Statistical Strategies to Deal With Missing Data Mixed-effects models, pattern-mixture models (PMMs), and selection models have been used in PRO data analysis of collected data Because the true treatment effect is unknown in previously collected data, it is impossible to evaluate the bias of the estimate using these data A simulation study is needed to evaluate the impact of missing data on the estimate of treatment effect Objective Perform simulations to assess the magnitude of the bias and the robustness of mixed-effects models, PMMs, and selection models under different mechanisms of missing not at random (MNAR) that are at different degrees of perturbation to the model assumptions Our primary interest is the estimate of treatment effect, which is also the interest of most clinical trials page 5
Definition of Treatment Effect Difference in E(y t ) between treatments - with baseline being adjusted Form of Response Variable Model Treatment Effect y 0 as a covariate Response score E(y t ) = β 0 + β 1 trt + β 2 t + β 3 trt t + β 4 y 0, t >0 Difference in E(y t ) at t>0 given a fixed y 0 : E[y t trt = 1, y 0 ] - E[y t trt = 0, y 0 ] = β 1 +β 3 t Change from baseline E(y t - y 0 ) = β 0 + β 1 trt + β 2 t + β 3 trt t+ β 4 y 0, t >0 Difference in E(y t ) at t>0 given a fixed y 0 : E[y t trt = 1, y 0 ] - E[y t trt = 0, y 0 ] = E[y t -y 0 trt = 1, y 0 ] - E[y t -y 0 trt = 0, y 0 ] = β 1 +β 3 t y 0 not as a covariate Response score E(y t ) = β 0 + β 1 trt + β 2 t + β 3 trt t, t 0 Difference in E(y t ) at t>0 minus difference in E(y 0 ): E[y t trt = 1] - E[y t trt = 0] { E[y 0 trt = 1] - E[y 0 trt = 0] } = β 3 t Change from baseline E(y t - y 0 ) = β 0 + β 1 trt + β 2 t + β 3 trt t, t > 0 Difference in E(y t ) at t>0 minus difference in E(y 0 ): E[y t -y 0 trt = 1] - E[y t -y 0 trt = 0] = β 1 +β 3 t Note: Trt is a 1/0 variable indicating treatment/no treatment; t is used to indicate the time variable and the time index for the dependent variable; t = 0 indicates baseline; y 0 = response at baseline; y t = response at t. page 6
Likelihood Method Model Likelihood Function Distribution Note Mixed f(y i, b i ) = f(y i b i ; β, φ) f(b i ; G) y i = X i β + Z i b i + ε i, i = 1,, N b i ~ N q (0, G) ε i ~ N ni (0, φ I ni ) b i ε i PMM f(y i, b i, s i ) = f(y i b i, s i ; β k, φ) f(b i ; G) f(s i ), given s i b i Selection f(y i, b i, r i ) = f(r i y i ) f(y i b i ; β, φ) f(b i ; G), given r i y i b i s i ~ multinomial (1, π i1,, π ik ) For missing pattern k, y i = X i β k + Z i b i + ε i r i = (r i1,, r ini ) T, where r ij y ij ~ Bernoulli (π ij ) Logit(π ij ) = ξ 0 + ξ 1 y ij π i1,, π ik are the proportion of subjects in all K missing patterns in the true population, which may depend on covariates, such as treatment, or on response π ij is the proportion of y ij being missing given the value of y ij in the true population. For MNAR, π ij is dependent on y ij, but it can also depend on response variables at other time points or on covariates page 7
Pattern-Mixture Model Step 1 Create K strata based on missing patterns Step 2 Estimate the pattern-specific regression parameters β k, k=1,,k Step 3 Obtain overall estimates (e.g., response, treatment effect) across all patterns π k does not depend on treatment (proportion in overall sample): E (Y) = E(Y k) π k π k depends on treatment (treatment-specific proportion): E (Y trt) = E(Y k,trt) π k,trt page 8
Simulation Data Generation 100 (scenarios 1 to 6) or 200 (scenarios 7 to 10) subjects with 4 time points Scenarios 1 to 6 y ij = β 0 + β 1 trt i + β 2 x i + β 3 time_2 ij + β 4 time_3 ij + β 5 time_4 ij + b i + ε ij, Scenarios 7 to 10 y ij = β 0 + β 1 trt i + β 2 x i + β 3 time_2 ij + β 4 time_3 ij + β 5 time_4 ij + β 6 pattern_1 i + β 7 pattern_2 i + β 8 pattern_3 i + β 15 time_2 ij trt i + β 16 time_3 ij trt i + β 17 time_4 ij trt i + β 12 time_2 ij pattern_2 i + β 13 time_2 ij pattern_3 i + β 14 time_3 ij pattern_3 i + β 9 trt i pattern_1 i + β 10 trt i pattern_2 i + β 11 trt i pattern_3 i + β 18 time_2 ij pattern_2 i trt i +β 19 time_2 ij pattern_3 i trt i + β 20 time_3 ij pattern_3 i trt i + b i + ε ij Where Treatment, trt i ~ bin(0.5) Baseline, x i ~ N(0,1) Random-effect, b i ~ N(0, 1) Measurement errors ε ij ~ N(0,1), i = 1,, N, j = 1,, n i, Missing data were created according to 10 different missing mechanisms page 9
Mechanism of Missing Data and Missing Rate in Simulation Data (Scenarios 1-6) Scenario Mechanism of Missing Data Setting 1 MCAR 20% of patients completing previous visit are missing randomly 2 MAR : missing depends on treatment Trt1: additional 30% missing at each visit Trt0: additional 10% missing at each visit Missing Pattern Average Missing Rate (%) at Each Time Point by Treatment Dropout Trt1: 0/21/36/48 Trt0: 0/20/37/50 Dropout Trt1: 0/29/50/65 Trt0: 0/9/19/26 3 MAR: dropout depends on x Logit(prob(r ij = 1)) = x i - 3, j = 2,3,4 Dropout Trt1: 0/25/40/51 Trt0: 0/24/40/51 4 MAR: missing depends on y at the previous visit 5a 5b MNAR: missing depends on y at the current visit MNAR: missing depends on y at the current visit 6 MCAR in trt0 MNAR in trt1 Logit(prob(r ij = 1)) = y i,j-1-3, j = 2,3,4 Dropout Trt1: 0/35/62/75 Trt0: 0/21/44/58 Logit(prob(r ij = 1)) = y ij 3, j = 2,3,4 Dropout Trt1: 0/50/69/79 Trt0: 0/36/54/64 Logit(prob(r ij = 1)) = y ij 3, j = 1, 2,3,4 Trt0: 20% of patients completing previous visit are missing randomly Trt1: Logit(prob(r ij = 1)) = y ij 3, j = 2,3,4 Intermittent + dropout Trt1: 33/50/51/49 Trt0: 21/33/34/35 Dropout Trt1: 0/50/70/80 Trt0: 0/20/36/49 MCAR = missing completely at random; MAR = missing at random; MNAR = missing not at random; r ij = indicator that y ij is missing; Trt = treatment; x i = baseline HRQOL score for subject i; y ij = HRQOL score for subject i at visit j. page 10
Mechanism of Missing Data and Missing Rate in Simulation Data (Scenarios 7-10) Scenario Mechanism of Missing Data Setting 7 MNAR-PMM: dropout pattern does not depend on treatment 15% in patten 1; 20% pattern 2; 25% pattern 3; 40% pattern 4 Missing Pattern Average Missing Rate (%) at Each Time Point by Treatment Dropout Trt1: 0/15/35/59 Trt0: 0/16/36/60 8 MNAR-PMM: dropout pattern depends on treatment Trt0: 25% in each pattern Trt1: 15% in patten 1; 20% pattern 2; 25% pattern 3; 40% pattern 4 Dropout Trt1: 0/15/35/59 Trt0: 0/26/51/76 9 MNAR: dropout pattern does not depend on treatment, treatment effect depends on pattern 15% in patten 1; 20% pattern 2; 25% pattern 3; 40% pattern 4 Dropout Trt1: 0/16/35/60 Trt0: 0/15/34/59 10 MNAR: dropout pattern depends on treatment, treatment effect depends on pattern Trt0: 25% in each pattern Trt1: 15% in patten 1; 20% pattern 2; 25% pattern 3; 40% pattern 4 Dropout Trt1: 0/16/35/60 Trt0: 0/24/49/74 MNAR = missing not at random; PMM = pattern-mixture model; Trt = treatment. page 11
Models Used for Estimation Scenarios 1 to 6 Mixed-effects model y ij = β 0 + β 1 trt i + β 2 x i + β 3 time_2 ij + β 4 time_3 ij + β 5 time_4 ij + b i + ε ij. PMM y ij = β 0 + β 1 trt i + β 2 x i + β 3 time_2 ij + β 4 time_3 ij + β 5 time_4 ij + β 6 pattern_1 i + β 7 pattern_2 i + β 8 pattern_3 i + β 9 trt i pattern_1 i + β 10 trt i pattern_2 i + β 11 trt i pattern_3 i + b i + ε ij. Selection model Logit(π ij ) = ξ 0 + ξ 1 y ij page 12
Models Used for Estimation Scenarios 7 to 10 Mixed-effects model y ij = β 0 + β 1 trt i + β 2 x i + β 3 time_2 ij + β 4 time_3 ij + β 5 time_4 ij + β 6 time_2 trt + β 7 time_3 trt + β 8 time_4 trt + b i + ε ij. PMM y ij = β 0 + β 1 trt i + β 2 x i + β 3 time_2 ij + β 4 time_3 ij + β 5 time_4 ij + β 6 pattern_1 i + β 7 pattern_2 i + β 8 pattern_3 i + β 15 time_2 ij trt i + β 16 time_3 ij trt i + β 17 time_4 ij trt i + β 12 time_2 ij pattern_2 i + β 13 time_2 ij pattern_3 i + β 14 time_3 ij pattern_3 i + β 9 trt i pattern_1 i + β 10 trt i pattern_2 i + β 11 trt i pattern_3 i + β 18 time_2 ij pattern_2 i trt i +β 19 time_2 ij pattern_3 i trt i + β 20 time_3 ij pattern_3 i trt i + b i + ε ij Selection model Logit(π ij ) = ξ 0 + ξ 1 y ij page 13
Point Estimate of Treatment Effect (100 Simulations, True Treatment Effect = 1) Scenario Mixed-Effects Model PMM (ov a ) PMM (trt b ) Selection Model 1: MCAR 1.004 1.004 1.005 1.000 2: MAR (treatment) 0.960 0.963 0.957 1.027 3: MAR (x) 0.987 0.997 0.989 0.998 4: MAR (y t-1 ) 1.029 0.734 c 1.076 c 1.116 c 5a: MNAR (y t ) 0.916 c 0.771 c 0.942 c 0.993 5b: MNAR (y t ) Intermittent missing 6: MCAR in Trt0, MNAR(y t ) in Trt1 0.793 c 0.706 c 0.785 c 0.974 0.796 c 0.732 c 0.898 c 0.931 c MCAR = missing completely at random; MAR = missing at random; MNAR = missing not at random; ov = overall ; PMM = pattern-mixture model; Trt = treatment. a Treatment effects estimated using proportion of dropout in overall sample. b Treatment effects estimated using treatment-specific proportion of dropout. c The 95% confidence interval does not cover the true value. page 14
Point Estimate of Treatment Effect (100 Simulations) Scenario 7: PMM, dropout does not depend on treatment, treatment effect does not depend on pattern 8: PMM dropout depends on treatment, treatment effect does not depend on pattern 9: PMM, dropout does not depend on treatment, treatment effect depends on pattern 10: PMM, dropout depends on treatment, treatment effect depends on pattern Time True Treatment Effect Mixed-Effects Model PMM (ov a ) PMM (trt b ) Selection Model 1 1 1.024 1.005 1.023 1.020 2 3 3.021 3.009 3.021 3.023 3 2.5 2.515 2.512 2.514 2.515 4 2 2.050 2.049 2.067 2.000 1 0.6 0.627 1.006 c 0.626 0.631 2 2.55 2.537 3.012 c 2.575 2.487 3 2.1 2.084 2.522 c 2.135 1.997 c 4 1.6 1.669 2.052 c 1.672 1.462 c 1 1.6 1.636 1.617 1.637 1.577 2 4.05 4.147 c 4.051 4.089 4.003 3 3.35 3.544 c 3.364 3.394 3.396 4 2.6 2.868 c 2.648 2.667 2.716 c 1 1.2 1.233 1.695 c 1.234 1.209 2 3.6 3.650 4.150 c 3.629 3.505 c 3 2.95 3.079 c 3.450 c 2.979 2.907 4 2.2 2.463 c 2.718 c 2.258 2.193 ov = overall ; PMM = pattern-mixture model; trt = treatment. a Treatment effects estimated using proportion of dropout in overall sample. b Treatment effects estimated using treatment-specific proportion of dropout. c The 95% confidence interval does not cover the true value. page 15
Summary When There Was a True Treatment Effect Scenario Mixed-Effects Model PMM (ov a ) PMM (trt b ) Selection Model 1 MCAR 2 MAR (treatment) 3 MAR (x) 4 MAR (y t-1 ) Worst 5a,b MNAR (y t ) Worst better than mixed 6 MCAR in trt0, MNAR(y t ) in trt1 Worst better than mixed better than mixed 7 PMM, dropout does not depend on treatment, treatment effect does not depend on pattern 8 PMM dropout depends on treatment, treatment effect does not depend on pattern 9 PMM, dropout does not depend on treatment, treatment effect depends on pattern 10 PMM, dropout depends on treatment, treatment effect depends on pattern Worst Later t Later t Worst ov = overall; PMM = pattern-mixture model; trt = treatment. a Treatment effects estimated using proportion of dropout in overall sample. b Treatment effects estimated using treatment-specific proportion of dropout. page 16
When There Was no True Treatment Effect (i.e., Parameters for Treatment Were 0) The bias was less than 0.10 for all models, except for scenario 6, in which treatments had different missing mechanisms In scenario 6, the bias from the mixed-effects model (-0.179) and PMM estimates using overall proportion (-0.158) was much larger than that from the selection model (-0.090) and PMM using treatment-specific proportion (-0.105) page 17
Conclusions Our simulation results suggest the following: The treatment effect, defined as the difference in the expected value of the response variable given the same baseline, was estimated without bias when the model assumption on the mechanism of missing data held Otherwise, the estimation of treatment effect was biased PMM using the treatment-specific proportion and selection model provided some correction of the estimate compared with mixed-effects model in several MNAR situations, even when the mechanism of missing data was not exactly the same as the model assumption page 18