In linear mixed-effects models, the Dunnett procedure is commonly employed for comparing multiple groups against a control group. While the influence of group number on the Dunnett test’s performance is well-documented, the effects of heteroscedasticity and unequal group sizes remain underexplored. This study investigates the robustness of the Dunnett test within linear mixed-effects models, particularly in the presence of heteroscedasticity, varying group sizes, and different group numbers. Through data simulation, we assessed performance using four key metrics: any-pairwise-power, all-pairwise-power, type I error rate, and false discovery rate. Our findings indicate that the procedure’s performance diminishes significantly when fewer than seven groups are involved, especially when group sizes are small and variances are heterogeneous. Under these circumstances, the type I error rate deviates from the nominal level, despite adequate power estimates being achieved.

Dunnett procedure, assumptions violation, test performance, control group, means comparison.

Received: June 16, 2024; Revised: July 19, 2024; Accepted: October 23, 2024; Published: November 7, 2024

How to cite this article: Codjo Emile Agbangba, Sika Fidele Tchando and Emmanuel Ehnon Gongnet, A simulation study on Dunnett test robustness to group size and heteroscedasticity in linear mixed-effects models, JP Journal of Biostatistics 24(3) (2024), 555-572. https://doi.org/10.17654/0973514324030

This Open Access Article is Licensed under Creative Commons Attribution 4.0 International License

References
[1] C. E. Agbangba, E. S. Aide, H. Honfo and R. G. Kakai, On the use of post-hoc tests in environmental and biological sciences: a critical review, Heliyon 10 (2024), e25131. https://doi.org/10.1016/j.heliyon.2024.e25131.
[2] G. P. Brooks and G. A. Johanson, Sample size considerations for multiple comparison procedures in ANOVA, Journal of Modern Applied Statistical Methods 10(1) (2011), 10.10.22237/jmasm/1304222940.
[3] R. W. Day and G. P. Quinn, Comparisons of treatments after an analysis of variance in ecology, Ecological Monographs 59(4) (1989), 433-463. 10.2307/1943075.
[4] H. Demirhan, N. A. Dolgun, Y. P. Demirhan and M. Ö. Dolgun, Performance of some multiple comparison tests under heteroscedasticity and dependency, J. Stat. Comput. Simul. 80(10) (2010), 1083-1100.
https://doi.org/10.1080/00949650902962253.
[5] C. W. Dunnett, A multiple comparison procedure for comparing several treatments with a control, J. Amer. Statist. Assoc. 50(272) (1955), 1096-1121.
[6] I. Einot and K. R. Gabriel, A study of the powers of several methods of multiple comparisons, J. Amer. Statist. Assoc. 70(351a) (1975), 574-583. https://doi.org/10.1080/01621459.1975.10482474.
[7] K. D. Gomez, C. Doetkott and R. C. Magel, A comparison of the false discovery rate with Dunnett’s test for a large number of treatments, Journal of Advance Research in Mathematics and Statistics 3(7) (2016), 20-26.
https://doi.org/10.53555/nnms.v3i7.548.
[8] M. J. Heise, S. K. Mon and L. C. Bowman, Utility of linear mixed effects models for event-related potential research with infants and children, Developmental Cognitive Neuroscience 54 (2022), 101070.
https://doi.org/10.1016/j.dcn.2022.101070.
[9] S. Holm, A simple sequentially rejective multiple test procedure, Scand. J. Statist. 6 (1979), 65-70. https://www.jstor.org/stable/4615733.
[10] L. A. Hothorn and M. Hasler, The Dunnett procedure with possibly heterogeneous variances, 2023. arXiv preprint arXiv:2303.09222.
https://doi.org/10.48550/arXiv.2303.09222.
[11] W. Jiang, J. Zhou and B. Liang, An improved Dunnett’s procedure for comparing multiple treatments with a control in the presence of missing observations, Mathematics 11(14) (2023), 3233. https://doi.org/10.3390/math11143233.
[12] C. T. Le and L. E. Eberly, Introductory Biostatistics, John Wiley & Sons, 2016.
[13] L. F. Petrinovich and C. D. Hardyck, Error rates for multiple comparison methods: some evidence concerning the frequency of erroneous conclusions, Psychological Bulletin 71(1) (1969), 43. https://doi.org/10.1037/h0026861.
[14] R Core Team, R: A language and environment for statistical computing, R Foundation for Statistical Computing, Vienna, Austria, 2021.
URL https://www.R-project.org/.
[15] S. A. Rusticus and C. Y. Lovato, Impact of sample size and variability on the power and type I error rates of equivalence tests: a simulation study, Practical Assessment, Research, and Evaluation 19(1) (2019), 11.
https://doi.org/10.7275/4s9m-4e81.
[16] D. Sauder, Examining the type I error and power of 18 common post-hoc comparison tests, Graduate Psychology: James Madison University, 2017. https://commons.lib.jmu.edu/cgi/viewcontent.cgi.
[17] L. E. Toothaker, Multiple Comparison Procedures, No. 89, Sage, 1993.
[18] B. T. West, K. B. Welch and A. T. Galecki, Linear Mixed Models: A Practical Guide Using Statistical Software, Chapman and Hall/CRC, 2022.