A practice-oriented guide to statistical inference in linear modeling for non-normal or heteroskedastic error distributions

Selecting an appropriate statistical method is a challenge frequently encountered by applied researchers, especially if assumptions for classical, parametric approaches are violated. To provide some guidelines and support, we compared classical hypothesis tests with their typical distributional assumptions of normality and homoskedasticity with common and easily accessible alternative inference methods (HC3, HC4, and six bootstrap methods) in the framework of ordinary least squares (OLS) regression. The method’s performances were assessed for four different regression models with varying levels of non-normality and heteroskedasticity of errors and five different sample sizes ranging from 25 to 500 cases. For each scenario 10,000 samples of observations were generated. Type I error and coverage rates, power, and standard error bias were examined to assess the methods’ performance. No method considered here performed satisfactorily on all accounts. Using HC3 or HC4 standard errors, or a wild bootstrap procedure with percentile confidence intervals, could yield reliable results in many, but not all, scenarios. We suppose that, in the case of assumption violations, researchers might refer to a method that performed best in a scenario most similar to their data situation. To aid selection of an appropriate method we provide tables comparing relative performances in all considered scenarios.