abstract
Purpose: A metamodel is a simplified approximation of the relationships between inputs and outputs in a simulation model. A challenge in metamodeling for policy analysis is that there are often multiple correlated outputs of interest: for example, costs and effects of different policy choices. We develop a framework for metamodeling that accommodates multivariate outcomes, and we compare alternatives for the choice of the base learning algorithm.
Methods: To generate metamodels for multivariate outcomes, we combined two algorithm adaptation methods – multi-target stacking (MTS) and regression chain with maximum correlation chain (RCCC) – together with different base learners including linear regression (LR), elastic net (EN) with second-order terms, Gaussian process regression (GPR), random forests (RFs), and neural networks (NNs). We optimized the integrated models using variable selection and hyperparameter tuning. We compared approaches in terms of model accuracy, efficiency, and interpretability. To illustrate the framework, we used inputs and outputs from an individual-based decision analytic simulation model of testing and treatment strategies for chronic hepatitis C virus in correctional settings. The training data set included 2,000 simulations, each summarized by 37 input variables and 60 output variables (5 economic and health outcomes for 6 alternative strategies measured over 2 time horizons).
Results: The output variables from the simulation model were heavily correlated (average rho=0.58). Without multioutput algorithm adaptation methods, in-sample fit (measured by R2) ranged from 0.881 for LR to 0.987 for GPR. Inclusion of multioutput algorithm adaptation methods increased R2by 0.004 on average across all base leaners, and variable selection and hyperparameter tuning added 0.009. In terms of model efficiency, simpler models such as LR, EN, and RF required minimal training and prediction time. GPR had short prediction time, but the training time was heavily affected by the size of the training dataset. NN had the longest training and prediction time. For model interpretability, LR and EN have clear advantages, but we also consider methods for improving interpretability of other models.
Conclusions: This study provides a framework for metamodeling in policy analyses with multivariate outcomes of interest. While the advantages and disadvantages of specific learning algorithms may vary across different modeling applications, we expect that the general framework presented here will have broad applicability to decision analytic models in health and medicine.
