Predictive quantification of surrogate model fidelity based on modal variations with sample density

It is generally challenging to quantify the fidelity of surrogate models without additional system evaluations. Standard error measures, such as the mean squared error and cross-validation error, often do not adequately capture the fidelity of the model trained using all available sample points. This paper introduces a new model-independent approach to quantify surrogate model fidelity, called Predictive Estimation of Model Fidelity (PEMF). In PEMF, intermediate surrogates are iteratively constructed over heuristic subsets of sample points. The median and the maximum errors estimated over the remaining points are used to determine the respective error distributions at each iteration. The estimated modes of the error distributions are represented as functions of the density of intermediate training points through nonlinear regression, assuming a smooth decreasing trend of errors with increasing sample density. These regression functions are then used to predict the expected median and maximum errors in the final surrogate model (trained using all available sample points). A Monotonic Trend criterion is defined to statistically test if the regression function is reasonably reliable in predicting the model fidelity, failing which a stable implementation of k-fold cross-validation (based on modal error) is used to predict the final surrogate error. To compare the accuracy and robustness of PEMF with that of the popular leave-one-out cross-validation, numerical experiments are performed using Kriging, RBF, and E-RBF models. It is observed that the model fidelities estimated by PEMF is up to two orders of magnitude more accurate and statistically more stable compared to those based on cross-validation.