A novel approach to simultaneous selection of surrogate models, constitutive kernels, and hyper-parameter values

Owing to the multitude of surrogate modeling techniques, developed in the recent years and the diverse characteristics offered by them, automated adaptive model selection approaches could be helpful in selecting the most suitable surrogate for a given problem. Surrogate selection could be performed at three different levels: (i) model type selection, (ii) basis (or kernel) function selection, and (iii) hyper-parameter selection where hyperparameters are those kernel parameters that are generally given by the users. Unlike the majority of existing model selection techniques, this paper explores the development of a method that performs selection coherently at all the three levels. In this context, the REES method is used to provide measures of the median and maximum errors of a candidate surrogate model. Two approaches are used for the 3-level selection; (i) A Cascaded approach performs each level in a nested loop in the order going from model-kernel-hyperparameters; (ii) A more advanced One-Step approach solves a MINLP to simultaneously optimize the model, kernel, and hyper-parameters. In both approaches, multiobjective optimization is performed to yield the best trade-offs between the estimated median and maximum errors. Candidate surrogates that are considered include (i) Kriging, (ii) Radial Basis Function (RBF), and (iii) Support Vector Regression (SVR), and multiple candidate kernels are allowed within these surrogate models. The 3-level REES-based model selection is compared with model selection based on error estimated on a large set of additional test points, for validation purposes. Numerical experiments on a 2-variable and 6-variable test problems, and wind farm power generation problem, show that the proposed approach provides unique flexibility in model selection and is also reasonably accurate when compared with selection based on errors estimated on additional test points.