Decision-Making Under Uncertainty for Multi-stage Pipelines: Simulation Studies to Benchmark Screening Strategies

Multi-stage screening pipelines are ubiquitous throughout experimental and computational science. Much of the effort in developing screening pipelines focuses on improving generative methods or surrogate models in an attempt to make each screening step effective for a specific application. Little focus has been placed on characterizing a generic screening pipeline’s performance with respect to the problem or problem parameters. Here, we develop methods to codify and simulate features and properties about the screening procedure in general. We outline and model common problem settings and identify potential opportunities to perform decision-making under uncertainty for optimizing the execution of screening pipelines. We then illustrate the developed methods through several simulation studies. We finally show how such studies can provide a quantification of the screening pipeline performance with respect to problem parameters, specifically identifying the significance of stage-wise covariance structure. We show how such structure can lead to qualitatively different screening behaviors and how screening can even perform worse than random in some cases.