Analyzing HPC Job Wait Times under Resource Scaling Using Historical Workload Data

Understanding the impact of hardware configuration, infrastructure investments, and operational policies on job wait times in high-performance computing (HPC) systems is a challenging problem primarily due to the lack of effective tools that are built on controlled, real-world observations across different system configurations. In this work, we present a data-driven framework that leverages historical job traces to estimate the impact of resource modifications on queue performance. Using job data collected from multiple HPC systems during periods with no hardware changes, we construct a synthetic scheduler that replays real job submission patterns and resource requests, while using ground-truth runtimes to model execution. This enables controlled, counterfactual evaluation of infrastructure changes without modifying production systems. We perform experiments by scaling system resources, including GPUs and CPU cores. Our results on GPU-based systems show that increasing GPU capacity leads to significant reductions in wait times, while CPU-only scaling provides minimal benefit. To quantify these effects, we introduce the Weighted Wait-Time Score (WWS), a bounded metric that captures both typical and tail wait-time behavior. Our experiments show that informed resource scaling improves WWS (up to 92% relative gain in WWS for low-baseline systems), capturing gains in both typical and tail wait-time behavior under this metric. We further formulate a cost-aware optimization framework to guide resource allocation under budget constraints. Our approach offers a data-driven way to evaluate HPC upgrades and supports future predictive optimization and better resource use. Overall, this work provides (i) a trace-driven framework for evaluating resource scaling effects, (ii) a bounded metric (WWS) for comparing wait-time performance, and (iii) a pathway to optimal user wait time optimization based resource allocation in HPC systems.