What Time Taught Us: Monitoring a Computing Technology Testbed Across Multiple Years

Testbeds for computing technologies play a crucial role in evaluating novel architectures and optimizing the user experience is critical to enabling meaningful and efficient use and testing of the platform. In this paper, we share our experience with Ookami, a computing technology testbed that has been operational for over four years, serving more than 500 users. Due to the novel nature of its technology and architecture, typical usage patterns differ significantly from those observed in conventional high-performance computing (HPC) systems. To monitor system performance and user interactions, we implemented a tracking approach that helps us assess project progress, identify beneficial and non-beneficial decisions, and continuously refine both system usage and user experience. Based on our findings, we provide recommendations for other testbed centers on key metrics to monitor and how to leverage collected data for system and user optimization. Additionally, we highlight aspects that in hindsight, would have been helpful, offering insights for future testbed development. This work serves as a practical guide for institutions managing similar testbeds, helping them make informed decisions on data collection, analysis, and operational improvements.