Choosing Between Remote I/O versus Staging in Large Scale Distributed Applications

Remote I/O and data staging are the most widely used data access methods for large scale distributed applications with non-local data sources. Application developers generally choose one method over the other intuitively without much scientific analysis specific to their applications since there is no generic model available that they can use for this purpose. Our goal is to develop such a model and to set guidelines for the application developers which will help them to choose the most appropriate data access method for their application. We define the parameters that potentially affect the end-to-end performance of the distributed applications which need to use remote and distributed data. We have developed a synthetic application in which we can compare the effects of different parameters such as input and output data size, data access pattern, and data access ratio on the performance of staging vs remote I/O. The results of our experiments with this synthetic application gives us hints on the generic model. Our preliminary results are presented in this paper.