EAGER: Stork Data Scheduler for Azure

This project will further develop and enhance the Stork Data Scheduler to support Azure cloud computing environment, and to mitigate the end-to-end data handling bottleneck in data-intensive cloud computing applications. Stork data scheduler has been very actively used in many data-intensive application areas including coastal hazard prediction and storm surge modeling; oil flow and reservoir uncertainty analysis; numerical relativity and black hole collisions; educational video processing and behavioral assessment; digital sky imaging; and multiscale computational fluid dynamics. Making Stork available on the Azure environment will enable this already existing user base to easily migrate to the Azure Cloud Computing platform as well as other Azure application groups benefiting from the large-scale data handling capabilities of Stork. The Stork Data Scheduler for Azure will make a distinctive contribution to cloud computing community because it focuses on planning, scheduling, monitoring and management of data placement tasks and application-level end-to-end optimization of networked I/O for petascale data-intensive applications. Unlike existing approaches, it will treat data resources and the tasks related to data access and movement as first class entities just like computational resources and compute tasks, and not simply the side effect of computation. Stork data scheduler for Azure will provide enhanced functionality for cloud computing such as data aggregation and connection caching; peer-to-peer and streamed data management; early error detection, classification, and recovery in data transfers; scheduled storage management; optimal protocol tuning; and end-to-end performance prediction services. The Stork data scheduler for Azure will dramatically change how domain scientists perform their research by rapidly facilitating sharing of large amounts of data in cloud computing environments.