TY - GEN
T1 - Network-aware end-to-end data throughput optimization
AU - Yildirim, Esma
AU - Kosar, Tevfik
PY - 2011
Y1 - 2011
N2 - The rapidly advancing optical networking technology allows us high-bandwidth connectivity up to 100Gbps these days. However, the end-users and their applications can only observe a fraction of this available bandwidth capacity due to inefficient transport protocols and other end-system bottlenecks such as disk I/O limitations, processor speed, and NIC restrictions. In this paper, we present a novel network-aware end-to-end throughput prediction and optimization framework which provides us with the best parameter combination (i.e. parallel stream, disk, and CPU numbers) to achieve the highest end-to-end throughput between two end-systems (i.e. clusters, data centers, parallel disk systems) possible. Our experiments show that the model and algorithm we have developed enable us to achieve close-to-optimal end-to-end throughput performance with negligible sampling and prediction overhead.
AB - The rapidly advancing optical networking technology allows us high-bandwidth connectivity up to 100Gbps these days. However, the end-users and their applications can only observe a fraction of this available bandwidth capacity due to inefficient transport protocols and other end-system bottlenecks such as disk I/O limitations, processor speed, and NIC restrictions. In this paper, we present a novel network-aware end-to-end throughput prediction and optimization framework which provides us with the best parameter combination (i.e. parallel stream, disk, and CPU numbers) to achieve the highest end-to-end throughput between two end-systems (i.e. clusters, data centers, parallel disk systems) possible. Our experiments show that the model and algorithm we have developed enable us to achieve close-to-optimal end-to-end throughput performance with negligible sampling and prediction overhead.
KW - Data parallelism
KW - GridFTP
KW - Optimization
KW - Parallel TCP streams
KW - Prediction
KW - Striping
UR - https://www.scopus.com/pages/publications/84857951633
U2 - 10.1145/2110217.2110221
DO - 10.1145/2110217.2110221
M3 - Conference contribution
AN - SCOPUS:84857951633
SN - 9781450311328
T3 - NDM'11 - Proceedings of the 2011 International Workshop on Network-Aware Data Management, Co-located with SC'11
SP - 21
EP - 30
BT - NDM'11 - Proceedings of the 2011 International Workshop on Network-Aware Data Management, Co-located with SC'11
T2 - International Workshop on Network-Aware Data Management, NDM'11, Held in Conjunction with the International Conference for High Performance Computing, Networking, Storage and Analysis, SC'11
Y2 - 14 November 2011 through 14 November 2011
ER -