A resource supply-demand based approach for automatic MapReduce job optimization

Jinjun Xiong; Dzung T. Phan; David Kung

doi:10.1109/HPCC-CSS-ICESS.2015.130

A resource supply-demand based approach for automatic MapReduce job optimization

Jinjun Xiong
, Dzung T. Phan
, David Kung

Department of Computer Science and Engineering

IBM

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

1 Scopus citations

Abstract

With the prevalence of big data, MapReduce has emerged as the most widely deployed computing framework for data analysts. This paper addresses MapReduce job performance optimization, targeting system latency reduction. We design a systematic method to optimize MapReduce job execution process by maximizing the utilization of computing resources. Through careful analysis of the mechanism behind Hadoop, the map-shuffle-reduce work-flow is formalized based on the resource supply-demand relations. Efficient and effective algorithms are developed to address the optimization using mixed integer nonlinear programming. Experiments on a ten-node cluster demonstrate that the proposed model achieves consistently improved performance, and significantly outperforms the system with default parameter setting.

Original language	English
Title of host publication	Proceedings - 2015 IEEE 17th International Conference on High Performance Computing and Communications, 2015 IEEE 7th International Symposium on Cyberspace Safety and Security and 2015 IEEE 12th International Conference on Embedded Software and Systems, HPCC-CSS-ICESS 2015
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	740-745
Number of pages	6
ISBN (Electronic)	9781479989362
DOIs	https://doi.org/10.1109/HPCC-CSS-ICESS.2015.130
State	Published - Nov 23 2015
Event	17th IEEE International Conference on High Performance Computing and Communications, IEEE 7th International Symposium on Cyberspace Safety and Security and IEEE 12th International Conference on Embedded Software and Systems, HPCC-ICESS-CSS 2015 - New York, United States Duration: Aug 24 2015 → Aug 26 2015

Publication series

Name	Proceedings - 2015 IEEE 17th International Conference on High Performance Computing and Communications, 2015 IEEE 7th International Symposium on Cyberspace Safety and Security and 2015 IEEE 12th International Conference on Embedded Software and Systems, HPCC-CSS-ICESS 2015

Conference

Conference	17th IEEE International Conference on High Performance Computing and Communications, IEEE 7th International Symposium on Cyberspace Safety and Security and IEEE 12th International Conference on Embedded Software and Systems, HPCC-ICESS-CSS 2015
Country/Territory	United States
City	New York
Period	08/24/15 → 08/26/15

Keywords

Big data
MapReduce
Performance tuning

Access to Document

10.1109/HPCC-CSS-ICESS.2015.130

Cite this

Xiong, J., Phan, D. T., & Kung, D. (2015). A resource supply-demand based approach for automatic MapReduce job optimization. In Proceedings - 2015 IEEE 17th International Conference on High Performance Computing and Communications, 2015 IEEE 7th International Symposium on Cyberspace Safety and Security and 2015 IEEE 12th International Conference on Embedded Software and Systems, HPCC-CSS-ICESS 2015 (pp. 740-745). Article 7336246 (Proceedings - 2015 IEEE 17th International Conference on High Performance Computing and Communications, 2015 IEEE 7th International Symposium on Cyberspace Safety and Security and 2015 IEEE 12th International Conference on Embedded Software and Systems, HPCC-CSS-ICESS 2015). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/HPCC-CSS-ICESS.2015.130

Xiong, Jinjun ; Phan, Dzung T. ; Kung, David. / A resource supply-demand based approach for automatic MapReduce job optimization. Proceedings - 2015 IEEE 17th International Conference on High Performance Computing and Communications, 2015 IEEE 7th International Symposium on Cyberspace Safety and Security and 2015 IEEE 12th International Conference on Embedded Software and Systems, HPCC-CSS-ICESS 2015. Institute of Electrical and Electronics Engineers Inc., 2015. pp. 740-745 (Proceedings - 2015 IEEE 17th International Conference on High Performance Computing and Communications, 2015 IEEE 7th International Symposium on Cyberspace Safety and Security and 2015 IEEE 12th International Conference on Embedded Software and Systems, HPCC-CSS-ICESS 2015).

@inproceedings{cb22930834d545dca47a784ee4a6f75f,

title = "A resource supply-demand based approach for automatic MapReduce job optimization",

abstract = "With the prevalence of big data, MapReduce has emerged as the most widely deployed computing framework for data analysts. This paper addresses MapReduce job performance optimization, targeting system latency reduction. We design a systematic method to optimize MapReduce job execution process by maximizing the utilization of computing resources. Through careful analysis of the mechanism behind Hadoop, the map-shuffle-reduce work-flow is formalized based on the resource supply-demand relations. Efficient and effective algorithms are developed to address the optimization using mixed integer nonlinear programming. Experiments on a ten-node cluster demonstrate that the proposed model achieves consistently improved performance, and significantly outperforms the system with default parameter setting.",

keywords = "Big data, MapReduce, Performance tuning",

author = "Jinjun Xiong and Phan, \{Dzung T.\} and David Kung",

note = "Publisher Copyright: {\textcopyright} 2015 IEEE.; 17th IEEE International Conference on High Performance Computing and Communications, IEEE 7th International Symposium on Cyberspace Safety and Security and IEEE 12th International Conference on Embedded Software and Systems, HPCC-ICESS-CSS 2015 ; Conference date: 24-08-2015 Through 26-08-2015",

year = "2015",

month = nov,

day = "23",

doi = "10.1109/HPCC-CSS-ICESS.2015.130",

language = "English",

series = "Proceedings - 2015 IEEE 17th International Conference on High Performance Computing and Communications, 2015 IEEE 7th International Symposium on Cyberspace Safety and Security and 2015 IEEE 12th International Conference on Embedded Software and Systems, HPCC-CSS-ICESS 2015",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "740--745",

booktitle = "Proceedings - 2015 IEEE 17th International Conference on High Performance Computing and Communications, 2015 IEEE 7th International Symposium on Cyberspace Safety and Security and 2015 IEEE 12th International Conference on Embedded Software and Systems, HPCC-CSS-ICESS 2015",

address = "United States",

}

Xiong, J, Phan, DT & Kung, D 2015, A resource supply-demand based approach for automatic MapReduce job optimization. in Proceedings - 2015 IEEE 17th International Conference on High Performance Computing and Communications, 2015 IEEE 7th International Symposium on Cyberspace Safety and Security and 2015 IEEE 12th International Conference on Embedded Software and Systems, HPCC-CSS-ICESS 2015., 7336246, Proceedings - 2015 IEEE 17th International Conference on High Performance Computing and Communications, 2015 IEEE 7th International Symposium on Cyberspace Safety and Security and 2015 IEEE 12th International Conference on Embedded Software and Systems, HPCC-CSS-ICESS 2015, Institute of Electrical and Electronics Engineers Inc., pp. 740-745, 17th IEEE International Conference on High Performance Computing and Communications, IEEE 7th International Symposium on Cyberspace Safety and Security and IEEE 12th International Conference on Embedded Software and Systems, HPCC-ICESS-CSS 2015, New York, United States, 08/24/15. https://doi.org/10.1109/HPCC-CSS-ICESS.2015.130

A resource supply-demand based approach for automatic MapReduce job optimization. / Xiong, Jinjun; Phan, Dzung T.; Kung, David.
Proceedings - 2015 IEEE 17th International Conference on High Performance Computing and Communications, 2015 IEEE 7th International Symposium on Cyberspace Safety and Security and 2015 IEEE 12th International Conference on Embedded Software and Systems, HPCC-CSS-ICESS 2015. Institute of Electrical and Electronics Engineers Inc., 2015. p. 740-745 7336246 (Proceedings - 2015 IEEE 17th International Conference on High Performance Computing and Communications, 2015 IEEE 7th International Symposium on Cyberspace Safety and Security and 2015 IEEE 12th International Conference on Embedded Software and Systems, HPCC-CSS-ICESS 2015).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - A resource supply-demand based approach for automatic MapReduce job optimization

AU - Xiong, Jinjun

AU - Phan, Dzung T.

AU - Kung, David

PY - 2015/11/23

Y1 - 2015/11/23

N2 - With the prevalence of big data, MapReduce has emerged as the most widely deployed computing framework for data analysts. This paper addresses MapReduce job performance optimization, targeting system latency reduction. We design a systematic method to optimize MapReduce job execution process by maximizing the utilization of computing resources. Through careful analysis of the mechanism behind Hadoop, the map-shuffle-reduce work-flow is formalized based on the resource supply-demand relations. Efficient and effective algorithms are developed to address the optimization using mixed integer nonlinear programming. Experiments on a ten-node cluster demonstrate that the proposed model achieves consistently improved performance, and significantly outperforms the system with default parameter setting.

AB - With the prevalence of big data, MapReduce has emerged as the most widely deployed computing framework for data analysts. This paper addresses MapReduce job performance optimization, targeting system latency reduction. We design a systematic method to optimize MapReduce job execution process by maximizing the utilization of computing resources. Through careful analysis of the mechanism behind Hadoop, the map-shuffle-reduce work-flow is formalized based on the resource supply-demand relations. Efficient and effective algorithms are developed to address the optimization using mixed integer nonlinear programming. Experiments on a ten-node cluster demonstrate that the proposed model achieves consistently improved performance, and significantly outperforms the system with default parameter setting.

KW - Big data

KW - MapReduce

KW - Performance tuning

UR - https://www.scopus.com/pages/publications/84961744661

U2 - 10.1109/HPCC-CSS-ICESS.2015.130

DO - 10.1109/HPCC-CSS-ICESS.2015.130

M3 - Conference contribution

AN - SCOPUS:84961744661

T3 - Proceedings - 2015 IEEE 17th International Conference on High Performance Computing and Communications, 2015 IEEE 7th International Symposium on Cyberspace Safety and Security and 2015 IEEE 12th International Conference on Embedded Software and Systems, HPCC-CSS-ICESS 2015

SP - 740

EP - 745

BT - Proceedings - 2015 IEEE 17th International Conference on High Performance Computing and Communications, 2015 IEEE 7th International Symposium on Cyberspace Safety and Security and 2015 IEEE 12th International Conference on Embedded Software and Systems, HPCC-CSS-ICESS 2015

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 17th IEEE International Conference on High Performance Computing and Communications, IEEE 7th International Symposium on Cyberspace Safety and Security and IEEE 12th International Conference on Embedded Software and Systems, HPCC-ICESS-CSS 2015

Y2 - 24 August 2015 through 26 August 2015

ER -

Xiong J, Phan DT, Kung D. A resource supply-demand based approach for automatic MapReduce job optimization. In Proceedings - 2015 IEEE 17th International Conference on High Performance Computing and Communications, 2015 IEEE 7th International Symposium on Cyberspace Safety and Security and 2015 IEEE 12th International Conference on Embedded Software and Systems, HPCC-CSS-ICESS 2015. Institute of Electrical and Electronics Engineers Inc. 2015. p. 740-745. 7336246. (Proceedings - 2015 IEEE 17th International Conference on High Performance Computing and Communications, 2015 IEEE 7th International Symposium on Cyberspace Safety and Security and 2015 IEEE 12th International Conference on Embedded Software and Systems, HPCC-CSS-ICESS 2015). doi: 10.1109/HPCC-CSS-ICESS.2015.130

A resource supply-demand based approach for automatic MapReduce job optimization

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this