Preemptive Hadoop jobs scheduling under a deadline

Li LIU, Yuan ZHOU, Ming LIU, Guandong XU, Xiwei CHEN, Dangping FAN, Qianru WANG

Research output: Chapter in Book/Report/Conference proceedingChapters

24 Citations (Scopus)

Abstract

MapReduce has become the dominant programming model in a cloud-based data processing environment, such as Hadoop. First In First Out (FIFO) is the default job scheduling policy of Hadoop, but it cannot guarantee that the job will be completed by a specific deadline. Research has been focused on developing deadline-based MapReduce schedulers by using the non-preemptive scheduling approach. However, compared with the non-preemptive scheduling approach, the preemptive scheduling approach has some advantages, such as the total completion time and slot utilization. In this paper, we first formulated the preemptive scheduling problem under deadline constraint, and then we proposed preemptive scheduling algorithms. To our knowledge we implemented the first real preemptive job scheduler to meet deadlines on Hadoop. The experimental results indicate that the preemptive scheduling approach is promising, which is more efficient than the non-preemptive one for executing jobs under a certain deadline. Copyright © 2012 IEEE.


Original languageEnglish
Title of host publicationProceedings of 2012 Eighth International Conference on Semantics, Knowledge and Grids, SKG 2012
Place of PublicationDanvers, MA
PublisherIEEE
Pages72-79
ISBN (Print)9780769547947
DOIs
Publication statusPublished - 2012

Citation

Liu, L., Zhou, Y., Liu, M., Xu, G., Chen, X., Fan, D., & Wang, Q. (2012). Preemptive Hadoop jobs scheduling under a deadline. In Proceedings of 2012 Eighth International Conference on Semantics, Knowledge and Grids, SKG 2012 (pp. 72-79), IEEE. https://doi.org/10.1109/SKG.2012.40

Fingerprint

Dive into the research topics of 'Preemptive Hadoop jobs scheduling under a deadline'. Together they form a unique fingerprint.