| Presentation | 2013-01-25 Enhancing Hadoop Performance by Network-Aware Task Scheduling Shun KATAOKA, Hyuma WATANABE, Masatoshi KAWARASAKI, |
|---|---|
| PDF Download Page |  PDF download Page Link |
| Abstract(in Japanese) | (See Japanese page) |
| Abstract(in English) | Hadoop uses MapReduce for distributed processing of large-scale data. Although a large amount of data transfer between Mappers and Reducers may be generated in a shuffle phase of the MapReduce processing, current Hadoop task-scheduling method allocates a task without taking this into account. As a result, job processing may be delayed because of the network bottleneck. In this paper, we investigate Hadoop data flow in detail and propose a new task-scheduling method that allocates a new task based on the observed amount of data transfer at each node. Job performance in the proposed task-scheduling method is evaluated in the real machine environment. |
| Keyword(in Japanese) | (See Japanese page) |
| Keyword(in English) | Hadoop / MapReduce / task scheduling / network / job performance |
| Paper # | IN2012-152 |
| Date of Issue |
| Conference Information | |
| Committee | IN |
|---|---|
| Conference Date | 2013/1/17(1days) |
| Place (in Japanese) | (See Japanese page) |
| Place (in English) | |
| Topics (in Japanese) | (See Japanese page) |
| Topics (in English) | |
| Chair | |
| Vice Chair | |
| Secretary | |
| Assistant | |
| Paper Information | |
| Registration To | Information Networks (IN) |
|---|---|
| Language | JPN |
| Title (in Japanese) | (See Japanese page) |
| Sub Title (in Japanese) | (See Japanese page) |
| Title (in English) | Enhancing Hadoop Performance by Network-Aware Task Scheduling |
| Sub Title (in English) | |
| Keyword(1) | Hadoop |
| Keyword(2) | MapReduce |
| Keyword(3) | task scheduling |
| Keyword(4) | network |
| Keyword(5) | job performance |
| 1st Author's Name | Shun KATAOKA |
| 1st Author's Affiliation | College of Media Arts, Science and Technology, University of Tsukuba() |
| 2nd Author's Name | Hyuma WATANABE |
| 2nd Author's Affiliation | Graduate School of Library, Information and Media Studies, University of Tsukuba |
| 3rd Author's Name | Masatoshi KAWARASAKI |
| 3rd Author's Affiliation | Graduate School of Library, Information and Media Studies, University of Tsukuba |
| Date | 2013-01-25 |
| Paper # | IN2012-152 |
| Volume (vol) | vol.112 |
| Number (no) | 393 |
| Page | pp.pp.- |
| #Pages | 6 |
| Date of Issue | |