Low Latency Data Stream Processing on Multi-Core CPU Environments |
|
Takanori Ueda , Sayaka Akioka ,Hayato Yamana |
|
|
 Takanori Ueda |
 Sayaka Akioka |
 Hayato Yamana |
This paper addresses stream data processing from the two perspectives of system architecture and algorithm. It introduces two techniques for efficient stream data processing on multi-core processors.
First, the authors develop an optimization technique based on dynamic programming for average latency problems.
The average latency problem is defined as a problem of optimization to minimize average latency by properly assigning operators of stream processing logic to multi-core processors.
Second, the authors design an efficient technique for dynamically re-assigning operators to multi-core processors with low overhead as the input stream rate changes over time.
These two techniques are very effective and are applicable to a number of advanced Big Data applications.
The experiments show that the performance of the prototype system scales well to the number of operators and that the system achieves high efficiency by dynamically re-assigning operators to multi-core processors.
As the authors write in their conclusion, they plan to integrate the above two techniques with QueueLinker, open source software program for stream data processing systems, and to make it available on the web.
We expect the extended QueueLinker to gain widespread use and to have a significant impact on a number of applications, worldwide.
