Indian Journal of Science and Technology

Abstract

Background/Objectives:Inthis paper,there are real-time stream &big data stream applications running ona large amount of clusters & live migration of data using intensive & interactive tasks. Methods/Statistical Analysis: Traditionally, there are many stream processing engines which are going to manage the processing data in this streaming scenario, ongoing stream processing the data must be handle due to faults & stragglers within a second-scale latency. At present, there are efficient stream processing engines which can avoid these faults & stragglers by using the stream processing engines like APACHE SPARK & APACHE FLINK. Findings: In the real-time data stream processing there are some problem occurrences while processing the data, those are data latency, fault-tolerance, mutable data-sets and more stragglers. To avoid all these issues there is an efficient stream processing engines with additional added mechanisms like dolly retreat mechanism for avoiding stragglers and process data efficiently. Application/Improvements: Now a day’s demand for stream processing is increasing a lot which may be real-time streaming or normal cluster data transformation. Data has to be processed fast, so that companies are integrating these stream processing engines to changing their business conditions effectively in realtime for analyze their data more effectively with help of Fast Stream Processing Engines.

Keywords: Fault-Tolerance, RDD, Second-Scale Latency and Immutable Datasets, Stragglers