Indian Journal of Science and Technology

Abstract

MapReduce, a programming model, allows parallel processing of large amount of data sets where various data mining techniques are not quite useful. It’s Map and Reduce functions can be customized by the developers according to their application. This paper gives an idea of MapReduce, its advantages and disadvantages. This paper also focuses on how MapReduce is used, how map and reduce computations are customized, implemented in several scenarios such as in medical field to generate medical reports by processing large medical data sets, stream processing and workflow scheduling in multi core processors, in distributed environment, for processing distributed data sets by using pilot abstractions. We also represent how MapReduce used for deduplication of files to save disk space in data centers. MapReduce based Pre-Post (MRPre-Post) a parallel data mining algorithm is adapted in Hadoop platform to achieve scalability. MapReduce is implemented in vHadoop (Virtual Hadoop), a scalable hadoop virtual cluster to process machine learning algorithms. The scenarios discussed in this paper help developers and researchers how to customize and use MapReduce in their applications.

Keywords: Big Data, Data Management, Distribute Environment, Iterative Computation, Large Scale Data Processing, MapReduce