A Survey on Utilization of Hadoop Framework with Machine Learning

Abstract

We are inundated with data from sensors, mobile devices, social media, commerce, and internet sources, among others. This enormous amount of data is accelerating its growth as the internet, e-commerce, and technology, most especially the Internet of Things, develop (IoT). The Internet of Things (IoT) is a phrase that refers to the process through which computers, smart gadgets, and other data-generating devices are connected via a network in order to transmit data. Thus, data is continuously generated and updated to reflect changes in all human areas and activities. This exponential increase in data has resulted in the coining of a new term and concept known as big data. Big data is needed to shed light on the connections between items, predict future trends, and give decision makers more information. However, the main issue at the moment is how to efficiently gather and analyze enormous quantities of varied and complex data. Machine learning techniques are the most commonly used methods for understanding and analyzing data and acquiring critical information in certain industries or applications. Traditional machine learning techniques cannot effectively address big data challenges on their own. This article provides an overview of the Hadoop and MapReduce framework as a platform that can be used by machine learning techniques to solve the concerns that have emerged about the design and implementation of big data systems. This article focuses on three types of machine learning techniques and their utilization on top of Hadoop framework: supervised machine learning regression, supervised machine learning classification and unsupervised machine learning clustering.