BIG DATA ANALYSIS ON YOUTUBE USING HADOOP and MAPREDUCE
Soma Hota, , , ,
Affiliations Amity School of Engineering and Technology - Computer Science Engineering, Amity University, Mumbai - Pune
We live in a digitalized world today. An enormous amount of data is generated from every digital service we use. This enormous amount of generated data is called Big Data. According to Wikipedia, Big data is a term for data sets that are so large or complex that traditional data processing application software is inadequate to deal with them .Big data challenges include capturing data, data storage, data analysis, search, sharing, transfer, visualization, querying, updating and information privacy. Google‘s video streaming services, YouTube, is one of the best examples of services which produces a huge quantity of data in a very short period. Data mining of such an enormous quantity of data is performed using Hadoop and MapReduce to measure performance. Hadoop is a system which provides a reliable shared storage of such huge datasets on the cloud and also provides an analysis system. The storage is provided by HDFS (Hadoop Distributed File System) and analysis by MapReduce. MapReduce is a programming model and an associated implementation for processing large data sets. This paper presents the algorithmic work on big data problem and its optimal solution using Hadoop cluster and HDFS for YouTube dataset storage and using parallel processing to process large data sets using Map Reduce programming framework. In this paper, we solve two problem statements using the YouTube dataset – top 5 video categories (genres) with the maximum number of videos uploaded and top 5 video uploaders on YouTube. A particularly distinguishing feature of this paper is its focus on analytics performed in unstructured data, which constitute 95% of big data.
Soma Hota (2018). BIG DATA ANALYSIS ON YOUTUBE USING HADOOP and MAPREDUCE. International Journal of Computer Engineering In Research Trends, 5(4), 98-104. Retrieved from http://ijcert.org/ems/ijcert_papers/V5I403.pdf
Keywords : Big Data definition, Data mining, YouTube data analysis, Hadoop, HDFS, MapReduce, unstructured dataset analysis.
1. Webster, John. "MapReduce: Simplified Data Processing on Large Clusters", "Search Storage", 2004. Retrieved on 25 March 2013. https://static.googleusercontent.com/media/research.g oogle.com/en//archive/mapreduce-osdi04.pdf
2. Bibliography: Big Data Analytics: Methods and Applications by Saumyadipta Pyne, B.L.S. Prakasa Rao, S.B. Rao
3. YOUTUBE COMPANY STATISTICS. https://www.statisticbrain.com/youtube-statistics/
4. Youtube.com @2017. YouTube for media. https://www.youtube.com/yt/about/press/
5. Big data;Wikipedia https://en.wikipedia.org/wiki/Big_data
6. Kallerhoff, Phillip. ―Big Data and Credit Unions: Machine Learning in Member Transactions https://filene.org/assets/pdfreports/301_Kallerhoff_M achine_Learning.pdf
7. Marr,Barnard.―Why only one of the 5 Vs of big data really matters http://www.ibmbigdatahub.com/blog/why-only-one-5-vs- big-data-really-matters
8. Resources Management Association (IRMA). 2016. Information. "Chapter 1 - Big Data Overview". Big Data: Concepts, Methodologies, Tools, and Applications, Volume I. IGI Global. http://common.books24x7.com/toc.aspx?bookid=114 046
9. Apache Hadoop
11. How To Analyze Big Data With Hadoop Technologies; 3pillarglobal.com. 2017 https://www.3pillarglobal.com/insights/analyze-big-data-hadoop-technologies Dean, S. Ghemawat, MapReduce: Simplified Data Processing on Large Clusters, in:
12. OSDI‘04, 6th Symposium on Operating Systems
13. Design and Implementation, Sponsored by USENIX, in cooperation with ACM SIGOPS, 2004, pp. 137– 150
14. Big Data Tutorial1:MapReduce https://wikis.nyu.edu/display/NYUHPC/Big+Data+T utorial+1%3A+MapReduce
15. MacLean,Diana.‖A Very Brief Introduction to MapReduce http://hci.stanford.edu/courses/cs448g/a2/files/map_r educe_tutorial.pdf
16. Edureka.‘Install Hadoo p:Setting up a single node cluster‘. https://www.edureka.co/blog/install-hadoop-single-node-hadoop-cluster
We have kept IJCERT is a free peer-reviewed scientific journal to endorse conservation. We have not put up a paywall to readers, and we do not charge for publishing. But running a monthly journal costs is a lot. While we do have some associates, we still need support to keep the journal flourishing. If our readers help fund it, our future will be more secure.