{"id":48519,"date":"2017-12-10T22:24:35","date_gmt":"2017-12-10T16:54:35","guid":{"rendered":"https:\/\/www.whizlabs.com\/?p=48519"},"modified":"2021-01-28T08:10:53","modified_gmt":"2021-01-28T08:10:53","slug":"do-you-need-hadoop-to-run-spark","status":"publish","type":"post","link":"https:\/\/www.whizlabs.com\/blog\/do-you-need-hadoop-to-run-spark\/","title":{"rendered":"Do You Need Hadoop to Run Spark?"},"content":{"rendered":"

Hadoop<\/a> and Apache Spark<\/a> both are today\u2019s booming open-source Big data frameworks. Though Hadoop and Spark don\u2019t do the same thing, however, they are inter-related. The need for Hadoop is everywhere for Big data processing. However, Hadoop has a major drawback despite its many important features and benefits for data processing. MapReduce which is the native batch processing engine of Hadoop is not as fast as Spark. <\/span><\/p>\n

And that\u2019s where Spark takes an edge over Hadoop. In addition to that, most of today’s big data projects demand batch workload as well as real-time data processing. Hadoop’s MapReduce isn’t cut out for it and can process only batch data. Furthermore, when it is time to low latency processing of a large amount of data, MapReduce fails to do that. Hence, we need to run Spark on top of Hadoop. With its hybrid framework and resilient distributed dataset (Spark RDD<\/a>), data can be stored transparently in-memory while you run Spark.<\/span><\/p>\n

\"Spark<\/a><\/p>\n

But does that mean there is always a need of Hadoop to run Spark? Let\u2019s look into technical detail to justify it.<\/span><\/p>\n

\n
\n

Table of Contents<\/p>\nToggle<\/span><\/path><\/svg><\/svg><\/span><\/span><\/span><\/a><\/span><\/div>\n