Friday, June 1, 2018

Spark Introduction

Apache Spark Introduction

We already know that when we have a massive volume of data, It won't be efficient and cost-effective to process it on a single computer. No matter how big and powerful is the individual machine, we will surely hit a bottleneck at some point. As an alternative solution, we need a distributed computing platform. A distributed computing platform is a cluster of computers and a set of services. These services pool the resources of the cluster members and offer a mechanism to accomplish massive data processing work.
Hadoop offered a revolutionary solution to this problem. Hadoop solved this problem in two parts.
  1. HDFS - Offering a distributed storage.
  2. Map Reduce - Offering a distributed computing engine.

However, creating map-reduce programs has been a black magic. The developer community found it hard to develop and manage map-reduce programs. Many of them also criticised map-reduce for its poor performance.

No comments:

Post a Comment