Apache Spark Introduction
We already know that when we have a massive volume of data, It won't be efficient and cost-effective to process it on a single computer. No matter how big and powerful is the individual machine, we will surely hit a bottleneck at some point. As an alternative solution, we need a distributed computing platform. A distributed computing platform is a cluster of computers and a set of services. These services pool the resources of the cluster members and offer a mechanism to accomplish massive data processing work.
Hadoop offered a revolutionary solution to this problem. Hadoop solved this problem in two parts.
Hadoop offered a revolutionary solution to this problem. Hadoop solved this problem in two parts.
-
HDFS - Offering a distributed storage.
-
Map Reduce - Offering a distributed computing engine.
However, creating map-reduce programs has been a black magic. The developer community found it hard to develop and manage map-reduce programs. Many of them also criticised map-reduce for its poor performance.
No comments:
Post a Comment