Apache Spark is one of the tools that you will have to learn whether you want to start implementing something with Big Data. Apache spark is also a very useful tool for other more sophisticated Big Data tools because it is integrated into other big data tools. For example you can choose Apache spark with Hadoop, Azure, AWS or other type of framework in which you can great a large amount of data. Thus. it is good for you to understand how much are Apache spark works and the different functions and functionality that you those Big Data Frameworks.
An easy way to understand Apache spark is by understanding how the data is manipulated from a Repository and delivered it into a more purified or filtered manner. the best way to understand this is by visualizing that you have data on one side. let’s suppose it is on the left side and you want to modify the data into a desired form which will be displayed on the right side. so the question here is what should I do to modify the data so that I can get the form on the right side.
The answer is simple and you are operations. These are the Apache spark operations that you will have to rely on so that you can modify the data, deliver the right amount data and design the right structure of data.
These two types of Apache spark operations are Transformations and actions.
Just park transformations are the most relevant way to manipulate daytime from the left side and transform it into our desired instructor on the right side. Transformations will allow you to give the right shape and quantity of your data. You can understand this by identifying that you will not be able to organize your data into a desired structure as transformations are functions that manipulate large volume of data.
There are two types of transformations which are considered lazy functions because they are not applied until actions are called in Apache Spark.
Narrow and wide transformations are operations that you will use to sort large amount of data. both of those Transformations use RDD ( resilient distributed dataset) which are immutable in nature and once you receive an RDD to sort data you will render one or more new RDDs after applying any type of transformation.
Transformations are characterized for having all the elements in a single partition coming from the same single parent RDD. the most common functions that are considered narrow transformations are map, filter, flatmap, Sample, Union and MapPartition.
Wide transformations are characterized my hopping all the elements in a single partition multiple parents RDDs. This type of transformations are section, distinct, reducedbykey, join, Cartesian, Groupbykey.
Once you have the desired structure of your data, you will be able to perform different actions so that you will be calling the action operations in Apache Spark. tell us the sequence manipulate your data will be first apply transformations and after that apply action operations.
Action operations render non-RDD structures and provide at a specific value obtained from the data that you are working with. They are used with executors in different clusters so that you are able to perform tasks in two different nodes of the clusters. action operators mainly work with executors to send data to the driver in cluster structure. function operations are count, fold, collect, aggregate, take, foreach and top.
As you can notice action operations are useful functions in which you can rely on to put into motion your data.
Apache spark is a little more complicated, however, you will be able to start working with Apache spark right away by understanding how to manipulate data with transformations and actions.
Learn more about these operations in the links below so that you can understand how each of these Transformations and actions work. you also will know about how these functions are applied and how they are used in combination with other different types of Transformations and actions.