While considering the multi-dimensional arrays as an example, there is a method that is capable of finding the common elements present within a multi-dimensional array - intersection_update(). This method is used in order to find the common or intersecting elements present within the same array which is multi-dimensional in nature. Let us consider an input output scenario and then proceed with a program. Input Output Scenarios Consider a 2D array which is multi-dimensional in nature. arr = [[1, 2, 3, 4], [3, 4, 5, 6], [7, 8, 3, 4], [4, 9, 8, 3], [4, 3, 10, 12]] The ... Read More

What is Concatenation of Arrays? The process of combining the arrays into a single array or merging the arrays into a single array is known as Concatenation of arrays. This mechanism can be done in many ways using several techniques. Let us discuss all techniques that help in concatenation of arrays in Python. Input Output Scenarios Consider three arrays to perform concatenation. arr1 = [“ Hello ”, “ World ”, “ My ”, “ Name ”, “ is ”, “ Python ”] arr2 = [“ Hello ”, “ World ”, “ My ”, “ Name ”, “ is ”, “ ... Read More

There are several techniques that helps us to check whether the given arrays are equal or not. The comparison of an array will not depend on the indices of the elements, it will only compare whether that particular element in one array is present in the other array or not. Let us discuss few techniques that compares two arrays and checks whether they are equal or not. There are several techniques that helps us to check whether the given arrays are equal or not. The comparison of an array will not depend on the indices of the elements, it will ... Read More

Applying conditions on a data frame can be very beneficial for a programmer. We can validate data to make sure that it fits our model. We can manipulate the data frame by applying conditions and filter out irrelevant data from the data frame which improves data visualization. In this article, we will perform a similar operation of applying conditions to a PySpark data frame and dropping rows from it. Pyspark offers real time data processing. It is an API of Apache spark which allows the programmer to create spark frameworks in a local python environment. Example Now that we ... Read More

In this article, we will discuss the different methods to drop rows from a data frame base on a one or multiple conditions. These conditions will be applied on the columns and the rows will be dropped accordingly. We will use pandas to create a data frame as it offers multiple functions to manipulate the data frame. We will also create a dataset which will act as a reference for the data frame although it is not mandatory to create one, we can also use a CSV file or any other document. Pandas support multiple file types including: “CSV”, ... Read More

A dataset consists of a wide variety of values. These values can be a “string”, “integer”, “decimal” “Boolean” or even a “data structure”. These datasets are extremely valuable and can be used in various purposes. We can train model, interpret results, produce a hypothesis and build applications with the help a dataset. However, sometimes a dataset can contain values that are not necessary for our purpose. These values are called “NaN” (not a number). In this article, we will be dealing with these “NaN” or missing values. Our objective is to drop to those rows that contain any ... Read More

When we are dealing with complex datasets, we require frameworks that can process data quickly and provide results. This is where PySpark comes into the picture. PySpark is a tool which was developed by the Apache community to process data in real time. It is an API which is used to create data frames and interpret results in our local python environment. The data frame can contain huge amount of information/data and in order to maintain the relevance of the data to be interpreted we make the required changes. In this article, we will manipulate a PySpark data frame ... Read More

The PySpark data frame is a powerful, real time data processing framework which was developed by the Apache Spark developers. Spark was originally written in “scala” programming language and in order to increase its reach and flexibility, several APIs were built. These APIs provided an interface which can be used to run spark applications on our local environment. One such API is known as PySpark which was developed for the python environment. The PySpark data frame also consists of rows and columns but the processing part is different as it uses in-system (RAM) computational techniques for processing the data. ... Read More

Pandas data frame is a very powerful data manipulation tool. It is a tabular data structure consisting of rows and columns. The size of this 2-D matrix can be variable depending upon the complexity of the dataset. We can use different type of sources to create a data frame ranging from databases to files. The columns in a pandas data frame represents a series of information and it can be an integer, float, or string. We can perform numerous operations on these columns including deletion, indexing, filtering etc. In this article, we will perform one such basic operation of ... Read More

PySpark is a tool designed by the Apache spark community to process data in real time and analyse the results in a local python environment. Spark data frames are different from other data frames as it distributes the information and follows a schema. Spark can handle stream processing as well as batch processing and this is the reason for their popularity. A PySpark data frame requires a session in order to generate an entry point and it performs on-system processing of the data (RAM). You can install PySpark module on windows using the following command – pip install pyspark ... Read More