Applying conditions on a data frame can be very beneficial for a programmer. We can validate data to make sure that it fits our model. We can manipulate the data frame by applying conditions and filter out irrelevant data from the data frame which improves data visualization. In this article, we will perform a similar operation of applying conditions to a PySpark data frame and dropping rows from it. Pyspark offers real time data processing. It is an API of Apache spark which allows the programmer to create spark frameworks in a local python environment. Example Now that we ... Read More

In this article, we will discuss the different methods to drop rows from a data frame base on a one or multiple conditions. These conditions will be applied on the columns and the rows will be dropped accordingly. We will use pandas to create a data frame as it offers multiple functions to manipulate the data frame. We will also create a dataset which will act as a reference for the data frame although it is not mandatory to create one, we can also use a CSV file or any other document. Pandas support multiple file types including: “CSV”, ... Read More

A dataset consists of a wide variety of values. These values can be a “string”, “integer”, “decimal” “Boolean” or even a “data structure”. These datasets are extremely valuable and can be used in various purposes. We can train model, interpret results, produce a hypothesis and build applications with the help a dataset. However, sometimes a dataset can contain values that are not necessary for our purpose. These values are called “NaN” (not a number). In this article, we will be dealing with these “NaN” or missing values. Our objective is to drop to those rows that contain any ... Read More

Generating random numbers is a common requirement in many programming applications. Golang offers built-in functionality to generate random numbers of different types. In this article, we will discuss how to generate random Int type numbers in Golang. What is Int Type? Int is a data type in Golang that represents an integer number. The size of the Int type depends on the architecture of the computer and can be either 32 or 64 bits. Int types are commonly used in programming applications for counting and indexing. Generating Random Int Type Numbers in Golang To generate a random Int type number ... Read More

When we are dealing with complex datasets, we require frameworks that can process data quickly and provide results. This is where PySpark comes into the picture. PySpark is a tool which was developed by the Apache community to process data in real time. It is an API which is used to create data frames and interpret results in our local python environment. The data frame can contain huge amount of information/data and in order to maintain the relevance of the data to be interpreted we make the required changes. In this article, we will manipulate a PySpark data frame ... Read More

In Golang, generating random numbers is a common task that is required in many applications. However, the type of random number generated is important as it affects the precision and range of the number. In this article, we will discuss how to generate random Float64 type numbers in Golang. What is Float64 Type? Float64 is a data type in Golang that represents a floating-point number with 64 bits of precision. It is a double-precision floating-point format that can store values with a larger range and higher precision than a Float32 type. Float64 types are commonly used in scientific and engineering ... Read More

The PySpark data frame is a powerful, real time data processing framework which was developed by the Apache Spark developers. Spark was originally written in “scala” programming language and in order to increase its reach and flexibility, several APIs were built. These APIs provided an interface which can be used to run spark applications on our local environment. One such API is known as PySpark which was developed for the python environment. The PySpark data frame also consists of rows and columns but the processing part is different as it uses in-system (RAM) computational techniques for processing the data. ... Read More

Pandas data frame is a very powerful data manipulation tool. It is a tabular data structure consisting of rows and columns. The size of this 2-D matrix can be variable depending upon the complexity of the dataset. We can use different type of sources to create a data frame ranging from databases to files. The columns in a pandas data frame represents a series of information and it can be an integer, float, or string. We can perform numerous operations on these columns including deletion, indexing, filtering etc. In this article, we will perform one such basic operation of ... Read More

PySpark is a tool designed by the Apache spark community to process data in real time and analyse the results in a local python environment. Spark data frames are different from other data frames as it distributes the information and follows a schema. Spark can handle stream processing as well as batch processing and this is the reason for their popularity. A PySpark data frame requires a session in order to generate an entry point and it performs on-system processing of the data (RAM). You can install PySpark module on windows using the following command – pip install pyspark ... Read More

A pandas data frame is a 2D data structure consisting of a series of entities. It is very useful in the analysis of mathematical data. The data is arranged in a tabular manner with each row behaving as an instance of the data. A Pandas data frame is special because it is empowered with numerous functions making it a very powerful programming asset. Each column in a data frame represents a series of information which is labelled. In this article, we will operate on these columns and discuss the various methods to drop columns in a pandas data frame. ... Read More