How to take a random row from a PySpark DataFrame?

In PySpark, working with large datasets often requires extracting a random row from a DataFrame for various purposes such as sampling or testing. However, the process of selecting a random row can be challenging due to the distributed nature of Spark.

In this article, we explore efficient techniques to tackle this task, discussing different approaches and providing code examples to help you effortlessly extract a random row from a PySpark DataFrame.

How to take a random row from a PySpark DataFrame?

Below are the Approaches or the methods using which we can take a random row from a PySpark Dataframe −

Approach 1: Using the orderBy and limit functions

One approach to selecting a random row from a PySpark DataFrame involves using the orderBy and limit functions. We can add a random column to the DataFrame using the rand function, then order the DataFrame by this column, and finally select the top row using the limit function.

To take a random row from a PySpark DataFrame using orderBy and limit, add a random column using rand(), order the DataFrame by the random column, and then use limit(1) to select the top row. This approach shuffles the rows randomly, allowing you to obtain a random row from the DataFrame.

Approach 2: Sampling using the sample function

Another approach is to use the sample function to randomly sample rows from the PySpark DataFrame. By specifying a fraction and setting withReplacement to False, we can sample a single random row.

To take a random row from a PySpark DataFrame using sampling, utilize the sample function. Specify a fraction of rows to sample, set withReplacement to False for unique rows, and provide a seed for reproducibility. This approach randomly selects a subset of rows from the DataFrame, allowing you to obtain a random row.

Approach 3: Converting DataFrame to RDD and sampling

PySpark DataFrames can be converted to RDDs (Resilient Distributed Datasets) for more flexibility. We can convert the DataFrame to an RDD and then use the takeSample function to retrieve a random row.

To take a random row from a PySpark DataFrame by converting it to an RDD, use the rdd property of the DataFrame to obtain the RDD representation. Then, apply the takeSample function on the RDD to randomly select a row. This approach allows you to directly sample from the RDD, effectively retrieving a random row from the original DataFrame.

These approaches provide different methods to extract a random row from a PySpark DataFrame. Depending on your specific requirements and the characteristics of your dataset, you can choose the approach that best suits your needs.

Below is the program example that shows all the approaches −

Example

from pyspark.sql import SparkSession
from pyspark.sql.functions import rand

# Create a SparkSession
spark = SparkSession.builder.getOrCreate()

# Create a sample DataFrame
data = [("Alice", 25), ("Bob", 30), ("Charlie", 35), ("David", 40), ("Eve", 45)]
df = spark.createDataFrame(data, ["Name", "Age"])

# Show the DataFrame
df.show()

# Approach 1: Using the `orderBy` and `limit` functions
df_with_random = df.withColumn("random", rand())
random_row_1 = df_with_random.orderBy("random").limit(1)
random_row_1.show()

# Approach 2: Using the `sample` function
random_row_2 = df.sample(withReplacement=False, fraction=0.1, seed=42)
random_row_2.show()

# Approach 3: Converting DataFrame to RDD and sampling
rdd = df.rdd
random_row_3 = rdd.takeSample(False, 1)
print(random_row_3)

Output

+-------+---+
|   Name|Age|
+-------+---+
|  Alice| 25|
|    Bob| 30|
|Charlie| 35|
|  David| 40|
|    Eve| 45|
+-------+---+

+-------+---+-------------------+
|   Name|Age|             random|
+-------+---+-------------------+
|Charlie| 35|0.27493566011232994|
+-------+---+-------------------+

+----+---+
|Name|Age|
+----+---+
| Bob| 30|
+----+---+

[Row(Name='Charlie', Age=35)]

The above examples illustrate different approaches to retrieving a random row from a PySpark DataFrame. Approach 1 uses the orderBy and limit functions to add a random column, sort the DataFrame by that column, and select the top row. Approach 2 utilizes the sample function to sample a fraction of the DataFrame's rows. Approach 3 involves converting the DataFrame to an RDD and using the takeSample function to retrieve a random row.

Conclusion

In conclusion, we explored different approaches to retrieve a random row from a PySpark DataFrame. Using the orderBy and limit functions, we added a random column, sorted the DataFrame, and selected the top row. Alternatively, we employed the sample function to sample a fraction of the DataFrame's rows.

Additionally, we discussed converting the DataFrame to an RDD and using the takeSample function to retrieve a random row. These methods provide flexibility and convenience for selecting random rows in PySpark, catering to different use cases and dataset sizes. Choose the approach that best suits your requirements for randomness and efficiency.