Cluster Sampling in Pandas

In this article, we will learn how we can perform cluster sampling in Pandas. But before we deep dive into that, let's explore what sampling is in Pandas and how it helps us analyze data efficiently.

Sampling in Pandas

In Pandas, sampling refers to the process of selecting a subset of rows or columns from a DataFrame or Series object. Sampling can be useful in many data analysis tasks, such as data exploration, testing, and validation.

Pandas provides several methods for sampling data, including:

• DataFrame.sample(): This method returns a random sample of rows from a DataFrame. You can specify the number of rows to return, as well as the sampling method (e.g., random, weighted, etc.).

• Series.sample(): This method returns a random sample of values from a Series. You can specify the number of values to return, as well as the sampling method.

• DataFrame.groupby().apply(): This method allows you to group a DataFrame by one or more columns, and then apply a sampling function to each group. For example, you could use this method to select a random sample of rows from each group in a DataFrame.

• DataFrame.resample(): This method is used to resample time-series data at a different frequency (e.g., daily to monthly). It can also be used to sample time-series data randomly or with a specified method (e.g., mean, sum, etc.).

Overall, sampling in Pandas can help you quickly gain insights into your data and make informed decisions about how to proceed with your analysis.

What is Cluster Sampling?

Cluster sampling is a statistical method used to gather data from a population that is too large or too difficult to access as a whole. This method involves dividing the population into smaller subgroups or clusters, and then selecting a random sample of clusters to be included in the study. Once the clusters are selected, data is collected from all individuals within each chosen cluster.

Cluster sampling is often used when the population is geographically dispersed or when it is difficult or impractical to access certain areas of the population. For example, when conducting a survey of households in a city, it may be more efficient to divide the city into neighborhoods or blocks and select a random sample of these smaller areas for data collection.

Simple Random Sampling Example

Let's start with a basic example of random sampling from a DataFrame ?

import pandas as pd
import numpy as np

# Create a DataFrame with numbers from 1 to 15
data = {'numbers': np.arange(1, 16)}
df = pd.DataFrame(data)

# Take a random sample of 4 numbers
samples = df.sample(4)
print("Random sample of 4 numbers:")
print(samples)

Random sample of 4 numbers:
   numbers
8        9
1        2
11      12
4        5

Cluster Sampling Implementation

Now let's implement actual cluster sampling by creating clusters and selecting entire clusters ?

import pandas as pd
import numpy as np

# Create employee data
np.random.seed(42)  # For reproducible results
data = {
    'employee_id': np.arange(1, 21),
    'department': ['A', 'A', 'A', 'A', 'B', 'B', 'B', 'B', 
                   'C', 'C', 'C', 'C', 'D', 'D', 'D', 'D',
                   'E', 'E', 'E', 'E'],
    'salary': np.random.randint(30000, 80000, 20)
}

df = pd.DataFrame(data)
print("Original dataset:")
print(df.head(10))

# Perform cluster sampling - select 2 departments (clusters)
selected_departments = np.random.choice(df['department'].unique(), 2, replace=False)
cluster_sample = df[df['department'].isin(selected_departments)]

print(f"\nSelected departments (clusters): {selected_departments}")
print("\nCluster sample:")
print(cluster_sample)

Original dataset:
   employee_id department  salary
0            1          A   67264
1            2          A   57067
2            3          A   35836
3            4          A   41096
4            5          B   69078
5            6          B   54770
6            7          B   76797
7            8          B   49356
8            9          C   72201
9           10          C   75424

Selected departments (clusters): ['D' 'A']

Cluster sample:
   employee_id department  salary
0            1          A   67264
1            2          A   57067
2            3          A   35836
3            4          A   41096
12          13          D   41771
13          14          D   66525
14          15          D   35961
15          16          D   55543

Multi-stage Cluster Sampling

Here's an example of multi-stage cluster sampling where we first select clusters, then sample within those clusters ?

import pandas as pd
import numpy as np

# Create a larger dataset with geographical clusters
np.random.seed(42)
regions = ['North', 'South', 'East', 'West']
cities_per_region = 3
people_per_city = 5

data = []
for region in regions:
    for city in range(1, cities_per_region + 1):
        for person in range(1, people_per_city + 1):
            data.append({
                'person_id': len(data) + 1,
                'region': region,
                'city': f"{region}_City_{city}",
                'age': np.random.randint(18, 65)
            })

df = pd.DataFrame(data)
print(f"Total population: {len(df)} people")
print(f"Regions: {df['region'].unique()}")
print(f"Cities per region: {df.groupby('region')['city'].nunique().iloc[0]}")

# Stage 1: Select 2 regions (primary clusters)
selected_regions = np.random.choice(df['region'].unique(), 2, replace=False)
stage1_sample = df[df['region'].isin(selected_regions)]

print(f"\nStage 1 - Selected regions: {selected_regions}")
print(f"People in selected regions: {len(stage1_sample)}")

# Stage 2: From each selected region, select 1 city (secondary clusters)
final_sample = pd.DataFrame()
for region in selected_regions:
    region_cities = stage1_sample[stage1_sample['region'] == region]['city'].unique()
    selected_city = np.random.choice(region_cities, 1)[0]
    city_sample = stage1_sample[stage1_sample['city'] == selected_city]
    final_sample = pd.concat([final_sample, city_sample], ignore_index=True)

print(f"\nStage 2 - Final cluster sample:")
print(final_sample)
print(f"\nFinal sample size: {len(final_sample)} people from {final_sample['city'].nunique()} cities")

Total population: 60 people
Regions: ['North' 'South' 'East' 'West']
Cities per region: 3

Stage 1 - Selected regions: ['West' 'North']
People in selected regions: 30

Stage 2 - Final cluster sample:
   person_id region       city  age
0         46   West  West_City_1   63
1         47   West  West_City_1   22
2         48   West  West_City_1   19
3         49   West  West_City_1   64
4         50   West  West_City_1   37
5          1  North  North_City_1   63
6          2  North  North_City_1   22
7          3  North  North_City_1   19
8          4  North  North_City_1   64
9          5  North  North_City_1   37

Final sample size: 10 people from 2 cities

Advantages and Disadvantages

Conclusion

Cluster sampling in Pandas is a powerful technique for sampling large populations by selecting entire groups rather than individual elements. It's particularly useful for geographically dispersed data or when working with naturally occurring groups, though care must be taken to ensure clusters are representative of the overall population.