Exploring Data Distribution

Data distribution analysis is a fundamental aspect of exploratory data analysis in data science and machine learning. Understanding how your data is distributed helps identify patterns, outliers, central tendencies, and the overall shape of your dataset. Python provides several powerful visualization tools to explore data distributions effectively.

Histograms and Density Plots

Histograms are the most popular graphical method for exploring data distribution. They use rectangular bars to represent the frequency of values within specific intervals called bins. A KDE (Kernel Density Estimation) plot shows the probability density function as a smooth curve.

Basic Histogram Example

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

# Create sample data
np.random.seed(42)
prices = np.random.normal(200000, 50000, 1000)
df = pd.DataFrame({'SalePrice': prices})

# Create histogram and density plot
fig, ax = plt.subplots(1, 2, figsize=(12, 5))

# Histogram
ax[0].hist(df['SalePrice'], bins=30, alpha=0.7, color='skyblue')
ax[0].set_xlabel('Sale Price')
ax[0].set_ylabel('Frequency')
ax[0].set_title('Histogram')

# Density plot
sns.kdeplot(data=df, x='SalePrice', ax=ax[1], fill=True)
ax[1].set_xlabel('Sale Price')
ax[1].set_ylabel('Density')
ax[1].set_title('Density Plot')

plt.tight_layout()
plt.show()

Using Different Bin Sizes

import seaborn as sns
import matplotlib.pyplot as plt

# Load sample dataset
penguins = sns.load_dataset("penguins")

# Create histogram with custom bins
plt.figure(figsize=(10, 6))
sns.histplot(data=penguins, x="bill_depth_mm", bins=15, alpha=0.7)
plt.title('Bill Depth Distribution')
plt.xlabel('Bill Depth (mm)')
plt.ylabel('Count')
plt.show()

Box Plots

Box plots (box-and-whisker plots) display data distribution through quartiles. They show the 25th percentile, median (50th percentile), 75th percentile, and outliers. The box represents the Interquartile Range (IQR) containing the middle 50% of the data.

Creating Box Plots

import seaborn as sns
import matplotlib.pyplot as plt

# Load dataset
tips = sns.load_dataset("tips")

# Create box plot
plt.figure(figsize=(8, 6))
sns.boxplot(x='day', y='total_bill', data=tips)
plt.title('Total Bill Distribution by Day')
plt.xlabel('Day of Week')
plt.ylabel('Total Bill ($)')
plt.show()

Violin Plots

Violin plots combine box plots with kernel density estimation. They show both the quartile information and the probability density of the data at different values, making them ideal for comparing distributions across categories.

import seaborn as sns
import matplotlib.pyplot as plt

# Load dataset
tips = sns.load_dataset("tips")

# Create violin plot
plt.figure(figsize=(8, 6))
sns.violinplot(x='day', y='total_bill', data=tips)
plt.title('Total Bill Distribution by Day (Violin Plot)')
plt.xlabel('Day of Week')
plt.ylabel('Total Bill ($)')
plt.show()

Comparison of Visualization Methods

Conclusion

Histograms, box plots, and violin plots are essential tools for exploring data distribution. Use histograms for frequency analysis, box plots for identifying outliers and quartiles, and violin plots for comparing probability distributions across categories. These visualizations provide crucial insights into data skewness, spread, and central tendencies.