How to Scale Pandas DataFrame Columns?

Scaling is the process of preprocessing the data in data analysis and ensuring that all the features in a dataset have similar ranges, making them more comparable and reducing the impact of different scales on machine learning algorithms. We can scale Pandas dataframe columns using methods like Min-max scaling, standardization, Robust scaling, and log transformation. In this article we will dive into the process of scaling pandas dataframe scaling using various methods.

Why Scaling is Important?

• Some features in the data may have larger values which can dominate when the analysis or model training is done. Scaling ensures that all the features have a similar range, preventing certain features from dominating the analysis.

• Some of the machine learning algorithms like the k-nearest neighbor, and support vector machines rely on distance-based calculations. Scaling ensures that the feature contributes equally to the calculations.

• Scaling also improves convergence rates and the performance of iterative algorithms like gradient descent algorithms.

• Features with similar scales become more easily comparable due to scaling.

Scaling Data frames columns using Pandas

Method 1: Using Min-Max Scaling

Min-Max scaling is also known as normalization. Using min-max scaling we can resize the data to a fixed range, typically between 0 and 1. The original distribution shape is preserved maintaining both the minimum and maximum values.

Syntax

 df[column_name] = (df[column_name] - df[column_name].min()) / (df[column_name].max() - df[column_name].min())

Here, Min-max scaling rescales the values of a column between 0 and 1 by subtracting the minimum value and dividing by the range (maximum value minus minimum value).

Example

In the below example, we use the min() and max() methods to calculate the minimum and maximum values of the column. Then, we can apply the scaling formula to rescale the values between 0 and 1.

import pandas as pd

# Create a sample DataFrame
data = {'Name': ['John', 'Emma', 'Tom', 'Emily', 'Mike'],
        'Salary': [50000, 80000, 45000, 70000, 90000]}
df = pd.DataFrame(data)

# Print the original DataFrame
print("Original DataFrame:")
print(df)
print()

#  Min-Max Scaling
def min_max_scaling(df, column_name):
    min_value = df[column_name].min()
    max_value = df[column_name].max()
    df[column_name] = (df[column_name] - min_value) / (max_value - min_value)

# Apply min-max scaling to 'Salary' column
min_max_scaling(df, 'Salary')

# Print the DataFrame after min-max scaling
print("DataFrame after Min-Max Scaling:")
print(df)
print()

Output

Original DataFrame:
    Name  Salary
0   John   50000
1   Emma   80000
2    Tom   45000
3  Emily   70000
4   Mike   90000

DataFrame after Min-Max Scaling:
    Name    Salary
0   John  0.111111
1   Emma  0.777778
2    Tom  0.000000
3  Emily  0.555556
4   Mike  1.000000

Method 2: Using the Standardization Technique

Standardization transforms the data to have zero mean and unit variance. Using standardization the data is centered around the mean and scaled based on standard deviation. The resulting distribution has a mean of 0 and a standard deviation of 1.

Syntax

df[column_name] = (df[column_name] - df[column_name].mean()) / df[column_name].std()

Here, Standardization transforms the values of a column to have zero mean and unit variance by subtracting the mean and dividing by the standard deviation.

Example

In the below example, a function named standardization is defined to perform standardization on a specified column. The function calculates the mean and standard deviation of the column and then applies the standardization formula to scale the values. Finally, the function is called on the 'Salary' column of the DataFrame, resulting in the values being standardized. The modified DataFrame is printed to show the result of the standardization process.

import pandas as pd

# Create a sample DataFrame
data = {'Name': ['John', 'Emma', 'Tom', 'Emily', 'Mike'],
        'Salary': [50000, 80000, 45000, 70000, 90000]}
df = pd.DataFrame(data)

# Print the original DataFrame
print("Original DataFrame:")
print(df)
print()

# standardization 
def standardization(df, column_name):
    mean_value = df[column_name].mean()
    std_value = df[column_name].std()
    df[column_name] = (df[column_name] - mean_value) / std_value

# Apply standardization to 'Salary' column
standardization(df, 'Salary')

# Print the DataFrame after standardization
print("DataFrame after Standardization:")
print(df)
print()

Output

Original DataFrame:
    Name  Salary
0   John   50000
1   Emma   80000
2    Tom   45000
3  Emily   70000
4   Mike   90000

DataFrame after Standardization:
    Name    Salary
0   John -0.883788
1   Emma  0.675838
2    Tom -1.143726
3  Emily  0.155963
4   Mike  1.195713

Method 3: Using Robust Scaling

Robust scaling is somewhat similar to standardization but it uses median and interquartile range (IQR) in place of mean and standard deviation. The outliers do not have much effect on Robust scaling so it is suitable for datasets with extreme values.

Syntax

df[column_name] = (df[column_name] - df[column_name].median()) / (df[column_name].quantile(0.75) - df[column_name].quantile(0.25))

Here, Robust scaling rescales the values of a column using the median and interquartile range (IQR), which is the range between the 25th percentile (Q1) and the 75th percentile (Q3).

Example

In the below example, a function called robust_scaling is defined to perform robust scaling on a specified column. Inside the function, the median, first quartile (q1), third quartile (q3), and interquartile range (IQR) of the column are calculated. The column values are then scaled by subtracting the median and dividing by the IQR. Finally, the function is applied to the 'Salary' column of the DataFrame, resulting in the values being robustly scaled. The modified DataFrame is printed to display the outcome of the robust scaling process.

import pandas as pd

# Create a sample DataFrame
data = {'Name': ['John', 'Emma', 'Tom', 'Emily', 'Mike'],
        'Salary': [50000, 80000, 45000, 70000, 90000]}
df = pd.DataFrame(data)

# Print the original DataFrame
print("Original DataFrame:")
print(df)
print()

# robust scaling
def robust_scaling(df, column_name):
    median_value = df[column_name].median()
    q1 = df[column_name].quantile(0.25)
    q3 = df[column_name].quantile(0.75)
    iqr = q3 - q1
    df[column_name] = (df[column_name] - median_value) / iqr

# Apply robust scaling to 'Salary' column
robust_scaling(df, 'Salary')

# Print the DataFrame after robust scaling
print("DataFrame after Robust Scaling:")
print(df)
print()

Output

Original DataFrame:
    Name  Salary
0   John   50000
1   Emma   80000
2    Tom   45000
3  Emily   70000
4   Mike   90000

DataFrame after Robust Scaling:
    Name    Salary
0   John -0.666667
1   Emma  0.333333
2    Tom -0.833333
3  Emily  0.000000
4   Mike  0.666667

Method 4: Using the Log Transformation Method

Log transformation uses a logarithmic function to apply non-linear scaling to the data. It reduces the impact of skewness in the data distribution which contains highly skewed data.

Syntax

import numpy as np
df[column_name] = np.log(df[column_name])

Here, Log transformation applies a logarithmic function to the values of a column, which can help reduce the impact of skewness and compress larger values towards smaller values.

Example

In the below example, a function called log_transformation is defined to perform a log transformation on a specified column. Within the function, the np.log() function from the NumPy library is used to apply the logarithmic transformation to the values in the column. The function is then applied to the 'Salary' column of the DataFrame, resulting in the values being transformed logarithmically. The modified DataFrame is printed to display the outcome of the log transformation process.

import pandas as pd

# Create a sample DataFrame
data = {'Name': ['John', 'Emma', 'Tom', 'Emily', 'Mike'],
        'Salary': [50000, 80000, 45000, 70000, 90000]}
df = pd.DataFrame(data)

# Print the original DataFrame
print("Original DataFrame:")
print(df)
print()


import numpy as np
# log transformation
def log_transformation(df, column_name):
    df[column_name] = np.log(df[column_name])

# Apply log transformation to 'Salary' column
log_transformation(df, 'Salary')

# Print the DataFrame after log transformation
print("DataFrame after Log Transformation:")
print(df)

Output

Original DataFrame:
    Name  Salary
0   John   50000
1   Emma   80000
2    Tom   45000
3  Emily   70000
4   Mike   90000

DataFrame after Log Transformation:
    Name     Salary
0   John  10.819778
1   Emma  11.289782
2    Tom  10.714418
3  Emily  11.156251
4   Mike  11.407565

Conclusion

In this article, we discussed how we can scale data frame columns in data processing and analysis. Scaling ensures that the features have similar ranges making them more comparable and reducing the impact of varying scales on the machine learning algorithm. We can use different scaling techniques like min-max scaling, standardization, robust scaling, and log transformation.