Python Pandas - Concatenating Categorical Data

---

---

Categorical data types are often used for efficient memory usage and performance when working with category values. Concatenating or merging categorical data can sometimes challenging, particularly when the categories involved are not unique.

Pandas provides efficient ways to concatenate/merge categorical data, optimizing memory usage and maintaining categorical properties. In this tutorial, we will learn how to merge or concatenate Series or DataFrame objects containing categorical data, with examples for various scenarios.

Concatenating Categorical Objects

When concatenating two Pandas Series or DataFrame objects with the same categorical data, will result the same category dtype. This ensures that the result uses the same efficient storage as the original data.

Example

This example demonstrates concatenating two Pandas Series objects with the same categorical data. Both Series objects contain the same categories ('cat' and 'dog'). After concatenation, the resulting Series retains the category dtype and has the same categories.

import pandas as pd

# Creating categorical Series
s1 = pd.Series(["cat", "dog"], dtype="category")
s2 = pd.Series(["cat", "dog", 'dog'], dtype="category")

# Display the Input Series objects
print("Input Series 1:")
print(s1)
print("\nInput Series 2:")
print(s2)

# Concatenating the Series
result = pd.concat([s1, s2])

print("\nSeries after concatenating:")
print(result)

When we run above program, it produces following result −

Input Series 1:
0    cat
1    dog
dtype: category
Categories (2, object): ['cat', 'dog']

Input Series 2:
0    cat
1    dog
2    dog
dtype: category
Categories (2, object): ['cat', 'dog']

Series after concatenating:
0    cat
1    dog
0    cat
1    dog
2    dog
dtype: category
Categories (2, object): ['cat', 'dog']

Concatenating with Different Categories

If you try to concatenate Series or DataFrame objects with different categories, Pandas will not automatically keep the category dtype. Instead, it will return the default object dtype, which is less memory efficient.

Example

This example demonstrates how the output dtype changes when concatenating two Pandas Series objects with the different categorical data.

import pandas as pd

# Creating categorical Series
s1 = pd.Series(["cat", "dog"], dtype="category")
# Series with different categories
s2 = pd.Series(["dog", "mouse", "cat"], dtype="category")

# Display the Input Series objects
print("Input Series 1:")
print(s1)
print("\nInput Series 2:")
print(s2)

# Concatenating the Series
result = pd.concat([s1, s2])

print("\nSeries after concatenating:")
print(result)

When we run above program, it produces following result −

Input Series 1:
0    cat
1    dog
dtype: category
Categories (2, object): ['cat', 'dog']

Input Series 2:
0      dog
1    mouse
2      cat
dtype: category
Categories (3, object): ['cat', 'dog', 'mouse']

Series after concatenating:
0      cat
1      dog
0      dog
1    mouse
2      cat
dtype: object

Concatenating with Different Data Types

When concatenating Pandas Series with categorical data of different data types (e.g., integers and floats), Pandas will infer the dtype of the concatenated result based on the values present.

Example

This example demonstrating output dtype changes while concatenating Pandas Series objects with the different data types, integer and float categories.

import pandas as pd

# Series with integer categories
int_cats = pd.Series([1, 2], dtype="category")

# Series with float categories
float_cats = pd.Series([3.0, 4.0], dtype="category")

# Display the Input Series objects
print("Input Series 1:")
print(int_cats)
print("\nInput Series 2:")
print(float_cats)

# Concatenating Series with different data types
result = pd.concat([int_cats, float_cats])

print("\nSeries after concatenating:")
print(result)

When we run above program, it produces following result −

Input Series 1:
0    1
1    2
dtype: category
Categories (2, int64): [1, 2]

Input Series 2:
0    3.0
1    4.0
dtype: category
Categories (2, float64): [3.0, 4.0]

Series after concatenating:
0    1.0
1    2.0
0    3.0
1    4.0
dtype: float64

Ensuring Categorical Type After Concatenation

If you want to ensure that the result is of category dtype after concatenation, you can use .astype("category"). This will force the result to use the more memory-efficient categorical type.

Example

Following example forcefully changes the concatenated result data type using the .astype("category") method.

import pandas as pd

# Creating categorical Series
s1 = pd.Series(["cat", "dog"], dtype="category")
# Series with different categories
s2 = pd.Series(["dog", "mouse", "cat"], dtype="category")

# Display the Input Series objects
print("Input Series 1:")
print(s1)
print("\nInput Series 2:")
print(s2)

# Concatenating Series and forcing category dtype
result = pd.concat([s1, s2]).astype("category")

print("\nSeries after concatenating:")
print(result)

Following is an output of the above code −

Input Series 1:
0    cat
1    dog
dtype: category
Categories (2, object): ['cat', 'dog']

Input Series 2:
0      dog
1    mouse
2      cat
dtype: category
Categories (3, object): ['cat', 'dog', 'mouse']

Series after concatenating:
0      cat
1      dog
0      dog
1    mouse
2      cat
dtype: category
Categories (3, object): ['cat', 'dog', 'mouse']

Concatenating categorical data in Pandas is efficient when categories align. You can use astype() or union_categoricals() functions for consistent category management.