How to shift a column in a Pandas DataFrame?

The shift() method in Pandas allows you to shift the values in a column up or down by a specified number of positions. This is useful for creating lagged variables or aligning time series data.

Syntax

shift(periods=1, freq=None, axis=0, fill_value=None)

Parameters

• periods ? Number of positions to shift. Positive values shift down, negative values shift up.
• axis ? 0 for shifting along rows (default), 1 for shifting along columns.
• fill_value ? Value to use for filling the newly created missing positions.

Basic Column Shifting

Let's create a DataFrame and demonstrate shifting columns ?

import pandas as pd

df = pd.DataFrame({
    'name': ['John', 'Jacob', 'Tom', 'Tim', 'Ally'],
    'marks': [89, 23, 100, 56, 90],
    'subjects': ["Math", "Physics", "Chemistry", "Biology", "English"]
})

print("Original DataFrame:")
print(df)

Original DataFrame:
    name  marks   subjects
0   John     89       Math
1  Jacob     23    Physics
2    Tom    100  Chemistry
3    Tim     56    Biology
4   Ally     90    English

Shifting Values Down

Shift the 'name' column down by 1 position ?

import pandas as pd

df = pd.DataFrame({
    'name': ['John', 'Jacob', 'Tom', 'Tim', 'Ally'],
    'marks': [89, 23, 100, 56, 90],
    'subjects': ["Math", "Physics", "Chemistry", "Biology", "English"]
})

# Shift name column down by 1
df['name'] = df['name'].shift(1)
print("After shifting 'name' down by 1:")
print(df)

After shifting 'name' down by 1:
   name  marks   subjects
0   NaN     89       Math
1  John     23    Physics
2  Jacob    100  Chemistry
3   Tom     56    Biology
4   Tim     90    English

Shifting Values Up

Use negative values to shift up ?

import pandas as pd

df = pd.DataFrame({
    'name': ['John', 'Jacob', 'Tom', 'Tim', 'Ally'],
    'marks': [89, 23, 100, 56, 90],
    'subjects': ["Math", "Physics", "Chemistry", "Biology", "English"]
})

# Shift subjects column up by 1
df['subjects'] = df['subjects'].shift(-1)
print("After shifting 'subjects' up by 1:")
print(df)

After shifting 'subjects' up by 1:
    name  marks   subjects
0   John     89    Physics
1  Jacob     23  Chemistry
2    Tom    100    Biology
3    Tim     56    English
4   Ally     90        NaN

Using fill_value Parameter

Replace NaN values with a custom fill value ?

import pandas as pd

df = pd.DataFrame({
    'name': ['John', 'Jacob', 'Tom', 'Tim', 'Ally'],
    'marks': [89, 23, 100, 56, 90]
})

# Shift with custom fill value
df['marks_shifted'] = df['marks'].shift(1, fill_value=0)
print("Shifting with fill_value=0:")
print(df)

Shifting with fill_value=0:
    name  marks  marks_shifted
0   John     89              0
1  Jacob     23             89
2    Tom    100             23
3    Tim     56            100
4   Ally     90             56

Comparison of Shift Operations

Conclusion

The shift() method is essential for time series analysis and creating lagged variables. Use positive values to shift down and negative values to shift up. The fill_value parameter helps replace NaN values with meaningful defaults.