Python Convert nested dictionary into flattened dictionary?

As the world embraces more unstructured data, we come across many formats of data where the data structure can be deeply nested like nested JSONs. Python has the ability to deal with nested data structure by concatenating the inner keys with outer keys to flatten the data. In this article we will take a nested dictionary and flatten it.

Using a Recursive Approach

In this approach we design a function to recursively process each item in the dictionary. We pass the dictionary, design a place holder for the output dictionary, the key and separator as parameters. We use the isinstance() to check if the next item itself is a dictionary and then pass it through the recursive call if it is also a dictionary.

Example

dictA = {
    "id": "0001",
    "name": "hotdog",
    "image": {
        "url": "images/0001.jpg",
        "thumbnail": {
            "url": "images/thumbnails/0001.jpg",
            "height,width": "2x4"
        }
    }
}

def dict_flatten(in_dict, dict_out=None, parent_key=None, separator="_"):
    if dict_out is None:
        dict_out = {}

    for k, v in in_dict.items():
        k = f"{parent_key}{separator}{k}" if parent_key else k
        if isinstance(v, dict):
            dict_flatten(in_dict=v, dict_out=dict_out, parent_key=k)
            continue

        dict_out[k] = v

    return dict_out

final_dict = dict_flatten(dictA)
print(final_dict)

The output of the above code is ?

{'id': '0001', 'name': 'hotdog', 'image_url': 'images/0001.jpg', 'image_thumbnail_url': 'images/thumbnails/0001.jpg', 'image_thumbnail_height,width': '2x4'}

Using pandas.json_normalize()

Pandas provides a built-in function json_normalize() that can flatten nested dictionaries with custom separators ?

Example

import pandas as pd

dictA = {
    "id": "0001",
    "name": "hotdog",
    "image": {
        "url": "images/0001.jpg",
        "thumbnail": {
            "url": "images/thumbnails/0001.jpg",
            "height,width": "2x4"
        }
    }
}

# Normalize and convert back to dictionary
df = pd.json_normalize(dictA, sep='_')
flattened_dict = df.to_dict(orient='records')[0]
print(flattened_dict)

The output of the above code is ?

{'id': '0001', 'name': 'hotdog', 'image_url': 'images/0001.jpg', 'image_thumbnail_url': 'images/thumbnails/0001.jpg', 'image_thumbnail_height,width': '2x4'}

Using Iterative Approach with Stack

An alternative non-recursive approach using a stack to avoid potential stack overflow issues ?

Example

dictA = {
    "id": "0001",
    "name": "hotdog",
    "image": {
        "url": "images/0001.jpg",
        "thumbnail": {
            "url": "images/thumbnails/0001.jpg",
            "height,width": "2x4"
        }
    }
}

def flatten_dict_iterative(nested_dict, separator='_'):
    result = {}
    stack = [(nested_dict, '')]
    
    while stack:
        current_dict, parent_key = stack.pop()
        
        for key, value in current_dict.items():
            new_key = f"{parent_key}{separator}{key}" if parent_key else key
            
            if isinstance(value, dict):
                stack.append((value, new_key))
            else:
                result[new_key] = value
    
    return result

flattened = flatten_dict_iterative(dictA)
print(flattened)

The output of the above code is ?

{'id': '0001', 'name': 'hotdog', 'image_url': 'images/0001.jpg', 'image_thumbnail_url': 'images/thumbnails/0001.jpg', 'image_thumbnail_height,width': '2x4'}

Comparison

Conclusion

Use the recursive approach for simple nested dictionaries. For data analysis, pandas.json_normalize() provides robust flattening capabilities. Choose the iterative stack method for very deeply nested structures to avoid recursion limits.