How to merge multiple files into a new file using Python?

Python makes it simple to create new files, read existing files, append data, or replace data in existing files. With the aid of a few open-source and third-party libraries, it can manage practically all of the file types that are currently supported.

We must iterate through all the necessary files, collect their data, and then add it to a new file in order to concatenate several files into a single file. This article demonstrates how to use Python to merge multiple files into a single file.

Using For Loop to Merge Multiple Files

A list of filenames or file paths to the necessary python files may be found in the Python code below. Next, the merged output file is either opened or created in write mode.

The list of filenames or file paths is then iterated over. Each file generates a file descriptor, reads its contents line by line, and then writes the information to the output file.

It adds a newline character, or \n, to the new file at the end of each line ?

# Create sample files to demonstrate merging
with open("file1.txt", "w") as f:
    f.write("Content of file 1\nLine 2 of file 1")

with open("file2.txt", "w") as f:
    f.write("Content of file 2\nLine 2 of file 2")

with open("file3.txt", "w") as f:
    f.write("Content of file 3\nLine 2 of file 3")

# Merge multiple files
file_names = ["file1.txt", "file2.txt", "file3.txt"]

with open("merged_file.txt", "w") as merged_file:
    for name in file_names:
        with open(name) as current_file:
            for line in current_file:
                merged_file.write(line)
        merged_file.write("\n")  # Add separator between files

# Display the merged content
with open("merged_file.txt", "r") as f:
    print(f.read())

Content of file 1
Line 2 of file 1

Content of file 2
Line 2 of file 2

Content of file 3
Line 2 of file 3

Using String Concatenation Method

In this approach, we read the entire content from each file into string variables, concatenate them together, and then write the combined data to a new file ?

# Create sample files
with open("sample1.txt", "w") as f:
    f.write("Hello from file 1\nThis is line 2")

with open("sample2.txt", "w") as f:
    f.write("Hello from file 2\nThis is another line")

# Read and merge using string concatenation
content1 = content2 = ""

# Reading information from the first file
with open('sample1.txt') as file:
    content1 = file.read()

# Reading information from the second file
with open('sample2.txt') as file:
    content2 = file.read()

# Merge files with separator
content1 += "\n"
content1 += content2

# Write merged content to new file
with open('merged_output.txt', 'w') as file:
    file.write(content1)

# Display the result
with open('merged_output.txt', 'r') as f:
    print(f.read())

Hello from file 1
This is line 2
Hello from file 2
This is another line

Using shutil Module for Binary Files

For binary files or when you need more efficient file operations, you can use the shutil module ?

import shutil

# Create sample files
with open("part1.txt", "w") as f:
    f.write("First part of the document\n")

with open("part2.txt", "w") as f:
    f.write("Second part of the document\n")

# Merge using shutil
with open("combined.txt", "wb") as output_file:
    for filename in ["part1.txt", "part2.txt"]:
        with open(filename, "rb") as input_file:
            shutil.copyfileobj(input_file, output_file)

# Display result
with open("combined.txt", "r") as f:
    print(f.read())

First part of the document
Second part of the document

Comparison of Methods

Conclusion

Use the for loop method for memory-efficient line-by-line merging of text files. For large files or binary data, the shutil module provides the most efficient solution. String concatenation works well for small files but can consume more memory.