Using sed to Replace a Multi-Line String

Introduction

Sed, or Stream Editor, is a powerful command-line tool that allows you to manipulate and transform text files. One of most common tasks that you may need to perform is to replace a multi-line string in a file with another string. In this article, we will explore how to use sed to accomplish this task.

Understanding Sed's s Command

Sed's s command is primary tool that you will use to perform text replacements. It takes following form −

s/pattern/replacement/flags

Here, pattern is a regular expression that matches text that you want to replace, replacement is text that you want to replace it with, and flags are optional flags that modify behavior of command.

Replacing a Multi-Line String with Sed

By default, sed operates on a line-by-line basis, meaning that it matches patterns and performs replacements on each line independently. However, with a little bit of trickery, we can use sed to replace multi-line strings as well.

Let's say that we have following multi-line string in a file named example.txt −

This is a multi-line
string that we want to replace
with a new string.

To replace this string with a new string, we can use following sed command −

sed -i ':a;N;$!ba;s/This is a multi-line
string that we want to replace
with a new string\./New string/g' example.txt

Let's break this command down −

• The -i flag tells sed to edit file in place, meaning that it will overwrite original file with modified version.

• The :a;N;$!ba; command is a sed idiom that reads entire file into pattern space, rather than processing it line-by-line.

• The s/This is a multi-line
  string that we want to replace
  with a new string\./New string/g command replaces multi-line string with new string.

Note that we need to use
to represent newlines in pattern that we want to match.

Using Variables to Make Command Readable

The sed command that we just used is quite long and difficult to read. To make it more readable, we can use variables to represent strings that we want to match and replace.

For example, let's say that we want to replace following multi-line string −

Lorem ipsum dolor sit amet,
consectetur adipiscing elit,
sed do eiusmod tempor incididunt
ut labore et dolore magna aliqua.

We can define variables for old and new strings like this −

OLD_STRING="Lorem ipsum dolor sit amet,
consectetur adipiscing elit,
sed do eiusmod tempor incididunt
ut labore et dolore magna aliqua."
NEW_STRING="New string"

And then use them in our sed command like this −

sed -i ":a;N;$!ba;s/${OLD_STRING}/${NEW_STRING}/g" example.txt

Using Regular Expressions to Match Multiple Strings

What if we want to replace multiple strings in a file, but we don't know exactly what those strings are? In this case, we can use regular expressions to match a pattern that covers all of strings that we want to replace.

For example, let's say that we have a file that contains a list of email addresses, like this −

user1@example.com
user2@example.com
user3@example.com

We want to replace all of email addresses in this file with a new email address, but we don't know exactly what old email addresses are.

In this case, we can use a regular expression to match all of email addresses in file, like this −

sed -i 's/[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}/newemail@example.com/g' example.txt

Let's break this command down −

• The regular expression [a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,} matches all valid email addresses.

• The s/ command replaces matched pattern with newemail@example.com.

• The g flag tells sed to perform replacement globally, meaning that it replaces all occurrences of pattern in file.

Using Hold Buffer to Preserve Data

Sometimes when we perform replacements with sed, we want to preserve some of data that we are replacing. For example, let's say that we have a file that contains a list of names and phone numbers, like this −

John Smith: (123) 456-7890
Jane Doe: (555) 555-1212

We want to replace phone numbers with a new phone number, but we want to preserve names. To do this, we can use hold buffer, which is a special buffer in sed that allows us to preserve data for later use.

We can use following sed command to replace phone numbers with a new phone number, while preserving names −

sed -i -e '/^.*:/{' -e 'h;s/[^:]*: //;s/([0-9]{3}) [0-9]{3}-[0-9]{4}/(555) 555-1234/;G;s/
/: /}' example.txt

Let's break this command down −

• The -e flag allows us to specify multiple sed commands.

• The /^.*:/ regular expression matches any line that contains a colon.

• The h command saves matched line to hold buffer.

• The s/[^:]*: // command removes name from line.

• The s/([0-9]{3}) [0-9]{3}-[0-9]{4}/(555) 555-1234/ command replaces phone number with a new phone number.

• The G command appends contents of hold buffer to pattern space.

• The s/
  /: / command adds name back to line, separated by a colon and a space.

Using Sed to Replace Multi-Line Strings

In some cases, you may need to replace a multi-line string with another multi-line string. This can be a bit trickier than simply replacing a single line of text, but sed makes it possible with a few special commands.

Let's say we have a file called example.txt that contains following multi-line string −

This is a multi-line
string that we want to
replace with another
multi-line string.

We want to replace this string with following multi-line string −

Here is our new
multi-line string
that we will use
to replace old one.

We can do this with following sed command −

sed -i -e ':a;N;$!ba;s/This is a multi-line
string that we want to
replace with another
multi-line string\./Here is our new
multi-line string
t that we will use
to replace old one\./g' example.txt

Let's break this command down −

• The -e flag allows us to specify multiple sed commands.

• The :a;N;$!ba command reads entire file into pattern space.

• The s/This is a multi-line
  string that we want to
  replace with another
  multi-line string\./Here is our new
  multi-line string
  t that we will use
  to replace old one\./g command replaces old multi-line string with new one.

Using Sed to Match Multiple Strings with Regular Expressions

In some cases, you may need to match multiple strings with a single sed command. This can be accomplished using regular expressions.

For example, let's say that we have a file called example.txt that contains following lines −

This is first line.
This is second line.
This is third line.

We want to replace all instances of word "line" with word "sentence". We can do this with following sed command −

sed -i 's/line/sentence/g' example.txt

Let's break this command down −

• The s/line/sentence/g command matches all instances of word "line" and replaces them with word "sentence".

Using Sed to Remove Blank Lines

In some cases, you may need to remove blank lines from a file. This can be accomplished using sed.

For example, let's say that we have a file called example.txt that contains following lines −

This is first line.
This is second line.
This is third line.

We want to remove all of blank lines from file. We can do this with following sed command −

sed -i '/^$/d' example.txt

Let's break this command down −

• The /^$/d command matches all lines that are empty (i.e., contain only a newline character) and deletes them.

Using Sed to Replace Multiple Strings in a Single Command

In some cases, you may need to replace multiple strings in a single sed command. This can be accomplished using multiple s commands separated by semicolons.

For example, let's say that we have a file called example.txt that contains following lines −

This is first line.
This is second line.
This is third line.

We want to replace all instances of word "line" with word "sentence" and all instances of word "first" with word "initial". We can do this with following sed command −

sed -i 's/line/sentence/g;s/first/initial/g' example.txt

Let's break this command down −

• The s/line/sentence/g command matches all instances of word "line" and replaces them with word "sentence".

• The s/first/initial/g command matches all instances of word "first" and replaces them with word "initial".

Conclusion

Sed is a powerful tool for performing text replacements and transformations. In this article, we explored how to use sed to replace multi-line strings, use variables to make command readable, match multiple strings with regular expressions, and use hold buffer to preserve data. With these techniques, you can efficiently manipulate and transform text files to meet your needs.