What is ESP in tunnel and transport mode and the difference between AH and ESP?

Computer NetworkInternetMCA

Encapsulating Security Payload (ESP) provides all encryption services in IPSec based on integrity for the payload and not for the IP header, confidentiality and authentication that using encryption, without authentication is strongly discouraged because it is insecure.

Any translations in readable message format into an unreadable format are encrypted and used to hide the message content against data tampering.

IPSec provides an open framework, such as SHA and MD5 for implementing industry standard algorithms.

Encryption/decryption allows only the sender and the authorized receiver to make the data to be received in readable form and only after the integrity verification process is complete, the data payload in the packet is decrypted.

IPSec uses a unique identifier for each packet, which is a data equivalent of a fingerprint and checks for packets that are authorized or not. It doesn't sign the entire packet unless it is being tunneled—ordinarily, for this IP data payload is protected, not the IP header, In Tunnel Mode, where the entire original IP packet is encapsulated with a new packet header added.

ESP in transport mode does not provide integrity and authentication for the entire IP packet.

ESP in Transport mode

In this, the ESP header is inserted after the original IP header. The ESP trailer is added after the IP trailer. ESP trailers contain the padding information. It is mostly used in a host-to-host scenario, where the data and security endpoints are the same.

The diagram for ESP in transport mode is as follows −

ESP in Tunnel mode

Tunnel mode encapsulation builds a new IP header containing the source and destination address of the security endpoints. In this mode, the outer IP header reflects the source and destination of the security endpoints, which might or might not be the same as the original source and destination IP address of the data connection.

The diagram for ESP in Tunnel mode is as follows −


The major differences between AH and ESP are as follows −

Difference BasisAuthentication HeaderEncapsulating Security Payload

It provides a mechanism for Sender Data Origin Authentication. So it cannot provide data Confidentiality/Encryption.

Provides Data Authentication and Data Privacy/Encryption and So it ensures both Confidentiality and Integrity for Packet Payload.
Difference in Authentication Process Coverage
It authenticates the entire IP packet, including the outer IP header.
It authenticates only the IP datagram portion of the IP packet.
Working through NATed network
It will not work through a NATed network as it hashes both the payload and header of a packet while NAT changes the IP header of a packet during translation.
It uses a hash algorithm for data integrity which does not include the IP header of the packet, thus ESP will work normally through a NATed device.
Updated on 15-Sep-2021 07:42:58