What is ESP in tunnel and transport mode and the difference between AH and ESP?

Encapsulating Security Payload (ESP) is one of the two main protocols in IPSec that provides encryption, authentication, and integrity services for IP packets. ESP protects the data payload through encryption while ensuring data integrity and authenticity using cryptographic algorithms.

ESP operates in two distinct modes: transport mode for host-to-host communication and tunnel mode for network-to-network communication. The choice of mode determines which parts of the IP packet are protected and how the packet structure is modified.

ESP in Transport Mode

In transport mode, the ESP header is inserted directly after the original IP header, and the ESP trailer is added after the data payload. This mode is primarily used for end-to-end communication between two hosts where both endpoints handle IPSec processing.

Key characteristics of transport mode:

  • Original IP header preserved − The source and destination IP addresses remain unchanged

  • Payload protection only − ESP encrypts and authenticates the data payload, not the IP header

  • Lower overhead − Less additional header information compared to tunnel mode

ESP Transport Mode Packet Structure Original IP Header ESP Header Encrypted Payload + ESP Trailer ESP Auth Protected by ESP (Encrypted and Authenticated)

ESP in Tunnel Mode

Tunnel mode creates an entirely new IP packet by encapsulating the original packet within new IP headers. This mode is commonly used in site-to-site VPN connections where security gateways handle IPSec processing on behalf of internal hosts.

Key characteristics of tunnel mode:

  • Complete packet encapsulation − The entire original IP packet becomes the payload of a new packet

  • New IP headers − Source and destination addresses reflect the security gateway endpoints

  • Enhanced security − Hides the original source and destination from network analysis

ESP Tunnel Mode Packet Structure New IP Header ESP Header Original IP Header Original Payload + ESP Trailer ESP Auth Protected by ESP (Encrypted and Authenticated) Entire original packet is encrypted and encapsulated

AH vs ESP Comparison

Authentication Header (AH) and ESP serve different security purposes in IPSec. While both provide authentication, ESP additionally offers encryption capabilities that AH lacks.

Feature Authentication Header (AH) Encapsulating Security Payload (ESP)
Primary Function Data origin authentication and integrity verification only Authentication, integrity, and confidentiality (encryption)
Authentication Coverage Authenticates entire IP packet including outer IP header Authenticates only the ESP payload portion
NAT Compatibility Incompatible with NAT devices due to IP header modification Works through NAT as it doesn't authenticate IP headers
Encryption No encryption provided Strong encryption for data confidentiality

Use Cases

ESP Transport Mode is ideal for secure host-to-host communication within trusted networks, while ESP Tunnel Mode is preferred for site-to-site VPN connections and securing traffic across untrusted networks. Organizations typically choose ESP over AH when data confidentiality is required alongside authentication.

Conclusion

ESP provides comprehensive IPSec security through both transport and tunnel modes, offering encryption and authentication services. Unlike AH, ESP ensures data confidentiality while maintaining compatibility with NAT devices, making it the preferred choice for most VPN implementations.

Updated on: 2026-03-16T23:36:12+05:30

6K+ Views

Kickstart Your Career

Get certified by completing the course

Get Started
Advertisements