Ipsec

IPSec Standards and Protocols AH and ESP Matt Norris

IPSec Standards and Protocols AH and ESP IPSec is one of several VPN standards that have allowed secure, low-cost connectivity and data transport between locations over unsecured communication lines. Although IPSec is most commonly used for the Internet, its use has been extended to secure communications within local area networks between client and server and server to server. When designing and implementing a VPN solution, it is important to understand that IPSec is not a single protocol but is comprised of many protocols that can be combined to provide varying levels of protection. The premise of choosing and combining different protocols makes IPSec extremely flexible and manageable if the implementer understands the primary protocols and connection modes used in the IPSec standard. The two protocols that, individually or in tandem, form the backbone of IPSec are Authentication Header AH) and Encapsulating Security Payload (ESP). The two modes that an IPSec connection, known as Security Association (SA), can operate in are Tunnel Mode or Transport Mode.

What Is a SA (Security Association)? The SA is the “connection” between the IPSec peers that is required for secure data exchange and is configured for either Transport Mode or Tunnel Mode. It is composed of the negotiated parameters that will be used for data handling. SA parameters include the IP address of the peer; encryption algorithm; protocol format (AH or ESP); and security parameter index (SPI), a unique number assigned to each SA and used to manage multiple SAs. Once negotiated, each peer stores the SA parameters locally in a SA database (SAD). The SA is unidirectional, meaning a separate SA is needed for each inbound and outbound connection and bidirectional traffic therefore requires two SAs.

IPSec Protocols: AH and ESP AH (Authentication Header) is a format protocol defined in RFC 2402 that provides data authentication, integrity, and nonrepudiation but does not provide data confidentiality. AH adds security to the communication stream by encrypting nonvolatile fields of the IP header and creating a message digest value at origination that can be re-created and compared at the destination. The message digest value is created through the application of a one-way hash of the IP header and data portion of the packet. After that message digest value is created, an encrypted AH header is inserted between the original IP header and data portion of the packet. The encrypted AH header includes the message digest value and authentication information for the packet. The IP packet is then sent to the IPSec peer. The IPSec peer hashes the IP header and data portion of the packet to create a message digest value and compares it to the hash value in the AH header that it has decrypted. If the message digest values match, it ensures that the packet has not been modified in transit. If a packet is determined to have been modified in transit, it is rejected. AH can be utilized in either Transport Mode or Tunnel Mode because it protects the outer IP header whether it is the original header in Transport Mode or a new header created in Tunnel Mode.

AH Process

Protected IP Header and AH Header

ESP is a format protocol defined in RFC 2406 that provides data confidentiality (through encryption) and is typically what we think of when deploying a “secure” VPN solution. ESP can optionally provide integrity and data origin authentication through the use of a hash and can provide replay attack protection. ESP adds security to the communication stream by encrypting the data payload when in Transport Mode or by encrypting and encapsulating the entire IP packet when in Tunnel Mode as described in the following. ESP supports the use of symmetric encryption algorithms, including DES, 3DES, and AES, for confidentiality and the use of MD5 HMAC and SHA1 HMAC for data authentication and integrity. Similar to AH authentication and integrity, ESP creates a hash at the point of origination that can be compared at the destination. But unlike AH, ESP cannot protect the unencrypted IP header--which is why ESP and AH are commonly combined to add another level of protection.

IPSec Connection Modes: Transport and Tunnel Transport Mode: Protection is provided for the data in the IP packet through encryption but not for the IP header information, which remains unchanged. Transport Mode adds only a few bytes of information to each IP packet, in the form of an IPSec header, and it allows for quality-of-service (QoS) management on the network. Transport Mode is typically used when end-to-end encryption is required and supported by the peers and is deployed between or within locations. Transport Mode IP Packet (Utilizing ESP)

Typical Transport Mode Data Exchange

Tunnel Mode: Protection is provided for the entire IP packet, which is encrypted and then encapsulated in a new IP packet including a new IP header and an IPSec header. Tunnel Mode is typically used on IPSec gateway devices such as firewalls, routers, and VPN appliances connecting remote locations such as branch offices. The gateway acts as an IPSec proxy for the clients that are located behind the device. Clients forward IP packets to the gateway in the clear. The gateway device then encrypts the packet and forwards it to an IPSec peer, which in turn decrypts the packet and forwards it to the destination client. Tunnel Mode IP Packet (Utilizing ESP)

New IP Header

IPSec Header

IP Header

DATA

IP Header (Encrypted)

DATA (Encrypted)

Original Packet

Tunnel Mode Packet

Typical Tunnel Mode Data Exchange

After the protocols (AH and ESP) and the modes of transportation (Transport or Tunnel) are understood, designing a secure communication stream can become a more manageable task. Of course, these steps are only a few in the overall process and the architect of the secure communication design should continue to understand all phases and processes involved in VPN creation and its ongoing maintenance.