Network Security – Application Layer


Various business services are now offered online though client-server applications. The most popular forms are web application and e-mail. In both applications, the client communicates to the designated server and obtains services.

While using a service from any server application, the client and server exchange a lot of information on the underlying intranet or Internet. We are aware of fact that these information transactions are vulnerable to various attacks.

Network security entails securing data against attacks while it is in transit on a network. To achieve this goal, many real-time security protocols have been designed. Such protocol needs to provide at least the following primary objectives −

  • The parties can negotiate interactively to authenticate each other.
  • Establish a secret session key before exchanging information on network.
  • Exchange the information in encrypted form.

Interestingly, these protocols work at different layers of networking model. For example, S/MIME protocol works at Application layer, SSL protocol is developed to work at transport layer, and IPsec protocol works at Network layer.

Application Layer

In this chapter, we will discuss different processes for achieving security for e-mail communication and associated security protocols. The method for securing DNS is covered subsequently. In the later chapters, the protocols to achieve web security will be described.

E-mail Security

Nowadays, e-mail has become very widely used network application. Let’s briefly discuss the e-mail infrastructure before proceeding to know about e-mail security protocols.

E-mail Infrastructure

The simplest way of sending an e-mail would be sending a message directly from the sender’s machine to the recipient’s machine. In this case, it is essential for both the machines to be running on the network simultaneously. However, this setup is impractical as users may occasionally connect their machines to the network.

Hence, the concept of setting up e-mail servers arrived. In this setup, the mail is sent to a mail server which is permanently available on the network. When the recipient’s machine connects to the network, it reads the mail from the mail server.

In general, the e-mail infrastructure consists of a mesh of mail servers, also termed as Message Transfer Agents (MTAs) and client machines running an e-mail program comprising of User Agent (UA) and local MTA.

Typically, an e-mail message gets forwarded from its UA, goes through the mesh of MTAs and finally reaches the UA on the recipient’s machine.

E-mail Infrastructure

The protocols used for e-mail are as follows −

  • Simple mail Transfer Protocol (SMTP) used for forwarding e-mail messages.

  • Post Office Protocol (POP) and Internet Message Access Protocol (IMAP) are used to retrieve the messages by recipient from the server.


Basic Internet e-mail standard was written in 1982 and it describes the format of e-mail message exchanged on the Internet. It mainly supports e-mail message written as text in basic Roman alphabet.

By 1992, the need was felt to improve the same. Hence, an additional standard Multipurpose Internet Mail Extensions (MIME) was defined. It is a set of extensions to the basic Internet E-mail standard. MIME provides an ability to send e-mail using characters other than those of the basic Roman alphabet such as Cyrillic alphabet (used in Russian), the Greek alphabet, or even the ideographic characters of Chinese.

Another need fulfilled by MIME is to send non-text contents, such as images or video clips. Due to this features, the MIME standard became widely adopted with SMTP for e-mail communication.

E-Mail Security Services

Growing use of e-mail communication for important and crucial transactions demands provision of certain fundamental security services as the following −

  • Confidentiality − E-mail message should not be read by anyone but the intended recipient.

  • Authentication − E-mail recipient can be sure of the identity of the sender.

  • Integrity − Assurance to the recipient that the e-mail message has not been altered since it was transmitted by the sender.

  • Non-repudiation − E-mail recipient is able to prove to a third party that the sender really did send the message.

  • Proof of submission − E-mail sender gets the confirmation that the message is handed to the mail delivery system.

  • Proof of delivery − Sender gets a confirmation that the recipient received the message.

Security services such as privacy, authentication, message integrity, and non-repudiation are usually provided by using public key cryptography.

Typically, there are three different scenarios of e-mail communication. We will discuss the methods of achieving above security services in these scenarios.

One-to-One E-mail

In this scenario, the sender sends an e-mail message to only one recipient. Usually, not more than two MTA are involved in the communication.

One-to-One E-mail

Let’s assume a sender wants to send a confidential e-mail to a recipient. The provision of privacy in this case is achieved as follows −

  • The sender and receiver have their private-public keys as (SPVT, SPUB) and (RPVT, RPUB) respectively.

  • The sender generates a secret symmetric key, KS for encryption. Though the sender could have used RPUB for encryption, a symmetric key is used to achieve faster encryption and decryption.

  • The sender encrypts message with key KS and also encrypts KS with public key of the recipient, RPUB.

  • The sender sends encrypted message and encrypted KS to the recipient.

  • The recipient first obtains KS by decrypting encoded KS using his private key, RPVT.

  • The recipient then decrypts message using the symmetric key, KS.

Confidential E-mail

If message integrity, authentication, and non-repudiation services are also needed in this scenario, the following steps are added to the above process.

  • The sender produces hash of message and digitally signs this hash with his private key, SPVT.

  • The sender sends this signed hash to the recipient along with other components.

Message Integrity Authentication and Non-repudiation
  • The recipient uses public key SPUB and extracts the hash received under the sender’s signature.

  • The recipient then hashes the decrypted message and now compares the two hash values. If they match, message integrity is considered to be achieved.

  • Also, the recipient is sure that the message is sent by the sender (authentication). And lastly, the sender cannot deny that he did not send the message (non-repudiation).

One-to-Multiple Recipients E-mail

In this scenario, the sender sends an e-mail message to two or more recipients. The list is managed by the sender’s e-mail program (UA + local MTA). All recipients get the same message.

One-to-Multiple Recipients E-mail

Let’s assume, the sender wants to send confidential e-mail to many recipients (say R1, R2, and R3). The provision of privacy in this case is achieved as follows −

  • The sender and all recipients have their own pair of private-public keys.

  • The sender generates a secret symmetric key, Ks and encrypts the message with this key.

  • The sender then encrypts KS multiple times with public keys of R1, R2, and R3, getting R1PUB(KS), R2PUB(KS), and R3PUB(KS).

  • The sender sends encrypted message and corresponding encrypted KS to the recipient. For example, recipient 1 (R1) receives encrypted message and R1PUB(KS).

  • Each recipient first extracts key KS by decrypting encoded KS using his private key.

  • Each recipient then decrypts the message using the symmetric key, KS.

For providing the message integrity, authentication, and non-repudiation, the steps to be followed are similar to the steps mentioned above in one-to-one e-mail scenario.

One-to-Distribution List E-mail

In this scenario, the sender sends an e-mail message to two or more recipients but the list of recipients is not managed locally by the sender. Generally, the e-mail server (MTA) maintains the mailing list.

The sender sends a mail to the MTA managing the mailing list and then the mail is exploded by MTA to all recipients in the list.

One-to-Distribution List E-mail

In this case, when the sender wants to send a confidential e-mail to the recipients of the mailing list (say R1, R2, and R3); the privacy is ensured as follows −

  • The sender and all recipients have their own pair of private-public keys. The Exploder Server has a pair of private-public key for each mailing list (ListPUB, ListPVT) maintained by it.

  • The sender generates a secret symmetric key Ks and then encrypts the message with this key.

  • The sender then encrypts KS with the public key associated with the list, obtains ListPUB(KS).

  • The sender sends encrypted message and ListPUB(KS). The exploder MTA decrypts ListPUB(KS) using ListPVT and obtains KS.

  • The exploder encrypts KS with as many public keys as there are members in the list.

  • The Exploder forwards the received encrypted message and corresponding encrypted KS to all recipients in the list. For example, the Exploder forwards the encrypted message and R1PUB(KS) to recipient 1 and so on.

Encrypted Message

For providing the message integrity, authentication, and non-repudiation the steps to be followed are similar as given in case of one-to-one e-mail scenario.

Interestingly, the e-mail program employing above security method for securing e-mail is expected to work for all the possible scenarios discussed above. Most of the above security mechanisms for e-mail are provided by two popular schemes, Pretty Good Privacy (PGP) and S/MIME. We discuss both in the following sections.


Pretty Good Privacy (PGP) is an e-mail encryption scheme. It has become the de-facto standard for providing security services for e-mail communication.

As discussed above, it uses public key cryptography, symmetric key cryptography, hash function, and digital signature. It provides −

  • Privacy
  • Sender Authentication
  • Message Integrity
  • Non-repudiation

Along with these security services, it also provides data compression and key management support. PGP uses existing cryptographic algorithms such as RSA, IDEA, MD5, etc., rather than inventing the new ones.

Working of PGP

Working of PGP
  • Hash of the message is calculated. (MD5 algorithm)

  • Resultant 128 bit hash is signed using the private key of the sender (RSA Algorithm).

  • The digital signature is concatenated to message, and the result is compressed.

  • A 128-bit symmetric key, KS is generated and used to encrypt the compressed message with IDEA.

  • KS is encrypted using the public key of the recipient using RSA algorithm and the result is appended to the encrypted message.

The format of PGP message is shown in the following diagram. The IDs indicate which key is used to encrypt KS and which key is to be used to verify the signature on the hash.

PGP Message Format

In PGP scheme, a message in signed and encrypted, and then MIME is encoded before transmission.

PGP Certificate

PGP key certificate is normally established through a chain of trust. For example, A’s public key is signed by B using his public key and B’s public key is signed by C using his public key. As this process goes on, it establishes a web of trust.

In a PGP environment, any user can act as a certifying authority. Any PGP user can certify another PGP user's public key. However, such a certificate is only valid to another user if the user recognizes the certifier as a trusted introducer.

Several issues exist with such a certification method. It may be difficult to find a chain leading from a known and trusted public key to desired key. Also, there might be multiple chains which can lead to different keys for desired user.

PGP can also use the PKI infrastructure with certification authority and public keys can be certified by CA (X.509 certificate).


S/MIME stands for Secure Multipurpose Internet Mail Extension. S/MIME is a secure e-mail standard. It is based on an earlier non-secure e-mailing standard called MIME.

Working of S/MIME

S/MIME approach is similar to PGP. It also uses public key cryptography, symmetric key cryptography, hash functions, and digital signatures. It provides similar security services as PGP for e-mail communication.

The most common symmetric ciphers used in S/MIME are RC2 and TripleDES. The usual public key method is RSA, and the hashing algorithm is SHA-1 or MD5.

S/MIME specifies the additional MIME type, such as “application/pkcs7-mime”, for data enveloping after encrypting. The whole MIME entity is encrypted and packed into an object. S/MIME has standardized cryptographic message formats (different from PGP). In fact, MIME is extended with some keywords to identify the encrypted and/or signed parts in the message.

S/MIME relies on X.509 certificates for public key distribution. It needs top-down hierarchical PKI for certification support.

Employability of S/MIME

Due to the requirement of a certificate from certification authority for implementation, not all users can take advantage of S/MIME, as some may wish to encrypt a message, with a public/private key pair. For example, without the involvement or administrative overhead of certificates.

In practice, although most e-mailing applications implement S/MIME, the certificate enrollment process is complex. Instead PGP support usually requires adding a plug-in and that plug-in comes with all that is needed to manage keys. The Web of Trust is not really used. People exchange their public keys over another medium. Once obtained, they keep a copy of public keys of those with whom e-mails are usually exchanged.

Implementation layer in network architecture for PGP and S/MIME schemes is shown in the following image. Both these schemes provide application level security of for e-mail communication.

Employability of S/MIME

One of the schemes, either PGP or S/MIME, is used depending on the environment. A secure e-email communication in a captive network can be provided by adapting to PGP. For e-mail security over Internet, where mails are exchanged with new unknown users very often, S/MIME is considered as a good option.

DNS Security

In the first chapter, we have mentioned that an attacker can use DNS Cache Poisoning to carry out an attack on the target user. Domain Name System Security Extensions (DNSSEC) is an Internet standard that can foil such attacks.

Vulnerability of Standard DNS

In a standard DNS scheme, whenever the user wants to connect to any domain name, his computer contacts the DNS server and looks up the associated IP address for that domain name. Once IP address is obtained, the computer then connects to that IP address.

In this scheme, there is no verification process involved at all. A computer asks its DNS server for the address associated with a website, the DNS server responds with an IP address, and your computer undoubtedly accepts it as legitimate response and connects to that website.

A DNS lookup actually happens in several stages. For example, when a computer asks for “”, a DNS lookup is performed in several stages −

  • The computer first asks the local DNS server (ISP provided). If ISP has this name in its cache, it responds else forwards the query to “root zone directory” where it can find “.com.” and root zone replies.

  • Based on the reply, the computer then asks the “.com” directory where it can find “”

  • Based on the information received, the computer inquires “” where it can find www.

Standard DNS Vulnerability

DNSSEC Defined

DNS lookup, when performed using DNSSEC, involves signing of replies by the responding entity. DNSSEC is based on public-key cryptography.

In DNSSEC standard, every DNS zone has a public/private key pair. All information sent by a DNS server is signed with the originating zone’s private key for ensuring authenticity. DNS clients need to know the zone’s public keys to check the signatures. Clients may be preconfigured with the public keys of all the top-level domains, or root DNS.

With DNSSEC, the lookup process goes as follows −

  • When your computer goes to ask the root zone where it can find .com, the reply is signed by the root zone server.

  • Computer checks the root zone’s signing key and confirms that it is the legitimate root zone with true information.

  • In the reply, the root zone provides the information on the signing key of .com zone server and its location, allowing the computer to contact the .com directory and ensuring it is legitimate.

  • The .com directory then provides the signing key and information for, allowing it to contact and verify that you are connected to the real, as confirmed by the zones above it.

  • The information sent is in the form of Resource Record Set (RRSets). The example of RRSet for domain “” in top-level “.com” server is shown in the following table.

Domain Name Time to live Type Value 86400 NS 86400 A 36..1.2.3 86400 KEY 3682793A7B73F731029CE2737D... 86400 SIG 86947503A8B848F5272E53930C...
  • The KEY record is a public key of “”.

  • The SIG record is the top-level .com server's signed hash of the fields NS, A, and KEY records to verify their authenticity. Its value is Kcompvt(H(NS,A,KEY)).

Thus, it is considered that when DNSSEC is fully rolled out, the user’s computer is able to confirm that DNS responses are legitimate and true, and avoid DNS attacks launched through DNS cache poisoning.


The process of securing e-mails ensures the end-to-end security of the communication. It provides security services of confidentiality, sender authentication, message integrity, and non-repudiation.

Two schemes have been developed for e-mail security: PGP and S/MIME. Both these schemes use secret-key and public-key cryptography.

Standard DNS lookup is vulnerable to the attacks such as DNS spoofing/cache poisoning. Securing DNS lookup is feasible through the use of DNSSEC which employs the public-key cryptography.

In this chapter, we discussed the mechanisms used at application layer to provide network security for end-to-end communication.