compsci 356 computer network architectures lecture 23 application layer protocols chapter 9 1 n.
Skip this Video
Download Presentation
Xiaowei Yang xwy@cs.duke

Loading in 2 Seconds...

play fullscreen
1 / 47

Xiaowei Yang xwy@cs.duke - PowerPoint PPT Presentation

  • Uploaded on

CompSci 356: Computer Network Architectures Lecture 23: Application Layer Protocols Chapter 9.1. Xiaowei Yang Overview. Application-layer protocols Applications vs. application-layer protocols Tailoring the protocol to the application Electronic mail

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Xiaowei Yang xwy@cs.duke' - jatin

Download Now An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
compsci 356 computer network architectures lecture 23 application layer protocols chapter 9 1

CompSci 356: Computer Network ArchitecturesLecture 23: Application Layer ProtocolsChapter 9.1

Xiaowei Yang

  • Application-layer protocols
    • Applications vs. application-layer protocols
    • Tailoring the protocol to the application
  • Electronic mail
    • E-mail messages, and multipurpose Internet Mail extensions (MIME)
    • E-mail addresses, and the role of DNS
    • E-mail servers and user agents
  • Electronic mail protocols
    • Transferring e-mail messages between servers
      • Simple Mail Transfer Protocol (SMTP)
    • Retrieving e-mail messages (POP, IMAP, and HTTP)
      • Client-side protocols
application layer protocols
Application-Layer Protocols
  • Network applications run on end systems
    • They depend on the network to provide a service
    • … but cannot run software on the network elements
  • Network applications run on multiple machines
    • Different end systems communicate with each other
    • Software is often written by multiple parties
  • Leading to a need to explicitly define a protocol
    • Types of messages (e.g., requests and responses)
    • Message syntax (e.g., fields, and how to delineate)
    • Semantics of the fields (i.e., meaning of the information)
    • Rules for when and how a process sends messages
    • Platform and programming language independent
application programs vs protocols
Application Programs vs. Protocols
  • Application-layer protocol is just one piece
    • Defining how the end hosts communicate
  • Example: World Wide Web
    • HyperText Transfer Protocol is the protocol
    • But the Web includes other components, such as document formats (HTML), Web browsers, servers, …
  • Example: electronic mail
    • Simple Mail Transfer Protocol (SMTP) is the protocol
    • But e-mail includes other components, such as mail servers, user mailboxes, mail readers
  • Open standards vs. vertical integration
    • Firefox vs. HTTP, Thunderbird vs. SMTP
the structure of a content rich application layer protocol
The structure of a content-rich application-layer protocol
  • A companion protocol that specifies the data format
  • A protocol that describes the interactions between two end systems
  • Examples
    • HTTP and HTML
    • SMTP and RFC 2822 (Internet Text Messages), and Multipurpose Internet Mail Extensions (MIME)
  • Simpler protocols may specify both data format and interactions in one protocol
    • Eg. DNS
      • Request/response
      • The record has a simple format
    • Telnet
    • FTP
protocols tailored to the application
Protocols Tailored to the Application
  • SMTP: sending e-mail to a remote mail server
    • Sending mail server transmits an e-mail message to a mail server running on a remote machine
    • Each server in the path adds its identifier to the message
    • Single TCP connection for control and data
    • SMTP replaced the earlier use of FTP for e-mail
  • HTTP: satisfying requests based on a global URL
    • Client sends a request with method, URL, and meta-data
    • … and the server applies the request to the resource and returns the response, including meta-data
    • Single TCP connection for control and data
protocols tailored to the application1
Protocols Tailored to the Application
  • Telnet: interacting with account on remote machine
    • Client simply relays user keystrokes to the server
    • … and server simply relays any output to the client
    • TCP connection persists for duration of the login session
    • Network Virtual Terminal (NVT) format for transmitting ASCII data, and control information (e.g., End-of-Line delimiter)
  • FTP: copying files between accounts
    • Client connects to remote machine, “logs in, and issues commands for transferring files to/from the account
    • … and server responds to commands and transfers files
    • Separate TCP connections for control and data
    • Control connection uses the same NVT format as Telnet
comparing the protocols
Comparing the Protocols
  • Commands and replies
    • Telnet sends commands in binary, whereas the other protocols are text based
    • Many of the protocols have similar request methods and response codes
  • Data types
    • Telnet, FTP, and SMTP transmit text data in standard U.S. 7-bit ASCII
    • FTP also supports transfer of data in binary form
    • SMTP uses MIME standard for sending non-text data
    • HTTP incorporates some key aspects of MIME (e.g., classification of data formats)
comparing the protocols continued
Comparing the Protocols (Continued)
  • Transport
    • Telnet, FTP, SMTP, and HTTP all depend on reliable transport protocol
    • Telnet, SMTP, and HTTP use a single TCP connection
    • … but FTP has separate control and data connections
  • State
    • In Telnet, FTP, and SMTP, the server retains information about the session with the client
    • E.g., FTP server remembers client’s current directory
    • In contrast, HTTP servers are stateless
      • Request/response
      • Cookies add state
reflecting on application layer protocols
Reflecting on Application-Layer Protocols
  • Protocols are tailored to the applications
    • Each protocol is customized to a specific need
  • Protocols have many key similarities
    • Each new protocol was influenced by the previous ones
    • New protocols commonly borrow from the older ones
  • Protocols depend on the same underlying substrate
    • Ordered reliable stream of bytes (i.e., TCP)
    • Domain Name System (DNS)
  • Relevance of the protocol standards process
    • Important for interoperability across implementations
    • Yet, not necessary if same party writes all of the software
    • …which is increasingly common (e.g., P2P software)
e mail message
E-Mail Message
  • E-mail messages have two parts
    • A header, in 7-bit U.S. ASCII text
    • A body, also represented in 7-bit U.S. ASCII text
  • Header
    • Lines with “type: value”
    • “To:”
    • “Subject: Hello!”
  • Body
    • The text message
    • No particular structure or meaning





e mail message format rfc 822
E-Mail Message Format (RFC 822)
  • E-mail messages have two parts
    • A header, in 7-bit U.S. ASCII text
    • A body, also represented in 7-bit U.S. ASCII text
  • Header
    • Series of lines ending in carriage return and line feed
    • Each line contains a type and value, separated by “:”
    • E.g., “To:” and “Subject: Hello”
    • Additional blank line before the body begins
  • Body
    • Series of text lines with no additional structure/meaning
    • Conventions arose over time (e.g., e-mail signatures)
limitation sending non text data
Limitation: Sending Non-Text Data
  • E-mail body is 7-bit U.S. ASCII
    • What about non-English text?
    • What about binary files (e.g., images and executables)?
  • Solution: convert non-ASCII data to ASCII
    • Base64 encoding: map each group of three bytes into four printable U.S.-ASCII characters
    • Uuencode (Unix-to-Unix Encoding) was widely used
      • Output an encoded text file
      • Begin and ending line shows the encoding algorithm
    • 1st:: uuencode –m cat.txt < lecture.ppt; -m: MIME Base64
    • 2nd: uuencode cat.txt < lecture.ppt.; historical algorithm
    • Uudecode produces cat.txt

begin-base64 644 cat.txt



begin 644 cat.txt



limitation sending multiple items
Limitation: Sending Multiple Items
  • Users often want to send multiple pieces of data
    • Multiple images, powerpoint files, or e-mail messages
    • Yet, e-mail body is a single, uninterpreted data chunk
  • Example: e-mail digests
    • Encapsulating several e-mail messages into one aggregate messages (i.e., a digest)
    • Commonly used on high-volume mailing lists
  • Conventions arose for how to delimit the parts
    • E.g., well-known separator strings between the parts
    • Yet, having a standard way to handle this is better
multipurpose internet mail extensions
Multipurpose Internet Mail Extensions
  • Additional headers to describe the message body
    • MIME-Version: the version of MIME being used
    • Content-Type: the type of data contained in the message
    • Content-Transfer-Encoding: how the data are encoded
  • Definitions for a set of content types and subtypes
    • E.g., image with subtypes gif and jpeg
    • E.g., text with subtypes plain, html, and richtext
    • E.g., application with subtypes postscript and msword
    • E.g., multipart for messages with multiple data types
  • A way to encode the data in ASCII format
    • Base64 encoding, as in uuencode/uudecode

MIME version

MIME-Version: 1.0

Subject: picture of my dog

From: Xiaowei Yang <>

To: Xiaowei Yang <>

Content-Type: multipart/mixed; boundary=00151759359af49a3e04a1271622


Content-Type: text/plain; charset=ISO-8859-1


This is a picture of my dog.




Content-Type: image/jpeg; name="Neo-Run.jpg"

Content-Disposition: attachment; filename="Neo-Run.jpg"

Content-Transfer-Encoding: base64

X-Attachment-Id: f_gmmpmaoi0


type and subtype

type and subtype

Ex: Email Msg

using MIME

Encoded data

distribution of content types
Distribution of Content Types
  • Content types in my own e-mail archive
    • Searched on “Content-Type”, not case sensitive
    • Extracted the value field, and counted unique types
    • At UNIX command line: grep -i Content-Type * | cut -d" " -f2 | sort | uniq -c | sort –nr
  • Out of 44343 matches
    • 25531: text/plain
    • 7470: multipart to send attachments
    • 4230: text/html
    • 759: application/pdf
    • 680: application/msword
    • 479: application/octet-stream
    • 292: image (mostly jpeg, and some gif, tiff, and bmp)
e mail addresses
E-Mail Addresses
  • Components of an e-mail address
    • Local mailbox (e.g., xwy or bob.flower)
    • Domain name (e.g.,
  • Domain name is not necessarily the mail server
    • Mail server may have longer/cryptic name
      • E.g., vs.
    • Multiple servers may exist to tolerate failures
      • E.g., vs. authdns{1,2,3,4}
  • Identifying the mail server for a domain
    • DNS query asking for MX records (Mail eXchange)
      • E.g., nslookup –q=mx, dig MX
    • Then, a regular DNS query to learn the IP address
mail servers and user agents









mail server

mail server

Mail Servers and User Agents
  • Mail servers
    • Always on and always accessible
    • Transferring e-mail to and from other servers
  • User agents
    • Sometimes on and sometimes accessible
    • Intuitive interface for the user
smtp store and forward protocol





mail server

mail server

SMTP Store-and-Forward Protocol
  • Messages sent through a series of servers
    • A server stores incoming messages in a queue
    • … to await attempts to transmit them to the next hop
  • If the next hop is not reachable
    • The server stores the message and tries again later
  • Each hop adds its identity to the message
    • By adding a “Received” header with its identity
    • Helpful for diagnosing problems with e-mail
example with received header
Example With Received Header


Received: by with SMTP id i17cs431026rvd; Wed, 14 Apr 2010 10:23:53 -0700 (PDT)

Received: by with SMTP id k3mr495928rvm.141.1271265832656; Wed, 14 Apr 2010 10:23:52 -0700 (PDT)

Return-Path: <>

Received: from ( []) by with ESMTP id bh13si1413883ibb.64.2010.; Wed, 14 Apr 2010 10:23:51 -0700 (PDT)

Received-SPF: pass ( domain of designates as permitted sender) client-ip=;

Authentication-Results:; spf=pass ( domain of designates as permitted sender); dkim=pass (test mode)

Received: from (localhost []) by (8.14.2/8.14.2) with ESMTP id o3EHNoWw023327 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for <>; Wed, 14 Apr 2010 13:23:50 -0400 (EDT)

X-DKIM: Sendmail DKIM Filter v2.8.3 o3EHNoWw023327 Authentication-Results:; dkim=pass (1024-bit key); x-dkim-adsp=none

Received: (from xwy@localhost) by (8.14.2/8.14.2/Submit) id o3EHNoVQ023326 for; Wed, 14 Apr 2010 13:23:50 -0400 (EDT)

X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on

X-Spam-Level: X-Spam-Status: No, score=-0.6 required=5.0 tests=AWL,DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DK_SIGNED,FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS shortcircuit=no autolearn=ham version=3.3.1

X-Spam-CMAE-Analysis: v=1.1 cv=Pto2NXHwdLBwzRB8+vd5Qd2kiNuFoxIMnIFOeK+fhJM= c=1 sm=0 a=aut-g5bx_9gA:10 a=e5d5pRNFdwYY0VJDlD0A:9 a=qoy35m6o3OoPNk0ReNInnvtYc2AA:4 a=wPNLvfGTeEIA:10 a=nlx8PpsGfoIU8a_ktGoA:9 a=n0XNdzWZCnTTayHvvAlEaa-HHpcA:4 a=HpAAvcLHHh0Zw7uRqdWCyQ==:117

Received: from ( []) by (8.14.2/8.14.2) with ESMTP id o3EHNmL0023285 for <>; Wed, 14 Apr 2010 13:23:48 -0400 (EDT)

X-DKIM: Sendmail DKIM Filter v2.8.3 o3EHNmL0023285 Received: by wyf23 with SMTP id 23so172877wyf.6 for <>; Wed, 14 Apr 2010 10:23:42 -0700 (PDT)

DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=gamma; h=domainkey-signature:mime-version:received:date:received:message-id :subject:from:to:content-type; bh=Xd14scK+5arOf0viCr7Xq/aCwNDuNRwZBCvFd/VF6jA=; b=iS7Cz75uCioxhPwiCCYexDG42vvUSD9/cpSDHQvXpxMOY1cL4RL3/IvOEvbEcit/j1 JoULkJzawiPY4dZ1W/mbU747gdG+1WYjLCGTR7+LTS+N2xy7CY74hcJvz52Ehq+zNbBj pyOzZRJFFlHp2ub7wUos56T4dvMyxwDPHgYjk= DomainKey-Signature: a=rsa-sha1; c=nofws;; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=qcC9huJFV2OjVf2/s3zD6ytA7w2YGCjaneLhX9VQG0/kjD32sRYknnLIGYmTpu7qNo pzmjuI85dkDWu42xVRpLb0TpVNdvwGQNqEFONLT6IkYxUUjqo97hbFEoJ1fecmAYCBfR pF2/GPy0znxe+cjahQGKzIHfI3O/55N10/HoE= MIME-Version: 1.0 Received: by with HTTP; Wed, 14 Apr 2010 10:23:42 -0700 (PDT) Date: Wed, 14 Apr 2010 13:23:42 -0400 Received: by with SMTP id w59mr4894770wee.186.1271265822165; Wed, 14 Apr 2010 10:23:42 -0700 (PDT) Message-ID: <> Subject: discussion list From: Xin Wu <>

To: Xiaowei Yang <>

Content-Type: multipart/alternative; boundary=0016e6d644c9becd43048435a332


multiple server hops
Multiple Server Hops
  • Typically at least two mail servers
    • Sending and receiving sides
  • May be more
    • Separate servers for key functions
      • Spam filtering
      • Virus scanning
    • Servers that redirect the message
      • From to
      • Messages to go through extra hops
    • Electronic mailing lists
      • Mail delivered to the mailing list’s server
      • … and then the list is expanded to each recipient
electronic mailing lists
Electronic Mailing Lists
  • Community of users reachable by one address
    • Allows groups of people to receive the messages
  • Exploders
    • Explode a single e-mail message into multiple messages
    • One copy of the message per recipient
  • Handling bounced messages
    • Mail may bounce for several reasons
    • E.g., recipient mailbox does not exist; resource limits
  • E-mail digests
    • Sending a group of mailing-list messages at once
    • Messages delimited by boundary strings
    • … or transmitted using multiple/digest format
simple mail transfer protocol





mail server

mail server





Simple Mail Transfer Protocol
  • Client-server protocol
    • Client is the sending mail server
    • Server is the receiving mail server
  • Reliable data transfer
    • Built on top of TCP (on port 25)
  • Push protocol
    • Sending server pushes the file to the receiving server
    • … rather than waiting for the receiver to request it
simple mail transfer protocol cont
Simple Mail Transfer Protocol (Cont.)
  • Command/response interaction
    • Commands: ASCII text
    • Response: three-digit status code and phrase
  • Synchronous
    • Sender awaits response from a command
    • … before issuing the next command
    • Though pipelining of commands was added later
  • Three phases of transfer
    • Handshaking (greeting)
    • Transfer of messages
    • Closure
scenario alice sends message to bob
1) Alice uses UA to compose message “to”

2) Alice’s UA sends message to her mail server; message placed in message queue

3) Client side of SMTP opens TCP connection with Bob’s mail server

4) SMTP client sends Alice’s message over the TCP connection

5) Bob’s mail server places the message in Bob’s mailbox

6) Bob invokes his user agent to read message



Scenario: Alice Sends Message to Bob













sample smtp interaction
Sample SMTP interaction
  • Show an example via telnet
  • Use dig to find the mail server
    • dig MX
  • telnet mailserver 25
sample smtp interaction1
Sample SMTP interaction

S: 220


S: 250 Hello, pleased to meet you


S: 250 Sender ok

C: RCPT TO: <>

S: 250 ... Recipient ok


S: 354 Enter mail, end with "." on a line by itself

C: Do you like ketchup?

C: How about pickles?

C: .

S: 250 Message accepted for delivery


S: 221 closing connection

try smtp for yourself
Try SMTP For Yourself
  • Running SMTP
    • Run “telnet servername 25” at UNIX prompt
    • See 220 reply from server
    • Enter HELO, MAIL FROM, RCPT TO, DATA commands
  • Thinking about spoofing?
    • Very easy
    • Just forge the argument of the “FROM” command
    • … leading to all sorts of problems with spam
  • Spammers can be even more clever
    • E.g., using open SMTP servers to send e-mail
    • E.g., forging the “Received” header
retrieving e mail from the server
Retrieving E-Mail From the Server
  • Server stores incoming e-mail by mailbox
    • Based on the “From” field in the message
  • Users need to retrieve e-mail
    • Asynchronous from when the message was sent
    • With a way to view the message and reply
    • With a way to organize and store the messages
  • In the old days…
    • User logged on to the machine where mail was delivered
    • Users received e-mail on their main work machine
influence of pcs on e mail retrieval
Influence of PCs on E-Mail Retrieval
  • Separate machine for personal use
    • Users did not want to log in to remote machines
  • Resource limitations
    • Most PCs did not have enough resources to act as a full-fledged e-mail server
  • Intermittent connectivity
    • PCs only sporadically connected to the network
    • … due to dial-up connections, and shutting down of PC
    • Too unwieldy to have sending server keep trying
  • Led to the creation of Post Office Protocol (POP)
post office protocol pop
Post Office Protocol (POP)
  • POP goals
    • Support users with intermittent network connectivity
    • Allow them to retrieve e-mail messages when connected
    • … and view/manipulate messages when disconnected
  • Typical user-agent interaction with a POP server
    • Connect to the server
    • Retrieve all e-mail messages
    • Store messages on the user’s PCs as new messages
    • Delete the messages from the server
    • Disconnect from the server
  • User agent still uses SMTP to send messages
pop3 protocol
Authorization phase

Client commands:

user: declare username

pass: password

Server responses



Transaction phase, client:

list: list message numbers

retr: retrieve message by number

dele: delete


POP3 Protocol

S: +OK POP3 server ready

C: user bob

S: +OK

C: pass hungry

S: +OK user successfully logged on

C: list

S: 1 498

S: 2 912

S: .

C: retr 1

S: <message 1 contents>

S: .

C: dele 1

C: retr 2

S: <message 1 contents>

S: .

C: dele 2

C: quit

S: +OK POP3 server signing off

limitations of pop
Limitations of POP
  • Does not handle multiple mailboxes easily
    • Designed to put user’s incoming e-mail in one folder
  • Not designed to keep messages on the server
    • Instead, designed to download messages to the client
  • Poor handling of multiple-client access to mailbox
    • Increasingly important as users have home PC, work PC, laptop, cyber café computer, friend’s machine, etc.
  • High network bandwidth overhead
    • Transfers all of the e-mail messages, often well before they are read (and they might not be read at all!)
interactive mail access protocol imap
Interactive Mail Access Protocol (IMAP)
  • Supports connected and disconnected operation
    • Users can download message contents on demand
  • Multiple clients can connect to mailbox at once
    • Detects changes made to the mailbox by other clients
    • Server keeps state about message (e.g., read, replied to)
  • Access to MIME parts of messages & partial fetch
    • Clients can retrieve individual parts separately
    • E.g., text of a message without downloading attachments
  • Multiple mailboxes on the server
    • Client can create, rename, and delete mailboxes
    • Client can move messages from one folder to another
  • Server-side searches
    • Search on server before downloading messages
web based e mail
Web-Based E-Mail
  • User agent is an ordinary Web browser
    • User communicates with server via HTTP
    • E.g., Gmail, Yahoo mail, and Hotmail
  • Reading e-mail
    • Web pages display the contents of folders
    • … and allow users to download and view messages
    • “GET” request to retrieve the various Web pages
  • Sending e-mail
    • User types the text into a form and submits to the server
    • “POST” request to upload data to the server
    • Server uses SMTP to deliver message to other servers
  • Easy to send anonymous e-mail (e.g., spam)
  • Application-layer protocols
    • Applications vs. application-layer protocols
    • Tailoring the protocol to the application
  • Electronic mail
    • E-mail messages, and multipurpose Internet Mail extensions (MIME)
    • E-mail addresses, and role of DNS
    • E-mail servers and user agents
  • Electronic mail protocols
    • Transferring e-mail messages between servers
      • Simple Mail Transfer Protocol (SMTP)
    • Retrieving e-mail messages (POP, IMAP, and HTTP)
      • Client-side protocols
  • A reliable data connection between two processes on top of UDP which is unreliable
    • Handle Packet drop, reordering, corruption
    • Provide flow control
                  • Sliding Window (Both sender and receiver)
                  • Two independent data links each connection:

Link 1

Process A

Process B

Link 2

what can the program do
What can the program do

On machine linux21, run:

xwy@linux21$./reliable 6666 linux22:5555

[listening on UDP port 6666]


On machine linux22, run:

xwy@linux22$./reliable 5555 linux21:6666

[listening on UDP port 5555]


  • reliable.c: functions you need to fill in
  • rlib.c & rlib.h:
    • the library supporting code
    • Just need to know which functions you should call
  • reference: a solution program for relable
  • tester: Use it to debug (-v option) and grade your program

xwy@linux22> ./tester ./reliable

data flow
Data Flow







Library Functions

Library Functions

You cannot use printf() in your program, because it would be considered as data receive from the sender. Use fprintf(stderr, “…”, …) instead

Sent out into network

Receive from network

reliable c
  • reliable_state (also defined as rel_t)
    • A data structure to maintain the connection state for one reliable connection
    • Should include two slide window (one for sending and one for receiving)
    • Add as much things as you need for the connection
  • rel_create(), rel_destroy(), rel_recvpkt(), rel_read(), rel_output(), rel_timer()
    • Six functions you need to implement
    • Refer to the lab1 tutorial for details
some hints
Some Hints
  • The original code actually support two running mode: single connection mode and server/client mode. Only the first one is required
  • Only when two data links both terminated with EOF from STDIN (Ctrl+D in Linux)should the connection be destroyed. Remember to report EOF to conn_output.
  • To handle the big/little-endian. Use htonl()/htons() to write a header and ntohl()/ntohs() to read a header.
more hints
More Hints
  • Use while loop in the rel_read(). If conn_input() returns -1, handle the EOF packet and break from the loop. If conn_input() returns 0, simply break from the loop. If the sender’s window is full, break from the loop even if there is still more data from conn_input().
  • Call rel_read() to ask for more data after you received some ack packets and some slots in sender’s window become vacant again.