Servers and Networking Basics

1. Servers and A Little Bit of Networking Jeff Chase Duke University

2. Unix process view: data A process has multiple channelsfor data movement in and out of the process (I/O). I/O channels (“file descriptors”) stdin Process stdout The channels are typed. tty stderr Each channel is named by a file descriptor. pipe Thread socket The parent process and parent program set up and control the channels for a child (until exec). Program Files

3. Servers and the cloud Where is your application? Where is your data? Where is your OS? networked server “cloud” Cloud and Software-as-a-Service (SaaS) Rapid evolution, no user upgrade, no user data management. Agile/elastic deployment on clusters and virtual cloud utility-infrastructure.

4. Networked services: big picture client host NIC device Internet “cloud” client applications kernel network software server hosts with server applications Data is sent on the network as messages called packets.

5. Sockets client intsd = socket(<internet stream>); gethostbyname(“www.cs.duke.edu”); <make a sockaddr_instruct> <install host IP address and port> connect(sd, <sockaddr_in>); write(sd, “abcdefg”, 7); read(sd, ….); • The socket() system call creates a socket object. • Other calls establish connections between socket pairs (e.g, connect). • A file descriptor for a connected socket is bidirectional. • Write bytes at one end; read returns them at the other end. • The readsyscall blocks if the (stream) socket is “empty”. • The writesyscall blocks if the (stream) socket is “full”. • Both read and write fail if there is no valid connection. socket A socketis a buffered channel for passing data between processes over a network.

6. Sockets: client/server example request “GET /images/fish.gif HTTP/1.1” reply client (initiator) server s = socket(…); bind(s, name); listen(s, 10); sd = accept(s); read(sd, request…); write(sd, reply…); close(sd); sd = socket(…); connect(sd, name); write(sd, request…); read(sd, reply…); close(sd);

7. Socket syscalls connect(csd, <IP address and port>). For a client: connect the socket named by descriptor csd to a server at the specified IP address and port. Block until the connection is established. bind(sd, <…port>). For a server: associate the socket named by descriptor sd with a port number reachable at an IP address of the host machine. Does not block, but may fail, e.g., if some other process is already bound to the port. listen(sd, qsize). For a server: indicate that the socket named by descriptor sd is a server socket. When a connect request arrives for its port, establish the connection and place it on the accept queue (unless the accept queue is full). Listen does not block: it merely sets some parameters on the socket. accept(sd, …). For a server: accept a connection from the accept queue for the server socket named by descriptor sd. Block if the accept queue is empty. Returns the IP address and port of the client for this connection, and a new socket descriptor csd for the connection. Given a socket descriptor csd for an established connection (from a completed connect or accept) a process may use write (or send) to send bytes to the connection peer, and may use read (or recv) to receive bytes sent by the peer.

8. catserver ... structsockaddr_insocket_addr; sock = socket(PF_INET, SOCK_STREAM, 0); memset(&socket_addr, 0, sizeofsocket_addr); socket_addr.sin_family = PF_INET; socket_addr.sin_port = htons(port); socket_addr.sin_addr.s_addr = htonl(INADDR_ANY); if (bind(sock, (structsockaddr *) &socket_addr, sizeofsocket_addr) < 0) { perror("bind failed"); exit(1); } listen(sock, 10); while (1) { intacceptsock = accept(sock, NULL, NULL); forkme(acceptsock, prog, argv); /* fork/exec cat */ close(acceptsock); } }

9. Web Server

10. Inside your Web server Server operations create socket(s) bind to port number(s) listen to advertise port wait for client to arrive on port (select/poll/epoll of ports) accept client connection read or recv request write or send response close client socket Server application (Apache, Tomcat/Java, etc) accept queue packet queues listen queue disk queue

11. Socket descriptors in Unix user space kernel space Disclaimer: this drawing is oversimplified file intfd pointer pipe Inbound traffic socket per-process descriptor table tty global port table “open file table” There’s no magic here: processes use read/write (and other syscalls) to operate on sockets, just like any Unix I/O object (“file”). A socket can even be mapped onto stdin or stdout. Deeper in the kernel, sockets are handled differently from files, pipes, etc. Sockets are the entry/exit point for the network protocol stack.

12. Ports • Each IP transport endpoint on a host has a logical port number (16-bit integer) that is unique on that host. • This port abstraction is an Internet Protocol concept. • Source/dest port is named in every IP packet. • Kernel looks at port to demultiplex incoming traffic. • What port number to connect to? • We have to agree on well-known ports for common services • Look at /etc/services • Ports 1023 and below are ‘reserved’ and privileged: generally you must be root/admin/superuser to bind to them. • Clients need a return port, but it can be an ephemeralport assigned dynamically by the kernel.

13. Ports and packet demultiplexing The IP network carries data packetsaddressed to a destination node (host named by IP address) and port. Kernel network stack demultiplexes incoming network traffic: choose process/socket to receive it based on destination port. Apps with open sockets Incoming network packets Network adapter hardware aka, network interface controller (“NIC”)

14. Wakeup from interrupt handler return to user mode trap or fault sleep queue ready queue sleep switch wakeup interrupt Example 1: NIC interrupt wakes thread to receive incoming packets. Example 2: disk interrupt wakes thread when disk I/O completes. Example 3: clock interrupt wakes thread after N ms have elapsed. Note: it isn’t actually the interrupt itself that wakes the thread, but the interrupt handler (software). The awakened thread must have registered for the wakeup before sleeping (e.g., by placing its TCB on some sleep queue for the event).

15. The network stack, simplified Internet client host Internet server host Client Server User code Sockets interface (system calls) TCP/IP TCP/IP Kernel code Hardware interface (interrupts) Hardware and firmware Network adapter Network adapter Global IP Internet Note: the “protocol stack” should not be confused with a thread stack. It’s a layering of software modules that implement network protocols: standard formats and rules for communicating with peers over a network.

16. The Internet concept wasn’t obviously compelling, at least not to everyone. It had to be marketed, even within the tech community. In 1986, the US National Science Foundation (NSF) opened the door to a commercial Internet (then NSFNET). IP support in sockets (Berkeley Unix) was widely used among academics. The driving force for adopting TCP/IP was a collection of Unix-oriented startups and upstarts arrayed against a few large companies with their own proprietary network standards. Insert “Power of TCP/IP” slide, /usr/net/87. (The poster in my office)

17. Stream sockets withTransmission Control Protocol (TCP) TCP/IP protocol sender TCP/IP protocol receiver TCB TCP user (application) user transmit buffers user receive buffers TCP send buffers (optional) COMPLETE SEND COMPLETE TCP rcv buffers (optional) RECEIVE TCP implementation transmit queue receive queue get data window data flow ack flow ack outbound packets inbound packets checksum checksum network path Integrity: packets are covered by a checksumto detect errors. Reliability: receiver acksreceived data, sender retransmits if needed. Ordering: packets/bytes have sequence numbers, and receiver reassembles. Flow control: receiver tells sender how much / how fast to send (window). Congestion control: sender “guesses” current network capacity on path.

18. Illustration only

19. Who governs the Internet? IANA: a department of ICANN. ICANN: a US nonprofit organization that is responsible for the coordination of…unique identifiers related to the namespaces of the Internet, and ensuring the network's stable and secure operation.

20. I/O syscalls: quick primer 1 read(fd, buf, len): read len bytes from the I/O object named by the descriptor fdand store it in the process VAS at address buf. Block until the data is available: • If fd is a file: wait for disk I/O to complete; e.g., the calling thread blocks awaiting a notify triggered by the disk interrupt handler. • If fd is a socket: block until the data arrives from the network peer; e.g., the calling thread blocks awaiting a notify triggered by the NIC interrupt handler after the packet arrives. (The recvsyscall is equivalent.) • If fd is a pipe: block until a writer writes the data into the write end of the pipe; e.g., the calling thread blocks awaiting a notify triggered by the write (like soda machine). If the object named by fd will never produce the data, then read returns an EOF: end of file. A read returns the number of bytes transferred: zero  EOF. • If fd is a file, and all bytes of the file have already been read through fd. • If fd is a stream socket or a pipe, and the writer(s) closed the other end.

21. I/O syscalls: quick primer 2 write(fd, buf, len): write len bytes to the I/O object named by the descriptor fd, fetching them from the process VAS at address buf. Generally, the write call is asynchronous / nonblocking: it returns immediately, and completes when the data arrives at the destination some time later. However, uncompleted writes require buffering in the system, and write blocks when some bound on the buffer memory is reached: • If fd is a file: block if there are “too many” pending writes in progress; e.g., the calling thread blocks until “enough” of those writes reach the disk. • If fd is a stream socket: block if there are “too many” packets in transit to the receiver; e.g., the calling thread blocks until the receiving process consumes “enough” of that data. (The sendsyscall is equivalent.) • If fd is a pipe: block if the kernel’s bounded buffer for the pipe is exhausted; e.g., the calling thread blocks until a reader consumes “enough” bytes, freeing up sufficient kernel buffer space for the write to complete. If bytes written to the object named by fd will never be consumed, then write returns an error. E.g., this may occur if the receiver(s) closed its end of the pipe or socket, or the network is unreachable.

22. Servers in “classic” Unix • Single-threaded processes • Blocking system calls • Synchronous I/O: calling process blocks until is “complete”. • Each blocking call waits for only a single kind of a event on a single object. • Process or file descriptor (e.g., file or socket) • Add signals when that model does not work. • Oops, that didn’t really help. • With sockets: add select system call to monitor I/O on sets of sockets or other file descriptors. • select was slow for large poll sets. Now we have various variants: poll, epoll, pollet, kqueue. None are ideal.

23. Services Network servicesrun as processes that listen for requests arriving on ports. Each network service receives requests at a designated port number (assigned and standardized by IANA). See /etc/services Inetd forks service processes on demand (lazily) on incoming connect requests for their ports. What if no process is listening on the destination port for a request? Before After

24. inetd • Classic Unix systems run an inetd“internet daemon”. • Inetd receives requests for standard services. • Standard services and ports listed in /etc/services. • inetd listens on all the ports and accepts connections. • For each connection, inetd forks a child process. • Child execs the service configured for the port. • Child executes the request, then exits. [Apache Modeling Project: http://www.fmc-modeling.org/projects/apache]

25. Request/reply messaging client server Remote Procedure Call (RPC) is one common example of this pattern. The Web is another. request compute reply

26. P2: break a simple web server • The web server is based on: • */c-samples/buggyserver.c • This server has a bug that makes it vulnerable to a stack smash attack (previously discussed). • Stack smash attacks may enable remote execution of code chosen by the attacker, to “own” the web server. • Each group gets their own instance to attack. If you crack it you get the points. • Test your talents, but please do not abuse them. • These attacks have unleashed untold pain into the world…and it never stops.

28. Stack smash defenses • Modern systems have various defenses. • NX: no-execute segments. The classic attack injects code onto a buffer that resides on the stack, and overwrites a return address to branch to the injected code. NX makes this harder by disabling execute privilege on the stack segment. Any attempt to execute the attacker’s code (or any code) on the stack generates a fault. • ASLR: address space layout randomization. The attacker guesses where the stack resides in order to overwrite a frame’s return address to branch to injected code. Randomizing the layout makes this harder. • These are disabled in the web server instances or p2.

29. Server listens on a socket structsockaddr_insocket_addr; sock = socket(PF_INET, SOCK_STREAM, 0); int on = 1; setsockopt(sock, SOL_SOCKET, SO_REUSEADDR, &on, sizeof on); memset(&socket_addr, 0, sizeofsocket_addr); socket_addr.sin_family = PF_INET; socket_addr.sin_port = htons(port); socket_addr.sin_addr.s_addr = htonl(INADDR_ANY); if (bind(sock, (structsockaddr *)&socket_addr, sizeofsocket_addr) < 0) { perror("couldn't bind"); exit(1); } listen(sock, 10); Illustration only

30. Accept loop: trivial example while (1) { int acceptsock = accept(sock, NULL, NULL); char *input = (char *)malloc(1024*sizeof (char)); recv(acceptsock, input, 1024, 0); int is_html = 0; char *contents = handle(input,&is_html); free(input); …send response… close(acceptsock); } If a server is listening on only one port/socket (“listener”), then it can skip the select/poll/epoll. Illustration only

31. Send HTTP/HTML response const char *resp_ok = "HTTP/1.1 200 OK\nServer: BuggyServer/1.0\n"; const char *content_html = "Content-type: text/html\n\n"; send(acceptsock, resp_ok, strlen(resp_ok), 0); send(acceptsock, content_html, strlen(content_html), 0); send(acceptsock, contents, strlen(contents), 0); send(acceptsock, "\n", 1, 0); free(contents); Illustration only

32. Servers and concurrency • Servers receive requests from many different clients. • Many clients send requests “at the same time”. • Servers should handle those requests concurrently. • Don’t leave a server CPU idle if there is a request to work on. • But how to do that with the classic Unix process model? • Unix had single-threaded processes and blockingsyscalls. • If a process blocks it can’t do anything else until it wakes up. • Systems with GUIs also face them. • What to do?

33. Event-driven programming • Event-driven programming is a design pattern for a thread’s program. • The thread receives and handles a sequence of typed messages or events. • Handle one event at a time, in order. • In its pure form the thread never blocks, except to get the next event. • Blocks only if no events to handle (idle). • We can think of the program as a set of handler routines for the event types. • The thread upcalls the handler to dispatchor “handle” each event. • A handler should not block: if it does, the thread becomes unresponsive to events. events Dispatch events by invoking handlers (upcalls).

34. Handling a Web request Accept Client Connection Read HTTP Request Header may block waiting on disk I/O may block waiting on network Find File Send HTTP Response Header Read File Send Data We want to be able to process requests concurrently.

35. Web server (serial process) • Option 1: could handle requests serially • Easy to program, but painfully slow (why?) WS Client 1 Client 2 R1 arrives Receive R1 Disk request 1a R2 arrives 1a completes R1 completes Receive R2

36. Web server (“pure” event-driven) • Option 2: use asynchronous I/O • Fast, but hard to program (why?) Disk WS Client 1 Client 2 R1 arrives Receive R1 Disk request 1a Start 1a R2 arrives Receive R2 Finish 1a 1a completes R1 completes

37. Web server (multiprogrammed) • Option 3: assign one thread per request • Where is each request’s state stored? Client 1 WS1 Client 2 WS2 R1 arrives Receive R1 Disk request 1a R2 arrives Receive R2 1a completes R1 completes

38. Events vs. threading • Classic Unix system call APIs are blocking. Requires multiple processes/threads to build responsive/efficient systems. • Even so, kernel networking and I/O stacks are mostly event-driven (interrupts, callbacks, event queues, etc.).Example: Windows I/O driver stack is a highly flexible event-driven system. • Some system call APIs may be non-blocking, i.e., asynchronous I/O. E.g., polling APIs like waitpid() with WNOHANG. This is an event-driven model: notify thread by an event when op completes. • Modern systems combine events and threading • Event-driven model is natural for GUIs, servers. • But to use multiple cores effectively, we need multiple threads. And every system today is a multicore system. • Multi-threading also enables use of blocking APIs without compromising responsiveness of other threads in the program.

39. Multi-process server architecture Find File Find File Send Header Send Header Read File Send Data Read File Send Data Accept Conn Accept Conn Read Request Read Request Process 1 … separate address spaces Process N

40. Multi-threaded server architecture Thread 1 Find File Send Header Read File Send Data Accept Conn Read Request … Thread N Find File Send Header Read File Send Data Accept Conn Read Request This structure might have lower cost than the multi-process architecture if threads are “cheaper” than processes.

41. Thread pool: idealized worker loop Magic elastic worker pool Resize worker pool to match incoming request load: create/destroy workers as needed. Handle one request, blocking as necessary. dispatch Incoming request queue When request is complete, return to worker pool. idle workers Workers wait here for next request dispatch. Workers could be processes or threads.

42. Ideal event poll API for thread pooling Poll() • Delivers: returns exactly one event (message or notification), in its entirety, ready for service (dispatch). • Idles: Blocks iffthere is no event ready for dispatch. • Consumes: returns each posted event at most once. • Combines: any of many kinds of events (a poll set) may be returned through a single call to poll. • Synchronizes: may be shared by multiple processes or threads ( handlers must be thread-safe as well).

43. Event/request queue We can synchronize an event queue with a monitor: a mutex/CV pair. worker loop handler Protect the event queue data structure itself with the mutex. Handle one event, blocking as necessary. dispatch Incoming event queue handler When handler is complete, return to worker pool. threads waiting on CV Workers wait on the CV for next event if the event queue is empty. Signal the CV when a new event arrives. This is a producer/consumer problem. handler

44. But what’s an “event”? • A system can use an event-driven design pattern to handle any kind of asynchronous event. • Arriving input (e.g., GUI clicks/swipes, requests to a server) • Notify that an operation started earlier is complete • E.g., I/O completion • Subscribe to events published/posted by other threads • Including status of children: stop/exit/wait, signals, etc. • You can use an “event” to represent any kind of message that drives any kind of action in the receiving thread. • In Android: intents, binder RPC, UI events • But the system must be designed for it, so that operations the thread requests do not block; the request returns immediately (“asynchronous”) and delivers a completion event later.

45. Android: threading model • An app is launched as a process when any of its components is first instantiated. • The process main thread is event-driven, e.g., by User Interface (UI) events. • Also called the “UI thread”. • UI toolkit code is not thread-safe, so it should execute only on the UI thread. • UI thread should not block (except for next event), or app becomes unresponsive. • UI thread also receives incoming intents, launches and tears down components, and receives various upcalls. • An app may spawn other background threads (workers) for other uses. • Binder RPC manages a thread pool. events

46. Threads in Android Three examples/models for use of threads in Android. • Main thread (UI thread): receives UI events and other upcall events on a single incoming message queue. Illustrates event-driven pattern: thread blocks only to wait for next event. • ThreadPool: an elastic pool of threads that handle incoming calls from clients: Android supports “binder” request/response calls from one application to another. When a request arrives, a thread from the pool receives it, handles it, responds to it, and returns to the pool to wait for the next request. • AsyncTask: the main thread can create an AsyncTask thread to perform some long-running activity without blocking the UI thread. The AsyncTask thread sends progress updates to the main thread on its message queue. These patterns are common in many other systems as well.

47. Adapted from http://developer.android.com/guide/components/processes-and-threads.html Summary: By default, all components of the same application run in the same process and thread (called the "main" thread). The main thread controls the UI, so it is also called the UI thread. If the UI thread blocks then the application stops responding to the user. You can create additional background threads for operations that block, e.g., I/O, to avoid doing those operations on the UI thread. The background threads can interact with the UI by posting messages/tasks/events to the UI thread. Details: When an application component starts and the application does not have any other components running, the Android system starts a new Linux process for the application with a single thread of execution called the main thread. All components that run in the same process are initialized by its main thread, and system upcalls to those components (onCreate, onBind, onStart,…) run on the main thread. The main thread is also called the UI thread. It is in charge of dispatching events to user interface widgets and interacting with elements of the Android UI toolkit. For instance, when the user touches a button on the screen, your app's UI thread dispatches the touch event to the widget, to set its pressed state and redraw itself. If you have operations that might require blocking, e.g., to perform I/O like network communication or database access, you should run them in separate threads. A thread that is not the UI thread is called a background thread or "worker" thread.

48. Adapted from http://developer.android.com/guide/components/processes-and-threads.html Your app should never block the UI thread. When the UI thread is blocked, no events can be dispatched, including drawing events. From the user's perspective, the application appears to “hang” or “freeze”. Even worse, if the app blocks the UI thread for more than a few seconds, Android presents the user with the infamous "application not responding" (ANR) dialog. The user might then decide to quit your application and uninstall it. In a correct Android program the UI thread blocks only to wait for the next event, when it has nothing else to do (it is idle). If you have an operation to perform that might block for any other reason, then you should arrange for a background/worker thread to do it. Additionally, the Android UI toolkit is not thread-safe: if multiple threads call a module that is not thread-safe, then the process might crash. A correct app manipulates the user interface only from a single thread, the UI thread. So: your app must not call UI widgets from a worker thread. So how can a worker thread interact with the UI, e.g., to post status updates? Android offers several ways for a worker to post operations to run on the UI thread. Note: this concept of a single event-driven main/UI thread appears in other systems too.

49. Android: AsyncTask [http://techtej.blogspot.com/2011/03/android-thread-constructs-part-3.html]

50. Summary/recap/segue • Servers need concurrency. The standard model today is multi-threaded servers with a thread pool. We can implement it with our thread primitives. • Clients need concurrency. It isn’t 1974 anymore, and byte streams just aren’t enough. Modern apps have GUIs and likely interact with one or more servers. And they need to be responsive. • Android apps “are servers”. They accept upcalls/events from the GUI, from the OS, and from other apps. So their threading models are similar to servers. • Android uses RPC to communicate across app boundaries. RPC is an important structuring element in modern distributed systems.