1 / 51

Building High Throughput, Multi-threaded Servers in C#/.NET

Building High Throughput, Multi-threaded Servers in C#/.NET. David Buksbaum. Who Am I?. Wall Street Wage Slave Developer – I still write code Tech-Lead – My bugs are harder to find Pointy Haired Manager – I live by the KISS principle. What have I built?. User Time to Near Real Time Systems

judithw
Download Presentation

Building High Throughput, Multi-threaded Servers in C#/.NET

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Building High Throughput, Multi-threaded Servers in C#/.NET David Buksbaum

  2. Who Am I? • Wall Street Wage Slave • Developer – I still write code • Tech-Lead – My bugs are harder to find • Pointy Haired Manager – I live by the KISS principle

  3. What have I built? • User Time to Near Real Time Systems • User Time: event time is based on user interaction • Near Real Time: event timing is as fast as possible without guarantee • Server to Server • Client - Server • Small Scale: 1-25 concurrent users • Large Scale: 250+ concurrent users • Distributed Systems (now called SOA)

  4. Agenda • C# Crash Course • Application Servers • Concepts • Connection Stack • Conclusion • Further Concepts

  5. C# Crash Course • [external]

  6. Application Servers • What I mean by App Servers • A stateful server to service client requests for high-level business functions in near user time • RRE Messaging Model (request, response, event) • Middle Tier Intra-Net Systems • May contain 1 or more logical servers • What I do not mean by App Servers • Not the back-end infrastructure systems • Broadcast Servers (pricing, curve servers, etc…) • Bus Servers (messaging platforms like Jabber) • Database Servers

  7. Concepts • Endian-ness • Does not matter unless your network is exactly ½ little and ½ big • Pick the model used more and make the other convert • Only the smaller group will have the performance hit • Most clients to your servers will be little-endian • Many middle-tier systems are Intel based • Solaris is being moved further back or out due to costs • Typical configurations will be WinTel clients with either WinTel or LinTel servers

  8. Concepts – cont. • Threading • Your server functionality will determine threading or not • Guideline: If you threads on WinTel, then thread count target should be (2 * CPU’s) + 1 • Explained in detail by Jeffery Richter in "Programming Server Side Applications for Windows 2000" • Exceptions: Lots of I/O bound threads for example, so use your judgment • Burst servers should be guided towards single threaded per connection (min concurrency) • Servers with minimal state, fast response time, and no persistent connections • HTTP 1.0 defaulted to a burst server (no keepalives) • What is a burst server? echo, time, etc… • Pattern: UDP or short lived TCP

  9. Concepts – cont. • Threading – cont. • Stateful servers need threads – so use them • State operations are slower than network operations, so minimize it in the network thread • If you use threads, use thread pools • Creating threads are expensive, so don’t do it • .NET has a built in thread pool • System.Threading.ThreadPool • ThreadPool.QueueUserWorkItem(new WaitCallback(myThreadFunc), myContextObject); • In large systems, use multiple specialized pools • Different Priorities, Order Guarantees, etc…

  10. Concepts – cont. • Queues • Rule 1: If you use threads – you use queues • Queues break performance coupling between two independent tasks • Task 1: Read / Write Data From / To Network • Task 2: Do Something with the Data • Queues allow for complex tasks to be sub-divided into simpler sub-tasks • These sub-tasks can be worked on concurrently (parallelism or grids) or synchronously • Reusability of sub-tasks between different parent tasks • eg: fetch from database, calculate distance between two points on a map, execute a trade • Simpler code is easier to build, debug, & maintain

  11. Concepts – cont. • Synchronicity • Blocking Synchronous I/O • Read or Write w/o timeouts • Select w/o timeouts • Non-Blocking Synchronous I/O • Read or Write w/ timeouts • Select w/ timeouts • Polling • Asynchronous I/O • BeginReceive or BeginSend • I/O Completion Ports & Overlapped I/O

  12. Concepts – cont. • Synchronicity – cont. • Which do you use? • Rule 2: Never, Ever, Use Blocking I/O • Server Threading model defines synchronicity strategy • Thread per connection  Non-Blocking Synchronous I/O • Thread pools  Asynchronous I/O • Clients can usually use Non-Blocking Synchronous I/O in a dedicated thread

  13. Concepts – cont. • State • Definition • Data and Methods encapsulating a logic block related information • An App Server has different types of state • Server State • State managed on a per logical server basis • Used anywhere the server needs to self administer itself • Usually the container of all other state in the server • Connection State • State managed on a per user connection basis • Used only in a context where the connection needs to be known • Reading and Writing to the network • Verifying credentials

  14. Concepts – cont. • State – cont. • Business State • Contains all the cached data used to support the business functionality • Provides access to all non-cached data transparently • eg: BusinessState.GetAllAccounts() may go to the cache or some external store. Follow good OO procedures here. • State Rules • Minimize surface area (less public methods), but no side effects • Everything public MUST be reentrant or thread-safe • Encapsulate – no public variables • Design for threading

  15. Concepts – cont. • State – cont. • Rule 3: State will dominate server design & performance • Questions to ask when defining state • Who needs access to it? • When do they need access to it? • Special Cases – frequent & pervasive • Logon, Logoff • Credentials Check

  16. Concepts – cont. • State – cont. • Techniques • Synchronized Collections • Equal punishment to readers & writers • Reader / Writer Locks • Favors reads over writes • External Data Store • Push the locking burden on an expert • Embedded • All state is passed between the client and server in every message • eg: URL’s, Cookies, etc… • Tokens • A token for looking up the state is passed between the client & server in every message • Usually used to add state to a stateless design (eg: HTTP) • Hybrids • Combinations of the above • eg: Tokens, Embedded, & External are used by many web servers

  17. Concepts – cont. • Performance • User Time vs. Actual Time • User Time is the time between the initiation of a process by the user on the client and when the result of that action is visible to the user • Actual Time is the actual time it takes to perform the given action on the server from receipt of a request to the sending of a response • Throughput of the server is often defined in terms of actual time (messages per second) • Rule 4a: The user does not care about Throughput • Rule 4b: User Time is all that matters in interactive systems

  18. Concepts – cont. • Performance – cont. • Users vs. Actions • How many concurrent users? • How many concurrent actions? • Similar to Time vs. Memory tradeoff • eq: 1 user with 100 actions or 100 users with 1 action

  19. Concepts – cont. • Case Example: Massive Multiplayer Online Role-Playing Games (MMORPG) – Everquest 2 • 2500 concurrent users per server • Each user requires significant state management • Location in the game world • Character statistics • Inventory • Relationships to other characters, non-player characters, organizations • State is a hybrid combining tokens & external data store • User time is between 11 & 140 ms, with the bulk of the users being at around 100 ms – derived from frame rates • Actual time peak is 25000 / second (2500 * 10) • Assumes all 2500 are not just concurrent, but active

  20. Questions?

  21. Architecture @ 50,000 Feet User Time Queue P1 Network Thread Pool P2 P3 Task Thread Pool Network Initial Response Time Actual Time • P1, P2 and P3 represent the areas that have the possibility for high contention due to locking or congestion • A fast initial response time, followed by incremental updates (streaming or events), can give the illusion of faster user time

  22. Protocol Stack @ 50,000 Feet Application • Wire • TCP, UDP, or other network protocol • Transport • Envelope to encapsulate the application message • Fields: • Length, Checksum, Flags, Data • Application • Message ID • Message Data Transport Wire

  23. Wire Protocols • UDP • Not reliable • Small blocks of data • Connectionless • TCP • Reasonably reliable • Any reasonable amount of data • No information about content

  24. Wire Protocols – cont. • Message Queues • Queue to Queue communications • Each end point has a queue • Messages are moved from one queue to another • A centralized queue can have 1 or more listeners in non-exclusive mode • A centralized queue in exclusive mode can have auto-failover • Reliability level is controllable • Products • JMS – Java Messaging Service • .NET bindings exists • Derivative works like Tibco’s EMS is a very performant version • Deployable on top of most network infrastructures • MSMQ • MS has .NET bindings in the System.Messaging namespace • Not available on all Windows platforms by default • Many organizations have restricted this product due to tight integration with the Windows platform

  25. Wire Protocols – cont. • Message Queues – cont. • Alternative Systems • Book: Message Passing Server Internals by Bill Blunden • Contains the design and implementation of Bluebox; an enterprise ready messaging queuing system in Java Others • Tibco Rendezvous • Basically a serialized hash table (key, item) • Jabber • Java framework originally geared for Instant Messaging, but now supporting robust bus based communications • Still XML based, so not a great fit for high performance

  26. Wire Protocols – cont. • Why not UDP? • Packet size limitations • Connectionless • Reliability • Why not TCP? • Reasonable reliability is not always good enough • Performance in broadcast scenarios (1 to many listeners)

  27. Wire Protocols – cont. • Why messaging systems? • Reliable & recoverable queues are great for when the message absolutely must get to its destination • Trading systems • Banking systems • Transactional • Queues can be used as the back bone of a system that support distributed functionality with distributed transactions (XA) • Your app sever can participate in a transaction that spans all systems between the client and the database

  28. Wire Protocols – cont. • Why messaging systems? – cont. • Support tools • Many messaging systems provide rich tools for sniffing and/or capturing the messages • Captured messages can be used for playback • Robust infrastructure • Messaging systems have been built by networking experts to accommodate a number of different infrastructures, topologies, and geographical idiosyncrasies • You have someone to scream at when it goes wrong • Rule 5: If you can, use a commercial transport

  29. Transport Protocol • Why do you need a protocol between the application and the wire? • Wire Insulation: The wire may change • TCP  Message Queues • UDP  TCP • The application protocol may not play nicely into a wire protocol • MB sized files • Streaming data • XML • Variable length records

  30. Transport Protocol – cont. • Remember our goal at the wire? • Get the data off the wire and onto the queue as fast as possible • We need information as we get the data off the wire to speed up processing • Length of the total message • Message data • Smallest transport protocol (4 bytes) • 4 byte length | n byte data block

  31. Transport Protocol – cont. • What else do we need? • We have: • Length • Data • We need (loosely used): • Message type information • Message ID • Sequencing information • Beginning • Middle • End • Flags or Options

  32. Transport Protocol – cont. • DIME Lite • Protocol created by Microsoft to allow for easier transmission of binary data over SOAP • Can be used independently of SOAP as a unidirectional transport block

  33. Transport Protocol – cont.

  34. Application Protocol • What is it? • It is anything meaningful to your servers business logic • Hash Tables • Schema’s can be used to define required keys • eg: Default schema may mandate key 100 in all messages • Key 100 might be the schema id for the rest of the message • Allows for complex record based pseudo-objects with mandatory, optional, default fields • Compatible cross language / platform • Efficient • Client objects can wrap hash table

  35. Application Protocol – cont. • What is it? – cont. • Serialized Objects • Complex message object or hierarchy of objects • Very rich distributed object functionality • Limited to .NET environment and possible subset of languages within the .NET world • eg: Java RMI, some transport for .NET Remoting • SOAP, Raw XML, or other text based transport • Schema or well defined set of tag / value pairs • String based • Changes require new memory block for the whole message since strings are immutable • Slow to parse (~350 / sec) [validating, msxml] • Even the fastest is less than 1000 / sec

  36. Application Protocol – cont. • What is it? – cont. • Hybrids • Serialized objects encoded in hash tables • Flexibility of hash tables • Expandable through new keys • Cross platform / language support • Versionable objects • Main limitation is the primitive types supported • Requires well known schemas

  37. Complete Example Simple RPC Client / Server

  38. Questions?

  39. Further Thoughts • Lock / Wait – Free Data Structures • Used in graphics pipelines and media streaming • Reduces contention • Aimed at small # of writers and any number of readers (where you would use Reader / Writer locks) • Protocols • Current trends: • Ignore wire performance and go for B2B protocols (SOAP, et al) • Text based protocols, such as ASCII encoded key / value pairs • Serialization • No simple way to work cross language / platform • Primitives are not always portable • Most research is focused on B2B

  40. Further Thoughts – cont. • Memory Pools • Pre-allocate pools of objects that can are reused rather than freed

  41. Rules Summary • Rule 1: If you use threads – you use queues • Rule 2: Never, Ever, Use Blocking I/O • Rule 3: State will dominate server design & performance • Rule 4a: The user does not care about Throughput • Rule 4b: User Time is all that matters in interactive systems • Rule 5: If you can, use a commercial transport

  42. References & Links • HTTP 1.0 – RFC 1945 • http://www.ietf.org/rfc/rfc1945.txt • HTTP 1.1 – RFC 2616 • http://www.ietf.org/rfc/rfc2616.txt • Echo Protocol – RFC 862 • http://ftp.rfc-editor.org/in-notes/rfc862.txt • Time Protocol – RFC 868 • http://ftp.rfc-editor.org/in-notes/rfc868.txt • Superseded by Network Time Protocol (NTP) • http://en.wikipedia.org/wiki/Network_time_protocol

  43. References & Links – cont. • Endianness • http://en.wikipedia.org/wiki/Endian • .NET Thread Pools • MSDN Docs for ThreadPool Class • http://msdn.microsoft.com/library/default.asp?url=/library/en-us/cpref/html/frlrfsystemthreadingthreadpoolclasstopic.asp • Stephen Toub’s Managed Thread Pool • http://www.gotdotnet.com/Community/UserSamples/Details.aspx?SampleGuid=bf59c98e-d708-4f8e-9795-8bae1825c3b6 • Enhanced version of Stephen Toub’s Thread Pool • http://www.codeproject.com/csharp/SmartThreadPool.asp

  44. References & Links – cont. • Overlapped I/O • http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dllproc/base/synchronization_and_overlapped_input_and_output.asp • I/O Completion Ports • http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/fs/i_o_completion_ports.asp

  45. References & Links – cont. • Message Queues • Message Passing Server Internals by Bill Blunden • http://www.amazon.com/gp/product/0071416382/103-6338215-6207008?v=glance&n=283155 • Jabber • http://www.jabber.org/ • DIME • http://msdn.microsoft.com/library/en-us/dnglobspec/html/draft-nielsen-dime-02.txt

  46. Lock / Wait Free Data Structures • http://www.audiomulch.com/~rossb/code/lockfree/ • http://appcore.home.comcast.net/

  47. Extra Slides

  48. using System; using System.Text; using System.Net; using System.Net.Sockets; namespace EchoServer { class Program { static void Main(string[] args) { try { . . . } catch(Exception x) { Console.WriteLine(x.ToString()); } } } } UDP Echo Server – the shell

  49. UDP Echo Server – the code try {// create a udp server on port 8666 UdpClient server = new UdpClient(8666); // create an end point that maps to any ip and any port IPEndPoint endPoint = new IPEndPoint(IPAddress.Any, 0); // receive data from our end point byte[] data = server.Receive(ref endPoint); // convert from bytes to string string msg = Encoding.ASCII.GetString(data); // display the message and who it is from Console.WriteLine( "Received message from {0} ==> {1}", endPoint.ToString(), msg); }

  50. UDP Echo Client in Java public static void main(String[] args) { try { // set up our locals InetAddress host = InetAddress.getByName("127.0.0.1"); int port = 8666; String msg = "Hello World from Java"; // get the bytes for string byte[] data = msg.getBytes(); // create a udp client DatagramSocket client = new DatagramSocket(); // create our data packet DatagramPacket packet = new DatagramPacket(data, data.length, host, port); // send the data client.send(packet); } catch(Exception x) { System.out.println(x.toString()); }; }

More Related