CSI 227

1. CSI 227 Windows Server 2003 Performance Monitor

2. Why Monitor Performance? • Detect network bottlenecks • Identify server performance problems • Plan the capacity of your servers and subnets • Set alerts so that you can address problems before they get out of hand • Create baselines when activity is low • Understand the effect of your workload on your resources

3. Why Monitor Performance? • Performance Monitor (perfmon) is a powerful program found in all Microsoft servers from NT to Windows Server 2003.  Very few people really know how to make effective use of Performance Monitor. It’s complicated and reports on system measurements, many of which are not easy to grasp. • The initial learning curve is steep.  The number of counters is, by itself, overwhelming. 

4. WhyMonitorPerformance ? • Make the best use of resources • Know where a small investment will produce the maximum gains • Understanding what components are actually doing (e.g., is the disk primarily reading or writing?) • Set alerts when resources are running low.  (e.g., if available memory is less than 10 MB)

5. The Big Four The processor Memory The Disks The Network

6. The best objects to monitor • The first skill to learn is how to choose the right objects and the best counters to monitor a particular problem.  To make sure you do not miss a vital clue, be methodical and always check the 'big four' resource objects: processor, memory, disk and the network.  • I will give tips on how to choose your own counters for some particular circumstances.

7. The best objects to monitor • Object – The system component you are watching (e.g., the CPU) • Counter – What aspect of the object you are watching (e.g., what percentage of the time it is busy)

8. Why Monitor Performance? • You should know how to create logs and then load the logs into Performance Monitor.

9. Identifying  the bottleneck • Once you have collected the data you need, you still need to interpret the logs.  The three skills that all good performance analysts need are: • A knowledge of network principles and server components • A natural talent for spotting something out of the ordinary • The experience and confidence to work with performance counters

10. Strategies • Adding counters is easy • Mastering Performance Monitor is hard

11. Whatis the bottleneck? • To discover a bottleneck, use Performance Monitor to unearth areas of high demand. For example, look at RAM, processor, disk or network demand. • Logging key counters can help you to anticipate problems or analyze what happened in the past.

12. Whatis a Baseline? • What is system performance like when no one is using it?  • Useful for capacity planning.  If you know what adding 100 users looks like, you can guess what adding another 100 will look like.

13. Whatyoulearn • You cannot help but learn how the operating system works when you measure its performance counters.  For example, you might notice that database servers use more memory than file and print servers and that web servers make heavy use of their disks.

14. First place to look? • Almost always the CPU. Is the CPU bogged down with too much to do? Is some particular process hogging all or most of the system’s resources?

15. Key Counter: % Processor Time • What percentage of the time is the processor busy?

16. Processor: % Processor Time • An overloaded processor has a distinctive and unmistakable performance monitor profile.  The % Processor Time trace looks like a curtain hanging down from an imaginary ceiling.  See Slide 28. • Textbooks say to look out for thresholds of between 70-85% for % Processor Time.  It is normal, however, for the trace to show a sharp increase when any program starts. You can safely ignore spikes. Be in alert for continuous high percentages.

17. Next place to look? • Memory: machines with plenty of RAM rarely have performance problems.  A bonus of plenty of memory is that, to a degree, abundant RAM compensates for strain on other resources.

18. Key memory counter: Available Bytes • The more available memory, the faster the server can respond. • Memory\Available bytes is a good counter to look at.  As long as the trace indicates more than 10MB of free memory, the server probably has sufficient RAM.

19. Key memory counter: Available Bytes • The diagram on the next slide shows a white descending line, and the legend confirms that Available bytes are down to 3MB.  Clearly this machine needs more memory. • Suppose a spreadsheet wants to start a new thread or a database needs to sort data. What each needs is memory.  • The operating system provides memory at least 100 times faster using RAM than it could using a disk based pagefile.  This is why a large pool of free memory is so important to an application server.

21. Counters • In general, no counter should be used in isolation • Spikes should be ignored, or at least played down

22. Memory: Pages/sec • Take care to distinguish between these two paging counters:  1) Pages/sec (Hard page faults)2) Page Faults/sec counter is likely to be at least twice the value of the above

23. Memory: Pages/sec • The less paging, the better your server's performance will be.  Memory: Pages/sec measures 'hard' page faults, in other words the page is nowhere in memory, so the VMM (Virtual Memory Manager) has to fetch the data from the pagefile on the disk. • Combined with low Available bytes, a pages/sec counter as high as 20 pages/sec is a clear indicator of a memory bottleneck

24. Memory: Page Faults/sec • Page faults/sec is the sum of hard and soft page faults.  Soft page faults are where the data is found elsewhere in RAM.  For example, Word has opened the spellchecker and now Outlook wishes to use it. There is no need for another call to the disk as the spellchecker is already in memory.

25. Paging File: % Usage • The paging file is very closely related to the physical RAM installed in the computer. Its purpose is to extend the amount of physical RAM and make it available to the system. Both services and installed applications can benefit from this 'extra' RAM, even though it is substantially different from the sticks that plug into the motherboard.

26. Paging File: % Usage • While the pagefile is less likely to be a bottleneck, it is easy to check.  If your system has multiple disks, you might confirm that your page file is on the faster disk or even split it over two disks. • Because the changes are so gradual, you are better off using this Page File % Usage counter as an alert, rather than a log.  A good alert threshold might be 70%.

27. ProcessBottlenecks • Processor usage (also called CPU usage) is the easiest bottleneck for performance monitor to detect.  What we are looking for is the percentage time that the processor is in use.  If the CPU is so busy that it cannot respond to requests, then the whole server's performance deteriorates rapidly.

28. ProcessBottlenecks • Database and email servers are the most likely to suffer from processor bottlenecks.  File and print servers are less likely to run short of CPU power.  • Large modern servers may have multiple processors making processor bottlenecks more rare than memory bottlenecks. 

29. ProcessBottlenecks • If you discover a processor bottleneck, you can use the process counters in Performance Monitor to identify which program or process is hogging the server.  Make sure you don’t include the idle process!

31. ProcessBottlenecks

32. ProcessBottlenecks

33. System: Processor Queue • The System: Processor Queue counter is like the load counter we looked at using top. It shows the number of processes waiting in line to be run.

34. System: Processor Queue

35. OtherCounters • You may also wish to examine DPCs Queue/sec.  This counter shows when the server was busy and so deferred processing a request.  High or intermittent bursts of Interrupts/sec could indicate a hardware problem or a loose component.

36. Process and Thread • If you find a processor bottleneck, you can pursue the cause by looking at the Process or even Thread object.  What you are looking for is which instance of the Process or Thread is responsible for exhausting the processor.  • This can be time consuming, especially with threads since there are so many of them.

37. System: %Privileged Time &Process: %Privileged Time • The Windows 2003 operating system can execute either in kernel mode, which shows up as %Privileged Time, or user mode which corresponds to %User Time. This means that activities of programs like SQL or Exchange are charged to %User Time. • Here is a combination which would point to an I/O bottleneck: • System %Privileged Time > 20% • PhysicalDisk %Usage > 55%. 

38. DPC • DPC means Deferred procedure calls - The processor is saying 'I am busy I will do this low priority task later.'  Processor %DPC Time > 50% is suspicious and may indicate a network adaptor bottleneck.

39. The Big Four • One company thought they had a problem with slow disks on a Windows 2003 Server.  Performance Monitor confirmed long queues and slow disk access times.  Their conclusion was that the bottleneck was the disk and so they bought faster disks.  Unfortunately, the slow response persisted.

40. The Big Four • After monitoring all the 'big four' performance objects, they found excessive paging, there was also less than 2MB of available bytes.  The true ailment was lack of memory, high disk usage was a symptom and not the cause.  The lesson: incomplete monitoring can mean a waste of time and money, so always record these four objects:- Memory, Processor, Disk and Network.

41. The Big Four • The Windows server roles most likely to experience disk problems are web servers with lots of graphics and files. • There are counters specifically for web servers.

42. Diskbottlenecks • In slide 43, performance monitor shows classic symptoms of a disk bottleneck.  The Disk write queue counter averages more than 2.  In fact the average is nearly 4 (with a peak of over 8). • To ensure that it was not a processor or memory bottleneck, we also recorded % processor time and available bytes.

44. Diskbottlenecks • As you can see from slide 42, the processor's average was below 30%.  If the processor were the bottleneck the trace would be over 80%.  On the other hand, if there was a memory shortage, available bytes should drop below 10MB.  The graph show there was always 70 MB of available memory.

45. Diskbottlenecks • There is one more deduction we can make from the queue data on the chart.  If you compare  the white line with the red line near the bottom, you can tell that the disk is writing more rather than reading.

47. Why Monitor? • In the next screenshot, performance monitor shows classic symptoms of a disk bottleneck.  You can see that the Disk write queue counter averages more than 2.  In fact the average is nearly 4 (with a peak of over 8).

49. Defragyourdisks • Once Windows disks fill to 70% capacity, they slow down dramatically.  The other side of the coin is that a defrag can cut queues in half.

50. BuyFasterDisks • The logical solution is to buy faster disks.  Go to your existing disk manufactures site and compare their figures with the data you collect for: • PhysicalDisk: Disk Read Byte /sec • PhysicalDisk: Writes /sec