Troubleshooting performance issues in a citrix virtualized environment
This presentation is the property of its rightful owner.
Sponsored Links
1 / 46

Troubleshooting Performance Issues in a Citrix Virtualized Environment PowerPoint PPT Presentation


  • 122 Views
  • Uploaded on
  • Presentation posted in: General

Troubleshooting Performance Issues in a Citrix Virtualized Environment. Kapil Ramlal Sr. Software Maintenance Engineer Daniel Lazar Lead Escalation Engineer. Agenda. XenServer Performance Overview Troubleshooting XenServer Performance Windows Application Architecture Primer

Download Presentation

Troubleshooting Performance Issues in a Citrix Virtualized Environment

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Troubleshooting performance issues in a citrix virtualized environment

Troubleshooting Performance Issues in a Citrix Virtualized Environment

Kapil Ramlal

Sr. Software Maintenance Engineer

Daniel Lazar

Lead Escalation Engineer


Agenda

Agenda

  • XenServer Performance Overview

  • Troubleshooting XenServer Performance

  • Windows Application Architecture Primer

  • Troubleshooting Virtual Machine Performance

  • Citrix Performance VM Demo

  • Q & A

Citrix Confidential - Do Not Distribute


Xenserver performance overview

XenServer Performance Overview


Xenserver performance overview1

XenServer Performance Overview

XenServer is designed to do one thing…

Performance is a function of “VM density”

All hypervisors have VM density limitations

Citrix XenServer

…Consolidate machine workloads

Citrix Confidential - Do Not Distribute


Xenserver performance overview2

XenServer Performance Overview

How do we determine optimal VM density for a host?

XenServer Hardware

Infrastructure, such as network and storage

Workload and sizing demands of the virtual machines

Native XenServer characteristics

Citrix Confidential - Do Not Distribute


Troubleshooting performance issues in a citrix virtualized environment

Dom0

DomU

DomU

Toolstack

App

App

App

App

Guest OS

Guest OS

netback

netfront

netfront

Native Driver

Xen Hypervisor

Host Machine (Hardware)


Xenserver performance overview3

XenServer Performance Overview

External Factors

Network

Storage

VM Workload and Sizing

Citrix Confidential - Do Not Distribute


Troubleshooting xenserver performance

Troubleshooting XenServer Performance


Troubleshooting performance issues in a citrix virtualized environment

Domain 0 Memory Management

for Dom0

for DomU

total Dom0

= memory required for n VMs

(

6 MB x n

352 MB

400 MB

346 MB

752 MB

6 MB

)

328 MB

6 MB

6 MB

6 MB

for DomU

+

= Total DomU “footprint”

for DomU

Total XenServer

Memory Pool

(ex. 12GB)

Dom0 Memory Pool

for DomU

752MB allows for about 60 VMs-per-host


Troubleshooting xenserver performance1

Troubleshooting XenServer Performance

  • What happens when we start more VMs than Dom0 has memory to manage?

    • Slow VM performance, poor user experience.

    • Slow response from XenAPI—takes longer to process tasks like starting, shutting down and migrating virtual machines.

    • It can cause XenServer host instability resulting in unpredictable behavior and potentially crashing the XenServer host machine!!

Citrix Confidential - Do Not Distribute


Troubleshooting xenserver performance2

Troubleshooting XenServer Performance

There are two common ways to monitor performance in XenServer

XenServer Command Line Interface

XenCenter Performance Tab

Citrix Confidential - Do Not Distribute


Troubleshooting xenserver performance3

Troubleshooting XenServer Performance

Using XenCenter

  • Good for ‘at a glance’ monitoring

  • Unwieldy for refined or customized performance testing

  • Difficult to use for historical trending

  • Data cannot be easily exported

  • Some types of information not gathered.

Citrix Confidential - Do Not Distribute


Troubleshooting xenserver performance4

Troubleshooting XenServer Performance

Performance monitoring commands

# top# Provides a dynamic real-time view of a running system.

Tasks: 68 total, 2 running, 65 sleeping, 0 stopped, 1 zombie

Cpu(s): 13.0%us, 33.6%sy, 0.0%ni, 1.0%id, 52.5%wa, 0.0%hi, 0.0%si, 0.0%st

Mem: 417792k total, 302832k used, 114960k free, 68384k buffers

Swap: 524280k total, 104k used, 524176k free, 80928k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

12857 65550 15 0 27784 3012 1280 D 19 0.6 0:00.57 qemu-dm

4679 root 12 -3 281m 16m 5308 S 12 3.4 3:47.85 xapi

5993 root 15 -3 6164 2276 1188 S 2 0.5 0:24.73 stunnel

1264 root 16 -4 2244 664 384 S 0 0.1 0:24.00 udevd

4641 root 15 0 16348 1936 952 S 0 0.4 0:01.25 xenstored

4650 root 15 0 12304 652 544 S 0 0.1 0:00.05 blktapctrl

12722 root 15 0 2188 1052 836 R 0 0.2 0:00.03 top

Citrix Confidential - Do Not Distribute


Troubleshooting xenserver performance5

Troubleshooting XenServer Performance

Performance monitoring commands

# xentop

# Displays real-time information about a Xen system and domains.

xentop - 17:24:33 Xen 3.3.1

4 domains: 1 running, 3 blocked, 0 paused, 0 crashed, 0 dying, 0 shutdown

Mem: 12580820k total, 7092880k used, 5487940k free CPUs: 8 @ 1600MHz

NAME STATE CPU(sec) CPU(%) MEM(k) MEM(%) MAXMEM(k) MAXMEM(%) VCPUS NETS ...

Domain-0 -----r 9849 65.5 417792 3.9 no limit n/a 8 0 ...

Win2K3-01 ------ 1 1.5 2097020 16.7 2106164 16.7 2 1 ...

Win2K3-02 ------ 1 4.3 2097020 16.7 2106164 16.7 2 1 ...

Win2K3-03 ------ 0 9.8 2097020 16.7 2106164 16.7 2 1 ...

Citrix Confidential - Do Not Distribute


Troubleshooting xenserver performance6

Troubleshooting XenServer Performance

What if I need to increase my VM-density-per-host?

We can tune XenServer to increase VM density. In a scalability study conducted by Citrix to determine the maximum number of Windows XP virtual desktops per host for XenServer 5.5 running XenDesktop 4 we were able to host 130 VMs per host.

Citrix Confidential - Do Not Distribute


Troubleshooting xenserver performance7

Troubleshooting XenServer Performance

This was achieved by making two key configuration changes to a default XenServer 5.5 installation.

  • Increased the amount of RAM assigned to Dom0 to 2.94GB from the default 752MB; increasing it enabled us to launch more desktop clients.

  • Increased the “Xen-heap” setting to take into account the large number of VMs on this single server host. This was done by adding "xenheap_megabytes=24" to the Xen command-line in /boot/extlinux.conf which resulted in an increase from the default of 16MB to 24MB.

Citrix Confidential - Do Not Distribute


Troubleshooting xenserver performance8

Troubleshooting XenServer Performance

Additional Information

Both the scalability study and instructions for increasing Dom0 memory limits are documented in the Citrix Knowledge Center here:

  • http://support.citrix.com/article/CTX124086 - XenServer Single Server Scalability with XenDesktop

  • http://support.citrix.com/article/CTX124259 - Adjusting Dom0 and Xenheap Setting in XenServer

Disclaimer: Your results may vary! This testing was done on very high-end equipment using Citrix best practices!

Citrix Confidential - Do Not Distribute


Troubleshooting xenserver performance9

Troubleshooting XenServer Performance

Troubleshooting commands - Storage

# iostat# Reports basic I/O stats for devices and partitions

# hdparm# Performs timed sequential reads

# dd# Simple, common block device copy utility

TIP: iSCSI storage throughput can usually be tied directly to network performance. If there is slow throughput for an iSCSI storage array, perform network diagnostics first!!

Citrix Confidential - Do Not Distribute


Troubleshooting xenserver performance10

Troubleshooting XenServer Performance

Troubleshooting commands - Network

# tcpdump # Dumps traffic on a network

  • http://support.citrix.com/article/CTX120869 - detailed instructions for using tcpdump.

    # netstat # Display network interface statistics

    # ifconfig # Display and configure network interfaces

TIP: You can always type ‘man’ followed by a Linux command name (i.e., ‘man netstat’) to get detailed help for the command.

Citrix Confidential - Do Not Distribute


Troubleshooting xenserver performance11

Troubleshooting XenServer Performance

Running Shell Scripts

  • Can capture customized data sets

  • Can be run over defined periods of time

  • Can be formatted specifically for reporting purposes.

  • Requires knowledge of Linux and shell scripting languages.

Citrix Confidential - Do Not Distribute


Troubleshooting xenserver performance12

Troubleshooting XenServer Performance

Additional Information

On the Citrix Knowledge Center you can find shell script examples, procedures and best practices for how to troubleshoot all aspects of a XenServer environment.

Some useful links to troubleshooting articles:

  • http://support.citrix.com/article/CTX124157

  • http://support.citrix.com/article/CTX121634

  • http://support.citrix.com/article/CTX122806

  • http://support.citrix.com/article/CTX120737

Citrix Confidential - Do Not Distribute


Windows application architecture primer

Windows Application Architecture Primer


Troubleshooting performance issues in a citrix virtualized environment

A process, in the simplest terms, is an executing program.

- Microsoft (2010)

Citrix Confidential - Do Not Distribute


Application basics

Application Basics

  • An application consists of one or more processes

  • Each process provides the resources needed to execute a program

  • One or more threads run in the context of the process

  • Each process is started with a single thread, often called the primary thread, but can create additional threads


Application basics1

Application Basics

  • A thread is the basic unit to which the operating system allocates processor time

  • Threads carry out the work of aprocess

  • All threads of a process share its virtual address space and system resources

  • Uses stack-based storage for handling data

What is a Thread?


Application basics2

Application Basics

Thread Stack

  • What is the Stack?

    • It’s temporary memory used by threads

    • It’s used to store function’s parameters and Local variables

Frame Pointer


Troubleshooting performance issues in a citrix virtualized environment

A closer look

Citrix Confidential - Do Not Distribute


Application basics3

Application Basics

User & Kernel Space

  • The Windows operating system can be conceptually divided into 2 parts:

    • User Space (User Mode)

    • Kernel Space (Kernel Mode)

  • Applications run in User Mode

  • System drivers run in Kernel Mode (Privileged Mode)


Troubleshooting performance issues in a citrix virtualized environment

USER SPACE

USER

APPLICATION

USER

APPLICATION

USER MODE

USER

APPLICATION

USER

APPLICATION

USER

APPLICATION

KERNEL SPACE

rusb2w2k.sys

keyboard.sys

win32k.sys

tcpip.sys

[…]


Troubleshooting virtual machine performance

Troubleshooting Virtual Machine Performance


Troubleshooting virtual machine performance1

Troubleshooting Virtual Machine Performance

  • Common performance related issues inside the VM:

    • High CPU

    • Disk/registry contention

    • High network utilization

    • Memory


High cpu

High CPU

  • Identify offending Thread (s)

  • Identify the top function call and its module

  • Capture user memory dump of offending process for analysis

  • Engage respective application vendor

ProcessExplorer can be used for live stack-trace viewing!


The windows performance tools

The Windows Performance Tools

  • Next generation performance monitoring from Microsoft

  • Track CPU usage, application start times, boot issues etc.

  • Identify common performance problems without a debugger

  • Included with Windows 7 SDK Download


The windows performance tools case study

The Windows Performance Tools: Case Study

ISSUE:

  • High CPU on wfica32.exe

    Methodology

  • Compare a 30-second sample of activity and compare to non-working ICA, working ICAand working RDP


The windows performance tools case study1

The Windows Performance Tools: Case Study

ICA test run where problem occurred

  • Notice that on this dual processor machine –1 processor is frequently at or very close to 100%.

  • Looking inside the above testing to see which instructions were being executed the most during the test –was wfica32.exe.


The windows performance tools case study2

The Windows Performance Tools: Case Study

  • Drilling into the calls of wfica32.exe, lead to theWindows function NtUserSetCursor() which results in calls to theigdkmd32.sys driver and then into the kernel –specifically the memcpy()function.


Memory dump collection

Memory Dump Collection

  • User dumps contain a snapshot of a process’ memory

  • Kernel dumps contain a snapshot of kernel memory space

  • A complete memory dump contains both the kernel and the entire user space


User dump collection

User Dump Collection

  • Configure a default post-mortem debugger:

    How to Set the NT Symbolic Debugger as a Default Windows Postmortem Debugger(CTX105888)

    How to Set WinDbg as a Default Windows Postmortem Debugger (CTX107528)

    Use Task Manager for manual dumps


System dump collection

To enable a complete memory dump using the registry:

Set the CrashDumpEnabled value to 1

HKLM\System\CurrentControlSet\Control\CrashControl

System Dump Collection

  • Small Memory Dump

    • Generally we avoid

  • Kernel Memory Dump

    • System crash

  • Complete Memory Dump

    • System unresponsive

Control Panel -> System->Advanced Tab -> Startup and Recovery


System dump collection1

System Dump Collection

  • Windows 7 introduced the Dedicated Dump Drive setting

  • Allows a pagefile to be configured on a dedicated drive for dump capture

  • Recommended to debug VM’s streamed through PVS

    How to Recover Windows Kernel Level Dump Files from Provisioned Target (CTX123642)

Citrix Confidential - Do Not Distribute


Demo citrix xenserver performance vm

Demo: Citrix XenServer Performance VM


Troubleshooting performance issues in a citrix virtualized environment

Q & A


Techedge survey video postings ppts

The TechEdge survey will be emailed out to end-user customers

If you complete the survey, you will be entered to win a $250 Amazon gift card. The winner will be announced June 1st.

View TechEdge videos & PPTs on the Knowledge Center by Monday, May17th http://support.citrix.com/techedge2010

TechEdge Survey, Video Postings & PPTs


  • Login