Debugging xenapp xendesktop l.jpg
This presentation is the property of its rightful owner.
Sponsored Links
1 / 64

Debugging XenApp & XenDesktop PowerPoint PPT Presentation


  • 522 Views
  • Updated On :
  • Presentation posted in: General

Debugging XenApp & XenDesktop. Lalit Kaushal Escalation Engineer EMEA. Agenda. Overview of Common Components Troubleshooting Utilities Common Issues Troubleshooting Tips. Overview of Common Components. Putting It All Together.  Find “best” virtual desktop.

Related searches for Debugging XenApp & XenDesktop

Download Presentation

Debugging XenApp & XenDesktop

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Debugging xenapp xendesktop l.jpg

Debugging XenApp & XenDesktop

Lalit Kaushal

Escalation Engineer EMEA


Agenda l.jpg

Agenda

  • Overview of Common Components

  • Troubleshooting Utilities

  • Common Issues

  • Troubleshooting Tips


Overview of common components l.jpg

Overview of Common Components


Putting it all together l.jpg

Putting It All Together

 Find “best” virtual desktop

 Acquire license and determine settings

 Authenticate

 Start VM

Desktop

Delivery Controller

SAN

PVS

 Register

 PXE-boot VM and stream OS

 Connect using ICA

 Log in

XenServer

Virtual Machines

Full range of authentication methods supported through web interface technology

 Apply profile

 Deliver apps

Full support for SmartAccess and ICA session policies

Active Directory with roaming profiles

XenApp


Common components l.jpg

Common Components

ICA Client

Web Interface

Active Directory

XML

IMA

DDC/ZDC (Although roles are a bit different)


Troubleshooting utilities l.jpg

Troubleshooting Utilities


Before you begin l.jpg

Understand the problem

Where is the problem

Network

Server (all servers / one server)

Client (one client machine/ one client version/client type)

Data Store problem (corruption / inconsistency / configuration)

Before you begin


Where to start l.jpg

Collect Information

Frequency? Can I reproduce?

Determine Possible Causes/Effects

Get dumps, logs

Tools

Determine necessary tools

Create a Setup

Debug

Tools and Information to solve problem

Where to start?


Solving the problem l.jpg

Determine accurate reproduction steps

Find appropriate starting point to debug

Crashes – Determine state (using global, stack, etc.)

Debug against working model

Use appropriate tools

Solving the problem


What tools are available l.jpg

WINDBG – Windows Debugger

CDFControl – CDF Tracing

FILEMON – File Monitoring

REGMON – Registry Monitoring

PROCEXP – Process Explorer

SYSTEMDUMP

What tools are available?


Cdfcontrol l.jpg

CDFControl


Process monitor l.jpg

Process Monitor

  • Combines Filemon and Regmon


Process explorer l.jpg

Process Explorer

  • Process Explorer shows handles and DLLs processes

  • Helpful to troubleshoot:

    • Memory Optimization issues

    • Application Streaming

    • Access issues

  • Process Explorer is available from Microsoft


Slide14 l.jpg

Network Trace - Packets

  • Sync Packet (SYN)

    • Start of TCP session. Three way handshake (Syn, Syn-Ack, Ack)

    • ICA session initialisation packets are transmitted next

  • Reset Packet (RST)

    • Something has gone wrong, TCP session failed, unhandled closure of session

  • Finish Packet (FIN)

    • Session is been closed in a handled manner

  • Push Packet (PSH)

    • Data is been sent to receiving process directly

  • Ack Packet (ACK)

    • Packet was received successfully by the remote device


Slide15 l.jpg

Session (Network trace)

Start of a session.

End of session


Dumps l.jpg

Dumps


Debugging l.jpg

User Mode versus Kernel Mode

The Windows operating system can be conceptually divided into 2 parts:

User Space (User Mode)

Kernel Space (Kernel Mode)

Applications run in User Mode

System drivers run in Kernel Mode (Privileged Mode)

Debugging


Slide18 l.jpg

USER SPACE

USER

APPLICATION

USER

APPLICATION

USER MODE

USER

APPLICATION

USER

APPLICATION

USER

APPLICATION

KERNEL SPACE

rusb2w2k.sys

keyboard.sys

win32k.sys

tcpip.sys

[…]


Dump and logs bsod l.jpg

Dump and Logs - BSOD

  • Microsoft definition: BSOD is a Fatal Exception Error or System failure

  • Fatal exception errors:

    • Access to an illegal instruction has been encountered

    • Invalid data or code has been accessed

    • The privilege level of an operation is invalid

  • In most cases the exception is non-recoverable

  • Dumps system memory to a file for debugging

    • Memory.dmp is placed on the System Drive

    • Requires free space equivalent to physical RAM + approx 12MB


Dumps logs types of dumps l.jpg

User dump – process memory

Live dump (snapshot)

Post-mortem dump (after crash)

Kernel dump – OS kernel memory

Manual dump

Post-mortem dump (after BSOD)

Complete dump – physical memory (kernel memory + processes)

Manual dump

Post-mortem dump (after BSOD)

Dumps & Logs - Types of Dumps


User dump l.jpg

Dr Watson

Debugger generates a log file (Drwtsn32.log) & User Dump (user.dmp) when an application exception or program error occur

Log file is cumulative, user.dmp overwritten

Set as the default debugger: drwtsn32.exe –I

User Dump

Generates memory dump of specific process

Microsoft Knowledge Base Article – 241215

User Dump

WINDOWS TASK MANAGER CAN CAPTURE USER DUMPS IN VISTA & 2008!!!


Testdefaultdebugger ctx111901 l.jpg

TestDefaultDebugger - CTX111901


Systemdump ctx111072 l.jpg

SystemDump - CTX111072

  • Can generate a dump from a session

  • No keyboard required

  • Command line option available

  • 32 / 64 bit

Description saved in dump


Dumpcheck ctx108825 l.jpg

DumpCheck - CTX108825

  • Citrix DumpCheck (Explorer Extension)


Common issues l.jpg

Common Issues


Common problems l.jpg

Common Problems

Server\Application Crash

Server\Application Hang

CPU Spikes

Web Interface Debugging


Server application crash l.jpg

Server\Application Crash


Capturing application crash dumps l.jpg

Capturing Application Crash Dumps

  • Some method of capturing the fault is needed

    • Ntsd - http://support.citrix.com/article/ctx108173

    • Windbg - http://support.citrix.com/article/ctx107528

    • Userdump - http://support.microsoft.com/kb/241215

    • Dr Watson – http://support.citrix.com/article/ctx103209

    • WER - http://www.microsoft.com/whdc/maintain/StartWER.mspx

  • Verify your chosen method works

    • TestDefaultDebugger – http://support.citrix.com/article/ctx111901

  • Have one of these methods enabled


Debugging tools for windows l.jpg

Debugging tools for Windows

  • Use tool analyze crash dumps

    • http://www.microsoft.com/whdc/devtools/debugging/debugstart.mspx

    • Latest version is part of WDK (620mb download)

    • Earlier version are available as standalone download


Windbg l.jpg

WinDbg

Symbols – Huh?


Symbols l.jpg

Symbols

  • .PDB – Program Database

    • Generated during compilation of the application by the vendor

    • Necessary to translate memory into something human readable..

      • 11010101001010101 =helloworld()

  • Microsoft symbols Server - Essential

    • http://msdl.microsoft.com/symbols

  • Citrix symbols

    • ftp://ftp.citrix.com/

    • http://ctxsym.citrix.com/symbols 

    • SRV*c:\symcache*http://msdl.microsoft.com/download/symbols;SRV*c:\symcache*http://ctxsym.citrix.com/symbols 


Tell windbg where to find the symbols l.jpg

Tell Windbg where to find the symbols


Analyzing crashes l.jpg

Analyzing crashes

  • Can use similar method for Kernel or User Dump analysis

    • !analyze –v

    • lmv m suspicousmodule

    • Update suspiciousmodule to latest version

    • Search if known stack trace

  • Look at stack functions

    • Understand what the code was trying to do when it crashed


Slide34 l.jpg

Systemdump_400000 makes a call into ntdll

Read upwards

  • Component names

    • DLL

    • EXE

    • SYSTEM Driver

Systemdump_400000 makes a call into USER32


Review of the stack l.jpg

Review of the stack

  • The top of the stack is the last function executed

    • What caused the crash

  • Look for non core OS components

    • Core OS module are usually not the fault

    • Closest to the top of the stack

    • Treat them as suspicious

  • Find out via lmv command

    • Version

    • Owner

    • Timestamp

Case Study: Using WinDbg to analyze IMA Crash


Case study l.jpg

Case Study

  • Issue Reported

    IMA frequently stopped unexpectedly on several server in the farm

  • Data Collected

    • Collected User Dump


Case study38 l.jpg

Case Study

  • Issue Reported

    IMA frequently stopped unexpectedly on several server in the farm

  • Data Collected

    • Collected User Dump

  • Analysis Done

    • !analyze –v

    • lmv m <modulename>

  • Resolution

    • Uninstall Oracle Client 9.2 and update to 10.2


Server application hangs l.jpg

Server\Application Hangs


Server hangs l.jpg

Server Hangs

  • Dumps are not created automatically

    • Full memory dumps are most useful

  • Need to force a dump

    • Systemdump - http://support.citrix.com/article/CTX111072

      • If server is not fully hung

    • Keyboard - http://support.microsoft.com/kb/244139/en-us

    • Hardware NMI Switch

    • Configure for full memory dump instead of kernel


Analyzing server hangs l.jpg

Analyzing Server Hangs

  • Automatic analysis

    • !analyze –v –hang

      • Not 100% reliable for full memory dumps

    • Lmv m suspectmodulename

  • Check for locks

    • The 3 step programme with two new commands

    • !locks

    • Look for exclusive waiters

      • Notice contention count

    • Look at the owner thread code

      • !thread <threadID>


Analyzing application hangs l.jpg

Analyzing application hangs

  • Force a crash of process

    • userdump.exe - http://support.citrix.com/article/ctx466627

    • Vista/2008 – Available from Task Manager

  • Same windbg commands again

  • Automatic analysis usually good

    • !analyze –v –hang

    • Try and understand what code is doing from function names

    • Might have to chase the hang from one process to another

Case Study: Using WinDbg to analyze Server hang


Case study43 l.jpg

Case Study

  • Issue Reported

    • XenApp server is hanging during logon

  • Data Collected

    • Collected Kernel Dump


Case study45 l.jpg

Case Study

  • Issue Reported

    • XenApp server is hanging during logon

  • Data Collected

    • Collected Kernel Dump

  • Analysis Done

    • !analyze –v -hang

    • !locks

  • Resolution

    • Involved Microsoft and recommended relevant Microsoft Hotfix


Cpu spikes l.jpg

CPU Spikes


Cpu spikes47 l.jpg

Try to define a pattern (leverage perfmon)

Determine offending Thread ID causing the spike (Process Explorer, QSlice)

Obtain UserDump of offending process immediately after (Userdump.exe, WinDbg.exe)

!runaway

WinDbg command to view thread times

Topmost thread is one to investigate

Use application spy to look at what the application is doing (TracePlus, Logger)

CPU Spikes

ProcDump – New Microsoft Tool!!!


Procdump l.jpg

Microsoft command-line utility

To monitor an application for CPU Spikes

Generate a dump during spikes

usage: procdump [-64] [[-c CPU usage] [-u] [-s seconds]] [-n exceeds] [-e] [-h] [-m commit usage] [-ma] [-o] [-r] [-t] < <process name or PID> [dump file]] | [-x <image file> <dump file> [arguments]>

C:\>procdump -c 20 -n 3 -o pnamain c:\dump\pnamain

ProcDump


Procdump demo l.jpg

ProcDump Demo


Debugging wi l.jpg

Debugging WI


Problem categories ui l.jpg

Problem categories - UI

Logic

Communication

UI

  • - Verify html code

  • Firebug

  • IE inspector

  • IE developer tools

  • “This looks wrong”

  • Accessibility

  • Section 508

  • Browsers’ incompatibilities


Problem categories logic l.jpg

Problem categories - Logic

UI

Logic

Communication

  • Weird or counter-intuitive behaviour

  • Spec says different thing

  • Configuration issues

  • WI trace file

  • Event log

  • Live http headers


Problem categories communication l.jpg

Problem categories - Communication

Communication

UI

Logic

  • When it’s not WI’s fault

  • New features

  • Performance issues

  • Capturing traffic (Wireshark, Fiddler)

  • Capturing ICA file


Troubleshooting tips l.jpg

Troubleshooting Tips


Troubleshooting tips55 l.jpg

Isolate the problem!

Does the issue affect Farm / Server / User ?

Farm – Try new farm / Data Store

Server – Try different server / Clean build

User – Try new user or Administrator

Does the issue affect ALL users ?

Is it the same in Fixed Window as Seamless ?

Does the problem happen via RDP ?

5 Why’s?

Troubleshooting Tips


Authentication issues l.jpg

Authentication Issues

  • What type of Authentication is configured?

  • Is Application Enumeration works?

  • Is Explicit\Prompt authentication works?

  • Is Kerberos enabled?

  • Capture Network Traffic


Licensing issues l.jpg

Licensing Issues

  • What’s the SA Date and is it valid for current Product?

    • License are not Product specific (2007.0131)

  • What’s LMC and ‘LMSTAT –a’ command showing?

  • Are you able to Telnet LS from XenApp box and vice versa?

  • Is customer using Citrix Option (Citrix.opt) file?

  • Is product going into Grace period? If not, what’s the error?

  • License Acquisition Error 500?


Session disconnection define the issue l.jpg

Session Disconnection - Define the issue

  • Is the issue server-side or client-side ?

    • Where is the network reset packet originating from (client/server)?

    • Use Network and/or CDF tracing to identify the type of disconnection which is occurring

    • If issue is intermittent enable Connection auditing events to define the rate of the issue and follow up with users

    • If client-side; check for network outages and process or device failures.

  • Narrow down steps required to reproduce the issue.

    • Rule out Keep-alives, other components, and timeouts.

    • Map out patterns. (users always disconnected when shadowing, etc.)

    • Identify if issue is related to users or group , subset of servers, network segment, build, or reported outages.


  • Session disconnection l.jpg

    Session Disconnection

    WsxBrokenConnection: Reason=2, Source=2

    • What side of the connection the disconnection:

      • 1 = Client

      • 2 = Server

    Reason why the disconnection occurred.

    wsxica 17896 04/17/2007 08:02:18.636 wsxica.c 2242 CDF_INFO WsxBrokenConnection: From WD: RequestedBPP: 0, SessionBPP: 0, Reason: 0 (overwriting context SessionBPP: 0)

    wsxica 17896 04/17/2007 08:02:18.636 wsxica.c 2260 CDF_INFO WsxBrokenConnection: Reason=2, Source=2

    wsxica 17896 04/17/2007 08:02:18.636 wsxica.c 2292 CDF_INFO WSXICA: BrokenConnection not terminate

    wsxica 17896 04/17/2007 08:02:18.636 wsxica.c 2299 CDF_INFO WsxBrokenConnection: open event Global\WFSHELL_DISCONNECT_1

    wsxica 17896 04/17/2007 08:02:18.636 wsxica.c 2313 CDF_INFO WSXICA: SetEvent disconnect

    wsxica 17896 04/17/2007 08:02:18.636 wsxica.c 2323 CDF_INFO WSXICA: release disconnect semaphore: Global\CPSVC_DISCONNECT_1

    wsxica 17896 04/17/2007 08:02:18.636 license.c 335 CDF_INFO ReleaseLicense: saved LogonId=1, fLimitChecksDone=1

    User’s session was disconnect from CMC


    Cdf tracing wsxbrokenconnection l.jpg

    CDF Tracing (wsxBrokenConnection)

    Disconnection matrix


    Brief troubleshooting guide l.jpg

    Brief Troubleshooting Guide

    • BTG to the rescue!

    • Topics include all current Citrix Products

    • Ensures basic information is collected

    • Helps to narrow down technical issues

    • Faster resolution times

    • Your feedback counts!


    Additional information l.jpg

    Additional Information

    • Citrix XenApp 5.0 for Windows Server 2008 Administrator's GuideKB Article CTX115519

    • Getting Started with Citrix XenApp 5.0KB Article CTX116418

    • Brief Troubleshooting GuideKB Article CTX106727

    • Troubleshooting Tools For Your Citrix EnvironmentKB Article CTX107572

    • Citrix XenApp 5.0 Installation GuideKB Article CTX116573

    • http://support.citrix.com/proddocs/


    Before you leave l.jpg

    Session surveys are available online at www.citrixsummit.com starting Thursday, 7 October

    Provide your feedback and pick up a complimentary gift card at the registration desk

    Download presentations starting Friday, 15 October, from your My Organiser Tool located in your My Synergy Microsite event account

    Before you leave…


  • Login