SharePoint Correlation IDs: Understanding and Troubleshooting

1. Richmond SharePoint User Group Toby McGrail Senior SharePoint Technical Architect Correlation or Bust? Febuary 2017

2. Agenda

3. Introduction Toby McGrail – Senior SharePoint Consultant for HP Enterprise Services. 10 Years SharePoint Experience Outside of SharePoint I have balanced lifestyle that includes my family, friends, and Fitness. Fitness and SharePoint are my passion. Over 25 years IT Experience

4. What is a Correlation ID? Correlation Ids are GUIDs assigned to events which transpire during the lifecycle of a resource request.  As problems occur, the Correlation Id is commonly surfaced within the context of an error when presented to the person initiating the request or through the Developer Dashboard. You may have seen a message like the ones below in SharePoint 2010 or 2013 SharePoint 2010 SharePoint 2013

5. Diagnostic Logging - Overview • The primary goal of monitoring is to ensure a healthy SharePoint Environment so that you can achieve service performance objectives such as short response time. • You can use the monitoring features from the SharePoint Central Administration and PowerShell scripts to monitor the SharePoint Environment and services. • Logs and reports track SharePoint Environment and service status. • You can read the logs from the logging database. The advantage of using logging database is that you can configure your view and export the logs to Excel. • The logs and reports from Central Administration help you understand how the SharePoint 2013 system is running, analyze and repair problems, and view metrics for the sites. • Log Levels • Trace Logs • Event Throttling

6. Log Levels – Event Logging • It is important that you choose an appropriate severity level. The severity level of an event is displayed in the Windows Event Log and is used by administrators and registered by monitoring tools to indicate how severe or important an event is. Choosing an appropriate level is a key part of the health and monitoring design for your component or system. • Now to the Levels • Critical Error - Events that demand the immediate attention of the system administrator. They are generally directed at the global (system-wide) level, such as System or Application. They can also be used to indicate that an application or system has failed or stopped responding. • Error - Events that indicate problems, but in a category that does not require immediate attention. • Warning -Events that provide forewarning of potential problems; although not a response to an actual error, a warning indicates that a component or application is not in an ideal state and that some further actions could result in a critical error. • Information - Events that pass noncritical information to the administrator, similar to a note that says: "For your information.“ • Verbose - Verbose status, such as progress or success messages.

7. Log Levels – Trace (ULS) Logging • When writing a trace log by using the ULS API, you must specify a severity level. The severity level is displayed in the ULS trace log and is commonly used by reporting or filtering tools. For this reason, it is important to choose an appropriate level. • Now to the Levels • Unexpected - Similar to an Assert (an assumption in code that a condition is true at a particular point), this message indicates that a logic check failed that is atypical, or the message returns an unexpected error code. These generally represent code bugs that should be investigated and fixed. • Monitorable - Traces that indicate a problem, but do not need immediate investigation. The intent is to collect data and analyze it over time, looking for problem trends. • High - General functional detail, the high priority events that happen in the environment. Examples include global configuration modifications, service start and stop, timer jobs completed, and so on • Medium - Useful to help support or test teams debug customer or environmental issues. These likely include messages indicating that individual features have succeeded or failed, such as creating a new list, modifying a page, and so on. • Verbose - Useful primarily to help developers debug low-level code failures. Not generally useful to anyone who does not have access to source code or symbols. Most event tracing that does not need to be enabled all the time should be set at the Verbose level.

8. Usage and Health Data Collection • SharePoint stores Usage and Health Information in Files and in a Database. • Consumes disk space and has a huge effect on Performance. Remember that these files can fill up server space if not configured correctly. Always remember to set a limit and don’t make it unlimited or you will see your disk space disappear rapidly • Something that needs to be managed closely and includes: • Health Data Collection – Lots of timer jobs to monitor and maintain • Log Collection – Timer Job to copy events from files into the Database

9. Health Analyzer • Identifies possible problems and gives the Farm Admin Possible solutions • Some of the Solutions have the Repair Now however in most cases they don’t work or are not “Best Practices” • Applies a set of rules that can be extended or in most Environments customized to the needs of the Farm • Rules are applied for some of the following categories • Security • Performance • Configuration • Availability • Timer Jobs perform these monitoring tasks and collect the monitoring data • Some of these notifications are not always helpful but more time consuming than anything else • Some of the alerts however are also very useful in finding potential issues that you would only find by monitoring the ULS Logs

10. Developer Dashboard • Don’t be fooled by the name its more a tool to help you troubleshoot problems and performance issues • Easily Troubleshoot Problems with Page Rendering • Three Types of modes that you need to be aware of • Off - Not Displayed • On – Rendering on Each and Every Page • OnDemand – Hidden until you manually click on the Developer Dashboard Icon • Granular Control on Visibility provided – Users that have Customization permissions by default • Great way to Monitor Custom Code when the Developer uses the SPMonitoredScope Tag – It’s a great idea to make your solutions use this tag. • Use PowerShell to enable DD. • $ds= [Microsoft.SharePoint.Administration.SPWebService]::ContentService.DeveloperDashboardSettings$ds.DisplayLevel = 'OnDemand'$ds.TraceEnabled = $true$ds.Update()``

11. Developer Dashboard There are 6 report sections which provide you with lots of stats and data .

12. Tools SharePoint Management Console Wireshark SharePoint Diagnostic Toolkit - SPDiag Developer Dashboard PAL – Performance Analysis of Logs SharePoint Diagnostic Studio Fiddler Performance Monitor ULS Viewer

13. PowerShell Commands When running updates don’t use the GUI use PowerShell PSConfig.exe -cmd upgrade -inplace b2b -force -cmd applicationcontent -install -cmd installfeatures Configure ULS and Data Collection through PowerShell Set-SPDiagnosticConfig -LogLocation D:\DiagnosticLogs Set-SPDiagnosticConfig –LogMaxDiskSpaceUsageEnabled PowerShell should be your best friend. Its faster and more reliable. Create custom script files to save you time. Example –Restart SPServices – Custom solution we have built to restart all SharePoint services throughout farm

14. Demo of Tools SharePoint Toolkit/Diagnostic Studio ULS Viewer Developer Dashboard PAL – Performance Analysis of Logs

15. Sharepoint toolkit/Diagnostic Studio To download the Toolkit go to the following URL SharePoint 2010 - http://go.microsoft.com/fwlink/p/?LinkId=196866 SharePoint 2013 – Not yet available but supposedly is in the works. Many third party have these available for a small fee. However the 2010 version works on 2013. SharePoint 2016 – Not available however just in 2013 it works with 2016

16. ULS Viewer

17. Crawl Logs Only visible within the Central Administration Site Relies on Crawl Log for Search Application Important to Monitor this log regular so that your crawl is effectively crawling all the content Remember to Look for Top Level Errors and fix them immediately Top Level documents especially start addresses Virtual Servers Content DB

18. Monitoring HTTP “Ping is a useful command but doesn’t help when troubleshooting Remember SharePoint implements custom error messages. (AKA the Correlation ID error message or the 2013 Error Message (Working on it) Most common error codes 404 and 401 can be hidden Develop a page that checks SharePoint Services

19. Custom Pages Create Custom Pages to allow for more in depth logging. Example HTTP Throttling for Performance Issue Custom Error Page to help Admin and Support with user with important data Corraleation ID Web Front End Server Time of Error User affected Log Name

20. IIS App Pools and Sites Common Issues with SharePoint App Pools IIS Resets not done correctly No Recycling or Restarting of App Pools IIS Website is stopped Create Task to have App Pools recycled daily and restarted once a week. Also have them restart automatically IIS Logging to see why App Pools and Sites have stopped or is not responding.

21. Counters and Thresholds Processor Utilization – Not to exceed 80 Percent but ideally under 50 Percent Available Memory – Greater then 10 Percent Disk Latency Less then 25 MS but ideal situation is 15 MS SQL Server is more like 10 MS

22. Network Troubleshooting SharePoint is Fast on Server but slow on client Slow only across VPN Clients Slow on Server and Client. Communication Issue with SQL Server is most likely the issue Networking Tools Microsoft Network Monitoring Wireshark

23. Client/Browser Issues Is the issue across the network or just one or very few users experience the issues Make sure that all clients are at Organization approved browser level SharePoint relies heavily on JavaScript Older Browser deliver poor user adoption and/or support IE9 and IE10 are much faster and more reliable Firefox Version 5 or later. Not all SharePoint features work in Firefox

24. Wrap Up Know your Environment – Troubleshooting starts here! Performance Baselines help detect and limit issues and problems Monitoring is the Key! Pay attention to Log Files – Both Event and ULS Logs. ULS Viewer should become your best friend next to PowerShell. Tools SPDiag – Helps Troubleshooting SharePoint PAL – Helps with Server Health Diagnose one issue at a time! Don’t always trust google when implementing a solution. Thoroughly test in dev and/or test environments before moving it to the Production Farm.

25. Questions Do you have any issues that you have seen that we have not covered My Blog http://tobymcgrail.com:2020/SPADMIN References PAL – http://pal.codeplex.com SharePoint Diagnostic Studio -http://technet.microsoft.com/en-us/library/hh144782.aspx TechNet Contact Information: Toby McGrail – toby.mcgrail@hp.com Twitter - @SPTOBY1

26. Note: Detailed instructions on how to replace sample pictures like the one shown here can be found in the speaker notes of this slide. Tip! Remember to remove this text box. Picture right with caption Brief caption or descriptive statement relating to picture

27. Note: Detailed instructions on how to replace sample pictures like the ones shown here can be found in the speaker notes of this slide. Tip! Remember to remove this text box. Two pictures with captions layout Brief caption or descriptive statement relating to picture Brief caption or descriptive statement relating to picture

28. Note: Detailed instructions on how to replace sample pictures like the ones shown here can be found in the speaker notes of this slide. Tip! Remember to remove this text box. Three pictures with captions layout Brief caption or descriptive statement relating to picture Brief caption or descriptive statement relating to picture Brief caption or descriptive statement relating to picture

29. Thank you Contact information

30. Note: The speaker notes for this slide include detailed instructions on how to customize and edit charts, like the one shown here, with your own data. Tip! Remember to remove this text box. Sample column chart

31. Note: The speaker notes for this slide include detailed instructions on how to customize and edit charts, like the one shown here, with your own data. Tip! Remember to remove this text box. Sample bar chart

32. Note: The speaker notes for this slide include detailed instructions on how to customize and edit charts, like the one shown here, with your own data. Tip! Remember to remove this text box. Sample stacked column chart

33. Note: The speaker notes for this slide include detailed instructions on how to customize and edit charts, like the one shown here, with your own data. Tip! Remember to remove this text box. Sample line chart

34. Note: The speaker notes for this slide include detailed instructions on how to customize and edit charts, like the one shown here, with your own data. Tip! Remember to remove this text box. Sample pie chart

35. Note: The speaker notes for this slide include detailed instructions on how to customize and edit tables, like the one shown here, with your own data. Tip! Remember to remove this text box. Sample table

36. Multi-level organization chart (without pictures) A Group lead Direct report Secondary report Indirect report INSERT NAME Insert title can run to two lines INSERT NAME Insert title can run to two lines A A Label 1 Label 2 Label 3 Label 4 Label 5 A INSERT NAME Insert title can run to two lines INSERT NAME Insert title can run to two lines INSERT NAME Insert title can run to two lines INSERT NAME Insert title can run to two lines INSERT NAME Insert title can run to two lines INSERT NAME Insert title can run to two lines INSERT NAME Insert title can run to two lines INSERT NAME Insert title can run to two lines INSERT NAME Insert title can run to two lines INSERT NAME Insert title can run to two lines INSERT NAME Insert title can run to two lines INSERT NAME Insert title can run to two lines INSERT NAME Insert title can run to two lines INSERT NAME Insert title can run to two lines INSERT NAME Insert title can run to two lines INSERT NAME Insert title can run to two lines INSERT NAME Insert title can run to two lines INSERT NAME Insert title can run to two lines INSERT NAME Insert title can run to two lines INSERT NAME Insert title can run to two lines INSERT NAME Insert title can run to two lines INSERT NAME Insert title can run to two lines INSERT NAME Insert title can run to two lines INSERT NAME Insert title can run to two lines

37. Multi-level organization chart (with pictures) A Group lead Direct report Secondary report Indirect report INSERT NAME Insert title can run to two lines INSERT NAME Insert title can run to two lines A A A Label 1 Label 2 Label 3 Label 4 INSERT NAME Insert title can run to two lines INSERT NAME Insert title can run to two lines INSERT NAME Insert title can run to two lines INSERT NAME Insert title can run to two lines INSERT NAME Insert title can run to two lines INSERT NAME Insert title can run to two lines INSERT NAME Insert title can run to two lines INSERT NAME Insert title can run to two lines INSERT NAME Insert title can run to two lines INSERT NAME Insert title can run to two lines INSERT NAME Insert title can run to two lines INSERT NAME Insert title can run to two lines INSERT NAME Insert title can run to two lines INSERT NAME Insert title can run to two lines INSERT NAME Insert title can run to two lines INSERT NAME Insert title can run to two lines INSERT NAME Insert title can run to two lines INSERT NAME Insert title can run to two lines INSERT NAME Insert title can run to two lines

38. Multi-level organization chart (without pictures) A Group lead Reporting team Indirect report INSERT NAME Insert title can run to two lines INSERT NAME Insert title can run to two lines A A A A A A Label 1 Label 2 Label 3 Label 4 Label 5 INSERT NAME Insert title can run to two lines INSERT NAME Insert title can run to two lines INSERT NAME Insert title can run to two lines INSERT NAME Insert title can run to two lines INSERT NAME Insert title can run to two lines INSERT NAME Insert title can run to two lines INSERT NAME Insert title can run to two lines INSERT NAME Insert title can run to two lines INSERT NAME Insert title can run to two lines INSERT NAME Insert title can run to two lines INSERT NAME Insert title can run to two lines INSERT NAME Insert title can run to two lines INSERT NAME Insert title can run to two lines INSERT NAME Insert title can run to two lines INSERT NAME Insert title can run to two lines INSERT NAME Insert title can run to two lines

39. HPE theme colors Refer to the PowerPoint How-to-Guide for detailed guidance on color usage Lights/Darks Accents and default chart color order Hyperlinks White R 255 G 255 B 255 Black R 0 G 0 B 0 HPE gray R 198 G 201 B 202 Dark gray R 128 G 130 B 133 HPE turquoise R 42 G 210 B 201 HPE purple R 97 G 71 B 103 HPE orange R 255 G 141 B 109 HPE dark steel R 95 G 122 B 118 HPE gray R 198 G 201 B 202 Dark gray R 128 G 130 B 133 HPE green R 1 G 169 B 130 HPE green R 1 G 169 B 130 Custom colors HPE green R 1 G 169 B 130 HPE bronze R 128 G 116 B 110 HPE slate R 66 G 85 B 99

40. Updating confidentiality footer Protecting company information All company information, whether sensitive or not, is company property and may only be used for the company’s business purposes. You have a duty to safeguard all company information from misuse or inappropriate disclosure to third parties. Under company policy, there are three standard labels for sensitive information which indicate the level of care to be exercised in handling such sensitive information: Private: “Private” information is information that has been prepared for Hewlett Packard Enterprise employees at the VP level or above. Examples of “Private” information include information that affects the company’s stock prices. “Private” information must be handled so that access to the information is granted only to a list of individuals maintained by a VP, e.g., through secure communications and authentication in the case of electronically posted information. Confidential: “Confidential” information may be shared with limited audiences within and outside the company based on a business need to know. Additionally, before “Confidential” information can be shared outside the company, the appropriate contractual protections must be put in place. “Confidential” information includes most information relating to company projects, such as technical data, pricing information, financial information, and business plans. Internal Use Only: “Internal Use Only” is the label for any information intended for widespread distribution within the company, e.g., to everyone within the company’s firewall or on any of the company’s internal sites. A good example of information receiving this label would be phone information for employees and most organization charts.

41. Updating confidentiality footer Protecting company information By default the new template does not have a customizable footer turned on. The footer can be used for confidentiality classification as needed. To add a footer: From the Insert tab, select the Header & Footer button Open the Header & Footer dialog and check the box for Footer Edit Footer text to include only the appropriate confidentiality classification required (according to previous slide) Private, Confidential or Internal Use Only Click Apply to All When updating older presentations that contain footer text, you will need to do the following: Turn off footers/slide numbers before copying slides into the new HPE template. Turn them back on after copying. Open an existing presentation and turn off footers/slide numbers: From the Insert tab, select the Header & Footer button On the Slide tab, remove the checkmarks next to Slide number and Footer Click Apply to All Copy and paste slides into the new HPE template From the Insert tab: Header and Footer dialog box: Edit footer text to include one of the three appropriate classifications: Private Confidential Internal Use Only

42. Sample library slides The following slides are sample visual layouts that have been customized to reflect HPE’s unique brand. Save time by copy/pasting a slide(s) into a new presentation.

43. Sample list Text here Text here • Supporting text here • Supporting text here • Supporting text here • Supporting text here

44. Sample list or process: transition slide 1 Supporting text here 2 Supporting text here 3 Supporting text here

45. Sample process Text here Text here Text here Text here Text here • Details • Details • Details • Details • Details

46. Sample process with pictures Text here Text here Text here Text here

47. Sample process: flow chart Box size and quantity may vary Text here Text here Text here Text here Text here • Supporting text here • Supporting text here • Supporting text here • Supporting text here • Supporting text here • Supporting text here • Supporting text here • Supporting text here • Supporting text here • Supporting text here

48. Sample process: flow chart Box size and quantity may vary Text here Text here Text here • Supporting text here • Supporting text here • Supporting text here • Supporting text here • Supporting text here • Supporting text here Text here

49. Sample timeline: chronological Year Year Year Year Year Date Details here Date Details here Date Details here Date Details here Date Details here Date Details here Date Details here Date Details here Date Details here

50. Sample timeline: product roadmap Task or project Task or project Task or project Task or project Task or project Task or project Task or project Task or project Task or project Task or project Task or project Task or project Task or project Task or project Task or project Task or project Text here  Text here