ensemble performance troubleshooting n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Ensemble Performance Troubleshooting PowerPoint Presentation
Download Presentation
Ensemble Performance Troubleshooting

Loading in 2 Seconds...

play fullscreen
1 / 33

Ensemble Performance Troubleshooting - PowerPoint PPT Presentation


  • 121 Views
  • Uploaded on

Ensemble Performance Troubleshooting. Three Problem Areas. Message Searching Management Portal Message Processing and Delivery. Message Searching. Message Searching. Some searches will always take a long time Routine searches should always be quick Why are queries slow?

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Ensemble Performance Troubleshooting' - nika


Download Now An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
three problem areas
Three Problem Areas
  • Message Searching
  • Management Portal
  • Message Processing and Delivery
message searching1
Message Searching
  • Some searches will always take a long time
  • Routine searches should always be quick
  • Why are queries slow?
    • Ensemble generate a bad query plan
    • The constraints are too broad
    • You don’t have an appropriate search table
finding the sql query
Finding the SQL query
  • Set ^Ens.Debug("UtilEnsMessages","sql")=1
  • New button ’Show Query’
  • Use the SQL management page to determine the execution plan
  • Warning : If the page times out this will be the wrong query!!!
  • If the page times out you must looked for the cache query
  • In 2014.2 it will show multiple recent queries even if he page times out
looking at the cached query
Looking at the cached query
  • ‘Keep source for cached queries’ must be turned on
  • Purge cached queries for Ens.MessageHeader in your namespace
  • Execute your search
  • Look at cached query and execution plan
when you have the query plan
When you have the query plan
  • If the plan is not very good
    • Report it to InterSystems!!!!
    • Tune table probably won’t help
    • Try varying the query – e.g. don’t only search session starts
  • If the plan is good and you are searching a lot of data
    • Narrow criteria
    • Consider adding additional search table fields
  • If the query takes longer than expected
    • See Vik’s presentation
management portal
Management Portal
  • If everything is slow
    • Look for system problems (PC, server or network)
  • If certain things are slow
    • Large Rules, Large Productions, Large DTL
    • Observe pattern of CPU activity on client and server
      • not easy on a busy system
    • If CPU is used on server, use PERFMON and MONLBL
    • Firebug shows JS callback to server
understand the workload
Understand The Workload
  • Run ^pButtons routinely
  • Capture message statistics routinely
    • Low message rates (~10 k per day) – SQL
    • High message rates (1 M per day) – custom report
  • Record disk space routinely
message processing and delivery1
Message Processing and Delivery
  • High Resource Usage?
  • Long Internal Latency?
  • Long External Latency?
three approaches
Three approaches
  • Analyze the message flow
  • Analyze the system
  • Guess
slide14

Ensemble

Business

Services

Inbound Adapters

slide15

Ensemble

Business

Processes

slide16

Ensemble

Business

Operations

Outbound Adapters

slide17

Ensemble

Business

Services

Business

Processes

Business

Operations

Inbound Adapters

Outbound Adapters

slide18

Ensemble

Business

Services

Business

Processes

Business

Operations

Inbound Adapters

Outbound Adapters

slide19

Ensemble

Business

Services

Business

Processes

Business

Operations

Inbound Adapters

Outbound Adapters

Message Body

Message Header

visual trace
Visual Trace
  • Gives time of message at each stage
  • Indentifies time spent in each BP
  • Identifies external latency for synchronous requests to BO
  • Time differences do include queuing
message header properties and sql
Message Header Properties and SQL

Source

Target

Message type

Time Created

Time Processed

Direction

Message ID

Session ID

….

….

SELECT ID, TimeCreated,

TimeProcessed from Ens.MessageHeader where SessionID=45972

slide22

Business

Services

Business

Processes

Business

Operations

time not included
Time not included
  • Inbound adapter
  • Business Service Execution
  • Asynchronous Business Operation Execution
other sources of webtiming
Other Sources of WebTiming
  • ^%ISCLOG
  • CSP Gateway Log
external latency
External Latency
  • Most common cause of poor throughput
  • Characterized by
    • Extended time in session trace
    • low CPU usage in BO
  • Can be helped by increasing pool size
    • Only if external resource can be run in parallel
internal latency
Internal Latency
  • Using or waiting on a resource
  • What is the CPU usage?
  • Look for lock contention
  • Examine system wide performance statistics
queues
Queues
  • Queuing is a symptom not a cause
  • Queuing across the board means a general resource is constrained
  • Localized Queuing highlights which component can’t handle the throughput
pool size greater than 1
Pool Size greater than 1
  • Multiple jobs obscure the picture
  • FIFO will not be guaranteed
  • Only helps if resource can be shared
  • Don’t increase unless you have a good reason
identifying busy configuration items
Identifying Busy Configuration Items
  • Use OS tools to identify busy jobs
  • Use Ensemble Management portal to link to configuration items
drilling into the system usage
Drilling into the system usage
  • ^PROFILE
  • ^PERFMON
    • Where is the application spending time
  • ^%SYS.MONLBL
    • Look at time ‘line by line’
key points
Key Points
  • Capture Metrics all the time
  • Use Cache tools to verify the system health
  • Characterize the problem
  • Drill down into specific issues
questions
Questions

You can reach me at loveluck@intersystems.com

Questions?