1 / 52

TBSM v6.1 Troubleshooting Events

TBSM v6.1 Troubleshooting Events. September, 2013 Joy Hawkins. Agenda. Technote UserID’s (Roles & Groups) TBSM Event Environment Event Results Walk-Through from GUI Enable Event Logging Results from event logging in trace.log How to use event logging filters. 2. Technote: (Events).

mignon
Download Presentation

TBSM v6.1 Troubleshooting Events

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. TBSM v6.1 Troubleshooting Events September, 2013 Joy Hawkins

  2. Agenda • Technote • UserID’s (Roles & Groups) • TBSM Event Environment • Event Results Walk-Through from GUI • Enable Event Logging • Results from event logging in trace.log • How to use event logging filters 2

  3. Technote: (Events) • MustGather - for IBM support to help analyze and understand a TBSM event issue? • http://www.ibm.com/support/docview.wss?uid=swg21598760

  4. Technote • UserID’s (Roles & Groups) Section Separator

  5. UserID’s - Roles & User Groups • (admin1) - (or use ‘tbsmadm’ from install) • Access Service Availability and Service Administration Pages • (tbsmAdminUserRole) assigned by default to tbsmAdmins Group • (user1) – (or use ‘tbsmuser” from install) • Access Service Availability Page • (tbsmReadOnlyUserRole) assigned by default to tbsmReadOnly Group

  6. UserID’s - Roles & User Groups (admin1) UserID’s access

  7. UserID’s - Roles & User Groups (user1) access

  8. UserID’s - Roles & User Groups Netcool_Omnibus_Admin Group • [ncw_user, ncw_gauges_editor,ncw_admin, ncw_dashboard_editor, netcool_rw]

  9. UserID’s - Roles & User Groups to run tool Netcool_Omnibus_User Group • [ncw_user, netcool_ro]

  10. UserID’s - Roles & User Groups • ISQWrite – omnibus privileges to run tool

  11. UserID’s - Roles & User Groups

  12. UserID’s (Roles & Groups) • TBSM Event Environment Section Separator

  13. TBSM Environment: (Templates)

  14. TBSM Environment: (Incoming Status Rule)

  15. TBSM Environment: (Preview data)

  16. TBSM Environment: (Identification fields tab)

  17. TBSM Environment: (ObjectServer is not Data Source)

  18. TBSM Environment: (Data Fetcher)

  19. TBSM Environment: (Data Fetcher’s View Data) Select DATAOPNED, LOWLEVELSERVICEID, REGION, SEVERITY, STATUS, SUMMARYTEXT, TICKETID, TICKETTYPE from TBSMDEMO.TICKETS where STATUS = ‘Open’ and SEVERITY >4 and REGION =‘US’

  20. TBSM Environment: (Rule’ Preview Data)

  21. TBSM Event Environment • Event Results Walk-Through from GUI Section Separator

  22. TBSM Environment: (Service Instances & send test Event)

  23. Event Results: (Service Administration Portlets)

  24. Event Results: (AEL)

  25. Event Results: (Event Details) Event Details

  26. Event Results Walk-Through from GUI • Enable Event Logging Section Separator

  27. Enabling Event Logging • Available from TBSM 4.2.1 FP3, or TBSM 6.1 FP1, or TBSM 6.1.1 GA. • Use setTraceLevel command to add the trace level for the EventLogging package. UNIX:$TBSM_HOME/bin/setTraceLevel.sh <tipadmin user> <tipadmin password> data –addtrace "EventLogging=finer" Windows:%TBSM_HOME%\bin\setTraceLevel.bat <tipadmin user> <tipadmin password> data -addtrace “EventLogging=finer“

  28. Enabling Event Logging • For best effect, set any other TBSM trace levels to “info” $TBSM_HOME/bin/setTraceLevel.sh <tipadmin user> <tipadmin password> data -addtrace "com.micromuse.sla.*=info:com.ibm.tbsm.*=info:EventLogging=finer”

  29. Enabling Event Logging • Other setTraceLevel.sh options $TBSM_HOME/bin/setTraceLevel.sh <tipadmin user> <tipadmin password> data -display $TBSM_HOME/bin/setTraceLevel.sh <tipadmin user> <tipadmin password> data -setstartup • Event Logging should not be enabled permanently on customer sites – turn it off when not required. $TBSM_HOME/bin/setTraceLevel.sh <tipadmin user> <tipadmin password> data -resetall OR $TBSM_HOME/bin/setTraceLevel.sh <tipadmin user> <tipadmin password> data -rmtrace "EventLogging=finer"

  30. Enable Event Logging • Results from event logging in trace.log Section Separator

  31. Collect new trace logs • Run <installdir>/tbsm/bin/collect_logs After running the utility on the TBSM Data server, the result will be a single zip file <installdir>/tbsm_logs_<datetimestamp>.zip • $TIP_HOME/TBSMProfile/logs/server1/ (trace.log, SystemOut.log, SystemErr.log)

  32. Trace log Entries • With tracing set to EventLogging. • The first entry is an event receipt – shows all fields in the Event Container. TBSM will use the filters to match the event to a service. • [7/22/13 13:42:10:946 EDT] 00000044 EventLogging2 RADEventProcessor execute "Prepared byTBSMOMNIbusEventReader"=(EventContainer=(RAD_TimeWindowStart=0, AlertKey=mail, Location=, RAD_TimeWindowEnd=0, RemotePriObj=, LocalRootObj=, RAD_FunctionName=, NmosEntityId=0, RemoteNodeAlias=, X733EventType=0, ReceivedWhileImpactDown=0, RAD_UserFunctionName=NCOMS564, RAD_TotalRawMetric=, TaskList=0, ProbeSubSecondId=0, RAD_ServiceID=0, RAD_TimeWindowLength=0, RemoteRootObj=, Tally=1, KeyField=564, NodeAlias=, RAD_NextThresholdRawMetric=, RAD_ServiceTypeName=, NmosDomainName=, RAD_LastCount=1, RAD_CurrentFunctionState=, RAD_FilterIDList=2,3, Serial=564, X733CorrNotif=, … • … • <also seen in same message, I can see…> • Identifier=steJoyTESTEvent110, • Node=Mail3PortC, • Class=0, • Summary=JoySendEvent3, • Severity=3,…), • <the message ends with the following…> • Policies={Main}, PolicyName=Main, ReportsPolicyName=Main, ActionNodeName={Main}, ActionType=1)

  33. Trace log Entries • Three parts of an Incoming Status Rule: (Discriminator, Filter Thresholds and Instance fields). 1.Discriminator (or Class) is used to set the RAD_FilterIDList on the event. [7/22/13 13:42:10:946 EDT] 00000044 EventLogging 2 MemoryBasedMatchingRawAttributeGetter getRawAttributeWrappersByEvaluatingFilters No Filters found in Event Container so must get filters. [7/22/13 13:42:10:946 EDT] 00000044EventLogging2 MemoryBasedMatchingRawAttributeGettergetRawAttributeWrappersByEvaluatingFilters Usingdiscriminator: 0.

  34. Trace log Entries 2.Threshold Filter. Each field in the Event Container is tested against the Filter fields, to check if they match the threshold. [7/22/13 13:42:10:946 EDT] 00000044 EventLogging 2 MemoryBasedMatchingRawAttributeGetter testThresholdMatch TestingAlertKeywith comparator = 'mail'. [7/22/13 13:42:10:946 EDT] 00000044 EventLogging 2 MemoryBasedMatchingRawAttributeGetter testThresholdMatch fieldMatched is: true. [7/22/13 13:42:10:947 EDT] 00000044 EventLogging 2 MemoryBasedMatchingRawAttributeGettertestThresholdMatch TestingSeveritywith comparator >= '3'. [7/22/13 13:42:10:947 EDT] 00000044 EventLogging 2 MemoryBasedMatchingRawAttributeGetter testThresholdMatch fieldMatched is: true. [7/22/13 13:42:10:946 EDT] 00000044 EventLogging 2 MemoryBasedMatchingRawAttributeGetteraddFilterIDToEvent NewRAD_FilterIDList for Event is to set to: 2. [7/22/13 13:42:10:946 EDT] 00000044 EventLogging 2 MemoryBasedMatchingRawAttributeGetterprintln testThresholdMatch Threshold matched and status is greater than 0.

  35. Trace log Entries 3.Instance field. TBSM reads the value for the Event fields from the event, and then returns a list of Service Instances which match this. [7/22/13 13:42:10:948 EDT] 00000044 EventLogging 2 AbstractMatchingInstanceGetter getInstanceIDFieldValueMappingForRawAttributeReturning field expression and value for attribute: {Node=Mail3PortC}. [7/22/13 13:42:10:948 EDT] 00000044 EventLogging 2 MemoryBasedMatchingInstanceGetter getMappingStringForRawAttribute Returning event string for raw attribute: Mail3PortC. [7/22/13 13:42:10:948 EDT] 00000044 EventLogging 2 GlobalInstanceFieldStore getCommaSeparatedInstanceIDs TagId is 85. attributeID is 1. eventMatchingString is Mail3PortC. List of Service Instance Ids which match raw attribute is: 89. [7/22/13 13:42:10:948 EDT] 00000044 EventLogging 2 MemoryBasedMatchingInstanceGetter getMatchingInstanceIDsFromDataStoreReturning matching instances: [89]. 2 1 3 4

  36. Trace log Entries • Update State in TBSM [7/22/13 13:42:10:948 EDT] 00000044 EventLogging 2 RADEventProcessor sendInstancesToStateModel1 Send to statemodel for ServiceInstanceName Mail3PortC ServiceinstanceID: 89. [7/22/13 13:42:10:948 EDT] 00000044 EventLogging 2 StateModelUpdater passToStateModel Statusis 3.0.

  37. Trace log Entries • Update Object Server service_deps table [7/22/13 13:42:10:948 EDT] 00000044 EventLogging 2 StateModelNodeAttributeState updateServiceDeps About to update service_deps in memory and at object server for SIB: Mail3PortC. [7/22/13 13:42:10:948 EDT] 00000044 EventLogging 2 ObjectServerUpdater updateServiceDeps updateEventIDs for Mail3PortC and eventInstanceID NCOMS:_:564.

  38. Trace log Entries • Update Service Name in Event [7/22/13 13:42:10:948 EDT] 00000044 EventLogging 2 StateModelUpdater setVarsInContexts RAD_ServiceNameis: Mail3PortC being set in Event Container.

  39. Trace log Entries • Propagate up parents [7/22/13 11:41:22:151 EDT] 00000036 EventLogging 2 StateModelNodeAttributeState stateChanged About to update parents for current SIB. [7/22/13 11:41:22:151 EDT] 00000036 EventLogging 2 StateModelNodeAttributeState updateParentInstance Updating state of parent:Mail3PortC. [7/22/13 11:41:22:153 EDT] 00000036 EventLogging 2 StateModelNodeAttributeState updateParentInstance Updating state of parent:MailServer3. [7/22/13 11:41:22:156 EDT] 00000036 EventLogging 2 StateModelNodeAttributeState updateParentInstance Updating state of parent:App-Root.

  40. Results from event logging in trace.log • How to use event logging filters Section Separator

  41. Event Logging with Filters Filters reduce the number of events which are eligible for Event Logging Filters are dynamic Filters are persistent Steps: First eventloggingfilter Table must created Then use radshell commands to set up filters for Event Logging

  42. Event Logging with Filters For TBSM 4.2.1 - create the postgres table eventloggingfilter Created by running the SQL file: createEventLoggingFilterTable.sql (This is shipped with TBSM 4.2.1 FP3) UNIX: cat ./createEventLoggingFilterTable.sql | $TBSM_HOME/bin/rad_db log Windows: type createEventLoggingFilterTable.sql | %TBSM_HOME%\bin\rad_db log

  43. Event Logging with Filters For TBSM 6.1.X - create the DB2 table eventloggingfilter From a command line start db2 db2 connect to TBSM(to connect to DB)

  44. Event Logging with Filters For TBSM 6.1.X - create the DB2 table eventloggingfilter Create Table create table TBSMBASE.eventloggingfilter ( filterfieldname varchar(255), filterfieldvalue varchar(1024) )

  45. Use radshell commands to set up filters for Event Logging Displays the current filter setting Clear the filter Adds fields and values to the current filter Removes a field from the filter Event Logging with Filters $TBSM_HOME/bin/rad_radshell displayEventLoggingFilter(); clearEventLoggingFilter(); addUpdateEventLoggingFilterValue(“fieldName”,”fieldValue”); removeEventLoggingFilterValue(fieldName);

  46. Filter examples: add a field to current filter: addUpdateEventLoggingFilterValue(“fieldName”,”fieldValue”); Only events with Node = “Mail3PortC” are logged, when the trace level for EventLogging package is finer addUpdateEventLoggingFilterValue(“Node”,”Mail3PortC”); Event Logging with Filters

  47. Event Logging with Filters Filter examples: add a field to current filter: addUpdateEventLoggingFilterValue(“fieldName”,”fieldValue”); • To change the filter, so that only events with Node = “Mail3PortC”,AND AlertKey = “mail”, are logged, when the trace level for EventLogging package is finer • addUpdateEventLoggingFilterValue(“Node”,”Mail3PortC”); • addUpdateEventLoggingFilterValue(“AlertKey”,”mail”);

  48. Event Logging with Filters Filter examples:

  49. Event Logging with Filters Filter examples: remove a field from current filter: removeEventLoggingFilterValue(fieldName); • To remove the Node fields from the filter (now filter will just match on AlertKey) • removeEventLoggingFilterValue(“Node”);

  50. Event Logging with Filters Filter examples: remove a field from current filter: clearEventLoggingFilter(); • Remove ALL filter fields – Logging will be done for all events if the trace level for EventLogging package is finer. • clearEventLoggingFilter();

More Related