TBSM v6.1 Troubleshooting Events
This presentation is the property of its rightful owner.
Sponsored Links
1 / 52

TBSM v6.1 Troubleshooting Events PowerPoint PPT Presentation


  • 44 Views
  • Uploaded on
  • Presentation posted in: General

TBSM v6.1 Troubleshooting Events. September, 2013 Joy Hawkins. Agenda. Technote UserID’s (Roles & Groups) TBSM Event Environment Event Results Walk-Through from GUI Enable Event Logging Results from event logging in trace.log How to use event logging filters. 2. Technote: (Events).

Download Presentation

TBSM v6.1 Troubleshooting Events

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Tbsm v6 1 troubleshooting events

TBSM v6.1 Troubleshooting Events

September, 2013

Joy Hawkins


Tbsm v6 1 troubleshooting events

Agenda

  • Technote

  • UserID’s (Roles & Groups)

  • TBSM Event Environment

  • Event Results Walk-Through from GUI

  • Enable Event Logging

  • Results from event logging in trace.log

  • How to use event logging filters

2


Tbsm v6 1 troubleshooting events

Technote: (Events)

  • MustGather - for IBM support to help analyze and understand a TBSM event issue?

  • http://www.ibm.com/support/docview.wss?uid=swg21598760


Tbsm v6 1 troubleshooting events

  • Technote

  • UserID’s (Roles & Groups)

Section Separator


Tbsm v6 1 troubleshooting events

UserID’s - Roles & User Groups

  • (admin1) - (or use ‘tbsmadm’ from install)

    • Access Service Availability and Service Administration Pages

    • (tbsmAdminUserRole) assigned by default to tbsmAdmins Group

  • (user1) – (or use ‘tbsmuser” from install)

    • Access Service Availability Page

    • (tbsmReadOnlyUserRole) assigned by default to tbsmReadOnly Group


Tbsm v6 1 troubleshooting events

UserID’s - Roles & User Groups

(admin1) UserID’s access


Tbsm v6 1 troubleshooting events

UserID’s - Roles & User Groups

(user1) access


Tbsm v6 1 troubleshooting events

UserID’s - Roles & User Groups

Netcool_Omnibus_Admin Group

  • [ncw_user, ncw_gauges_editor,ncw_admin, ncw_dashboard_editor, netcool_rw]


Tbsm v6 1 troubleshooting events

UserID’s - Roles & User Groups

to run tool

Netcool_Omnibus_User Group

  • [ncw_user, netcool_ro]


Tbsm v6 1 troubleshooting events

UserID’s - Roles & User Groups

  • ISQWrite – omnibus privileges to run tool


Tbsm v6 1 troubleshooting events

UserID’s - Roles & User Groups


Tbsm v6 1 troubleshooting events

  • UserID’s (Roles & Groups)

  • TBSM Event Environment

Section Separator


Tbsm v6 1 troubleshooting events

TBSM Environment: (Templates)


Tbsm v6 1 troubleshooting events

TBSM Environment: (Incoming Status Rule)


Tbsm v6 1 troubleshooting events

TBSM Environment: (Preview data)


Tbsm v6 1 troubleshooting events

TBSM Environment: (Identification fields tab)


Tbsm v6 1 troubleshooting events

TBSM Environment: (ObjectServer is not Data Source)


Tbsm v6 1 troubleshooting events

TBSM Environment: (Data Fetcher)


Tbsm v6 1 troubleshooting events

TBSM Environment: (Data Fetcher’s View Data)

Select

DATAOPNED, LOWLEVELSERVICEID, REGION, SEVERITY, STATUS, SUMMARYTEXT, TICKETID, TICKETTYPE

from TBSMDEMO.TICKETS

where STATUS = ‘Open’ and SEVERITY >4 and REGION =‘US’


Tbsm v6 1 troubleshooting events

TBSM Environment: (Rule’ Preview Data)


Tbsm v6 1 troubleshooting events

  • TBSM Event Environment

  • Event Results Walk-Through from GUI

Section Separator


Tbsm v6 1 troubleshooting events

TBSM Environment: (Service Instances & send test Event)


Tbsm v6 1 troubleshooting events

Event Results: (Service Administration Portlets)


Tbsm v6 1 troubleshooting events

Event Results: (AEL)


Tbsm v6 1 troubleshooting events

Event Results: (Event Details)

Event Details


Tbsm v6 1 troubleshooting events

  • Event Results Walk-Through from GUI

  • Enable Event Logging

Section Separator


Enabling event logging

Enabling Event Logging

  • Available from TBSM 4.2.1 FP3, or TBSM 6.1 FP1, or TBSM 6.1.1 GA.

  • Use setTraceLevel command to add the trace level for the EventLogging package.

    UNIX:$TBSM_HOME/bin/setTraceLevel.sh <tipadmin user> <tipadmin password> data –addtrace "EventLogging=finer"

    Windows:%TBSM_HOME%\bin\setTraceLevel.bat <tipadmin user> <tipadmin password> data -addtrace “EventLogging=finer“


Enabling event logging1

Enabling Event Logging

  • For best effect, set any other TBSM trace levels to “info”

    $TBSM_HOME/bin/setTraceLevel.sh <tipadmin user> <tipadmin password> data -addtrace "com.micromuse.sla.*=info:com.ibm.tbsm.*=info:EventLogging=finer”


Enabling event logging2

Enabling Event Logging

  • Other setTraceLevel.sh options

    $TBSM_HOME/bin/setTraceLevel.sh <tipadmin user> <tipadmin password> data -display

    $TBSM_HOME/bin/setTraceLevel.sh <tipadmin user> <tipadmin password> data -setstartup

  • Event Logging should not be enabled permanently on customer sites – turn it off when not required.

    $TBSM_HOME/bin/setTraceLevel.sh <tipadmin user> <tipadmin password> data -resetall

    OR

    $TBSM_HOME/bin/setTraceLevel.sh <tipadmin user> <tipadmin password> data -rmtrace "EventLogging=finer"


Tbsm v6 1 troubleshooting events

  • Enable Event Logging

  • Results from event logging in trace.log

Section Separator


Tbsm v6 1 troubleshooting events

Collect new trace logs

  • Run <installdir>/tbsm/bin/collect_logs

    After running the utility on the TBSM Data server, the result will be a single zip file <installdir>/tbsm_logs_<datetimestamp>.zip

  • $TIP_HOME/TBSMProfile/logs/server1/ (trace.log, SystemOut.log, SystemErr.log)


Tbsm v6 1 troubleshooting events

Trace log Entries

  • With tracing set to EventLogging.

    • The first entry is an event receipt – shows all fields in the Event Container. TBSM will use the filters to match the event to a service.

  • [7/22/13 13:42:10:946 EDT] 00000044 EventLogging2 RADEventProcessor execute "Prepared byTBSMOMNIbusEventReader"=(EventContainer=(RAD_TimeWindowStart=0, AlertKey=mail, Location=, RAD_TimeWindowEnd=0, RemotePriObj=, LocalRootObj=, RAD_FunctionName=, NmosEntityId=0, RemoteNodeAlias=, X733EventType=0, ReceivedWhileImpactDown=0, RAD_UserFunctionName=NCOMS564, RAD_TotalRawMetric=, TaskList=0, ProbeSubSecondId=0, RAD_ServiceID=0, RAD_TimeWindowLength=0, RemoteRootObj=, Tally=1, KeyField=564, NodeAlias=, RAD_NextThresholdRawMetric=, RAD_ServiceTypeName=, NmosDomainName=, RAD_LastCount=1, RAD_CurrentFunctionState=, RAD_FilterIDList=2,3, Serial=564, X733CorrNotif=, …

  • <also seen in same message, I can see…>

    • Identifier=steJoyTESTEvent110,

    • Node=Mail3PortC,

    • Class=0,

    • Summary=JoySendEvent3,

    • Severity=3,…),

  • <the message ends with the following…>

  • Policies={Main}, PolicyName=Main, ReportsPolicyName=Main, ActionNodeName={Main}, ActionType=1)


Tbsm v6 1 troubleshooting events

Trace log Entries

  • Three parts of an Incoming Status Rule:

    (Discriminator, Filter Thresholds and Instance fields).

    1.Discriminator (or Class) is used to set the RAD_FilterIDList on the event.

[7/22/13 13:42:10:946 EDT] 00000044 EventLogging 2 MemoryBasedMatchingRawAttributeGetter getRawAttributeWrappersByEvaluatingFilters No Filters found in Event Container so must get filters.

[7/22/13 13:42:10:946 EDT] 00000044EventLogging2 MemoryBasedMatchingRawAttributeGettergetRawAttributeWrappersByEvaluatingFilters Usingdiscriminator: 0.


Tbsm v6 1 troubleshooting events

Trace log Entries

2.Threshold Filter. Each field in the Event Container is tested against the Filter fields, to check if they match the threshold.

[7/22/13 13:42:10:946 EDT] 00000044 EventLogging 2 MemoryBasedMatchingRawAttributeGetter testThresholdMatch TestingAlertKeywith comparator = 'mail'.

[7/22/13 13:42:10:946 EDT] 00000044 EventLogging 2 MemoryBasedMatchingRawAttributeGetter testThresholdMatch fieldMatched is: true.

[7/22/13 13:42:10:947 EDT] 00000044 EventLogging 2 MemoryBasedMatchingRawAttributeGettertestThresholdMatch TestingSeveritywith comparator >= '3'.

[7/22/13 13:42:10:947 EDT] 00000044 EventLogging 2 MemoryBasedMatchingRawAttributeGetter testThresholdMatch fieldMatched is: true.

[7/22/13 13:42:10:946 EDT] 00000044 EventLogging 2 MemoryBasedMatchingRawAttributeGetteraddFilterIDToEvent NewRAD_FilterIDList for Event is to set to: 2.

[7/22/13 13:42:10:946 EDT] 00000044 EventLogging 2 MemoryBasedMatchingRawAttributeGetterprintln testThresholdMatch Threshold matched and status is greater than 0.


Tbsm v6 1 troubleshooting events

Trace log Entries

3.Instance field. TBSM reads the value for the Event fields from the event, and then returns a list of Service Instances which match this.

[7/22/13 13:42:10:948 EDT] 00000044 EventLogging 2 AbstractMatchingInstanceGetter getInstanceIDFieldValueMappingForRawAttributeReturning field expression and value for attribute: {Node=Mail3PortC}.

[7/22/13 13:42:10:948 EDT] 00000044 EventLogging 2 MemoryBasedMatchingInstanceGetter getMappingStringForRawAttribute Returning event string for raw attribute: Mail3PortC.

[7/22/13 13:42:10:948 EDT] 00000044 EventLogging 2 GlobalInstanceFieldStore getCommaSeparatedInstanceIDs TagId is 85. attributeID is 1. eventMatchingString is Mail3PortC. List of Service Instance Ids which match raw attribute is: 89.

[7/22/13 13:42:10:948 EDT] 00000044 EventLogging 2 MemoryBasedMatchingInstanceGetter getMatchingInstanceIDsFromDataStoreReturning matching instances: [89].

2

1

3

4


Tbsm v6 1 troubleshooting events

Trace log Entries

  • Update State in TBSM

[7/22/13 13:42:10:948 EDT] 00000044 EventLogging 2 RADEventProcessor sendInstancesToStateModel1 Send to statemodel for ServiceInstanceName Mail3PortC ServiceinstanceID: 89.

[7/22/13 13:42:10:948 EDT] 00000044 EventLogging 2 StateModelUpdater passToStateModel Statusis 3.0.


Tbsm v6 1 troubleshooting events

Trace log Entries

  • Update Object Server service_deps table

[7/22/13 13:42:10:948 EDT] 00000044 EventLogging 2 StateModelNodeAttributeState updateServiceDeps About to update service_deps in memory and at object server for SIB: Mail3PortC.

[7/22/13 13:42:10:948 EDT] 00000044 EventLogging 2 ObjectServerUpdater updateServiceDeps updateEventIDs for Mail3PortC and eventInstanceID NCOMS:_:564.


Tbsm v6 1 troubleshooting events

Trace log Entries

  • Update Service Name in Event

[7/22/13 13:42:10:948 EDT] 00000044 EventLogging 2 StateModelUpdater setVarsInContexts RAD_ServiceNameis: Mail3PortC being set in Event Container.


Tbsm v6 1 troubleshooting events

Trace log Entries

  • Propagate up parents

[7/22/13 11:41:22:151 EDT] 00000036 EventLogging 2 StateModelNodeAttributeState stateChanged About to update parents for current SIB.

[7/22/13 11:41:22:151 EDT] 00000036 EventLogging 2 StateModelNodeAttributeState updateParentInstance Updating state of parent:Mail3PortC.

[7/22/13 11:41:22:153 EDT] 00000036 EventLogging 2 StateModelNodeAttributeState updateParentInstance Updating state of parent:MailServer3.

[7/22/13 11:41:22:156 EDT] 00000036 EventLogging 2 StateModelNodeAttributeState updateParentInstance Updating state of parent:App-Root.


Tbsm v6 1 troubleshooting events

  • Results from event logging in trace.log

  • How to use event logging filters

Section Separator


Event logging with filters

Event Logging with Filters

Filters reduce the number of events which are eligible for Event Logging

Filters are dynamic

Filters are persistent

Steps:

First eventloggingfilter Table must created

Then use radshell commands to set up filters for Event Logging


Event logging with filters1

Event Logging with Filters

For TBSM 4.2.1 - create the postgres table eventloggingfilter

Created by running the SQL file: createEventLoggingFilterTable.sql (This is shipped with TBSM 4.2.1 FP3)

UNIX:

cat ./createEventLoggingFilterTable.sql | $TBSM_HOME/bin/rad_db log

Windows:

type createEventLoggingFilterTable.sql | %TBSM_HOME%\bin\rad_db log


Event logging with filters2

Event Logging with Filters

For TBSM 6.1.X - create the DB2 table eventloggingfilter

From a command line start db2

db2

connect to TBSM(to connect to DB)


Event logging with filters3

Event Logging with Filters

For TBSM 6.1.X - create the DB2 table eventloggingfilter

Create Table

create table TBSMBASE.eventloggingfilter

(

filterfieldname varchar(255),

filterfieldvalue varchar(1024)

)


Event logging with filters4

Use radshell commands to set up filters for Event Logging

Displays the current filter setting

Clear the filter

Adds fields and values to the current filter

Removes a field from the filter

Event Logging with Filters

$TBSM_HOME/bin/rad_radshell

displayEventLoggingFilter();

clearEventLoggingFilter();

addUpdateEventLoggingFilterValue(“fieldName”,”fieldValue”);

removeEventLoggingFilterValue(fieldName);


Tbsm v6 1 troubleshooting events

Filter examples: add a field to current filter:

addUpdateEventLoggingFilterValue(“fieldName”,”fieldValue”);

Only events with Node = “Mail3PortC” are logged, when the trace level for EventLogging package is finer

addUpdateEventLoggingFilterValue(“Node”,”Mail3PortC”);

Event Logging with Filters


Tbsm v6 1 troubleshooting events

Event Logging with Filters

Filter examples: add a field to current filter:

addUpdateEventLoggingFilterValue(“fieldName”,”fieldValue”);

  • To change the filter, so that only events with Node = “Mail3PortC”,AND AlertKey = “mail”, are logged, when the trace level for EventLogging package is finer

  • addUpdateEventLoggingFilterValue(“Node”,”Mail3PortC”);

  • addUpdateEventLoggingFilterValue(“AlertKey”,”mail”);


Tbsm v6 1 troubleshooting events

Event Logging with Filters

Filter examples:


Tbsm v6 1 troubleshooting events

Event Logging with Filters

Filter examples: remove a field from current filter:

removeEventLoggingFilterValue(fieldName);

  • To remove the Node fields from the filter (now filter will just match on AlertKey)

  • removeEventLoggingFilterValue(“Node”);


Tbsm v6 1 troubleshooting events

Event Logging with Filters

Filter examples: remove a field from current filter:

clearEventLoggingFilter();

  • Remove ALL filter fields – Logging will be done for all events if the trace level for EventLogging package is finer.

  • clearEventLoggingFilter();


Tbsm v6 1 troubleshooting events

The filter set up by the addUpdateEventLoggingFilterValue radshell command, is only used when the trace level for the EventLogging package is set to finer.

If EventLogging package is not set, the radshell filter has NO effect.

Event Logging with Filters- Turn off afterwards


  • Login