Tivoli Workload Scheduler Distributed

1. Tivoli Workload Scheduler Distributed Managing Events in a Tivoli Workload Scheduler Environment

2. Introduction Using the Event-Driven Workload Automation you can: • Solve many workload automation scenarios • Perform on-demand workload automation • Carry out a predefined set of actions in response to events Event- driven workload automation performs on-demand workload automation and plan-based job scheduling. This automation defines rules that can trigger on-demand workload automation. The object of event-driven workload automation in Tivoli Workload Scheduler is to carry out a predefined set of actions in response to events that occur in the environment. The focus of this session will be to demonstrate how you can solve workload automation scenarios by using event-driven workload automation.

3. Event Event Event Event-Driven Workload Automation basic concepts An Event-Driven Workload Automation Rule is a composition of: • Events • E.g. “file x is created”, “message xyz issued in a log-file“, “Job x abended with RC=12” , “TWS agent unlinked”, “An email is received”, “Event xyz issued on SAP systemx” • Events Correlation Conditions • E.g. Filter, Events Sequence, Events Set • Actions • E.g. “Submit a TWS jobStream” “Send an e-mail”, “Send An event to T/EC”, Write a message in the message log Correlation rule Sample scenarios • Submit a Job Stream when a defined set of events is received • When file /abc/file1 is FTP’ed on machine-x and at least one of the TWS job* running on the Workstation machine-x is in successful status, then start the TWS jobstream-y • Scenario #2 – Trigger a credit transfer transaction and notify result • If msg “EGS0243 Transaction A terminated” is issued in logfile /var/log1 on machine-x, start TWS job_exec_tr1 • If event “job_exec_tr1=succesfull” is received in 1 hour send an Informational email to joe.smith@company.com otherwise send an Error email to the same user Action Action

4. Event Event Event Correlation rule Action Action Event Rule Concepts • The main concept of the event-driven scheduling/notification architecture is the Event Rule • The Event Rule object is a composition of: • Events e.g. “The file x on machine y is created” • Events Correlation Conditions e.g.“event_x and event_y are received in a specific or random order”, • Actionse.g “submit job_x on machine_y”, “send an email to joe_smith@abc.com”, • The Event Rule can be associated to a: • Validity time interval • Status: draft or complete

6. Event Event Event Correlation rule Action Action Event Rule modeling The supported Events are: • File created, file modification completed, file deleted, log message written • A message is issued on a log-file (e.g. may be an application log file) • TWS specific monitors • Job/jobstream status changed and return code condition • Status of TWS agents (linked/unlinked, started/stopped) • Application server status changed • Prompt status changed • Generic event. May also be an event generated via a Command Line Interface that will be provided by TWS and that can be used by customer to generate custom events • SAP batch related events (using TWS 8.4 for Applications)

7. Event Event Event Correlation rule Action Action Event Rule modeling The supported Actions are: • Submit a TWS Job, submit a TWS Job Stream, submit ad-hoc TWS Job • Answer a TWS Prompt • Send an e-mail • Send an EIF event to T/EC, ITM or NetCool • Write a message in the message log

8. Deploy (Manual/ Scheduled) Create Event Rule (not built yet) Builds Rule Editor Rules Builder Event Rule (built) Events, Correlation Rule, Actions Ah-Hoc Job submission Event-Rules Engine Deploys Perform Actions Jobs status notification • send Email • send T/EC event Events Monitors Listen & Correlate • Modeling • Create, Modify, Delete, List • Starts the monitoring activity • Activates the events receiving • Activates the correlation and actions triggering • Activation & Deploy • Query and browse event rule instances • Drill down on actions executed for a specific event rule • Monitoring & Management Event-Rules: the Life Cycle big picture EDWA life cycle

9. Can be done using the Tivoli Dynamic Workload Console (aka TWS Web UI) and the TWS Command Line Interface (composer) • Documented APIs are provided Modeling Activation & Deploy • It can be automated using the option manager (optman) • A new customization of the planman CLI is provided for manual or scheduled activation • Can be done both using the Tivoli Dynamic Workload Console (aka TWS WebUI) and the TWS Command Line Interface (aka TWS composer CLI) • Documented APIs are provided Monitoring & Management User Interfaces

10. Creating an event rule: General information While in draft status, an event rule is not deployed in Tivoli Workload Scheduler. When draft status is changed to complete, the event rule is compiled and deployed. 10

11. Adding events

12. Actions Tivoli Workload Scheduleraction: Submit predefined job, job stream or ad hoc job Reply to prompt Notification action: Send an e-mail Forward an event to Tivoli Enterprise Console Log a message to file Custom action: Run custom commands on the event processor server

13. Specifying actions

14. Possible customer scenarios • Scenario #1 – Simple notification • You define the following Event Rule • When jobs ABC*#* terminates in error, thensend an e-mail to the john.smith@mycorp.com. In the subject and the body of the e-mail the text includes the key of the job ended in error. • The Event Rule is valid from Dec 1st to Dec 31th, in the time-window 18:00-22:00 EST • You save the Event Rule that is automatically activated by the TWS • TWS starts monitoring all the specified jobs, and as soon as one of the specified jobs ends in error, it send the notification e-mail • You receive the e-mail and check the status of the jobs.

15. Possible customer scenarios • Scenario #2 – Trigger TWS Agents status • You define the following Event Rule • When any the Workstation CPU1 is unlinked and the Workstation CPU1 is not linked back within 10 minutes, send an Error email to bill_miller@company.com • You save the Event Rule that is automatically activated by the TWS • TWS starts monitoring the status of the Workstation CPU1 • As soon as the Workstation status is unlinked and the TWS starts the 10 minutes timeout • If the event “CPU1 linked” is not received in 10 minutes, TWS sends the Error email • Bill Miller receives the Error email, goes to the TWS Event Rule management interface, queries the actions/rules that was triggered in the last 10 minutes and from there navigates to the CPU1 instance where can perform a first problem analysis

16. Possible customer scenarios • Scenario #3 – Submit a Job Stream when FTP completed • You define the following Event Rule • When a file is created in directory /abc on machine_1and the same file is no more modified, then start the TWS jobstream-y • The Event Rule is valid from Dec 1st to Dec 31th, in the time-window 18:00-22:00 EST • You save the Event Rule that is automatically activated by the TWS • TWS starts monitoring file /abc/file1 on machine_1, and as soon as the file is created and it is not anymore in use, it submits the jobstream-y • You check if the Event Rule was triggered in the last two hours and see its status

17. Possible customer scenarios • Scenario #4 – Trigger long duration Jobs based on timeout • You define the following Event Rule • When the event “job-x=exec” and the event “job-x=succ/abend” are received in 5 minutes TWS should reply Yes to prompt-1 and starts the jobstream-z otherwise it should send an Informational email to the tws-oper@company.com alerting for a Job in late • You save the Event Rule in draft status. After few days the user reedits the rule, changes the email recipient and save it as not-draft • The Event Rule is automatically activated • Once the job-x status is exec, TWS starts the 5 minutes timeout • If the event “job-y successful or abend” is not received in 5 minutes, TWS sends the Informational email otherwise it replies Yes to prompt-1 and submits the jobstream-z

18. Event is matched? CFG file FileMonitor Events Workflow SSM reads CFG and monitors the system EIF reads trap and sends event to TWS yes Add FileMonitor rule Create .cfg file for SSM agent no Create the rule with TWS TWS creates the cfg file CFG file is deployed to SSM SSM reads the file and monitors the system When the event is matched the SSM generates a trap EIF listens to the trap and sends event to TWS TWS executes the action correlated with rule

19. Composer new features • Is possible to specify in “composer new” command which kind of object you want to add. • For example: -composer new eventRule • Composer is able to manage eventRule objects with commands: • create – extract • delete • display • list - print • lock - unlock • modify • new • rename Local encoding is used by default For XML encoding eventRule definition is completely written in XML unlike other TWS objects written in Scheduling Language

20. Planman command line: event rule activation • Is possible to perform the build and deployment of the event rules using directly the planman CLI: • planman deploy [-scratch] • where: • -scratch option builds again all rules defined in the database. • This is an example of the output running “planman deploy”.

21. Planman command line : event rule activation • planman command line activates all valid event rules and deploys the monitoring configuration to the specific agents. • Theevent rules that are not in draft status and are in the valid time interval are periodically (the frequency interval can be customized) checked for changes by TWS that automatically builds or refreshes the internal correlation and monitoring information for those rules. To get this result a new optman keyword has been created: deploymentFrequency / df = 5 Note: The default value is 5 minutes.

22. Monitor triggered actions

23. Monitoring Events

24. Conman new features (1/2) • The enhancements are provided by new commands that you can use to: • Download the latest monitoring configuration for the event monitoring engine on the workstation, using deployconf command (permission to “start” action on the specified workstation is required in the security file): • deployconf [domain!]workstation • Manage the monitoring process using startmon and stopmon commands (permissions to “start” and “stop” actions on the specified workstation are required in the security file): • startmon [domain!]workstation [;noask] • stopmon [domain!]workstation [;wait] [;noask]

25. Conman new features (2/2) • Start and stop the WebSphere Application Server process using startappserver and stopappserver commands (permissions to “start” and “stop” actions on the specified workstation are required in the security file): • startappserver [domain!]workstation [;wait] • stopappserver [domain!]workstation [;wait] • Manage the new event processing server used in event-driven workload automation using starteventprocessor, stopeventprocessor and switcheventprocessor commands (permissions to “start” and “stop” actions on the specified workstation are required in the security file): • startevtproc [domain!]workstation • stopevtproc [domain!]workstation • switchevtproc [domain!]workstation

26. EDWA Using the TDWC Web ConsoleCreating or editing an Event Rule

27. Engine on which the Rule is created/edited Draft/Complete status Validity interval Timeout conditions Event correlation types Common attributes correlation Timeout actions Event Rule Editor: Overview The Event Rule Editor is the tool you use to create and edit Event Rules You open the Rule Editor directly from the TDWC portfolio: - Workload Definition - Event Management - New Event Rule Or you open it by selecting an Event Rule from a query table and click “Edit” Events Actions

28. Event Rule Editor: Overview The Event Rule Editor connects to a specific engine to store or retrieve the Event Rule. As soon as you select an engine, the Editor connects to the engine and loads the list of event and action plug-ins available on it. When the connection is successful, the Editor opens in its entirety. The Event Rule, whether existing or a new one, is locked on the TWS database and is kept locked until the Editor is “Closed”. You can then save the progresses on the Rule multiple times while you are working on it. The Rule will be kept locked all the time. Select an Engine Connection and click Go

29. Dates and times expressed here will be referred to the TWS engine Time zone unless otherwise specified here The Rule will be valid every day between the specified dates, but only within the hours specified here. An empty value for one or both extremes means midnight. A Daily end prior to Daily start is a valid range and represents a range that crosses midnight. In this case refer to the manuals for what are the effects on the first and last validity day. The Rule will be valid between the specified dates. An empty value for one or both extremes means indefinite. Event Rule Editor: General Information This section contains general information about the Event Rule and its validity period. A “Draft” Rule will not be activated or deployed, so it will not generate events and will not run actions. A “Non Draft”, or “Complete” Rule will be activated and deployed at next “Planman –deploy” (either scheduled or manual) and will be active within its Validity Period. The changes you make to a Rule (including changing the “Draft” flag) will be effective only after the next deploying of the Rule.

30. Add an Event by clicking here Select the type of correlation: • Single event, • Unordered Set of events, • Ordered Sequence of events Click here to change Complete Information available clicking here Remove an Event by clicking here To expand/collapse a section click the title bar Event Rule Editor: Events

31. Enable Timeout and specify a value here Select properties common to all events here. They must all match for the rule to be satisfied Event Rule Editor: Timeout and Correlation

32. The Properties are displayed below Select an Event by clicking it to specify its properties A name is generated and assigned to each Event automatically. You can change it here Multiple, semi-colon-separated, values not allowed Wildcards are allowed for this property Remove Optional properties clicking here A non-empty value is required Specify a comparison operator Add optional properties from this list Event Rule Editor: Event Properties Event Properties specify a filtering condition for events. A condition on a property is specified with a comparison operator and one or more values. The values are in OR condition. Wildcards may or not be allowed. Filtering on some properties may be mandatory. You can add optional properties from the drop-down list. A property can be added multiple times. These are in AND condition.

33. Use Machine-readable format to use variables as input to scripts, programs, etc… Select an Event from the list of events currently added to the rule, and then one of its Properties. A Variable pointing to the selected property will be inserted at the end of the current input field A variable chooser helps inserting variables. Variables are used to take values from Event properties to use into Action properties You can use Variables for any Action Property Event Rule Editor: Actions and Variables Actions Properties do not specify a comparison operator, because they are actually parameters for the Action Some properties are mandatory. You can add optional properties from the drop-down list. A property can be specified only once.

34. Specify search criteria and press Search. The list of results appears below. Click on an item to select it, or change the criteria and search again. Event Rule Editor: Lookup of TWS objects When specifying properties related to TWS objects, a Lookup facility helps selecting an existing object in the TWS database. Lookup is available for Jobs, Job Streams and Workstations. Looking-up an object will fill-in all the object properties in the Action, not only the property next to which the button appears. Press the Lookup button here

35. Switch by clicking here Event Rule Editor: Timeout Actions If you enabled Timeout, switch to the Timeout Actions tab to work with these

36. Event Rule Editor: User Input Validation An Event or Action box turns RED if there are outstanding errors with it. Click the box to resolve them in the Properties section Errors are detected while you are typing and are shown below the corresponding field

37. Event Rule Editor: User Input Validation Errors related to invalid ranges are outlined with a box around the corresponding fields

38. Event Rule lifecycle management • Event Rules • Associate Event filters with response and timeout Actions • Can be listed using a query task • On the resulting table, you can drill-down to see a rule’s properties in read-only, and perform administrative actions on it

39. Event Monitoring using the TDWC Web Console

40. Event Rule monitoring • Event Rule Instances • Represent event rules that have been matched • An Event Rule Instance includes information like event rule name, date/time when it was matched and the list of actions that were triggered, whether the rule was satisfied or timed-out and other • An Event-Rule Instance is created each time the event conditions contained in the an event-rule are matched or you have defined timeout actions and the rule times-out

41. Event Rule monitoring • Triggered Actions • Represent actions that have been run as result of an event rule being matched • A triggered action instance includes information like action type, action name, date/time when it was executed and if it was executed successfully or not • A triggered action instance is created for each action that is executed when an event rule is matched

42. Event Rule monitoring • Operator Messages • An Operator Message is a particular type of action instance that includes a textual field containing the message text • Operator Messages represent another view on the Triggered Actions list, specific for actions of the “Log Message” type • This view shows each message that was logged as result of a Log Message action with a more event console-like look&feel • You will be able to see for each Operator Message a corresponding record also in the Triggered Actions view

43. Event Rule monitoring • Workstations • The “workstations in plan” view has been enhanced to show event monitoring information • You can see whether a workstation is the Event Processor Server • You can see whether the SSM Monitoring Agent is running • You can see whether the workstation has received the latest monitoring configuration • You can perform administrative actions on the selected workstation(s)

44. Troubleshooting EDWA

45. Global Options • General options for EDWA: • enEventDrivenWorkloadAutomation (ed)enables or disables the event driven workload automation featuredefault = YES (EDWA is enabled) • deploymentFrequency (df)specifies how often event rule updates are applied on the event processor and deployed to the TWS agentsdefault = 5 (minutes) • enEventProcessorHttpsProtocol(eh)enables or disables use of the HTTPS protocol to connect to the event processor serverdefault = YES (HTTPS is enabled) • eventProcessorEIFPort(ee)port of the Tivoli Event Integration Facility (EIF) for communication with the event processor serverdefault = 31131 • logHistory (lh)number of days for which history of triggered rule instances, action runs and logged messages are kept in the TWS databasedefault = 10 • logCleanupFrequency (lc)specifies how often to look for history data older than logHistory days, which must be deleteddefault = 5 (minutes)

46. Global Options • Options used by MailSender action plugin: • smtpServerName (sn)hostname or IP address of the SMTP server through which outgoing e-mails are delivereddefault = localhost • smtpServerPort (sp)port of the SMTP server through which outgoing e-mails are delivered default = 25 • smtpUseSSL (us)enables or disables use of Secure Sockets Layer (SSL) to secure the connection to the SMTP server default = NO (SSL is disabled) • smtpUseTLS (tl)enables or disables use of Transport Layer Security (TLS) to secure the connection to the SMTP serverdefault = NO (TLS is disabled) • smtpUseAuthentication (ua)enables or disables user/password authentication against the SMTP serverdefault = NO (authentication is disabled) • smtpUserName (un)name of the user to be used to authenticate with the SMTP server (if authetication is enabled)default = name of the tws_user • smtpUserPassword (up)password of the user to be used to authenticate with the SMTP server (if authetication is enabled)default = not set • mailSenderName (ms)name of the mail sender to be used for e-mails sent by TWSdefault = TWS

47. Local Options • Options added to the localopts file for EDWA: • AUTOSTART MONMAN • Tells whether or not the monman process for monitoring of generated events should be automatically started at the startup of this CPU • the monman process can be also controlled manually by using the “conman startmon”and “conman stopmon”commands •  default is YES • CAN BE EVENT PROCESSOR • Tells whether or not this CPU can take responsibility of being an event processor (e.g. as a result of a “conman switcheventprocessor” command) • only the master domain manager and the backup master can be event processors •  default is YESfor master DM and backup master DM, NOfor other CPUs

48. Troubleshooting/Diagnostics (EDWA) Scenario: An event rule does not trigger the required action. • Check if the event management feature is enabled • Use the optman command line to verify : • enEventDrivenWorkloadAutomation / ed = YES • Action: If the property is set to NO, run the command: optman chg ed=YES and run JnextPlan –for 0000 • Check if the event processor is installed, up and running, and correctly configured • Run conman showcpus • CPUID RUN NODE LIMIT FENCE DATE TIME STATE METHOD DOMAIN • CPU_MASTER 11 *WNT MASTER 0 0 09/03/07 09:51 I JW MDEA MASTERDM • FTA1 11 WNT FTA 0 0 LT MASTERDM

49. Troubleshooting/Diagnostics (EDWA) • The STATE field has a lower-case e If the STATE field has a lower case e, the event processor is installed but not running. Action: Start the event processor by conman: conman startevtproc • The STATE field has no M If the STATE field has no M, monman is not running. Action: Start monman process by conman: conman startmon • The STATE field has no D If the STATE field has no D, the current monitoring package configuration is not deployed. Action: Force the deployment by conman: conman deploy Note: If there is neither an upper-case E nor a lower-case e, the event processor is not installed. The event processor is installed by default on the master domain manager and backup master domain manager. If you are working on either, then the installation did not complete correctly.

50. Troubleshooting/Diagnostics (EDWA) • Check if the rule has been added to the monitoring configuration on the workstation. • Check if the rule is present in the workstation monitoring configuration by running the command: conman <workstation>;getmon Monitoring configuration for CPU_MASTER: **************************************************** *** Package date : 2007/09/03 07:48 GMT *** **************************************************** TEST1::FileMonitor#FileCreated:C:\TEMP\FILE5.TXT ON CPU_MASTER; TEST1::TWSObjectsMonitor#JobSubmit:* # * . TEST*; • Check in the <TWS-home>/monconf if the configuration is present • Check if the rule is active: Use Web UI interface or composer command line to verify the state of the rule: • For example by composer -list er=@ Event Rule Name Type Draft Status Updated On Locked By ------------------ --------- ----- ------- ---------- -------------- TEST1 filter N active 09/03/2007 –