1 / 57

Bart Drewes - PowerPoint PPT Presentation

  • Uploaded on

Bart Drewes. Product Manager VERITAS Software. Inside the Black Box. Tuning Oracle E-Business Suite Applications. Agenda. Introduction/ERP Overview Success Criteria for ERP Performance Management Some Real World Experiences Starting Point to ERP Tuning Closing Summary.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Bart Drewes' - wendell

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Bart drewes l.jpg

Bart Drewes

Product ManagerVERITAS Software

Inside the black box l.jpg

Inside the Black Box

Tuning Oracle E-Business Suite Applications

Agenda l.jpg

  • Introduction/ERP Overview

  • Success Criteria for ERP Performance Management

  • Some Real World Experiences

  • Starting Point to ERP Tuning

  • Closing Summary

Why erp applications l.jpg
Why ERP Applications?

  • Shorten the time and expense (theoretically!) required to deliver high quality business solutions

  • Buy (vs. build) functionality whenever and wherever possible

  • Avoid the expense and risk of building (and supporting) applications from scratch

  • Allow focus on core business functions

  • Minimize efforts which dilute core competency

  • Maintain competitiveness through employment of latest technology

Erp application characteristics l.jpg
ERP Application Characteristics

  • Wide-ranging core functionality that can be customized to meet specific customer requirements

  • Support for multiple platforms to accommodate today’s heterogeneous computing environments

  • Modular design to enable customers to “mix & match” application components to fit their functional business requirements

A need for performance l.jpg
A Need for Performance

  • Because ERP solutions are typically implemented to automate mission-critical core business functions, the requirement for high performance and throughput must be met

  • E-business modules further increase the exposure of poor performance

  • ERP-based applications inherently tend to be the most transaction – and throughput – intensive of your entire portfolio

Built in performance l.jpg
Built-in Performance?

  • Well…

    Unfortunately, the requirement of high performance cannot be explicitly designed into the application as a core functional attribute

  • But why not…?

    The problem stems from the fact that each customer environment is inherently different – different customizations, usage patterns, data volumes, etc.

Performance limitations from erp design l.jpg
Performance Limitations from ERP Design

  • Wide-ranging, customizable core functionality

    • Each customer/site will customize uniquely

  • Support for multiple platforms

    • Tuning is often platform- and release-specific

  • Modular design in support of “mix & match”

    • Modular application components

    • Modular architectures

Complex multi tier architectures l.jpg




Complex, Multi-tier Architectures

Web Server






“The More Tiers,

the More Tears!”

Common perception on erp tuning l.jpg
Common Perception on ERP Tuning

  • It’s no use

  • Vendors do not allow

  • How can you tune black boxes?

  • Changes cause more problems

  • No way to measure performance gain

  • Would never catch up with dynamic business changes

  • Just follow the vendor’s best practice guidelines

Success factors basic erp performance management l.jpg
Success Factors - Basic ERP Performance Management

  • Holistic view of application performance behavior

  • Ability to measure end-to-end response time, and segment into application tiers

  • Continuous performance monitoring with little overhead

  • Visibility to application structure

  • Correlation of information across application tiers

  • Ability to track true user activities across tiers

  • Ability to track application module performance across tiers

  • In-context drill down for root cause analysis

  • Quick identification of performance bottleneck

Success factors advanced erp performance management l.jpg
Success Factors - Advanced ERP Performance Management

  • Capture of historical performance data to enable:

    • Analysis of problems in the past

    • Baseline behavior for exception analysis

    • Trend analysis

    • Capacity planning

    • Building of Application knowledge base

  • Ability to do proactive performance management

  • Simulation for performance impact from changes

  • Understanding of application usage patterns in relation to business requirement changes

Veritas i 3 l.jpg



Where is the problem?


Do I or will I have a performance problem?




How is performance tracking over time?

How do I fix the problem?




Web Servers

App Servers




Case study l.jpg

Oracle Financials 11i running on a 4-tier architecture

Two Form Servers balanced by the Forms Metric Server

Eight worldwide offices are connected to the system

Case Study

The System

Oracle Database Server

Web Clients


Web Server

Form Servers

Slide16 l.jpg

StartPoint shows the system architecture. It provides us with a dashboard view of the system’s health

Each sphere represents one tier of the architecture.

The atmosphere represents the alerts that relate to the tier

We detect a performance alert in the Oracle Application server


Slide17 l.jpg

with a dashboard view of the system’s healthTop Activity’ alert is part of the ‘Performance’ alert metric set. It indicates that an OA activity response time has breached the performance threshold

Slide18 l.jpg

We were able to drill down with a dashboard view of the system’s health‘in context’ to find the problematic Form that breached the response time.

We found the Transaction form

This form exceeded the response time that we set for it by 15% of its normal response time

Slide19 l.jpg

We are looking at the Oracle Applications Savvy in the Oracle Apps tier

We see most of the processing time is in the Oracle Database tier

We have the database users that called the Transaction form

Slide20 l.jpg

We are in the Oracle Workspace Oracle Apps tier

This displays the Oracle Activity associated with the Transaction Form

SQL Statements associated with Transaction Form.

One SQL statement has high response time

Slide21 l.jpg

We selected the most time consuming SQL statement and looked at it using the Over Time view

It has increased dramatically.

Let us drill down in context on this statement

Slide23 l.jpg

We have drilled down and are able to see the Form, the Users, the longest running statements

We see this statement is the eight longest running statement over the long term

Slide26 l.jpg

We uncover that the table RA_CUSTOMER_TRX_ALL has recently grown significantly. This table is accessed by the long running query

Slide27 l.jpg

The Over Time graph verifies this grown significantly. This table is accessed by the long running query

Slide30 l.jpg

These are the columns in the Index grown significantly. This table is accessed by the long running query

Slide31 l.jpg

The number of Distinct values have not changed although the table size has significantly increased

Slide34 l.jpg

Look at the behavior of the Insert statement over time. RA_CUSTOMER_TRX_ALL table

This statement consumed major I/O resources on the 31st August

Slide35 l.jpg

To find what activated the Insert statement we associate it to the Forms. It show us that it is the RAXTRX.exe (Autoinvoice Import Program)

Slide36 l.jpg

To improve the situation we use the to the Forms. It show us that it is the RAXTRX.exe (Autoinvoice Import Program)SmarTune feature and receive index recommendations

Slide37 l.jpg

Simulate Changes check the affect of adding the index, both positive or negative for all statements

Slide38 l.jpg

Verify the changes by looking at the response time statistics of the original problematic statement

Slide39 l.jpg

The Over Time graph verifies that the response time has returned to the normal value after the index implementation

Slide40 l.jpg

The Over Time graph verifies that the response time has returned to the normal value after the index implementation

Slide41 l.jpg

Alerts are gone returned to the normal value after the index implementation

Case study42 l.jpg
Case Study returned to the normal value after the index implementation

  • Detect

    • Identify an upsurge in form response time

  • Find

    • Problem is associated with slowed down performance of the database

  • Focus

    • An SQL statement has deteriorated in performance due to increases in table size

  • Improve

    • A suitable index was introduced

  • Verify

    • The index has improved SQL time, and consequently the performance of the problematic form

Starting point to erp tuning l.jpg
Starting Point To ERP Tuning returned to the normal value after the index implementation

  • Majority of performance challenges end up in database tier

  • Start measuring and collecting performance indicators ASAP

  • Set reasonable targets based on measurable performance indicators

  • Identify top 10 tuning targets (by user, by business transaction, or other application entity)

Large return recognizable through database tuning oracle as example l.jpg
Large Return Recognizable Through Database Tuning - Oracle as Example

  • Object Design and Maintenance

  • Query Optimization

    • Object statistics

    • Indexing

  • Environmental/Configuration Considerations

  • Lifecycle Issues

Object design and maintenance dos l.jpg
Object Design and Maintenance — DOs as Example

  • Review base table and index growth - monitor extents and fragmentation - frequently; preferably weekly

  • If possible, anticipate (and plan for) occasional rebuilds of your database (after lots of testing)

  • Consider a larger database block size - 8K is a good nominal value for large sites/implementations

  • Make sure your busy tables and indexes are load-balanced for optimal I/O (try to isolate large transaction-intensive objects in their own tablespaces)

  • Try to educate your users on the importance of workload management - distribute log files and teach them how to interpret them

Object design and maintenance dos ii l.jpg
Object Design and Maintenance — as ExampleDOs (II)

  • Pay special attention to temporary and interface tables - monitor their size very closely, and truncate them as often as possible to reclaim space and reset the high water mark in the header block

    • This can have a dramatic effect on full table scans!

  • Monitor your index usage patterns (more on this later)

Object design and maintenance don ts l.jpg
Object Design and Maintenance — as ExampleDON’Ts

  • Never accept the default storage parameters

    • Diligent review should be standard policy

  • Don’t expect “linear” growth

    • Data volume

    • Transaction volume

    • Server capacity

    • Users’ expertise

    • User demand on performance

Query optimization l.jpg
Query Optimization as Example

  • Cost-based optimization is prevalent

    • Tables must be ANALYZEd periodically to maintain accurate data distribution statistics

    • Check standard ANALYZE utility scripts packaged with ERP administration function to ensure sufficient sample size for ESTIMATE option

    • Histogram for certain tables may be critical for success

  • Track explain plan change over time from object statistics changes

Indexing most power tuning key l.jpg
Indexing – Most Power Tuning Key as Example

  • Heavy indexing is a common design feature –

    • Result from implicitly “generic” design approach

    • Monitor usage

    • Evaluate search efficiency

    • Informed decision to drop indexes

    • Informed decision to add appropriate indexes

  • Can’t change SQL text from standard ERP modules -- but can influence optimizer with proper index design

Indexing ii l.jpg
Indexing (II) as Example

  • All “standard” tuning practices with regard to indexes still apply in ERP environment:

    • Try to enhance clustering whenever and wherever possible (via reloads or CLUSTER creation)

    • Re-organize indexes as frequently as is feasible

    • Capitalize on larger block size (as previously mentioned) to reduce b-tree depth

    • Physically separate indexes from base table - separate spindles, not just LVs

    • Keep your statistics as current as possible to enhance the effectiveness of CBO hints

Environmental configuration considerations l.jpg
Environmental/Configuration Considerations as Example

  • Don’t “under-instance”, i.e., make sure you have enough Apps. database instances to effectively manage your entire lifecycle (more on this later)

  • If possible, try to maintain pseudo-equivalence between instances, i.e., make your test instances as close to production as possible, ideally with on-demand or short latency refresh capabilities

  • Size your systems with absolutely as much physical memory as you can afford - and then immediately budget for more!

Environmental configuration considerations ii l.jpg
Environmental/Configuration Considerations (II) as Example

  • Try to keep ad-hoc and OLAP reporting queries off your production instance; consider a short-latency reporting instance, preferably on a separate machine

  • “Pin to Win!” - Design a package pinning strategy

  • Adjust the PCTFREE parameter for static data, such as financial data for closed accounting periods

  • For tables which are subjected to high concurrent INSERT activity, consider building them with more than one free list

  • Tables with high UPDATE activity may benefit from a higher INITRANS value, although this does increase block-level overhead

Environmental configuration considerations iii l.jpg
Environmental/Configuration Considerations (III) as Example

  • Some influencing INIT.ORA parameters:

    • CURSOR_SPACE_FOR_TIME - when set to TRUE, prevents private SQL statements in the library cache from being de-allocated until application cursor is closed; use to enhance on-line transaction performance, bear in mind that you will probably want to increase the size of the shared pool library cache and the number of open cursors (watch out for ORA-01000 errors)

    • DB_FILE_MULTIBLOCK_READ_COUNT - set to a minimum of 32; this will greatly enhance the performance of full table scans (remember that the maximum for this value is OS-dependent, and calculated as OS maximum IO size/database block size)

Lifecycle issues l.jpg
Lifecycle Issues as Example

  • Formalize your change management policies and practices before you go live; some bureaucracy is not necessarily a bad thing...

  • Make sure you have a very thorough code/design review with your implementation consultants before they walk out the door - document everything! - and don’t forget database object parameters

  • If you are going to do any post-production customizations, make sure you have at least 3 (preferably full-sized) instances for migration - development, test, and QA

    • Need a separate instance specifically for validating patches

Lifecycle issues ii l.jpg
Lifecycle Issues (II) as Example

  • Remember - new/changed code will impact your maintenance schedule as well (i.e., defrags, rebuilds, extent monitoring, etc.)

  • Tricks of the Trade - you might be able to justify the additional capacity required by your full-scale development/testing database(s) as part of a disaster recovery strategy

Slide56 l.jpg

& as Example