Tuning and scalability for sonic
Download
1 / 47

Tuning and Scalability for Sonic - PowerPoint PPT Presentation


  • 517 Views
  • Uploaded on

Tuning and Scalability for Sonic. Analyzing, testing and tuning JMS/ESB performance. Jiri De Jagere. Solution Engineer EMEA. D I S C L A I M E R. D I S C L A I M E R. Setting Performance Expectations.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Tuning and Scalability for Sonic' - PamelaLan


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Tuning and scalability for sonic l.jpg

Tuning and Scalability for Sonic

Analyzing, testing and tuning JMS/ESB performance

Jiri De Jagere

Solution Engineer EMEA


Setting performance expectations l.jpg

D I S C L A I M E R

D I S C L A I M E R

Setting Performance Expectations

  • System performance is highly dependent on machine, application and product version. Performance levels described here may or may not be achievable in a specific deployment.

  • Tuning techniques often interact with specific features, operating environment characteristics and load factors. You should test every option carefully and make your own decision regarding the relative advantages of each approach.

Session ID: Session Title


Agenda l.jpg
Agenda

Analyzing, testing and tuning JMS/ESB performance

  • Methodology

  • Analysis

  • Testing

  • Tuning

Session ID: Session Title


Performance engineering terms l.jpg

Expert Tip: Limit scope to those test components that

are critical to performance and under your control

Performance Engineering Terms

“Platform”

“System Metric”

“Load” = “Sessions” * “Delivery Rate”

“System Under Test”

R

V

“Test Harness”

  • “Variable” =

    • client param,

    • app param,

    • system param

V

V

“Latency” = ReceiveTime – SendTime

“Test Components”

“External

Components”

Session ID: Session Title


Concepts partitioning resource usage l.jpg

% CPU

=

( Overhead

X

Message rate )

Load

svcs

(writes/msg)

(msg/sec)

(Writes/sec)

Bottom-Up Rule: Test and tune each component before you

test and tune the aggregate.

Concepts: Partitioning Resource Usage

  • Partitionable resources can be broken down as the sum of the contributions of each test component on the system

  • Total resource usage is limited by system capacity

    • Becomes the bottleneck as it nears 100%

    • Goal is linear scalability as additional resource is added

    • Vertical versus Horizontal scalability

  • Total latency is the sum across all resource latencies, i.e.:

    • Latency = CPU_time + Disk_time + Socket_time + wait_sleep

Session ID: Session Title


Concepts computer platform resources l.jpg
Concepts: Computer Platform Resources

CPU time

Memory

(in use, swap)

# Threads

Network I/O

(send/receive)

Disk I/O

(read/write)

Favorite tools: task mgr,perfmon, top, ping –s, traceroute

  • Use level of detail appropriate to the question being asked

  • Machine resource (such as %CPU) artifacts:

    • side effects from uncontrolled applications

    • timing of refresh interval

    • correlation with test intervals

  • Scaling across different platforms and resource types

Session ID: Session Title


The performance engineering project l.jpg

Expert Tip: Schedule daily meetings to share results and

reprioritize test plan.

The Performance Engineering Project

  • For each iteration

  • Test performance vs goal

  • Identify likeliest area for gain

Test

Analyze

Tune

  • Startup tasks

  • Define project completion goals

  • Staff benchmarking skills

  • Acquire test harness

The Project is Goal Driven

Session ID: Session Title


Performance project skills l.jpg
Performance Project Skills

Requirements Expert

  • SLA/QoS levels – minimal & optimal

  • Predicted load – current & future

  • Distribution topology

    Integration Expert

  • Allowable design options

  • Cost to deploy

  • Cost to maintain

    Testing Expert

  • Load generation tool

  • Bottleneck diagnosis

  • Tuning and optimization

R.E.

Cost/Benefit

Load/Distribution

SOLUTION

I.E.

T.E.

Design Options

Session ID: Session Title


Tools for a messaging benchmark l.jpg

Component

Platform

Tools for a Messaging Benchmark

System Under Test

  • Configurator – creates conditions to bring system under test into known state

  • Harness – the platforms and components whose performance response is being measured

  • Analyzer – tools and procedures to make meaningful conclusions based on result data.

Test

Analyzer

Test

Harness

Test

Configurator

Session ID: Session Title


Performance project timeline l.jpg

Design

Application Tuning

1 2 3 4 5 6 7 8

Performance Project Timeline

System Test

Development

Project

Deployment Plan

Service Dev

Sizing

Process Dev

Performance

Project

Perf Prototype

Launch

Week

Session ID: Session Title


Agenda11 l.jpg
Agenda

Analyzing, testing and tuning JMS/ESB performance

  • Methodology

  • Analysis

  • Testing

  • Tuning

Session ID: Session Title


Performance analysis l.jpg
Performance Analysis

  • Performance scenarios

    • Requirements

    • Goals

  • System characterization

    • Platforms

    • Architecture

  • Test cases

Session ID: Session Title


Performance scenario specification l.jpg

Rule of Thumb: Focus on broker loads over 10 msg/sec or 1 MByte/sec,

and service loads over 10,000 per hour.

Performance Scenario Specification

  • First, triage performance-sensitive processes

    • substantial messaging load and latency requirements

    • impact of resource-intensive services

  • Document only process messaging & services

    • leave out Application specific logic – this is a prototype

  • Set specific messaging performance goals

    • Message rate and size

    • Minimum and average latency required

  • Try to quantify actual value and risk

    • Why this use case matters

Session ID: Session Title


Example performance scenario specification l.jpg
Example: Performance scenario specification MByte/sec,

  • Overall project scope:

    • Project goals and process

    • Deployment environment

    • System architecture

  • For each Scenario:

    • Description of business process

    • Operational constraints (QoS, recovery, availability)

    • Performance goals, including business benefits

Session ID: Session Title


System characterization l.jpg
System Characterization MByte/sec,

  • Scope current and future hardware options available to the project

  • Identify geographical locations, firewalls and predefined service hosting restrictions

  • Define predefined Endpoints and Services

  • Define data models and identify required conversions and transformations.

Session ID: Session Title


Platform configuration specification l.jpg

Rule of Thumb: MByte/sec,

LAN 15 – 150 MBytes / second

Disk: 2 – 10 MBytes / second

XSLT: 200 – 300 KBytes / second

Platform configuration specification

Network

bandwidth

latency

speed

Field

DMZ

DMZ

CPU

number

type

speed

Memory

size

speed

Firewall

cryptos

latency

Disk

type

speed

Session ID: Session Title


Architecture spec service distribution l.jpg

Integration Broker MByte/sec,

Adapter

Adapter

ERP

Tracking Service

SFA

SCM

SCM

Finance

PoS

CRM

CRM

Field

Front Office

Back Office

Partner

Architecture Spec: Service distribution

ESB

ESB

ESB

Partner ESB

  • Identify services performance characteristics

  • Identify high-bandwidth message channels

  • Decide which components can be modified

  • Annotate with message load estimates

Session ID: Session Title


Architecture spec data integration l.jpg

? MByte/sec,

Data Schemas 1…n

Data Schemas 2…m

Expert Tip: Transform tools vary in efficiency:

XSLT – slowest (but most standard)

Semantic modeler – generally faster (e.g. Sonic DXSI)

Custom text service – fastest, but not as flexible

Architecture Spec: Data Integration

  • Approximate the complexity of data schemas

  • Identify performance critical transformation events

  • Estimate message size

  • Identify acceptable potential services

Session ID: Session Title


Platform profile real time messaging l.jpg

Rule of Thumb: Real-time, 1 KB messages, broker performance is about 1,000 to 10,000 msg/sec for each 1 gHz cpu power.

Platform Profile: Real-time messaging

System resource limitations

90%

5%

% capacity

20%

70%

Session ID: Session Title


Platform profile queued requests l.jpg

Rule of Thumb: Queued msgs, 1 KB messages, broker performance is about 100 to 1,000 msg/sec for each 1 gHz cpu power.

Platform Profile: Queued requests

System resource limitations

50%

85%

40%

20%

% capacity

Session ID: Session Title


Specifying test cases l.jpg
Specifying Test Cases performance is about 100 to 1,000 msg/sec for each 1 gHz cpu power.

  • Factors to include:

    • Load, sizes, complexity of messaging

    • Range of scalability to try (e.g. min/max msg size)

    • Basic ESB Process model

    • Basic distribution architecture

  • Details to avoid:

    • Application code (unless readily available)

    • Detailed transformation maps

  • Define relevant variables:

    • Fixed factors

    • Test Variables

    • Dependent measures

Session ID: Session Title


Typical test variables l.jpg

Expert Tip: External JMS client variables are easily managed performance is about 100 to 1,000 msg/sec for each 1 gHz cpu power.

with the Test Harness.

Typical test variables

  • JMS Client variables:

    • Connection / session usage

    • Quality of Service

    • Interaction mode:

    • Message size and shape

  • ESB container variables

    • Service provisioning and parameters

    • Endpoint / Connection parameters

    • Process implementation and design

Session ID: Session Title


Example test case specification l.jpg
Example: Test Case Specification performance is about 100 to 1,000 msg/sec for each 1 gHz cpu power.

For each identified Test Case there is a section specifying the following:

  • Overview of test

    • How this use case relates to the scenario

    • Key decision points being examined

  • Functional definition

    • How to simulate the performance impact

    • Description of ESB processes and services

    • Samples messages

    • Design alternatives that will be compared

  • Test definition

    • Variables manipulated

    • Variables measured

  • Completion criteria

    • Throughput and latency goals

    • Issues and options that may merit investigation

Session ID: Session Title


Agenda24 l.jpg
Agenda performance is about 100 to 1,000 msg/sec for each 1 gHz cpu power.

Analyzing, testing and tuning JMS/ESB performance

  • Methodology

  • Analysis

  • Testing

  • Tuning

Session ID: Session Title


Testing l.jpg
Testing performance is about 100 to 1,000 msg/sec for each 1 gHz cpu power.

  • Think about the approach you want to take

    • Bottom-up vs Top-down

  • Create a controllable environment

    • Simulate non-essential services

    • Use a testing tool for simulating clients

  • Evaluate the results

Session ID: Session Title


Simulating clients with test harness l.jpg

Broker performance is about 100 to 1,000 msg/sec for each 1 gHz cpu power.

System Under Test

Request

Test

Harness

JNDI

Connection

Msg

Pool

Reply

Message

Generation

Simulating clients with Test Harness

  • File or JNDI connection configuration

  • Producer / Consumer parameters

  • Message generation

Session ID: Session Title


Evaluating the results l.jpg

Expert Tip: Spreadsheets are excellent for documenting and interpreting

the results

Evaluating the results

  • Test Harness output:

    • Throughput (msg/sec)

    • Latency (msecs per round trip)

    • Message size (bytes)

  • System measures:

    • CPU usage (%usr, %sys)

    • Disk usage (writes/sec)

  • Broker metrics:

    • Messaging rate (bytes/second)

    • Peak queue size (bytes)

Session ID: Session Title


Evaluating the results28 l.jpg

Expert Tip: Spreadsheets are excellent for documenting and interpreting

the results

Evaluating the results

  • Evaluate against the goals

  • Perform a bottleneck analysis

  • Compare across test runs

Session ID: Session Title


Agenda29 l.jpg
Agenda interpreting

Analyzing, testing and tuning JMS/ESB performance

  • Methodology

  • Analysis

  • Testing

  • Tuning

Session ID: Session Title


The art of tuning l.jpg
The Art of Tuning interpreting

Pimp my ride!

  • Art of tuning

    • Change only 1 parameter at a time

    • Take your time for it

    • Always check the same things (even reboot/restart after each run)

    • Know what you are measuring

Session ID: Session Title


Java tuning options l.jpg

Rule of Thumb: On windows platforms, the Sun 1.5.0 JVM is interpreting

10% to 50% slower than the default IBM 1.4.2 JVM.

Java Tuning Options

  • ‘Fastest’ JVM depends a little on the application and a lot on the platform

  • VM heap

  • Garbage Collection:

    • default settings good for optimal throughput

    • use advanced (jdk4 or later) GC options to maximize latency

Session ID: Session Title


Broker tuning options l.jpg
Broker Tuning Options interpreting

  • Sources of broker CPU cost

    • CPU cost of i/o

      • Network sockets

      • Log/Data disks

    • Web Service protocol

      • Web Service security

    • Security

      • Authorization

      • Message or channel encryption

Session ID: Session Title


Broker tuning options33 l.jpg
Broker Tuning Options interpreting

  • Sources of broker disk overhead

    • Message recovery log

      • Persistent message store

      • Might not be used if other guarantee mechanisms work

    • Message data store

      • Disconnected consumer

      • Queue save threshold overflow

    • Flow to disk overflow

    • Message storage overhead depends on disk sync behavior

      • Explicit file synchronization ensures data retained on crash

      • Tuning options at disk, filesystem, JVM and Broker levels

Session ID: Session Title


Broker tuning parameters l.jpg

Rule of Thumb: For non-trivial queues, multiply default interpreting

settings by 10 to 100.

Broker Tuning Parameters

  • Core Resources:

    • JVM heap size

    • JVM thread, stack limits

    • DRA, HTTP and Transaction threads

    • TCP settings

  • Message storage:

    • Queue size and save threshold

    • Pub/sub buffers

    • Message log and store

  • Message management

    • Encryption

    • Flow control and flow-to-disk

    • Dead message queue management

  • Connections

    • Mapping to NIC’s

    • Timeouts

    • Load balancing

Session ID: Session Title


Messaging tuning options l.jpg

Expert Tip: With CAA configured, Best Effort service is equivalent

to At Least Once, with substantially lower overhead.

Messaging Tuning Options

Implement optimal QoS for speed versus precision

(Based on CAA brokers and fault-tolerant connections)

Session ID: Session Title


Messaging tuning options36 l.jpg

Msg equivalent

Msg

Msg

Msg

Broker

Ack

Ack

Messaging Tuning Options

Use message batching to accelerate message streams

Consumer

  • Message transfer overhead is generally fixed

  • Hidden ack messages amenable to tuning:

    • AsyncNonPersistent mode decouples ack latency

    • Transaction Commit allows 1 ack per N messages

    • DupsOK ack mode allows ‘lazy’ ack from consumer

    • Pre-Fetch Count allows batched transmit to consumer

  • ESB Design option: send one multi-part message instead of N individual messages

Producer

Session ID: Session Title


Flow control l.jpg

Point to Point equivalent

Potential

Receiver

Sender

Queue

Potential

Receiver

Flow Control

  • Flow Control in PtP:

    • If the Maximum Size of a Queue is reached, Flow Control is kicking in

    • Sender seems to be hanging but is only waiting for space in the Queue

    • Flow Control can be switched off and replaced by an Exception in your code.

Session ID: Session Title


Flow control38 l.jpg

Publish and Subscribe equivalent

Subscriber

Publisher

Topic

Subscriber

Flow Control

  • In Pub/Sub, the working of Flow Control is more difficult to predict

    • Each Subscriber gets a Buffer (Outgoing Buffer Size)

    • Sonic will try to only have 1 copy of the message in memory

    • When the buffer is reaching a threshold, Flow Control is issued.

    • Threshold depends on message priority

Session ID: Session Title


Flow control39 l.jpg

Outgoing Buffer equivalent

Outgoing Buffer

Pub/Sub

TEXT

TEXT

TEXT

TEXT

TEXT

TEXT

TEXT

TEXT

TEXT

TEXT

Flow Control

Ack

FLOW CONTROL

Only a pointer is inserted in the Buffer but the calculated size is the message size

Publish and Subscribe

Danger !!!

Slow Subscriber may result in the following situation

Subscriber

Publisher

Topic

Subscriber

Ack

Session ID: Session Title


Client tuning options l.jpg

Rule of Thumb: Up to 500 queues per Broker and equivalent

10,000 topics per broker.

Client Tuning Options

  • Reuse JMS objects to reduce setup cost

    • Objects with client and broker footprint

      • Connection

      • Session

      • Sender / Receiver / Publisher / Subscriber

    • Coding best practices

      • Share connections across sessions

      • Share sessions across producers / consumers

        • Not across threads!!

Session ID: Session Title


Esb tuning options l.jpg
ESB Tuning Options equivalent

X-scaling: Multiple Listeners

Y-scaling: Multiple JVMs

Z-scaling: Multiple Machines

Session ID: Session Title


Esb tuning options42 l.jpg

equivalent

Receive Msg

Broker

Dispatch Outbox

Instantiate Proc

Marshall Msg

Unmarshall Msg

Send Msg

Call onMessage

Dispatch Outbox

Unmarshall Msg

Marshall Msg

Call onMessage

Send Msg

ESB Tuning Options

Inter-Container Messaging

Intra-Container Messaging

v7.5: better!

faster!

Session ID: Session Title


Esb tuning options43 l.jpg
ESB Tuning Options equivalent

  • Transformations

  • XML Server

  • BPEL Server

  • Database Service

  • DXSI Service

Session ID: Session Title


In summary l.jpg
In Summary equivalent

  • Know what you’re doing!

  • If you don’t, get someone in who does!

  • Use good old plain common sense

Session ID: Session Title


For more information go to l.jpg
For More Information, go to… equivalent

  • PSDN

    • White paper: Benchmarking Enterprise Messaging Systems

    • Sonic Test Harness

  • Documentation:

    • Progress Sonic MQ Performance Tuning Guide

Session ID: Session Title


Slide46 l.jpg

Questions? equivalent

Session ID: Session Title


Slide47 l.jpg

Thank you for equivalentyour time

Session ID: Session Title


ad