Tuning and Scalability for Sonic - PowerPoint PPT Presentation

Tuning and Scalability for Sonic

Analyzing, testing and tuning JMS/ESB performance

Jiri De Jagere

Solution Engineer EMEA

D I S C L A I M E R

D I S C L A I M E R

• System performance is highly dependent on machine, application and product version. Performance levels described here may or may not be achievable in a specific deployment.
• Tuning techniques often interact with specific features, operating environment characteristics and load factors. You should test every option carefully and make your own decision regarding the relative advantages of each approach.

Session ID: Session Title

Analyzing, testing and tuning JMS/ESB performance

• Methodology
• Analysis
• Testing
• Tuning

Session ID: Session Title

Expert Tip: Limit scope to those test components that

are critical to performance and under your control

“Platform”

“System Metric”

“Load” = “Sessions” * “Delivery Rate”

“System Under Test”

R

V

“Test Harness”

• “Variable” =
• client param,
• app param,
• system param

V

V

“Latency” = ReceiveTime – SendTime

“Test Components”

“External

Components”

Session ID: Session Title

% CPU

∑

=

( Overhead

X

Message rate )

Load

svcs

(writes/msg)

(msg/sec)

(Writes/sec)

Bottom-Up Rule: Test and tune each component before you

test and tune the aggregate.

• Partitionable resources can be broken down as the sum of the contributions of each test component on the system
• Total resource usage is limited by system capacity
• Becomes the bottleneck as it nears 100%
• Goal is linear scalability as additional resource is added
• Vertical versus Horizontal scalability
• Total latency is the sum across all resource latencies, i.e.:
• Latency = CPU_time + Disk_time + Socket_time + wait_sleep

Session ID: Session Title

CPU time

Memory

(in use, swap)

# Threads

Network I/O

(send/receive)

Disk I/O

(read/write)

Favorite tools: task mgr,perfmon, top, ping –s, traceroute

• Use level of detail appropriate to the question being asked
• Machine resource (such as %CPU) artifacts:
• side effects from uncontrolled applications
• timing of refresh interval
• correlation with test intervals
• Scaling across different platforms and resource types

Session ID: Session Title

Expert Tip: Schedule daily meetings to share results and

reprioritize test plan.

• For each iteration
• Test performance vs goal
• Identify likeliest area for gain

Test

Analyze

Tune

• Startup tasks
• Define project completion goals
• Staff benchmarking skills
• Acquire test harness

The Project is Goal Driven

Session ID: Session Title

Requirements Expert

• SLA/QoS levels – minimal & optimal
• Predicted load – current & future
• Distribution topology

Integration Expert

• Allowable design options
• Cost to deploy
• Cost to maintain

Testing Expert

• Load generation tool
• Bottleneck diagnosis
• Tuning and optimization

R.E.

Cost/Benefit

Load/Distribution

SOLUTION

I.E.

T.E.

Design Options

Session ID: Session Title

Component

Platform

System Under Test

• Configurator – creates conditions to bring system under test into known state
• Harness – the platforms and components whose performance response is being measured
• Analyzer – tools and procedures to make meaningful conclusions based on result data.

Test

Analyzer

Test

Harness

Test

Configurator

Session ID: Session Title

Design

Application Tuning

1 2 3 4 5 6 7 8

System Test

Development

Project

Deployment Plan

Service Dev

Sizing

Process Dev

Performance

Project

Perf Prototype

Launch

Week

Session ID: Session Title

Analyzing, testing and tuning JMS/ESB performance

• Methodology
• Analysis
• Testing
• Tuning

Session ID: Session Title

• Performance scenarios
• Requirements
• Goals
• System characterization
• Platforms
• Architecture
• Test cases

Session ID: Session Title

Rule of Thumb: Focus on broker loads over 10 msg/sec or 1 MByte/sec,

and service loads over 10,000 per hour.

• First, triage performance-sensitive processes
• substantial messaging load and latency requirements
• impact of resource-intensive services
• Document only process messaging & services
• leave out Application specific logic – this is a prototype
• Set specific messaging performance goals
• Message rate and size
• Minimum and average latency required
• Try to quantify actual value and risk
• Why this use case matters

Session ID: Session Title

• Overall project scope:
• Project goals and process
• Deployment environment
• System architecture
• For each Scenario:
• Description of business process
• Operational constraints (QoS, recovery, availability)
• Performance goals, including business benefits

Session ID: Session Title

• Scope current and future hardware options available to the project
• Identify geographical locations, firewalls and predefined service hosting restrictions
• Define predefined Endpoints and Services
• Define data models and identify required conversions and transformations.

Session ID: Session Title

Rule of Thumb:

LAN 15 – 150 MBytes / second

Disk: 2 – 10 MBytes / second

XSLT: 200 – 300 KBytes / second

Network

bandwidth

latency

speed

Field

DMZ

DMZ

CPU

number

type

speed

Memory

size

speed

Firewall

cryptos

latency

Disk

type

speed

Session ID: Session Title

Integration Broker

Adapter

Adapter

ERP

Tracking Service

SFA

SCM

SCM

Finance

PoS

CRM

CRM

Field

Front Office

Back Office

Partner

ESB

ESB

ESB

Partner ESB

• Identify services performance characteristics
• Identify high-bandwidth message channels
• Decide which components can be modified
• Annotate with message load estimates

Session ID: Session Title

?

Data Schemas 1…n

Data Schemas 2…m

Expert Tip: Transform tools vary in efficiency:

XSLT – slowest (but most standard)

Semantic modeler – generally faster (e.g. Sonic DXSI)

Custom text service – fastest, but not as flexible

• Approximate the complexity of data schemas
• Identify performance critical transformation events
• Estimate message size
• Identify acceptable potential services

Session ID: Session Title

Rule of Thumb: Real-time, 1 KB messages, broker performance is about 1,000 to 10,000 msg/sec for each 1 gHz cpu power.

System resource limitations

90%

5%

% capacity

20%

70%

Session ID: Session Title

Rule of Thumb: Queued msgs, 1 KB messages, broker performance is about 100 to 1,000 msg/sec for each 1 gHz cpu power.

System resource limitations

50%

85%

40%

20%

% capacity

Session ID: Session Title

• Factors to include:
• Load, sizes, complexity of messaging
• Range of scalability to try (e.g. min/max msg size)
• Basic ESB Process model
• Basic distribution architecture
• Details to avoid:
• Application code (unless readily available)
• Detailed transformation maps
• Define relevant variables:
• Fixed factors
• Test Variables
• Dependent measures

Session ID: Session Title

Expert Tip: External JMS client variables are easily managed

with the Test Harness.

• JMS Client variables:
• Connection / session usage
• Quality of Service
• Interaction mode:
• Message size and shape
• ESB container variables
• Service provisioning and parameters
• Endpoint / Connection parameters
• Process implementation and design

Session ID: Session Title

For each identified Test Case there is a section specifying the following:

• Overview of test
• How this use case relates to the scenario
• Key decision points being examined
• Functional definition
• How to simulate the performance impact
• Description of ESB processes and services
• Samples messages
• Design alternatives that will be compared
• Test definition
• Variables manipulated
• Variables measured
• Completion criteria
• Throughput and latency goals
• Issues and options that may merit investigation

Session ID: Session Title

Analyzing, testing and tuning JMS/ESB performance

• Methodology
• Analysis
• Testing
• Tuning

Session ID: Session Title

• Think about the approach you want to take
• Bottom-up vs Top-down
• Create a controllable environment
• Simulate non-essential services
• Use a testing tool for simulating clients
• Evaluate the results

Session ID: Session Title

Broker

System Under Test

Request

Test

Harness

JNDI

Connection

Msg

Pool

Reply

Message

Generation

• File or JNDI connection configuration
• Producer / Consumer parameters
• Message generation

Session ID: Session Title

Expert Tip: Spreadsheets are excellent for documenting and interpreting

the results

• Test Harness output:
• Throughput (msg/sec)
• Latency (msecs per round trip)
• Message size (bytes)
• System measures:
• CPU usage (%usr, %sys)
• Disk usage (writes/sec)
• Broker metrics:
• Messaging rate (bytes/second)
• Peak queue size (bytes)

Session ID: Session Title

Expert Tip: Spreadsheets are excellent for documenting and interpreting

the results

• Evaluate against the goals
• Perform a bottleneck analysis
• Compare across test runs

Session ID: Session Title

Analyzing, testing and tuning JMS/ESB performance

• Methodology
• Analysis
• Testing
• Tuning

Session ID: Session Title

Pimp my ride!

• Art of tuning
• Change only 1 parameter at a time
• Take your time for it
• Always check the same things (even reboot/restart after each run)
• Know what you are measuring

Session ID: Session Title

Rule of Thumb: On windows platforms, the Sun 1.5.0 JVM is

10% to 50% slower than the default IBM 1.4.2 JVM.

• ‘Fastest’ JVM depends a little on the application and a lot on the platform
• VM heap
• Garbage Collection:
• default settings good for optimal throughput
• use advanced (jdk4 or later) GC options to maximize latency

Session ID: Session Title

• Sources of broker CPU cost
• CPU cost of i/o
• Network sockets
• Log/Data disks
• Web Service protocol
• Web Service security
• Security
• Authorization
• Message or channel encryption

Session ID: Session Title

• Sources of broker disk overhead
• Message recovery log
• Persistent message store
• Might not be used if other guarantee mechanisms work
• Message data store
• Disconnected consumer
• Queue save threshold overflow
• Flow to disk overflow
• Message storage overhead depends on disk sync behavior
• Explicit file synchronization ensures data retained on crash
• Tuning options at disk, filesystem, JVM and Broker levels

Session ID: Session Title

Rule of Thumb: For non-trivial queues, multiply default

settings by 10 to 100.

• Core Resources:
• JVM heap size
• JVM thread, stack limits
• DRA, HTTP and Transaction threads
• TCP settings
• Message storage:
• Queue size and save threshold
• Pub/sub buffers
• Message log and store
• Message management
• Encryption
• Flow control and flow-to-disk
• Dead message queue management
• Connections
• Mapping to NIC’s
• Timeouts
• Load balancing

Session ID: Session Title

Expert Tip: With CAA configured, Best Effort service is equivalent

to At Least Once, with substantially lower overhead.

Implement optimal QoS for speed versus precision

(Based on CAA brokers and fault-tolerant connections)

Session ID: Session Title

Msg

Msg

Msg

Msg

Broker

…

…

Ack

Ack

Use message batching to accelerate message streams

Consumer

• Message transfer overhead is generally fixed
• Hidden ack messages amenable to tuning:
• AsyncNonPersistent mode decouples ack latency
• Transaction Commit allows 1 ack per N messages
• DupsOK ack mode allows ‘lazy’ ack from consumer
• Pre-Fetch Count allows batched transmit to consumer
• ESB Design option: send one multi-part message instead of N individual messages

Producer

Session ID: Session Title

Point to Point

Potential

Receiver

Sender

Queue

Potential

Receiver

• Flow Control in PtP:
• If the Maximum Size of a Queue is reached, Flow Control is kicking in
• Sender seems to be hanging but is only waiting for space in the Queue
• Flow Control can be switched off and replaced by an Exception in your code.

Session ID: Session Title

Publish and Subscribe

Subscriber

Publisher

Topic

Subscriber

• In Pub/Sub, the working of Flow Control is more difficult to predict
• Each Subscriber gets a Buffer (Outgoing Buffer Size)
• Sonic will try to only have 1 copy of the message in memory
• When the buffer is reaching a threshold, Flow Control is issued.
• Threshold depends on message priority

Session ID: Session Title

Outgoing Buffer

Outgoing Buffer

Pub/Sub

TEXT

TEXT

TEXT

TEXT

TEXT

TEXT

TEXT

TEXT

TEXT

TEXT

Ack

FLOW CONTROL

Only a pointer is inserted in the Buffer but the calculated size is the message size

Publish and Subscribe

Danger !!!

Slow Subscriber may result in the following situation

Subscriber

Publisher

Topic

Subscriber

Ack

Session ID: Session Title

Rule of Thumb: Up to 500 queues per Broker and

10,000 topics per broker.

• Reuse JMS objects to reduce setup cost
• Objects with client and broker footprint
• Connection
• Session
• Sender / Receiver / Publisher / Subscriber
• Coding best practices
• Share connections across sessions
• Share sessions across producers / consumers
• Not across threads!!

Session ID: Session Title

X-scaling: Multiple Listeners

Y-scaling: Multiple JVMs

Z-scaling: Multiple Machines

Session ID: Session Title

…

Receive Msg

Broker

Dispatch Outbox

Instantiate Proc

Marshall Msg

Unmarshall Msg

Send Msg

Call onMessage

…

…

Dispatch Outbox

Unmarshall Msg

Marshall Msg

Call onMessage

…

Send Msg

Inter-Container Messaging

Intra-Container Messaging

v7.5: better!

faster!

Session ID: Session Title

• Transformations
• XML Server
• BPEL Server
• Database Service
• DXSI Service
• …

Session ID: Session Title

• Know what you’re doing!
• If you don’t, get someone in who does!
• Use good old plain common sense

Session ID: Session Title

• PSDN
• White paper: Benchmarking Enterprise Messaging Systems
• Sonic Test Harness
• Documentation:
• Progress Sonic MQ Performance Tuning Guide

Session ID: Session Title

Questions?

Session ID: Session Title

Thank you foryour time

Session ID: Session Title