slide1 l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Chicago, IL PowerPoint Presentation
Download Presentation
Chicago, IL

Loading in 2 Seconds...

play fullscreen
1 / 51

Chicago, IL - PowerPoint PPT Presentation


  • 652 Views
  • Uploaded on

®. C20. Using Multicast in real life scenarios. Murtuza Choilawala – Global Response Team (GRT). Chicago, IL. April 24-27, 2006. Using Multicast in real life scenarios. Presented by Murtuza Choilawala - GRT. Agenda. Introduction Multicast Distribution Overview

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

Chicago, IL


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
slide1

®

C20

Using Multicast in real life scenarios

Murtuza Choilawala – Global Response Team (GRT)

Chicago, IL

April 24-27, 2006

using multicast in real life scenarios

Using Multicast in real life scenarios

Presented by Murtuza Choilawala - GRT

agenda
Agenda
  • Introduction
  • Multicast Distribution Overview
  • Multicast Log Files and Commands
  • Configuring and Tuning Multicast
  • Multicast Troubleshooting
  • Multicast Performance Tuning
  • Multicast Performance Evaluation
  • Conclusion
introduction
Introduction
  • Tivoli Multicast became available with
    • Tivoli Management Framework v4.1.x
    • Tivoli Configuration Manager v4.2.x
  • Multicast data distribution can be a powerful solution when distributing bulk data to numerous targets in a large-scale environment
  • There are some considerations in multicast implementation
    • To make multicast stable and reliable
    • To improve multicast performance
  • Multicast very much depends on the environment such as network or background traffic
    • Some customer’s parameter settings may not work for other customers
    • Performance tuning approach will be different for each environment
    • Each customer has to find the best configuration for their environment
multicast distribution overview6
Multicast Distribution Overview
  • Preparation Phase:
    • Step 1: Prepare Files to be Distributed
    • Step 2: Create Software Package
  • Data Distribution Phase:
    • Step 3: Distribute Software Package
    • Step 4: Resend Dropped Packets
  • Installation Phase:
    • Step 5: Install Software Package
multicast distribution overview7

1

RMTPOpenReceiver()

3

RMTPOpenSender()

2

RMTPConnectReceiver()

CONN

RMTPConnectSender()

4

CACK

RMTPAcceptConnection()

5

Data Xfr

RMTPSendData()

6

RMTPReceiveData()

7

POLL

NACK

8

Data Xfr

ACK

9

RELEASE

RACK

RMTPClose()

RMTPClose()

Multicast Distribution Overview
  • Data Distribution Phase (CONN – DT – REL)
multicast log file and commands
Multicast Log File and Commands
  • To perform multicast troubleshooting or performance tuning, multicast log file and command output information are mandatory
    • Log files: Multicast distribution details
    • Command outputs: Multicast configurations, parameters, and distribution status
  • It is possible to gather multicast log files and command output by using scripts.
    • The script will enable you to automate the process of collecting log files and command outputs for each distribution
  • Need to understand which log files and command outputs are required for multicast troubleshooting and performance tuning
multicast log files
Multicast Log Files
  • mcast.log File
    • Most important log file for Multicast troubleshooting and performance tuning
    • Contains communication flow and detailed data transmission log information
    • Located at $DBDIR/mcast.log on repeaters
    • Debug level = 3 is recommended when tuning or troubleshooting multicast (default = 1)
  • lcfd.log File
    • Contains detailed data transmission process on a specific endpoint
    • Contains endpoint information about endpoint methods that are invoked on a specific endpoint via downcall
    • Located at $LCF_DATDIR/lcfd.log on endpoints
    • Debug level = 3 is recommended when tuning or troubleshooting multicast (default = 1)
multicast log files10
Multicast Log Files
  • gatelog File
    • Contains detailed information of endpoint methods execution and endpoint login
      • Software package installation
      • Starting multicast receivers
    • Located at $DBDIR/gatelog on gateways
    • Debug level = 9 is recommended when tuning or troubleshooting multicast
  • Software Distribution Log/Trace Files
    • Can be used for software package installation troubleshooting
    • Log file is located at $BINDIR/../SWDIS/WORK/
    • Trace file is located at $LCF_DATDIR/../../swdis/1/ on endpoints
    • Trace level = 5 is recommended when troubleshooting (default = 0)
  • epmgrlog File
    • Can be used for endpoint troubleshooting
    • Located at $DBDIR/epmgrlog on Tivoli Server
multicast commands
Multicast Commands
  • wmcast Command
    • Can be used for configuring multicast parameters
      • wmcast –s <repeater> [keyword=value…]
    • Can be used for browsing current multicast configuration
      • wmcast –s <repeater>
    • Can be used for verifying connections (multicast ping)
      • wmcast –p {<repeater> | all}
multicast commands12
Multicast Commands
  • wmcast –s Command Output

dtwtout: 3000 (ms)

relwtout: 2000 (ms)

connwtout: 2000 (ms)

connrtry: 5

dtrtry: 10

pollrtry: 5

relrtry: 5

ttl: 5

blocksize: 1460 (Bytes)

backofftm: 100 (ms)

repeat: 2

rcvbufsz: 250000 (Bytes)

sndbufsz: 250000 (Bytes)

mc_netload: 500000 (Bytes/sec)

recv_timeout: 1800000 (ms)

port: 9499

returnIP: 0.0.0.0

mcadvert: 224.0.1.118

mchigh: 224.2.255.255

mclow: 224.2.128.0

ifsrcaddr: 0.0.0.0

ifrcvaddrs: 0.0.0.0

mc_debug_level: 1

multicast commands13
Multicast Commands
  • wmdist: Configuring Parameters
    • Can be used for repeater configuration
      • wmdist –s <repeater> [keyword=value…]
      • Make multicast available on a specific repeater with one of the following parameters:
        • repeater_multicast
        • endpoint_multicast
        • default_multicast
        • fail_unavailable
    • Can be used for browsing current repeater configuration
      • wmdist –s <repeater>
multicast commands14
Multicast Commands
  • wmdist –s Command Output

repeater_id: 1554731128.1.577

rpt_dir: C:/Tivoli/db/ishii1.db/tmp/

permanent_storage: TRUE

max_sessions_high: 5

max_sessions_medium: 10

max_sessions_low: 40

disk_max: 500 (MB)

mem_max: 64 (MB)

send_timeout: 300 (secs)

execute_timeout: 600 (secs)

notify_interval: 30 (mins)

conn_retry_interval: 900 (secs)

retry_ep_cutoff: 7200 (secs)

net_load: 500 (KB/sec)

packet_size: 16 (KB)

target_netload: 0 (KB/sec)

debug_level: 3

repeater_multicast: FALSE

endpoint_multicast: FALSE

default_multicast: FALSE

SLOW_LINK: FALSE

multicast commands15
Multicast Commands
  • wmdist Command: Display Distribution Status
    • Can be used for displaying distribution status for each distribution
      • wmdist –l
    • Can be used for displaying distribution status for each target
      • wmdist –e <dist_id>
    • For more detailed distribution status:
      • wmdist –liv <dist_id>
multicast commands16
Multicast Commands
  • wmdist –l Command Output

Name Distribution ID Targets Completed Successful Failed

mcast_test^1 1978508757953160036 4 1( 25%) 0( 0%) 1( 25%)

mcast_test^2 1757544609953651663 1 1(100%) 0( 0%) 1(100%)

  • wmdist –e <dist_id> Command Output

Name Status Start Time End Time

tivoli CANCELED 2003.02.06 20:11:50 2003.02.06 20:59:01

a24922 SUCCESSFUL 2003.02.06 20:39:59 2003.02.06 20:44:32

a24927 SUCCESSFUL 2003.02.06 20:39:58 2003.02.06 20:44:28

a25065 SUCCESSFUL 2003.02.06 20:40:00 2003.02.06 20:45:01

a25592 SUCCESSFUL 2003.02.06 20:40:00 2003.02.06 20:44:52

a25593 SUCCESSFUL 2003.02.06 20:39:58 2003.02.06 20:44:49

a25594 SUCCESSFUL 2003.02.06 20:40:00 2003.02.06 20:44:23

  • The elapsed time includes software package installation process

Elapsed Time = Data Transmission Process + Installation Process

configuring multicast
Configuring Multicast
  • Send a multicast ping (wmcast –p) to all targets with the default parameters
  • Ensure that multicast can be enabled between between sender and receiver
  • If some receivers do not respond to multicast ping, the receiver must have some problems

STEP 1

Sending Multicast Ping

Sending Data via Multicast

STEP 2

  • Send data via multicast with the default parameters
  • Check the distribution status with wmdist and look into the problems with the appropriate log files if the distribution failed

Tuning Multicast Parameters

  • Ensure that data can be distributed via multicast without any errors
  • Tune multicast parameters and configurations to improve multicast distribution performance and throughput

STEP 3

step1 sending multicast ping
Step1: Sending Multicast Ping
  • Data Stream of multicast ping between sender and receiver is the same as when transmitting data to the target via multicast
  • Multicast ping allows us to make sure whether multicast is enabled between sender and receiver
  • If multicast ping fails for a target, multicast data transmission to the target cannot be completed
  • Multicast ping is the first thing to do and should be exercised before sending data via multicast
    • wmcast –p

a00028 Multicast ping was successful.

a00068 Multicast ping was successful.

a00069 Multicast ping was successful.

a00062 Multicast ping was successful.

a00009 Multicast ping was successful.

a00062 Multicast ping was successful.

a00010 Multicast ping failed.

a00053 Multicast ping was successful.

step 1 sending multicast ping

START

wmcast –p

Success?

Yes

No

Check Multicast Configuration

Check Receiver Status/Reachability

Check Software Prerequisites

Collect Multicast Log Files

Check Miscellaneous

Check Network Configuration

GO TO STEP2

Step 1: Sending Multicast Ping
  • Recommended Checkpoints for Multicast Ping Problems
step 1 sending multicast ping20
Step 1: Sending Multicast Ping
  • If multicast ping fails on an endpoint, make sure of a couple of check points and take a look at log files and messages corresponding to the endpoint
    • Network and endpoint connectivity
    • mcast.log, lcfd.log file, etc.
    • Endpoint is unreachable
    • Receiver process is not running
    • Multicast is not enabled
    • TTL (Time-to-Live) configuration Typical multicast ping problemsM
    • Multicast advertisement setting for the gateway matches the endpoint

<MCAST_ADVERTISEMENT>

<MCAST_TEST_ADVT>MCAST_TEST_ADVT</MCAST_TEST_ADVT>

<SENDER>1808441727.1</SENDER>

<PACKAGE_ID>testAdvert</PACKAGE_ID>

<PACKAGE_VERSION>testAdvert</PACKAGE_VERSION>

<DISTRIBUTION_ID>testAdvert</DISTRIBUTION_ID></MCAST_ADVERTISEMENT>

step 1 sending multicast ping21

1

RMTPOpenReceiver()

3

RMTPOpenSender()

2

RMTPConnectReceiver()

RMTPConnectSender()

4

RMTPAcceptConnection()

5

RMTPSendData()

6

RMTPReceiveData()

7

8

9

RMTPClose()

RMTPClose()

Step 1: Sending Multicast Ping

Multicast Data Transmission Flow

CONN

connwtout * connrtry

CACK

Data Xfr

pollwtout * pollrtry

POLL

NACK

dtwtout * dtrtry

Data Xfr

ACK

relwtout * relrtry

RELEASE

RACK

step 1 sending multicast ping22
Step 1: Sending Multicast Ping
  • Multicast Data Transmission (Multicast Ping) Flow: Log Messages in lcfd.log File

Feb 04 18:25:35 3 RMTP 64 Receive CONN packet from [10.94.142.128:39513] ackcnt[0]

Feb 04 18:25:35 3 RMTP 64 Set back-off time[9] - range[100]

Feb 04 18:25:35 3 RMTP 64 << CLOSEDCONN -------------------------------------->>

Feb 04 18:25:35 3 RMTP 64 Datasize[289] Msgsize[478405499] Msgnum[1] MsgID[0] Blocksize[7300] Blocknum[1]

Feb 04 18:25:35 3 RMTP 64 Set destination port[39513] connection_id[44860]

Feb 04 18:25:35 2 MCASTR 64 Entering decrypt_in_place...

Feb 04 18:25:35 2 MCASTR 64 Leaving decrypt_in_place, rc = 0.

Feb 04 18:25:35 3 MCASTR 64 Advertisement After ConnectReceiver:

Feb 04 18:25:35 3 MCASTR 64 RMTP version: 4 Datasize: 289

Feb 04 18:25:35 3 MCASTR 64 Optional Data: <?xml version="1.0" ?>

<MCAST_ADVERTISEMENT>

<MCAST_TEST_ADVT>MCAST_TEST_ADVT</MCAST_TEST_ADVT>

<SENDER>1808441727.1</SENDER>

<PACKAGE_ID>testAdvert</PACKAGE_ID>

<PACKAGE_VERSION>testAdvert</PACKAGE_VERSION>

<DISTRIBUTION_ID>testAdvert</DISTRIBUTION_ID></MCAST_ADVERTISEMENT>

  • Multicast (including multicast ping) exchanges XML format data that contains information of the distribution data in the same manner as actual data transmission flow (CONN – DT – REL) before transmitting actual data (i.e. software package). This process is called multicast advertisement and multicast ping also performs multicast advertisement at the beginning of the distribution
step 1 sending multicast ping23

Feb 04 18:25:45 3 RMTP 64 Receive DT packet from [10.94.142.128:39513]

Feb 04 18:25:45 3 RMTP 64 Receive POLL[1007] packet from [10.94.142.128:39513] cyclecnt[1] ackcnt[5]

Feb 04 18:25:45 3 RMTP 64 Send ACK[1002] to [10.94.142.128:39513]

  • Sender sent DT (Data), and then receiver received POLL from the sender. The entire data was received and the receiver sent ACK to the sender

Feb 04 18:25:35 3 RMTP 64 Receive CONN packet from [10.94.142.128:39513] ackcnt[0]

Feb 04 18:25:35 3 RMTP 64 Receive CONN packet from [10.94.142.128:39513] ackcnt[0]

Feb 04 18:25:35 3 RMTP 64 Send CACK[1009](YES) to [10.94.142.128:39513]

  • Receiver received CONN and then the receiver sent CACK to the sender as response
Step 1: Sending Multicast Ping

Feb 04 18:25:57 3 RMTP 64 Receive REL packet from [10.94.142.128:39513]

Feb 04 18:25:57 3 RMTP 64 Receive REL packet from [10.94.142.128:39513]

Feb 04 18:25:57 3 RMTP 64 Send RACK[100b] to [10.94.142.128:39513]

Feb 04 18:25:57 3 RMTP 64 << CLOSED ------------------------------------------>>

  • Receiver received REL from sender. As a result, the receiver sent RACK and the multicast data transmission (multicast ping) was completed
step 2 sending data via multicast
Step 2: Sending Data via Multicast
  • If Step 1 is completed, send actual data via multicast
    • Start with the default parameters
    • To perform in-depth analysis, send a large data (i.e. 500MB or larger)
  • To make log analysis efficient, minimize log messages
    • Disable ‘Retry Unicast’ option
    • Stop receiver process on targets that you do not want to send data
  • Check the distribution status with wmdist –e <dist_id> command
  • If distribution fails
    • To analyze the distribution and detect the problems, gather appropriate log information
    • Check typical multicast distribution problems
  • If distribution is successfully completed
    • To tune multicast distribution performance, modify appropriate parameters and check the results
step 2 sending data via multicast25
Step 2: Sending Data via Multicast
  • Recommended Checkpoints for Multicast Data Distribution Problems

START

Multicast

Data Distribution

Success?

Yes

No

Check ADVT_TIMEOUT Error

Check ABORT Error

Check execute_timeout Error

Set backofftm Properly

Check Miscellaneous

GO TO STEP3

step 3 tuning multicast parameters
Step 3: Tuning Multicast Parameters
  • Once multicast becomes stable, multicast performance tuning needs to be done
  • Multicast performance tuning will very much depend on environment; so there is no standard
  • Starting with mc_netload parameter tuning is recommended
  • Key points in multicast performance tuning
    • Environmental Characteristics: OS, Number of targets, Distribution requirements, The other applications running on the same box or network, etc.
    • Network Characteristics: Type of network (WAN, LAN, etc.), Network bandwidth, Background traffic, etc.
    • Distribution Characteristics: Data size, Distribution frequency, Distribution time (day or night), Typical distribution, etc.
step 3 tuning multicast parameters27
Step 3: Tuning Multicast Parameters

START

  • Recommended Checkpoints for Multicast Performance Tuning

Tune mc_netload

Any Large

Software Package?

No

Distribute via

Satellite or WAN?

No

Yes

Check Depot and Staging Config

Yes

Tune blocksize and messagesize (*)

Tune Multicast Buffer

More than

50 Endpoint

Targets?

No

Tune Miscellaneous Configurations

Yes

Tune OS and Network

Increase max_sessions

END

multicast troubleshooting
Multicast Troubleshooting
  • Typical multicast distribution problems
    • Multicast timeout (ABORT) error
    • Wait timeout (ADVT_TIMEOUT) error
    • Execute_timeout error
    • Time-to-Live (TTL) count
  • The bottom line is to make multicast distribution successful distribute to all targets and make multicast distribution stable
multicast timeout abort error
Multicast Timeout (ABORT) Error
  • Multicast timeout (ABORT) error is one of the most typical problems in multicast data distribution
    • Especially in a large-scale environment
  • ABORT error will occur when
    • Multicast dropped a lot of packets during communication
    • Multicast reached its timeout
    • A network problem occurred during communication
  • Receiver cannot hear sender, or sender cannot hear receiver during multicast distribution
    • As a result multicast reaches timeout on the sender side
multicast timeout abort error30
Multicast Timeout (ABORT) Error

How ABORT error occurs during multicast distribution

  • Multicast communication starts
  • Sender sends CONN to receiver
  • Sender does not receive any response from receiver
  • Sender retries sending CONN several times (defined in parameters)
  • Sender cannot receive a response from receiver until it reaches timeout (connrty * connwtout)
  • Sender sends an ABORT packet
  • When receiver sees the ABORT packet, receiver quits listening and the multicast communication between sender and receiver terminates
multicast timeout abort error31
Multicast Timeout (ABORT) Error
  • ABORT error usually occurs in connection phase (CONN – CACK) or advertisement phase
    • During multicast connection phase, all receivers were sending back CACK’s to the sender within a short period and it dropped a lot of packets
  • If ABORT error occurs, you will see log message
    • mcast.log file
    • lcfd.log file

Feb 04 20:00:39 3 RMTP 257 Receive ABORT[1006] packet from [10.94.142.128:42018] nameid[1808441727(10.161.202.107) 174]

Feb 04 20:00:39 3 RMTP 257 << CLOSED ------------------------------------------>>

multicast timeout abort error32
Multicast Timeout (ABORT) Error
  • To resolve ABORT error, the following parameters need to be tuned
    • backofftm
      • wmcast –s <repeater_name> backofftm=time
    • Timeout (connwtout, pollwtout, dtwtout, and relwtout)
    • Retry (connrtry, pollrtry, dtrtry, and relrtry)
      • wmcast –s <repeater_name> connwtout=time
  • Especially backofftm, connwtout and connrtry parameters are key because ABORT error usually occurs in the early phase of multicast communication
multicast timeout abort error33
Multicast Timeout (ABORT) Error
  • Typical ABORT error during connection phase in a large environment
    • Sender tries to complete multicast connection with all receivers within a short period
    • Network is overloaded and it causes a timeout error
    • Need to spread the data out during multicast connection by backofftm, connwtout, and connrtry
  • ABORT error does not occur

in a small environment

multicast timeout abort error34
Multicast Timeout (ABORT) Error
  • backofftm parameter
    • Can be set which will tell receivers to pick a random number between 0 ~ backofftm and wait that amount of time before sending a reply
    • The backofftm spreads the packets out.
    • This reduces congestion and gives sender time to easily handle the packets from receivers
    • As a result, it will improve the

throughput of the entire distribution

    • The backofftm should be less than 65535
    • When increasing backofftm, the *tout settings (connwtout, dtwtout, and relwtout) should be larger than backofftm. Otherwise timeout will occur
    • If there are numerous targets, the default backofftm (100 ms) may be too short to handle multicast communication
advt timeout error
ADVT_TIMEOUT Error
  • Wait Timeout (ADVT_TIMEOUT) is receiver side timeout
  • ADVT_TIMEOUT occurs when receiver reaches timeout after multicast advertisement phase
  • If multicast data transmission terminated with an error on an endpoint and you do not see ABORT packet in the lcfd.log file, it might be caused by ADVT_TIMEOUT
    • If you see the following message in the lcfd.log file, it indicates that ADVT_TIMEOUT occurred during multicast communication

Feb 04 18:25:13 3 RMTP 97 Wait timeout occurred with [15500] msec. Timeout value [15000]

advt timeout error36
ADVT_TIMEOUT Error
  • How to resolve ADVT_TIMEOUT
    • ADVT_TIMEOUT is defined in mcast_receiver.cfg file on endpoint
      • By default ADVT_TIMEOUT = 15000 (ms)
    • ADVT_TIMEOUT can be resolved by increasing ADVT_TIMEOUT in mcast_receiver.cfg file
    • Try ADVT_TIMEOUT = 20000 or higher and see what happens
      • If you still see ADVT_TIMEOUT, then increase ADVT_TIMEOUT furter
      • If you increase connwtout and connrtry, ADVT_TIMEOUT also will need to be increased

ADVT_TIMEOUT >= connwtout * connrtry

advt timeout error37
ADVT_TIMEOUT Error
  • mcast_receiver.cfg file
    • Endpoint will look at mcast_receiver.cfg file on gateway anytime multicast receiver is started
    • If the version of gateway’s mcast_receiver.cfg is different from the endpoint’s, then the endpoint will download the gateway version
    • mcast_receiver.cfg file is located at $BINDIR/../lcf_bundle.40/bin/<interp>/TMF/LCF
    • Modifying parameters in mcast_receiver.cfg is not recommended unless you encounter ADVT_TIMEOUT. Normally default parameters should work for your environment
advt timeout error38
ADVT_TIMEOUT Error
  • How to modify mcast_receiver.cfg file
    • If multicast is not enabled on the gateway
      • Modify the mcast_receiver.cfg on the gateway, then when multicast is enabled the endpoint will download the updated mcast_receiver.cfg
    • If multicast is already enabled on the gateway
      • Modify the mcast_receiver.cfg, and the next time the endpoint logins, it will get the updates

Or

      • Modify the mcast_receiver.cfg, then run the following commands. This will restart receivers, which will cause them to download the updated mcast_receiver.cfg
        • wmdist –s <repeater_name> endpoint_multicast=FALSE
        • wmdist –s <repeater_name> endpoint_multicast=TRUE
    • On the Endpoint : Make a copy of mcast_receiver.cfg and name it mcast_receiver.conf for changes to be used on an individual endpoint:

$LCF_DATDIR/mcast/mcast_receiver.conf

BUT ONLY INCLUDE THOSE VARIABLES THAT ARE TO BE OVERRIDDEN:

For example, on an endpoint with multiple NICs, it may be necessary to specify IFADDR. After copying mcast_receiver.cfg to mcast_receiver.conf, remove (or comment out with "#") all other variables except IFADDR.

execute timeout error
Execute_timeout Error
  • Execute_timeout error is one of typical problems in multicast installation phase especially when distributing large data
  • Execute_timeout occurs only after multicast data transmission is completed
  • Execute_timeout is used to specify the amount of time that an endpoint method has to return a result after data distribition
  • When sending a large software package, the software package installation (copy) process may hit execute_timeout
  • In multicast distribution execute_timeout can occur even if there is no user program defined in the software package
  • When execute_timeout occurs, the distribution status is changed from WAITING to INTERRUPTED
  • The following log message can be seen in lcfd.log file

1 21 21:15:42 Q cm_operation_ep2 mdist: Unable to find entry for segment 1gb^1.

1 21 21:15:42 Q MethInit method returned.

execute timeout error40
Execute_timeout Error
  • How to resolve execute_timeout error
    • Execute_timeout can be configured via wmdist command
      • wmdist –s execute_timeout=time
    • By default, execute_timeout = 300 (sec)
    • If you planning to distribute large software packages (greater than 1 GB), increase execute_timeout and ensure that execute_timeout is large enough
      • Perform distribution test with 2 GB software package (distribution itself cannot be larger than 2GB)
      • For instance, start with execute_timeout=2000
time to live ttl count
Time-to-Live (TTL) Count
  • wmcast command allow you to specify Time-to-Live (TTL) count
    • wmcast –s <repeater_name> ttl=count
    • The default value is 5
  • When a packet is passed a router, TTL count is decremented; when it reaches 0, the packet is dropped
    • Need to specify a number larger than the number of routers between sender and receiver (255 is maximum)
  • In a large-scale environment, the default value is sometimes too small
  • Check how many hops your network requires prior to distribution (i.e. by ping command)
    • Some network device, such as VLAN, may deal with TLL count in a different manner
multicast performance tuning
Multicast Performance Tuning
  • Recommended performance tuning points in multicast data distribution
    • Use different Advertisement address along with mclow-mchigh range
    • Multicast bandwidth rate (mc_netload)
    • Multicast data transmission and MTU
    • Local disk copy
    • Multicast buffer
    • Operating systems, network, etc.
  • Look for the best configurations and parameters for your environment
mcadvert mchigh mclow
MCADVERT, MCHIGH, MCLOW
  • In large env changing MCADVERT, MCHIGH and MCLOW will improve performance
  • Provides separate groups for targets below different gateways
  • Example:

Mcadvert: 224.0.1.118  224.10.1.1

Mclow: 224.2.1.18  224.10.1.2

Mchigh 224.2.1.255  224.10.1.255

  • Change the above values to different subnets on different gateways
  • All three addresess can be changed using following commands

wmcat –s <repeater_name> mcadvert=value

wmcat –s <repeater_name> mchigh=value

wmcat –s <repeater_name> mclow=value

multicast bandwidth mc netload
Multicast Bandwidth (mc_netload)
  • Multicast data transmission speed can be configured via mc_netload parameter
    • wmcat –s <repeater_name> mc_netload=value
    • By default mc_netload = 500,000 (bytes/sec)
  • Tuning multicast bandwidth (mc_netload) is good starting point for multicast performance tuning
  • If mc_netload is set too high, many packets will be dropped and the dropped packets need to be resent which causes performance to suffer
  • Look for the best bandwidth for your environment
    • Link speed
    • Software package size
    • Background traffic
multicast bandwidth mc netload45
Multicast Bandwidth (mc_netload)

How to determine the best value for mc_netload parameter

  • Start with the default value (500 kb/s) and measure data distribution performance
  • Increase and decrease mc_netload, and measure data distribution performance. For instance, try mc_netload=600 (kb/s) and 400 (kb/s)
  • If you see significant improvement with either mc_netload=400 (kb/s) or 600 (kb/s), repeat step 2 with different bandwidth and see the differences
  • To find best value for mc_netload, try a couple software packages (data size) for step 1–3
  • If you do not see significant difference between step 1 and step 2, the default value is the appropriate value for the environment
    • A large software package is recommended for mc_netload tuning
    • mc_netload tuning should be done in production network because background traffic will affect multicast data distribution performance
    • To make further analysis, capture all distribution data such as mcast.log, wmcast, wmdist command output, etc.
    • There are network router settings to increase the priority of UDP packets and this setting may affect multicast performance
multicast bandwidth mc netload46
Multicast Bandwidth (mc_netload)
  • Analyzing dropped packets
    • To recognize your network characteristics, analyzing dropped packets is recommended
      • Leverage the distribution log files captured during mc_netload tuning test
    • Key information can be seen in multicast log files
      • Elapsed time of distribution (not including installation)
      • How many times data was resent
    • This information should be able to help you in configuring and tuning mc_netload properly
local disk copy

Receive

Multicast

Local Disk Copy
  • When distributing a large software package, the installation process will take a long time
    • Multicast receiver stores the entire data in a local file (depot) , then the data is copied to target directory
  • To improve installation process performance, secure enough free disk space
    • Enable multicast to store at least twice the size of distribution data, preferably three times or more
    • Avoid disk fragmentation on target endpoints
  • Multicast installation process internally performs a ‘local copy’ from the depot to the installation target directory
    • Software package is uncompressed during the installation
    • By default, depot is located at $LCF_DATDIR/depot
multicast buffer
Multicast Buffer
  • Tivoli multicast provides parameters that enable you to configure buffer size of UDP socket on sender and receiver sides
  • rcvbufsz and sndbufsz parameter on sender
    • The default value is 250,000 bytes
    • rcvbufsz and sndbufsz can be configured via wmcast command
      • wmcast –s <repeater_name> rcvbufsz=size
  • RCVBUFSZ and SNDBUFSZ parameter on receiver
    • Defined in mcast_receiver.cfg file and the default value is 98304 bytes
    • Increasing RCVBUFSZ may result in fewer dropped packets and therefore a reduced distribution time
  • If you see the following message in receiver side log file, it would be better to increase buffer size. This message indicates that receiver dropped packets because of buffer flow
  • Multicast buffer is related to setting of UDP socket buffer on operating system
    • Different operating systems have different limits for the buffer size
    • Multicast buffer size cannot be larger than operating systems’ limit
  • Multicast buffer tuning is optional and should be done after multicast data distribution is stabilized

Jun 30 15:07:57 3 RMTPERR [RMTPtblWriteUserData_R] buffer size is too small [421]

multicast performance evaluation
Multicast Performance Evaluation
  • When evaluating Tivoli multicast performance, probably the following are the typical scenarios
    • Before and after performance tuning
    • Multicast and unicast
    • Tivoli multicast and the other product’s multicast
  • Multicast data transfer rate may not seem high. However, assuming there are numerous distribution targets, you should consider how long the same data distribution takes via unicast
  • As long as you have numerous targets, multicast should be able to show you better throughput than unicast, especially when a large data is distributed
conclusion
Conclusion
  • Multicast very much depends on the environment
  • Start with default values and configuration
  • Perform multicast tuning and testing in the production environment
  • Understand the characteristics of the network, distributed data, and targets
slide51

Questions ???

Name: Murtuza Choilawala– Tivoli Global Response Team

Email : murtuza@us.ibm.com