1940 East 6th Street 11th Floor Cleveland Ohio 44114 Tel : 216.589.9626 Fax : 216.589.9639 Email: infocampuse

1. 1940 East 6th Street � 11th Floor � Cleveland � Ohio 44114 �Tel : 216.589.9626 � Fax : 216.589.9639 � Email: info@campuseai.org

2. 1940 East 6th Street � 11th Floor � Cleveland � Ohio 44114 �Tel : 216.589.9626 � Fax : 216.589.9639 � Email: info@campuseai.org Topic Outline Prerequisites and Audience Introduction What is Apache Jmeter: Basics of where and how? What are key functions available? Building Tests Monitoring and Measurements Methods and Styles of building tests Investigation and Troubleshooting Reporting Future Road Map - issues GOAL: Advanced Understanding, Lessons Learned, and Best Practices

3. 1940 East 6th Street � 11th Floor � Cleveland � Ohio 44114 �Tel : 216.589.9626 � Fax : 216.589.9639 � Email: info@campuseai.org Audience and Pre-requisites to this subject Fairly strong technical personnel System Administrators System Analysts DBAs Developers Knowledge of Apache/ Web Server technologies Understanding of HTTP (HyperText Transfer Protocol), URLs, etc Understanding of basic networks concepts and terms People to explain all major system components Firewalls, Routers, Networks, Web servers, Applications, Database Servers, SANS Assumes basic knowledge of load testing, stress testing, and tools Based on JMETER, but applicable to all tools (Mercury Web/Load Runner, WAS, etc) Review of existing training materials from CampusEAI Download jmeter or have access to using stress tool.

4. 1940 East 6th Street � 11th Floor � Cleveland � Ohio 44114 �Tel : 216.589.9626 � Fax : 216.589.9639 � Email: info@campuseai.org Introduction of the issue and purpose of stress testing Why do we stress test? When should we stress test? What is difference between stress and load testing? Limitations of JMETER vs other approaches. Define objectives of testing and purpose. Types of Stress / Load tests, practices, purposes

5. 1940 East 6th Street � 11th Floor � Cleveland � Ohio 44114 �Tel : 216.589.9626 � Fax : 216.589.9639 � Email: info@campuseai.org Purpose and Objective Of Testing Define objectives of testing and purpose. Can we have 1,000 people login to server per hour? What about testing per day? Unique user identities, How do we handle users and passwords? Why is my server running slowly? Find or investigate slow processes, pages, or situations Login process, email page, some specific situation, or generally slow system Preparing for production go-live? Before going live testing and creation of standards (level of standards) Base line of testing performance, start early and understand the �deltas� or changes Testing existing or new deployment functionality for breaks or problems Software development Testing of portlets or components Targeted stress testing, functional, load, and investigative. More than just throwing 1,000 users on in 30 minutes (which has limited benefit.) Stress testing is more expansive, exhaustive, and detailed oriented

6. 1940 East 6th Street � 11th Floor � Cleveland � Ohio 44114 �Tel : 216.589.9626 � Fax : 216.589.9639 � Email: info@campuseai.org What is Apache JMETER? Apache JMeter is a 100% pure Java desktop application designed to load test functional behavior and measure performance. It was originally designed for testing Web Applications but has since expanded to other test functions. Jmeter requires you to know how to �configure� your page samples, which has to be done manually, not a task for the faint of heart. Therefore, obtain a copy of �bad boy� which will record and save your clicks and keystrokes so that it can be �played� back as Jmeter scripts. Jmeter is fully documented. Jmeter can be used to simulate a heavy load on a server, network or object to test its strength or to analyze overall performance under different load types. Jmeter has a graphical analysis of performance or to test your server/script/object behavior under heavy concurrent load. C:\jakarta-jmeter-2.2\docs There are different versions and they may cause problems from older versions to new. Where to download from: http://jakarta.apache.org/jmeter/

7. 1940 East 6th Street � 11th Floor � Cleveland � Ohio 44114 �Tel : 216.589.9626 � Fax : 216.589.9639 � Email: info@campuseai.org What tools are available? CampusEAI Knowledge Base 3 Trainings at uCleveland Introduction to jmeter scripts Ceai jmeter script for testing Using Bad Boy script to record key strokes Newly Created Jmeter-tool-2007v1.PPT This document gives the specifics of the jmeter tool elements or functions. Training from Jmeter site and download C:\jakarta-jmeter-2.2\docs C:\jakarta-jmeter-2.2\docs\demos **** whole list 14 - AuthManagerTestPlan.jmx plans

8. 1940 East 6th Street � 11th Floor � Cleveland � Ohio 44114 �Tel : 216.589.9626 � Fax : 216.589.9639 � Email: info@campuseai.org Understand Your Architecture and system components Understand your Architecture Layout Hardware Configuration Sizing Components Print diagram out Name components Purpose of Testing Which component What portlet? Which user profile? Log files Enterprise Manager

9. 1940 East 6th Street � 11th Floor � Cleveland � Ohio 44114 �Tel : 216.589.9626 � Fax : 216.589.9639 � Email: info@campuseai.org What are the components and where are their log files? Portal Components J2EE = OC4J_Portal for PPE (dedicated � note: do not deploy portlets here ) Originally Database (local) vs. Web (remote) Providers Apache is OHS (aka: Oracle HTTP Server) Web Cache OC4J_Provider (separate unique j2ee for portlets) Metadata database (ASDB) OC4J_Provider Setup separate J2EE installed instance dedicated to running Portlets Separate from default OC4J_Portal j2ee instance Some schools setup another farm instance Developers can access and manage farm EAR file deployment segmented Separate login and administration for EM Requires JPDK libraries deployed

10. 1940 East 6th Street � 11th Floor � Cleveland � Ohio 44114 �Tel : 216.589.9626 � Fax : 216.589.9639 � Email: info@campuseai.org Login Process, Portal Pages, and Caching Login process on any system is most costly Target Login, password changes, and logoff in your testing plans. Login as a costly event, fortunately only occurs once per user session Unfortunately, in higher education portals, Login is done many times per day Profile of a �typical� college portal servers One school has 4,000 students and 12,000 logins per day! Students login at breakfast, after 10:30 class, lunch, and before or after dinner. Weekends especially Saturday night no load or access� (must be sleeping early for church on Sunday) Profile your schools loads and user access (use projections or guess -imates if necessary) Portal pages are looped. Login and log off should only once per �typical user�. Define and analyze your user or test situations profile of usage.

11. 1940 East 6th Street � 11th Floor � Cleveland � Ohio 44114 �Tel : 216.589.9626 � Fax : 216.589.9639 � Email: info@campuseai.org Startup and memory settings C:\jmeter-2\bin> java -Xms512m -Xmx1024m -XX:NewSize=128m -XX:MaxNewSize=128m -XX:SurvivorRatio=8 -XX:TargetSurvivorRatio=50 -XX:MaxTenuringThreshold=2 -XX:MaxLiveObjectEvacuationRatio=20 -Dsun.rmi.dgc.client.gcInterval=600000 -Dsun.rmi.dgc.server.gcInterval=600000 -XX:PermSize=64m -XX:MaxPermSize=64m -verbose:gc -XX:+PrintTenuringDistribution -jar ApacheJMeter.jar -n -t wsrp-testonly.jmx Large tests need more increase �HEAP MEMORY� = -XX:MaxNewSize=512m

12. 1940 East 6th Street � 11th Floor � Cleveland � Ohio 44114 �Tel : 216.589.9626 � Fax : 216.589.9639 � Email: info@campuseai.org Threads/ Loops/ User Variables Threads simulate the # of browsers Ramp effects avoids overloading Loop Counter Loops Logins are typically 1 loop Stress is created by looping specific pages User Defined Variables Use the ${customName} Variables can be static or dynamic See �extractors� to dynamically page drive values � these values can be used parameters.

13. 1940 East 6th Street � 11th Floor � Cleveland � Ohio 44114 �Tel : 216.589.9626 � Fax : 216.589.9639 � Email: info@campuseai.org Extraction & Header Request Objects HTTP Request - User �Agent header field User-Agent request-header field : Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; InfoPath.1; .NET CLR 2.0.50727) Extractor: Regular Expression & XPATH (newer) Regular Expression Extractor Site2ptoken = VALUE="(.+?)� my /portal/pls/portal/PORTAL.wwptl_login.show_site2pstoretoken A bit complicated.

14. 1940 East 6th Street � 11th Floor � Cleveland � Ohio 44114 �Tel : 216.589.9626 � Fax : 216.589.9639 � Email: info@campuseai.org Assertion � Finding Errors �inside� the page. Assertions � used to find errors �inside� the content or response of page. Assert or verify that a page response contains or doesn�t contain certain things. Why: Cause we could get a HTTP Response 200 � but have page error or portlet error. Various types of Assertions. Response Timings (if page takes more than 7,500 milli-seconds) then consider this a failure Size Page less or greater than a certain amount of bytes (page should be so big) assert on that. XPath Assertion xml parser, namespaces, search and find

15. 1940 East 6th Street � 11th Floor � Cleveland � Ohio 44114 �Tel : 216.589.9626 � Fax : 216.589.9639 � Email: info@campuseai.org FAQ: Multiple JVMs running from Desktop We long term stress/load testing from a desktop computer, try to run multiple JVMs. (hence you'll have to double click on the jmeterm.bat file multiple times.) Further you may need to enhance the -Xmx setting for more virtual memory in the JVM for jmeter, or disable the "listeners" like View result tree (pig for memory) and other reporting elements. What do you have threads set to? i usually try to keep them below 25 threads for GUI desktop activity. (per jvm) From a personal laptop, it�s typical for about 10,000 logins per hour. Actually achieved nearly 21,000 logins per hour from my laptop on our LAN network. many factors influence this number

16. 1940 East 6th Street � 11th Floor � Cleveland � Ohio 44114 �Tel : 216.589.9626 � Fax : 216.589.9639 � Email: info@campuseai.org Simulating User Think Time and timers How does think time work? What is your timer set to? (Simulate the users think time in milli-secs 1/1000 of a second) usually 750 ms (less than a 1 second) use the Gaussian Random timer. Set constant to 750 and deviation to 300. This is base plus random, divide the random portion by 2 and add this to the constant amount for the average �think time�. If I have 10 threads, I'll set rampup to 31secs (basically about 3+ seconds per number of threads) some people dont do this... it's not really required with a low number thread count like 1-5. As you go towards 10 or higher, I would put some number here like my example of 3 secs per thread. If you start 3 Jmeter JVMs on your laptop (assuming you have decent lap/desktop) and you have threads set to 20 then thats 20*3=60 logins per loop. If you have 10 portal pages clicks per loop. Assuming page responses take about 1 sec or less, (portal pages even with ugly portlets should not take more than 1 or 2 secs) unless under severe loading conditions. (logins (post to /sso/auth will take 2-5 secs, plus +5 secs under loading conditions) continue with math. 60 logins per 3 jmeter loads loop. 12 secs per loop (1 sec per page + 2 sec login) 5 loops per minute. 60 minutes in hour, 60 * 5 * 60 = 18,000 logins per hour. 1 sec think is really like normal users 10 secs that�s 10x= 180,000 portal page views per hour. you'll have some page failures, but it should not be consistent (random pages) and should be less than 0.1 percent. (investigate anything greater a few)

17. 1940 East 6th Street � 11th Floor � Cleveland � Ohio 44114 �Tel : 216.589.9626 � Fax : 216.589.9639 � Email: info@campuseai.org Types of Testing Development Testing Used for testing developed code or sub-systems. The objective of this testing is evaluate errors, �bugs�, bad programming, or unknown situations that the developer, system administrator did not foresee. Troubleshooting Problems or bugs that can be investigated using stress testing. Hidden or �random� problems can be exposed and investigated. (very difficult and time consuming, but when these situations occur, people are desperate, this process is necessary to help �find� the problem. Production server ready go-live Standardized test that measures base line issues. Get the loading test base lines.

18. 1940 East 6th Street � 11th Floor � Cleveland � Ohio 44114 �Tel : 216.589.9626 � Fax : 216.589.9639 � Email: info@campuseai.org Portlet Stress Testing Turn off page caching on each pages for testing the backend portlets. Stress focuses more then load test. Look at function and how components behave. Put a single portlet on single page, test that page as solo- see the results Put portlets together as a menagerie � see how they behave. Log in a large user set Difference between 6 users login 1000 and 6,000 unique usernames. Start with 1 user (Thread) � test and eliminate all errors. Setup

19. 1940 East 6th Street � 11th Floor � Cleveland � Ohio 44114 �Tel : 216.589.9626 � Fax : 216.589.9639 � Email: info@campuseai.org Identify Targets Identify Target Page Name Identify Target Page ID (10.1.4 is human readable URLs) Testing size, number of loops, type and amount of stress Loops and Threads Assertions: finding good words that persistent or found on every version of page Negative assertions: look for occurrence of error or exception term

20. 1940 East 6th Street � 11th Floor � Cleveland � Ohio 44114 �Tel : 216.589.9626 � Fax : 216.589.9639 � Email: info@campuseai.org Large User Counts � Users and Passwords User1 � user500 test script. Perl script to generate ldif file of sample users ldapadd -D cn=orcladmin -w 'welcome1' -c -v -f users.ldif create.pl � edit your institution�s RND Loop for next 1 to 2000; open (OUT, ">users.ldif� print OUT qq~ dn: cn=$user,cn=users,dc=campuseai,dc=org cn: $user sn: $user uid: $user mail: fjohnson@campuseai.org objectclass: orcluserv2 objectclass: orcluser objectclass: organizationalperson objectclass: inetorgperson objectclass: person objectclass: top userpassword: welcome1

21. 1940 East 6th Street � 11th Floor � Cleveland � Ohio 44114 �Tel : 216.589.9626 � Fax : 216.589.9639 � Email: info@campuseai.org Methods to increment users Small number of users for exact testing Setup each users name and password Good for about 5-10, max 20 users painful after that cause its manual entry There are several methods to simulate users. script create user1..user6000 Put username text with the counter element value other mechanisms: Xml source files Authorization Manager

22. 1940 East 6th Street � 11th Floor � Cleveland � Ohio 44114 �Tel : 216.589.9626 � Fax : 216.589.9639 � Email: info@campuseai.org Oracle Login Page Sample Request Basicially the same but different between versions 9.0.4, 10.1.2, and 10.1.2.0.2 (no news on new infra 10.1.4.1) 9.0.4 has either the /pls/orasso login or the servlet /sso/auth to login The pls means pl-sql module interface, strictly type set of parameters The /sso/auth is the OC4J_SECURITY module 10.1.2 higher the /pls/orasso redirects to /sso/auth servlet anyway. ${site2ptoken} variable = /portal/pls/portal/PORTAL.wwptl_login.show_site2pstoretoken

23. 1940 East 6th Street � 11th Floor � Cleveland � Ohio 44114 �Tel : 216.589.9626 � Fax : 216.589.9639 � Email: info@campuseai.org Measurement and Monitoring System and Resource Monitoring Memory RAM, SAR command, top command, vmstat command Unix cron is good system for recording over time. vmstat 60 60 >testResults1hr.log

24. 1940 East 6th Street � 11th Floor � Cleveland � Ohio 44114 �Tel : 216.589.9626 � Fax : 216.589.9639 � Email: info@campuseai.org PLSQL to access key information SQL INTO Key Oracle Tables SELECT COUNT(*), TO_CHAR(log_date, 'HH24') FROM orasso.WWSSO_AUDIT_LOG_TABLE$ WHERE log_date LIKE TO_DATE('11/11/2005','MM/DD/YYYY') AND message = 'Login Successful' GROUP BY TO_CHAR(log_date, 'HH24'); Create web based display on SQL select into orasso.WWSSO_AUDIT_LOG_TABLE$

25. 1940 East 6th Street � 11th Floor � Cleveland � Ohio 44114 �Tel : 216.589.9626 � Fax : 216.589.9639 � Email: info@campuseai.org Enterprise Manager # of Logins per Hour and per Day Oracle Enterprise manager has several reporting screens This screen shows login attempts.

26. 1940 East 6th Street � 11th Floor � Cleveland � Ohio 44114 �Tel : 216.589.9626 � Fax : 216.589.9639 � Email: info@campuseai.org Look at the network loading Look at the network loading Evaluate key interfaces Where likely bandwidth bottlenecks.

27. 1940 East 6th Street � 11th Floor � Cleveland � Ohio 44114 �Tel : 216.589.9626 � Fax : 216.589.9639 � Email: info@campuseai.org Things Jmeter can do that we never explored - RoadMap Automatic Site traversal or Spidering LDAP stress/load testing Database stress/load testing Extensible java API for additional functionality Javascript testing?? � newest issue Rhino project (Netscape last coolness java implementation of javascript) Run javascript from java command line Simulate javascript browser functions Test new advanced features automatically

28. 1940 East 6th Street � 11th Floor � Cleveland � Ohio 44114 �Tel : 216.589.9626 � Fax : 216.589.9639 � Email: info@campuseai.org Understanding JMeter client-side results Main report is the aggregate report, it shows summary data of the pages tested. Keep your URLs (names of the HTTP Request Sample�s name) � readable in shorter space Typically, use hostname and shorten version of path (my /pagegroup/pagename) Max and 90% column are important fields, as the error field. Errors should be investigated using the tree sample

29. 1940 East 6th Street � 11th Floor � Cleveland � Ohio 44114 �Tel : 216.589.9626 � Fax : 216.589.9639 � Email: info@campuseai.org Tree Sampler Click the switch on the left of the sample list panel � open up the sub-tabs Red items indicate failures � click on Requests look at details Review the response area for errors or messages Sample results only show basic http errors like 404 or 500

30. 1940 East 6th Street � 11th Floor � Cleveland � Ohio 44114 �Tel : 216.589.9626 � Fax : 216.589.9639 � Email: info@campuseai.org SSL Keystore popup When doing SSL enabled sites, the security algorithm needs a random number Only needs it ONCE first time in. If you set ramp up high enough you can type gibberish random �seed� value in once. Otherwise, each time jmeter (HttpClient libraries) attempts a new page and jmeter asks for this. 1 trick (start with only 1 thread then switch to more) 2nd trick ramp time 3rd trick type quickly

31. 1940 East 6th Street � 11th Floor � Cleveland � Ohio 44114 �Tel : 216.589.9626 � Fax : 216.589.9639 � Email: info@campuseai.org CampusEAI Consortium Overview Stress Testing using JMETER CampusEAI Consortium (CEAI) Non-Profit Corporation Established in 2003 by 14 research universities in the United States including: More than 200 Colleges and University Members in 9 Countries including: Shared Services and Technology Transfer, Incubation & Commercialization = More than 150 Full-time Staff Offices in Cleveland, Ohio and New Delhi, India Data Center(s)