LG-EDS System Consulting

1. LG-EDS SystemConsulting Oct 20-21 Sung-Ik Son

2. Toplology wds1 RS/6000 Trade Server INFORMIX DB 7.24 UC6/ JDBC Driver 1.5 128.2.104.21 RS/6000 Trade Server eND/ISS (eb1) wds2 cluster 128.2.104.21 128.2.104.21 128.2.104.21 128.2.104.21 RS/6000 Stock Server 128.2.250.245 wds3 128.2.104.20 eND/ISS (eb2) RS/6000 Trade Server 128.2.104.32 128.2.250.246 wds4 Load Balancer and ISS Monitor SUN SC250 128.2.104.21 Load balanced server and ISS agent

3. Installation Checkpoint • If you want to continue with WAS 2.0.x, definitely upgrade to 2.0.3.1. It's available on the WebSphere intranet web site. http://w3.software.ibm.com/webservers/html/downloads.html • It is a couple of fixes (one in particular for Japan) plus we were able to put the Advanced Edition (timebombed) out there, so we dropped the Patch install. We just wanted to get the complete install of Advanced on the web, so people could pick up IHS 1.3.6 plug-in and NES 3.6.2 plug-in.

4. If IBM HTTPServer 1.3.3 is upgraded to 1.3.6, then the corresponding plugin file under <IBMWebAS_root>/plugins/<platform> needs to be renamed, before installing the Service Pack. On UNIX, RENAME mod_ibm_app_server.so TO mod_ibm_app_server_136.so Also, update the IBM HTTPServer configuration file, httpd.conf, under the configuration directory, to reflect the new plugin name in the following directive: LoadModule ibm_app_server_module

5. On UNIX, ensure that the following environment variablesare set (type the entire thing on a single line): <IBMWebAS_root>/plugins/<platform>:/usr/lib export LD_LIBRARY_PATH=<JDK directory>/lib/<platform>/native_threads: export LIBPATH=$LD_LIBRARY_PATH

6. AIX Performance Tuning • the NetScape default configuration is single-process, multi-threaded. Under default configurations, Netscape on AIX have not produced significantly different results. • In-process may provide some small performance improvements on some configurations, but the drawbacks typically outweigh any small performance gains. We recommend keeping the default out-of-process setting and focusing on performance improvements elsewhere. If you do choose in-process, please be aware that we have seen performance problems running servlets in process on 2-way and 4-way NT configurations. ose.mode = out (default)

7. The number of active threads within the Netscape Enterprise server is controlled by the “Maximum number of simultaneous requests” parameter set in the Enterprise Server Manager. This setting corresponds to the RqThrottle parameter in magnus.conf and defaults to 128. Once the server reaches this limit, it will stop servicing new connections until it has finished with the old connections. If this setting is too low your server may be throttled and response time may degrade. • The best way to tell if your Web Server is being throttled is to use perfdump stats and look at the WaitingThreads count. If this number is getting close to 0 or is 0, then the server is not accepting new connections right now. At the same time, this probably means that the number of BusyThreads is very close to the limit. Also check to see if the number of ActiveThreads is close to the limit. If so, the server is probably limiting itself. • The min thread limit is a goal for how many threads the server attempts to keep in the WaitingThreads state. This number is just a goal; the number of actual threads in this state may go slightly above or below this value. This parameter is RqThrottleMinPerSocket in magnus.conf.

8. Once you have selected out of process, you have three choices, local, remote, or remote_java. In all AIX cases examined to date, “local” performs faster, in many cases significantly. ose.outofproc.transport.type=local (default) • The next tuning points are the number of threads in use within WebSphere itself. The link.cache.size parameter is the communication channel between the web server and the WebSphere engine, and the thread pool is the number of threads within the WebSphere engine. The thread pool size is a new parameter in WebSphere 2.02 and was not documented. Typically, the cache.size should be larger than the threadpool.size. Ose.outofproc.link.cache.size - 125 (default is 50) Ose.outofproc.threadpool.size - 90 (4-way AIX), 105 (12-way AIX), 75 (4-way Solaris)

9. the latest 1.1.n level will provide better performance as JVM technology is continually improving. • The JVM offers several tuning parameters that will impact the performance of WebSphere (which is primarily a java application), as well as your own application. The JIT and garbage collection both have significant impacts on performance. In all cases, you want to run with JIT on and asynch garbage collection on. You can turn off class garbage collection to enable more class reuse, and prevent the JIT from being garbage collected. The JVM parameters to are also set through the WebSphere properties: Java.compiler= (leave this blank for the default setting of JIT on). Java.asyncgc =true (default) Java.classgc = false (false will enable more class reuse; default is true)

10. Java mx and ms are used to set the maximum heap, and starting heap for the JVM. For the best performance runs, set these values to be the same. This will prevent the heap from dynamically being massaged during performance runs. The default, 128 meg, is a good starting value for smaller (4-way and under) machines. 256 meg is a good initial setting for machine with more than four processors. Make sure to convert from meg to bytes. Java.mx = 134217728 (4-way or lower; default; 268435456 for bigger machines) Java.ms = 134217728 (same as java.mx; default is 16M) • Java oss and ss set java and native thread size stacks within the JVM. Both of these parameters should be at least 819200 or bigger. Java.oss = 819200 (8M; default is 4M) Java.ss = 819200 (8 M; default is 1M)

11. Too much heap dramatically impacts garbage collection times • Too little heap chokes the application

12. Files descriptors are a limited resource in Unix Pre-AIX 4.3.1 = 2,000 File Handles available AIX 4.3.1 = 32,000 File Handles available • WAS defines a pool of available threads • Default: 50 concurrent threads • Modify threads in the bootstrap.properties file • ose.outofproc.threadpool.size=50 • More threads does not necessarily increase performance • Excessive threads increases swapping overhead • Set other system pools to appropriate sizes • HTTP thread pool sufficient for static and dynamic traffic • DB connection pool large enough to handle dynamic requests

13. What is connection manager? The connection manager lets you control and reduce the resources used by your Web-based applications. Web-based applications accessing data servers incur higher and less predictable overhead than non-Web applications, because Web users connect and disconnect more frequently. Often, more resources are spent connecting and disconnecting than are spent during the interactions themselves. Users interactions are typically shorter, due to the "surfing" nature of the Internet. Users are often from outside the company (Internet, rather than intranet), which make usage volumes larger and more difficult to predict.

14. The connection manager spreads the connection overhead across several user requests by establishing a pool of connections, which user servlets can use. Put another way, each user request incurs the overhead of only a fraction of the cost of a connect/disconnect. After the initial resources are spent to produce the connections in the pool, additional connect/disconnect overhead is insignificant because the existing connections are reused repeatedly. • The servlets use the connection pool as follows: When a user makes a request over the Web to a servlet, the servlet uses an existing connection from the pool, meaning the user request does not incur the overhead of a data server connect. When the request is satisfied, the servlet returns the connection to the connection manager pool for use by other servlets. The user request therefore does not incur the overhead of a data server disconnect. • The connection manager also lets you control the number of concurrent connections to a data server product. This is very useful if the data server license agreement limits you to a certain number of users. You could create a pool for the data server and set the Maximum Connections parameter for the connection manager pool equal to the maximum number of users permitted by your data server product license. This approach does not guarantee compliance if you connect to the data server with other programs that bypass the connection manager.

15. The connection manager maintains a pool of open data server connections to specific data server products. Each data server can have one or more identical or non-identical pools. Multiple data servers can be supported by one running instance of the connection manager. The figure illustrates the typical interactions between the connection manager and a servlet seeking to use a connection from the connection manager's connection pool. The following list traces the sequence followed when a connection is available when requested. As you will see later, the sequence changes if a connection is not available when the servlet requests one.

16. The connection manager, which runs under the Application Server, is loaded by the Application Server when the first servlet tries to communicate with the connection manager. The connection manager stays loaded as long as the Application Server is running. • The Application Server passes a user request to a servlet. • The servlet uses methods of the connection manager to request a connection from the pool. • The pool gives the servlet a connection. • The servlet uses the connection to talk directly to the data server, using the standard APIs for the specific data server. • The data server returns data through the connection to the servlet. • When the servlet ends communications with the data server, the servlet returns the connection to the pool for use by another user request. • The servlet sends the response back through the Application Server to the user.

17. A connection is not always available from the pool when the servlet requests it. In such cases, the connection manager communicates directly with the data server. The connection manager: • Requests a new connection (point 9 in the figure) • Adds the connection to the pool (point 10) • The connection manager will not add a new connection to the pool if the specified maximum number of connections in the pool has been reached. • Gives the new connection to the servlet (point 4)

18. Performance features • Creating a new connection for the pool is a high overhead task and the new connection will use resources on the data server machine. Therefore, the connection manager seeks to maximize the probability that a servlet can get an existing connection from the pool. At the same time, the connection manager must minimize the idle connections in the pool, because they are a significant waste of resources. • The connection manager works with the servlet to perform these minimizing and maximizing tasks. For optimal performance, choose the proper settings for the connection manager parameters. • The connection manager maintains the verify timestamp, last-used timestamp, and in-use flag of each connection. When a servlet first gets a connection, the connection's verify timestamp and last-used timestamp are set to the current time and the connection's in-use flag is set to true.

19. The connection manager can be configured to take a connection away from a servlet that has not used the connection for a length of time specified by the connection manager Maximum Age parameter. If the servlet is going to use the connection to communicate with the data server multiple times over an extended period, you may want to add code to the servlet to verify that it still owns the connection just before each use of the connection. • The connection manager removes idle connections from the pool, because they waste resources. To decide which connections are idle, the connection manager checks the connection flags and timestamps in a periodic reap of the connection pool:

20. The connection manager looks at the last-used timestamp of the in-use connections. If the time between the last-used and current time exceeds the Maximum Age configuration parameter, the connection is assumed to be an orphan connection, meaning the owning servlet has died or is otherwise unresponsive. The orphan connection is returned to the pool for use by another servlet, its in-use flag is set to false and its verify and last-used timestamps are set to the current time. • The connection manager examines connections not in use by any servlet - those connections whose in-use flags are false. If the time between the last-used and current time exceeds the Maximum Idle Time configuration parameter, the connection is assumed idle. Idle connections are removed from the pool, down to the lower limit specified by Minimum Connections configuration parameter.

21. IBMConnMgrTest.java sample • // ******************************************************************* • // * IBMConnMgrTest.java - test the connection manager * • // ******************************************************************* • import java.io.*; • import javax.servlet.*; • import javax.servlet.http.*; • import java.sql.*; • import java.util.*; • import com.ibm.servlet.connmgr.*; • public class IBMConnMgrTest extends HttpServlet • { • // **************************************************************** • // * Variables * • // **************************************************************** • // Use to communicate with connection manager. • static IBMConnMgr connMgr = null; • // Use later in init() to create JDBC connection specification. • static IBMConnSpec spec = null; // the spec • static String DbName = null; // database name • static String Db = "db2"; // JDBC subprotocol for DB2 • static String poolName = "JdbcDb2"; // from Webmaster • static String jdbcDriver = "COM.ibm.db2.jdbc.app.DB2Driver"; • static String url = null; // constructed later • static String user = null; // user and password could • static String password = null; // come from HTML form • static String owner = null; // table owner • // Name of property file used to complete the user, password, • // DbName, and table owner information at runtime. ".properties" • // extension assumed. • static final String CONFIG_BUNDLE_NAME = "login";

22. // **************************************************************** // * Initialize servlet when it is first loaded * // **************************************************************** public void init(ServletConfig config) throws ServletException { super.init(config); try { // Get information at runtime (from an external property file // identified by CONFIG_BUNDLE_NAME) about the database name // and the associated database user and password. This // information could be provided in other ways. It could be // hardcoded within this application, for example. PropertyResourceBundle configBundle = (PropertyResourceBundle)PropertyResourceBundle. getBundle(CONFIG_BUNDLE_NAME); DbName = configBundle.getString("JDBCServlet.dbName"); url = "jdbc:" + Db + ":" + DbName; user = configBundle.getString("JDBCServlet.dbUserid"); password = configBundle.getString("JDBCServlet.dbPassword"); owner = configBundle.getString("JDBCServlet.dbOwner"); } catch(Exception e) { System.out.println("read properties file: " + e.getMessage()); }

23. try { // ********** // * STEP 1 * // ********** // Create JDBC connection specification. spec = new IBMJdbcConnSpec (poolName, // pool name from Webmaster “JdbcInformix” true, // waitRetry true jdbcDriver, // Remaining four “com.informix.jdbc.ifxDriver” url, // parameters are “jdbc:informix:sqli://128.2.1.100:9998/…” user, // specific for a “informix” password); // JDBC connection. “informix” // ********** // * STEP 2 * // ********** // Get a reference to the connection manager. Connect to the Connection Manager connMgr = IBMConnMgrUtil.getIBMConnMgr(); } catch(Exception e) { System.out.println("set connection spec, get connection manager: " + e.getMessage()); } }

24. // **************************************************************** // * Respond to user GET request * // **************************************************************** public void doGet(HttpServletRequest req, HttpServletResponse res) { IBMJdbcConn cmConn = null; Connection dataConn = null; Vector firstNameList = new Vector(); try { // ********** // * STEP 3 * // ********** // Get an IBMJdbcConn object (cmConn) meeting JDBC // connection specs, from the connection manager pool. cmConn = (IBMJdbcConn)connMgr.getIBMConnection(spec); // ********** // * STEP 4 * // ********** // Get a Connection object (dataConn). This is // an object from the java.sql package and it is used // for JDBC access. dataConn = cmConn.getJdbcConnection(); dataConn.setAutoCommit(false); // will be discussed later

25. // ********** // * STEP 5 * // ********** // Run DB query - create a Vector of first names of all people // whose last name is 'PARKER'. // Standard JDBC coding follows. Change the query for your // specific situation. Statement stmt = dataConn.createStatement(); String query = "Select FirstNme " + "from " + owner + ".Employee " + "where LASTNAME = 'PARKER'"; ResultSet rs = stmt.executeQuery(query); while(rs.next()) { firstNameList.addElement(rs.getString(1)); } // Invoke close() on stmt, which also closes rs, freeing // resources and completing the interaction. You must // not, however, close the dataConn object. It must // remain open and under the control of connection manager // for possible use by other requests to this servlet or // to other servlets. rs.close(); stmt.close(); // If you now want to use the dataConn object again for // another interaction, you may want to make sure that you // still own the associated cmConn object. Use the // verifyIBMConnection() method to do this, which also // updates a last-used timestamp. If the // verifyIBMConnection() method fails, you will need // to get new instances of the cmConn and dataConn objects // before you can use them. The actual need to use // the verifyIBMConnection() method depends on the // Maximum Age parameter set for the connection manager // and the length of time that you anticipate your // servlet will need to use the cmConn and dataConn // objects. } catch(Exception e) { System.out.println("get connection, process statement: " + e.getMessage()); } Do not use prepareStatement() for dynamic SQL

26. // ********** // * STEP 6 * // ********** // Release the connection back to the pool. finally { if(cmConn != null) { try { cmConn.releaseIBMConnection(); } catch(IBMConnMgrException e) { System.out.println("release connection: " + e.getMessage()); } } }

27. // ********** // * STEP 7 * // ********** // Prepare and return HTML response - say Hello to everyone // whose last name is 'Parker' and address them with their first // and last name. res.setContentType("text/html"); // Next three lines prevent dynamic content from being cached // on browsers. res.setHeader("Pragma", "no-cache"); res.setHeader("Cache-Control", "no-cache"); res.setDateHeader("Expires", 0); try { ServletOutputStream out = res.getOutputStream(); out.println("<HTML>"); out.println("<HEAD><TITLE>Hello DBWorld</TITLE><HEAD>"); out.println("<BODY>"); if(firstNameList.isEmpty()) { out.println("<H1>Nobody named Parker</H1>"); } else { for(int i = 0; i < firstNameList.size(); i++) { out.println("<H1>Hello " + firstNameList.elementAt(i) + " Parker</H1>"); } } out.println("</BODY></HTML>"); out.close(); } catch(IOException e) { System.out.println("HTML response: " + e.getMessage()); } } }

28. Programming Suggestion • Avoid expensive string manipulation • shortens Garbage Collection Cycle • Reduce CPU burden • String1 = “string2”+myObhect.getName()+”String3”; • Avoid dynamic String Allocation • String a = “Hi”+name; • Make Static any static elements • Avoid expensive Java classes and methods (ex: DateFormat) • Avoid PrintWriter • Expensive NLS support overhead • Required for engagements using NLS/DBCS • Solution: Use stream

29. Servlet Performance Tuning Suggestion The JVM running the servlets is expected to simultaneously handle dozens, if not hundreds, of threads, each executing a servlet. These coexisting servlets have to share the resources of the JVM in a way that normal applications do not. • Avoid the unnecessary creation of objects • wastes memory • wastes a fair amount of time • Most JVMs have a global object heap that must be locked for each new memory allocation. While any servlet is creating a new object or allocation additional memory, no other servlet can do so.

30. Servlet Performance Tuning Suggestion • Don’t Append by Concatenation. Use the append() method of StringBuffer instead bad example) String output ; output += “<title>”; output +=“Hello, :+ user; output += “</title>”; correct way) StringBuffer buf = new StringBuffer(); buf.append(“<title>”); buf.append(“Hello, “).append(user); buf.append(“</title>”); output = buf.toString(); • Limit Synchronization

31. // **************************************************************** // * Initialize servlet when it is first loaded * // **************************************************************** public void init(ServletConfig config) throws ServletException { super.init(config); try { // Get information at runtime (from an external property file // identified by CONFIG_BUNDLE_NAME) about the database name // and the associated database user and password. This // information could be provided in other ways. It could be // hardcoded within this application, for example. PropertyResourceBundle configBundle = (PropertyResourceBundle)PropertyResourceBundle. getBundle(CONFIG_BUNDLE_NAME); DbName = configBundle.getString("JDBCServlet.dbName"); url = "jdbc:" + Db + ":" + DbName; user = configBundle.getString("JDBCServlet.dbUserid"); password = configBundle.getString("JDBCServlet.dbPassword"); owner = configBundle.getString("JDBCServlet.dbOwner"); } catch(Exception e) { System.out.println("read properties file: " + e.getMessage()); } Example

32. bad good

33. Servlet Performance Tuning Suggestion • Limit Synchronization • Servlets that implement SingleThreadModel don’t need to synchronize access to their instance variables • Servlets are multi-threaded • Each request receives its own instance of the servlet • Sharing variables among the instances forces serialization • Serialization creates bottlenecks • Enormous negative impact on application performance • Sharing variables also produces unpredictable results • Shared variable state is arbitrary • Better Practice: • Use Session State or a persistent store to maintain state

34. Servlet Serialization Multi-Threaded Model

35. Servlet Serialization Single-Threaded Model

36. Drawbacks of Each Model • Multi-threading impacts non-local variables • Race conditions • Serialization overhead • Single-threading requires more resources • Extra memory required for additional instances • Can reduce sharing of system resources • Database connections • Best Practice: Multi-threaded model • Use Session State and Pooling to share resources

37. JDBC Connection Sharing - The Wrong Way (Multi-threaded Model) public class MyJDBCServlet extends HttpServlet{ Connection con; public void init(ServletConfig config) throws ServletException { super.init(config); try { Class.forName("COM.ibm.db2.jdbc.app.DB2Driver"); con = DriverManager.getConnection("jdbc:db2:sample"); } catch (Exception e) { con = null; } } Connection is shared by all thread All threads share the same db connection Poor practice: Unpredictable results Serialized access - performance bottleneck

38. JDBC Connection Sharing - A Right Way (Single-threaded Model) public class MyJDBCServlet extends HttpServlet implements SingleThreadModel{ Connection con; public void init(ServletConfig config) throws ServletException { super.init(config); try { Class.forName("COM.ibm.db2.jdbc.app.DB2Driver"); con = DriverManager.getConnection("jdbc:db2:sample"); } catch (Exception e) { con = null; } } One instance of the class per thread Each thread has its own connection Drawbacks: Licensing: One connection per servlet instance Sharing: Connections held by inactive servlet instances Use Connection Manager for managed sharing

39. Using a JDBC Connection - The Right Way (All Models) public void init(ServletConfig config) throws ServletException { super.init(config); try { // Create JDBC connection specification. spec = new IBMJdbcConnSpec(...); // Get a reference to the connection manager. connMgr = IBMConnMgrUtil.getIBMConnMgr(); } catch(Exception e) {...} } Uses the WAS Connection Manager IBM and JDBC connections local to thread One connection per thread Pool size centrally managed Idle connections returned to pool public void doGet(HttpServletRequest req, HttpServletResponse res) { IBMJdbcConn cmConn = null; Connection dataConn = null; try { // Get an IBMJdbcConn object (cmConn) from the cm pool. cmConn = (IBMJdbcConn)connMgr.getIBMConnection(spec); // Get a Connection object (dataConn). dataConn = cmConn.getJdbcConnection(); .....

40. Maintaining State Information - The Wrong Way (Multi-threaded Model) Variable shared by all threads public class MyCountServlet extends HttpServlet { int count = 0; Hashtable myHashTable = new Hashtable(); public void service (HttpServletRequest req, HttpServletResponse res) throws ServletException, IOException { count++; ... myHashTable.putValue("firstValue", aValue); } } Variable shared by all threads and serialized Update is unpredictable Serialized access causes bottlenecks

41. Maintaining State Information - A Better Way public class MyCountServlet extends HttpServlet { public void service (HttpServletRequest req, HttpServletResponse res) throws ServletException, IOException { .... HttpSession session = req.getSession(true); if (session == null) hwcount= "MyCountServlet: Cannot create session as expected!"; else { Integer value = (Integer)session.getValue("counter"); if (value == null) { value = new Integer(1); myCount = "MyCountServlet: 1 (from new session)"; } else { value = new Integer(value.intValue() + 1); myCount = "My count: " + value + " (from existing session)"; } session.putValue("counter", value); .... State information held in Session State

42. Database Performance • Disable autocommit mode • Do not use the prepareStatement method for dynamic SQL • caused unnecessary traffic flow • Always close connection, result set, and statement

43. Disable autocommit Auto-commit mode indicates to the database whether to issue an execute and commit after every SQL statement. Being in auto-commit mode can be expensive in terms of time and processing effort if, for example, you are repeating the same statement with different bind variables. By default, new connection objects are in auto-commit mode. However, you can disable auto-commit mode with the setAutoCommit() method of the connection object (java.sql.Connection). In auto-commit mode, the commit occurs either when the statement completes or the next execute occurs, whichever comes first. In the case of statements returning a ResultSet, the statement completes when the last row of the ResultSet has been retrieved or when the ResultSet has been closed. In more complex cases, a single statement can return multiple results as well as output parameter values. Here, the commit occurs when all results and output parameter values have been retrieved. If you disable auto-commit mode (setAutoCommit(false)), then the JDBC driver groups the connection's SQL statements into transactions that it terminates by either a commit() or rollback() statement.

44. Example • // Load the Oracle JDBC driver • DriverManager.registerDriver(new oracle.jdbc.driver.OracleDriver()); • // Connect to the database • // You can put a database hostname after the @ sign in • // the connection URL. • Connection conn = • DriverManager.getConnection ("jdbc:oracle:oci8:@", • "scott", "tiger"); • // It's faster when auto commit is off • conn.setAutoCommit (false); • // Create a Statement • Statement stmt = conn.createStatement (); • ...

45. prepareStatement vs createStatement <Example> String qry=“SELECT * FROM EMPLOYEE”; PreparedStatement stmt = con.prepareStatement(qry); : Statement stmt= con.createStatement(); ResultSet rs = stmt.executeQuery(qry);

46. Memory Leak protection If you receive messages that you are running out of cursors or that you are running out of memory, make sure that all of your Statement and ResultSet objects are explicitly closed. If you do not explicitly close your result set and statement objects, serious memory leaks could occur. You could also run out of cursors in the database. Closing a result set or statement releases the corresponding cursor in the database. Similarly, you must explicitly close Connection objects to avoid leaks and running out of cursors on the server side. When you close the connection, the JDBC driver closes any open statement objects associated with it, thus releasing the cursor objects on the servers side.

47. Minimize size and complexity • Large servlets impact performance • Long code paths increase response time • Large memory footprint impacts WAS performance • Better Practice: Compose a web application from multiple servlets • Each application task assigned its own servlet • Each servlet returns a few basic page choices • Servlets share information via Session State/persistent store • Benefits: • Faster individual servlet performance • Improved site maintenance