Thursday, August 20, 2009

Performance Tuning for Apache Tomcat

At SpringOne 2009, Mark Thomas gave a talk about Tomcat configuration. Mark mentions that 80% of performance problems will be in the application rather than in tomcat. Having said that, Mark shares some tomcat optimizations:

1. Logging: Turn off logging to std and only log to a file. Add rotation by limiting file size and # of file.

2. Connector Tuning: depends on application, client(web of mobile), tcp, http keep alive, and sll. Keep alive uses one tcp connection for multiple http requests. Modern pages uses over 100 requests per page. Blocking IO connectors use a thread per connection. For sticky sessions, layer 4 load balancer uses client ip and port (not always available when isps proxy user requests). Layer 7 load balancer understands http headers and tomcat puts markers for sticky session in header.

a. BIO – Java blocking connector. Stable (use as default)

b. Native (APR) – non blocking, Open SSL (use when need SSL or high concurrency with keep alive). Not stable on Solaris so use NIO instead.

c. Java non blocking IO – JSSE based SSL

Tuning:

a. MaxThreads: maximum number of concurrent requests (in BIO, it is the max number of open connections). Set to 200 – 800. Start with 400 and increase if low CPU usage or decrease with high usage

b. MaxKeepAliveRequests: Use 1 (turn off) or 100. Used to clean up and close unused requests. Disable for BIO with high concurrency, layer 4 load balancer, no ssl. Enable for ssl, APR/NIO, layer 7 load balancer. Disable if using httpd and have it on same box as tomcat

c. ConnectionTimeOut. Typical value is 3000 (3 seconds). Default of 20000 (20s) is too high. Increase for slow clients or layer 7 load balancer with connection pool and keep alive.

3. Content Cache: caches static content. Configured with CacheMaxSize in KB (10240), CacheTLL in ms(5000), CacheMaxFileSize in KB (512).


4. JVM:

a. Xmx/-Xmx used to define size of java heap. Aim to set as low as possible to leave enough space for all other processes. Setting too high can also lead to very long GC cycles. If app needs 200M set it for 250 or 300M.

b. –XX:NewSize/-XX:NewRatio set to 25%-33% of total java heap size. No need to change, but if you need to go into these details, set NewRatio over NewSize.

c. -XX:MaxGCPauseMillis –XX:MaxGCMinorPauseMillis. Goal to end up with more frequent shorter GC.

Reference: http://blog.sun.com/watt/resource/jvm-options-list.htm

5. Scaling: State management and failover. Setup cluster in httpd.conf, enable manager in httpd.conf, add sticky session in tomcat by adding unique name to jvmRoute in server.xml and then add unique route name to httpd balance member and add sticky session to proxy pass. For adding session replication, most of the time, cost of setting it up, coding for it and maintaining it is not worth it.

Mark wraps up with some hints. He recommends setting up clustering in development. He also recommends using 3 nodes instead of 2 to test load balancing and node failure to make sure load of one node is distributed equally on the remaining nodes. Mark also notes that redeployment can lead to memory leaks.


This presentation is available on InfoQ at http://www.infoq.com/presentations/Tuning-Tomcat-Mark-Thomas

Monday, August 17, 2009

Navigating the Rapids - Real World Lessons in Adopting Agile

Sam Newman talks about lessons learned while transitioning clients to agile practices. He 1st recalls a story where he failed to educate stakeholders upfront that he will share with them the good news and the bad news on a regular basis in hope that they can step in and help with the bad news when needed. Next he discusses how when adopting agile productivity 1st goes down as the team is adapting to the new practices. Patience is needed before there is a clear sign of productivity increase. Finally he describes how different clients prefer to see data in different ways.

He summarizes the lessons learned as:

1. Educate stakeholders about early signs of pain

2. Bite of something small – Earn trust early: Need to show that things are better (hard numbers, metrics).

3. Track your data and know how to present it

This presentation is available on InfoQ at  http://www.infoq.com/presentations/navigating-agile-rapids

Saturday, August 15, 2009

The Dancing Agile Elephant: IBM Software Group’s Transition to Agile and Lean Development

At QCON San Francisco 2008, Sue McKinney gave a presentation on IBM’s agile adoption strategy. She started out by giving an overview of some of the business and operational challenges at IBM. These include innovating the business to differentiate and capture new value, heighten responsiveness and closer linkage to customers, and improved time to value. On the operations side, issues included better workload management, improved quality, improved project development cycle times, improved predictability on schedule, and making better use of resources to be more productive

Next Sue described the IBM environment. IBM has a global team, with different companies brought together by acquisition. There are 500 development teams in 5 divisions. Teams are as large as 800 or as small as 20. Very few teams are collocated. They are highly security conscious. They use many tools (due to acquisitions), and many platforms.

Sue describes how IBM software development transformation went from waterfall in the 1980s (rigid, late feedback, slow reaction to market changes) to iterative (customized RUP, community source and component reuse, emphasis on consumability), to agile (global reach, SOA, agile practices, outside-in development, tool and not rules, continuous learning and adaptive planning

Sue mentions how she sold agile as a way to deal with uncertainty, and to respond to changing markets. Agile gives them transparency to check where things are going and enables them to take a course correction if needed by meeting with stakeholders and deciding what change to make. She kept processes to deal with complexity (team size, geo distribution, compliance or regulatory issues) and move from lightweight to things that are more thorough and allow for traceability.

Next Sue lists some things to consider before getting started. These include getting management support, picking strong experienced leaders, picking the right project as a proof point (project with high visibility and some risk), providing the right education, tooling and governance, ability to allow change to occur, and keeping it simple.

Sue defined Agile as short time boxed iteration with stakeholder feedback. This creates automatic constraints (iteration length) that force the team to find defects earlier and address them. The team becomes more responsive and there is always a sense of urgency. There is also a notion of transparency with demos at the end of each iteration. The constraints forces teams to optimize and eliminate waste (automate). Share holder feedback causes us to focus on the essentials.

The initial project was a team of 200 people over 4 sites with 1 week iteration. 1st 2 months, put together infrastructure to enable the team (test case harnesses, continuous integration, static code analysis), then 1 week iterations began. The software was published for regular consumption and received regular feedback. For sustainability (220 out of 500 teams are using agile), they created a practices for distributed development: