Performance Tuning for Apache Tomcat

At SpringOne 2009, Mark Thomas gave a talk about Tomcat configuration. Mark mentions that 80% of performance problems will be in the application rather than in tomcat. Having said that, Mark shares some tomcat optimizations:

1. Logging: Turn off logging to std and only log to a file. Add rotation by limiting file size and # of file.

2. Connector Tuning: depends on application, client(web of mobile), tcp, http keep alive, and sll. Keep alive uses one tcp connection for multiple http requests. Modern pages uses over 100 requests per page. Blocking IO connectors use a thread per connection. For sticky sessions, layer 4 load balancer uses client ip and port (not always available when isps proxy user requests). Layer 7 load balancer understands http headers and tomcat puts markers for sticky session in header.

a. BIO – Java blocking connector. Stable (use as default)

b. Native (APR) – non blocking, Open SSL (use when need SSL or high concurrency with keep alive). Not stable on Solaris so use NIO instead.

c. Java non blocking IO – JSSE based SSL

Tuning:

a. MaxThreads: maximum number of concurrent requests (in BIO, it is the max number of open connections). Set to 200 – 800. Start with 400 and increase if low CPU usage or decrease with high usage

b. MaxKeepAliveRequests: Use 1 (turn off) or 100. Used to clean up and close unused requests. Disable for BIO with high concurrency, layer 4 load balancer, no ssl. Enable for ssl, APR/NIO, layer 7 load balancer. Disable if using httpd and have it on same box as tomcat

c. ConnectionTimeOut. Typical value is 3000 (3 seconds). Default of 20000 (20s) is too high. Increase for slow clients or layer 7 load balancer with connection pool and keep alive.

3. Content Cache: caches static content. Configured with CacheMaxSize in KB (10240), CacheTLL in ms(5000), CacheMaxFileSize in KB (512).


4. JVM:

a. Xmx/-Xmx used to define size of java heap. Aim to set as low as possible to leave enough space for all other processes. Setting too high can also lead to very long GC cycles. If app needs 200M set it for 250 or 300M.

b. –XX:NewSize/-XX:NewRatio set to 25%-33% of total java heap size. No need to change, but if you need to go into these details, set NewRatio over NewSize.

c. -XX:MaxGCPauseMillis –XX:MaxGCMinorPauseMillis. Goal to end up with more frequent shorter GC.

Reference: http://blog.sun.com/watt/resource/jvm-options-list.htm

5. Scaling: State management and failover. Setup cluster in httpd.conf, enable manager in httpd.conf, add sticky session in tomcat by adding unique name to jvmRoute in server.xml and then add unique route name to httpd balance member and add sticky session to proxy pass. For adding session replication, most of the time, cost of setting it up, coding for it and maintaining it is not worth it.

Mark wraps up with some hints. He recommends setting up clustering in development. He also recommends using 3 nodes instead of 2 to test load balancing and node failure to make sure load of one node is distributed equally on the remaining nodes. Mark also notes that redeployment can lead to memory leaks.


This presentation is available on InfoQ at http://www.infoq.com/presentations/Tuning-Tomcat-Mark-Thomas

Navigating the Rapids - Real World Lessons in Adopting Agile

Sam Newman talks about lessons learned while transitioning clients to agile practices. He 1st recalls a story where he failed to educate stakeholders upfront that he will share with them the good news and the bad news on a regular basis in hope that they can step in and help with the bad news when needed. Next he discusses how when adopting agile productivity 1st goes down as the team is adapting to the new practices. Patience is needed before there is a clear sign of productivity increase. Finally he describes how different clients prefer to see data in different ways.

He summarizes the lessons learned as:

1. Educate stakeholders about early signs of pain

2. Bite of something small – Earn trust early: Need to show that things are better (hard numbers, metrics).

3. Track your data and know how to present it

This presentation is available on InfoQ at  http://www.infoq.com/presentations/navigating-agile-rapids

The Dancing Agile Elephant: IBM Software Group’s Transition to Agile and Lean Development

At QCON San Francisco 2008, Sue McKinney gave a presentation on IBM’s agile adoption strategy. She started out by giving an overview of some of the business and operational challenges at IBM. These include innovating the business to differentiate and capture new value, heighten responsiveness and closer linkage to customers, and improved time to value. On the operations side, issues included better workload management, improved quality, improved project development cycle times, improved predictability on schedule, and making better use of resources to be more productive

Next Sue described the IBM environment. IBM has a global team, with different companies brought together by acquisition. There are 500 development teams in 5 divisions. Teams are as large as 800 or as small as 20. Very few teams are collocated. They are highly security conscious. They use many tools (due to acquisitions), and many platforms.

Sue describes how IBM software development transformation went from waterfall in the 1980s (rigid, late feedback, slow reaction to market changes) to iterative (customized RUP, community source and component reuse, emphasis on consumability), to agile (global reach, SOA, agile practices, outside-in development, tool and not rules, continuous learning and adaptive planning

Sue mentions how she sold agile as a way to deal with uncertainty, and to respond to changing markets. Agile gives them transparency to check where things are going and enables them to take a course correction if needed by meeting with stakeholders and deciding what change to make. She kept processes to deal with complexity (team size, geo distribution, compliance or regulatory issues) and move from lightweight to things that are more thorough and allow for traceability.

Next Sue lists some things to consider before getting started. These include getting management support, picking strong experienced leaders, picking the right project as a proof point (project with high visibility and some risk), providing the right education, tooling and governance, ability to allow change to occur, and keeping it simple.

Sue defined Agile as short time boxed iteration with stakeholder feedback. This creates automatic constraints (iteration length) that force the team to find defects earlier and address them. The team becomes more responsive and there is always a sense of urgency. There is also a notion of transparency with demos at the end of each iteration. The constraints forces teams to optimize and eliminate waste (automate). Share holder feedback causes us to focus on the essentials.

The initial project was a team of 200 people over 4 sites with 1 week iteration. 1st 2 months, put together infrastructure to enable the team (test case harnesses, continuous integration, static code analysis), then 1 week iterations began. The software was published for regular consumption and received regular feedback. For sustainability (220 out of 500 teams are using agile), they created a practices for distributed development:

Realistic About Risk: Software Development with Real Options

At QCON London 2008, Olav Maassen and Chris Matts gave a talk about Real Options. They started out with a quick exercise where the audience was divided up into teams and each team had to write down who they thought would finish task A 1st and who would finish task B 1st. then they gave detailed instructions of each task and asked the teams to complete them. At the end, none of the teams picked the correct winner, where as both Olav and Chris picked correctly. The point of the exercise was that Olav and Chris made their decisions later where as the teams made their decision early even before knowing what the tasks were.

They then compared the risk profile of an agile project (many short increments) vs. the risk profile of waterfall project (one long increment). People who are risk averse prefer the waterfall model even though the agile risk profile appears safer.

Next they explain real options which is an approach that allow optimal decisions with the current context. It has two aspects: Math and Psychology. Using math to price real options is hard, but the results of the math tell us that:

1. Options have value

2. Options expire

3. Never commit early unless you know why

When making a decision, you can be right, wrong, or uncertain. People hate uncertainty so much that they will rather be wrong than uncertain. From a psychological aspect, rather than saying don’t make the decision now, say let’s make the decision under these conditions and circumstances (postponed till a future date) “let’s do it when…”

They wrap up by showing how real options apply in IT. No big upfront design and defer decisions too last responsible moment. Pair programming gives you options by sharing knowledge and making you less dependent on one person. Also, by assign juniors 1st to projects this leaves seniors (valuable skills) to coach and mentor with an option to work on emergency or high priority projects that come up later. When using an MS project plan, it gives us the zero probability date. Before that date there is no chance of delivering early. On that date you have zero percent probability of delivering and then the distribution builds from there. A study shows that to get 90% probability you need to multiply the given estimates by 4.

This presentation is available on InfoQ at  http://www.infoq.com/presentations/software-with-real-options

Coaching and Scaling Agility

At QCON San Francisco 2008, David Hussman gave a presentation on coaching. He defines coaching as helping plan products, helping with iterative delivery, helping tune and improve, and helping to build community. Coaching is about guiding people from how to why Shu, Ha, Ri or practice, improvise, and evolve. Coaching gigs and styles vary greatly. They can be prescriptive where this is what you should do or descriptive where this is what I have seen work.

When coaching large communities, it is important to understand that there is no recipe. Each community is unique. David recommends coaching respectful change where change must happen with people and not to them. Try to understand who people are, how they work right now, what’s working for them, and what motivates them to change. Try to find some set of practices, a way to bound as a community, some way to talk about products, some way to deliver it, and some way to tune it.

Then David talks about providing real education by building a library, providing pragmatic (need to try and experience things) training, coaching classes, facilitating training, and teaching TDD-refactoring.

David talks about chartering which covers who is involved, what we are trying to get done, what are the risks, timeline, etc… David uses chartering to find common goals and build a collective groove. Here is what we are trying to do, here is how we know when we are done, this is who is building it, etc...

Next David discusses scaling core practices. He recommends creating pragmatic product roadmaps (3, 6, 12, 18 months), pairing beyond programming (business person, testing person, and development person), radiating information, making issues visible, and promoting improvisation.

Then David describes two situations: many teams, many products or many teams one product. They need to be working in cross cutting concerns. Also we need to build customer communities composed of people that have interest, passion, knowledge, and some decision making capability.

Lastly, David discusses issues with large distributed communities. Conference calls are NOT just like being there. One option is building whole sub-teams around common goals and common understandings.

This presentation is available on InfoQ at http://www.infoq.com/presentations/coaching-scaling-agility

Behavior Driven Development

At QCON 2008, Dan North gave a talk about BDD. Dan start by stating that projects fail because the products come in too late, cost too much, do the wrong thing, are unstable in production and crash every day, break the rules, or the code is impossible to work with.

Next Dan compares the bottom up and top down approached to delivering software. Frederick Taylor’s scientific management theory kind of says that people are lazy and stupid and as a result we need to make their work simple and we need to constantly monitor them. We need to separate work from management. Top down process on delivering software is based on premise that people downstream are increasingly pluggable and prone to make mistakes so we need to front load our process with all the smart stuff so there is less risk of something going wrong as we go further along. Since the price of defects increase the later it is discovered, we need big upfront design(BUFD). Plan, requirements specs, functional design, detailed design, test plans is hedging against the exponential cost of change, but this is in turn creates it and thus creates a reinforcing loop. There is no cause and effect, they both cause one another.

Meanwhile Deming’s premise is that people generally want to do a good job and take pride in their work and they respond well to an encouraging environment. Bottom up process builds business objects to represent business domain and then produces an architecture to wire them together.

Next Dan shares statistics that say that 60% of features are never or rarely used, 30% are sometimes used, and 10% to 15% are often or always used.

Then Dan explains that Behavior Driven Development is about implementing an application by describing it from the point of view of its stake holders. Only focus on high-value features, flatten the cost of change of anything at any stage, prioritize often, change often, adapt to feedback and continuously learn. Pull requirements as needed, evolve the design, code that can change, frequent code integration, frequent regression tests

BDD is derivative from XP (TDD, CI), Domain Driven Design, Acceptance Test Driven Planning (estimation based on building one scenario on top of another), Neurolinguistic programming (NLP), systems thinking.



BDD is getting the words right, enough is enough and more is waste while less is irresponsible, agreeing on DONE, outside-in (more systemic approach), interactions (series of interaction between people with different skills and software is an output of these interactions). People over process.

Dan next elaborates:

PMP or CSM?

All these acronyms can be confusing...