At Agile 2009, Brian Shoemaker and Nancy Van Schooenderwoert gave a presentation about safety critical agile projects. They start about by describing how software can contribute to poor safety like in chemical plants, power stations, aviation systems, and medical devices. Many think the solution is do things in a control and sequential matter. In traditional planning, there is a lot of emphasis on upfront planning. In agile you have to have a sense of direction. A product backlog has stories which are requirements that are still negotiable.
Next they discuss some benefits of Agile. It has 2 great strengths: fast time to market, ability to hit a moving target (tracer bullets). Agile teams bring the certainty of project costs forward in time. In traditional development, code gets more brittle with time. More effort is needed to create the same feature later. In general velocity goes down with time as the code base gets more and more complicated (Non-linear effort vs. results). In agile, if we finish off what we commit to in every iteration (delivering small features), we can keep the effort vs. results curve almost linear. If we want to estimate the future based on past performance, a conservative estimate is to use the lowest velocity of the team of the past couple of iterations. Lastly, agile development takes away of the hiding of bad news.
They then cover how risk management benefits from iteration. With iterations, we analyze risk early and often. Requirements and hazards converge when we have positive stories and negative stories prioritized in the product backlog. Hazards are often caught in context. Analysis can be done using a classical risk ranking matrix of probability (high, occasional, low, remote) and severity (major, moderate, minor). Acceptability is ranked using Unacceptable (mitigation required), As low as reasonably practical (mitigate as reasonable, risk decision must be document and reviewed), and Negligible (acceptable without review). Another technique is the fault tree analysis which tracks what is the bad thing that can happen. You start with effect and decide what could be the cause. Another approach is failure mode effect and criticality analysis where you build up from components and decide what can fail in each component.
Next they describe 5 types of failure:
1. Direct failure: software flaw is in normal correct use of system causes or permits incorrect dosage or energy to be delivered to patient.
2. Permitted misuse: software does not reject or prevent entry of data in a way that a) is incorrect according to user instructions, and b) can result in incorrect calculation or logic, and consequent life-threatening or damaging therapeutic action.
Friday, October 2, 2009
Tuesday, September 15, 2009
Deliberate Practice in Software Development
At Agile 2009, Mary Poppendieck gave a talk on the theory behind craftsmanship. She starts out by giving some stats on Hockey Players from the book Outliers. 40% of elite Canadian hockey players were born between January and March, 30% April – June, 20% July – September, 10% October – December. The cut-off date is the end of the calendar year. So the ones born in December are at a disadvantage since they will be the youngest. The same applies for European football players.Mary then defines talent (nature) as abilities, powers, and gifts bestowed upon a person; natural endowments; general intelligence or mental power. Also, talent (nurture) is a special innate or developed aptitude for an activity. Talent is overrated. The most accomplished people need around ten years of deliberate practice before becoming world class (ten-year rule).
Next Mary defines deliberate practice: Identify a skill that needs improvement, devise a focused exercise designed to improve the skill, practice repeatedly, obtain immediate feedback and adjust accordingly, focus on pushing the limits expecting repeated failures, practice regularly and intensely up to 3 hours a day.
Then Mary elaborates on the four elements of deliberate practice:
1. Mentor: masters, teachers, deeply knowledgeable and involved. They hire and grow people, review and guide work, set technical standards, and ensure technical excellence.
2. Challenge: do it frequently and it will become easy, test/integrate/release early and often, change the code frequently, assign challenging work assignment, focus on constant improvement of the product, process, and people. Don’t get comfortable, get better.
3. Feedback: Immediate feedback: Design reviews with mentors providing guidance, common code ownership with peers providing visibility, TDD, CI, customer feedback at every iteration, escaped defect feedback and analysis. Focus on what’s not perfect so it can be improved. You have to ask constantly “why are we doing this?” Bad news first!
4. Dedication: skill development takes time in place, time to learn, and time to invent. Also need motivation, and career paths for technical experts.
Mary wraps up by stressing the importance of clean code and going over the Software Craftsmanship manifesto. We value not only working software but also well crafted software, not only responding to change, but also steadily adding value, not only individuals and interactions, but also a community of professionals, not only customer collaboration but also productive partnerships.This presentation is available on InfoQ at http://www.infoq.com/presentations/poppendieck-deliberate-practice-in-software-development
Integration Tests are a Scam
At Agile 2009, Joe Rainsberger gave a talk about integration tests. Joe states that integration tests (end to end) are a scam. He defines integration tests as is any test were the result of the test depends on the correctness of more than one interesting behavior (multiple objects, multiple unrelated methods, complex enough to be tested on their own).
Next Joe covers some basics system object classifications from Domain Driven Design:
1. Values: Simple like string or number, transient, equality depends on value rather identity, light weight enough to be created when needed and thrown away when not
2. Entity: values with entity semantics have essential identifier and end up getting stored somewhere, equal if ids are equal.
3. Services: Stateless action of some kind.
Joe explains that any test that verifies 1 entity or service behavior possibly with multiple values involved is as focused a test as we can expect. Values are our inputs and values are our outputs. We do not want to test multiple entity or service behavior at the same time as this will lead to long running test suite. These result in slow feedback that eventually lead to no feedback since we stop running them. The tests lose their power of a pool of change detectors. The later a mistake is found the more costly it becomes. This also leads to a false sense of security. Over a long running time, the value of having the tests is roughly the same as not having tests at all.
Tests consist of arrange (put together the things you want to test), act (invoke what you to test), assert (check result). Next Joe discusses the problems with excessive test setup that are usually needed for integration tests. Due to all the setup, we end up having multiple asserts per test or even worse, multiple actions per test. Alternatively we copy/paste setup into multiple tests. The hard coding of dependency is making it impossible to check something without executing something else that we do not want to check right now.
Next Joe explains that when integration test fails it is not clear where the failure is. Integration test are slower than focus tests. # tests needed is the product of the code path(4*3*5=60 test to be thorough). The # of test is not exponential but combinatoric. In reality if we do a really good job we end up with less than 1 % coverage. It is like the lottery, when the odds of winning are 1 in 40 million, buying 10000 tickets is not that much better than buying 10. When we need 4 million tests, writing 20000 tests is not better than 2000.
Next Joe covers some basics system object classifications from Domain Driven Design:
1. Values: Simple like string or number, transient, equality depends on value rather identity, light weight enough to be created when needed and thrown away when not
2. Entity: values with entity semantics have essential identifier and end up getting stored somewhere, equal if ids are equal.
3. Services: Stateless action of some kind.
Joe explains that any test that verifies 1 entity or service behavior possibly with multiple values involved is as focused a test as we can expect. Values are our inputs and values are our outputs. We do not want to test multiple entity or service behavior at the same time as this will lead to long running test suite. These result in slow feedback that eventually lead to no feedback since we stop running them. The tests lose their power of a pool of change detectors. The later a mistake is found the more costly it becomes. This also leads to a false sense of security. Over a long running time, the value of having the tests is roughly the same as not having tests at all.
Tests consist of arrange (put together the things you want to test), act (invoke what you to test), assert (check result). Next Joe discusses the problems with excessive test setup that are usually needed for integration tests. Due to all the setup, we end up having multiple asserts per test or even worse, multiple actions per test. Alternatively we copy/paste setup into multiple tests. The hard coding of dependency is making it impossible to check something without executing something else that we do not want to check right now.
Next Joe explains that when integration test fails it is not clear where the failure is. Integration test are slower than focus tests. # tests needed is the product of the code path(4*3*5=60 test to be thorough). The # of test is not exponential but combinatoric. In reality if we do a really good job we end up with less than 1 % coverage. It is like the lottery, when the odds of winning are 1 in 40 million, buying 10000 tickets is not that much better than buying 10. When we need 4 million tests, writing 20000 tests is not better than 2000.
Monday, September 14, 2009
I Come to Bury Agile, Not to Praise it
Alistair Cockburn
gave a talk at Agile 2009 about software development. He explains that developing software is about people making concrete ideas in an economic context. It is about people inventing, deciding, communicating, solving problems and creating solutions that they don’t yet understand and keep changing, expressing ideas in primitive languages to an interpreter unforgiving of errors, making decisions with limited resources and where every choice has economic consequences. All of engineering and team design patterns kind of fall into this definition.Next Alistair describes one pillar of software development as a cooperative game. Games have positions, moves, and strategies. Some are competitive like chess which is finite and goal-directed or poker which is open-ended. Others are cooperative like rock climbing which is finite and goal-directed, or jazz and music which are open ended games. Some fall in the middle like both like organizational survival and career management. These are infinite.
He explains that software development is cooperative, finite and goal oriented (funding model): 1 round is a deployment of the system. IT system is cooperative and open ended games. They live until we get tired of them, retire them, rebuild them or buy a commercial system. Product Line Management maps to infinite games. After one release 3.0, a competitor comes out with something new, and we follow our release with a .x release just to stay in the game and so on.
The game has two conflicts, to deliver the software and to setup for the next game by refactoring architecture, adding tests, making sure juniors are ready to lead, and adding documentation, etc. Both are competing for finite resources. Only three moves are allowed: invent, decide, and communicate. Also, the situations (almost) never repeat. The strategy that worked on last project, odds are it might not work on the next project. We have to always be alert.
Next Alistair states that we have to adapt to your situation based on the number of people to coordinate and the criticality to the application. Communication and control structure change and strategies, setup, conventions need to change accordingly. Face to face is the most effective form of communication. When there are question and answer the best form of communication is people at whiteboard, then people on phone, then people on chat. If there are no question and answer then best is videotape, then paper.
Alistair describes distance as expensive. If we assume that 2 programming in pairs communicating once every 20 minutes cost nothing then 12 people on same floor but in different rooms cost 100K/yr penalty in communication speed, and 12 people on different floors cost 300K/yr penalty in communication speed. People issues determine a project’s speed. Setup a project so people can detect when something is not right. They need to care enough do something about it and effectively pass along the information. Speed of the project is the speed of which ideas moves between minds (attitude and mechanics for communication).
Thursday, August 20, 2009
Performance Tuning for Apache Tomcat
At SpringOne 2009, Mark Thomas gave a talk about Tomcat configuration. Mark mentions that 80% of performance problems will be in the application rather than in tomcat. Having said that, Mark shares some tomcat optimizations:
1. Logging: Turn off logging to std and only log to a file. Add rotation by limiting file size and # of file.
2. Connector Tuning: depends on application, client(web of mobile), tcp, http keep alive, and sll. Keep alive uses one tcp connection for multiple http requests. Modern pages uses over 100 requests per page. Blocking IO connectors use a thread per connection. For sticky sessions, layer 4 load balancer uses client ip and port (not always available when isps proxy user requests). Layer 7 load balancer understands http headers and tomcat puts markers for sticky session in header.
a. BIO – Java blocking connector. Stable (use as default)
b. Native (APR) – non blocking, Open SSL (use when need SSL or high concurrency with keep alive). Not stable on Solaris so use NIO instead.
c. Java non blocking IO – JSSE based SSL
Tuning:
a. MaxThreads: maximum number of concurrent requests (in BIO, it is the max number of open connections). Set to 200 – 800. Start with 400 and increase if low CPU usage or decrease with high usage
b. MaxKeepAliveRequests: Use 1 (turn off) or 100. Used to clean up and close unused requests. Disable for BIO with high concurrency, layer 4 load balancer, no ssl. Enable for ssl, APR/NIO, layer 7 load balancer. Disable if using httpd and have it on same box as tomcat
c. ConnectionTimeOut. Typical value is 3000 (3 seconds). Default of 20000 (20s) is too high. Increase for slow clients or layer 7 load balancer with connection pool and keep alive.
3. Content Cache: caches static content. Configured with CacheMaxSize in KB (10240), CacheTLL in ms(5000), CacheMaxFileSize in KB (512).
4. JVM:
a. Xmx/-Xmx used to define size of java heap. Aim to set as low as possible to leave enough space for all other processes. Setting too high can also lead to very long GC cycles. If app needs 200M set it for 250 or 300M.
b. –XX:NewSize/-XX:NewRatio set to 25%-33% of total java heap size. No need to change, but if you need to go into these details, set NewRatio over NewSize.
c. -XX:MaxGCPauseMillis –XX:MaxGCMinorPauseMillis. Goal to end up with more frequent shorter GC.
Reference: http://blog.sun.com/watt/resource/jvm-options-list.htm
5. Scaling: State management and failover. Setup cluster in httpd.conf, enable manager in httpd.conf, add sticky session in tomcat by adding unique name to jvmRoute in server.xml and then add unique route name to httpd balance member and add sticky session to proxy pass. For adding session replication, most of the time, cost of setting it up, coding for it and maintaining it is not worth it.
Mark wraps up with some hints. He recommends setting up clustering in development. He also recommends using 3 nodes instead of 2 to test load balancing and node failure to make sure load of one node is distributed equally on the remaining nodes. Mark also notes that redeployment can lead to memory leaks.
This presentation is available on InfoQ at http://www.infoq.com/presentations/Tuning-Tomcat-Mark-Thomas
1. Logging: Turn off logging to std and only log to a file. Add rotation by limiting file size and # of file.
2. Connector Tuning: depends on application, client(web of mobile), tcp, http keep alive, and sll. Keep alive uses one tcp connection for multiple http requests. Modern pages uses over 100 requests per page. Blocking IO connectors use a thread per connection. For sticky sessions, layer 4 load balancer uses client ip and port (not always available when isps proxy user requests). Layer 7 load balancer understands http headers and tomcat puts markers for sticky session in header.
a. BIO – Java blocking connector. Stable (use as default)
b. Native (APR) – non blocking, Open SSL (use when need SSL or high concurrency with keep alive). Not stable on Solaris so use NIO instead.
c. Java non blocking IO – JSSE based SSL
Tuning:
a. MaxThreads: maximum number of concurrent requests (in BIO, it is the max number of open connections). Set to 200 – 800. Start with 400 and increase if low CPU usage or decrease with high usage
b. MaxKeepAliveRequests: Use 1 (turn off) or 100. Used to clean up and close unused requests. Disable for BIO with high concurrency, layer 4 load balancer, no ssl. Enable for ssl, APR/NIO, layer 7 load balancer. Disable if using httpd and have it on same box as tomcat
c. ConnectionTimeOut. Typical value is 3000 (3 seconds). Default of 20000 (20s) is too high. Increase for slow clients or layer 7 load balancer with connection pool and keep alive.
3. Content Cache: caches static content. Configured with
4. JVM:
a. Xmx/-Xmx used to define size of java heap. Aim to set as low as possible to leave enough space for all other processes. Setting too high can also lead to very long GC cycles. If app needs 200M set it for 250 or 300M.
b. –XX:NewSize/-XX:NewRatio set to 25%-33% of total java heap size. No need to change, but if you need to go into these details, set NewRatio over NewSize.
c. -XX:MaxGCPauseMillis –XX:MaxGCMinorPauseMillis. Goal to end up with more frequent shorter GC.
Reference: http://blog.sun.com/watt/resource/jvm-options-list.htm
5. Scaling: State management and failover. Setup cluster in httpd.conf, enable manager in httpd.conf, add sticky session in tomcat by adding unique name to jvmRoute in server.xml and then add unique route name to httpd balance member and add sticky session to proxy pass. For adding session replication, most of the time, cost of setting it up, coding for it and maintaining it is not worth it.
Mark wraps up with some hints. He recommends setting up clustering in development. He also recommends using 3 nodes instead of 2 to test load balancing and node failure to make sure load of one node is distributed equally on the remaining nodes. Mark also notes that redeployment can lead to memory leaks.
Monday, August 17, 2009
Navigating the Rapids - Real World Lessons in Adopting Agile
Sam Newman talks about lessons learned while transitioning clients to agile practices. He 1st recalls a story where he failed to educate stakeholders upfront that he will share with them the good news and the bad news on a regular basis in hope that they can step in and help with the bad news when needed. Next he discusses how when adopting agile productivity 1st goes down as the team is adapting to the new practices. Patience is needed before there is a clear sign of productivity increase. Finally he describes how different clients prefer to see data in different ways.
He summarizes the lessons learned as:
1. Educate stakeholders about early signs of pain
2. Bite of something small – Earn trust early: Need to show that things are better (hard numbers, metrics).
3. Track your data and know how to present it
This presentation is available on InfoQ at http://www.infoq.com/presentations/navigating-agile-rapids
He summarizes the lessons learned as:
1. Educate stakeholders about early signs of pain
2. Bite of something small – Earn trust early: Need to show that things are better (hard numbers, metrics).
3. Track your data and know how to present it
This presentation is available on InfoQ at http://www.infoq.com/presentations/navigating-agile-rapids
Saturday, August 15, 2009
The Dancing Agile Elephant: IBM Software Group’s Transition to Agile and Lean Development
At QCON San Francisco 2008, Sue McKinney gave a presentation on IBM’s agile adoption strategy. She started out by giving an overview of some of the business and operational challenges at IBM. These include innovating the business to differentiate and capture new value, heighten responsiveness and closer linkage to customers, and improved time to value. On the operations side, issues included better workload management, improved quality, improved project development cycle times, improved predictability on schedule, and making better use of resources to be more productive
Next Sue described the IBM environment. IBM has a global team, with different companies brought together by acquisition. There are 500 development teams in 5 divisions. Teams are as large as 800 or as small as 20. Very few teams are collocated. They are highly security conscious. They use many tools (due to acquisitions), and many platforms.
Sue describes how IBM software development transformation went from waterfall in the 1980s (rigid, late feedback, slow reaction to market changes) to iterative (customized RUP, community source and component reuse, emphasis on consumability), to agile (global reach, SOA, agile practices, outside-in development, tool and not rules, continuous learning and adaptive planning
Sue mentions how she sold agile as a way to deal with uncertainty, and to respond to changing markets. Agile gives them transparency to check where things are going and enables them to take a course correction if needed by meeting with stakeholders and deciding what change to make. She kept processes to deal with complexity (team size, geo distribution, compliance or regulatory issues) and move from lightweight to things that are more thorough and allow for traceability.
Next Sue lists some things to consider before getting started. These include getting management support, picking strong experienced leaders, picking the right project as a proof point (project with high visibility and some risk), providing the right education, tooling and governance, ability to allow change to occur, and keeping it simple.
Sue defined Agile as short time boxed iteration with stakeholder feedback. This creates automatic constraints (iteration length) that force the team to find defects earlier and address them. The team becomes more responsive and there is always a sense of urgency. There is also a notion of transparency with demos at the end of each iteration. The constraints forces teams to optimize and eliminate waste (automate). Share holder feedback causes us to focus on the essentials.
The initial project was a team of 200 people over 4 sites with 1 week iteration. 1st 2 months, put together infrastructure to enable the team (test case harnesses, continuous integration, static code analysis), then 1 week iterations began. The software was published for regular consumption and received regular feedback. For sustainability (220 out of 500 teams are using agile), they created a practices for distributed development:
Next Sue described the IBM environment. IBM has a global team, with different companies brought together by acquisition. There are 500 development teams in 5 divisions. Teams are as large as 800 or as small as 20. Very few teams are collocated. They are highly security conscious. They use many tools (due to acquisitions), and many platforms.
Sue describes how IBM software development transformation went from waterfall in the 1980s (rigid, late feedback, slow reaction to market changes) to iterative (customized RUP, community source and component reuse, emphasis on consumability), to agile (global reach, SOA, agile practices, outside-in development, tool and not rules, continuous learning and adaptive planning
Sue mentions how she sold agile as a way to deal with uncertainty, and to respond to changing markets. Agile gives them transparency to check where things are going and enables them to take a course correction if needed by meeting with stakeholders and deciding what change to make. She kept processes to deal with complexity (team size, geo distribution, compliance or regulatory issues) and move from lightweight to things that are more thorough and allow for traceability.
Next Sue lists some things to consider before getting started. These include getting management support, picking strong experienced leaders, picking the right project as a proof point (project with high visibility and some risk), providing the right education, tooling and governance, ability to allow change to occur, and keeping it simple.
Sue defined Agile as short time boxed iteration with stakeholder feedback. This creates automatic constraints (iteration length) that force the team to find defects earlier and address them. The team becomes more responsive and there is always a sense of urgency. There is also a notion of transparency with demos at the end of each iteration. The constraints forces teams to optimize and eliminate waste (automate). Share holder feedback causes us to focus on the essentials.
The initial project was a team of 200 people over 4 sites with 1 week iteration. 1st 2 months, put together infrastructure to enable the team (test case harnesses, continuous integration, static code analysis), then 1 week iterations began. The software was published for regular consumption and received regular feedback. For sustainability (220 out of 500 teams are using agile), they created a practices for distributed development:
Subscribe to:
Posts (Atom)