At Agile 2008 David Anderson talks about agile principles, practices and the community. David covers multiple topics that reveal how CMMI, Kanban and Real Options Theory help in making organizations scale, lean, and of high maturity.
David mentions that trying to scale agile across an enterprise is forcing a change on the community. We are trying to expand agile to the entire IT department and even more to other areas like marketing and recruiting. We need a wider definition of agile that is not narrowly focused on software development. We have the manifesto and the principles, but it is 7 years old and in badly need of an update. David stresses that we should not underestimate the value of what the manifesto gave us, but it is not everything of what it means to be agile. David controversially states that the agile manifesto is superstitiously declared.
Next David clears up the ideas behind the manifesto and how can we capture them as a model in 3 underlying paradigms
1. High trust, high social capital, highly collaborative culture
David describes the different cultures between the traditional and agile communities. The traditional community was rooted in government and developed a body of knowledge from defense. The high cost of failure (money, lives, politics) makes people conservatives, risk averse, and have low trust. On the other hand, the agile community was started by thought leaders from start-up companies building web sites. They were innovating daily, empowering people to release new functionality with an assumed zero cost of failure. They realized that in order to innovate at a fast pace, there was no time to write requirement and negotiate contracts. It was better to let people work and adopt a failure tolerant attitude. This was a highly collaborative approach with a high trust liberal culture. This led to the 3rd value of the agile manifesto, customer collaboration (high trust) over contract negotiation (low trust). Self organization or empowering, delegating and tolerating failure through adoption of high trust culture is better than contract and audits in low trust culture.
2. Perfect is the enemy of good enough.
David mentions that CMMI taught us that project success came from organizational maturity. Level 1 is chaos, Level 2 is repeatable and Level 3 is a defined process. At this level we should be able to deliver successfully every time. So the assumption was that we need to pursue perfection and eliminate ambiguity and uncertainty. We need accurate plans, analysis, estimates, and diagramming. By the time we are done, they are already out of date. This led to the agile manifesto value of working software over comprehensive documentation.
Perfection also led to account for everything that can go wrong and create 10K pages of process documentation and hire companies to help with creating and documenting these huge processes. This led to the agile manifesto value of individuals and interactions over processes and tools.
David concludes that working software over comprehensive documentation and value of individuals and interactions over processes and tools can be summed up with “perfect is the enemy of good enough”. These 2 statements rebel against the pursuit of perfection and ask us to embrace uncertainty.
3. Knowledge work is perishable
Next David compares bananas to requirement documents and notes that they are both perishable. Requirements documents are perishable because humans forget, market strategies change, strategic position change, new competitors enter markets, and economic conditions change. David also notes that lead time is vital to business competitiveness so we need short cycle time. This led to the agile manifesto statement of responding to change over following a plan.
The model includes the 3 main paradigms 1) high trust, high social capital, highly collaborative culture, 2) perfect is the enemy of good enough, and 3) knowledge work is perishable. These were derived from the agile manifesto and can be scientifically proven. The model also includes craftsmanship, sustainable level, reflect and adjust, waste elimination which are derived from the agile principles.
Next David describes lean development and its focus on value, flow and pull.
Lean focuses on value (customer intimacy is important to understand value), value is contextual and context is temporal, the cost of delivery is high because of the uncertainty of future value. Lean has the concept of the cost of delay.
Lean focuses on flow – remove impediments, do root cause analysis and make sure it does not happen again. Inventory is not an asset, it is a liability.
David concludes that value and flow represent the same idea as responding to change over following a plan
Lean also has the concept of pull. Limiting work in progress and balancing demand against throughput. Pull ensures that no one is overloaded. This concept maps to the sustainable pace principle.
Modern western management are obsessed with the relentless pursuit of efficiency. Since there is always a setup cost, clean up cost, and coordination cost (communication cost) then economy of scale or larger batches. This works in manufacturing because coordination costs do not grow but this does not work in software because coordination costs grow nonlinearly with batch size.
Toyota did not have space for the inventory produced by large batches. Those constraints forced them to go with smaller batches. They realized that smaller batches allowed the right product to be brought to the market at the right time realizing more value for the business. Small batches avoid waste and make a business more responsive to the market. Large batches force discounts to get rid of unwanted inventory.
David concludes that elimination of waste is a lean paradigm opposite of economy of scale. We value elimination of waste over achieving economy of scale, and we value delivery of value over reduction of cost.
Next David compares agile and lean. Agile does not include elimination of waste and pull, but does include value, flow and kaizen idea. Agile model has bunch of ideas that do not exist in Lean like perfect is the enemy of good enough, and high trust, high social capital, highly collaborative culture.
Next David discusses Kanban. He does not go into details but gives a brief overview and compares and contrasts it with agile. Kanban is a lean solution for value, flow and pull. Kanban means signal cards. It limits work in progress and creates a pull system. Kanban does not use timeboxed iterations and almost completely eliminates planning and estimation. Kanban covers value flow and pull and has been shown to enable continuous improvements. It also covers agile ideas that do not appear in lean. Things like perfect is the enemy of good enough are covered by constant incremental delivery, frequent releases of working code, transparency of work in progress, frequent prioritization of input queue and tolerance for failure. It covers high trust, high social capital, highly collaborative culture through a kaizen culture of transparency, visual controls and kanban signaling (provides for self organization). It covers knowledge work is perishable because it is designed to optimize cycle time through exposure and elimination of waste by concentrating of the value stream. Craftsmanship is covered by stop the line idea. Division of labor is ok. It encourages pride of workmanship and high craft ethics. Sustainable pace is covered by the limit on work in progress which ensures the team is never overloaded.
David concludes that Kanban meets all criteria for lean and agile methods.
David then states that Kanban gives the agile community a convenient way of embracing division of labor without losing our agile values. At large enterprise scale, division of labor is inevitable, it isn’t practical to hire large workforce of highly skilled top performing generalist, nor is it economical. In any value chain, there must be some division of labor. Division of labor has marginal utility. Flow pushes against division of labor, but marginal utility, cost and other economical concerns push back. So David concludes that division of labor is inevitable.
Next, David predicts that by embracing lean thinking to its fullest, software will change to a software supply chain of software product lines and software cloud services. Software product line is the idea of describing software assets in XML and produces 80% of re-usable assets that you can then finish and compile. Software cloud services are web services enabled in the cloud. These two will allow for software re-use and a means of exchanging value. Modeling will be back in fashion and re-use will drive the next wave of productivity improvements. We can get 10x improvement through re-use. But we can also get 4x to 10x improvement through agile and lean methods. This leads to 100x improvement in productivity. Software development will not be off shored. Software that is of high value, highly differentiated, and requires high customer involvement will be built locally. Software that is of low value and undifferentiated will be built offshore.
Next David covers CMMI and states that it is still relevant. Enterprise clients need to be more predictable, have low variability and have short lead times, and be data driven and less subjective and reactionary. These business objectives require them to have the organizational high maturity of CMMI level 4. Most agile teams are good at putting in place what is required to get CMMI level 2 and 3. These are things like project planning, verification, validation, project monitoring and control, process definition, and requirement development. However, agile teams do not achieve what is need for CMMI level4 like decision analysis and resolutions, integrated project management, training management, supplier agreement management, and issue and risk management. To achieve a high maturity model we need a shift in mind set in the community. CMMI model has 20 years behind it. CMMI is relevant to agile community.
Lean has its roots in the work of Deming’s system of profound knowledge. CMMI and agile have common goals and are compatible. CMMI level 2 and level 3 and the anti patterns that resulted from them make them are not as compatible with agile, but this is not true for high maturity level 4. Predictability allows you to be more agile because of a shorter cycle. It allows a faster response to change. Long term institutionalized learning is aligned with agile aspiration of continuous improvements. Agile teams rely of subjective analysis and fail to embrace quantitative analysis. Retrospective are not data driven. There needs to be quantitative analysis. Processes needs be institutionalized. High maturity is not achieved by imposing power but by self organization.
David again concludes that Kanban is a method of achieving agile and high maturity.
David next quickly covers Real Options Theory. He states that real options change everything. Through out what you now about project planning and scheduling, risk planning, portfolio management etc. and start again. The theory asks us to not try to predict the future, have options available, know how long you have to pick an option and push decisions to the last possible moment. People like to be right, but we prefer to be wrong rather than uncertain.
David wraps up by stating that the agile community needs a formal mechanism to embrace change in the agile world. We need a model for agility and an institutional home and an agile program with formal meetings and publications. We need to be open minded, embrace innovation, expand coverage and eliminate more bad paradigms. We need to embrace management organizational skills as essential to furthering our goals. We need to be open to CMMI. We need to recognize that division of labor is essential but it does not mean an end to craftsmanship ethics. The agile community should be inclusive, innovative and an open society that embraces change.
This presentation is available on InfoQ at http://www.infoq.com/presentations/Agile-Directions-David-Anderson
