Organizational Agility

The Economist Intelligence Unit published a paper entitled "Organizational agility: how businesses can survive and thrive in turbulent times." The paper is based on in depth interviews and surveys of 349 executives around the world on the benefits, challenges and risks associated with creating a more agile organization.

The report finds that organizational agility is a core differentiator in today’s rapidly changing business environment. Agility may also be linked to profitable growth as research conducted at MIT suggests that agile firms grow revenue 37% faster and generate 30% higher profits than non-agile companies.

Yet most companies admit they are not flexible enough to compete successfully. The report finds that internal barriers stall agile change efforts and the main obstacles to business responsiveness are slow decision-making, conflicting departmental goals and priorities, risk-averse cultures and silo-based information.

Technology can play an important supporting role in enabling organizations to become more agile. Technology should function as a change agent in the use and adoption of best-in-class knowledge sharing processes, so companies can improve their use of critical data.

The report concludes there are a number of steps that management can consider to lighten the burden of agile transformation:

1. Minimizing excess spending and non-core programs so companies can better direct limited resources to satisfying customer expectations. 
2. Minimizing information silos so business leaders can improve collaboration inside and outside their enterprise and better align departmental goals and performance measures with overall strategy. 
3. Integrate and automate fundamental knowledge-sharing processes to improve decision-making, convert information into insight and enable IT to advance an organization’s ability to problem-solve. 

The complete report and survey results can be found here.

Comparative Agility

So how are you doing at adopting agile? You decided to go Agile. You got the training and coaching. Six months have passed. Are you where you should be? Are you excelling in some areas? Do you need to improve in others? How well are you doing compared to your competitors?

To answers these questions, Kenny Rebin and Laurie Williams presented Comparative Agility at Agile 2010. Comparative Agility is an assessment that evaluated multiple dimensions of agility that lead to actionable recommendations. The assessment framework consists of 7 dimensions covering 3 to 6 characteristics per dimension for a total of 125 questions.

The seven dimensions represent broad classifications of changes to be expected of a team or organization as it becomes more agile. The seven dimensions are:

• Teamwork
• Requirements
• Planning
• Technical practices
• Quality
• Culture
• Knowledge creation

Questions are asked to assess a team's score on each characteristic and answers are in the form of:

• More true than false
• Neither true nor false
• More false than true
• False

The philosophy behind the assessment is that organizations do not need to be perfect but they need to be better than their competitors. The assessment is not used to determine maturity levels. It simply answers “How am I doing compared to my competitors?”

Want to find out how you are doing? Take the survey and find out more at www.ComparativeAgility.com

Accelerating Your Organization’s Agile Adoption

Bryan Campbell and Robbie Mac Iver gave a talk at Agile 2010 about Agile Adoption. They started out by discussing how acquiring a skill requires time, practice and a mentor. They then defined 7 stages of agile expertise based on the work of Meiler Page-Jones:

• Innocent: unaware of agile techniques
• Aware: aware and seeking to learn more
• Apprentice: ready to apply their skills to a real project
• Practitioner: leap from classroom projects to those of real world complexity
• Journeyman: agile techniques embedded natural way of working
• Master: range of real-world project experiences; ability to teach these techniques to Apprentices
• Researcher: sharing knowledge with a broader community; champion to further extend the benefits of agile techniques

Next they identify the risk and challenges from moving from one stage to the next. Moving from innocent to aware can happen fairly quickly. Moving from apprentice, to practitioner, to journeyman requires more effort and will take longer; however, it is during these stages that you get the greatest increases in productivity. Reaching the master and researcher stage will take even longer and not many companies see value in having employees reach this level. The tipping point is usually when more than 50% of the organization is at the practitioner level and more than 75% is operating at the apprentice level.

Next Bryan and Robbie discuss the J-curve effect where an apprentice struggles in adapting his new skills to real situations and reverts back to old techniques. This causes a dip in the skills progression which results in a decrease of productivity and a risk that crossing to the next level might stall or eventually fail. This is where having access to an experience coach or mentor is crucial to overcome the J curve and achieve a successful agile adoption.

Bryan and Robbie recommend 2 techniques that can help accelerate agile adoption:

• The Breadth approach focuses on developing a solid foundation of best practices and refining them over time. This works best in the move from innocent to aware, or from practitioner to journeyman.
• The Depth approach is focused on a more active engagement/participation of mentors on a real project. It is more of a deep dive and is best for crossing the J-curve from apprentice to practitioner.

They next cover agile leadership guidelines. They recommend

• Addressing culture and values first and then practices will generally follow
• Working in ways that embrace change and adjusting methods to fit the project
• Creating teams of advanced citizens to ensure team dynamics
• Influencing team decisions by setting movable boundaries

Bryan and Robbie wrapped up by presenting each group with different real life scenarios and having each group discuss different ways of resolving them.

The scenarios can be found at http://www.robbiemaciver.com/documents/presentations/A2010-Agile%20Maturity%20-%20Problem%20Scenarios.pdf

The presentation slides are available at http://www.robbiemaciver.com/documents/presentations/A2010-Agile%20Maturity%20-%20Presentation.pdf

Kanban Adoption at SEP

Chris Shinkle gave a talk about Kanban adoption at SEP during the Agile 2009 conference. Chris described the Dreyfus Model of Skill Acquisition and showed how the team at SEP progressed from one stage to the other as they learned and applied Kanban.
Stage 1 – Novice: Little or no previous experience. It is about following context free rules. No discretionary judgment. Want to accomplish a goal, not learn. Team had clear understanding of project, clear priority of work items, and clearly see progress from the introduction of Kandan boards and standup meetings.

Stage 2 – Advanced Beginner: Still rules based, but start understanding rules within context and based on past experience. Team understood WIP (work in progress) limits, began to collaborate and identify bottlenecks and areas for improvement.

Stage 3 – Competent: People recognize patterns and principles. Rule sets become more rules of thumb. They start to establish guidelines and seek out and solve problems. They see actions in terms of long term plans and goals. The team felt a sense of ownership. They started pulling in alternate practices to optimize the process and solve specific problems. They began to discover and understand lean principles themselves. Things like cause and effect of flow, value, quality, quality, and waste.

Stage 4 – Proficient: People seek out and want to understand the big picture. They perceive a complete system instead of a set of individual parts. They can learn from the experience of others and take full advantage of reflection and feedback and correct previous poor task performance. The team starts to focus on throughput and reducing cycle time. They began to focus on optimizing the whole and reducing the cost of delay and WIP limits. Kaizen moments became more common place.

Stage 5 – Expert: They can look at each situation immediately and intuitively take appropriate action. They know what to focus on and can distinguish between important details and irrelevant ones. Chris’s teams haven’t reached this stage yet.

Chris believes that it is ok to talk about principles but teaching principles does not equate to acquiring a new skill. Kanban provides a framework where principles can be introduced. Then the process in and of itself is going to help encourage those behaviors and allow people to understand them at their own pace.

Next Chris shares the lessons learned:

1. Start by teaching practices, not principles

2. Don’t set WIP limits too low for a new team

3. Keep rules about moving tokens/cards simple

4. WIP is strongly correlated to product quality.

Finally, Chris concludes by stating that Kanban teams mature in a way consistent with the Dryfus model. Kanban is an effective tool for teaching lean principles and managing change in an organization. There multiple levels of maturity and at each level certain behavior guide your focus.

This presentation is available on InfoQ at  http://www.infoq.com/presentations/kanban-at-sep

Navigating the Rapids - Real World Lessons in Adopting Agile

Sam Newman talks about lessons learned while transitioning clients to agile practices. He 1st recalls a story where he failed to educate stakeholders upfront that he will share with them the good news and the bad news on a regular basis in hope that they can step in and help with the bad news when needed. Next he discusses how when adopting agile productivity 1st goes down as the team is adapting to the new practices. Patience is needed before there is a clear sign of productivity increase. Finally he describes how different clients prefer to see data in different ways.

He summarizes the lessons learned as:

1. Educate stakeholders about early signs of pain

2. Bite of something small – Earn trust early: Need to show that things are better (hard numbers, metrics).

3. Track your data and know how to present it

This presentation is available on InfoQ at  http://www.infoq.com/presentations/navigating-agile-rapids

The Dancing Agile Elephant: IBM Software Group’s Transition to Agile and Lean Development

At QCON San Francisco 2008, Sue McKinney gave a presentation on IBM’s agile adoption strategy. She started out by giving an overview of some of the business and operational challenges at IBM. These include innovating the business to differentiate and capture new value, heighten responsiveness and closer linkage to customers, and improved time to value. On the operations side, issues included better workload management, improved quality, improved project development cycle times, improved predictability on schedule, and making better use of resources to be more productive

Next Sue described the IBM environment. IBM has a global team, with different companies brought together by acquisition. There are 500 development teams in 5 divisions. Teams are as large as 800 or as small as 20. Very few teams are collocated. They are highly security conscious. They use many tools (due to acquisitions), and many platforms.

Sue describes how IBM software development transformation went from waterfall in the 1980s (rigid, late feedback, slow reaction to market changes) to iterative (customized RUP, community source and component reuse, emphasis on consumability), to agile (global reach, SOA, agile practices, outside-in development, tool and not rules, continuous learning and adaptive planning

Sue mentions how she sold agile as a way to deal with uncertainty, and to respond to changing markets. Agile gives them transparency to check where things are going and enables them to take a course correction if needed by meeting with stakeholders and deciding what change to make. She kept processes to deal with complexity (team size, geo distribution, compliance or regulatory issues) and move from lightweight to things that are more thorough and allow for traceability.

Next Sue lists some things to consider before getting started. These include getting management support, picking strong experienced leaders, picking the right project as a proof point (project with high visibility and some risk), providing the right education, tooling and governance, ability to allow change to occur, and keeping it simple.

Sue defined Agile as short time boxed iteration with stakeholder feedback. This creates automatic constraints (iteration length) that force the team to find defects earlier and address them. The team becomes more responsive and there is always a sense of urgency. There is also a notion of transparency with demos at the end of each iteration. The constraints forces teams to optimize and eliminate waste (automate). Share holder feedback causes us to focus on the essentials.

The initial project was a team of 200 people over 4 sites with 1 week iteration. 1st 2 months, put together infrastructure to enable the team (test case harnesses, continuous integration, static code analysis), then 1 week iterations began. The software was published for regular consumption and received regular feedback. For sustainability (220 out of 500 teams are using agile), they created a practices for distributed development:

Crafting an Agile Adoption Strategy

Amr Elssamadisy gave a talk at QCON 2008 about agile adoption strategies. He started out by defining what is valuable? Is it being agile, design, requirements, time to market, user satisfaction, meetings? Value is different based on the context. It is what you want to get out of it. It should be your goal for adopting an agile practice. This is how you will measure your success.

Next Amr talk about a learning cycle of 1) goal, 2) process, 3) test, stop, and learn, apply lessons learned, and repeat until success. He then simplifies agile software development to recognizing and responding to change (learning, requirements, team structure, new practices), feedback practices and communication practices for the team (iterations stand up meetings, retrospective), and technical practices for developers (TDD, refactoring, pair programming).

Next Amr gives examples of some business values: reduce time to market, increase quality, increase flexibility, increase product utility, increase visibility, reduce cost, increase product lifetime. This is followed with some examples of some smells: Us vs. Them, Customer asks for everything, direct input from customer unrealistic, management is surprised, bottlenecked resources, churning projects, hundreds of bugs, hardening phase needed.

Each of these problems is addressed by different set of practices or patterns. A pattern is a problem solution pair within a context.

Amr then goes through some patterns:

1. Decrease time to market: We need to apply Iterations. For that we need a product backlog and a well defined definition of done. The backlogs improve product utility and increase visibility by enabling high quality feedback as the goals are met and reviewed regularly. They decrease time to market because the prioritized list enables negotiation of scope and early release with the most valuable functionality.

• a. Backlog context: You are on a team that decides to adopt agile including iterations. You decide to go away from big upfront design. Team has needed expertise to expand and evolve requirements
• b. Backlog adoption: Customer flushes out coarse grained requirements ahead of the iteration planning meeting. At the beginning of each iteration, team should have enough info to roughly estimate and begin development. Items chosen for development make up the iteration backlog.
• c. Backlog smells: Estimation paralysis, techies take over, multiple non cross functional teams have trouble working from one backlog

2. Increase quality: implement test driven development which needs refactoring which in turn needs collective code ownership and automated developer tests (Test first development or test last development). Refactoring increases flexibility and the product lifetime by enabling and encouraging developers to change the design of the system as needed. Quality to market and costs are reduced because continuous refactoring keeps the design from degrading over time, ensuring that the code is easy to understand, maintain, and change.

• a. Refactoring context: Development team is practicing automated developer tests. Requirement is not well supported by the current design or you want to make the design cleaner.
• b. Refactoring adoption: Just do it keeping in mind that it is a practice not a tool. Start automated developer tests until you are comfortable writing tests for all of the code. Adopt team code ownership. Agree on how to handle broken tests that result from refactoring. Read Martin Fowler book on refactoring and a book on TDD that is exercise driven.
• c. Refactoring smells: don’t get carried away and refactor just for the sake of refactoring. It does not deliver direct business value. Many missed small refactoring build up over time causing the need for a large refactoring. Beware of code ownership issues and pride that might lead to “commit wars”.

This presentation is available on InfoQ at  http://www.infoq.com/presentations/adopting-agile-practices

Measuring Agile in the Enterprise: 5 Success Factors for Large Scale Agile Adoption

At Agile 2008 Michael Mah discussed Measuring Agile in the Enterprise. He starts by mentioning that Business will have to soon make a decision to reduce costs. 1) ship development to India, 2) adopt agile or 3) ship to India and adopt agile.

Michael then shares metrics (size, time, effort, defects, and productivity) collected from agile projects against industry averages. The data was mainly from BMC and Follet and looked at cost, schedule, defects, and staffing showing better results for the mature agile teams. Then Michael did a trendline assessment of staffing, schedule, effort, and defects. The data showed the agile project team size is fairly typical, bug rates are significantly lower (they start out higher but fall way down as team matures), and as a whole achieve faster speed. Productivity index (size/(time*effort)) improves the longer you’ve been doing agile, so the sooner you start the better.

Finally Michael interviews Walter Bodwell and describes the BMC secret sauce for success:

1. Buy in: need VP or higher senior executive sponsor, scrum master training, and core team energized and passionate

2. Staying releasable: have nightly builds/tests, 2 week iteration demos, frequent, rigorous code reviews

3. Dusk to dawn team work: Communication techniques for information flow, Wikis, video conferencing, periodic onsite meetings, collocated release planning, scrum of scrum meetings

4. Backlogs: One master backlog and multiple backlog management, one setup for user stories across teams, requirements architect to interface product management and R&D

5. Holding back the waterfall: Use Test Driven Development, retrospective meetings to not regress into old waterfall habits, outside source to audit the process

 This presentation is available on InfoQ at http://www.infoq.com/presentations/5-Success-Factors-Michael-Mah

Succeeding with Agile: A guide to Transitioning

Mike Cohn gave a talk at Agile 2007 about how to succeed in Agile based on his experiences with different clients transitioning and successfully transforming their organization.

Mike starts out by addressing why organizational change is hard:

1. It’s not top down where a strong leader sets a vision and people follow it, and it is not bottom up where team start and others follow until you run into groups that are powerful enough to squelch transition.

2. Best practices are tempting, but once we identify something as best, we stop changing. We have to keep inspecting and adapting otherwise we are not agile.

3. The transition has to also go with a transition in the development process

4. We cannot predict how the organization is going to respond. We want to effect the organizational change in an incremental manner.

Mike describes how if we break things up into small pieces, work on the smaller more manageable pieces and then try to put them back together we do not necessarily get the whole picture. We need to think of organization as complex adaptive system. We need to look at success as achieving a good fit with the environment instead of as closing the gap with the desired state. We have to acknowledge that agents are closing local goals and closing local gaps. They are thinking about their own job security and their own vacation schedule. It is ok to set vision, but each agent is pursuing their own local goals and they are taking local actions and they all interact.

Mike next contrasts the traditional model of change vs. the complex adaptive model for change.

1. Traditionally, behavior is controllable and predictable. In the complex model we cannot control behavior.

2. Traditionally, we can establish directions. . In the complex model direction is established by the moving groups.

3. Traditionally there is clear cause and effect. In the complex model, there is a ripple effect.

4. Traditionally, relationships are directive. In the complex model, relationships are empowering and can go in both directions.

5. Traditionally, focus is on efficiency. In the complex model we are more interested in responding to change.

6. Traditionally, decisions are based on fact and data. In the complex model, decisions are based on patterns and tensions between groups.

7. Traditionally, leaders are the experts. In the complex model, everybody can help lead. Leaders are there to facilitate and support.

Dealing with Organizational Challenges to Agile Adoption

At Qcon 2007, Joseph Pelrine addressed issues dealing with agile adoption.


He started out by describing how an organization is composed of multiple teams with some being flexible and others rigid. The flexible teams have resonance and will quickly sync up with you. Using resonance can help you bring change. However, too much resonance is a bad thing. It’s like how running down hill too fast can make you fall down. The rigid teams Don’t want to change, do not like what you are doing and do not want to move. They will resist your process. You can use this as a buffer so you do not start tripping up. However, too much resistance is not good.

Next Joseph discusses how management is still stuck in Newtonian concepts and mechanics of work where the whole is the sum of the parts. He describes how we deal with different tasks and issues by providing 4 categories based on cause and effect:

1. Simple: Cause and effect is known. Switching a light switch turns light off and on.

2. Complicated: Cause and effect is hidden but knowable. Pressing a button turns a computer on. Pressing button turns on phone.

3. Complex: Cause and effect is retrospective coherence. Cause and effect emerges and system self organizes, but cause and effect is only clear when looking back and in hindsight.

4. Chaos: No patterns of cause and effect. There is no discernable cause and effect.
Between these is a black hole of stuff we haven’t figure out yet. It will stay there until we know which category it belongs too.


Most tasks that management deal with and most deterministic coding tasks are on the side on simple or complicated (know cause and effect). The hard part are thing on the other side like figuring out what customer need and dealing with people. We’ve been taught how to work with 1st set, but are still floundering around with the other set.

Next, Joseph discusses different ways of thinking

1. Process engineering: There is order. Cause and effect is knowable. We can look it up or it or we can call up an expert. This is great for waterfall. When there is order, you can know everything in advance and because agents obey rules you can predict behavior. This is known as process engineering (Taylor).

Evolving Agile

Scott Ambler gave a talk at JAVAPOLIS 07 about Evolving Agile. He started out by describing Moore’s Adoption Curve which is like a bell curve and starts out with innovators, then early adopters, peeks with early majority, start its decline with late majority and dies down with laggards. Most new technologies/processes do not manage to cross the chasm from early adopters to early majority and never make it. The innovators are always excited about new things and are ready to jump in. The early adopters believe they can get a competitive advantage so they try it. The early majority see others succeeding so they get in. The late majority need to see similar organizations doing the same thing before they step in, while the laggards need to see case studies and solid proof before committing. In most cases, the late majority and laggards come way too late to catch up to the competition.

Next Scott provides some stats from surveys done on dr dobb (www.ddj.com).

The survey shows the about 70% have adopted agile in some way. Teams that are collocated have higher success rates than distributed teams. Smaller teams have better success rates than larger teams. The survey also shows that:

• Quality: 87% believe high quality is more important than on time and budget
• Scope: 87% believe meeting actual needs of stakeholders is more important than building to spec
• Money: 80% believe that providing the best ROI is more important than delivering under budget.
• Staff: 75% believe that having a healthy workplace is more important than delivering on time and on budget
• Schedule: 61% believe that delivering when the system is ready to be shipped is better than delivering on schedule

Next Scott moved to cover some items often skipped when talking agile.
On any project there needs to be project initiation where the team and stakeholders are determined, funding is secured, a team room is setup. This also includes initial scope modeling.
Spend some time getting initial requirements, architecture modeling. You always need to answer
What are you going to build? How much will it cost? How long is it going to take? How are you going to build it?

Other issues include release into production, pilot, documentation, training, end of phase testing, sign-off, final testing environment, data conversion. Also one needs to understand production needs (operations and support)

Test driven development is not meant to scale. One should use the right tool for the right job. When working with small issue, one can do just in time specs by writing tests followed by code to make the tests pass. When dealing with big issues then we need to draw sketches, use index cards, etc. For validation, we can run tests to ensure code still works, but this is smoke testing or confirmation testing. It is good, but not enough. It validates to our understanding of the requirements. This makes the assumption that stakeholders understand what they want and can communicate that to us. We need investigative testing. TDD tests to spec. Investigative testing is testing outside the box (key strokes, integration, usability, break the system).