4 Parts to Delivery

In a previous post, I discussed how Jez Humble made the case for the need for continuous delivery and DevOps. In this post I’ll cover the changes that need to be implemented to enable DevOps.

Below is how Jez describes the 4 parts to delivery:
1. Culture
2. Automation
3. Measurement
4. Sharing

Culture is the most important part of DevOps. We need to make sure development and Ops are not treating each other as rivals and are instead working together frequently and efficiently. Ops need to be involved early on in the development process. They need to be in the inception meetings and in the demos. They need to know what is coming down the pipeline and provide their requirements in terms of monitoring, scalability, archiving, etc… Developers also need to rotate through the operations department. They need to be on call in case of production problems. They need to feel the pain and impact of not delivering production ready applications. Incentives between the development and operations department need to be reversed. Development needs to be measure on stability and operations needs to be measured on throughput. This gives development incentive to deliver production ready applications while encouraging operations to push applications into production.

Through automation, we can reduce the cost of making changes to the application and thus enable frequent delivery of software. This enables us to make smaller incremental changes that are less risky than big bang approaches. We want to create a deployment pipeline and automate the entire delivery process from build, deploy, test to release. Provisioning of infrastructure as well as database migrations and deployments need to be also automated so that we can always test in production like environment. All this creates transparency with a complete audit trail from when a change was introduced into source control until it was released into production.

Measurements are key to continuous improvement. We need measurements for all teams.
Are we generating business value? How many new users do we have? How many new orders?
Are we producing quality products? How many bugs are there? What changes are breaking things?
Do we have the right infrastructure? How often does a failure happen? How long to restore service? From these different measures we can decide on what areas need improvement.

Techniques like CI, TDD, and refactoring that developers use to improve their code can also be used to improve the Ops environment. Having the development team sharing and working with the operations team to improve their environment further builds trust and improves the relationship between the two until gradually the border between the two disappears.

All this is a major improvement on the traditional silo in the organizations. By building trust, aligning incentives, building a deployment pipeline, and measuring our processes, we can now eliminate the hardening and stabilization phases that tend to extend project plans. As the 2 groups move closer together, done no longer means dev complete but instead means released and in use. Developers now write production ready code and deploy to production like environments from day one. The conception to production cycle gets dramatically reduced to a matter of days and the business can now receive fast feedback and plan based on real user data.

In my next post I’ll describe how this enables organizations to align themselves around products instead of projects. Stay tuned...

Adapted from  Enterprise DevOps: Breaking Down the Barriers between Development and IT Operations @ADP East 2011

Technical Practices Turning the Agile Dial to Eleven

Craig Smith and Paul King gave an experience report at Agile 2008 about technical agile practices. They started out with an initial goal of 100% of code coverage for unit test, all production code paired and test driven, minimal design up front with appreciation for when such design made sense, customer focused outcomes, full continuous integration, daily pair rotation, continuous improvements through retrospectives, high levels of automation (installers, vmware instances), and light weight metrics to monitor progress.

Using quality metrics forced team to have a common way of doing things. They tracked duplications, and enforced 0% duplication (4 lines limit). They also tracked method and class complexity & size and kept extremely simple classes and code with approximately 10 lines per method, 80 lines per class.

For testing boundaries used for 3rd party apis, they used an IDE plugin to create static boundaries, used Groovy to automatically eliminate checked exception through a language feature, and used auto boundaries of an interface

To ease the use of mock objects, they built a framework to avoid boiler plate code. They had the IDE convert code into expectations, and used annotation for instance/mock creation.

They implemented auto checking for nulls for public methods and public constructors, auto checking getters and setters, and checked for project wide rules to do with final, equals and hashcode. Other numerous types of classes had further checks for immutable, serializable, data, and domain.

They disposed of interaction based tests. The IDE could recreate them whenever refactoring was required.

They wrapped up by discussing future direction of acceptance-TDD or functional-TDD, evolving how they use TDD and CI (groovy, easyB), leveraging a broader base of testing approaches (jester, AllPairs, Theories, Grids), further using DSL, and further using Groove, or Scala.

This presentation is available on InfoQ at http://www.infoq.com/presentations/Super-Agile-Craig-Smith-and-Paul-King

Measuring Agile in the Enterprise: 5 Success Factors for Large Scale Agile Adoption

At Agile 2008 Michael Mah discussed Measuring Agile in the Enterprise. He starts by mentioning that Business will have to soon make a decision to reduce costs. 1) ship development to India, 2) adopt agile or 3) ship to India and adopt agile.

Michael then shares metrics (size, time, effort, defects, and productivity) collected from agile projects against industry averages. The data was mainly from BMC and Follet and looked at cost, schedule, defects, and staffing showing better results for the mature agile teams. Then Michael did a trendline assessment of staffing, schedule, effort, and defects. The data showed the agile project team size is fairly typical, bug rates are significantly lower (they start out higher but fall way down as team matures), and as a whole achieve faster speed. Productivity index (size/(time*effort)) improves the longer you’ve been doing agile, so the sooner you start the better.

Finally Michael interviews Walter Bodwell and describes the BMC secret sauce for success:

1. Buy in: need VP or higher senior executive sponsor, scrum master training, and core team energized and passionate

2. Staying releasable: have nightly builds/tests, 2 week iteration demos, frequent, rigorous code reviews

3. Dusk to dawn team work: Communication techniques for information flow, Wikis, video conferencing, periodic onsite meetings, collocated release planning, scrum of scrum meetings

4. Backlogs: One master backlog and multiple backlog management, one setup for user stories across teams, requirements architect to interface product management and R&D

5. Holding back the waterfall: Use Test Driven Development, retrospective meetings to not regress into old waterfall habits, outside source to audit the process

 This presentation is available on InfoQ at http://www.infoq.com/presentations/5-Success-Factors-Michael-Mah

Fostering Software Craftsmanship in a Corporate Setting

Scott Dillman gave a talk at Agile 2008 about fostering software craftsmanship in a corporate setting. Scott defined software craftsmanship as taking responsibility, continuous learning, rejecting specialization, pride in quality work, passing on knowledge, and meeting professional standards (test driven development, continuous integration, coding patterns and practices).

Next Scott described 3 steps to evolve a novice software developer into a craftsman.

1. Evaluate the current state of the organization through interviews, surveys and metrics. Surveys can be reused at a later date to evaluate progress. Collecting metrics from automated builds like % passing builds, builds/day, build duration, fail build reasons, code coverage, code complexity. Track historical trends.

2. Educate via pair programming, centralized resources (wiki, educational content, knowledge base, moderated technical forum, book reports, tools and code repository), educational sessions(teasers, lightning sessions, tech talks, workshops), craftsmanship day, 10% time and on team mentoring. To get upper management buy in, highlight benefits, cite case studies, link to corporate goals. Upper management has to allocate time and recognize and reward participation.

3. Measure success by interviews, surveys, metrics, and performance reviews. Make sure reviews have goals that motivate and educate, are relevant, specific, actionable, and verifiable. Have the right people perform the reviews, do 360 reviews, and encourage and reward participation.

This presentation is available on InfoQ at http://www.infoq.com/presentations/Craftsmanship-Scott-Dillman

Prioritizing Product Backlog

At Agile 2008, Mike Cohn gave a talk about product backlog prioritization. Mike starts out by discussing the right level of prioritizing. A well formed product backlog should take the form of an iceberg. Small on the top (sprints), a little bigger down the middle (releases), and much bigger down the bottom (future releases). Mike believes that about 10% of current iteration should involve planning and prioritizing for the next iteration.

He then describes 3 levels of user stories. A user story which is a description of desired functionality told from the perspective of user or client, epics which are very large user stories and themes which are a collection of related user stories. An example of an epic might be as VP of marketing, I want to select type of campaign (direct mail, tv, radio, etc) so that I can identify and repeat profitable ones. This can then move up the iceberg and become as VP of marketing, I want to see direct mailings when reviewing historical campaigns. Mike recommends prioritizing at the theme or epic level. It is a lot of work to prioritize at the user story level and not very meaningful.

Mike then covers 4 techniques for prioritization:

1. Kano analysis:

Kano identifies 3 types of features to measure customer satisfaction:

• 1. Exciter or delighter: Exciter - feature you did not know you wanted until you saw it.
• 2. Mandatory or baseline: must be present for users to be satisfied
• 3. Linear: the more of it the better

Agile Estimating and PlanningAgile Estimating and PlanningThis can be done by expert opinion or surveying a small group of users (20 to 30) and asking both functional and dysfunctional questions like how do you feel if a feature is included and how do you feel if a feature is absent. The available answers are standard: I like it, I expect it, I’m neutral, I can live with it, and I dislike it. The results can then be matched to the Kano matrix to find out if a feature is an exciter, mandatory or linear. Others might fall in a questionable, reverse, or indifferent category. Mike recommends prioritizing by including all mandatory, some linear and leaving room for a few exciters.



2. Theme screening:

Identify 5 to 9 selection criteria for what is important in the next release (Importance to existing customers, competiveness, generating revenue, etc…). Identify a baseline theme that is likely to be in the next release and one that everybody understands (not the best one). For each theme, score the theme based on each selection criteria and compare it to the baseline theme using +1, 0, or -1. Add up the results for each theme and then rank them. Open the highest ranking themes and look at individual stories and use expert judgment on them.

Agile and Beyond – The Power of Aspirational Teams

At Agile 2008, Tim gave a talk on appreciative inquiry. Tim starts by defining appreciative inquiry as the study and exploration of what gives life to human systems when they function at their best. It suggests strengths, hopes and successes are transformational. We control a lot more of our destiny than we are aware of. Agile is similar. It is not just about process and practices but is a belief in achievement. Software development always brings up bad news. We need to think of what we do good in software.

Next Tim asks who defends the honor of the agile manifesto and principles. It’s easy to just adopt some of the principles and skip others. It is easy to assume that we cannot do all the practices and then not aspire for more. The best teams adopt all practices because they know that all the practices are dependent on each other.

Tim then covers several areas to influence appreciative inquire:

1. Working together: Pulling the load together, having fun, being excited about achieving.

2. Project chartering: It helps team understand the parameters you work in like defining success and determining who is part of the community. It also helps self organize over a common vision and goal and forms a contract that binds together the rules of the teams/practices.

3. Group sharing: encourages new ideas, gets more people involved in work.

4. Story cards and planning boards. Puts focus on collaboration. Use magnetic boards, make it fun, try magnetic avatars. Also, instead of statuses of in development, in test, in QA, use started, in progress, done which encompasses the different roles. Story cards are placeholders for a bigger conversation. Important to follow format of Story cards “as role, I want, so that”. Also get good pens to encourage good card writing (dark and thick ). Physical cards much better than huddling over a spreadsheet. If not hand written, print them out.

5. Visualize progress - Burn down chart vs. burn up chart. In general we like to see thing go up kind like a profit graph. For forecasting, simply counting story cards done vs. story cards not complete can be enough. Graph can still be drawn manually.

6. Gold cards: give 2 per developers to use for working on areas where they can innovate.

10 Ways to Screw Up With Scrum and XP

At Agile 2008, Henrik Kniberg gave a presentation on 10 ways to screw up despite Scrum and XP

1. Believing the Hype: That is, believing that if you go agile, all problems will disappear. You throw out everything you are doing now even if it works and then get the latest and best tools and focus on perfection.

2. Definition of Done: But you said you were done? Not having a definition, or not obeying the definition, or having done be outside of the teams control (no access to production). Good examples of Done are: unit and integration tested, acceptance tested and deployed to demo site. Or Releasable (acceptance tested, release notes written) and no increased technical cost.

3. Velocity: Not having one makes it hard to create a release plan. Having one and misusing it by comparing different team velocities. Or having a yo-yo velocity where it is high one sprint and very low on another sprint (indicates spending a lot of time fixing bugs).

4. Retrospectives: We are very busy so we skip it. And if we do it, changes and recommendations are ignored and not implemented. Having unwanted people in the meeting where the team feels uncomfortable and do not participate. Punishing the team for bad changes (the whole point is to try something and later evaluate it to see if things improved).

5. Team commitment: Team is pressured, Team is not sitting together, team does not track and learn and keeps repeating the same mistakes. Team is always undercommiting (team always looks good) or always over committing or has a velocity of 0 (doing a lot of stuff but not finishing stories to the end).

6. Technical Debt: No time to write unit tests or refactor code. This results in duplicate code, unreadable code, lack of code coverage and all add to technical debt. Technical debt slows team down. Things become harder. This also leads to inappropriate solutions like a big bang re-write or fixing the product instead of the process.

7. Teamwork: Fixed roles (no one helping anyone else), personal backlogs (results with attitude of at least I finished my stuff!), not helping each others, personal incentive models, implementing all stories in parallel, management interference (team not self managing).

8. Product backlog/product owner and customer: Product owner does not have time to maintain product backlog so team does not have one or it is not prioritized. Product owner does not know the system or does not have the authority and thus cannot make decisions. Having multiple product owners that do not agree with each other. Product owner becoming a bottleneck where no one can do something unless they talk to the Product owner 1st. Product owner not always available.

9. Mergophobia: Fear of merging code. Not integrating early and often, No branching policies, not taking responsibility (everybody ignoring team rules regarding code checkin),

10. Spring backlog or taskboard: Does not exist, too far from the team (not easily accessible), too complicated (many states to keep track off ) result in it not being used during daily scrum, not owned by the team, no burndown, not updated daily, warning signs ignored.

11. Worry about 1 – 10. Problems are normal. Don’t panic. Don’t despair. Tackle them gradually, prioritize and try to improve sprint after sprint.

This presentation is available on InfoQ at http://www.infoq.com/presentations/Fail-Scrum-Henrik-Kniberg

Future Directions for Agile

At Agile 2008 David Anderson talks about agile principles, practices and the community. David covers multiple topics that reveal how CMMI, Kanban and Real Options Theory help in making organizations scale, lean, and of high maturity.

David mentions that trying to scale agile across an enterprise is forcing a change on the community. We are trying to expand agile to the entire IT department and even more to other areas like marketing and recruiting. We need a wider definition of agile that is not narrowly focused on software development. We have the manifesto and the principles, but it is 7 years old and in badly need of an update. David stresses that we should not underestimate the value of what the manifesto gave us, but it is not everything of what it means to be agile. David controversially states that the agile manifesto is superstitiously declared.

Next David clears up the ideas behind the manifesto and how can we capture them as a model in 3 underlying paradigms

1. High trust, high social capital, highly collaborative culture

David describes the different cultures between the traditional and agile communities. The traditional community was rooted in government and developed a body of knowledge from defense. The high cost of failure (money, lives, politics) makes people conservatives, risk averse, and have low trust. On the other hand, the agile community was started by thought leaders from start-up companies building web sites. They were innovating daily, empowering people to release new functionality with an assumed zero cost of failure. They realized that in order to innovate at a fast pace, there was no time to write requirement and negotiate contracts. It was better to let people work and adopt a failure tolerant attitude. This was a highly collaborative approach with a high trust liberal culture. This led to the 3rd value of the agile manifesto, customer collaboration (high trust) over contract negotiation (low trust). Self organization or empowering, delegating and tolerating failure through adoption of high trust culture is better than contract and audits in low trust culture.

2. Perfect is the enemy of good enough.

David mentions that CMMI taught us that project success came from organizational maturity. Level 1 is chaos, Level 2 is repeatable and Level 3 is a defined process. At this level we should be able to deliver successfully every time. So the assumption was that we need to pursue perfection and eliminate ambiguity and uncertainty. We need accurate plans, analysis, estimates, and diagramming. By the time we are done, they are already out of date. This led to the agile manifesto value of working software over comprehensive documentation.

“We Suck Less” is Not Enough

At Agile 2008, David Douglas and Robin Dymond described the current state of agile adoption and discussed what the legacy of agile will be. They mentioned Forester research from 2007 that says 17% of North American and European companies and are now using agile. They however noted that there is a lot of confusion on what agile really is and described the Nokia test to find out if a team is agile:

1. Iteration must be time boxed to less than 4 weeks

2. SW features must be tested and working at end of each iteration

3. Iteration must start before spec is complete

4. Do you know who the product owner is

5. There is a product backlog prioritized by business value

6. Product backlog has estimates done by team

7. Team knows there velocity

8. There are no managers disrupting the work

They then warned that the attitude of we suck less now than before sets the bar really low. The business will be satisfied and will stop when things get a little better than before, but they are not even close to how much better they can get. Moving from ok to great is a journey that takes time and effort. There are levels of agility that companies can get to based on their level of commitment.

Next they mentioned that there a perception that agile is just a set of tools and practices like Rapid Application Development or that it is cheap, easy, and runs on its own, or that it is a project solution instead of a system solution. Agile is an entire work system. It can produce 5x+ productivity gains, requires organizational redesign, requires significant change management component. We as agilest are currently feeding this perception by training teams on techniques and consciously encouraging growth over quality.



Next, they describe several large organizations that adopted agile and then moved away from it. There were early warning signs of agile adoption stagnation like early declaration of success, increased time slicing and borrowing developers for “important” project, lack of commitment or investment, flight exodus of change agents, executive support waning, continued emphasis on cost reduction without the corresponding ROI and continual disconnect between IT and business and a lack of understanding on how IT delivers value to the business.

The Development of a New Car at Toyota

Kenji Hiranabe gave a presentation at Agile 2008 about Lean at Toyota. He talks about Nobuaki Katayama, former Chief Engineer at Toyota, and how they build new cars. There are 3 phases to car building:

1. Planning and concept development: concept, style, market research, pre-development, cost and profit target. Spend a lot of time in this phase. Start out with a good concept, then visit dealers and users and get feedback to see if this concept work and then redo. The next 2 phases are management of milestones.

2. Real car development: Design prototyping, evaluation

3. Production and sales:

Managing big projects:

1. Planning and concept management: project backbone, clear appealing points are needed, 1st concept and plan has to be refined until satisfaction, discuss multiple opinions from wide views, rules to avoid immature “go”, calm evaluation with 3rd person

2. Development process management: needs to be a strong dedicated leader with org that supports him, with clear milestones, and supporting people to achieve them, rules to avoid proceeding before achieving them, mechanics to avoid “too late”, trigger of 3rd person, honest communication (face to face), bad news 1st rule, early and reliable backup plan.

3. Resource management: people and money, work hard does not work, visualization estimate based on actual record, make projects slim. Cost: Visualize the target cost, leader owns buffer, assign target cost based on actual record, value sense of completion, avoid sense of being “forced to do”, plan in detail and follow up and adjust seeing the whole

4. Cost management: Reduce cost means save profit so that the company can grow, everyone and every division think about global competition and cost reduction. Culture of do not complain but cooperate. Analyze past and present cost. Use it as base cost data. Reduction of cost is accumulated step by step by everybody. Value engineering is the technical competence. Each team is responsible for reducing cost.

5. Benchmarking: it’s about knowing competitors strategy. Not copy others techniques, but focus of parts that are better than ours. Set performance targets including ahead and behind time difference. Study designs of 2 to 3 years ago and develop product 1 to 2 years away. Use it for continuous competiveness.

6. Leader attitude: Leadership, teamwork, macroview, passion are good. Dictator, one man shop, leave all to subordinates, charismatic are bad for leadership.


Kenji then contrasts “do it until it’s done” vs. “do it until the time”. Mr. Katayama explained that for phase 1 use do it until it is done and for phases 2 and 3, do it until time.

Kenji wraps up by mentioning the Nobuaki Katayama values consistency and integrity over passion and strong leadership, values engineering thinking, and tries to understand new ideas.

This presentation is available on InfoQ at http://www.infoq.com/presentations/Toyota-Kenji-Hiranabe