Command and Control vs. Agile

In her book, Coaching Agile Teams, Lyssa Adkins nicely maps out the transition in beliefs that traditional project managers have to go through when adopting agile.


Lyssa explains that the command and control project management beliefs on the left need to be replaced with the agile beliefs on the right:

Project management belief	Replaced with
We can plan the work and work the plans.	Planning is essential, plans are useless.
The triple constraints can be traded off for one another to correct for unknowns.	Time and budget are held constant. Only scope flexes.
The plan gets more accurate over time as we flush out the project through phases of activity: requirements, design, testing, and so on.	The plan gets more accurate over time because it is constantly reviewed and tuned up based on the teams actual performance.
Delivering on time, within budget and on scope equals success.	Clients getting the business value they need is the only value of success.
Scope can be locked down with later discoveries being handled as change requests against the schedule end date.	Scope remains flexible and changes of any kind are welcomed even late in the schedule.
Controlling through the project plan is my job.	Controlling through a plan is not possible; releasing the team into the safety of agile is my only measure of control. So I coach the team to use agile well.
Completing tasks and delivering deliverables indicate progress and value delivered.	Only delivered end products indicate progress and value delivered.

The Agile Manifesto – 10 Years Later

At the 2011 Agile Development Practices West, Martin Fowler reflected on where we are in the agile movement 10 years after the creation of the Agile manifesto. Martin looked at what was going on back in 2000. We were in a world of badly managed software. A group of people felt that to clean up this mess we needed big methodology, planned driven methodology, or an engineering approach to software development with lots of documents and lots of processes (waterfall).

Others had success with a different approach. They believed in rapid iteration, lots of collaboration, and an evolutionary approach to requirements, design, and architecture. They felt that they were not getting heard and were getting trampled by big methodology approach.

This group got together in Snowbird 10 years ago. The idea for the meeting originated a year earlier when the group got together at a workshop at Kent Beck’s home. Kent described XP as a set of practices governed by a set of principles and values. However, there were some that liked the values or principles, but did not like the practices.

This led to the Snowbird meeting where they concentrated on the values and discussed the different approaches like XP, SCRUM and Feature Driven Development. Uncle Bob felt the group needed a manifesto as a statement of what they believed in. In the end, the people that wrote the manifesto were just the people that were free to meet up that week.

Now 10 years later, a lot of the people that where there at the creation feel a sense of unhappiness with Agile and the manifesto.

Martin believes there are 2 main reasons for this.

The 1st is due to the inevitable consequence of success. The idea of Agile spreads out but the true meaning gets diffused. He refers to this as semantic diffusion. Anything that is successful spreads faster than the semantics can follow it. Martin stresses that we have to keep reminding people about the core concepts.

The 2nd reason is that the Agile revolution is still in its infancy. We are in the early days of a very long running change. Martin remembers the OO revolution. It started in the late 60s, it was put together in the 70s, had a good definition of what OO is in the 80s with small talk and took another 20 years or so before OO became main stream in languages like java, C++ and C#. Still today, 40 years on, people think they are doing OO, but they really just have procedures and naked data structures and not true OO.

Martin thinks Agile will take even longer than OO because it affects more people. It’s not just developers, but it is testers, BAs, managers, etc…

Martin then argues that people should still care about agile. He remembers how the manifesto was written.

The group came up with 8 valuable useful good things in software development. The idea was to arrange them in pairs so that one useful thing was more valuable than the other. The guiding light was the drive to the engineering methodologies. Flipping the values around was the value system of the engineering methodologists.

A lot of companies care about processes and tools over individuals because they want to be able to move people around.


The design and documentation is important to them because they believe that once it is produced they can give it to a bunch of developers that can code it up.

You have to have contracts in place and decided. The conversations are based on what is in the contract.

Following a plan was important because the definition of success was based on whether they stuck to the plan.

Martin argues that if you don’t care about Agile then you can be equally happy in a flipped or agile environment. The order within each pair of values does not matter.

Martin wraps up by looking to the future. He sees 2 emerging areas for Agile. The 1st is Agile User interface design where the UI evolves through customer, designer and developer collaboration instead of handing the UI to a team and then asking the developers to code what was designed (even if the developers see much simpler and faster solutions).

The second is the concept of continuous delivery and improving the process of how we move code from development to test to production.

This presentation is available on youtube at http://www.youtube.com/watch?v=p5Qj75nJPEs (minute 50)

The Semmelwies Reflex

At the 2011 Agile Development Practices West conference, Linda Rising gave a keynote address entitled “Deception and Estimating, how we fool ourselves”. In the presentation, Linda referred to the The Semmelwies Reflex which is a metaphor for the reflex-like rejection of new knowledge because it contradicts entrenched norms, beliefs or paradigms. It is based on the theory that once scientists have crafted a hypothesis they are reluctant to let go.

Linda quotes Max Plank:

“A new scientific truth does not triumph by convincing its opponents and making them see the light, but rather because its opponents eventually die, and a new generation grows up that is familiar with it”.

Linda then explains that we have to wait for the old ideas to die, physically die, and she amusingly wonders “How many waterfall people must pass on before Agile takes over!!!”

Non-Determinism and Testing

At the keynote address of the 2011 Agile Development Practices West, Martin Fowler gave a talk on non- deterministic tests. These are tests that when you run them they pass, but if you run them again (without changing any code), they fail. They are also referred to as flaky tests or intermittent failures. The tests seem to pass or fail in a random fashion.

Martin asserts that non-deterministic tests are useless. The whole point of a regression suite is to be a bug detection mechanism. Its value is that you get immediate feedback when you make a mistake. The time between making a mistake and realizing that you did is very short. And because it is short, you can quickly figure out what you did wrong and fix it. With non-deterministic tests you get unreliable information.

Martin argues that non-deterministic tests are worse that useless. They are very dangerous. They are like an infection that infects the entire suite. One failing test will mark the entire suite as red. Usually we will dig in to see which test failed and fix it, but once we start seeing red regularly, we will assume the failure is due to the non-deterministic test and ignore the failure of the entire suite. This makes the entire suite flaky and the entire test suite becomes useless.

Martin purposes setting up a quarantine area for flaky tests. If we pull the tests from the suite, then the suite will remain reliable. The number of tests in quarantine need to be limited and fixed as soon as possible. Martin recommends setting some limits like no more than 3 tests can be in quarantine or a test in quarantine should be fixed within a week.

Martin next describes some causes for non-deterministic tests.

1. Lack of isolation: Tests depend on the order that are run. These are very hard to debug because when a test fails because of the order it ran in the suite, you usually re-run in isolation and then it passes. The real cause of the problem is in another test which is green and very hard to locate. There are two possible solutions:

• Track dependencies: Track the order in which the tests need to run. This is very hard to manage and maintain.
• Isolation: Martin prefers this solution. You can apply a clean-up strategy. That is, make sure there is a tear down that destroys anything you have created. An even better approach is to use a blank slate. Always start with a clean data set. The disadvantage here is that setting up the blank slate can be time consuming.

2. Asynchrony: Testing asynchrony is difficult. A common approach is to use what Martin calls a bare sleep where we introduce a sleep statement.

//pseudo-code
someFunction
sleep(aWhile)
assert(answer)

If we sleep too long, the test will run very slowly. If we reduce the sleep, then when we move to a different machine or if the machine is overloaded, then the tests will fail, so coming up with the correct number to sleep is very hard. Martin warns us against using this approach and recommends another solution using polling:

//pseudo-code
someFunction
timeLimit=Time.now + timeout
while(!answer.recieved) {
  if (Time.now > waitLimit) {
    throw new TestTimeoutException
  }
  sleep (pollingInterval)
}
assert(answer)

Here we have the sleep inside the polling and a timeout interval. You can keep the sleep interval very low and adjust the timeout. You have two times to play with (global constants).

Martin also discusses another approach using callbacks

//pseudo-code
someFunction
callback = function
{ assert(answer) }
answer.ready(callback)
asserts.add(callback, timeout)

The callback method itself contains the verification. This solution depends on the testing framework that you are using.


3. Interaction with remote services: There are many things that can go wrong when dealing with a remote service that have nothing to do with your code (network down or slow, limited availability, unstable test data).

Martin recommends using a a test double. This way, you control the data the service has access to, you can control changes, and ensure fast connections. However, some don’t believe this is testing the real service because we cannot keep the double consistent with the remote service. To solve this problem, Martin recommends that we add an integration contract test that does not run as part of the main build. This test will probably run nightly against the actual service and ensures that the signature of the double is in synch with the actual service.

Martin concludes by warning us that flaky tests are a common problem that can become very dangerous if not addressed immediately.

This presentation is available on youtube at http://www.youtube.com/watch?v=p5Qj75nJPEs

Martin also discusses non-deterministic tests on his blog http://martinfowler.com/articles/nonDeterminism.html

Software Craftsmanship - It's an Imperative

Thanks to all that attended my presentation on Software Craftsmanship.

Below are the slides and notes from the presentation.

Software Craftsmanship, Beyond the Hype

Corey Haines gave a talk about software craftsmanship at QCON London 2010.Corey describes how in December 2008, a group of aspiring software craftsmen got together and tried to solve some problems they were facing. They discussed how to become better and how to bring new people into software careers. They came up with a statement of things that they believe in and crafted the manifesto of Software Craftsmanship which states:

• Not only working software but also well crafted software

• Not only responding to change but also steadily adding value

• Not only individuals and interactions but also a community of professionals

• Not only customer collaboration, but also productive partnerships

Corey’s talk focuses on well crafted software and a community of professionals.There are a lot of schools of software craftsmanship that all revolve around the principles of the manifesto. Corey expands on the thoughts of the Chicago school of software craftsmanship. They felt that we are heading in a bad direction. The state of software development was going downhill. People came up with idea on developing quality software. They came up with Scrum. However, Scrum did not prescribe any practices. Ken Schwaber said that we made a fundamental assumption that was wrong. The assumption was that developers were smart enough to come up with their own practices. However, developers had spent their careers working in large cycles. They were used to it. They were accustomed to waiting for 9 months before coding or before QA and now they had to suddenly figure out how to do things in 30 days or even in 2 weeks.

XP had the practices that support the idea of short feedback loops. However, XP is hard. TDD is hard. When you introduce TDD, productivity goes down. Overtime, it will go back up again, but it is hard to get started.

Corey states that Agile is about developing quality software. We need something that focuses on developing quality software developers. This is what software craftsmanship is. We need to give people the tools to be better.

But what are these tools? And what does it mean to be better? This depends on each individual. Don’t let somebody else tell you what it means. You have to figure it out for yourself. Find out what you need to improve on to produce better software. Picture your ideal. Picture your reality when under pressure. The difference between the two is a measure of how much your code sucks. Use this to see how you are improving.

Corey continues by explaining how do we know what those ideals are? This is where we each have to go through the stages of being an apprentice, a journeyman and eventually a craftsman.

Apprentice is going in and finding somebody who is willing to take you under their wing and show you the way. You can spend time learning effective ways of developing software (TDD, evolutionary design, capturing requirements). You work to learn those ways. You internalize a set of practices that are time tested and proven. You do this by working on software. Not by sitting in a classroom.

Once you have those practices, then you look out for other people out there that are doing it differently. See how you can integrate them. This is the concept of journeyman. It is not about travelling the world. It is about participating in user groups, reading books, observing others, integrating what you have learned and moving forward.

Corey wraps up by describing the latest movements that resulted from the manifesto and how it made it easier for people to roll out some ideas on how to practice.

• Code Katas: Dave Thomas from pragmatic programmers came up with code katas, which were small problems to solve. Uncle Bob switched that to practice on a solution instead. He related it to martial arts and how you repeat small motions and practice them until they become natural reflexes. When you are in this situation, this is what you do. An example might be practicing String replacement using regex. Do it over and over again until it becomes a reflex. The site Katacast.com has various screencasts known as katacast that show folks practicing a small kata.
• Conferences: The Software craftsmanship conferences were established and 2 are held each year, one in the UK and one in the US.
• Code retreats: Retreats started worldwide. This is where developers get together on a Saturday for a full day of practice. They work on Conway’s game of life using a technique known as “TDD as if you meant it” which focuses on TDD being all about evolutionary design. Pair-up, work on it for 45 minutes. Then delete your code, swap pairs and do it again.
• User groups: Several Software Craftsmanship User groups started.
• Craftsman Swaps: A couple of companies conducted craftsman swaps. This is where 2 companies swap an employee for a week. The employees learn the practices of another company and come back and try to improve their own environment.
• Craftsman Journeys: Similar to a craftsman swap, this is where you just go to a company for a week and learn what they do. So instead of going to a conference, a company will give you time off to go and work and learn what others are doing.
• Craftsman spikes: These are side projects that you can use to practice craftsmanship. Companies offers employees 20% time to work on these side projects.

This presentation is now available on infoQ at http://www.infoq.com/presentations/Software-Craftsmanship-Beyond-The-Hype

Bold Statements

During the key note address at the 2010 Agile DoD conference, Scott Ambler made some bold statements. Below is a recap of some of his most interesting points.

There is overwhelming evidence that defining requirements in detail upfront is phenomenally bad practice, unethical at best in the IT world.

A changed requirement late in development cycle is a competitive advantage provided you can act on it.

Change management is really about change prevention. People that follow change management are worried about scope creep and are trying to prevent it. These people fundamentally do not understand what their jobs are as IT professionals. When you prevent requirements from changing, then yes you might be building something to spec, but you are now building something that people do not want and you are charging them for it. That is not ethical.

The best way for communication is face to face communication. If you want to increase the risk on your IT projects, write documentation and create a detailed specification.

Repeatable processes are bureaucracy. What people really want are repeatable results. They want high quality. They want systems that meet their needs in a timely matter. Delivering repeatable results requires discipline.

Agile is highly disciplined. You have to distinguish between bureaucracy and discipline. Bureaucracy is not discipline. Filling up paperwork, documenting requirements, documenting detailed architecture, reviewing them, and getting people to sign off on them, all this is not discipline. This is bureaucracy.

Real disciplined is producing high quality software every few weeks. It is refactoring and cleaning up your code as you go. It is about professionals validating their own work to the best of their ability. Don’t just throw stuff over to QA.

Agile teams are self organizing generalists. Like the military, guys on the ground are allowed to make decisions and adjust. Specialists are risky. If the specialists are gone, everybody is in trouble. We should avoid hand-offs between specialists.

Agile teams are easier to manage than traditional teams. In agile world, you cannot hide anymore, you have to produce.

There is nothing new about Agile. Most ideas of Agile can be traced back to the 60s and 70s. Agilists focus on the things that work and avoid the things that do not work.

There is nothing special about you. Do not come up with excuses for not adopting agile.