The Semmelwies Reflex

At the 2011 Agile Development Practices West conference, Linda Rising gave a keynote address entitled “Deception and Estimating, how we fool ourselves”. In the presentation, Linda referred to the The Semmelwies Reflex which is a metaphor for the reflex-like rejection of new knowledge because it contradicts entrenched norms, beliefs or paradigms. It is based on the theory that once scientists have crafted a hypothesis they are reluctant to let go.

Linda quotes Max Plank:

“A new scientific truth does not triumph by convincing its opponents and making them see the light, but rather because its opponents eventually die, and a new generation grows up that is familiar with it”.

Linda then explains that we have to wait for the old ideas to die, physically die, and she amusingly wonders “How many waterfall people must pass on before Agile takes over!!!”

Non-Determinism and Testing

At the keynote address of the 2011 Agile Development Practices West, Martin Fowler gave a talk on non- deterministic tests. These are tests that when you run them they pass, but if you run them again (without changing any code), they fail. They are also referred to as flaky tests or intermittent failures. The tests seem to pass or fail in a random fashion.

Martin asserts that non-deterministic tests are useless. The whole point of a regression suite is to be a bug detection mechanism. Its value is that you get immediate feedback when you make a mistake. The time between making a mistake and realizing that you did is very short. And because it is short, you can quickly figure out what you did wrong and fix it. With non-deterministic tests you get unreliable information.

Martin argues that non-deterministic tests are worse that useless. They are very dangerous. They are like an infection that infects the entire suite. One failing test will mark the entire suite as red. Usually we will dig in to see which test failed and fix it, but once we start seeing red regularly, we will assume the failure is due to the non-deterministic test and ignore the failure of the entire suite. This makes the entire suite flaky and the entire test suite becomes useless.

Martin purposes setting up a quarantine area for flaky tests. If we pull the tests from the suite, then the suite will remain reliable. The number of tests in quarantine need to be limited and fixed as soon as possible. Martin recommends setting some limits like no more than 3 tests can be in quarantine or a test in quarantine should be fixed within a week.

Martin next describes some causes for non-deterministic tests.

1. Lack of isolation: Tests depend on the order that are run. These are very hard to debug because when a test fails because of the order it ran in the suite, you usually re-run in isolation and then it passes. The real cause of the problem is in another test which is green and very hard to locate. There are two possible solutions:

• Track dependencies: Track the order in which the tests need to run. This is very hard to manage and maintain.
• Isolation: Martin prefers this solution. You can apply a clean-up strategy. That is, make sure there is a tear down that destroys anything you have created. An even better approach is to use a blank slate. Always start with a clean data set. The disadvantage here is that setting up the blank slate can be time consuming.

2. Asynchrony: Testing asynchrony is difficult. A common approach is to use what Martin calls a bare sleep where we introduce a sleep statement.

//pseudo-code
someFunction
sleep(aWhile)
assert(answer)

If we sleep too long, the test will run very slowly. If we reduce the sleep, then when we move to a different machine or if the machine is overloaded, then the tests will fail, so coming up with the correct number to sleep is very hard. Martin warns us against using this approach and recommends another solution using polling:

//pseudo-code
someFunction
timeLimit=Time.now + timeout
while(!answer.recieved) {
  if (Time.now > waitLimit) {
    throw new TestTimeoutException
  }
  sleep (pollingInterval)
}
assert(answer)

Here we have the sleep inside the polling and a timeout interval. You can keep the sleep interval very low and adjust the timeout. You have two times to play with (global constants).

Martin also discusses another approach using callbacks

//pseudo-code
someFunction
callback = function
{ assert(answer) }
answer.ready(callback)
asserts.add(callback, timeout)

The callback method itself contains the verification. This solution depends on the testing framework that you are using.


3. Interaction with remote services: There are many things that can go wrong when dealing with a remote service that have nothing to do with your code (network down or slow, limited availability, unstable test data).

Martin recommends using a a test double. This way, you control the data the service has access to, you can control changes, and ensure fast connections. However, some don’t believe this is testing the real service because we cannot keep the double consistent with the remote service. To solve this problem, Martin recommends that we add an integration contract test that does not run as part of the main build. This test will probably run nightly against the actual service and ensures that the signature of the double is in synch with the actual service.

Martin concludes by warning us that flaky tests are a common problem that can become very dangerous if not addressed immediately.

This presentation is available on youtube at http://www.youtube.com/watch?v=p5Qj75nJPEs

Martin also discusses non-deterministic tests on his blog http://martinfowler.com/articles/nonDeterminism.html

Software Craftsmanship - It's an Imperative

Thanks to all that attended my presentation on Software Craftsmanship.

Below are the slides and notes from the presentation.

Software Craftsmanship, Beyond the Hype

Corey Haines gave a talk about software craftsmanship at QCON London 2010.Corey describes how in December 2008, a group of aspiring software craftsmen got together and tried to solve some problems they were facing. They discussed how to become better and how to bring new people into software careers. They came up with a statement of things that they believe in and crafted the manifesto of Software Craftsmanship which states:

• Not only working software but also well crafted software

• Not only responding to change but also steadily adding value

• Not only individuals and interactions but also a community of professionals

• Not only customer collaboration, but also productive partnerships

Corey’s talk focuses on well crafted software and a community of professionals.There are a lot of schools of software craftsmanship that all revolve around the principles of the manifesto. Corey expands on the thoughts of the Chicago school of software craftsmanship. They felt that we are heading in a bad direction. The state of software development was going downhill. People came up with idea on developing quality software. They came up with Scrum. However, Scrum did not prescribe any practices. Ken Schwaber said that we made a fundamental assumption that was wrong. The assumption was that developers were smart enough to come up with their own practices. However, developers had spent their careers working in large cycles. They were used to it. They were accustomed to waiting for 9 months before coding or before QA and now they had to suddenly figure out how to do things in 30 days or even in 2 weeks.

XP had the practices that support the idea of short feedback loops. However, XP is hard. TDD is hard. When you introduce TDD, productivity goes down. Overtime, it will go back up again, but it is hard to get started.

Corey states that Agile is about developing quality software. We need something that focuses on developing quality software developers. This is what software craftsmanship is. We need to give people the tools to be better.

But what are these tools? And what does it mean to be better? This depends on each individual. Don’t let somebody else tell you what it means. You have to figure it out for yourself. Find out what you need to improve on to produce better software. Picture your ideal. Picture your reality when under pressure. The difference between the two is a measure of how much your code sucks. Use this to see how you are improving.

Corey continues by explaining how do we know what those ideals are? This is where we each have to go through the stages of being an apprentice, a journeyman and eventually a craftsman.

Apprentice is going in and finding somebody who is willing to take you under their wing and show you the way. You can spend time learning effective ways of developing software (TDD, evolutionary design, capturing requirements). You work to learn those ways. You internalize a set of practices that are time tested and proven. You do this by working on software. Not by sitting in a classroom.

Once you have those practices, then you look out for other people out there that are doing it differently. See how you can integrate them. This is the concept of journeyman. It is not about travelling the world. It is about participating in user groups, reading books, observing others, integrating what you have learned and moving forward.

Corey wraps up by describing the latest movements that resulted from the manifesto and how it made it easier for people to roll out some ideas on how to practice.

• Code Katas: Dave Thomas from pragmatic programmers came up with code katas, which were small problems to solve. Uncle Bob switched that to practice on a solution instead. He related it to martial arts and how you repeat small motions and practice them until they become natural reflexes. When you are in this situation, this is what you do. An example might be practicing String replacement using regex. Do it over and over again until it becomes a reflex. The site Katacast.com has various screencasts known as katacast that show folks practicing a small kata.
• Conferences: The Software craftsmanship conferences were established and 2 are held each year, one in the UK and one in the US.
• Code retreats: Retreats started worldwide. This is where developers get together on a Saturday for a full day of practice. They work on Conway’s game of life using a technique known as “TDD as if you meant it” which focuses on TDD being all about evolutionary design. Pair-up, work on it for 45 minutes. Then delete your code, swap pairs and do it again.
• User groups: Several Software Craftsmanship User groups started.
• Craftsman Swaps: A couple of companies conducted craftsman swaps. This is where 2 companies swap an employee for a week. The employees learn the practices of another company and come back and try to improve their own environment.
• Craftsman Journeys: Similar to a craftsman swap, this is where you just go to a company for a week and learn what they do. So instead of going to a conference, a company will give you time off to go and work and learn what others are doing.
• Craftsman spikes: These are side projects that you can use to practice craftsmanship. Companies offers employees 20% time to work on these side projects.

This presentation is now available on infoQ at http://www.infoq.com/presentations/Software-Craftsmanship-Beyond-The-Hype

Bold Statements

During the key note address at the 2010 Agile DoD conference, Scott Ambler made some bold statements. Below is a recap of some of his most interesting points.

There is overwhelming evidence that defining requirements in detail upfront is phenomenally bad practice, unethical at best in the IT world.

A changed requirement late in development cycle is a competitive advantage provided you can act on it.

Change management is really about change prevention. People that follow change management are worried about scope creep and are trying to prevent it. These people fundamentally do not understand what their jobs are as IT professionals. When you prevent requirements from changing, then yes you might be building something to spec, but you are now building something that people do not want and you are charging them for it. That is not ethical.

The best way for communication is face to face communication. If you want to increase the risk on your IT projects, write documentation and create a detailed specification.

Repeatable processes are bureaucracy. What people really want are repeatable results. They want high quality. They want systems that meet their needs in a timely matter. Delivering repeatable results requires discipline.

Agile is highly disciplined. You have to distinguish between bureaucracy and discipline. Bureaucracy is not discipline. Filling up paperwork, documenting requirements, documenting detailed architecture, reviewing them, and getting people to sign off on them, all this is not discipline. This is bureaucracy.

Real disciplined is producing high quality software every few weeks. It is refactoring and cleaning up your code as you go. It is about professionals validating their own work to the best of their ability. Don’t just throw stuff over to QA.

Agile teams are self organizing generalists. Like the military, guys on the ground are allowed to make decisions and adjust. Specialists are risky. If the specialists are gone, everybody is in trouble. We should avoid hand-offs between specialists.

Agile teams are easier to manage than traditional teams. In agile world, you cannot hide anymore, you have to produce.

There is nothing new about Agile. Most ideas of Agile can be traced back to the 60s and 70s. Agilists focus on the things that work and avoid the things that do not work.

There is nothing special about you. Do not come up with excuses for not adopting agile.

Continoues Delivery

Jez Humble talks about delivering software fast and reliably at a DevOps conference. He refers to Mary and Tom Poppendieck's book Implementing Lean Software Development and asks: “How long would it take your organization to deploy a change that involved just one single line of code? Do you do this on a repeatable, reliable basis?”  To get an idea of what that is, checkout the bottom part of code.flickr.com. It say flckr was last deployed x hours ago, including y changes by z people.

Jez recalls a situation where their 1st attempt to deploy took them 2 weeks. He was working for an ISP provider with a team of 60 developers. They had setup file caching which worked great on windows development environment but failed when deployed onto the production solaris cluster. The problem was that they had made assumptions for windows that did not apply to solaris.

Jez explains that to do real testing, we need to be in a production like environment.  In agile environments we are very good at doing analysis, development, testing, and demos in a team environment but then we pass things over to QA and IT operations. These hand-offs are an anti pattern. The team needs to include everyone involved in delivery and deployment needs to be fully automated. If it takes 2 weeks to deploy then it takes more than 2 weeks to get feedback.

Jez explains that releasing frequently is important for 3 main reasons:

1. Fast feedback: Tight cycle between thinking of an idea, releasing some software that represents it, getting feedback from users, and very quickly releasing new versions of software. Example: Flckr started out as gaming, but then realized the people are using it for photo sharing.

2. Reduce risk: With a big bang release, the individual amount of change is large. If releasing frequently, the delta is very small. It is easy to figure out where the problem is and it is easy to roll back if needed.

3. Real project progress: New software is only done when it is released. Before being released it is not delivering value.

To achieved continuous delivery, start with value stream mapping. Talk with everyone involved in delivery, measure time taken doing vs. waiting and then create an automated system that embodies the process. Every time anyone makes a change, run unit test and run acceptance test on production like environment. Everyone has to collaborate to get changes into production as quickly as possible. Use cycle time as a metric for measuring the productivity of the team.

When you are setup for continuous delivery, releases are no longer tied to operational constraints. Business can decide when to release.

Jez refers to research done by Forrester that some business will run internally as if they are start-ups. Business units will act as VC’s. They have funding and can provide it to projects which the business wants to invest in. They run projects as a product team not a project team. The key is not to disband the team when the project goes lives. We need to keep the team going as long as the product is in production.

Jez then goes over some principles for continuous delivery:

• Repeatable, reliable process for releasing software. Releases should be push buttons.
• Almost everything should be automated. We cannot automate exploratory testing and user acceptance testing, but we can automate almost everything else.
• Keep everything in version control.
• If it hurts, do it more often. Pain will force you to change your process and automate things.
• Build quality in. Everyone is responsible for testing. Developers, operations, etc.
• Done means released. Done means delivering value. We are not delivering value until the software is released.
• Everyone is responsible for delivery. Cross functional teams. Everyone should work together to fix bugs.
• Continuous improvements. Focus on biggest bottlenecks and improve those and then move on to the next ones and so on.

Next Jez cover practices:

• Only build binaries once. Can’t gurantee that the thing that you are releasing is the same thing that you are testing. Separate out config stuff from binaries. 
• Deploy the same way to every environment. Create package and use puppet to deploy to every environment.
• Smoke test your deployment. Check that everything you depend on is in place. Ping them!
• Keep your environment similar. That is, make sure the process to manage the environments is similar. 
• If anything fails, stop the line. Everyone should focus on fixing the problem.

Jez continues by briefly discussing continuous integration, testing, canary releasing, and data migration:

Continuous integration:

• Continuous is more often than you think. Do not use branches for features. Always check in to the main branch. 
• If you must branch (moving to a new architecture), then branch by abstraction and use a config option to choose which implementation to use.
• Check in new features to the main line, but hide them to turn them off if not ready for production

Testing:

• Unit tests, integration tests, and system tests: Focus on technology and programming practice.
• Functional acceptance tests: end to end system tests that verify business value.
• Non functional acceptance tests: performance, scaling …
• Showcases, usablility, explarotarty testing: this is the manual stuff that testers should be spending their time on. Everything else should be automated.

Canary releasing: Release new version of software to small subset of nodes, then route subset of users to these new nodes. This helps with A/B testing, and performance testing.

Data migration: Data migration needs to also be part of the deployment process. It needs to be automated. Data migration can be abstracted (using views, triggers) so that it works with multiple versions of your application.

Next, Jez explores objections to continuous delivery:

• Locking down – need change control: Reason is that things go wrong. But having visibility into what is coming and changing and control to rollback reduces this need to lock things down. 
• Compliance: comply using automation over documentation
• Auditing: see who does what.
• Make it easy to remediate outages

Finally Jez emphasizes that people are the key. Get everyone together from the beginning, keep meeting, make it easy for everyone to see what’s happening and keep continuously improving.

This presentation is now available on infoq at http://www.infoq.com/presentations/Continuous-Delivery

Comparative Agility

So how are you doing at adopting agile? You decided to go Agile. You got the training and coaching. Six months have passed. Are you where you should be? Are you excelling in some areas? Do you need to improve in others? How well are you doing compared to your competitors?

To answers these questions, Kenny Rebin and Laurie Williams presented Comparative Agility at Agile 2010. Comparative Agility is an assessment that evaluated multiple dimensions of agility that lead to actionable recommendations. The assessment framework consists of 7 dimensions covering 3 to 6 characteristics per dimension for a total of 125 questions.

The seven dimensions represent broad classifications of changes to be expected of a team or organization as it becomes more agile. The seven dimensions are:

• Teamwork
• Requirements
• Planning
• Technical practices
• Quality
• Culture
• Knowledge creation

Questions are asked to assess a team's score on each characteristic and answers are in the form of:

• More true than false
• Neither true nor false
• More false than true
• False

The philosophy behind the assessment is that organizations do not need to be perfect but they need to be better than their competitors. The assessment is not used to determine maturity levels. It simply answers “How am I doing compared to my competitors?”

Want to find out how you are doing? Take the survey and find out more at www.ComparativeAgility.com