Software Craftsmanship, Beyond the Hype

Corey Haines gave a talk about software craftsmanship at QCON London 2010.Corey describes how in December 2008, a group of aspiring software craftsmen got together and tried to solve some problems they were facing. They discussed how to become better and how to bring new people into software careers. They came up with a statement of things that they believe in and crafted the manifesto of Software Craftsmanship which states:

• Not only working software but also well crafted software

• Not only responding to change but also steadily adding value

• Not only individuals and interactions but also a community of professionals

• Not only customer collaboration, but also productive partnerships

Corey’s talk focuses on well crafted software and a community of professionals.There are a lot of schools of software craftsmanship that all revolve around the principles of the manifesto. Corey expands on the thoughts of the Chicago school of software craftsmanship. They felt that we are heading in a bad direction. The state of software development was going downhill. People came up with idea on developing quality software. They came up with Scrum. However, Scrum did not prescribe any practices. Ken Schwaber said that we made a fundamental assumption that was wrong. The assumption was that developers were smart enough to come up with their own practices. However, developers had spent their careers working in large cycles. They were used to it. They were accustomed to waiting for 9 months before coding or before QA and now they had to suddenly figure out how to do things in 30 days or even in 2 weeks.

XP had the practices that support the idea of short feedback loops. However, XP is hard. TDD is hard. When you introduce TDD, productivity goes down. Overtime, it will go back up again, but it is hard to get started.

Corey states that Agile is about developing quality software. We need something that focuses on developing quality software developers. This is what software craftsmanship is. We need to give people the tools to be better.

But what are these tools? And what does it mean to be better? This depends on each individual. Don’t let somebody else tell you what it means. You have to figure it out for yourself. Find out what you need to improve on to produce better software. Picture your ideal. Picture your reality when under pressure. The difference between the two is a measure of how much your code sucks. Use this to see how you are improving.

Corey continues by explaining how do we know what those ideals are? This is where we each have to go through the stages of being an apprentice, a journeyman and eventually a craftsman.

Apprentice is going in and finding somebody who is willing to take you under their wing and show you the way. You can spend time learning effective ways of developing software (TDD, evolutionary design, capturing requirements). You work to learn those ways. You internalize a set of practices that are time tested and proven. You do this by working on software. Not by sitting in a classroom.

Once you have those practices, then you look out for other people out there that are doing it differently. See how you can integrate them. This is the concept of journeyman. It is not about travelling the world. It is about participating in user groups, reading books, observing others, integrating what you have learned and moving forward.

Corey wraps up by describing the latest movements that resulted from the manifesto and how it made it easier for people to roll out some ideas on how to practice.

• Code Katas: Dave Thomas from pragmatic programmers came up with code katas, which were small problems to solve. Uncle Bob switched that to practice on a solution instead. He related it to martial arts and how you repeat small motions and practice them until they become natural reflexes. When you are in this situation, this is what you do. An example might be practicing String replacement using regex. Do it over and over again until it becomes a reflex. The site Katacast.com has various screencasts known as katacast that show folks practicing a small kata.
• Conferences: The Software craftsmanship conferences were established and 2 are held each year, one in the UK and one in the US.
• Code retreats: Retreats started worldwide. This is where developers get together on a Saturday for a full day of practice. They work on Conway’s game of life using a technique known as “TDD as if you meant it” which focuses on TDD being all about evolutionary design. Pair-up, work on it for 45 minutes. Then delete your code, swap pairs and do it again.
• User groups: Several Software Craftsmanship User groups started.
• Craftsman Swaps: A couple of companies conducted craftsman swaps. This is where 2 companies swap an employee for a week. The employees learn the practices of another company and come back and try to improve their own environment.
• Craftsman Journeys: Similar to a craftsman swap, this is where you just go to a company for a week and learn what they do. So instead of going to a conference, a company will give you time off to go and work and learn what others are doing.
• Craftsman spikes: These are side projects that you can use to practice craftsmanship. Companies offers employees 20% time to work on these side projects.

This presentation is now available on infoQ at http://www.infoq.com/presentations/Software-Craftsmanship-Beyond-The-Hype

Bold Statements

During the key note address at the 2010 Agile DoD conference, Scott Ambler made some bold statements. Below is a recap of some of his most interesting points.

There is overwhelming evidence that defining requirements in detail upfront is phenomenally bad practice, unethical at best in the IT world.

A changed requirement late in development cycle is a competitive advantage provided you can act on it.

Change management is really about change prevention. People that follow change management are worried about scope creep and are trying to prevent it. These people fundamentally do not understand what their jobs are as IT professionals. When you prevent requirements from changing, then yes you might be building something to spec, but you are now building something that people do not want and you are charging them for it. That is not ethical.

The best way for communication is face to face communication. If you want to increase the risk on your IT projects, write documentation and create a detailed specification.

Repeatable processes are bureaucracy. What people really want are repeatable results. They want high quality. They want systems that meet their needs in a timely matter. Delivering repeatable results requires discipline.

Agile is highly disciplined. You have to distinguish between bureaucracy and discipline. Bureaucracy is not discipline. Filling up paperwork, documenting requirements, documenting detailed architecture, reviewing them, and getting people to sign off on them, all this is not discipline. This is bureaucracy.

Real disciplined is producing high quality software every few weeks. It is refactoring and cleaning up your code as you go. It is about professionals validating their own work to the best of their ability. Don’t just throw stuff over to QA.

Agile teams are self organizing generalists. Like the military, guys on the ground are allowed to make decisions and adjust. Specialists are risky. If the specialists are gone, everybody is in trouble. We should avoid hand-offs between specialists.

Agile teams are easier to manage than traditional teams. In agile world, you cannot hide anymore, you have to produce.

There is nothing new about Agile. Most ideas of Agile can be traced back to the 60s and 70s. Agilists focus on the things that work and avoid the things that do not work.

There is nothing special about you. Do not come up with excuses for not adopting agile.

Continoues Delivery

Jez Humble talks about delivering software fast and reliably at a DevOps conference. He refers to Mary and Tom Poppendieck's book Implementing Lean Software Development and asks: “How long would it take your organization to deploy a change that involved just one single line of code? Do you do this on a repeatable, reliable basis?”  To get an idea of what that is, checkout the bottom part of code.flickr.com. It say flckr was last deployed x hours ago, including y changes by z people.

Jez recalls a situation where their 1st attempt to deploy took them 2 weeks. He was working for an ISP provider with a team of 60 developers. They had setup file caching which worked great on windows development environment but failed when deployed onto the production solaris cluster. The problem was that they had made assumptions for windows that did not apply to solaris.

Jez explains that to do real testing, we need to be in a production like environment.  In agile environments we are very good at doing analysis, development, testing, and demos in a team environment but then we pass things over to QA and IT operations. These hand-offs are an anti pattern. The team needs to include everyone involved in delivery and deployment needs to be fully automated. If it takes 2 weeks to deploy then it takes more than 2 weeks to get feedback.

Jez explains that releasing frequently is important for 3 main reasons:

1. Fast feedback: Tight cycle between thinking of an idea, releasing some software that represents it, getting feedback from users, and very quickly releasing new versions of software. Example: Flckr started out as gaming, but then realized the people are using it for photo sharing.

2. Reduce risk: With a big bang release, the individual amount of change is large. If releasing frequently, the delta is very small. It is easy to figure out where the problem is and it is easy to roll back if needed.

3. Real project progress: New software is only done when it is released. Before being released it is not delivering value.

To achieved continuous delivery, start with value stream mapping. Talk with everyone involved in delivery, measure time taken doing vs. waiting and then create an automated system that embodies the process. Every time anyone makes a change, run unit test and run acceptance test on production like environment. Everyone has to collaborate to get changes into production as quickly as possible. Use cycle time as a metric for measuring the productivity of the team.

When you are setup for continuous delivery, releases are no longer tied to operational constraints. Business can decide when to release.

Jez refers to research done by Forrester that some business will run internally as if they are start-ups. Business units will act as VC’s. They have funding and can provide it to projects which the business wants to invest in. They run projects as a product team not a project team. The key is not to disband the team when the project goes lives. We need to keep the team going as long as the product is in production.

Jez then goes over some principles for continuous delivery:

• Repeatable, reliable process for releasing software. Releases should be push buttons.
• Almost everything should be automated. We cannot automate exploratory testing and user acceptance testing, but we can automate almost everything else.
• Keep everything in version control.
• If it hurts, do it more often. Pain will force you to change your process and automate things.
• Build quality in. Everyone is responsible for testing. Developers, operations, etc.
• Done means released. Done means delivering value. We are not delivering value until the software is released.
• Everyone is responsible for delivery. Cross functional teams. Everyone should work together to fix bugs.
• Continuous improvements. Focus on biggest bottlenecks and improve those and then move on to the next ones and so on.

Next Jez cover practices:

• Only build binaries once. Can’t gurantee that the thing that you are releasing is the same thing that you are testing. Separate out config stuff from binaries. 
• Deploy the same way to every environment. Create package and use puppet to deploy to every environment.
• Smoke test your deployment. Check that everything you depend on is in place. Ping them!
• Keep your environment similar. That is, make sure the process to manage the environments is similar. 
• If anything fails, stop the line. Everyone should focus on fixing the problem.

Jez continues by briefly discussing continuous integration, testing, canary releasing, and data migration:

Continuous integration:

• Continuous is more often than you think. Do not use branches for features. Always check in to the main branch. 
• If you must branch (moving to a new architecture), then branch by abstraction and use a config option to choose which implementation to use.
• Check in new features to the main line, but hide them to turn them off if not ready for production

Testing:

• Unit tests, integration tests, and system tests: Focus on technology and programming practice.
• Functional acceptance tests: end to end system tests that verify business value.
• Non functional acceptance tests: performance, scaling …
• Showcases, usablility, explarotarty testing: this is the manual stuff that testers should be spending their time on. Everything else should be automated.

Canary releasing: Release new version of software to small subset of nodes, then route subset of users to these new nodes. This helps with A/B testing, and performance testing.

Data migration: Data migration needs to also be part of the deployment process. It needs to be automated. Data migration can be abstracted (using views, triggers) so that it works with multiple versions of your application.

Next, Jez explores objections to continuous delivery:

• Locking down – need change control: Reason is that things go wrong. But having visibility into what is coming and changing and control to rollback reduces this need to lock things down. 
• Compliance: comply using automation over documentation
• Auditing: see who does what.
• Make it easy to remediate outages

Finally Jez emphasizes that people are the key. Get everyone together from the beginning, keep meeting, make it easy for everyone to see what’s happening and keep continuously improving.

This presentation is now available on infoq at http://www.infoq.com/presentations/Continuous-Delivery

Comparative Agility

So how are you doing at adopting agile? You decided to go Agile. You got the training and coaching. Six months have passed. Are you where you should be? Are you excelling in some areas? Do you need to improve in others? How well are you doing compared to your competitors?

To answers these questions, Kenny Rebin and Laurie Williams presented Comparative Agility at Agile 2010. Comparative Agility is an assessment that evaluated multiple dimensions of agility that lead to actionable recommendations. The assessment framework consists of 7 dimensions covering 3 to 6 characteristics per dimension for a total of 125 questions.

The seven dimensions represent broad classifications of changes to be expected of a team or organization as it becomes more agile. The seven dimensions are:

• Teamwork
• Requirements
• Planning
• Technical practices
• Quality
• Culture
• Knowledge creation

Questions are asked to assess a team's score on each characteristic and answers are in the form of:

• More true than false
• Neither true nor false
• More false than true
• False

The philosophy behind the assessment is that organizations do not need to be perfect but they need to be better than their competitors. The assessment is not used to determine maturity levels. It simply answers “How am I doing compared to my competitors?”

Want to find out how you are doing? Take the survey and find out more at www.ComparativeAgility.com

Code Retreat

At agile 2010, Rob Park introduced us to code retreats. This is where software craftsmen from all over get together to do several iterations of pair programming and test-driven development. Developers pair up to try to solve Conway's Game of Life. Each iteration is 45 minutes, followed by a retrospective. Before each new iteration, code is deleted and pairs are swapped. The idea is to experiment and try coding with bottom-up or top down, using no ifs or no loops, etc. Practice the fundamentals of TDD by coding using “TDD as if you meant it”:

1. Write exactly 1 failing test
2. Write implementation code in the test to make the test pass.
3. Create new implementation methods only by extracting out the working code from the test.
4. Create these new methods in the same test class.
5. Extract these new methods from the test class into implementation classes.
6. Refactor as required.
7. Repeat

The rules for Conway's Game of Life are:

•  Any live cell with fewer than two live neighbours dies, as if caused by under-population.
•  Any live cell with more than three live neighbours dies, as if by overcrowding.
•  Any live cell with two or three live neighbours lives on to the next generation.
•  Any dead cell with exactly three live neighbours becomes a live cell, as if by reproduction.

The first generation is created by applying the above rules simultaneously to every cell in the seed—births and deaths occur simultaneously, and the discrete moment at which this happens is sometimes called a tick (in other words, each generation is a pure function of the preceding one). The rules continue to be applied repeatedly to create further generations.

Others have tried retreats using games like tic-tac-toe or Go. Find out more at http://www.coderetreat.com/how-it-works.html or check out the next code retreat coming to a neighborhood near you at coderetreat.ning.com/events

New Year's Resolution: Get Fit Using Agile Workstations

Every New Year, I come up with a New Year's resolution list, and it almost always includes getting in better shape by joining a gym. Unfortunately, I never seem to follow through on it. But how about getting fit at work by using an Agile Workstation?

Recently, Matt, a colleague of mine, setup a make shift standing work station. He stacked some books on top of some boxes to lift up his keyboard and mouse. He also placed his monitor higher up on a shelf. His argument was that this gives him better posture and reduces neck pain and is better for his overall health. Everyday he codes standing up for about 2 to 3 hours and then goes back to a more traditional seated coding position.

This reminded me of a presentation given by Doug Bradbury at Agile 2010 entitled “Walk and Code”. Doug presented data that showed that we sit significantly more and stand significantly less on our work days than on our leisure days. Another study showed that obese individuals were seated on average 2 hours longer per day than lean individuals. Yet another study showed that men and women had a 17% and 37% higher chance of death when sitting more than 6 hours per day as compared to those sitting less than 3 hours per day. When we factor in sitting and not exercising then the numbers jump to an alarming 48% and 95% respectively. So clearly we need to exercise during our work week. But does standing or walking count?

Doug next describes the research done by Dr. James Levine on non-exercise activity thermogenesis (NEAT) which is the calories people burn during everyday activities like walking, standing or even fidgeting. Dr. Levine is studying the effect of NEAT on energy expenditure and overall good health. Simply walking at a slow 1 mph can burn an extra 100 cal/hr. Dr. Levine came up with an adjustable treadmill desk where people can walk and work, stand, or sit. He believes this can help in both health and productivity as people stay motivated, alert and focused. However, this adjustable workstation cost $4000+.

So like Matt, Doug setup his own make shift treadmill/desk by buying a used treadmill from ebay and a desk from IKEA. After 22 weeks he had walked 240 miles (average of 11 mi/week) by just walking and coding!

So are you up for it? I found a treadmill desk add-on on amazon for $400. If this all sounds too extreme and expensive then try replacing your chair with an exercise ball or bicycle, or simply take the stairs instead of the elevators, park far from the office, and take a walk at lunch. Good Luck and Happy New Year!

 

To learn more about NEAT check out http://mayoresearch.mayo.edu/levine_lab/about.cfm

To find out more on how Dough constructed his workstation, check out his blog http://blog.8thlight.com/articles/2010/2/25/walk-and-code

Here are links to studies referenced in Dough's presentation:

http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2783690/
 and http://aje.oxfordjournals.org/content/172/4/419.abstract

The Ultimate Wall Board

What’s on your wall board? Atlassian ran a competition for the ultimate wall board. Ole Højriis Kristensen from the Vodafone web team won the competition by integrating a physical Scrum board with Atlassian’s JIRA/Greenhopper issue tracking/PM tool.

Ole’s solution keeps the physical board in sync with their project management tool. He setup a printer below the scrum board. Whenever a new story is added to JIRA, a card is printed out with a reference to the JIRA ticket number. This receipt is then placed in a plastic card that has an RFID and a magnet and then the card is placed on the board.

Whenever a developer is ready to work on a story, the developer swipes the card and moves it to the next column. They also have avatar cards for each developer that they swipe to indicate who is assigned to the story. Any card movement on the board automatically updates the corresponding story in JIRA. A camera snaps a picture of the developer that is moving the card and it is attached as a comment to the JIRA issue indicating who modified the ticket. The board also kicks off the build/deploy process for stories that are ready for testing and sends out tweets to inform the team and QA. There is also a projector that continuously displays burn down charts and velocity from JIRA data.

Even when an issue is updated directly through JIRA, the board constantly reminds users (via voice prompts) to move the corresponding card to its correct position!

Additional awards were given to a couple of other boards. Check them out at http://ultimatewallboard.com/