Spiral Matrix Kata

In the May DC Software Craftsmanship meeting we tackled the Spiral Matrix kata. The instructions for this one are simple and are best illustrated by the examples below:

3 by 4 matrix:

4 by 4 matrix:

3 by 5 matrix:

The Kata in the Hat

At Agile 2011, Emmanuel Gaillot and Jonathan Perret performed a coding Dojo entitled "The Kata in The Hat." Emmanuel and Jonathan are behind the original coding dojos in Paris. Usually, katas are performed by practicing TDD and applying baby steps. The main kinds of Katas include:

1. Prepared Kata  where someone comes prepared with a full solution to the Kata and presents it to the group.
2. Randori Kata where the group rotates pairs so solve a particular problem.

The Kata in The Hat is a twist on these katas. Here, the hosts had a hat in which audience members can write-in rules and requests and drop them in the hat. The presenters start with a main idea, but every five minutes one paper is pulled out of the hat, the instructions are read and the timer is reset. The hosts continue to pair program, but now they incorporate the new requests into their code. And so on... The interesting part is seeing how they can finish each request by doing the simplest thing that can possibly work. They quickly adjust and keep the code moving as they try to solve the main problem no matter what is thrown at them. This type of Kata is definitely not for the beginners, but it’s a great show that is fun, entertaining and educational. If they are ever touring at a conference near you, I highly recommend you attend.

"The big idea we want to get across is that programming can be beautiful, and that through creativity and beauty and poetry programmers have much more power than they think."
 http://thekatainthehat.com/ 

Importance of Well Designed Software

At the 2011 Agile Development Practices West conference, Martin Folwer addressed trade-able quality.

Martin explains that quality is something we can trade-off for other things. We often hear managers saying “We need to put less effort on quality so we can build more features for our next release.” We also do this every day. We do it when we buy a car. Sure we want the fancy car, but we go for the cheaper one and spend the extra savings on something else. This is a natural part of how we think about the things that we buy every day.

In software, there are visible and invisible quality features. For example, with a UI, you can feel the difference. But internal quality is not visible to the users. As a buyer, if I can’t see it, why do I want it? In that case, I will always pick the cheaper option.

Next Martin introduces the Design Stamina Hypothesis to illustrate why we should care about internal design of software.

Figure 1 - Stamina Hypothesis

The graph shows functionality over time.

The blue line shows bad design. We start out strong, but fade and slow down over time.
The red line shows good design. We start out strong and over time, we can keep a steady pace and even speed up.

With the clean system, a year from now it will have another 5 new features. The bad design system will probably only have 1 or 2 new features as they struggle with their bad code base.

Next Martin introduces the term technical debt which was originally coined by Ward Cunningham. Complexity that we don’t need causes extra effort to add new features. The unnecessary complexity in your code base is like the principal of a debt and the extra cost that you pay for adding new features is like the interest on the debt. This metaphor is useful when talking with non technical people. When building new software, you can choose a quick and dirty approach, increase your debt and increase your interest payment. Or you can pay down the principal and reduce interest payments.

Martin then explains the reasons we get into technical debt.

Figure 2 - Technical Debt Quadrant

1. Prudent/Deliberate: “We must ship now and deal with consequences” – This is a conscious decision. I have a deadline and I’m prepared to tradeoff some quality to meet that deadline.

2. Inadvertent/Reckless: People have no clue about software design.

3. Deliberate/Reckless: Very close to prudent/deliberate however, it is done without understanding what is going on. The decision to trade-off quality (debt) for speed only makes sense when we are below the design pay-off line (where the 2 red and green lines cross in figure 1). The key is to understand how far away the design pay-off line is. It is probably in the order of weeks and not in the order of months. It is a lot shorter than what people tend to think it is. If I am close to the point where I go over the design pay-off line, then there is no point in sacrificing design for speed because them I will end up losing both.

4. Prudent/Inadvertent – Here the financial metaphor breaks down. This case is when we’ve taken on debt without noticing it. It is because we learn new things as we code and when we look back at what we coded we say “Now we know how we should have done it”.

This presentation is available on youtube at http://www.youtube.com/watch?v=p5Qj75nJPEs (starts at minute 29:05)

Martin also discusses technical debt on his blog http://www.martinfowler.com/bliki/DesignStaminaHypothesis.html and http://www.martinfowler.com/bliki/TechnicalDebtQuadrant.html

Non-Determinism and Testing

At the keynote address of the 2011 Agile Development Practices West, Martin Fowler gave a talk on non- deterministic tests. These are tests that when you run them they pass, but if you run them again (without changing any code), they fail. They are also referred to as flaky tests or intermittent failures. The tests seem to pass or fail in a random fashion.

Martin asserts that non-deterministic tests are useless. The whole point of a regression suite is to be a bug detection mechanism. Its value is that you get immediate feedback when you make a mistake. The time between making a mistake and realizing that you did is very short. And because it is short, you can quickly figure out what you did wrong and fix it. With non-deterministic tests you get unreliable information.

Martin argues that non-deterministic tests are worse that useless. They are very dangerous. They are like an infection that infects the entire suite. One failing test will mark the entire suite as red. Usually we will dig in to see which test failed and fix it, but once we start seeing red regularly, we will assume the failure is due to the non-deterministic test and ignore the failure of the entire suite. This makes the entire suite flaky and the entire test suite becomes useless.

Martin purposes setting up a quarantine area for flaky tests. If we pull the tests from the suite, then the suite will remain reliable. The number of tests in quarantine need to be limited and fixed as soon as possible. Martin recommends setting some limits like no more than 3 tests can be in quarantine or a test in quarantine should be fixed within a week.

Martin next describes some causes for non-deterministic tests.

1. Lack of isolation: Tests depend on the order that are run. These are very hard to debug because when a test fails because of the order it ran in the suite, you usually re-run in isolation and then it passes. The real cause of the problem is in another test which is green and very hard to locate. There are two possible solutions:

• Track dependencies: Track the order in which the tests need to run. This is very hard to manage and maintain.
• Isolation: Martin prefers this solution. You can apply a clean-up strategy. That is, make sure there is a tear down that destroys anything you have created. An even better approach is to use a blank slate. Always start with a clean data set. The disadvantage here is that setting up the blank slate can be time consuming.

2. Asynchrony: Testing asynchrony is difficult. A common approach is to use what Martin calls a bare sleep where we introduce a sleep statement.

//pseudo-code
someFunction
sleep(aWhile)
assert(answer)

If we sleep too long, the test will run very slowly. If we reduce the sleep, then when we move to a different machine or if the machine is overloaded, then the tests will fail, so coming up with the correct number to sleep is very hard. Martin warns us against using this approach and recommends another solution using polling:

//pseudo-code
someFunction
timeLimit=Time.now + timeout
while(!answer.recieved) {
  if (Time.now > waitLimit) {
    throw new TestTimeoutException
  }
  sleep (pollingInterval)
}
assert(answer)

Here we have the sleep inside the polling and a timeout interval. You can keep the sleep interval very low and adjust the timeout. You have two times to play with (global constants).

Martin also discusses another approach using callbacks

//pseudo-code
someFunction
callback = function
{ assert(answer) }
answer.ready(callback)
asserts.add(callback, timeout)

The callback method itself contains the verification. This solution depends on the testing framework that you are using.


3. Interaction with remote services: There are many things that can go wrong when dealing with a remote service that have nothing to do with your code (network down or slow, limited availability, unstable test data).

Martin recommends using a a test double. This way, you control the data the service has access to, you can control changes, and ensure fast connections. However, some don’t believe this is testing the real service because we cannot keep the double consistent with the remote service. To solve this problem, Martin recommends that we add an integration contract test that does not run as part of the main build. This test will probably run nightly against the actual service and ensures that the signature of the double is in synch with the actual service.

Martin concludes by warning us that flaky tests are a common problem that can become very dangerous if not addressed immediately.

This presentation is available on youtube at http://www.youtube.com/watch?v=p5Qj75nJPEs

Martin also discusses non-deterministic tests on his blog http://martinfowler.com/articles/nonDeterminism.html

Software Craftsmanship - It's an Imperative

Thanks to all that attended my presentation on Software Craftsmanship.

Below are the slides and notes from the presentation.

Software Craftsmanship, Beyond the Hype

Corey Haines gave a talk about software craftsmanship at QCON London 2010.Corey describes how in December 2008, a group of aspiring software craftsmen got together and tried to solve some problems they were facing. They discussed how to become better and how to bring new people into software careers. They came up with a statement of things that they believe in and crafted the manifesto of Software Craftsmanship which states:

• Not only working software but also well crafted software

• Not only responding to change but also steadily adding value

• Not only individuals and interactions but also a community of professionals

• Not only customer collaboration, but also productive partnerships

Corey’s talk focuses on well crafted software and a community of professionals.There are a lot of schools of software craftsmanship that all revolve around the principles of the manifesto. Corey expands on the thoughts of the Chicago school of software craftsmanship. They felt that we are heading in a bad direction. The state of software development was going downhill. People came up with idea on developing quality software. They came up with Scrum. However, Scrum did not prescribe any practices. Ken Schwaber said that we made a fundamental assumption that was wrong. The assumption was that developers were smart enough to come up with their own practices. However, developers had spent their careers working in large cycles. They were used to it. They were accustomed to waiting for 9 months before coding or before QA and now they had to suddenly figure out how to do things in 30 days or even in 2 weeks.

XP had the practices that support the idea of short feedback loops. However, XP is hard. TDD is hard. When you introduce TDD, productivity goes down. Overtime, it will go back up again, but it is hard to get started.

Corey states that Agile is about developing quality software. We need something that focuses on developing quality software developers. This is what software craftsmanship is. We need to give people the tools to be better.

But what are these tools? And what does it mean to be better? This depends on each individual. Don’t let somebody else tell you what it means. You have to figure it out for yourself. Find out what you need to improve on to produce better software. Picture your ideal. Picture your reality when under pressure. The difference between the two is a measure of how much your code sucks. Use this to see how you are improving.

Corey continues by explaining how do we know what those ideals are? This is where we each have to go through the stages of being an apprentice, a journeyman and eventually a craftsman.

Apprentice is going in and finding somebody who is willing to take you under their wing and show you the way. You can spend time learning effective ways of developing software (TDD, evolutionary design, capturing requirements). You work to learn those ways. You internalize a set of practices that are time tested and proven. You do this by working on software. Not by sitting in a classroom.

Once you have those practices, then you look out for other people out there that are doing it differently. See how you can integrate them. This is the concept of journeyman. It is not about travelling the world. It is about participating in user groups, reading books, observing others, integrating what you have learned and moving forward.

Corey wraps up by describing the latest movements that resulted from the manifesto and how it made it easier for people to roll out some ideas on how to practice.

• Code Katas: Dave Thomas from pragmatic programmers came up with code katas, which were small problems to solve. Uncle Bob switched that to practice on a solution instead. He related it to martial arts and how you repeat small motions and practice them until they become natural reflexes. When you are in this situation, this is what you do. An example might be practicing String replacement using regex. Do it over and over again until it becomes a reflex. The site Katacast.com has various screencasts known as katacast that show folks practicing a small kata.
• Conferences: The Software craftsmanship conferences were established and 2 are held each year, one in the UK and one in the US.
• Code retreats: Retreats started worldwide. This is where developers get together on a Saturday for a full day of practice. They work on Conway’s game of life using a technique known as “TDD as if you meant it” which focuses on TDD being all about evolutionary design. Pair-up, work on it for 45 minutes. Then delete your code, swap pairs and do it again.
• User groups: Several Software Craftsmanship User groups started.
• Craftsman Swaps: A couple of companies conducted craftsman swaps. This is where 2 companies swap an employee for a week. The employees learn the practices of another company and come back and try to improve their own environment.
• Craftsman Journeys: Similar to a craftsman swap, this is where you just go to a company for a week and learn what they do. So instead of going to a conference, a company will give you time off to go and work and learn what others are doing.
• Craftsman spikes: These are side projects that you can use to practice craftsmanship. Companies offers employees 20% time to work on these side projects.

This presentation is now available on infoQ at http://www.infoq.com/presentations/Software-Craftsmanship-Beyond-The-Hype

Code Retreat

At agile 2010, Rob Park introduced us to code retreats. This is where software craftsmen from all over get together to do several iterations of pair programming and test-driven development. Developers pair up to try to solve Conway's Game of Life. Each iteration is 45 minutes, followed by a retrospective. Before each new iteration, code is deleted and pairs are swapped. The idea is to experiment and try coding with bottom-up or top down, using no ifs or no loops, etc. Practice the fundamentals of TDD by coding using “TDD as if you meant it”:

1. Write exactly 1 failing test
2. Write implementation code in the test to make the test pass.
3. Create new implementation methods only by extracting out the working code from the test.
4. Create these new methods in the same test class.
5. Extract these new methods from the test class into implementation classes.
6. Refactor as required.
7. Repeat

The rules for Conway's Game of Life are:

•  Any live cell with fewer than two live neighbours dies, as if caused by under-population.
•  Any live cell with more than three live neighbours dies, as if by overcrowding.
•  Any live cell with two or three live neighbours lives on to the next generation.
•  Any dead cell with exactly three live neighbours becomes a live cell, as if by reproduction.

The first generation is created by applying the above rules simultaneously to every cell in the seed—births and deaths occur simultaneously, and the discrete moment at which this happens is sometimes called a tick (in other words, each generation is a pure function of the preceding one). The rules continue to be applied repeatedly to create further generations.

Others have tried retreats using games like tic-tac-toe or Go. Find out more at http://www.coderetreat.com/how-it-works.html or check out the next code retreat coming to a neighborhood near you at coderetreat.ning.com/events

Bad Code, Craftsmanship, Engineering, and Certification

At QCON 2010, Robert Martin gave a talk about Bad Code, Craftsmanship, Engineering, and Certification. Uncle Bob started the talk by showing an example of bad code which consisted of two massive classes with hundred of methods. He asked why do we write bad code that we know will impede us? And wondered if it is due to deadlines, laziness, boredom, or job security. Bob stresses that the way we go fast as professionals is to do the best job we possible can. He encourages us to wear a green band on our wrist that symbolizes a commitment to write the best code we can at all times.

Uncle Bob then makes the following recommendations:

1.      Apply the boy scout rule: Always leave the camp ground cleaner than the way you found it. That is, whenever you work on a class or function, spend some extra time to refactor it and clean it a little.

2.      Apply Extract till you drop: Function size should be small. Keeping extracting until you can no longer extract the function. This takes all the concepts in the function puts them in well named places.

3.      No argument is the best argument. Functions should have the smallest number of arguments possible. Try not to exceed 2 arguments. Do not pass Booleans as arguments; instead create two well named functions.

4.      Use long names to be as descriptive as possible.

5.      Avoid duplication. Do not copy and paste.


Next Uncle Bob goes over the Software Craftsmanship Manifesto which states:

1.      We value not only working software but also well crafted software

2.      Not only responding to change, but also steadily adding value.

3.      Not only individuals and interactions, but also a community of professionals.

4.      Not only customer collaboration but also productive partnerships.


He then covers the disciplines that the manifesto is based on. These include:

1.      TDD – verifies that our code works and also makes it possible for us to refactor and constantly improve the code over time.

2.      CI: Use tools like Hudson, CruiseControl, Bamboo, and TeamCity. If the build fails, immediately fix the build.

3.      Pairing: If you don’t want to code pair, at least make sure the team is communicating, sharing and working closely together.

4.      QA should find nothing

5.      Have pride in workmanship

To convince others, the best thing you can do is do the best job you can and be a role model. Soon others will observe that you are outperforming the rest and will realize why.

Uncle Bob next covers the engineering practices of small cycles feedback, source code as the design, and design patterns. He wraps up by discussing certification and how they over promise and under deliver. Instead he recommends actually talking to potential new hires and have several team members talk to them as well. Let them show you code samples. Ask for references and check those references.

This presentation is available on InfoQ at http://www.infoq.com/presentations/Robert-C.-Martin-Bad-Code

From Good to Great Developer

At Jfocus, Chris Hedgate gave a talk great developers. Chris warns that a good developer can be fast at coding and solving problems, but in long run, he will slow the team down due to un-maintainable code. Great developers are passionate about code and make sure the code they write does not rot over time.

Chris stresses that there needs to be awareness of the benefits of design patterns, TDD, refactoring, readable code. Total cost of software development is development plus maintenance. Maintenance starts as soon as we write the code instead of at the end after it is delivered. This way productivity is maintained at the same pace instead of declining. Existing code must be continuously maintained and all code must be easy to understand and modify.

Next Chris defines habit as the intersection of: Skills (how), knowledge (why), desire (want to). Chris describes the 4 stages of competence:

1. Unconscious incompetent (we don’t know that we don’t know, we don’t know that we can’t and we don’t see a need)

2. Consciously incompetent (we see something we do not know and we see a benefit from it. We can learn and get better at it).

3. Consciously competent: We work by ourselves; we can perform and can teach.

4. Unconsciously competent: habits are achieved, it’s automatic.

We train with book, lectures, and courses. However if they are at Unconscious incompetent then they need to be inspired and moved to consciously incompetent.

Chris suggests that to move people from good to great, we need to inspire people by leading by example and using simple tools, and involve people so that they move along with you from good to great. There need to be

1. Attitude: simplify, improve, modify, never leave code in worse shape than how your found it.

2. Tools: name things (variables, methods), eliminate duplication, pair programming, study circles, reflect and discuss, code reviews.

Chris wraps up by recommending we think about how to guide the team. Lead by example. Be a role model. If the team gets better, productivity will not only remain steady but increase over time.

Deliberate Practice in Software Development

At Agile 2009, Mary Poppendieck gave a talk on the theory behind craftsmanship. She starts out by giving some stats on Hockey Players from the book Outliers. 40% of elite Canadian hockey players were born between January and March, 30% April – June, 20% July – September, 10% October – December. The cut-off date is the end of the calendar year. So the ones born in December are at a disadvantage since they will be the youngest. The same applies for European football players.

Mary then defines talent (nature) as abilities, powers, and gifts bestowed upon a person; natural endowments; general intelligence or mental power. Also, talent (nurture) is a special innate or developed aptitude for an activity. Talent is overrated. The most accomplished people need around ten years of deliberate practice before becoming world class (ten-year rule).

Next Mary defines deliberate practice: Identify a skill that needs improvement, devise a focused exercise designed to improve the skill, practice repeatedly, obtain immediate feedback and adjust accordingly, focus on pushing the limits expecting repeated failures, practice regularly and intensely up to 3 hours a day.

Then Mary elaborates on the four elements of deliberate practice:

1. Mentor: masters, teachers, deeply knowledgeable and involved. They hire and grow people, review and guide work, set technical standards, and ensure technical excellence.

2. Challenge: do it frequently and it will become easy, test/integrate/release early and often, change the code frequently, assign challenging work assignment, focus on constant improvement of the product, process, and people. Don’t get comfortable, get better.

3. Feedback: Immediate feedback: Design reviews with mentors providing guidance, common code ownership with peers providing visibility, TDD, CI, customer feedback at every iteration, escaped defect feedback and analysis. Focus on what’s not perfect so it can be improved. You have to ask constantly “why are we doing this?” Bad news first!

4. Dedication: skill development takes time in place, time to learn, and time to invent. Also need motivation, and career paths for technical experts.

Mary wraps up by stressing the importance of clean code and going over the Software Craftsmanship manifesto. We value not only working software but also well crafted software, not only responding to change, but also steadily adding value, not only individuals and interactions, but also a community of professionals, not only customer collaboration but also productive partnerships.

This presentation is available on InfoQ at  http://www.infoq.com/presentations/poppendieck-deliberate-practice-in-software-development

The Ethics of Error Prevention

Michael Feathers gave a talk at JAOO about the ethics of error prevention. He mentions that preventing errors in applications have many different solutions, but most of the time we only need to pick one or two. He then moves on and coves 5 techniques:

1. Abstraction: Use of object oriented languages and higher end languages.

2. Design by contract: document precondition and post-conditions. The clients of the routine are obligated to fulfill them.

3. Clean room engineering: discipline of annotating code that proves to yourself that the code you are writing is correct.

4. Test driven development: write a test for a new capability, compile, fix compile error, run test and see it fail, write code, run the test and see it pass, refactor as needed, repeat.

5. Pair programming/software inspections: Fagan inspection: planning, overview, preparation, meeting, rework (looping back to planning), follow-up

Michael wraps up by mentioning that each of these techniques forces us to focus on what we are doing and steers us away from complication. They each trigger contemplation. The craft of software development is not about languages or tools. It is about practices. Quality is in the intangibles.

This presentation is available on InfoQ at http://www.infoq.com/presentations/error-prevention-ethics

Craftsmanship and Ethics

Bob Martin gave a talk about craftsmanship and ethics and wondered if we as developers have a profession. He defined a profession where as a body we can agree on a set of rules to follow. With the rise of agile techniques, developers have become more disciplined. Bob covers the following disciplines:

a. Short iterations (1 or 2 weeks). Plan, code, testing, documentation (complete cycle resulting in deployable software.

b. Don’t wait for definition: instead participate in the definition process by demonstrating working code.

c. Abstract away volatility: look at what is likely to change and separate it from what is not likely to change. (do not mix gui with business rules).

d. Commission instead of omission. It is better to experiment than to wait.

e. Decouple from others: write stubs to ensure your code can run and to define your interface

f. Never be blocked: always find some way to make progress

g. Avoid turgid viscous architectures: they do more to impede progress than it is to help. Built to create perfect architecture to solve all problems that may come up. Go with simple architectures.

h. Incremental improvement: check in the code better than when we checked it out.

i. No grand redesigns: they generally do not work very well. “tiger” team constantly trying to catch up with maintenance team

j. Progressive widening: Add a small feature from top to bottom (GUI to database),

k. Progressive deepening: Get something working in one layer and then move it down to other layers. 1st make it work, then make it right, then make it fast.

l. Don’t write bad code: It does not only slow down others months from now but will slow you down immediately.

m. Clean code: Only way to go fast is to slow down a minute and write clean code. Readable, obvious, small functions, good names

n. TDD: Follow 3 rules: 1) not allowed to write production code until you have a failing unit test, 2) you are not allowed to write more of unit test than is sufficient to fail (not compile is failing), 3) you are not allowed to write more production code than is sufficient to pass. This forces us to keep the code executing all the time. Tests make the software flexible and maintainable.

o. QA should find nothing: our attitude and approach should be that QA should find nothing.

p. 100% code coverage: strive for constantly achieving 100% code coverage. Managers should not use this number. Professionals should use it.

q. Avoid debugging: Don’t jump into the debugger. Look at the code 1st. Use TDD. Debugger should be last resort.

r. Manual test scripts are immoral: Playing with system should be explorative testing. The other vast majority of testing should be automated.

s. Definition of Done: All the tests are passing.

t. Test through the right interface: Do not test business rules through the gui. Test gui connected to a dummy (mock) layer.

u. Apprenticeship: We have to grow the new members of the team and transfer knowledge to them

v. Use good tools: open source community care about the code they write, like svn

w. Green band on wrist: A symbol of ethics. “I will be a professional developer”

This presentation is available on InfoQ at http://www.infoq.com/presentations/craftmanship-ethics

Fostering Software Craftsmanship in a Corporate Setting

Scott Dillman gave a talk at Agile 2008 about fostering software craftsmanship in a corporate setting. Scott defined software craftsmanship as taking responsibility, continuous learning, rejecting specialization, pride in quality work, passing on knowledge, and meeting professional standards (test driven development, continuous integration, coding patterns and practices).

Next Scott described 3 steps to evolve a novice software developer into a craftsman.

1. Evaluate the current state of the organization through interviews, surveys and metrics. Surveys can be reused at a later date to evaluate progress. Collecting metrics from automated builds like % passing builds, builds/day, build duration, fail build reasons, code coverage, code complexity. Track historical trends.

2. Educate via pair programming, centralized resources (wiki, educational content, knowledge base, moderated technical forum, book reports, tools and code repository), educational sessions(teasers, lightning sessions, tech talks, workshops), craftsmanship day, 10% time and on team mentoring. To get upper management buy in, highlight benefits, cite case studies, link to corporate goals. Upper management has to allocate time and recognize and reward participation.

3. Measure success by interviews, surveys, metrics, and performance reviews. Make sure reviews have goals that motivate and educate, are relevant, specific, actionable, and verifiable. Have the right people perform the reviews, do 360 reviews, and encourage and reward participation.

This presentation is available on InfoQ at http://www.infoq.com/presentations/Craftsmanship-Scott-Dillman