The Dancing Agile Elephant: IBM Software Group’s Transition to Agile and Lean Development

At QCON San Francisco 2008, Sue McKinney gave a presentation on IBM’s agile adoption strategy. She started out by giving an overview of some of the business and operational challenges at IBM. These include innovating the business to differentiate and capture new value, heighten responsiveness and closer linkage to customers, and improved time to value. On the operations side, issues included better workload management, improved quality, improved project development cycle times, improved predictability on schedule, and making better use of resources to be more productive

Next Sue described the IBM environment. IBM has a global team, with different companies brought together by acquisition. There are 500 development teams in 5 divisions. Teams are as large as 800 or as small as 20. Very few teams are collocated. They are highly security conscious. They use many tools (due to acquisitions), and many platforms.

Sue describes how IBM software development transformation went from waterfall in the 1980s (rigid, late feedback, slow reaction to market changes) to iterative (customized RUP, community source and component reuse, emphasis on consumability), to agile (global reach, SOA, agile practices, outside-in development, tool and not rules, continuous learning and adaptive planning

Sue mentions how she sold agile as a way to deal with uncertainty, and to respond to changing markets. Agile gives them transparency to check where things are going and enables them to take a course correction if needed by meeting with stakeholders and deciding what change to make. She kept processes to deal with complexity (team size, geo distribution, compliance or regulatory issues) and move from lightweight to things that are more thorough and allow for traceability.

Next Sue lists some things to consider before getting started. These include getting management support, picking strong experienced leaders, picking the right project as a proof point (project with high visibility and some risk), providing the right education, tooling and governance, ability to allow change to occur, and keeping it simple.

Sue defined Agile as short time boxed iteration with stakeholder feedback. This creates automatic constraints (iteration length) that force the team to find defects earlier and address them. The team becomes more responsive and there is always a sense of urgency. There is also a notion of transparency with demos at the end of each iteration. The constraints forces teams to optimize and eliminate waste (automate). Share holder feedback causes us to focus on the essentials.

The initial project was a team of 200 people over 4 sites with 1 week iteration. 1st 2 months, put together infrastructure to enable the team (test case harnesses, continuous integration, static code analysis), then 1 week iterations began. The software was published for regular consumption and received regular feedback. For sustainability (220 out of 500 teams are using agile), they created a practices for distributed development:

Realistic About Risk: Software Development with Real Options

At QCON London 2008, Olav Maassen and Chris Matts gave a talk about Real Options. They started out with a quick exercise where the audience was divided up into teams and each team had to write down who they thought would finish task A 1st and who would finish task B 1st. then they gave detailed instructions of each task and asked the teams to complete them. At the end, none of the teams picked the correct winner, where as both Olav and Chris picked correctly. The point of the exercise was that Olav and Chris made their decisions later where as the teams made their decision early even before knowing what the tasks were.

They then compared the risk profile of an agile project (many short increments) vs. the risk profile of waterfall project (one long increment). People who are risk averse prefer the waterfall model even though the agile risk profile appears safer.

Next they explain real options which is an approach that allow optimal decisions with the current context. It has two aspects: Math and Psychology. Using math to price real options is hard, but the results of the math tell us that:

1. Options have value

2. Options expire

3. Never commit early unless you know why

When making a decision, you can be right, wrong, or uncertain. People hate uncertainty so much that they will rather be wrong than uncertain. From a psychological aspect, rather than saying don’t make the decision now, say let’s make the decision under these conditions and circumstances (postponed till a future date) “let’s do it when…”

They wrap up by showing how real options apply in IT. No big upfront design and defer decisions too last responsible moment. Pair programming gives you options by sharing knowledge and making you less dependent on one person. Also, by assign juniors 1st to projects this leaves seniors (valuable skills) to coach and mentor with an option to work on emergency or high priority projects that come up later. When using an MS project plan, it gives us the zero probability date. Before that date there is no chance of delivering early. On that date you have zero percent probability of delivering and then the distribution builds from there. A study shows that to get 90% probability you need to multiply the given estimates by 4.

This presentation is available on InfoQ at  http://www.infoq.com/presentations/software-with-real-options

Coaching and Scaling Agility

At QCON San Francisco 2008, David Hussman gave a presentation on coaching. He defines coaching as helping plan products, helping with iterative delivery, helping tune and improve, and helping to build community. Coaching is about guiding people from how to why Shu, Ha, Ri or practice, improvise, and evolve. Coaching gigs and styles vary greatly. They can be prescriptive where this is what you should do or descriptive where this is what I have seen work.

When coaching large communities, it is important to understand that there is no recipe. Each community is unique. David recommends coaching respectful change where change must happen with people and not to them. Try to understand who people are, how they work right now, what’s working for them, and what motivates them to change. Try to find some set of practices, a way to bound as a community, some way to talk about products, some way to deliver it, and some way to tune it.

Then David talks about providing real education by building a library, providing pragmatic (need to try and experience things) training, coaching classes, facilitating training, and teaching TDD-refactoring.

David talks about chartering which covers who is involved, what we are trying to get done, what are the risks, timeline, etc… David uses chartering to find common goals and build a collective groove. Here is what we are trying to do, here is how we know when we are done, this is who is building it, etc...

Next David discusses scaling core practices. He recommends creating pragmatic product roadmaps (3, 6, 12, 18 months), pairing beyond programming (business person, testing person, and development person), radiating information, making issues visible, and promoting improvisation.

Then David describes two situations: many teams, many products or many teams one product. They need to be working in cross cutting concerns. Also we need to build customer communities composed of people that have interest, passion, knowledge, and some decision making capability.

Lastly, David discusses issues with large distributed communities. Conference calls are NOT just like being there. One option is building whole sub-teams around common goals and common understandings.

This presentation is available on InfoQ at http://www.infoq.com/presentations/coaching-scaling-agility

Behavior Driven Development

At QCON 2008, Dan North gave a talk about BDD. Dan start by stating that projects fail because the products come in too late, cost too much, do the wrong thing, are unstable in production and crash every day, break the rules, or the code is impossible to work with.

Next Dan compares the bottom up and top down approached to delivering software. Frederick Taylor’s scientific management theory kind of says that people are lazy and stupid and as a result we need to make their work simple and we need to constantly monitor them. We need to separate work from management. Top down process on delivering software is based on premise that people downstream are increasingly pluggable and prone to make mistakes so we need to front load our process with all the smart stuff so there is less risk of something going wrong as we go further along. Since the price of defects increase the later it is discovered, we need big upfront design(BUFD). Plan, requirements specs, functional design, detailed design, test plans is hedging against the exponential cost of change, but this is in turn creates it and thus creates a reinforcing loop. There is no cause and effect, they both cause one another.

Meanwhile Deming’s premise is that people generally want to do a good job and take pride in their work and they respond well to an encouraging environment. Bottom up process builds business objects to represent business domain and then produces an architecture to wire them together.

Next Dan shares statistics that say that 60% of features are never or rarely used, 30% are sometimes used, and 10% to 15% are often or always used.

Then Dan explains that Behavior Driven Development is about implementing an application by describing it from the point of view of its stake holders. Only focus on high-value features, flatten the cost of change of anything at any stage, prioritize often, change often, adapt to feedback and continuously learn. Pull requirements as needed, evolve the design, code that can change, frequent code integration, frequent regression tests

BDD is derivative from XP (TDD, CI), Domain Driven Design, Acceptance Test Driven Planning (estimation based on building one scenario on top of another), Neurolinguistic programming (NLP), systems thinking.



BDD is getting the words right, enough is enough and more is waste while less is irresponsible, agreeing on DONE, outside-in (more systemic approach), interactions (series of interaction between people with different skills and software is an output of these interactions). People over process.

Dan next elaborates:

Agilists and Architects: Allies not Adversaries

Martin Fowler and Rebecca Parsons gave a talk at QCON about architects in an agile environment. Architects worry about their jobs and where they fit in agile. Architects try to achieve reuse, documentation, see what is happening and agile can help architects by providing:

1. Transparency and visibility into progress and therefore can react in a reasonable way.

2. Up to date specification of functionality through strong emphasis on testing.

3. Results in adaptable code. Reuse after the fact rather than planned. You at least know that it is useful once and then adopt it in other places.

To make it work in agile, we have to look at architects as the customer. As a customer, architect need to prioritize decisions and need to specify what they mean and remove ambiguity. Architects also need to participate as team members. Architects need to be able to code. This will give them more information to be able to make adequate decisions. It will increase trust between them and the developers.

This presentation is available on InfoQ at  http://www.infoq.com/presentations/agilists-and-architects

Just You Wait

Kent Beck gave a talk at QCON about current trends and where they are going. He covers these trends in the following themes:

1. Communication: Information sharing (twitter), information collecting (logs, recordings), more frequent releases

2. Simplification: flat data(Amazon simple db, Google large table), Data parallel(Hadoop, Map reduce), Screen-less computing

3. Unintended consequences: Energy usage (small devices, sustainability), privacy (privacy is going away)

4. Disappearing: “Free” or differed revenue model (ads are out, need to pay for things we find valuable), reuse, status

5. New Approaches: design (good design valuable to enable frequent releases), tests (automated, need to catch mistakes early)

Kent wraps up by asking what have I accomplished in the past years and what will I accomplish in the years to come?

This presentation is available on InfoQ at http://www.infoq.com/presentations/just-you-wait

Crafting an Agile Adoption Strategy

Amr Elssamadisy gave a talk at QCON 2008 about agile adoption strategies. He started out by defining what is valuable? Is it being agile, design, requirements, time to market, user satisfaction, meetings? Value is different based on the context. It is what you want to get out of it. It should be your goal for adopting an agile practice. This is how you will measure your success.

Next Amr talk about a learning cycle of 1) goal, 2) process, 3) test, stop, and learn, apply lessons learned, and repeat until success. He then simplifies agile software development to recognizing and responding to change (learning, requirements, team structure, new practices), feedback practices and communication practices for the team (iterations stand up meetings, retrospective), and technical practices for developers (TDD, refactoring, pair programming).

Next Amr gives examples of some business values: reduce time to market, increase quality, increase flexibility, increase product utility, increase visibility, reduce cost, increase product lifetime. This is followed with some examples of some smells: Us vs. Them, Customer asks for everything, direct input from customer unrealistic, management is surprised, bottlenecked resources, churning projects, hundreds of bugs, hardening phase needed.

Each of these problems is addressed by different set of practices or patterns. A pattern is a problem solution pair within a context.

Amr then goes through some patterns:

1. Decrease time to market: We need to apply Iterations. For that we need a product backlog and a well defined definition of done. The backlogs improve product utility and increase visibility by enabling high quality feedback as the goals are met and reviewed regularly. They decrease time to market because the prioritized list enables negotiation of scope and early release with the most valuable functionality.

• a. Backlog context: You are on a team that decides to adopt agile including iterations. You decide to go away from big upfront design. Team has needed expertise to expand and evolve requirements
• b. Backlog adoption: Customer flushes out coarse grained requirements ahead of the iteration planning meeting. At the beginning of each iteration, team should have enough info to roughly estimate and begin development. Items chosen for development make up the iteration backlog.
• c. Backlog smells: Estimation paralysis, techies take over, multiple non cross functional teams have trouble working from one backlog

2. Increase quality: implement test driven development which needs refactoring which in turn needs collective code ownership and automated developer tests (Test first development or test last development). Refactoring increases flexibility and the product lifetime by enabling and encouraging developers to change the design of the system as needed. Quality to market and costs are reduced because continuous refactoring keeps the design from degrading over time, ensuring that the code is easy to understand, maintain, and change.

• a. Refactoring context: Development team is practicing automated developer tests. Requirement is not well supported by the current design or you want to make the design cleaner.
• b. Refactoring adoption: Just do it keeping in mind that it is a practice not a tool. Start automated developer tests until you are comfortable writing tests for all of the code. Adopt team code ownership. Agree on how to handle broken tests that result from refactoring. Read Martin Fowler book on refactoring and a book on TDD that is exercise driven.
• c. Refactoring smells: don’t get carried away and refactor just for the sake of refactoring. It does not deliver direct business value. Many missed small refactoring build up over time causing the need for a large refactoring. Beware of code ownership issues and pride that might lead to “commit wars”.

This presentation is available on InfoQ at  http://www.infoq.com/presentations/adopting-agile-practices

Responsive Design

Kent Beck gave a talk at QCON 2008 about Responsive Design. Kent starts out by defining design as beneficially relating elements. Design has elements. The elements have relationships, and the relationships can be beneficial or not. He states that there should be an ongoing investment in design. Design should happen at the right moment to enable a steady flow of features.

Kent moves on to briefly cover some design values, patterns, principles strategies, refactorings, successions and data that help achieve the goal of steady flow of features.

1. Values: simplicity, feedback, community

2. Patterns: most decision are not based on the domain, but are based on dealing with a computer. Having access to a wide library of patterns makes me more effective (vocabulary, efficiency). Do not waste time on originality for problems that do not require it.

3. Principles: There are universal principles like don’t repeat yourself, and then there more specific ones.

4. Strategies: Move your design in safe steps.

• a. Leap: break it up into tiny steps so you can make a leap in safe steps.
• b. Parallel: Operate 2 designs in parallel for a while.
• c. Stepping Stone: If cannot do something in a safe step, build a little component or service to help make progress towards a safe step.
• d. Simplification: Solve a simplified version 1st without any constraints, then slowly add constraints and continue to solve problem

5. Refactoring:

• a. Bidirectional: extract method and inline method, extract component and inline methods. All refactorings are bi-directional.
• b. Fluid: It is not from here to here, but more a sequence of steps.
• c. First class: Refactoring are 1st class.

6. Succession: important sequences of design that happen over and over again. Like if you know that you need to deal with n elements, deal with one element now and then transform it to deal with n elements when you need to.

7. Data: understand metrics patterns to justify advice.

Kent concludes by reminding us that the goal is to find a way to continually invest in design to more closely approximate this steady flow of features to create value.

This presentation is available on InfoQ at  http://www.infoq.com/presentations/responsive-design

Beyond Agile: Cultural Patterns of Software Organizations

At QCON London 2008, Marc Evers and Wilem Van Den Ende gave a talk about cultural patterns. They start out by mentioning that cultural patterns can help make sense of what’s happening. It helps us understand various subcultures that exist in an organization, and can predict conflicts. It helps put agile in perspective and helps us adapt our change strategy to particular situations. They then cover 6 cultural patterns:

1. Routine – We follow our standard procedures (except when we panic). Bring order to disorder. Management by controlling. Process oriented. Working in a well known context. It is very predictable, but difficult to improve productivity

2. Variable – We do whatever we feel like at the moment. Value Craftsmanship, foster innovation. It’s characterized by close cooperation between customers and developers, craftsmanship, hands off management. Performance and quality is totally dependent on individuals. It works when starting with a few customers developing custom built software until the number of clients increase to 8 or so. Once they go beyond that organizations switch to routine. But when something goes wrong, then they go back to variable. Variable produces fast delivery and good relationship between customer and developer.

3. Steering – We choose among our routines by the result they produce. Make extraordinary things ordinary. It’s characterized by feedback control, results oriented, trust based, act early, act small, XP, and Scrum. Testing and feedback play an important role. It uses feedback. You can improve as you go. Can be light weight.

Moving towards steering:

a. Mental models: Have more work to do. Increase hours/week to try to finish work. However, more hours per week lead to fatigue which results in more defects and thus increases work to do.

b. Visibility: Charts and boards on the board.

c. Stability: Need to have a stable velocity.



4. Oblivious – We’re not aware that we’re developing software. No separation between user and developer. Highly adaptive, highly customer oriented. Customer and developer is the same person. You always get what you want as long as you can make it

5. Anticipating - We establish routines based on our past experience with them. The art of the long view. Pay more attention to long term planning and changing your processes. Consciously managing change, process oriented, always improving your processes (if it ain’t broke, fix it). Practices (retrospectives, scenario planning, risk management). Lean Software development. It makes steering culture more predictable with a conscious process of managing change

6. Congruent – everyone is involved in improving everything all the time. It is a culture of ongoing reflection and improvement.


They see some similarity with CMM where level 0 is Oblivious, 1 Variable, 2 Routine, 3 Steering, 4 Anticipating, and level 5 Congruent. They conclude by stressing that you should find the patterns that fit your context.

This presentation is available on InfoQ at http://www.infoq.com/presentations/beyond-agile

Born to Cycle

Linda Rising gave a talk at QCON London 2008 entitled “Born To Cycle”. This talk is very similar to “Perfection is an unrealistic goal”. Linda does add a couple of interesting experiments:

1. Rats in a maze. During REM sleep rats were transferring information about what they learned about the maze. The next morning the rats navigated the maze and found the cheese.

2. Two groups (humans) were taught a task. The group that took a nap improved more than the group that stayed awake. After a night’s sleep, both groups were at the same level.

3. One group was taught a task and then 6 to 8 hours later, a second task. The next day, the group did not improve on either task.

4. Two groups were taught a task. Both were taught a second task, but one group took a nap in the interim. No improvements were noticed later in the day, but the next morning, the nap group had improved at both tasks.

This presentation is available on InfoQ at http://www.infoq.com/presentations/born-to-cycle

Agile Mashups

Rachel Davies gave a talk about Agile Mashups at QCON London 2008. Rachel mentions that organizations trying to adopt agile can be confused by the different agile methods (Scrum, XP, Crystal, DSDM, Lean). These methods are simplified ideas and practices to make it easy to transmit and understand. It will help teams get started but they have to fill in the gaps. A lot of teams create their own agile combinations. A good way to do that is through retrospectives.

Next Rachel goes over some typical agile practices like standup meetings, sprints, user stories, release plans, TDD, velocity, burndown charts, team boards, retrospectives, continuous integration. Most teams adopt these practices but struggle with pair programming, product increment, and collocation. Often the teams are self organizing and cross functional and include 5 to 10 members with at least one tester. The customer role is split between someone who is making priority calls and someone who is explaining the domain. Agile project manager or scrum master are responsible for facilitating meetings, shielding the team, working with the team to remove obstacles. Sometimes they also do project management activities like reporting progress and preparing the road ahead.

Rachel stresses the importance of setting up a visual space to see what the team is working on at the moment. The team gathers around the project board for their daily stand up meeting.

Projects start with an iteration Zero that creates the release plan, sets up an infrastructure, architecture, and for some initial estimation. The typical cycle is 2 weeks. Half of 1st day is spent planning, then development, and at the last day there is a demo and then a retrospective. A lot of teams start the sprint on Thursday. Lots of teams have a preplanning meeting. The meeting at the beginning of the iteration is for estimation and task creation while the meeting to decide what goes into a sprint happens a couple of days earlier. Demos need to involve product owner and a wider set of stakeholders. It’s ok to have breathing spaces between iterations. It is also ok to have polish iterations before a major release.

Rachel wraps up by recommending that we read multiple agile books and use them as a source of ideas and not as “religious” texts. Projects are varied and each might need a different approach. We need to inspect, adapt and evolve.

This presentation is available on InfoQ at  http://www.infoq.com/presentations/agile-mashups

Agility: Possibilities at a Personal Level

Linda Rising gave a talk about personal agility at QCON San Francisco 2008. The talk was about the effects and history of caffeine. In the industrial age, people were having caffeine for breakfast. They boiled the water to make it safe to drink and then had coffee or tea. Coffee, tea, clock, and factories appeared at the same time. Before that we used to have beer for breakfast (Europeans were consuming 3l beer /person/day). We used to wake up and sleep based on the sun and seasons. We now have to adapt and cope with a work schedule set by a clock and not daylight or the natural sleep cycle. Inventions of the clock and availability of caffeine changed lives.

Next Linda explained that caffeine blocks the effect of adenosine (one of the body’s natural sleeping pills) and keeps us awake. The average person takes about 3.5 hrs to metabolize caffeine. It takes longer for thinner, or pregnant women. Newborns cannot metabolize caffeine. Nicotine moderates the mood, extends attention, and doubles the rate of caffeine metabolism.

Then Linda mentioned that without adequate sleep, we are not at our best, physically, mentally or emotionally. We have come to believe that sleep is a waste of time and makes us overall les productive. As a result, we are sleep deprived and our brains show visible signs of premature aging.

Linda shared research that shows that caffeine is not better than breaks. It improves “vigilance tasks” – prolonged attention, little physical activity. But a good night’s sleep improves performance, mood, and alertness better than caffeine and the benefits last longer. For simple tasks, caffeine improves performance. But on complex tasks, extroverts’ performance tends to improve, while introverts’ tends to get worse.

Finally, Linda showed how spiders on different drugs performed (marijuana, chloral hydrate, Benzedrine and caffeine). The spider web of a spider on caffeine had no architecture, no regularity, no structures what so ever and was rather erratic.

Linda wraps up by questioning if agile is the new caffeine. It is energizing, stimulating, fun and addictive. It also has side effects: irritable, restless, anxious, and sleepless and teams get themselves into a lot of hot water! Is what is good for teams good for us? Linda wonders if we can do a better job instead of living our lives as we did in the industrial age.

This presentation is available on InfoQ at http://www.infoq.com/presentations/agility-personal-level-possibilities

Teamwork is an Individual Skill

At QCON 2008, Christopher Avery talked about teamwork. He started out by stating that we all live in a world of shared responsibility. A team is a result. It emerges from an opportunity for a shared responsibility. He explained that the biggest problems are within departments and other departments, within team functions and other functions. This gives the attitude of you are my problem and I am your problem. Agile is moving to a different culture of

1. Collaboration, partnering, trust

2. Openness, transparency, visibility

3. Adaptive, iterative, evolving

4. Awareness, learning, facing reality

5. Humaneness & performance

Chris then demonstrated playing 4 by 4 tic tac toe with the goal of maximizing your score while taking turns. The game showed that the only way to maximize your score is to maximize the other’s score. It does not have to be win lose. It can be win/win

Next, Chris covered the 3 phases of power from the power of economics and organizing

1. Power over: authority power

2. Power to/by: exchange power – power of the vote, power of the budget, power to barter

3. Power with: Integrative power – ability to use only ideas and actions to attract other people to you to accomplish something far greater than you could do by yourself.

We want more of power over and power to rather than power with even though they are scarce and limited, while power with is available in virtually unlimited abundance.

Chris mentioned that we have far more power and ability to get things done, produce value, and make changes than we give ourselves credit for.

Managers in Scrum

Roman Pichler gave a talk at QCON 2008 about the role of managers in Scrum. He starts out by describing a typical organization as a hierarchical structure with a command and control management paradigm. Managers receive reports, make decisions and give orders. Subordinates comply with orders and execute and then report back. Managers are removed from the day to day work and lose touch with what is going on and makes it difficult to make decisions. Managers are accountable for decisions and results. Subordinates are doing things they might not always agree with. Authority and responsibility are separated across 2 roles. This makes it difficult to have buy-in, accountability, ownership, and to learn and improve.

Next, Roman mentions that under Scrum the outlook on management is different. It is about helping the people doing the real work in adding value to the product, and creating the right environment for them to succeed. He then covers some management practices in Scrum:

1. Servant-leadership: lead others by serving them. Serve 1st, lead 2nd. Help the team and its members grow and develop. Respect individuals, honor effort and goodwill. Help create the right work environment.

2. Empirical Management: As a manager, make decisions on the basis of facts and empirical evidence. Report and numbers are not sufficient. You need to see what is happening for yourself. Create transparency and be able to inspect and adapt. Managers engage with employees to understand what’s happening where the actual work is done. Ask questions, share observations, make helpful suggestions to assist and guide but do not micro manage.

3. Empowerment: Delegate decision making authority to the lowest possible level. Collaboration instead of command and control. Authority and responsibility are united.

4. Quality first: Quality is built into the product from the start. Encourage and empower the team to identify and rectify problems together with their root causes.

5. Continuous improvement: Challenge the status quo on an ongoing basis. Identify and remove wasteful activities. It is about continuous innovation and change. It is a learning non-judgmental, non blaming approach.

6. Standarization: Standards developed by the team and then communicated to the rest of the organization

Next Roman goes over some transition techniques:

1. Focus everyone on the customer needs. Consider the entire value stream and a void sub optimization.

2. Systematically remove overburden: Overburden decreases morale and robs people of creativity. Give people slack so they have time to reflect and continuously improve. Avoid overburden by limiting demand on capacity and capability.

3. Promote team work: help create effective teams and foster creativity and collaboration

4. Clear the way: remove impediments promptly and anticipate new impediments.

5. Be a Scrum champion: teach, encourage and guide. Be a role model and walk the talk.

Finally, Roman summarizes by mentioning that the good news is that there is plenty left for managers to do in Scrum. The Management culture must change profoundly from telling people what to do to supporting and guiding them.

This presentation is available on InfoQ at http://www.infoq.com/presentations/managers-in-scrum

A Kanban System for Software Engineering

At QCON London 2008, David Anderson gave a talk about Kanban. David starts by mentioning he had some failures in institutionalizing agile changes and failing to scale it to a significant size. He then had success with Kanban. Kanban system allows for focus on quality, reduces or limits work in progress, balances demand against throughput, prioritizes.

David then describes a case study at Microsoft where a team was at CMMI level 5 and producing high quality software however they had a huge backlog and items constantly delyed. David shows how implementing Kanban helped this team improve productivity by 200%.

Instead of having an agile transition plan and force a team to use a particular agile process, David now advocates starting from where you are now and create a culture that encourages people to improve, teach them lean principles, teach them about waste and bottlenecks, and then have the team figure out on how to get better.

David then covers another case study at Corbis and describes Kanban in more details. Kanban can work with specialist teams of analysts, developers, testers, QA, etc. However, it puts a limit on the amount of items that each team can process. There are no iterations in Kanban. It is more of a continues flow with a release every 2 weeks, but the release content is bound and published only 5 days prior. Items that take longer than 2 weeks are just not released. Prioritization meetings are held every week (inputs and outputs are not in sync in a cycle). Whenever there is an empty slot, an item from the backlog is added. Prioritization and competition for an empty slot eventually evolves into collaboration. No estimation is done on individual items, with the effort to estimate turned back into increased productivity in analysis, coding, testing, etc. The Kanban white board gives visibility into process issues (transaction cost of releases, transfer though stages, bottlenecks, ragged flow). Daily stand up helps eliminate impediments affecting productivity and lead time. Kanban also allows for scaling standup meetings. Large teams can go through tickets on the board and ask if anyone knows of something impeding the progress of the ticket.

David summarizes that the Kanban method enabled:

1. Culture Change: trust, empowerment, collaborative team working and focused on quality.

2. Policy Changes: No estimating, late binding release scope, late binding prioritization

3. Regular delivery cadence: Releases become routine.

4. Cross functional collaboration

5. Self regulating process robust to gaming and abuse

6. Continuous improvement: Increase throughput, high quality, process continually evolving, kanban limits empirically adjusted.

7. Little Management Overhead: Little to no involvement in day to day

This presentation is available on InfoQ at  http://www.infoq.com/presentations/kanban-for-software

Concurrency – Past and Present

At QCON London 2008, Brian Geotz gave a talk on concurrency. Brian starts by mentioning that concurrency is hard. It is unnatural, error prone and untestable, hard for most programmers to use. He then explains the basics concepts of concurrency. A thread is a sequential process with its own program counter and call stack. Thread share VM wide resources such as memory, file handles, and security credentials. The pros of threads is fine grained data sharing between threads. However, that is also its disadvantage. Threads execute concurrently unless you use explicit locking to ensure that threads take turns accessing critical resources (synchronize key word in java).

If mutable data is shared between threads, all access requires synchronization. Failing to follow this rule results in data race which results in reading the wrong value or writing the wrong value. This generally occurs whenever there is a if something then do something or ready-modify-write. In the 1st case, another thread might change the value between the time we do the check and the time we call the method. In the second case, 1 statement is actually 3 statements like x++. Again here another thread might read the same value and then increment it by just 1 when it should have gotten incremented by 2.

Adding synchronized on methods in the class that modify data is a start (account with debit/credit), but then we have to be aware of the locking mechanism when composing operations in other classes (accountManager with transfer).

Programmers have to specify which operations are required to be atomic. But when doing so, this might lead to deacklock. Example, transferMoney(me, you, 100) and transferMoney(you, me, 50). One way to deal with this, is to induce a lock ordering. Example, always acquire lock 1st on the smaller value.

If from

synchronize(from) {

synchronize(to) {

}

}


Next Brian explains how this is getting more and more complicated. The main reason is because there is a fundamental tension between concurrency and OO design. OO encourage encapsulation and hiding of implementation details. OO also encourages composition, but composing thread safe objects requires knowing how they implement locking in order to participate in their locking protocols and know how to avoid deadlocks. However, the language is hiding the implementation details.

Next Brian offers some solutions to this problem. One solution is software transactional memory (STM). Explicit locks are replaced by transactional boundaries that define atomic blocks. The VM then provides locking nesting semantics and choose a locking strategy. This is under research and promising but not yet available.

Threads and locks are just one model for concurrency. Lock based concurrency rules hold locks when accessing shared mutable state and hold locks for the duration of atomic operations. Alternatives are don’t mutate state or don’t share state. Functional languages like Haskell or JOCaml have no mutable state. In java, we should get in the habit of making everything final unless you have a clear need to make it mutable. In Erlang everything is an Actor (like a lightweight thread). It has a designated behavior for when a message is received. No shared state. Scala has a similar actor library.

Brian concludes that in java, we can try to restore predictability by limiting concurrent interactions to well defined points, limiting sharing, limiting immutability. Concurrency is hard, so minimize the amount of code that has to deal with concurrency (isolate concurrency in concurrent components such as blocking queues and isolate code that accesses shared state in frameworks), use immutable objects wherever you can. Sometimes it is cheaper to share a non thread safe object by copying than to make it thread safe.


This presentation is available on InfoQ at http://www.infoq.com/presentations/goetz-concurrency-past-present