Friday, August 29, 2008

Agile and Beyond – The Power of Aspirational Teams

At Agile 2008, Tim gave a talk on appreciative inquiry. Tim starts by defining appreciative inquiry as the study and exploration of what gives life to human systems when they function at their best. It suggests strengths, hopes and successes are transformational. We control a lot more of our destiny than we are aware of. Agile is similar. It is not just about process and practices but is a belief in achievement. Software development always brings up bad news. We need to think of what we do good in software.

Next Tim asks who defends the honor of the agile manifesto and principles. It’s easy to just adopt some of the principles and skip others. It is easy to assume that we cannot do all the practices and then not aspire for more. The best teams adopt all practices because they know that all the practices are dependent on each other.

Tim then covers several areas to influence appreciative inquire:

1. Working together: Pulling the load together, having fun, being excited about achieving.

2. Project chartering: It helps team understand the parameters you work in like defining success and determining who is part of the community. It also helps self organize over a common vision and goal and forms a contract that binds together the rules of the teams/practices.

3. Group sharing: encourages new ideas, gets more people involved in work.

4. Story cards and planning boards. Puts focus on collaboration. Use magnetic boards, make it fun, try magnetic avatars. Also, instead of statuses of in development, in test, in QA, use started, in progress, done which encompasses the different roles. Story cards are placeholders for a bigger conversation. Important to follow format of Story cards “as role, I want, so that”. Also get good pens to encourage good card writing (dark and thick ). Physical cards much better than huddling over a spreadsheet. If not hand written, print them out.

5. Visualize progress - Burn down chart vs. burn up chart. In general we like to see thing go up kind like a profit graph. For forecasting, simply counting story cards done vs. story cards not complete can be enough. Graph can still be drawn manually.

6. Gold cards: give 2 per developers to use for working on areas where they can innovate.

Thursday, August 28, 2008

Concurrency – Past and Present

Java Concurrency in PracticeAt QCON London 2008, Brian Geotz gave a talk on concurrency. Brian starts by mentioning that concurrency is hard. It is unnatural, error prone and untestable, hard for most programmers to use. He then explains the basics concepts of concurrency. A thread is a sequential process with its own program counter and call stack. Thread share VM wide resources such as memory, file handles, and security credentials. The pros of threads is fine grained data sharing between threads. However, that is also its disadvantage. Threads execute concurrently unless you use explicit locking to ensure that threads take turns accessing critical resources (synchronize key word in java).

If mutable data is shared between threads, all access requires synchronization. Failing to follow this rule results in data race which results in reading the wrong value or writing the wrong value. This generally occurs whenever there is a if something then do something or ready-modify-write. In the 1st case, another thread might change the value between the time we do the check and the time we call the method. In the second case, 1 statement is actually 3 statements like x++. Again here another thread might read the same value and then increment it by just 1 when it should have gotten incremented by 2.

Adding synchronized on methods in the class that modify data is a start (account with debit/credit), but then we have to be aware of the locking mechanism when composing operations in other classes (accountManager with transfer).

Programmers have to specify which operations are required to be atomic. But when doing so, this might lead to deacklock. Example, transferMoney(me, you, 100) and transferMoney(you, me, 50). One way to deal with this, is to induce a lock ordering. Example, always acquire lock 1st on the smaller value.

If from

synchronize(from) {

synchronize(to) {

}

}


Next Brian explains how this is getting more and more complicated. The main reason is because there is a fundamental tension between concurrency and OO design. OO encourage encapsulation and hiding of implementation details. OO also encourages composition, but composing thread safe objects requires knowing how they implement locking in order to participate in their locking protocols and know how to avoid deadlocks. However, the language is hiding the implementation details.

Next Brian offers some solutions to this problem. One solution is software transactional memory (STM). Explicit locks are replaced by transactional boundaries that define atomic blocks. The VM then provides locking nesting semantics and choose a locking strategy. This is under research and promising but not yet available.

Threads and locks are just one model for concurrency. Lock based concurrency rules hold locks when accessing shared mutable state and hold locks for the duration of atomic operations. Alternatives are don’t mutate state or don’t share state. Functional languages like Haskell or JOCaml have no mutable state. In java, we should get in the habit of making everything final unless you have a clear need to make it mutable. In Erlang everything is an Actor (like a lightweight thread). It has a designated behavior for when a message is received. No shared state. Scala has a similar actor library.

Brian concludes that in java, we can try to restore predictability by limiting concurrent interactions to well defined points, limiting sharing, limiting immutability. Concurrency is hard, so minimize the amount of code that has to deal with concurrency (isolate concurrency in concurrent components such as blocking queues and isolate code that accesses shared state in frameworks), use immutable objects wherever you can. Sometimes it is cheaper to share a non thread safe object by copying than to make it thread safe.


This presentation is available on InfoQ at http://www.infoq.com/presentations/goetz-concurrency-past-present

10 Ways to Screw Up With Scrum and XP

Scrum and XP from the Trenches (Enterprise Software Development)At Agile 2008, Henrik Kniberg gave a presentation on 10 ways to screw up despite Scrum and XP

1. Believing the Hype: That is, believing that if you go agile, all problems will disappear. You throw out everything you are doing now even if it works and then get the latest and best tools and focus on perfection.

2. Definition of Done: But you said you were done? Not having a definition, or not obeying the definition, or having done be outside of the teams control (no access to production). Good examples of Done are: unit and integration tested, acceptance tested and deployed to demo site. Or Releasable (acceptance tested, release notes written) and no increased technical cost.

3. Velocity: Not having one makes it hard to create a release plan. Having one and misusing it by comparing different team velocities. Or having a yo-yo velocity where it is high one sprint and very low on another sprint (indicates spending a lot of time fixing bugs).

4. Retrospectives: We are very busy so we skip it. And if we do it, changes and recommendations are ignored and not implemented. Having unwanted people in the meeting where the team feels uncomfortable and do not participate. Punishing the team for bad changes (the whole point is to try something and later evaluate it to see if things improved).

5. Team commitment: Team is pressured, Team is not sitting together, team does not track and learn and keeps repeating the same mistakes. Team is always undercommiting (team always looks good) or always over committing or has a velocity of 0 (doing a lot of stuff but not finishing stories to the end).

6. Technical Debt: No time to write unit tests or refactor code. This results in duplicate code, unreadable code, lack of code coverage and all add to technical debt. Technical debt slows team down. Things become harder. This also leads to inappropriate solutions like a big bang re-write or fixing the product instead of the process.

7. Teamwork: Fixed roles (no one helping anyone else), personal backlogs (results with attitude of at least I finished my stuff!), not helping each others, personal incentive models, implementing all stories in parallel, management interference (team not self managing).

8. Product backlog/product owner and customer: Product owner does not have time to maintain product backlog so team does not have one or it is not prioritized. Product owner does not know the system or does not have the authority and thus cannot make decisions. Having multiple product owners that do not agree with each other. Product owner becoming a bottleneck where no one can do something unless they talk to the Product owner 1st. Product owner not always available.

9. Mergophobia: Fear of merging code. Not integrating early and often, No branching policies, not taking responsibility (everybody ignoring team rules regarding code checkin),

10. Spring backlog or taskboard: Does not exist, too far from the team (not easily accessible), too complicated (many states to keep track off ) result in it not being used during daily scrum, not owned by the team, no burndown, not updated daily, warning signs ignored.

11. Worry about 1 – 10. Problems are normal. Don’t panic. Don’t despair. Tackle them gradually, prioritize and try to improve sprint after sprint.

This presentation is available on InfoQ at http://www.infoq.com/presentations/Fail-Scrum-Henrik-Kniberg

Sunday, August 24, 2008

Future Directions for Agile

At Agile 2008 David Anderson talks about agile principles, practices and the community. David covers multiple topics that reveal how CMMI, Kanban and Real Options Theory help in making organizations scale, lean, and of high maturity.

David mentions that trying to scale agile across an enterprise is forcing a change on the community. We are trying to expand agile to the entire IT department and even more to other areas like marketing and recruiting. We need a wider definition of agile that is not narrowly focused on software development. We have the manifesto and the principles, but it is 7 years old and in badly need of an update. David stresses that we should not underestimate the value of what the manifesto gave us, but it is not everything of what it means to be agile. David controversially states that the agile manifesto is superstitiously declared.

Next David clears up the ideas behind the manifesto and how can we capture them as a model in 3 underlying paradigms

1. High trust, high social capital, highly collaborative culture

David describes the different cultures between the traditional and agile communities. The traditional community was rooted in government and developed a body of knowledge from defense. The high cost of failure (money, lives, politics) makes people conservatives, risk averse, and have low trust. On the other hand, the agile community was started by thought leaders from start-up companies building web sites. They were innovating daily, empowering people to release new functionality with an assumed zero cost of failure. They realized that in order to innovate at a fast pace, there was no time to write requirement and negotiate contracts. It was better to let people work and adopt a failure tolerant attitude. This was a highly collaborative approach with a high trust liberal culture. This led to the 3rd value of the agile manifesto, customer collaboration (high trust) over contract negotiation (low trust). Self organization or empowering, delegating and tolerating failure through adoption of high trust culture is better than contract and audits in low trust culture.

2. Perfect is the enemy of good enough.

David mentions that CMMI taught us that project success came from organizational maturity. Level 1 is chaos, Level 2 is repeatable and Level 3 is a defined process. At this level we should be able to deliver successfully every time. So the assumption was that we need to pursue perfection and eliminate ambiguity and uncertainty. We need accurate plans, analysis, estimates, and diagramming. By the time we are done, they are already out of date. This led to the agile manifesto value of working software over comprehensive documentation.

Tuesday, August 19, 2008

“We Suck Less” is Not Enough

At Agile 2008, David Douglas and Robin Dymond described the current state of agile adoption and discussed what the legacy of agile will be. They mentioned Forester research from 2007 that says 17% of North American and European companies and are now using agile. They however noted that there is a lot of confusion on what agile really is and described the Nokia test to find out if a team is agile:

1. Iteration must be time boxed to less than 4 weeks

2. SW features must be tested and working at end of each iteration

3. Iteration must start before spec is complete

4. Do you know who the product owner is

5. There is a product backlog prioritized by business value

6. Product backlog has estimates done by team

7. Team knows there velocity

8. There are no managers disrupting the work

They then warned that the attitude of we suck less now than before sets the bar really low. The business will be satisfied and will stop when things get a little better than before, but they are not even close to how much better they can get. Moving from ok to great is a journey that takes time and effort. There are levels of agility that companies can get to based on their level of commitment.

Next they mentioned that there a perception that agile is just a set of tools and practices like Rapid Application Development or that it is cheap, easy, and runs on its own, or that it is a project solution instead of a system solution. Agile is an entire work system. It can produce 5x+ productivity gains, requires organizational redesign, requires significant change management component. We as agilest are currently feeding this perception by training teams on techniques and consciously encouraging growth over quality.



Next, they describe several large organizations that adopted agile and then moved away from it. There were early warning signs of agile adoption stagnation like early declaration of success, increased time slicing and borrowing developers for “important” project, lack of commitment or investment, flight exodus of change agents, executive support waning, continued emphasis on cost reduction without the corresponding ROI and continual disconnect between IT and business and a lack of understanding on how IT delivers value to the business.

Sunday, August 17, 2008

The Development of a New Car at Toyota

Inside the Mind of Toyota: Management Principles for Enduring GrowthKenji Hiranabe gave a presentation at Agile 2008 about Lean at Toyota. He talks about Nobuaki Katayama, former Chief Engineer at Toyota, and how they build new cars. There are 3 phases to car building:

1. Planning and concept development: concept, style, market research, pre-development, cost and profit target. Spend a lot of time in this phase. Start out with a good concept, then visit dealers and users and get feedback to see if this concept work and then redo. The next 2 phases are management of milestones.

2. Real car development: Design prototyping, evaluation

3. Production and sales:

Managing big projects:

1. Planning and concept management: project backbone, clear appealing points are needed, 1st concept and plan has to be refined until satisfaction, discuss multiple opinions from wide views, rules to avoid immature “go”, calm evaluation with 3rd person

2. Development process management: needs to be a strong dedicated leader with org that supports him, with clear milestones, and supporting people to achieve them, rules to avoid proceeding before achieving them, mechanics to avoid “too late”, trigger of 3rd person, honest communication (face to face), bad news 1st rule, early and reliable backup plan.

3. Resource management: people and money, work hard does not work, visualization estimate based on actual record, make projects slim. Cost: Visualize the target cost, leader owns buffer, assign target cost based on actual record, value sense of completion, avoid sense of being “forced to do”, plan in detail and follow up and adjust seeing the whole

4. Cost management: Reduce cost means save profit so that the company can grow, everyone and every division think about global competition and cost reduction. Culture of do not complain but cooperate. Analyze past and present cost. Use it as base cost data. Reduction of cost is accumulated step by step by everybody. Value engineering is the technical competence. Each team is responsible for reducing cost.

5. Benchmarking: it’s about knowing competitors strategy. Not copy others techniques, but focus of parts that are better than ours. Set performance targets including ahead and behind time difference. Study designs of 2 to 3 years ago and develop product 1 to 2 years away. Use it for continuous competiveness.

6. Leader attitude: Leadership, teamwork, macroview, passion are good. Dictator, one man shop, leave all to subordinates, charismatic are bad for leadership.


Kenji then contrasts “do it until it’s done” vs. “do it until the time”. Mr. Katayama explained that for phase 1 use do it until it is done and for phases 2 and 3, do it until time.

Kenji wraps up by mentioning the Nobuaki Katayama values consistency and integrity over passion and strong leadership, values engineering thinking, and tries to understand new ideas.

This presentation is available on InfoQ at http://www.infoq.com/presentations/Toyota-Kenji-Hiranabe

Saturday, August 16, 2008

The Yawning Crevasse of Doom

Patterns of Enterprise Application ArchitectureMartin Fowler and Dan North gave a talk at QCON London 2007 about communication gaps.

They started about by describing the cost of communications gaps between customers, users and developers. This cost is obvious in systems that completely fail, but sometimes the cost is not apparent like when about 70% of features delivered are hardly or never used. In this case there is also a cost of failed opportunity to create something more valuable instead. These are costs that we do not even notice.

They suggest 2 approaches to close the communication gap:

1. Ferry men approach: A group of people act as intermediary. They are ferried between the business folks and the developers and act as translators. The business people never talk to the developers.

2. Bridge approach: Provide structure and mechanism for direct communication between developers and business people.

Dan and Martin prefer to use bridges than to ferry men. Dan gives an example of how different solutions can result out of this direct communication or even entire new requirements can appear. He recalls working on a project were the business folks required some printing functionality. When probed, it was discovered that printing was needed to key in data from the printout into another system. Instead of printing and rekeying, Dan suggested to simply sending the data directly to the other system. This was a simple solution that would not have come out if there wasn’t any direct communication so that the developer could really understand what the problem is.

They give 2 reasons for preferring the bridge or the ferry.

1. Efficiency of communication: Use analyst to enable everyone to share knowledge instead of just being a conveyer of information.

2. Caring and Motivation. Having direct contact, you can actually see who you are making a difference for. That is more motivating then looking at a requirements document.

Next, Dan and Martin discuss what techniques effect communication and help us build the bridge:



Sunday, August 10, 2008

Succeeding with Agile: A guide to Transitioning

Succeeding with Agile: Software Development Using ScrumMike Cohn gave a talk at Agile 2007 about how to succeed in Agile based on his experiences with different clients transitioning and successfully transforming their organization.

Mike starts out by addressing why organizational change is hard:

1. It’s not top down where a strong leader sets a vision and people follow it, and it is not bottom up where team start and others follow until you run into groups that are powerful enough to squelch transition.

2. Best practices are tempting, but once we identify something as best, we stop changing. We have to keep inspecting and adapting otherwise we are not agile.

3. The transition has to also go with a transition in the development process

4. We cannot predict how the organization is going to respond. We want to effect the organizational change in an incremental manner.

Agile Estimating and PlanningMike describes how if we break things up into small pieces, work on the smaller more manageable pieces and then try to put them back together we do not necessarily get the whole picture. We need to think of organization as complex adaptive system. We need to look at success as achieving a good fit with the environment instead of as closing the gap with the desired state. We have to acknowledge that agents are closing local goals and closing local gaps. They are thinking about their own job security and their own vacation schedule. It is ok to set vision, but each agent is pursuing their own local goals and they are taking local actions and they all interact.

Mike next contrasts the traditional model of change vs. the complex adaptive model for change.

1. Traditionally, behavior is controllable and predictable. In the complex model we cannot control behavior.

2. Traditionally, we can establish directions. . In the complex model direction is established by the moving groups.

3. Traditionally there is clear cause and effect. In the complex model, there is a ripple effect.

4. Traditionally, relationships are directive. In the complex model, relationships are empowering and can go in both directions.

5. Traditionally, focus is on efficiency. In the complex model we are more interested in responding to change.

6. Traditionally, decisions are based on fact and data. In the complex model, decisions are based on patterns and tensions between groups.

7. Traditionally, leaders are the experts. In the complex model, everybody can help lead. Leaders are there to facilitate and support.

Friday, August 1, 2008

Developing Expertise: Herding Racehorses, Racing Sheep

Dave Thomas gave a talk at QCON London 2007 on how teams can write software better. He starts out by saying that picking a new tool or methodology will not help unless we address the root of the problem. That is, software is not written by tools, frameworks, methodology, or languages. It is written by people. In the 70s, there were 5.5 bugs per 1000 lines of code. In the 90s there were 5.6 bugs per. Even though languages changed (from cobol to c++), methodologies changed (from adhoc to PMBOK), and tools changed (from punch cards, to IDEs), there was no improvements in the number of bugs. So to make the software better, we need to make the people better.

Next, Dave looks at how to make the team better and improve their performance. Dave mentions that teams consist of members with different backgrounds and expertise, yet we like to treat them all the same. They all follow the same procedures even though they work differently and have totally set of needs.

Dave then explains the Dreyfus model for skill acquisition and its stages. The model suggest that most people start out as a novice needing rules and procedures until they gain experience and then work of intuition. Intuition is knowing something without knowing how you know it. Things just pop into your head and you have no idea where they came from. The model defines 5 stages:

1. Novice: You are doing something for the 1st time. You have no idea what you are doing. No interested in learning. You want detail instructions on how to solve the problem and get the job done. You are vulnerable to confusion and don’t know how to respond to mistakes. Novices require short term goals. As you do something over and over again you assimilate it (kind of like driving). It become below the level of consciousness and you get to become an advanced beginner.

2. Advanced beginner: you start trying things on your own, but still need to be told what to do. You have difficulty troubleshooting. You want information fast. You can place some advice in context. You begin formulating some principles but without holistic understanding. You are too busy concentrating on what they are doing. They cannot step back. Things that can help are pairing programming and unit testing.



3. Competent: You have big picture. You can do things for yourself. You seek out expert user advice. You are conscious and look for long terms plans and goals. You can troubleshoot on your own.

4. Proficient: You realize that there is a bigger domain. You are frustrated by oversimplified information. You will self correct previous poor task performance. You will learn from experiences of others. (It is like 3 year old always asking why?)

5. Expert: You are a primary source of knowledge and information. You continually look for better methods. You work primarily from intuition not reason. Forcing to follow set rules degrades performance. You require autonomy. You require the big picture.