Is Being Agile Your Favorite Thing?

Here's a look at a team that loves being Agile...

Measuring Agile in the Enterprise: 5 Success Factors for Large Scale Agile Adoption

At Agile 2008 Michael Mah discussed Measuring Agile in the Enterprise. He starts by mentioning that Business will have to soon make a decision to reduce costs. 1) ship development to India, 2) adopt agile or 3) ship to India and adopt agile.

Michael then shares metrics (size, time, effort, defects, and productivity) collected from agile projects against industry averages. The data was mainly from BMC and Follet and looked at cost, schedule, defects, and staffing showing better results for the mature agile teams. Then Michael did a trendline assessment of staffing, schedule, effort, and defects. The data showed the agile project team size is fairly typical, bug rates are significantly lower (they start out higher but fall way down as team matures), and as a whole achieve faster speed. Productivity index (size/(time*effort)) improves the longer you’ve been doing agile, so the sooner you start the better.

Finally Michael interviews Walter Bodwell and describes the BMC secret sauce for success:

1. Buy in: need VP or higher senior executive sponsor, scrum master training, and core team energized and passionate

2. Staying releasable: have nightly builds/tests, 2 week iteration demos, frequent, rigorous code reviews

3. Dusk to dawn team work: Communication techniques for information flow, Wikis, video conferencing, periodic onsite meetings, collocated release planning, scrum of scrum meetings

4. Backlogs: One master backlog and multiple backlog management, one setup for user stories across teams, requirements architect to interface product management and R&D

5. Holding back the waterfall: Use Test Driven Development, retrospective meetings to not regress into old waterfall habits, outside source to audit the process

 This presentation is available on InfoQ at http://www.infoq.com/presentations/5-Success-Factors-Michael-Mah

Forging a New Alliance

At ThoughtWorks quarterly conference, Scott Shaw and Martin Fowler discussed the communication gap that exists between IT and the business side. Originally, IT department was created to centralize desktop administration and to gather MIS data. IT managed billing systems, accounting system or data processing on a centralized computer system. Things changed with the introduction of the PC. A broader range of business functionality came under IT functionality. Martin predicts that custom application provision will see IT department effectively disappear. Commodity standard stuff will move to the cloud and get hosted in a data center while custom software will move closer to the business side and the way to thrive in the IT world is to move closer to the business people.

Next they discuss some disturbing trends in IT that result in this communication divide between business and IT.

1. Slow to market: They note statistics that show about 34% of IT projects are successful. Here success is defined as on time and on budget. Martin notes that success should really be measured as providing value. They also show an input value stream that shows a typical example in which out of 90 days, 18 days are spent adding value and 70 days are spent in waste (waiting for handoffs). To have a successful rapid feedback loop, you need to have minimal latency.

2. Bloatware: This investment in large software platform that is intended to make it more responsive to the business. They give examples showing the same platforms growing to 200x their original size over the past 15 years, yet they mostly still provide the same business value.

3. War for talent: People fleeing the industry. The job is really boring. We took thinking and the design part away from people writing the code. There is no sense of ownership or context around what they are doing.

Next they cover some encouraging trends:

Collaborative team: Face to face communication, joint ownership of outcome, multi skilled people where developers can code and learn business domain (Make jog more interesting because of variety of task), and continuous improvements (plan, do, check). This results in better responsiveness, richer communication, higher quality, and improved staff retention.

Rise of craftsmanship: Master apprentice relationship. Attention to tools and quality and simplicity. Polyglot programmers. Software is all about design. Design is important because of its role in the long term of things. We should not see software as a cycle of building something and then it is delivered and we do not see it again. There should not be a disconnect between building and supporting something. Looking at long term brings out importance of design. It is important to have a code base that can absorb changes and gives the right degree of long term horizon to think of investments.

Perpetual beta: get software out there for people to try, fund things incrementally. Get away from project mentality and move towards product support mentality.

User experience design: 65% of features are rarely or never used Standish group. Bring user experience design into the iterative cycle of user needs analysis, collaborative design with users, design refinements, and test with users.

Domain Driven Design: enhance communication of internals of software. Use same language between business people and developers.

Domain Specific languages: makes the code readable to the business side making it easier for them to understand the code and improves communication.

They also gave several examples of case studies where these trends have helped out. They conclude that today business and IT are separate. In 2020 business and IT will come much closer to together. How are we going to survive the post IT world? What techniques and approaches are we going to use?

This presentation is available on InfoQ at  http://www.infoq.com/presentations/New-Alliance-Shaw-Fowler

Fostering Software Craftsmanship in a Corporate Setting

Scott Dillman gave a talk at Agile 2008 about fostering software craftsmanship in a corporate setting. Scott defined software craftsmanship as taking responsibility, continuous learning, rejecting specialization, pride in quality work, passing on knowledge, and meeting professional standards (test driven development, continuous integration, coding patterns and practices).

Next Scott described 3 steps to evolve a novice software developer into a craftsman.

1. Evaluate the current state of the organization through interviews, surveys and metrics. Surveys can be reused at a later date to evaluate progress. Collecting metrics from automated builds like % passing builds, builds/day, build duration, fail build reasons, code coverage, code complexity. Track historical trends.

2. Educate via pair programming, centralized resources (wiki, educational content, knowledge base, moderated technical forum, book reports, tools and code repository), educational sessions(teasers, lightning sessions, tech talks, workshops), craftsmanship day, 10% time and on team mentoring. To get upper management buy in, highlight benefits, cite case studies, link to corporate goals. Upper management has to allocate time and recognize and reward participation.

3. Measure success by interviews, surveys, metrics, and performance reviews. Make sure reviews have goals that motivate and educate, are relevant, specific, actionable, and verifiable. Have the right people perform the reviews, do 360 reviews, and encourage and reward participation.

This presentation is available on InfoQ at http://www.infoq.com/presentations/Craftsmanship-Scott-Dillman

Prioritizing Product Backlog

At Agile 2008, Mike Cohn gave a talk about product backlog prioritization. Mike starts out by discussing the right level of prioritizing. A well formed product backlog should take the form of an iceberg. Small on the top (sprints), a little bigger down the middle (releases), and much bigger down the bottom (future releases). Mike believes that about 10% of current iteration should involve planning and prioritizing for the next iteration.

He then describes 3 levels of user stories. A user story which is a description of desired functionality told from the perspective of user or client, epics which are very large user stories and themes which are a collection of related user stories. An example of an epic might be as VP of marketing, I want to select type of campaign (direct mail, tv, radio, etc) so that I can identify and repeat profitable ones. This can then move up the iceberg and become as VP of marketing, I want to see direct mailings when reviewing historical campaigns. Mike recommends prioritizing at the theme or epic level. It is a lot of work to prioritize at the user story level and not very meaningful.

Mike then covers 4 techniques for prioritization:

1. Kano analysis:

Kano identifies 3 types of features to measure customer satisfaction:

• 1. Exciter or delighter: Exciter - feature you did not know you wanted until you saw it.
• 2. Mandatory or baseline: must be present for users to be satisfied
• 3. Linear: the more of it the better

Agile Estimating and PlanningAgile Estimating and PlanningThis can be done by expert opinion or surveying a small group of users (20 to 30) and asking both functional and dysfunctional questions like how do you feel if a feature is included and how do you feel if a feature is absent. The available answers are standard: I like it, I expect it, I’m neutral, I can live with it, and I dislike it. The results can then be matched to the Kano matrix to find out if a feature is an exciter, mandatory or linear. Others might fall in a questionable, reverse, or indifferent category. Mike recommends prioritizing by including all mandatory, some linear and leaving room for a few exciters.



2. Theme screening:

Identify 5 to 9 selection criteria for what is important in the next release (Importance to existing customers, competiveness, generating revenue, etc…). Identify a baseline theme that is likely to be in the next release and one that everybody understands (not the best one). For each theme, score the theme based on each selection criteria and compare it to the baseline theme using +1, 0, or -1. Add up the results for each theme and then rank them. Open the highest ranking themes and look at individual stories and use expert judgment on them.

Agile and Beyond – The Power of Aspirational Teams

At Agile 2008, Tim gave a talk on appreciative inquiry. Tim starts by defining appreciative inquiry as the study and exploration of what gives life to human systems when they function at their best. It suggests strengths, hopes and successes are transformational. We control a lot more of our destiny than we are aware of. Agile is similar. It is not just about process and practices but is a belief in achievement. Software development always brings up bad news. We need to think of what we do good in software.

Next Tim asks who defends the honor of the agile manifesto and principles. It’s easy to just adopt some of the principles and skip others. It is easy to assume that we cannot do all the practices and then not aspire for more. The best teams adopt all practices because they know that all the practices are dependent on each other.

Tim then covers several areas to influence appreciative inquire:

1. Working together: Pulling the load together, having fun, being excited about achieving.

2. Project chartering: It helps team understand the parameters you work in like defining success and determining who is part of the community. It also helps self organize over a common vision and goal and forms a contract that binds together the rules of the teams/practices.

3. Group sharing: encourages new ideas, gets more people involved in work.

4. Story cards and planning boards. Puts focus on collaboration. Use magnetic boards, make it fun, try magnetic avatars. Also, instead of statuses of in development, in test, in QA, use started, in progress, done which encompasses the different roles. Story cards are placeholders for a bigger conversation. Important to follow format of Story cards “as role, I want, so that”. Also get good pens to encourage good card writing (dark and thick ). Physical cards much better than huddling over a spreadsheet. If not hand written, print them out.

5. Visualize progress - Burn down chart vs. burn up chart. In general we like to see thing go up kind like a profit graph. For forecasting, simply counting story cards done vs. story cards not complete can be enough. Graph can still be drawn manually.

6. Gold cards: give 2 per developers to use for working on areas where they can innovate.

Concurrency – Past and Present

At QCON London 2008, Brian Geotz gave a talk on concurrency. Brian starts by mentioning that concurrency is hard. It is unnatural, error prone and untestable, hard for most programmers to use. He then explains the basics concepts of concurrency. A thread is a sequential process with its own program counter and call stack. Thread share VM wide resources such as memory, file handles, and security credentials. The pros of threads is fine grained data sharing between threads. However, that is also its disadvantage. Threads execute concurrently unless you use explicit locking to ensure that threads take turns accessing critical resources (synchronize key word in java).

If mutable data is shared between threads, all access requires synchronization. Failing to follow this rule results in data race which results in reading the wrong value or writing the wrong value. This generally occurs whenever there is a if something then do something or ready-modify-write. In the 1st case, another thread might change the value between the time we do the check and the time we call the method. In the second case, 1 statement is actually 3 statements like x++. Again here another thread might read the same value and then increment it by just 1 when it should have gotten incremented by 2.

Adding synchronized on methods in the class that modify data is a start (account with debit/credit), but then we have to be aware of the locking mechanism when composing operations in other classes (accountManager with transfer).

Programmers have to specify which operations are required to be atomic. But when doing so, this might lead to deacklock. Example, transferMoney(me, you, 100) and transferMoney(you, me, 50). One way to deal with this, is to induce a lock ordering. Example, always acquire lock 1st on the smaller value.

If from

synchronize(from) {

synchronize(to) {

}

}


Next Brian explains how this is getting more and more complicated. The main reason is because there is a fundamental tension between concurrency and OO design. OO encourage encapsulation and hiding of implementation details. OO also encourages composition, but composing thread safe objects requires knowing how they implement locking in order to participate in their locking protocols and know how to avoid deadlocks. However, the language is hiding the implementation details.

Next Brian offers some solutions to this problem. One solution is software transactional memory (STM). Explicit locks are replaced by transactional boundaries that define atomic blocks. The VM then provides locking nesting semantics and choose a locking strategy. This is under research and promising but not yet available.

Threads and locks are just one model for concurrency. Lock based concurrency rules hold locks when accessing shared mutable state and hold locks for the duration of atomic operations. Alternatives are don’t mutate state or don’t share state. Functional languages like Haskell or JOCaml have no mutable state. In java, we should get in the habit of making everything final unless you have a clear need to make it mutable. In Erlang everything is an Actor (like a lightweight thread). It has a designated behavior for when a message is received. No shared state. Scala has a similar actor library.

Brian concludes that in java, we can try to restore predictability by limiting concurrent interactions to well defined points, limiting sharing, limiting immutability. Concurrency is hard, so minimize the amount of code that has to deal with concurrency (isolate concurrency in concurrent components such as blocking queues and isolate code that accesses shared state in frameworks), use immutable objects wherever you can. Sometimes it is cheaper to share a non thread safe object by copying than to make it thread safe.


This presentation is available on InfoQ at http://www.infoq.com/presentations/goetz-concurrency-past-present