Thursday, December 20, 2007

Agile Development Practical Experiences

Johan Lybaert gave a talk at JAVAPOLIS 07 about practical experiences in agile development. He started out by emphasizing that picking a certain methodology needs to take into account the type of project, organization culture, team members and client. Once a methodology is selected, it should be constantly evaluated and adapted.

With Agile, team members require certain soft skills. These include courage, responsibility, stand up meetings, direct communication, working in pairs, working with different members, thinking before acting, does what they say, looking for feedback and able to accept feedback, likes to keep it simple, and likes to have fun.

Johan covers some lessons learned:

  1. Team members work on a variety of tasks
  2. Configuration and build is common across projects and is provided as a service
  3. 2 week sprints better than 4 week sprints
  4. Investment in continuous learning is a must (coaching, info sessions, reading groups, workshops)
  5. Use Agile PM instruments (Velocity graph, backlog, high level plan with milestones,

Johan then moves to cover 10 best practices:
  1. User stories: An agreed high level analysis document should be the starting point. Just in time preparation for user stories is key, Review and sign off by customer at sprint design walkthrough essential for good scope management, involvement of developer and architect in the preparation is important for domain model design, technical options and constraints.
  2. On site customer: If can’t have on site customer all the time, then have a proxy customer fill in. Getting Demo feedback from the customer is not enough. Try to get the customer involved in exploratory tests after each sprint.
  3. Stand up meeting: Let the team agree what should be the best time for meeting (usually beginning of working day). Always have it at the same time and same place. Avoid having members report to the leader. Make sure they are discussing things amongst themselves. Avoid problem solving in the meeting. Avoid having multiple status meetings. Have scrum of scrums.
  4. Testing: teach TDD (not natural for developers), unit test becomes the documentation, automation of gui test and integration test will have very important influence on build duration, keep tests independent, regression tests outside core application.
  5. Collective ownership: technology diversity is a challenge, keep project wiki updated, the team has to respect code standards, extra education and information session is required, not everyone can master all aspects from day one, assign owners for each technical to act as coach for the others
  6. Continuous integration: automated, have multiple build environments, build manager necessary in large team, one successful build per day.
  7. Refactoring: give them the courage (it will pay off), educate the team (use real smells as examples), refactor if you do not understand the code, refactor only code you are working on, only refactor when it hurts, late refactoring costs a lot, is often perceived as conflicting with iteration scope.
  8. Design: need a lead architect, assign a design coach, involve experts in 1st technical implementations, courage.
  9. Pair programming: 6 dev + 1 coach, change pairs every half day, combine experience and junior developers, exchange people each sprint, pair when you are at start phase, have new technology, have difficult business.

Evolving Agile

Agile Modeling: Effective Practices for eXtreme Programming and the Unified ProcessScott Ambler gave a talk at JAVAPOLIS 07 about Evolving Agile. He started out by describing Moore’s Adoption Curve which is like a bell curve and starts out with innovators, then early adopters, peeks with early majority, start its decline with late majority and dies down with laggards. Most new technologies/processes do not manage to cross the chasm from early adopters to early majority and never make it. The innovators are always excited about new things and are ready to jump in. The early adopters believe they can get a competitive advantage so they try it. The early majority see others succeeding so they get in. The late majority need to see similar organizations doing the same thing before they step in, while the laggards need to see case studies and solid proof before committing. In most cases, the late majority and laggards come way too late to catch up to the competition.

Next Scott provides some stats from surveys done on dr dobb (

The survey shows the about 70% have adopted agile in some way. Teams that are collocated have higher success rates than distributed teams. Smaller teams have better success rates than larger teams. The survey also shows that:
  • Quality: 87% believe high quality is more important than on time and budget
  • Scope: 87% believe meeting actual needs of stakeholders is more important than building to spec
  • Money: 80% believe that providing the best ROI is more important than delivering under budget.
  • Staff: 75% believe that having a healthy workplace is more important than delivering on time and on budget
  • Schedule: 61% believe that delivering when the system is ready to be shipped is better than delivering on schedule
Next Scott moved to cover some items often skipped when talking agile.
On any project there needs to be project initiation where the team and stakeholders are determined, funding is secured, a team room is setup. This also includes initial scope modeling.
Spend some time getting initial requirements, architecture modeling. You always need to answer
What are you going to build? How much will it cost? How long is it going to take? How are you going to build it?

Other issues include release into production, pilot, documentation, training, end of phase testing, sign-off, final testing environment, data conversion. Also one needs to understand production needs (operations and support)

Test driven development is not meant to scale. One should use the right tool for the right job. When working with small issue, one can do just in time specs by writing tests followed by code to make the tests pass. When dealing with big issues then we need to draw sketches, use index cards, etc. For validation, we can run tests to ensure code still works, but this is smoke testing or confirmation testing. It is good, but not enough. It validates to our understanding of the requirements. This makes the assumption that stakeholders understand what they want and can communicate that to us. We need investigative testing. TDD tests to spec. Investigative testing is testing outside the box (key strokes, integration, usability, break the system).

Monday, December 10, 2007

The Technology Platform: Building Block for Innovation

At QCON London 2007, Werner Vogels gave a talk on technology platform. The Amazon technology platform services 5 kinds of clients:

1. retail customers

2. Retail market place sellers

3. Enterprise Retail Customers

4. Associates

5. Web Service Developers

Werner describes the original Amazon business model as increase in selection, leads to more customers, which leads to more sellers and so on. This results in a fly wheel in motion which leads to growth. Economy of scale leads to lower cost structure and thus lower prices which feeds back into more customers which gets an additional swing in the fly wheel.

Amazon started out with selling books. Due to the enormous selection, books are the ideal candidates to sell online.

From books, there was a move for more breadth and depth (screws, springs, grocery, movies, TVs).

Werner explains how to extended selection, Amazon invites other vendor onto the platform, even if they offer the same products at lower prices. These vendors get to take advantage of reviews and customer feedback. Amazon takes in xml feeds containing item and inventory descriptions and pushes back information about transactions.

In terms of Services, Amazon offers webstores, fulfillment, and selling on Amazon.

Ecommerce platform: opening up the data through web services. It will drive traffic back to application.

Clients can pick and choose the services of the platform that they want

Friday, November 30, 2007

Role of Leadership Mary Poppendieck

Leading Lean Software Development: Results Are not the PointAt Agile 2007, Mary Poppendieck gave a talk entitled ‘The Role of Leadership’. Mary started out with a little history going back to 1911 and the book ‘Principles of Scientific Management’ by Fredrick Winslow Taylor. The book came out about the same time as the 1st Ford assembly line of 1913 and addressed how to make workers more efficient. Mary totally disagrees with the book. Its premise is that workers will do as little as possible, do not care about quality and are not smart enough to know the best way to do the job. To improve efficiency, experts need to define the best way to do a job by breaking down the job into parts and finding the best way to do each part. Workers get paid extra to follow the method determined by the expert. The workers are very happy because they get a higher pay and the employers are very happy because they get higher profit. A lot of these early management thoughts have made their way into current western management practices.

Next Mary covers Charles Allen and his 4 step method of industrial training (Preparation, Presentation, Application, and Testing). To train, get ready and figure out the job, present it and have them do it then do some testing to make sure they know how to do it and follow up on a regular basis until the workers get it. If the learner hasn’t learned then the teacher hasn’t taught. On job training is the best. Let’s train supervisors and teach them how to train the people working for them and cascade it down. This method was tested in the war and proved successful.

20 years later (1940), there was another war and Training Within Industry (TWI) was used. It adapted Allen’s approach to take an incredibly inexperienced workforce and train them. The idea was to train line supervisors 1st by teaching them on job instruction (how to train), job methods (how to improve), and relations (how to treat people/solve problems). The method resulted in impressive productivity, but was abandoned in the US after the war ended. However, it got exported to Japan to help rebuild the economy.

The premise of TWI is the 5 skills of a good supervisor:
1. Knowledge of work: Have to know how to do the job.
2. Knowledge of responsibility: Have to understand policy, regulation, rules, etc
3. Skill in instructing: Have to be skillful instructors to pass knowledge onto others.
4. Skill in improving methods to enhance quality and quantity
5. Skill in leading

Next Mary moves to 1950 and the Toyota Production System. Taiichi Ohno studied US auto manufacturing and realized that Japan has to improve its manufacturing to catch up with the US but also realized that Japanese workers are not dumb and can figure out the best way to do the job without being told. The Toyota Production System was based on 3 keys:

Saturday, November 10, 2007

Why I don’t like Mondays

Collaboration Explained: Facilitation Skills for Software Project LeadersJean Tabaka gave a talk at Agile 2007 entitled ‘Why I Don’t Like Mondays’. Most hate Mondays because of long meetings. The talk is mainly about managing and facilitating meetings. Jean starts by listing 10 common meeting dysfunctions:

1. Meetings are repetitive: They are all the same. If the same topic is being discussed over and over again, it means that decisions from last time are not being tracked.

2. The same people do all the talking: This leads to others thinking why am I here?

3. Subjects are beaten to death again again: Decisions need to be honored.

4. We come to decisions just to get the meeting over.

5. I don’t have time to code because I am in too many meeting: Notion is that if I am not coding I am not working. In Agile, part of your job is not to code.

6. We have too many people in our meeting: The meeting should include the necessary people needed to get to yes or make a commitment. Having too many people is a sign of distrust and a need for control. These people are standing in the way of commitments.

7. We have too few people in our meeting: Missing product owner, customer representative, tester are too busy. Database developers cannot attend meeting. If all these members are important to decision making they should be in the meeting.

8. With constant stream of meetings, we are treated like machinery not people.

9. Our demos and reviews never bring about anything new: These meetings are not for acceptance. Acceptance should have occurred before hand. The demo is to demonstrate progress. It’s an opportunity to act like a team. What are we doing and where are we heading?

10. All decisions were made outside of the meeting. We just had the meeting to be told what we are doing and what to agree to: This indicates that there is still a lot of control going on. Technical architect is deciding what plan should be and what estimates are. Moving from command and control to collaboration is required.

Sunday, August 12, 2007

Availability & Consistency

At QCON London 2007, Werner Vogels gave a talk about Availability & Consistency and how the CAP theorem ruins it all. Werner defines scalability by as adding resources and getting a performance improvement proportional to the resources added. Also, if you add resources to improve redundancy, it must not hurt performance. A scalable service is resilient, becomes more cost effective when it grows, is capable of handling heterogeneity (as tech improves, architecture needs to improve), and is operationally efficient (# people necessary to run a node needs to go down as you scale up).

Next Werner explains the principles for scalable service design:

· Decentralize: avoid single point of failure. No centralized components.

· Asynchrony: Make progress under all circumstances even if some parts are not working. Work locally and not worry about the rest.

· Autonomy: Each node should be able to make decision purely based on local state.

· Controlled concurrency: Reduce concurrency as much as possible. Funnel things through single points, change data design to avoid fancy locking.

· Controlled parallelism: Control traffic going to each node so that there is capacity both in CPU and I/O left to do other tasks.

· Decompose into small well-understood building blocks. Same with teams. If it takes more than 10 people then the team is too big (2 pizza team). Knowledge gets shared automatically. If larger, need to have meetings.

· Symmetry: All nodes should do exactly the same thing.

· Failure tolerant:

· Local responsibility:

· Recovery built-in:

· Simplicity:

Sunday, July 15, 2007

Agile Styles: Lean

Leading Lean Software Development: Results Are not the PointMary Poppendieck covers Lean Software Development at Agile 2006. She mentions that Lean came into being in 1990 based on book ‘The Machine that Changed the World’. The book compares and shows the advantages of Japanese auto manufacturing vs. USA manufacturing. Most of the Lean practices trace their history back to Toyota. Originally it used to be Just In Time (JIT) and then later it became Lean. It’s used it manufacturing, operations and supply chain. We can’t apply it directly to software development, but we can use the Lean principles and look closed at Lean practices in product development. These principles end up deriving the agile practices.

Next Mary discusses some fundamental principles of Lean. She starts with eliminating waste. The trick here is to decide what waste it. She states that it is the customer’s view of waste that matters. MUDA is Japanese for waste or anything that does not add customer value.

Then Mary mentions that Testing is not for finding defects. In lean the job of testing is to prevent defects. If you have a quality process, you are building quality into the code. If you routinely have test and fix cycles, it means you are testing too late. Your process is defective. Final verification should be used to verify that something is working as expected and not for debugging. Defects are a management problem. We should look at defects as caused by a system which allows defects instead of just caused by developers.

A fundamental principle of Lean is to respect people. Toyota’s success was based on their ability to harness intellect of ordinary employees.

Lean looks at the whole picture. Software by itself is useless. It needs to be embedded in a business process, hardware, or an activity to become useful. The goal of software development is to support the development of a complete product (a process that helps customers get job done). The team should include business people not just developers.

Next Mary covers the process of iterative development. She also mentions that if churn or requirement change is as high then you are writing requirements too soon. You should move requirements gathering later into the process.

Lean has quality built in by having:
1. Standards: Architecture, Conventions, Tools. These should constantly be challenged and changed.
2. Continuous improvement: Improve the process, refactor the code.
3. Synchronization: Merge early and merge often. Configuration management, automated build, one click build, continuous integration, nested synchronizations, stop and fix if something is wrong
4. Frequent deployment. Small releases, automated deployment, automated installation.

Agile Styles: Dynamic Systems Development Method - DSDM

Jean Tabaka gave an overview of DSDM at Agile 2006. Jean begins by giving us some background on DSDM which stands for dynamic systems development method. DSDM has 9 principles. There is nothing remarkable about them other than that business people sat together and agreed on them.

1. Active users involvement is imperative

2. The team must be empowered to make decisions

3. The focus is on frequent delivery of products

4. Fitness for business purpose is the essential criterion for acceptance of deliverables

5. Iterative and incremental development is necessary to converge on an accurate business solution

6. All changes during development are reversible

7. Requirements are baselined at a high level

8. Testing is integrated throughout the life cycle

9. Collaboration and cooperation between all stakeholders is essential.

Collaboration and cooperation is one area of differentiation of DSDM. DSDM is very specific about having very clear stakeholder relationships that are explicitly brought in. This is really built into the method and it is not just be hopeful that stakeholders will like it.

Jean then presents the ‘3 pizzas and the cheese’ picture which represents the phases of DSDM. It has phases beyond development. There is the pre-project phase which is a feasibility study and a business study that addresses thing like how a project gets started and where the funding is coming from. Then there is Functional Model iterations, Design and build iterations, and Implementation iterations. Finally there is the post project phase which addresses things like what happens after deployment.

Next, Jean talks about the core agile components of DSDM. They include the 9 principles, the fixed time box, the collaborative and facilitated workshops, and the Prioritized requirement list or PRL. Like other agile methods, in DSDM the iron triangle is flipper over. Time and resources are fixed. Requirements are managed through prioritization and inspection and adaptation. Requirements are baselined at a high level usually using MoSCoW technique (Must have, Should have, Could have, Would like to have it) and delivery is between 2 and 6 weeks.

Other aspects of DSDM include being founded by business owners as opposed to software developers or engineers. It has specific integration points with business processes. It is phase driven. It is consortium-based which gives access to continuous growth of the method. It has licensing fees and multiple certifications which seems appealing to large enterprises.

Some of its benefits include documentation since 1995, continuously evolving, guidance maintained and updated in multiple languages, certification recognized worldwide, and provides a nice enterprise wrapper.

Jean wraps up by pointing us to the source as

This presentation is available on InfoQ at

Saturday, June 30, 2007

An Introduction to Agile Leadership

Adrenaline Junkies and Template Zombies: Understanding Patterns of Project BehaviorAt Agile 2006, Tim Lister gave a talk on Agile Leadership. Tim starts by defining a process. It is what you do not do naturally. Process is interesting when your instinct to do something natural is not optimal. Tim makes an analogy with swimming. Our natural instinct in water is head up and paddle like mad. Afterwards, we learn to sip air, put our head in the water, and come up when we need to. In freestyle competitions, most swimmers choose the crawl technique because it has become a proven process for short distances. A true test of a process is what happens when we are under stress and pressure. What do we do? How do we behave? How to we build? How do we deliver? We should follow the process when things get rough and not only in the good times.

Next Tim defines a leader as someone who by dint of words or deeds influences the behavior of others. A leader inspires, or leads by example. Leadership is not by position. It moves around based on the topic or problem at hand. It will naturally flow to the appropriate person. Position is mostly irrelevant. A PMs jobs is to cultivate leadership.

Tim then moves on to give his own interpretation of the Agile Project Leadership Network’s Declaration of interdependence.

1. We increase return on investment by making continuous flow of value our focus.
At least with the time we had, we delivered as many of the most valuable features as we could. All features are accepted, even low value features. However, they just get added to the log and sit there because the client prioritizes items of higher value.

2. We deliver reliable results by engaging customers in frequent interactions and shared ownership.
Deliver reliable results means we make pace visible early. Agile makes work rate or velocity obvious with small increments. You know when you are behind schedule. This eliminates surprises. Engaging customers means we force interaction into a partnership. Knowledge of the customer is always needed. Team needs customer domain expertise.

3. We expect uncertainty and manage for it through iterations, anticipation, and adaptation.
Requirements emerge. They are not gathered. Systems need a gestation period. Anticipation: Why anticipate that the customer always knows what they want. It is rare that we know exactly what we need.

4. We unleash creativity and innovation by recognizing that individuals are the ultimate source of value, and creating an environment where they can make a difference.
This means pushing decisions down and out and making the workplace safe. That is, find the right home for every decision. Don’t make a decision that you are not competent to make even though you have the authority to make it. Some decisions need to go down to the developer. Some decisions need to go out to the customer.
Great creativity and innovation is when people start working together on half baked ideas. One person has an idea and others jump in with their own ideas to complement it.
The teams recognize their performance. They don’t care what the outside world thinks.

Friday, May 18, 2007

Agile Styles: Crytsal

Agile Software Development: The Cooperative Game (2nd Edition)Alistair Cockburn introduces us to Crytsal in a presentation give at Agile 2006. He starts by discussing Japanese Ikido and the concepts of Shu, Ha, and Ri. He states that people learn things in 3 stages:

1. Shu or following: Show me one way that works. Once I know it, I’ll move on to others
2. Ha or breaking away: Learn limitations of technique A and look for others that complement it or competes with it
3. Ri or fluency: Improvise, slice, shift, or combine techniques in a moment

Alistair recommends getting the starter level kit for all the agile styles. He mentions that a methodology is a reflection of the methodology author and people pick a methodology because the style of the author resonates with them.

Alistair then introduces us to Crystal be stating that it aims to be the lightest, least intrusive set of rules that puts a project in the safety zone. Its purpose is to keep people informed and keep them from hurting each other. Its nature is enable people to follow a set of conventions that gets updated regularly. Its philosophy is that people differ in their working styles, projects differ in needs, and software development is communication-intensive, experiment-based, needs lots of feedback in all directions, less is generally better, techniques and technologies change over time, and people learn in class or on the job and not from methodology.

Crystal Clear: A Human-Powered Methodology for Small TeamsAlistair shares a study he conducted which concluded that people centric methodologies do better than process centric methodology. Methodology is nothing more than a set of conventions that people agree to follow. He recommends adjusting a methodology based on the team and project like having a meeting every month and tuning the methodology based on what was learnt. As the people on the team change, the conventions of the team change. As the project evolves from beginning, middle to end, the strategies and conventions also change.

Alistair then defines the Crystal family of methodologies Clear, Yellow, Orange, and Red. Each one is better suited depending on the project needs.

Crystal is a family with genetic code:
1. Mindset: Not modeling, but cooperative game where people help each other to succeed. There are 2 goals: The primary is to delivers software, and the secondary is to setup for the next game. These two goals conflicting with each other.
2. Design priorities: Project safety (first and foremost), then development efficiency process habitability (tolerance). Will people use the suggestions.
3. Principles:
  • a. interactive face to face communication is most efficient,
  • b. methodology weight is costly,
  • c. use heavier methodology for large groups,
  • d. use more ceremony for more criticality,
  • e. use more feedback and communication instead with fewer intermediate deliverables
  • f. Discipline, skills, understanding counter process, formality, documentation
  • g. Efficiency is expendable at non bottleneck activities

4. Crystal’s Project Properties
  • a. Part of genetic code: Frequent delivery, close communication, reflective improvement.
  • b. Good ideas: Personal safety (freedom to speak without fear of reprisal), focus (knowing what to develop and having time for it), easy access to expert users, technical environment with frequent integration, automated testing, configuration management.
5. Starter Techniques:
  • a. Methodology shaping
  • b. Reflection workshop (2 columns 1st: keep these, problem areas, 2nd :try these)
  • c. Blitz planning
6. Crystal sample methodology: Crystal clear, Crystal orange

To get started with crystal, Alistair recommends finding your place on the project chart, picking a frequency of delivery, focusing on the top 3 properties, starting work, and every month doing a reflection workshop and deciding what to change.

Alistair concludes by mentioning that to know if you are doing Crystal look for close communication, see users every week, deliver software every 1 to 3 months, monitor project by quality of communication and moral of community, actively reflect on team quality and working habit.
This presentation is available on InfoQ at

Agile Styles: Feature Driven Development

David Anderson gave a talk on Feature Driven Development at Agile 2006. FDD is highly effective and produces high quality in a faster time to market. It increases productivity and quality and enhances team work. In FDD there is no overtime, no time tracking, no GANTT charts, no task tracking, and no time on task estimates.

FDD methodology involves developing an overall model at a high level and then building a feature list. Planning then proceeds by feature where each feature is designed in detail and then built in small code batches.

David defines a feature as a tiny piece of client valued functionality which can be delivered in less than 2 man weeks, typically 2 days. There are 4 types of features: User Interface, Problem Domain or Business Logic, System Interface, and Data Management or Persistence. A feature is written using the following format:
action result [of|to|from|for] object
Example: Calculate the interest for the bank account. This directly maps to the method name, return value, and class name. David then explains relating a feature to the model. A feature is a method on the domain model.

David then explains relating a feature to the model. A feature is a method on the domain model. There is 1 UML sequence diagram per feature. A feature set is related to a moment-interval on the domain model. All features in a set touch the same pink class on the model. A subject area is related to a chain of moment-interval.

Next David covers the Law of Demeter which ensures loose coupling by stating that classes should only depend on their immediate neighbors. This allows for postponed component definition and making decision at the last responsible moment. This enables componentizations of service orientation which allows building one big code base and breaking into components later and not having to refactor because interfaces between components are defined as the method signature on the classes.

Components are color coded pink, yellow, green and blue. They are re-usable enterprise components. The pinks and yellows are re-usable across multiple greens (the core enterprise components). The greens and blues are re-usable across discrete Enterprise Applications modeled as sequence of pinks.

FDD includes the concept of class or code ownership. This ensures that a class is not doing something it is not supposed to do from a design perspective. Classes are consistent and the api is concise. It also fosters collaboration and team work because if a feature is messaging a class owned by someone else, you need to work with that person.

David then compares bananas to requirement documents. He stresses that they are both perishable and thus cycle time is very important. He introduces CPW( Chief programmer work package) which is a collection or batch of features which can logically be grouped for development simultaneously and can be delivered within 2 weeks or less (each feature must be less than 2 weeks, each CPW must be less than 2 weeks). There is no timebox iteration in FDD. It is however long it takes, but must be less than 2 weeks.

Next feature teams are defined as dynamically formed per feature under guidance of chief programmer. All owners of relevant code are part of the team. This emphasizes team work and ensures that multiple minds collaborate on design and development.

David then provides some productivity statistics of using FDD and briefly covers configuration management in FDD using promotions groups instead of branch and merge techniques. Finally David gives some examples of dashboards used on his projects.
This presentation is available on InfoQ at

Monday, May 7, 2007

Role of Testing and QA in Agile Software Development

Agile Modeling: Effective Practices for eXtreme Programming and the Unified ProcessScott Ambler gave a talk entitled “The Role of Testing and QA in the Agile Software Development” at TASSQ. The talk however covered many topics and was not necessarily geared to testing and QA.

Scott starts by giving an overview of agile software development. Agile development involves open communication, enhanced collaboration, and embraces change. It results in the creation of working systems that meet the changing needs of stakeholders and produces high quality software with the highest value possible for the stakeholder’s investment.

Agile values individuals and interactions over processes and tools, working software over comprehensive documentation, customer collaboration over contract negotiation, and responding to change over following a plan. There are 12 principles in Agile which Scott briefly goes over.

After that Scott dispels some misinformation about agile. These include:
  1. 1. Many people confuse agility with code and fix (hacking). Many hackers claim to be agile. These are not agilest. Do they have a test bed? Do they know the stakeholders?
  2. 2. Agilist don’t model. Agilist don’t document. Not true. Agile documentation and modeling is done just in time and avoids unnecessary documentations.
  3. 3. ISO/CMM/6sigma therefore we cannot be agile. These presents some challenges’ but it is still possible to be agile even in these types of organizations.
  4. 4. Agility does not scale. This has been proven to be false. Example is the Eclipse project.
  5. 5. Agilist ignore Enterprise concerns. This area is still not well addressed by the agile community.

Next Scott moves on to cover Agile techniques. These include:

    Agile Database Techniques: Effective Strategies for the Agile Software Developer (Wiley Application Development)
  1. 1. Regular deployment of working software: The only measure of success is delivering working software.
  2. 2. Pair programming: This is a hard sell to management, but the data shows that it increases productivity. People learn from each other quickly. Forces programmers to code. Reduces down time (facebook, email, browsing, etc…).
  3. 3. Active stakeholder participation: They provide information and make decisions in a timely manner.
  4. 4. Model with others – equivalent to pair programming. Collective ownership concept. More people understand what is going on. Leads to more collaboration and less documentation
  5. 5. Refactoring – Small changes to code that improves design and retains the behavioral semantics of code. Enables evolving development assets in a controlled manner, enabling design to remain high quality.
  6. 6. Database refactoring – simple change to database schema that preserves the behavioral semantics and the informational semantics.
  7. 7. Test driven development. Write test 1st, write code to pass test.
  8. 8. Continuous regression testing – Whenever someone checks code in.
  9. 9. Test as primary artifacts – acceptance tests are considered to be primary requirements artifacts and unit tests are considered to be detailed design artifacts.
  10. 10. Continuous Integration: Frequent builds
  11. 11. Follow Guidance – Coding practices and Modeling practices

Friday, April 20, 2007

Modifiability: or is there design in Agility

Patterns of Enterprise Application ArchitectureMartin Fowler lead a panel discussion at QCON2007 that included Dan North, Fred George, David Farley, Eric Doernenburg, and Ian Cartright. The main themes of the discussion were:

1. Do the simplest thing that can possibly work. Simple does not mean easy, quick or stupid. Simple means it is simple to use. Do the simple thing well before moving into the clever step.

2. Push decisions to last responsible moment. Always ask do I really need to make this decision now or can I put it off for later. Is a decision reversible? If it is then it is not a big deal. But if it is not, postpone it until the latest possible. Early up front design can lead to surprising results. Let the design come on its own with each iteration.

Refactoring: Improving the Design of Existing Code3. Domain Driven Design is natural home for OO. OO is about creating a software simulation of the problem we are trying to solve. The domain model should not know about the presentation layer or the data persistence layer. The domain model will change. Try to get a simulation as clean as possible without technical dependencies. Programmers use the same language as the business person. If the domain model is complex, design the simplest representation possible to get the job done and add to it as more requirements come along. Every time circumstances change and invalidate your model, fix the model. You should be able to articulate the architecture and be able to say this does not fit anymore.

4. Use encapsulation. Ensure classes keep secrets, use dependency injection.

5. Using Test Driven Development drives better design by making it more modular. You benefit from the solution as well as the testing. Having an automated unit tests and acceptance test enables change even in the database schema.

6. Mitigate risks by starting out on parallel path to test out different alternatives.

7. Have basic design beliefs like refactoring, design patterns, design principles, MVC, isolating db from model, focusing on model. Following these believes might not be the quickest way but it is the cleanest.

This presentation is available on InfoQ at

Tuesday, April 10, 2007

Java EE Class Loading Architectures

At JavaOne Conference, Ernie Svehla gave a talk about JEE class loading. Ernie explains that classes can be loaded implicitly as a result of a reference, instantiation or inheritance or explicitly by for example a call to Class.forName(). A class is defined by its name and its associated class loaded. There are 2 key class loading concepts.

1. Isolation: ensures that classes used by one application cannot be seen by another

2. Delegation: defer a request to load the class to a parent before attempting to load the class myself.

Core class loaders include the bootstrap which load the java.lang.* files from rt.jar, the extension which loads jar files contained in jre/lib/ext, and system or application which loads classes on the class path and shared jars. This creates a hierarchy of class loaders. A user class calls its class loader which delegates to its parent (system). If system cannot find the class, it delegates to its parents and so on (system->extension->bootstrap). If class is still not found, the user class loader then tries to load the class or throws an exception.

Different JEE app servers have different hierarchies and algorithms for loading classes. EJBs, WARS, WEB-INF, manifest, etc. Depending on where classes are, an app might need to be restarted or it can simply be a hot deploy.

Ernie then recommends some packaging guidelines. He recommends leveraging EARs. They should be self contained, include all dependent jars but minimize dependency on core environment like jdbc jars. He recommends packaging common jars needed by wars with ejb jars and to allow wars to be self contained.

Next Ernie covers different application servers and explains how each implements class loading. After that, Ernie gives some examples of the most common errors due to class loading.

1. ClassNotFoundException: Reference to class is to high in the hierarchy or using incorrect string in ClassLoader.loadClass(className)

2. NoClassDefFound: Class existed at compile time, but is no longer available (.class file is deleted or not copied properly in build)

3. ClassCastException: Class loaded by 2 different class loaders (WAR and EJB) or attempting to cast an object to a non compatible type

4. ClassCirculatoryError: Circulatory dependency or a class and one of its dependents are both dependent on a third class and different class loaders are used to load that 3rd class.

5. Class path issues: improper url specification when using URLClassLoader. url ending with slash references a directory. If no slash, it is referring to a jar file with the name of directory.

6. Class visibility issues: A class cannot be seen due to the class loader hierarchy.

Then Ernie explains how to create a custom class loader by overriding the loadClass() method associated with java.lang.ClassLoader.

Ernie wraps up reminding us that we need to understand how java class loading works; we need to understand how specific JEE servers works and we need to pay attention to how classes are packaged.

This presentation is available on InfoQ at

Thursday, March 15, 2007

Database Refactoring

Refactoring Databases: Evolutionary Database DesignAt Agile 2006, Scott Ambler gave a talk on Database refactoring. Scott stresses on the need to be able to evolve our database schema and mentions that it is technically possible but it might not be culturally. Scott defines database refactoring as a simple change to the design of the database schema that does not change behavioral semantics or information semantics. Then Scott gives an overview of database refactoring and discusses the steps involved in the refactoring. These include: Verifying that refactoring is needed, choosing the right refactoring method, deprecating the original schema, writing and running tests, changing the schema, migrating the data, updating external apps, re-running tests, announcing the refactoring, and versioning your work.

When refactoring, there needs to be a deprecation period or transition window. Not all apps will be able to apply the required changes all at once so the old schema and new schema need to run in parallel. Both the refactoring and scaffolding need to be deployed into production. The scaffolding ensures that the old schema and new schema are kept in sync. After a transition window, the original schema and scaffolding are removed and the refactoring is complete.

Scott then gives examples of refactoring techniques. These include: renaming a column, replacing a column, merging columns, removing a view, introducing a default value, replacing 1 to many relationship with an associative table, and introducing a surrogate key. In general, the techniques involve keeping the old schema, adding the new schema along with a trigger to keep the two schemas in sync. At a later date (after the transition period), the old schema and trigger are dropped.

Next Scott covers 6 ways to enable database refactoring.

  1. Accept the situation: Scott mentions that we need to accept the fact that the data is an important asset and that we should be able to change the data schema. The database should not be an anchor around our neck. Trivial things should be trivial to do. We have to be able to do what needs to be done and no tools will magically fix the schema for us.
  2. Teamwork: Developers need to learn data skills. They should understand the fundamentals of database modeling. They should know how to normalize. Also, DBAs need to understand design patterns. Operations must be flexible.
  3. Sandboxes: Developers need their own database instance or table space to work with. There needs to be development sandbox, project integration sandbox, demo sandbox, qa sandbox, and production.
  4. Configuration management: All system artifacts need versions( DDL scripts, test data, test scripts, source code, models, documents).
  5. Database Regression Testing: Testing should be at the interface to database (input/output), and also inside the database (stored procedures, views, triggers, constraints). Example, testing cascading deletes or null values.
  6. Encapsulate database access: not all apps will access the db the same way. Some will use Frameworks, some might use direct SQL, etc.

Then Scott covers the steps to deploy database refactoring:
  1. Sandboxes: Each team works in its own set of sandboxes.
  2. Deployment Process: Each team run regression tests, deploy code, deploy database  refactoring, re-run test, back out if it does not work.
  3. Versioning issues: There will be different version of the database for different team sandboxes. Some are behind, some are ahead. Each database refactoring needs to be a script with incremental version numbers. There should be table in the database that keeps track of its current version.
  4. Deployment Windows: Operations can use windows to merge database refactoring scripts from different teams into 1 before rolling into production

Finally Scott leaves us with some parting thoughts. He mentions that if we can refactor application code we should be able to refactor database schemas. Refactoring a database is harder, but the tools are getting better. Don’t underestimate how difficult database refactoring is.
This presentation is available on InfoQ at

Friday, January 26, 2007

Agile Restrospectives – Making Good Teams Great!

Agile Retrospectives: Making Good Teams GreatEsther Derby and Diana Larsen gave a presentation at Google Tech Talk about agile retrospectives. They explained how retrospectives are tied to the agile practices of inspect and adapt. It fits into the feedback cycle, looking at what you are doing, see if it is working or whether you need to do some changes and try things in a different way. Retrospectives address effectiveness of methods, engineering practices and team work.

They then describe 5 steps of retrospectives:

1. Set the stage: get everybody’s head in the game.

2. Gather data: what happened and how we responded.

3. Generate insights: make meaning of the data.

4. Decide what to do: same, differently, or try new things.

5. Close the retrospective.

Behind Closed Doors: Secrets of Great Management (Pragmatic Programmers)Retrospectives are ongoing and a linked process that flows naturally at the end of each iteration and into the planning part of next iteration. People think that they don’t have time for all of this, but these 5 stages can be done in about an hour. They recommend that you don’t skip these stages as you might compromise the results.

Next they cover each stage in details:

1. Set the stage: Specify a goal on what we are going to look at and an agreement on how we will work together.
  • a. Have a goal like let’s look at what’s working and what’s not working. This is good to start out with, but after a while it becomes boring. So then set a new goal like let’s look at how well we are applying coding standards, refactoring, xp practices, etc…
  • b. Set a working agreement: Inquiry rather than advocacy, Dialogue rather than debate, Conversation rather than argument, Understanding rather than defending

2. Gather Data: Think together as a group and look back over the iteration. Create a timeline and put up events that are important. Try to have a common understanding of the what went on and get a fuller picture of what happened. Also add an emotional graph tracking energy (high, medium, low) along with the events (avoid using the word ‘feeling’).