Modifiability: or is there design in Agility

Martin Fowler lead a panel discussion at QCON2007 that included Dan North, Fred George, David Farley, Eric Doernenburg, and Ian Cartright. The main themes of the discussion were:

1. Do the simplest thing that can possibly work. Simple does not mean easy, quick or stupid. Simple means it is simple to use. Do the simple thing well before moving into the clever step.

2. Push decisions to last responsible moment. Always ask do I really need to make this decision now or can I put it off for later. Is a decision reversible? If it is then it is not a big deal. But if it is not, postpone it until the latest possible. Early up front design can lead to surprising results. Let the design come on its own with each iteration.

3. Domain Driven Design is natural home for OO. OO is about creating a software simulation of the problem we are trying to solve. The domain model should not know about the presentation layer or the data persistence layer. The domain model will change. Try to get a simulation as clean as possible without technical dependencies. Programmers use the same language as the business person. If the domain model is complex, design the simplest representation possible to get the job done and add to it as more requirements come along. Every time circumstances change and invalidate your model, fix the model. You should be able to articulate the architecture and be able to say this does not fit anymore.

4. Use encapsulation. Ensure classes keep secrets, use dependency injection.

5. Using Test Driven Development drives better design by making it more modular. You benefit from the solution as well as the testing. Having an automated unit tests and acceptance test enables change even in the database schema.

6. Mitigate risks by starting out on parallel path to test out different alternatives.

7. Have basic design beliefs like refactoring, design patterns, design principles, MVC, isolating db from model, focusing on model. Following these believes might not be the quickest way but it is the cleanest.

This presentation is available on InfoQ at http://www.infoq.com/presentations/modifiability-fowler

Java EE Class Loading Architectures

At JavaOne Conference, Ernie Svehla gave a talk about JEE class loading. Ernie explains that classes can be loaded implicitly as a result of a reference, instantiation or inheritance or explicitly by for example a call to Class.forName(). A class is defined by its name and its associated class loaded. There are 2 key class loading concepts.

1. Isolation: ensures that classes used by one application cannot be seen by another

2. Delegation: defer a request to load the class to a parent before attempting to load the class myself.

Core class loaders include the bootstrap which load the java.lang.* files from rt.jar, the extension which loads jar files contained in jre/lib/ext, and system or application which loads classes on the class path and shared jars. This creates a hierarchy of class loaders. A user class calls its class loader which delegates to its parent (system). If system cannot find the class, it delegates to its parents and so on (system->extension->bootstrap). If class is still not found, the user class loader then tries to load the class or throws an exception.

Different JEE app servers have different hierarchies and algorithms for loading classes. EJBs, WARS, WEB-INF, manifest, etc. Depending on where classes are, an app might need to be restarted or it can simply be a hot deploy.

Ernie then recommends some packaging guidelines. He recommends leveraging EARs. They should be self contained, include all dependent jars but minimize dependency on core environment like jdbc jars. He recommends packaging common jars needed by wars with ejb jars and to allow wars to be self contained.

Next Ernie covers different application servers and explains how each implements class loading. After that, Ernie gives some examples of the most common errors due to class loading.

1. ClassNotFoundException: Reference to class is to high in the hierarchy or using incorrect string in ClassLoader.loadClass(className)

2. NoClassDefFound: Class existed at compile time, but is no longer available (.class file is deleted or not copied properly in build)

3. ClassCastException: Class loaded by 2 different class loaders (WAR and EJB) or attempting to cast an object to a non compatible type

4. ClassCirculatoryError: Circulatory dependency or a class and one of its dependents are both dependent on a third class and different class loaders are used to load that 3rd class.

5. Class path issues: improper url specification when using URLClassLoader. url ending with slash references a directory. If no slash, it is referring to a jar file with the name of directory.

6. Class visibility issues: A class cannot be seen due to the class loader hierarchy.

Then Ernie explains how to create a custom class loader by overriding the loadClass() method associated with java.lang.ClassLoader.

Ernie wraps up reminding us that we need to understand how java class loading works; we need to understand how specific JEE servers works and we need to pay attention to how classes are packaged.

This presentation is available on InfoQ at http://www.infoq.com/presentations/java-classloading-architectures-ernie-svehla

Database Refactoring

At Agile 2006, Scott Ambler gave a talk on Database refactoring. Scott stresses on the need to be able to evolve our database schema and mentions that it is technically possible but it might not be culturally. Scott defines database refactoring as a simple change to the design of the database schema that does not change behavioral semantics or information semantics. Then Scott gives an overview of database refactoring and discusses the steps involved in the refactoring. These include: Verifying that refactoring is needed, choosing the right refactoring method, deprecating the original schema, writing and running tests, changing the schema, migrating the data, updating external apps, re-running tests, announcing the refactoring, and versioning your work.

When refactoring, there needs to be a deprecation period or transition window. Not all apps will be able to apply the required changes all at once so the old schema and new schema need to run in parallel. Both the refactoring and scaffolding need to be deployed into production. The scaffolding ensures that the old schema and new schema are kept in sync. After a transition window, the original schema and scaffolding are removed and the refactoring is complete.

Scott then gives examples of refactoring techniques. These include: renaming a column, replacing a column, merging columns, removing a view, introducing a default value, replacing 1 to many relationship with an associative table, and introducing a surrogate key. In general, the techniques involve keeping the old schema, adding the new schema along with a trigger to keep the two schemas in sync. At a later date (after the transition period), the old schema and trigger are dropped.

Next Scott covers 6 ways to enable database refactoring.

1. Accept the situation: Scott mentions that we need to accept the fact that the data is an important asset and that we should be able to change the data schema. The database should not be an anchor around our neck. Trivial things should be trivial to do. We have to be able to do what needs to be done and no tools will magically fix the schema for us.
2. Teamwork: Developers need to learn data skills. They should understand the fundamentals of database modeling. They should know how to normalize. Also, DBAs need to understand design patterns. Operations must be flexible.
3. Sandboxes: Developers need their own database instance or table space to work with. There needs to be development sandbox, project integration sandbox, demo sandbox, qa sandbox, and production.
4. Configuration management: All system artifacts need versions( DDL scripts, test data, test scripts, source code, models, documents).
5. Database Regression Testing: Testing should be at the interface to database (input/output), and also inside the database (stored procedures, views, triggers, constraints). Example, testing cascading deletes or null values.
6. Encapsulate database access: not all apps will access the db the same way. Some will use Frameworks, some might use direct SQL, etc.

Then Scott covers the steps to deploy database refactoring:

1. Sandboxes: Each team works in its own set of sandboxes.
2. Deployment Process: Each team run regression tests, deploy code, deploy database  refactoring, re-run test, back out if it does not work.
3. Versioning issues: There will be different version of the database for different team sandboxes. Some are behind, some are ahead. Each database refactoring needs to be a script with incremental version numbers. There should be table in the database that keeps track of its current version.
4. Deployment Windows: Operations can use windows to merge database refactoring scripts from different teams into 1 before rolling into production

Finally Scott leaves us with some parting thoughts. He mentions that if we can refactor application code we should be able to refactor database schemas. Refactoring a database is harder, but the tools are getting better. Don’t underestimate how difficult database refactoring is.
This presentation is available on InfoQ at http://www.infoq.com/presentations/ambler-database-refactoring

Agile Restrospectives – Making Good Teams Great!

Esther Derby and Diana Larsen gave a presentation at Google Tech Talk about agile retrospectives. They explained how retrospectives are tied to the agile practices of inspect and adapt. It fits into the feedback cycle, looking at what you are doing, see if it is working or whether you need to do some changes and try things in a different way. Retrospectives address effectiveness of methods, engineering practices and team work.

They then describe 5 steps of retrospectives:

1. Set the stage: get everybody’s head in the game.

2. Gather data: what happened and how we responded.

3. Generate insights: make meaning of the data.

4. Decide what to do: same, differently, or try new things.

5. Close the retrospective.

Retrospectives are ongoing and a linked process that flows naturally at the end of each iteration and into the planning part of next iteration. People think that they don’t have time for all of this, but these 5 stages can be done in about an hour. They recommend that you don’t skip these stages as you might compromise the results.

Next they cover each stage in details:

1. Set the stage: Specify a goal on what we are going to look at and an agreement on how we will work together.

• a. Have a goal like let’s look at what’s working and what’s not working. This is good to start out with, but after a while it becomes boring. So then set a new goal like let’s look at how well we are applying coding standards, refactoring, xp practices, etc…

• b. Set a working agreement: Inquiry rather than advocacy, Dialogue rather than debate, Conversation rather than argument, Understanding rather than defending

2. Gather Data: Think together as a group and look back over the iteration. Create a timeline and put up events that are important. Try to have a common understanding of the what went on and get a fuller picture of what happened. Also add an emotional graph tracking energy (high, medium, low) along with the events (avoid using the word ‘feeling’).

Typical Pitfalls in Agile Development

Jutta Eckstein gave a talk at JAVAPOLIS 06 where she covered 8 typical pitfalls in agile development:

1. Practices over values. Do not insist on specific practices over values. Use practices that support the team and the team’s value system. Figure out what the goal of the practice is, is this our goal? Does the goal support our value system? How can we achieve this?
2. Iterate then measure. You need to learn from the iteration. Get feedback, improve and adapt
3. Missing result-orientation. Slice features so that they can be completed in an iteration and they can provide business value.
4. Release plan. At end every iteration, check and update release plan
5. Interruptions are norm. Keep iterations length the right size to be able to handle interruptions.
6. Importance of integration is often ignored. Ensure integration has always the highest priority and has dedicated staff.
7. Wasted times. Keep stand up meeting on short and on track. Avoid endless meetings, status meetings, etc.
8. Missing courage. Unable to deal with dependencies of time, scope, quality, resource. Clarify dependencies and show options. Learn to say no.

How To Design A Good API And Why It Matters

At Javapolis, Joshua Bloch gave a presentation about designing a good API. Joshua starts by defining why designing a good API matters. An API is as asset. A public API if forever. You have once chance to get it right. If it is well written, it will capture and hook customers, while a badly written one will result in an endless barrage of support calls. Good software is modular and each module should have a well defined API. A good API is one that satisfies the requirements of the appropriate audience. It is easy to learn and use while hard to misuse. The code is readable, maintainable, and extendable.

Next Joshua explains the process of API design. Gather requirements in terms of use cases. Start with a short 1 page spec. Bounce it around as many people as possible and flesh it out as you gain confidence. Then write to your API early and often, even before it’s implemented or fully specified. The code will act as examples to use in unit tests. If writing an SPI (plug-in for multiple interfaces), make sure you try on 3 different implementations. You can’t please everyone so instead aim to displease everyone equally. Expect to make mistakes and flush those out after a few years of real use.

Then Joshua covers some general principles:

· API should do one thing and do it well. Functionality should be easy to name.

· API should be as small as possible. When in doubt, leave it out, as it is easy to add it later, but you can never remove it. Look for a good power to weight ratio.

· Implementation details should not impact API

· Minimize accessibility of everything. Make classes and members as private as possible. Allow classes to be used, understood, built, tested, and debugged independently.

· Names matter. Avoid cryptic abbreviations. Be consistent. Strive for symmetry.

· Documentation matters. Document religiously. Document every class, interface, method,

constructor, parameter, and exception.

· Consider Performance consequences of API decision on performance. Bad decisions include making types mutable, using constructors instead of factories, and using implementation instead of interface



· API must coexist peacefully with platform. Obey naming conventions, avoid obsolete types, mimic patterns and take advantage of core API and language features while avoiding language pitfalls.

Next Joshua covers class design and emphasizes minimizing mutability, subclassing only when it makes sense (Liskov), and designing and document for inheritance, otherwise prohibiting it.

For method design:

· Don’t make the client do anything the module could do

· Don’t violate the principle of least astonishment

· Fail Fast- report errors as soon as possible after they occur.

· Provide programmatic access to all data available string form

· Overload with care: just because you can does not mean you should.

· Use consistent parameter ordering across methods.

· Avoid long parameter lists

· Avoid return values that demand exceptional processing: return zero length array or empty collection instead of null

Finally for exception design, throw exceptions to indicate exceptional conditions and don’t force clients to use exceptions for control flow, favor unchecked exceptions and include failure capture information in exceptions

This presentation is available on InfoQ at http://www.infoq.com/presentations/effective-api-design

Agile Quality: A Canary in a Coal Mine

At Agile 2006, Ken Schwaber gave a talk about Agile Quality. Ken starts by describing the way product owners and teams should work. He mentions some statistics showing that on average, 35% of requirements change and 65% of what is delivered is rarely or never used. So product owners have to be skilled in requirements management and make sure the product backlog focuses on the highest value, importance, and risk stories 1st.
Traditionally customers have no visibility into what’s happening until the project is almost done.
In iterative developments, customers have visibility into the project and can control risk
Every sprint, some cumulative business value is delivered. At some point, enough ROI is achieved that the customer can get rid of functionality of lowered earned value.

Ken moves on to describe the place that product owners and teams come from using waterfall and discusses some of the bad techniques for delivering on time and on cost:
1.    Reduce quality
2.    Change definition of done to somewhat done and add stabilization, user acceptance, alpha and pre-release phases

Next Ken describes how scrum falls apart if these habits persist and how organizations have fallen apart because of these habits.

He starts by listing some of quality reduction techniques:
1.    Overtime (studies shows that increasing a working day increases defects)
2.    Cut unit testing, acceptance, and performance testing
3.    Skip design and code reviews
4.    Don’t follow standards
5.    Don’t refactor

These techniques lead to a scrum only including analysis, design, coding and some testing. Every sprint something is left behind and not done. Instead it should be optimized so that in every sprint everything is done. Ask the team to define done for every sprint and daily scrum. And done should include everything.

Ken mentions that a major problem most organization face is how to add new functionality tied to legacy system or to synchronize new functionality to core functionality. The main problem is that the core functionality has no test harness, is very fragile (a simple change breaks something else), and usually only few people still know and are willing to work on it. Thus, the velocity of core functionality team is a lot less than velocity of new functionality team.

Investigating this problem further reveals that as teams bend to pressure to meet dates, they drop quality and deliver incremental crap, and within 4 or 5 years, the team’s velocity is down all the way to 1 because the software is in such bad shape.

Ken emphasizes that we should have the courage to do the right thing and not drop quality. Dropping quality to meet a date is not a customer or product owner decision. It’s a CFO decision because of the impact on the company. Software is an organizational asset and decision to cut quality must be made by executive management and reflected in financial statements.

To get out of this mess, companies end up rebuilding core functionality into each function, building new stuff and not worrying about core functionality, dropping functionality, or allocating more developers to the core team in an attempt to build up and increase the velocity of core functionality.

To undo these habits, Ken suggests that the ScrumMaster teaches the product owner how to manage by value and that he not allow the team to present anything that is not done. The CEO should be apprised of the root cause of the problem and should provide support.

Ken finishes by giving an example of SCRUM being used at Primavera and concludes that one should have the courage to do the right thing and that it is a professional responsibility that makes you feel better and is for the good of the company.
This presentation is available on InfoQ at http://www.infoq.com/presentations/agile-quality-canary-coalmine