At Agile 2009, Joe Rainsberger gave a talk about integration tests. Joe states that integration tests (end to end) are a scam. He defines integration tests as is any test were the result of the test depends on the correctness of more than one interesting behavior (multiple objects, multiple unrelated methods, complex enough to be tested on their own).
Next Joe covers some basics system object classifications from Domain Driven Design:
1. Values: Simple like string or number, transient, equality depends on value rather identity, light weight enough to be created when needed and thrown away when not
2. Entity: values with entity semantics have essential identifier and end up getting stored somewhere, equal if ids are equal.
3. Services: Stateless action of some kind.
Joe explains that any test that verifies 1 entity or service behavior possibly with multiple values involved is as focused a test as we can expect. Values are our inputs and values are our outputs. We do not want to test multiple entity or service behavior at the same time as this will lead to long running test suite. These result in slow feedback that eventually lead to no feedback since we stop running them. The tests lose their power of a pool of change detectors. The later a mistake is found the more costly it becomes. This also leads to a false sense of security. Over a long running time, the value of having the tests is roughly the same as not having tests at all.
Tests consist of arrange (put together the things you want to test), act (invoke what you to test), assert (check result). Next Joe discusses the problems with excessive test setup that are usually needed for integration tests. Due to all the setup, we end up having multiple asserts per test or even worse, multiple actions per test. Alternatively we copy/paste setup into multiple tests. The hard coding of dependency is making it impossible to check something without executing something else that we do not want to check right now.
Next Joe explains that when integration test fails it is not clear where the failure is. Integration test are slower than focus tests. # tests needed is the product of the code path(4*3*5=60 test to be thorough). The # of test is not exponential but combinatoric. In reality if we do a really good job we end up with less than 1 % coverage. It is like the lottery, when the odds of winning are 1 in 40 million, buying 10000 tickets is not that much better than buying 10. When we need 4 million tests, writing 20000 tests is not better than 2000.
Next Joe defines basic correctness as given the myth of perfect technology (no limit on memory and time), do we compute the right answer? He stresses that he is not talking about customer tests that business people write to make sure features exists. Customer tests tend to be end to end, but are not exhaustive tests.
Not dealing with performance, scalability, reliability
Joe’s thesis is when showing basic correctness, write only focused object tests (test one thing at a time). Every test will run in about 5 m second. 2 to 5000 tests in 10 seconds.
His approach is to start with basic focused object tests. When testing interaction with other services, put an interface in front of it and perform collaboration tests and contract tests. Collaboration tests test conversation between the object under test and the interfaces it talks to. The goal is to answer 2 key questions: do I ask collaborators the right questions, can I handle their responses. Contract tests go in the opposite direction and answer can you handle the question, and can you respond properly. Test doubles are used as an implementation of an interface that behaves similarly enough to the real thing. These include mocks which are used to test invocation with the right parameters and Mock which are used to return responses.
The number of test needed in this scenario is 3+4+5+a small constant. The total tests are down from 4 million to 10000.
Joe concludes that simple state based tests, collaboration test and contract test give us arbitrarily high confidence in the correctness of the system. There is no need for integration test during the design phase. When there is a mistake (misplaced contract or collaboration test), convert integration tests into a collaboration or contract test and then you can throw away the integration test.
This presentation is available on InfoQ at http://www.infoq.com/presentations/integration-tests-scam
