Dark Views

AgileUnit is a new testing framework which puts the testing code and all your assumptions into the business logic.

The main advantage is that you can make sure that your assumptions are met as the customer uses your product. To drive this home: The tests are running while the code is in production and while real customers use it.

At first glance, this sound reasonable but there are a few catches:

1. Test code doesn’t stop early when you disable it. This means that the AgileUnit code makes your product slower at runtime. Each added test will make the situation worse. I have a feeling that each test will read the property file from disk – every time the method under test is run.
2. User input isn’t tested, only the predefined test cases are run.
3. Only primitive types plus some numeric types (Date, BigDecimal) can be tested. If your methods work with lists, maps or other complex data structures, then AgileUnit is not for you.
4. Test failures will show you the result of a computation but not the arguments. In the “sum” example, you get an error that the “input” (which is actually the result of the sum() method) is less than 0 but the inputs of the sum() method are not listed so there is no way to know why it failed (just that it did).
5. Tests are configured with huge property files (which you don’t have to edit by hand) but which are pretty unreadable. This also leads to cluttered test code. Things that I can do in a single line of code in JUnit take 20-30 lines of code and properties with AgileUnit.
6. The documentation is confusing, outdated and riddled with spelling mistakes.
7. Test code uses static variables which means you can’t use it in multi-threaded code.
8. The configuration for the tests is not in the project but in the home directory of the user. All configurations for all projects are in the same folder which means you can’t work on two versions of the same project at the same time and you can’t put these files under version control.
9. The sample project doesn’t work.

Conclusion: The idea to make sure expectations are met after a product has been deployed to production is intriguing but AgileUnit is not the solution. Some of the issues listed above could probably be fixed easily (lazy loading of properties, replacing the property files with a more powerful DSL) but the overall feeling isn’t promising. The code looks clean but for me, the project has too many problems which cry “this won’t get far.”