Dark Views

Google asks to “Just Say No to More End-to-End Tests” – just go and read it.

The suggestion in the document is to have a testing pyramid. A few (slow, expensive, dangerous) End-to-End (E2E) tests, more integration tests (IT) and a whole lot of unit tests.

My approach is to aim for 100% unit tests by breaking down the E2E and integration tests into chains of unit tests.

Example: You want to test whether you can save a UI field in a database.

The naive approach would be to create the UI, create a DB, simulate user input, click the save button, check the data in the database, update the UI afterwards.

My approach is to cut this into individual steps and use the output of test X as input for X+1. For unit tests, output is always a constant, so this doesn’t create dependencies between the tests except for the fact that some are the logical next steps. Tests can still be executed in any order.

But back to the example. The first set of tests would put various values in the UI, simulating user input. The tests would just read the values from the fields and compare them against expected values.

The next set of tests would be for the input validation. These tests would reuse some of the “expected output” values from the previous tests. No UI would be necessary (the code to test the display of validation messages would be in another set). We’re only interested in “do we get the correct error for input value FOO” where FOO is a constant.

The input validation tests can be grouped into two sets: One where the input value passes the validation and another (probably a much bigger) where the validation fails.

For all the inputs where the validation succeeds, we write a test that writes this value (again taken from a constant) into the database.

Lastly, to update the UI, we write a set of tests which examine the state of the database. And another set which tests that a certain state appears correctly in the UI.

As you can see, at no time we have a runtime dependency of tests. Tests work in any order. If the output of one test changes, only a small number of tests need to be updated (if at all). The IDE will help you locate all tests which are connected.

Instead of using the constant directly in an assert inside of a test, you can refer to it from an annotation. Now you can write tests that make sure that an output value is also used as an input somewhere.

If a test fails, only a single test fails. If the validation changes and a value isn’t valid anymore, the DB test might still pass. This is good: We don’t want code to fail unless it’s really broken. The DB interface might be able to write “illegal” values into the database but the validation layer is responsible to catch them. Don’t mix responsibilities.

In E2E or integration tests, a single failure somewhere in the whole chain of operations breaks the whole test. With the approach outlined above, a single failure just breaks very few tests. Each test still runs only a small number of code lines, making it easier to locate the cause of the problem. Tests are fast, you can run tests for a subset of the chain – why test the database when you’re improving validation? Or start the whole UI? You can run fast tests all the time and mark slow ones (UI, DB) for “run only on CI server“.