Good and Bad Tests

16. January, 2017

How do you distinguish good from bad tests in your code?

Check these criteria. Good tests

  • Nail down expectations
  • Monitor assumptions
  • Help to locate the cause of a failure
  • Document usage patterns
  • Allow to change code
  • Allow to verify changes
  • Are short (LOC + time)

Bad tests

  • Waste development time
  • Execute many, many lines of code
  • Prevent code changes
  • Need more time to write than the code they test
  • Need a lot of code to set up
  • Take ages to execute
  • Are hard to run

Expectations

There are a lot of checks in your compiler. Those help to catch mistakes you make. Do the same with your tests. There are a lot of things that compilers don’t check: File encodings, existence of files, existence of config options, types of config options.

Use tests to nail down your expectations. Read config files and validate the odd option.

Create a test which collects the whole configuration of your program and checks it against a known state. Check that each config option is set only once (or at least that it has the same value in all places).

When you need to translate your texts, add tests which make sure that you have all the texts that you need, that texts are unique, etc.

Assumptions

Convention over configuration only works when everyone agrees what the convention is. Conventions are assumptions. Your brain has to know them since they are no longer in the code. If this approach fails for you, write a test that validates your assumptions.

Check that code throws the exceptions that you expect.

If you have found a bug in a framework and added a workaround, add a test which fails when the bug is fixed. Add a comment “If this test fails, you can remove the workaround.”

Speed

The world speeds up. No one can afford slow tests, tests that are hard to understand, hard to maintain, tests that get in the way of Get-Things-Done™. Make sure you can run your tests at the touch of a button. Make sure they never fail unless they should. Make sure they fail when they should. Make sure they are small (= execute fast, few lines to understand, little code to write, easy to change, cheap to delete). Ten tests, each asserting a single fact, are better than one test that asserts ten facts. If your tests run for more than ten seconds, you lose.

Documentation

There is code rot. But long before that, there is documentation rot. Who has time to update the comments and documentation after a code change?

Why not document code usage in tests? Tests tell you the Truth™. Why give someone 100 pages of words they can’t trust when you can give them 100 unit tests they can execute?

Conclusion

Make your life easier. Stop wasting time in your debugger, begging for production log files, running code in your head. Write a good test today, it will watch your back for as long as the project lives. Write a thousand good tests and they will be like an army of angels, warding you from suffering, every time you press that button.


TNBT – Creating Tests from the Debugger

20. July, 2015

From my series “The Next Best Thing“:

Often, you will find yourself in a debugger, trying to follow some insanely complicated code to find the root cause of a bug.

I would like to see a button in my IDE which reads “New Test Case”. It should example the current program state, determine (with my help) what part of the code I want to test and then copy the current state into a unit test. In a second step, I could then trim down the unit tests but I would have all the input, all necessary dependencies would be there, correctly initialized.

It would also be great if the IDE would track the state of the code from which the unit test was generated. If the code changes too much (indicating that the unit test might become outdated), I’d like to see that. Or maybe the IDE could figure out by itself when code tested in such a way deviates “too much.”

Along the same lines, the IDE should be able to inject probes into the product code. As I click buttons and enter data in the UI, the IDE should generate a series of unit tests under the hood as described here. If you’re using frameworks like Spring, the tests should come with minimal (or mocked) dependencies.


Replacing Integration With Unit Tests

15. July, 2015

Google asks to “Just Say No to More End-to-End Tests” – just go and read it.

The suggestion in the document is to have a testing pyramid. A few (slow, expensive, dangerous) End-to-End (E2E) tests, more integration tests (IT) and a whole lot of unit tests.

My approach is to aim for 100% unit tests by breaking down the E2E and integration tests into chains of unit tests.

Example: You want to test whether you can save a UI field in a database.

The naive approach would be to create the UI, create a DB, simulate user input, click the save button, check the data in the database, update the UI afterwards.

My approach is to cut this into individual steps and use the output of test X as input for X+1. For unit tests, output is always a constant, so this doesn’t create dependencies between the tests except for the fact that some are the logical next steps. Tests can still be executed in any order.

But back to the example. The first set of tests would put various values in the UI, simulating user input. The tests would just read the values from the fields and compare them against expected values.

The next set of tests would be for the input validation. These tests would reuse some of the “expected output” values from the previous tests. No UI would be necessary (the code to test the display of validation messages would be in another set). We’re only interested in “do we get the correct error for input value FOO” where FOO is a constant.

The input validation tests can be grouped into two sets: One where the input value passes the validation and another (probably a much bigger) where the validation fails.

For all the inputs where the validation succeeds, we write a test that writes this value (again taken from a constant) into the database.

Lastly, to update the UI, we write a set of tests which examine the state of the database. And another set which tests that a certain state appears correctly in the UI.

As you can see, at no time we have a runtime dependency of tests. Tests work in any order. If the output of one test changes, only a small number of tests need to be updated (if at all). The IDE will help you locate all tests which are connected.

Instead of using the constant directly in an assert inside of a test, you can refer to it from an annotation. Now you can write tests that make sure that an output value is also used as an input somewhere.

If a test fails, only a single test fails. If the validation changes and a value isn’t valid anymore, the DB test might still pass. This is good: We don’t want code to fail unless it’s really broken. The DB interface might be able to write “illegal” values into the database but the validation layer is responsible to catch them. Don’t mix responsibilities.

In E2E or integration tests, a single failure somewhere in the whole chain of operations breaks the whole test. With the approach outlined above, a single failure just breaks very few tests. Each test still runs only a small number of code lines, making it easier to locate the cause of the problem. Tests are fast, you can run tests for a subset of the chain – why test the database when you’re improving validation? Or start the whole UI? You can run fast tests all the time and mark slow ones (UI, DB) for “run only on CI server“.


SoCraTes Day: Testing the Impossible

21. June, 2015

I’m back from SoCraTes Day Switzerland where I help a Code&Hack session called “Testing the Impossible”.

The session is based on this Mercurial repository de.pdark.testing.

Transcript

  • 20150619_105103-socrates-day-testing-the-impossible-page1Space Shuttle – They Write the Right Stuff
    • Why wasn’t the bug caught by QA?
    • Why did the bug escape dev?
  • Personal Bug Diary
  • Team Culture: Strengths va. Blame Game
  • Miscommunication with Customers
  • Anyone can press the “Red Button
  • Make failures cheap. Fail fast.
  • I’m wrong. I have learned something.
  • Bug-.hunting takes time and mindshare.
  • If you found a bug, write a test for it

How do I test randomness?

  1. Test small pieces
  2. Fix things that varies (IDs, timestamps, etc.)

How do I test a DB?

  • 20150619_105103-socrates-day-testing-the-impossible-page2Layer between DB and App code which allows to switch between real DB and mock
  • Verify generated SQL instead of executing it
    • Fast
    • Allows to test thousands of combinations easily
    • Useful for code that builds search queries
  • Run SQL against real DB during the night (CI server)
  • H2 embedded DB can emulate Oracle, MySQL, PostgreSQL, …
  • Test at various depths (speed vs. accuracy)
  • Testing Walrus dives deep!

The Difference Between Unit and Integration Tests

14. February, 2013

A unit test checks a certain feature of the software. When it fails, you usually know exactly where to look for a place to fix the problem. They are short, compact and fast.

They come at a cost, though: You have to replicate the setup code in your test.

This is an important point. To use a feature of your software, you always have some set up to do. This set up has to exist in your production code. For unit tests, you will have to copy some parts of this code into your tests because usually, the setup of the production code simply isn’t fine grained or flexible enough to be useful for tests. Think tests which check the error handling: Your production code usually can’t build mock objects that raise errors when certain methods are being invoked.

An integration test reuses the production configuration. It tests many features at once, in the same order and with the same or at least a very similar environment that your final application will use. They are high level and often much easier to set up. But that comes at a cost:

  • They are slow
  • When they fail, there will be many places which could cause the issue, so fixing them is more expensive
  • They break more often because you have more dependencies
  • Setting up a test for the “success” scenario will be simple but injecting mock objects to cause exact error states will be much harder

So be aware what kind of test you’re writing.


Eclipse Suddenly Takes Long to Start Unit Tests

10. May, 2012

When starting a JUnit test in Eclipse suddenly takes ages (the process starts quickly as you can see in the Console view but it takes ages until the tree of tests appears in the JUnit view), you might experience troubles with IPv6.

The background of the issue: Modern OSs assign your network cards two addresses, one for the old IPv4 and one for the new IPv6. When IPv6 isn’t configured correctly, Eclipse will try to connect via this route and it will take some 30 seconds for Eclipse and the JUnitRunner process to begin talking to each other.

The quick fix is to disable IPv6 or to tell Java to prefer IPv4:


Testing Requirements

30. August, 2011

When developing software, you’re constantly faced with two issues:

  1. Get all the requirements
  2. Make sure the requirements are correct

When writing code, we use tests to make sure we meet all the goals (completeness and correctness) but how do you test requirements?

By writing unit tests against a model of your requirements. There are two approaches to model requirements in a way useful for both the business and the development: Requirements Modeling Framework (RMF) and AlphaSimple.

See rafael.chaves’s blog post “Modeling requirements the pragmatic way (or When xUML meets xUnit)” for a more detailed introduction to the idea.