Chained Unit Tests – CUT

29. March, 2023

The CUT approach allows to test logically related parts or to gradually replace integration tests with pure unit tests.

Let’s start with the usual app: There is a backend server with data and a frontend application. Logically speaking, those are connected but the backend is using a Java and the frontend uses TypeScript. At first glance, the only way to test this is to

  1. Set up a database with test data.
  2. Start a backend server.
  3. Configure the backend to talk to the database.
  4. Start the frontend.
  5. Configure the frontend to talk to the test backend.
  6. Write some code which executes an operation in the frontend to test the whole.

There are several problems with this:

  • If the operation changes the database, you sometimes have to undo this before you can run the next test. The usual example is a test which checks the rendering of a table of users and another test which creates a new user.
  • The test executes millions of lines of code. That means a lot of causes for failures which are totally unrelated to the test. The tests are flaky.
  • If something goes wrong, you need to analyze what happened. Unlike with unit tests, the problem can be in many places. This takes much more time than just checking the ~ 20 lines executed by a standard unit test.
  • It’s quite a lot of effort to make sure you can render the table of users.
  • It’s very slow.
  • Some unrelated changes can break these tests since they need the whole application.
  • Plus several more but we have enough for the moment.

CUT is an approach that can help here.

Step 1: Rendering in the Frontend

Locate the code which renders the table. Ideally, it should look like this:

  1. Fetch list of elements from backend using REST
  2. Render each element

Change this code in such a way that the fetching is done independent of the rendering. So if you have:

renderUsers() {
    const items = fetchUsers();
    return items.map((it) => renderUser(it));
}

replace that with this:

renderUsers() {
    const items = fetchUsers();
    return renderUserItems(items);
}
renderUserItems(items) {
     return items.map((it) => renderUser(it));
}

At first glance, this doesn’t look like an improvement. We have one more method. The key here is that you can now call the render method without fetching data via REST. Next:

  1. Start the test system.
  2. Use your browser to connect to the test system.
  3. Open the network tab.
  4. Open the users table in your browser.
  5. Copy the response of fetchUsers() into a JSON file.
  6. Write a test that loads the JSON and which calls renderUserItems().

This now gives you a unit test which works even without a running backend.

We have managed to cut the dependency between frontend and backend for this test. But soon, the test will give us a false result: The test database will change and the frontend test will run with outdated input.

Step 2: Keeping the test data up-to-date

We could use the steps above to update the test data every time the test database changes. But a) that would be boring, b) we might forget it, c) we might overlook that a change affects the test data, d) it’s tedious, repetitive manual work. Let’s automate this.

  1. Find the code which produces the JSON that fetchUsers() asks for.
  2. Write a unit test that connects to the test database, calls the code and compares the result with the JSON file in the frontend project.

This means we now have a test which fails when the JSON changes. So in theory, we can notice when we have to update the JSON file. There are some things that are not perfect, though:

  • If the test fails, you have to replace the content of the JSON file manually.
  • It needs a running test database.
  • The test needs to be able to find the JSON file which means it must know the path to the frontend project.

Step 2 a: Update the JSON file

There are several solutions to this:

  • Use an assertion that your IDE recognizes and which shows a diff when the test fails. That way, you can open the diff, check the changes, copy the new output, open the JSON file, paste the new content. A bit tedious but if you use keyboard shortcuts, it’s just a few key presses and it’s always the same procedure.
  • Add a flag (command line argument, System property, environment variable) which tells the test to overwrite the JSON when the test fails (or always, if you don’t care about wear&tear of your hardware). Since all your source code is under version control, you can check see the diff there and commit or revert.
    • Optional: If the file doesn’t exist, create it. This is a bit dangerous but very valuable when you have a REST endpoint with many parameters and you need lots of JSON files. That way, the first version gets created for you and you can always use the diff/copy/paste pattern.

You probably have concerns that mistakes could slip through when people mindlessly update the JSON without checking all the changes, especially when there are a lot.

In my experience, this doesn’t matter. For one, it will rarely happen.

If you have code reviews, then it should be caught there.

Next, you have the old version under version control, so you can always go back and fix the issue. Fixing it will be easy because you now have a unit test that shows you exactly what happens when you change the code.

Remember: Perfection is a vision, not a goal.

Step 2 b: Cut away the test database

Approaches to achieve this from cheapest to most expensive:

  • Fill the test database from CSV files. Try to load the CSV in your test instead of connecting to a real database.
  • Use an in-memory database for the test. Use the same scripts to set up the in-memory database as the real test database. Try to load only the data that you need.
    • If the two databases have slightly different syntax, load the production script and then patch the differences in the test to make the same script work for both.
  • Have a unit test that can create the whole test database. The test should verify the contents and dump the database in a form which can be loaded by the in-memory database.
  • Use a Docker image for the test database. The test can then run the image and destroy the container afterwards.

Step 2 c: Project organization

To make sure the backend tests can find the frontend files, you have many options:

  • Use a monorepo.
  • Make sure everyone checks out the two projects in the same folder and using the same names. Then, you can just go one up from the project root to find the other project.
  • Use an environment variable, System property or config file to specify the path. In the last case, make sure the name of the config file contains the username (Java: System property user.name) so every developer can have their own copy.

What else can you do?

There are several more things that you can add as needed:

  • Change fetchUsers() so you can get the URL it will try to fetch from. Put the URL into a JSON file. Load the JSON in the backend and make sure there is a REST endpoint which can handle this URL. That way, you can test the request and make sure the fetching code in the frontend keeps working.
  • If you do this for every REST endpoint, you can compare the list from the tests against the list of actual endpoints. That way, you can delete unused endpoints or find out which ones don’t have a test, yet.
  • You can create several URLs with different parameters to make sure the fetching code works in every case.

Conclusion

The CUT approach allows you to replace complex, slow and flaky integration tests with fast and stable unit tests. At first, it will feel weird to modify files of another project from a unit test or even trying to connect the two projects.

But there are several advantages which aren’t obvious:

  1. You now have test data for the default case. You can create more test cases by copying parts of the JSON, for example. This means you no longer have to keep all edge cases in your test database.
  2. This approach works without understanding what the code does and how it works. It’s purely mechanical. So it’s a great way to start writing tests for an unknown project.
  3. This can be added to existing projects with only small code changes. This is especially important when the code base has few or no tests since every change might break something.
  4. This is a cheap way to create test data for complex cases, for example by loading the JSON and then duplicating the rows to to trigger paging in the UI rendering. Or you can duplicate the rows and the randomize some fields to get more reasonable test data. Or you can replace some values to test cases like very long user names.
  5. It gives you a basis for real unit tests in the frontend. Just identify the different cases in the JSON and pick one example for each case. For example, if you have normal and admin users and they look different, then you need two tests. If there is special handling when the name is missing, add one more test for that. Either get the backend to create the fragments of the JSON for you or load the original JSON and then filter it. Make sure you fail the test when expected item is no longer in the list.
  6. The first test will be somewhat expensive to set up. But after that, it will be cheap to add more tests, for example for validation and error handling, empty results, etc.

Why chained unit test? Because they connect different things in a standard way like the links of a chain.

From a wider perspective, they allow to verify that two things will work together. We use the same approach routinely when we expect the compiler to verify that methods which we call exist and that the parameters are correct. CUT allows to do the same for other things:

  • Code and end user documentation.
  • Code and formulas in Excel files.
  • Validation code which should work exactly the same in frontend and backend.

Good and Bad Tests

16. January, 2017

How do you distinguish good from bad tests in your code?

Check these criteria. Good tests

  • Nail down expectations
  • Monitor assumptions
  • Help to locate the cause of a failure
  • Document usage patterns
  • Allow to change code
  • Allow to verify changes
  • Are short (LOC + time)

Bad tests

  • Waste development time
  • Execute many, many lines of code
  • Prevent code changes
  • Need more time to write than the code they test
  • Need a lot of code to set up
  • Take ages to execute
  • Are hard to run

Expectations

There are a lot of checks in your compiler. Those help to catch mistakes you make. Do the same with your tests. There are a lot of things that compilers don’t check: File encodings, existence of files, existence of config options, types of config options.

Use tests to nail down your expectations. Read config files and validate the odd option.

Create a test which collects the whole configuration of your program and checks it against a known state. Check that each config option is set only once (or at least that it has the same value in all places).

When you need to translate your texts, add tests which make sure that you have all the texts that you need, that texts are unique, etc.

Assumptions

Convention over configuration only works when everyone agrees what the convention is. Conventions are assumptions. Your brain has to know them since they are no longer in the code. If this approach fails for you, write a test that validates your assumptions.

Check that code throws the exceptions that you expect.

If you have found a bug in a framework and added a workaround, add a test which fails when the bug is fixed. Add a comment “If this test fails, you can remove the workaround.”

Speed

The world speeds up. No one can afford slow tests, tests that are hard to understand, hard to maintain, tests that get in the way of Get-Things-Done™. Make sure you can run your tests at the touch of a button. Make sure they never fail unless they should. Make sure they fail when they should. Make sure they are small (= execute fast, few lines to understand, little code to write, easy to change, cheap to delete). Ten tests, each asserting a single fact, are better than one test that asserts ten facts. If your tests run for more than ten seconds, you lose.

Documentation

There is code rot. But long before that, there is documentation rot. Who has time to update the comments and documentation after a code change?

Why not document code usage in tests? Tests tell you the Truth™. Why give someone 100 pages of words they can’t trust when you can give them 100 unit tests they can execute?

Conclusion

Make your life easier. Stop wasting time in your debugger, begging for production log files, running code in your head. Write a good test today, it will watch your back for as long as the project lives. Write a thousand good tests and they will be like an army of angels, warding you from suffering, every time you press that button.


TNBT – Creating Tests from the Debugger

20. July, 2015

From my series “The Next Best Thing“:

Often, you will find yourself in a debugger, trying to follow some insanely complicated code to find the root cause of a bug.

I would like to see a button in my IDE which reads “New Test Case”. It should example the current program state, determine (with my help) what part of the code I want to test and then copy the current state into a unit test. In a second step, I could then trim down the unit tests but I would have all the input, all necessary dependencies would be there, correctly initialized.

It would also be great if the IDE would track the state of the code from which the unit test was generated. If the code changes too much (indicating that the unit test might become outdated), I’d like to see that. Or maybe the IDE could figure out by itself when code tested in such a way deviates “too much.”

Along the same lines, the IDE should be able to inject probes into the product code. As I click buttons and enter data in the UI, the IDE should generate a series of unit tests under the hood as described here. If you’re using frameworks like Spring, the tests should come with minimal (or mocked) dependencies.


Replacing Integration With Unit Tests

15. July, 2015

Google asks to “Just Say No to More End-to-End Tests” – just go and read it.

The suggestion in the document is to have a testing pyramid. A few (slow, expensive, dangerous) End-to-End (E2E) tests, more integration tests (IT) and a whole lot of unit tests.

My approach is to aim for 100% unit tests by breaking down the E2E and integration tests into chains of unit tests.

Example: You want to test whether you can save a UI field in a database.

The naive approach would be to create the UI, create a DB, simulate user input, click the save button, check the data in the database, update the UI afterwards.

My approach is to cut this into individual steps and use the output of test X as input for X+1. For unit tests, output is always a constant, so this doesn’t create dependencies between the tests except for the fact that some are the logical next steps. Tests can still be executed in any order.

But back to the example. The first set of tests would put various values in the UI, simulating user input. The tests would just read the values from the fields and compare them against expected values.

The next set of tests would be for the input validation. These tests would reuse some of the “expected output” values from the previous tests. No UI would be necessary (the code to test the display of validation messages would be in another set). We’re only interested in “do we get the correct error for input value FOO” where FOO is a constant.

The input validation tests can be grouped into two sets: One where the input value passes the validation and another (probably a much bigger) where the validation fails.

For all the inputs where the validation succeeds, we write a test that writes this value (again taken from a constant) into the database.

Lastly, to update the UI, we write a set of tests which examine the state of the database. And another set which tests that a certain state appears correctly in the UI.

As you can see, at no time we have a runtime dependency of tests. Tests work in any order. If the output of one test changes, only a small number of tests need to be updated (if at all). The IDE will help you locate all tests which are connected.

Instead of using the constant directly in an assert inside of a test, you can refer to it from an annotation. Now you can write tests that make sure that an output value is also used as an input somewhere.

If a test fails, only a single test fails. If the validation changes and a value isn’t valid anymore, the DB test might still pass. This is good: We don’t want code to fail unless it’s really broken. The DB interface might be able to write “illegal” values into the database but the validation layer is responsible to catch them. Don’t mix responsibilities.

In E2E or integration tests, a single failure somewhere in the whole chain of operations breaks the whole test. With the approach outlined above, a single failure just breaks very few tests. Each test still runs only a small number of code lines, making it easier to locate the cause of the problem. Tests are fast, you can run tests for a subset of the chain – why test the database when you’re improving validation? Or start the whole UI? You can run fast tests all the time and mark slow ones (UI, DB) for “run only on CI server“.


SoCraTes Day: Testing the Impossible

21. June, 2015

I’m back from SoCraTes Day Switzerland where I help a Code&Hack session called “Testing the Impossible”.

The session is based on this Mercurial repository de.pdark.testing.

Transcript

  • 20150619_105103-socrates-day-testing-the-impossible-page1Space Shuttle – They Write the Right Stuff
    • Why wasn’t the bug caught by QA?
    • Why did the bug escape dev?
  • Personal Bug Diary
  • Team Culture: Strengths va. Blame Game
  • Miscommunication with Customers
  • Anyone can press the “Red Button
  • Make failures cheap. Fail fast.
  • I’m wrong. I have learned something.
  • Bug-.hunting takes time and mindshare.
  • If you found a bug, write a test for it

How do I test randomness?

  1. Test small pieces
  2. Fix things that varies (IDs, timestamps, etc.)

How do I test a DB?

  • 20150619_105103-socrates-day-testing-the-impossible-page2Layer between DB and App code which allows to switch between real DB and mock
  • Verify generated SQL instead of executing it
    • Fast
    • Allows to test thousands of combinations easily
    • Useful for code that builds search queries
  • Run SQL against real DB during the night (CI server)
  • H2 embedded DB can emulate Oracle, MySQL, PostgreSQL, …
  • Test at various depths (speed vs. accuracy)
  • Testing Walrus dives deep!

Eclipse Finance Day 2014: Testing business applications with RCPTT

3. November, 2014

RCP Testing Tool (RCPTT) is an Open Source tool for UI testing of Eclipse-based applications. During the demo by Ivan Inozemtsev, I got the impression that they thought of everything:

  • There is a recorder so you can quickly create a test case by clicking thought the application
  • There is an assertion builder where you can say “this element should be red”, “this checkbox needs to be checked”, “this image must be visible”
  • There is special support for all kinds of widgets like text editors where “position” is row/column (i.e. cursor position) instead of mouse coordinates or character offsets. The test API allows for checks like “is styled like a keyword”.
  • If the UI is resized after the tests have been recorded, the tool will try to calculate the relative click point. Unless the UI element is special (like a graph editor) where it has different heuristics (like when you clicked in an empty area but now, there is something below at the coordinate). If everything else fails, the testing tool will try to resize the UI element to the recorded size.
  • There is a compact, extensible DSL to describe the test cases. The DSL supports procedures so you can reuse test code.

The tool was fast and stable during the demo. It could reset Eclipse’s workspace to a certain state (open perspective, open editors, open projects) – I’m currently thinking about using it to create the default workspace for our development team.

One more things from the demo: Black box testing is a myth or at least a dream. A testing tool for anything complex needs to know internal details (like Eclipse’s background jobs). Otherwise the tests will either fail randomly when some background job didn’t complete in time or they will be slow (since each of them will be sprinkled with “WAIT 1 MINUTE” instructions) or they will be sprinkled with application specific “wait” instructions.


Eclipse Finance Day 2014: Automating user interface tests with behavior-driven development (BDD)

3. November, 2014

The last two talks were about testing.

“Automating user interface tests with behavior-driven development (BDD)” by Jose Badeau and Dietmar Stoll used an Xtext-based DSL to connect requirements, wireframes and test descriptions so they can validate each other. In the example, they removed a field from the wireframe mockup and the Eclipse editor for the test cases showed errors where they were used in the DSL.

The demo showed once again how much power lies in connecting information from different sources. If you remember, then this was the key feature which set Eclipse apart from all the other Java tools in 2001: It could bring you to the place where something was defined by pressing F3 or Ctrl+Click. If you changed one file, it would instantly compile all the other files and show you any problems it would find. Having a tool which actively searches for problems saved and still saves a lot of time.

 


Agile For Prudes

12. November, 2013

The article “WORKING IN A WANNABE-AGILE TEAM” points out a common problem in agile: It really exposes you and most people simply are prude.

Unlike many people want to make you believe, they are aware of their own flaws and how much a certain process humiliates them – it’s a skill everyone adopts at an early age, and therefore almost completely subconscious.

Since there is no way to be agile without looking at the team’s issues, the solution is to offer them something else instead.

In my experience, the easiest foot to get into the door is testing. When customers ask for features, ask “How would you know that it’s working correctly? Can you give me an example?” Yay, acceptance tests for free.

People struggling with a piece of code that fails all the time in interesting ways? “How do you know it’s wrong? What would be right? Maybe we could write a piece of code that makes sure it stays right from now on?” Yay, unit testing.

“Can you write a test?” “You can’t test that!” “…. You wrote software that can’t be tested? Seriously?” “… No, of course you could test it but …”

The best part: It focuses on solutions. When suggesting tests, no one can get into the blame game. Everyone can get involved. Customers, managers, developers, everyone understands tests. And they offer the most value for the least investment.

When people have started testing, they become interested in other things as well: Agile planning. Listening to the customer. That’s when you can start to change the culture – you now have some trust that you can spend.


Jazoon 2013 – Spock: boldly go where no test has gone before

24. October, 2013

Jazoon 2013 badgeIf you look at your tests, do you see a lot of repetitive patterns? Or are you looking for a way to express your intent more easily? Then, Spock might be for you. The talk “Spock: boldly go where no test has gone before” by Andres Almiray (slides on slideshare) gave a good introduction to the framework. Here is an example:

class HelloSpock extends spock.lang.Specification {
    def "length of Spock's and his friends' names"() {
        expect:
        name.size() == length

        where:
        name     | length
        "Spock"  | 5
        "Kirk"   | 4
        "Scotty" | 6
    }
}

In a nutshell, this creates three tests, running the assertion name.size() == length for each tuple of values. Here is an example how to test a stack.

But this is just one of many ways which you can use to describe tests that Spock should execute for you. The page “WhySpock” lists 10 reasons.


Selenium vs. ZK

30. August, 2013

Testing ZK applications using Selenium can be a drag. Selenium offers a lot of tool to test traditional request-response cycle applications. But it relies heavily on stable element IDs and submitting whole forms.

ZK, on the other hand, sends just a skeleton page to the browser and from then, builds everything with JavaScript. Instead of posting a whole form, it submits individual field values (to trigger validation). And it assigns random IDs to reach element.

At first glance, the two don’t seem to be a perfect match. But there are a couple of simple tools that will make your life much more simple.

Getting Stable IDs for Tests

One solution here is to write an ID generator which always returns the same IDs for each element. But this is tedious, error-prone and sometimes impossible.

A better solution is to attach a custom attribute to some elements which doesn’t change. A beacon. If you have a central content area, then being able to find that will make your life much more simple because whatever else you might seek – it must be a child of the main content.

To do that, add a XML namespace:

    <zk xmlns:cl="client" xmlns:ca="client/attribute">

to the top of your ZUL file, you can now use a new attribute ca:data-test-id="xxx" to set a fixed attribute on an element. In Selenium code, you can now locate this element using this code:

    By.xpath( "//div[@data-test-id = 'xxx']" )

I suggest to use this sparingly in order not to bloat your DOM. But a few of them for fast moving targets will make your tests much more stable.

Debugging the DOM

Sometimes, your life would be much more simple if you could see what the DOM was when your test failed. Here is a simple trick to get a HTML fragment from the browser:

    protected JavascriptLibrary javascript = new JavascriptLibrary();

    public String dump( WebElement element ) {
        return (String) javascript.executeScript( driver,
            "return arguments[0].innerHTML", element );
    }

You can now use JTidy and JDOM to convert the fragment first into W3C XML nodes and then into JDOM elements:

    public org.w3c.dom.Document asDom( WebElement parent ) {
        String html = dump( parent );
        Tidy tidy = new Tidy();
        tidy.setShowWarnings( false );
        tidy.setErrout( new PrintWriter( new NullOutputStream() ) );
        org.w3c.dom.Document dom = tidy.parseDOM(
           new StringReader( html ), null );
        return dom;
    }

    public static org.jdom2.Document asJDOM( org.w3c.dom.Document content ) {

        // JDOM chokes on: org.jdom2.IllegalNameException: The name "html PUBLIC "-//W3C//DTD HTML 4.01//EN"" is not legal for JDOM/XML DocTypes: XML names cannot contain the character " ".
        Node docType = content.getDoctype();
        if( null != docType ) {
            content.removeChild( docType );
        }

        DOMBuilder builder = new DOMBuilder();
        org.jdom2.Document jdomDoc = builder.build( content );

        return jdomDoc;
    }

Another great use case for this is a default exception handler for tests which saves a screenshot and a copy of the DOM at the time a test fails. No more guessing why something didn’t happen.


%d bloggers like this: