Jazoon 2012: Large scale testing in an Agile world

28. June, 2012

Alan Ogilvie is working at a division of IBM responsible for testing IBM’s Java SE product. Some numbers from his presentation:

  • A build for testing is about 500MB (takes 17 min to download to a test machine)
  • There are 20 different versions (AIX, Linux, Windows, z/OS * x86, power, zSeries)
  • The different teams create 80..200 builds every day
  • The tests run on heaps from 32MB to 500GB
  • They use hardware with 1 to 128+ cores
  • 4 GC policies
  • More than 1000 different combinations of command line options
  • Some tests have to be repeated a lot of time to catch “1 out of 100” failures that happen only very rarely

That amounts to millions of test cases that run every month.

1% of them fail.

To tame this beast, the team uses two approaches:

  1. Automated failure analysis that can match error messages from the test case to known bugs
  2. Not all of the tests are run every time

The first approach makes sure that most test failures can be handled automatically. If some test is there to trigger a known bug, that shouldn’t take any time from a human – unless the test suddenly succeeds.

The second approach is more interesting: They run only a small fraction of the tests every time the test suite is started. How can that possibly work?

If you run a test today and it succeeds, you will have some confidence that it still works today. You’re not 100% sure but, well, maybe 99.5%. So you might skip this test today and mark it as “light green” in the test results (as opposed to “full green” for a test that has been run this time).

What about the next day? You’re still 98% sure. And the day after that? Well, our confidence is waning fast, so we’re still pretty sure – 90%.

The same goes for tests that fail. Unless someone did something about them (and requested that this specific test is run again), you can be pretty sure that the test would fail again. So it gets light red unlike the tests that failed today.

This way, most tests only have to be run once every 4-5 days during development.

Why would they care?

For a release, all tests need to be run. That takes three weeks.

They really can’t possibly run all tests all the time.


Mocking AJAX in jQuery

31. May, 2011

When developing small web applications, it would be great if I could mock AJAX requests.

Apparently, Jonathan Sharp had the same problem and created a solution: Mock Your Ajax Requests with Mockjax for Rapid Development

Very nice. Thank you!

 

 

 


JaCoCo: Successor of EclEmma

15. March, 2011

If you’re using EclEmma to check the code coverage, you should have a look at JaCoCo. If you’re not using code coverage, yet, you really should.

JaCoCo (Java Code Coverage Library) is the successor of EclEmma where the developers applied all the lessons learned during the time with EclEmma.


Feeling like Hercules?

21. October, 2010

In the ancient mythology, Hercules slew the Hydra, a monster which could grow a new head for each one cut off. Does that remind you of your programming job? The punchline is: Hercules didn’t conquer his opponent with brute force but with brains.

So if your product sprouts two new bugs for each you fix, it’s time to take the fingers away from the keyboard and start rubbing your head.

The solution? Unit tests. Each test makes sure that a piece of code works as it should. Even if each tests looks dump and futile, in a mass, they make sure that all the code which you didn’t change still behaves as you expect.

Kudos to Thomas E. Deutsch for the idea of the Hydra.


Dependency Injection and Reference Passing

21. July, 2010

When you start using dependency injection (DI), you probably come from the painful world of singletons. Singletons are a lie. When we were doing structured programming (remember? What we did before OO?), that was called “global variable” and everyone knew they were bad. But hey, OO came along and we had the same problem and to solve it … we used global variables. Only we didn’t name them that. We said “It’s a Singleton!” and everybody was happy.

Except that the mighty singleton has the same problems as the global variable – because they are the same thing.

A solution was sought and DI was invented. When people start to use DI, they are still in the “Singleton” mind because you can’t get rid of an idea that has served you (more or less) well over many years. Since a human can’t simply forget what he’s been doing for a long time (it’s traditional), singletons leaked into DI leading to odd design which felt wrong.

Software developers are paid for their brains. If something feels wrong, it usually is. Most of the early code we come up with then starting with DI violates the Law of Demeter.

A common solution to the problems with many singletons is to replace them with a single singleton (for example one which loads and offers the application context in Spring). While this is convenient, we still have a global variable left.

Another solution is to write constructors that take 27 parameters so you can pass in all the parameters. If you avoid that trap, then your class will have 27 setters. Holy Ugly, Batman.

How to solve that? Use more DI. Most of the 27 ex-singletons will be passed on to other worker classes. So instead of passing on the singletons to create the worker classes deep down in the code, create the worker classes using DI (so the DI framework can fill in all the ex-singletons they need) and then pass in the 2-3 workers.

For some code, see this article: Dependency Injection Myth: Reference Passing


Jazoon 2010, day 3

7. June, 2010

The last post of the series (day 1, day 2).

The Gaia satellite and Data Processing by William O’Mullane

The day started with some astronomy. Gaia is another effort for a complete sky survey (like it’s predecessor Hiparcos). It’s a “cheap” mission by ESA which costs “only” 600 Million Euros (most space missions start at 1 Billion). It’s interesting how they keep pushing the limits today. Gaia will orbit L2, 1.5 million km away from Earth (the Moon is only 300’000km away). If something goes wrong, there is no way to fix it (which is why most systems are redundant except for the main mirror, for example). The main camera has 170 CCD chips. A huge effort is taken to determine the exact position of the satellite, it’s rotation speed, precise orientation, the position of the components (like the main camera in relation to the main mirror).

It will generate such an enormous amount of data that most of it will have to be thrown away on the craft before it is downloaded. The data will be available to anyone … anyone who can store a couple of petabytes at least (1 petabyte = 1000 terrabytes).

The mind boggles 🙂

Essentials of Testing: The Tools You Need to Know by Bettina Polasek and Marco Cicolini

My topic 🙂 The talk showed how they selected a couple of tools from all the available ones out there by functionality, how they support and complete each other and how well they are supported. Here is the list they came up with:

  • jDepend to know who uses what
  • GlassFish as a means to run J2EE tests out- and inside a container
  • HtmlUnit for testing web pages outside the browser (faster than Selenium but doesn’t catch all kinds of errors)
  • Selenium for testing web pages inside the browser. Slower than HtmlUnit but can test browser specific quirks.
  • PMD to keep your code clean.
  • FEST stands for “Fixtures for Easy Software Testing” and is a library to make testing more simple. For example, you’ll find code here to test Swing UIs or mock the classes you need outside the scope of your test.
  • Mockito, a mocking framework. It takes mocking to the next level with fluent interfaces.

Rapid Application Development with Apache Wicket by Andrew Lombardi

Again a tough one. I’d have loved to attend Using Software Metrics to detect refactorings by Thomas Haug.

Wicket is another web framework for Java. “Why another one?” I hear you wail. Because they all suck? Being an Apache project, Wicket tries to suck less. It’s fully mavenized and builds upon a component framework (the simple ones are built in and you find extensions on wicketstuff.org).

Wicket revolves around the idea that you give it a plain HTML (with almost no extensions) and a piece of Java which connects parts of the HTML with the code so you can simply render your pages from these components.

I’m not 100% sold, yet. Wicket was started around 2005 and has been under the radar most of the time. This can mean that it doesn’t solve all the problems. We’ll see.

Lunch break.

Building DSLs with Eclipse by Peter Friese

I’m a huge fan of code generation. A lot of code that we write is actually pretty repetitive and I really miss my preprocessor from the good old C days. Of course, today, it’s called Model Driven Development and we use XML and model transformation and EMF and the like but still.

Peter showed how to build a small DSL with Eclipse Xtext and generate code with the help of Xpand.

Kids, when you play with DSLs, always remember:

  1. They should be limited. Don’t build general-purpose programming languages with it. Less is more (less time spent debugging and hair pulling in this case).
  2. Know what you want to achieve. The tools won’t help you there.
  3. Know your tools. You can write Towers of Hanoi in SQL but it’s not as much fun as using JavaFX instead.

After the talk, I had a long chat with Peter about DSL debuggers. To make them work, we must have (at least) the following information:

  • Position in the input stream
  • Which DSL rule was applied
  • Which template contributed code
  • The state of the session at this time (values of all parameters, etc)
  • All user supplied transformations which were applied
  • Position in the output stream
    1. Only this huge amount of data will allow us to create meaningful “stack traces” if we want to debug DSL modeling problems.

      Spring ROO – A New Level of Enterprise Java Productivity by Eberhard Wolff

      I skipped most of the talk because I juggled ideas with Peter about DSL debuggers.

      Most of my problems are still there and will be solved by Roo 1.1 (couldn’t find out a release date for that).

      My major objection with Roo: It actually generates all the code. With Grails, for example, I get a controller but the class is empty. This makes it obvious where the defaults are being used. Roo, OTOH, copies a whole slew of code and files into your project when you create a new controller. This is code that you don’t know but which you’ll have to maintain.

      High Performance File IO: the Perl/Java battle by Daniel Eichhorn and Stefan Rufer

      How well does Java fare against Perl when it comes to filtering files if you use NIO? It seems that for big files, say 500MB, Java is just 25% slower which amounts to 24s vs 19s. 24 seconds to process a 500MB file twice isn’t that bad, is it?

      Mifos – the Grameen Foundation’s Java-based Microfinance application by Michael Vorburger

      Motto: Making the world better one line of code at a time. If you heard about microfinance, this is a software which helps to run it. Next time you find yourself with some time at your hands, how about helping fight poverty and join the Mifos open source project?

      Software in the service of handicapped people: Research & Development at Otto Bock by Hans-Willem van Vliet

      Along the same lines as Mifos, Otto Bock tries to make the world better by helping disabled people with wheelchairs and prostheses.

      It was interesting to hear how complex something like a smart leg is and how much people still want to look like everyone else.

      And That’s a Wrap

      With that, Jazoon 2010 ended. For me, it wasn’t as exciting as the last three years, mainly because the keynotes were somewhat weak. Well, see you all back in 2011.


Jazoon 2010, day 2

7. June, 2010

This is my report of day 2 (see my posts about day 1).

Total Cost of Ownership by Ken Schwaber

This talk was basically about Scrum and the fact that you can’t get something of value for free. Or to put it another way around: If you save some time today by quickly hacking up a feature, you’re gonna pay in the future. There is even an interest on this, so the later you have to pay, the more expensive it will get.

So the next time your boss asks you to do something “quick”, ask him whether (s)he is aware of the total cost and whether (s)he is really willing to pay it.

Unleash your processor(s) by Václav Pech

We all know that CPU’s don’t get faster, they just reproduce faster. PCs sold today have 4 cores (and each core can execute two threads at the same time). In 2012, intel is planning to release a 50 core chip and that’s just peanuts to what you can find on your graphics card (which can have roughly 600 very simple CPUs on a single chip).

The main problem here is that we, as humans, are extremely good at parallel processing at the hardware level (most of our body continues to works while we talk, think, eat, etc.) but we’re extremely bad when thinking about parallel problems.

Concepts from the HPC world and mainframes come to the rescue: Actors, Fork/Join, Parallel Arrays, Agents and Dataflow.

The idea is to get away from the tedious synchronization and use data structures which are already thread-safe and then write simple algorithms which are invoked by a framework on an as-needed basis. Imagine you have a huge amount of images to scale to thumbnails. The algorithm is always the same and it works independent of the input. So you can allocate a number of generic worker threads. Each of them gets a copy of the algorithm at runtime plus the arguments (image and thumbnail file name).

Then you have an algorithm to traverse the directory tree which produces the input and output file names. Instead of doing everything yourself, you take a parallel array and add file names as your tree workers discover them. This will trigger the thumbnail workers.

The interesting thing here: No synchronization. You don’t even write the threads. All you do is a single call:

inParallel (filenames, thumbnailer);

The thumbnailer is just as simple:

public class Thumbnailer extends Actor {
    public void act (Object item) {
        File input = ((File[])item)[0];
        File output = ((File[])item)[0];
        ... insert favorite scaler here ...
    }
}

See? No synchronized, volatile or extends Thread. Can’t wait? Check out JSR-166y

JavaFX: Designer developer workflow by Martin Gunnarsson and Pär Sikö

Tough one since I couldn’t decide where to go. HTML 5 with WebSockets? Maven 3.0?

Mouth-wetting talk about what you could do with software if everyone was just a little bit more open. In the talk, they showed how you could draw something in Photoshop and then export the design and use it directly in JavaFX.

It also showed some of the new features of JavaFX 1.3 which seems to follow the historical model of Java: 1.2 is the first version which is really usable. But it’s nice to see some progress in the Java world at last. I just wished we’d have got these five years ago when it would have mattered 🙂

Lunch break. 🙂

JavaFX – The condemned live longer by Andreas Fürer, David Sauter and Daniel Seiler

Along the same lines as the previous talk but this time, it shows some of the dark sides of JavaFX. Mostly boils down to: If you want to do fancy graphics in JavaFX, just do it. If you want to use it for more traditional UIs , then think again. Everything but the most simple components are still missing and those which exist sometimes have ugly bugs. 😦

My conclusion: Immature, oversold technology (or in managese: Sun’s bold response to the threads imposed by RCP and Flex/AIR).

Patterns and Practices in Hibernate by Patrycja Wegrzynowicz

Hibernate might be the most successful OR mapper for Java but it’s not the most simple (which is partly because of the documentation and that the problem itself isn’t very simple). I own the standard book about Hibernate, too, and I can agree that it explains in detail all the great features of Hibernate but it doesn’t answer “Why would I use that?”

Patrycja did a code review of the examples in the book and came up with lots of small problems and a couple of major ones (like code which doesn’t lock the rows in the correct order leading to an illegal state in the database). I talked to her and she wants to put the results of her code review online. This would mean we’d get a project with correct examples for using Hibernate.

She also mentioned SOLID which is an acronym made up from acronyms and stands for:

S – SRP (Single responsibility principle),
O – OCP (Open/closed principle),
L – LSP (Liskov substitution principle),
I – ISP (Interface segregation principle),
D – DIP (Dependency inversion principle)

This blog also explains it very well.

Migration to JPA – real life experience by Jan Sliwa

It’s always interesting when marketing hits the real world and all those buzz words are stripped to the bones. Jan talked about how to build a Java application which connects data centers all over Europe which contain sensitive data (medical records). To make the data secure, they applied a simple solution: The personal data is stored on the computers of the responsible doctor and only the medical files are saved on the servers. This means that the medical data itself is anonymous.

Two of the problems they encountered were:

  1. Creating an EntityManagerFactory is expensive. When do I open/close one? Is one enough for the whole application? Do I need a pool?
  2. How do I know whether an object is detached from the session?

He also talked about problems during testing. Maybe he should read my blog more often 🙂

Managed JPA in an OSGi framework – getting the best of both worlds by Tim Ward

OSGi is a framework for the paranoid. By default, it hides everything. So how do you expose your model to both the JPA framework and all the other places where it is used?

Tim explains the problems they encountered and how they solved them. My conclusion: For your problem, OSGi is not the solution. Spring and similar frameworks have shown how to do DI properly and Maven has shown how to handle dependencies. OSGi more and more feels like a remnant from the cold war where no one trusted anyone.

That’s for all for day 2. Next: day 3.


env.js has a new website

17. May, 2010

env.js has a new website at http://www.envjs.com/.

Just in case you don’t know why you need it:

Envjs is a simulated browser environment written in javascript.

If you still don’t know: You can use env.js to simulate a web browser in unit tests (among other things).


How do I make them test?

9. March, 2010

On The Daily WTF was a nice story: Unit Tested. It’s a story how someone tried developers to test their code and failed.

It’s always the same story and always the same reason: For a change in behavior, expect a year for it to become a habit. Under the best conditions.

So how do you introduce tests to developers? Here are some tips:

1. When they fix a bug, ask “Are you sure the fix is good? How do you know? Ah, you ran the code? Good. Then run the code once more but from a little test.”

2. When they discuss a new feature and can’t decide which way to go, say “How about you write a test for the feature and then try to modify the code so it makes the test pass. This way, we’ll see quickly which way works better”

3. Write some tests yourself. Make sure they run and if they fail, talk to the guy who made them fail. Help them to fix the problem. The bad solution would be to ridicule them, or punish them. Everyone makes mistakes. The tests are there to help to make less mistakes.

If the tests become another problem in the daily routine, They won’t generate enough positive energy for anyone to bother. Result: Failure.

If you meet resistance, ask yourself: What problem do I solve? Or am I just being religious? Do I test for the sake of testing? Or do the tests make everything better for everyone? Do they reduce stress or increase stress? Why?

All these questions will have different answers depending on your specific situation and the answers will help you to find your solution. Paper can’t think, you can. If some rule (out of a book or from your boss) just doesn’t make sense, then it’s your turn to come up with a solution.

There is just one thing to keep in mind and that’s the goal: Tests must help. Everything else will follow.


Testing the Impossible: Shifting IDs

7. October, 2009

So you couldn’t resist and wrote tests against the database. With Hibernate, no less. And maybe some Spring. And to make things more simple, toString() contains the IDs of the objects in the database. Now, you want to check the results of queries and other operations using my advice how to verify several values at once.

During your first run, your DB returns some values, you copy them into a String. When you run the tests again, the tests fail because all the IDs have changed. Bummer. Now what? Just verify that the correct number of results was returned?

Don’t. assertEquals (3, list.size()) doesn’t give you a clue where to search for the error when the assert fails.

The solution is to give each ID a name. The name always stays the same and when you look at the test results, you’ll immediately know that the ID has been “fixated” (unlike when you’d replace the number with another, for example). Here is the code:

import java.util.*;

/** Helper class to fixate changing object IDs */
public class IDFixer
{
    private Map<String, String> replacements = new HashMap<String, String> ();
    private Map<String, Exception> usedNames = new HashMap<String, Exception> ();
    
    /** Assign a name to the number which follows the string <code>id=</code> in the object's
     * <code>toString()</code> method. */
    public void register (Object o, String name)
    {
        if (o == null)
            return;
        
        String s = o.toString ();
        int pos = s.indexOf ("id=");
        if (pos != -1)
            pos = s.indexOf (',', pos);
        if (pos == -1)
            throw new RuntimeException ("Can't find 'id=' in " + s);
        String search = s.substring (0, pos + 1);
        
        pos = search.lastIndexOf ('=');
        String replace = search.substring (0, pos + 1) + name + ",";
        
        add (name, search, replace);
        
        registerSpecial (o, name);
    }

    /** Add a search&replace pattern. */
    protected void add (String name, String search, String replace)
    {
        if (usedNames.containsKey (replace))
            throw new RuntimeException ("Name "+name+" is already in use", usedNames.get (replace));
        
        //System.out.println ("+++ ["+search+"] -> ["+replace+"]");
        usedNames.put (replace, new Exception ("Name was registered here"));
        replacements.put (search, replace);
    }
    
    /** Allow for special mappings */
    protected void registerSpecial (Object o, String name)
    {
        // NOP
    }

    /** Turn a <code>Collection</code> into a <code>String</code>, replacing all IDs with names. */
    public String toString (Collection c)
    {
        StringBuilder buffer = new StringBuilder (10240);
        String delim = "";
        for (Object o: c)
        {
            buffer.append (delim);
            delim = "\n";
            buffer.append (o);
        }
        return toString (buffer);
    }
    
    /** Turn a <code>Map</code> into a <code>String</code>, replacing all IDs with names. */
    public String toString (Map m)
    {
        StringBuilder buffer = new StringBuilder (10240);
        String delim = "";
        for (Iterator iter=m.entrySet ().iterator (); iter.hasNext (); )
        {
            Map.Entry e = (Map.Entry)iter.next ();
            
            buffer.append (delim);
            delim = "\n";
            buffer.append (e.getKey ());
            buffer.append ('=');
            buffer.append (e.getValue ());
        }
        return toString (buffer);
    }
    
    /** Turn an <code>Object</code> to a <code>String</code>, replacing all IDs with names.
     * 
     *  <p>If the object is a <code>Collection</code> or a <code>Map</code>, the special collection handling methods will be called. */
    public String toString (Object o)
    {
        if (o instanceof Collection)
        {
            return toString ((Collection)o);
        }
        else if (o instanceof Map)
        {
            return toString ((Map)o);
        }
        
        if (o == null)
            return "null";

        String s = o.toString ();
        for (Map.Entry<String, String> entry: replacements.entrySet ())
        {
            s = s.replace (entry.getKey (), entry.getValue ());
        }
        
        return s;
    }

    public boolean knowsAbout (String key)
    {
        return replacements.containsKey (key);
    }
}

To use the IDFixer, call register(object, name) to assign a name to whatever follows the number after id=. The method will search for this substring in the result of calling object.toString().

If you call IDFixer.toString(object) for a single object or a collection, it will replace id=number with id=name everywhere. If you have references between objects, you can register additional pattern to be replaced by overriding registerSpecial().

To make it easier to compare lists and maps in assertEquals(), each item gets its own line.