Jazoon 2012: Large scale testing in an Agile world

28. June, 2012

Alan Ogilvie is working at a division of IBM responsible for testing IBM’s Java SE product. Some numbers from his presentation:

  • A build for testing is about 500MB (takes 17 min to download to a test machine)
  • There are 20 different versions (AIX, Linux, Windows, z/OS * x86, power, zSeries)
  • The different teams create 80..200 builds every day
  • The tests run on heaps from 32MB to 500GB
  • They use hardware with 1 to 128+ cores
  • 4 GC policies
  • More than 1000 different combinations of command line options
  • Some tests have to be repeated a lot of time to catch “1 out of 100” failures that happen only very rarely

That amounts to millions of test cases that run every month.

1% of them fail.

To tame this beast, the team uses two approaches:

  1. Automated failure analysis that can match error messages from the test case to known bugs
  2. Not all of the tests are run every time

The first approach makes sure that most test failures can be handled automatically. If some test is there to trigger a known bug, that shouldn’t take any time from a human – unless the test suddenly succeeds.

The second approach is more interesting: They run only a small fraction of the tests every time the test suite is started. How can that possibly work?

If you run a test today and it succeeds, you will have some confidence that it still works today. You’re not 100% sure but, well, maybe 99.5%. So you might skip this test today and mark it as “light green” in the test results (as opposed to “full green” for a test that has been run this time).

What about the next day? You’re still 98% sure. And the day after that? Well, our confidence is waning fast, so we’re still pretty sure – 90%.

The same goes for tests that fail. Unless someone did something about them (and requested that this specific test is run again), you can be pretty sure that the test would fail again. So it gets light red unlike the tests that failed today.

This way, most tests only have to be run once every 4-5 days during development.

Why would they care?

For a release, all tests need to be run. That takes three weeks.

They really can’t possibly run all tests all the time.


Jazoon 2012: Development Next – And Now For Something Completely Different?

28. June, 2012

Dave Thomas gave the keynote speech (link) about how technology seems to change all around us just to show up the same, old problems over and over again.

Things that I took home:

  • In the future, queries will me more important than languages
  • Big data is big

Some comments from me:

How often were you irritated by how source code from someone else looked? I don’t mean sloppy, I mean indentation or how they place spaces and braces. In 2012, it should be possible to separate the model (source code) from the view (text editor) – why can’t my IDE simply show me the source in the way that I like and keep the source code in a nice, common format? (see Bug 45423 – [formatting] Separate presentation from formatting)

And how often have you wondered “Which parts of the code call Locale.getDefault() either directly or indirectly?”

How often did you need to make a large-scale change in the source code which could have been done with a refactoring in a few minutes – but writing the refactoring would have taken days because there simply are no tools to quickly write even simple refactorings in your IDE?

Imagine this: You can load the complete AST of your source code into a NoSQL database. And all the XML files. And the configuration files. And the UML. And everything else. And then create links between those parts. And query those links … apply a piece of JavaScript to each matching node …

Customers of my applications always want new business reports every day. It takes way too long to build these reports. It takes too much effort to change them. And it’s impossible to know whether a report is correct because no one can write test cases for them.


e4 is here

28. June, 2012

Unless you live under a rock or you’re not using Eclipse, you can’t possibly have missed it: Eclipse Juno (or 4.2)

If you’re unsure what happened to 3.8 or 4.0 and 4.1, here is the story in a nutshell: A small team of Eclipse developers was concerned about the state of the platform (platform is the part of Eclipse that knows what a plug-in is, how to find, install and load them and arrange them in a neat way called “perspective”). When the platform was developed 2004, it was based on VisualAge. Don’t ask. Like all “visual” tools, it had … issues. Many of them were hardcoded into the platform. For example, there are many singletons in there. Many. Testing the platform is either a nightmare or slow or both.

The e4 team decided that something needed to be done. Only, it was risky. And it would take a long time. Which meant that the PMCs probably wouldn’t approve.

Therefore, they came up with a cunning plan: Develop a new platform (e4) alongside the official Eclipse releases. e4 would get a “compatibility layer” that would allow to run the old junk (like the JDT) but it would also allow to write plug-ins in a new, clean, understandable way. For example, in 3.x, you can use this code to get the current selection:

ISelection selection = getSite().getWorkbenchWindow().getSelectionService().getSelection();
Object item = ((IStructuredSelection)selection).getFirstElement();

The same code looks a bit different in e4:

@Inject
void setSelection(@Optional @Named(IServiceConstants.ACTIVE_SELECTION) Contact contact) {
    ...
}

e4 will call this method every time the user selects an instance of Contact in the UI.

The whitepaper gives some more details. Lars Vogel wrote a pretty long Eclipse 4 RCP tutorial.

After two years, e4 was deemed stable enough to make it the default release.

Conclusion: e4 doesn’t get rid of all the problems (p2, OSGi and SWT are still there) but it’s a huge step forward.

Related links:


Java Pitfalls: Covariant Return Types

25. June, 2012

Sometimes, covariant return types (= methods overriding existing methods which return a sub-type) can bite you. Especially when you want them to have annotations.

Ralf Stuckert wrote a good blog post about the reasons: The Covariant Return Type Abyssal


When Agile Fails …

18. June, 2012

When an agile approach fails, then remember rule #0: No dogmas.

Agile is all about being non-dogmatic. If a rule doesn’t work in your case, find something that works.

You need UML? Use it. UML only slows you down? Drop it. UML might have some value for you in some specific cases? Apply it in a smart way.


Re: Curing the Problem of Software Patents, by Michael Risch

16. June, 2012

Michael Risch wrote a longer piece on software patents and why we should try to make software patents work: Curing the Problem of Software Patents

Here is my answer:

I think the main problem with the patent system isn’t the ideas behind but that some people have started to abuse it. And those who didn’t are being driven to join the brawl for no other reason than protect themselves.

You quoted “Software is hardware in every sense that matters for patents.” Let’s have a deeper look at that. Software is a formalized way to write recipes (as in cooking). Software development is a translation process (as in language interpreter). We take ideas from customers and turn them into detailed instructions to be executed by a moron. A human could do it but since it would be boring, we use a computer. Note that software isn’t mathematics – it’s language (as in English). It’s very limited (because the computer is so very dumb) but we’re more authors than mathematicians.

Now, my questions are:

  1. Can you patent the work of an online interpreter?
  2. How about the words that come out of her/his mouth?
  3. What about the words that she/he hears and translates?

My answers:

#1: No. The process of translating something is a craft. Crafts as such are not patentable.

#2: Since the interpreter doesn’t add anything of value to the input (they should stay true to what is given them), the output can’t be more or less than the input. It’s the input in another form but by itself, it doesn’t make the input more or less patentable.

#3 This leads to the question: Can you patent a recipe?

And the answer here is, sadly, yes. Which leads to the question: Can you patent any recipe?

No: The recipe has to be, well, “outstanding” in some way. Unfortunately, there is no universal standard what “outstanding” means. When I teach a newbie, they will think my methods are outstanding but they are just by relative comparison of what they know and what I know.

Since “outstanding” doesn’t seem to be a good tool to make a decision, what else do we have?

Damage. One of the roots of patent law is the idea to bring more justice to the world. It was invented for the specific purpose to protect the work of “helpless” inventors so they wouldn’t be ripped off all the time.

This makes sense if you, say, develop a new medicine. As of today, you need to spend around one billion dollars for a new medicine. Without patent law, it simply wouldn’t make sense for corporations to take the involved risks, so it makes sense to apply it here.

But software patents are in a completely different league. They cost $20’000 or less to produce and they can cause hundreds of billions of damage when they are used as a weapon – which is the sole and main purpose of software patents.

Since software patents are used solely to damage society as a whole (forcing companies to invest in them, costs of defending yourself against a lawsuit no matter if it’s justified or not, costs of canceling abusive patents, costs in lost revenue when you can’t sell your product because a competitor wants more market share), they need to be outlawed.

If you fail in this task, then no new computer related products will be sold in the USA by … well … 2013 because everyone will be suing everyone and all money will flow into courts and to patent lawyers. No more software development will happen because it’s just too expensive/dangerous. Maybe someone will find a patent to “display text remotely” and take down Google or the whole Internet (at least the part that runs in the USA).

The lawsuits between Apple and Samsung should be a warning shot. These only exist to give one company a bigger market share in a saturated market. Better product doesn’t count nor how “inventive” it is. It’s just the logical, inevitable conclusion what happens when all players follow the rules that we made.

Or Oracle vs. Google. Oracle came in demanding several billion dollars and got nothing. But if they had a different judge, the outcome could have been completely different. Is that what we want?

This seems to damage companies like Swype (see “An Example” in his post). But does it really?

First of all, someone could copy their idea. But customers would only buy the copy when it was better for most of them. Do we really want to protect something that most people think worse?

Copying an idea usually leads to a new, very similar but still different idea. The “thief” adds his own wisdom to it. We call that process “learning”. Isn’t that something that we should support?

If Swype can’t patent their idea, they can still make a product and sell it. If they fail, some money will be lost. But if they succeed, no other company can sue them for billions of dollars for no other reason than to slow them down. Isn’t that better than the other way around?

We say that the Internet Year is only three months. New ideas spring up so quickly by now that the old, sluggish patent system can never keep up. Do we want to slow down progress (might be a good idea) *and* feed the trolls? How much is an idea worth protecting if it’s outdated in 9 months?

How much more money will we make/safe if software patents are outlawed and all this money goes back into R&D?

Related:


If You Want Them to RTFM, Make a Better FM

16. May, 2012

If you’re interested in the conflicts between software and documentation, Alex Lagarde wrote a great post about that: Intent Discovery – Part 1 : the intents behind softwares.

It explains what the different (and conflicting) goals are between the “technical space” (source code), “model space” (design and architecture) and “document space” (what people need to use the software successfully). And he promises some solutions with the new Eclipse project “Intent” (and the second post in the series).


Eclipse Suddenly Takes Long to Start Unit Tests

10. May, 2012

When starting a JUnit test in Eclipse suddenly takes ages (the process starts quickly as you can see in the Console view but it takes ages until the tree of tests appears in the JUnit view), you might experience troubles with IPv6.

The background of the issue: Modern OSs assign your network cards two addresses, one for the old IPv4 and one for the new IPv6. When IPv6 isn’t configured correctly, Eclipse will try to connect via this route and it will take some 30 seconds for Eclipse and the JUnitRunner process to begin talking to each other.

The quick fix is to disable IPv6 or to tell Java to prefer IPv4:


Sharing Source Code

1. May, 2012

One would think that “open source” is all about sharing. But that’s a misconception. Example: Try to use some GPL‘d code in your non-GPL OSS project. Oh, the humanity.

So called “proprietary” software at least believes in “buying love”. They won’t show you the source but for a price, you can at least use their work without many questions asked.

OSS is different. If you use the wrong license, you must be a moron (proof: You’re using a different license than me. QED) Nobody wants to share their hard work with morons!

Especially not since the process to select the “perfect” OSS license is so painful. You need to read legalese, try to understand it, reason with the nice smiling person on the other side of the padded wall (a.k.a “outside”) that you’re not insane – the rest of the world is and you can prove it.

Ever tried to get some OSS project to share their code under second license? It’s a lot of fun – unless you’re serious. Then … it’s not so much fun.

Why I’m ranting?

I spend a lot of time on stackoverflow. It’s cool. It’s full of source.

But can you use any piece of that source code in your OSS project?

Are you sure?

You are. Splendid. Do you really think a lawyer would see this the same way?


How To Disable IPv6 In Ubuntu

30. April, 2012

If you’re getting errors from squid or your web browser with some weird IP address, chances are that squid is trying to connect to a server which supports IPv4 and IPv6.

To disable IPv6 on Ubuntu (also Kubuntu), add this line to the end of /etc/sysctl.conf:

net.ipv6.conf.all.disable_ipv6 = 1

Restart your computer or reload sysctl.conf by executing: sudo sysctl -p

To check, run: ip a | grep inet6

When IPv6 is disabled, this will give no output.

Source: Quick Tip: How To Disable IPv6 In Ubuntu For Better Internet Speed