SoCraTes Day: Testing the Impossible

21. June, 2015

I’m back from SoCraTes Day Switzerland where I help a Code&Hack session called “Testing the Impossible”.

The session is based on this Mercurial repository de.pdark.testing.


  • 20150619_105103-socrates-day-testing-the-impossible-page1Space Shuttle – They Write the Right Stuff
    • Why wasn’t the bug caught by QA?
    • Why did the bug escape dev?
  • Personal Bug Diary
  • Team Culture: Strengths va. Blame Game
  • Miscommunication with Customers
  • Anyone can press the “Red Button
  • Make failures cheap. Fail fast.
  • I’m wrong. I have learned something.
  • Bug-.hunting takes time and mindshare.
  • If you found a bug, write a test for it

How do I test randomness?

  1. Test small pieces
  2. Fix things that varies (IDs, timestamps, etc.)

How do I test a DB?

  • 20150619_105103-socrates-day-testing-the-impossible-page2Layer between DB and App code which allows to switch between real DB and mock
  • Verify generated SQL instead of executing it
    • Fast
    • Allows to test thousands of combinations easily
    • Useful for code that builds search queries
  • Run SQL against real DB during the night (CI server)
  • H2 embedded DB can emulate Oracle, MySQL, PostgreSQL, …
  • Test at various depths (speed vs. accuracy)
  • Testing Walrus dives deep!

Eclipse Finance Day 2014: Testing business applications with RCPTT

3. November, 2014

RCP Testing Tool (RCPTT) is an Open Source tool for UI testing of Eclipse-based applications. During the demo by Ivan Inozemtsev, I got the impression that they thought of everything:

  • There is a recorder so you can quickly create a test case by clicking thought the application
  • There is an assertion builder where you can say “this element should be red”, “this checkbox needs to be checked”, “this image must be visible”
  • There is special support for all kinds of widgets like text editors where “position” is row/column (i.e. cursor position) instead of mouse coordinates or character offsets. The test API allows for checks like “is styled like a keyword”.
  • If the UI is resized after the tests have been recorded, the tool will try to calculate the relative click point. Unless the UI element is special (like a graph editor) where it has different heuristics (like when you clicked in an empty area but now, there is something below at the coordinate). If everything else fails, the testing tool will try to resize the UI element to the recorded size.
  • There is a compact, extensible DSL to describe the test cases. The DSL supports procedures so you can reuse test code.

The tool was fast and stable during the demo. It could reset Eclipse’s workspace to a certain state (open perspective, open editors, open projects) – I’m currently thinking about using it to create the default workspace for our development team.

One more things from the demo: Black box testing is a myth or at least a dream. A testing tool for anything complex needs to know internal details (like Eclipse’s background jobs). Otherwise the tests will either fail randomly when some background job didn’t complete in time or they will be slow (since each of them will be sprinkled with “WAIT 1 MINUTE” instructions) or they will be sprinkled with application specific “wait” instructions.

Eclipse Finance Day 2014: Automating user interface tests with behavior-driven development (BDD)

3. November, 2014

The last two talks were about testing.

“Automating user interface tests with behavior-driven development (BDD)” by Jose Badeau and Dietmar Stoll used an Xtext-based DSL to connect requirements, wireframes and test descriptions so they can validate each other. In the example, they removed a field from the wireframe mockup and the Eclipse editor for the test cases showed errors where they were used in the DSL.

The demo showed once again how much power lies in connecting information from different sources. If you remember, then this was the key feature which set Eclipse apart from all the other Java tools in 2001: It could bring you to the place where something was defined by pressing F3 or Ctrl+Click. If you changed one file, it would instantly compile all the other files and show you any problems it would find. Having a tool which actively searches for problems saved and still saves a lot of time.


Eclipse Finance Day 2014: Modernisation of Software

3. November, 2014

In his talk “Modernisation of Software“, Georg Pietrek of itemis AG shows the key points when you need to modernize software (= make large, disruptive changes). Karsten Thoms then demonstrated those points in a practical example in his talk “Model based migration of Oracle Forms applications”.

  1. Analyze the old code
  2. Create DSLs that describe the pieces of the old code which you want to migrate
  3. Write a small demo what the new app should look like
  4. Develop code generators which take the DSLs to generate the demo

After you have proven that you have covered all important bases this way, you can expand the DSLs (or rather write more code using the DSLs) to migrate the rest of the old code. You can reuse the small demo from step #3 to test any changes to the DSLs and code generators on a small code base.

The big advantage of using models to modernize is that manual migration often runs into a death spiral: Developers start with little knowledge (especially, when the original developers aren’t around anymore). They learn as they migrate code. But as this process advances, it becomes more and more expensive to fix early mistakes. With the model driven approach, you fix the bug in the code generator and thousands of lines of code are corrected at the push of a button. This means you can make abstraction work in your favor.

The advantage over exiting tools to migrate an old code base is that the model based approach is more flexible.

A quote from the talk: “Software doesn’t rot but it’s environment changes.” Eventually, the software no longer meets the needs.

Eclipse Finance Day 2014: What Finance Systems can learn from Embedded Systems

3. November, 2014

Thomas Schuetz from Protos Software showed some surprising similarities between embedded and finance systems: Both need to run a long period of time without human interaction, they shouldn’t show “odd” behavior, and you can’t simply shut them down to look for a bug.

Granted, lives are much less at risk when a financial software crashes (as opposed to, say, a pacemaker). So at first glance, the strict safety rules which apply to embedded systems seem too strict for financial software. But safety is built on top of reliability. And we very much want reliability in any system we build.

An important tool here is tracing. Tracing is the pedantic brother of logging. The goal is to collect enough data to simulate the state of the system at any point in time.

In a demonstration, he showed a demo for project eTrice. In a mix of a textual and UI editors, he created a simple application with two objects that could send data back and forth. Since everything is based on EMF, changes on one side are immediately reflected on the other. As a free bonus, you get a sequence diagram of the whole process by clicking a button after the application has finished.

Eclipse Finance Day 2014: User-Driven Open Source Communities, auch für den Finanzsektor

3. November, 2014

Matthias Stürmer from the University of Bern explained what kinds of OSS communities exist, how they work and why the financial sector should look at them.

For one, investment into open source software is always paying back: No matter what anyone else does, you will always at least get back what you invested. As an example, several public departments needed to improve Microsoft Office support in LibreOffice/OpenOffice. Not a single one of them could afford the CHF 50,000 which this would cost. But all of them could.

Another important point is that OSS never goes out of maintenance. Sure, there are many abandoned projects but that doesn’t stop anyone from taking the source and continuing the work. By definition, no one cares about an abandoned OSS project.

If you need help with OSS, there is a directory of companies which sell support:

Eclipse Finance Day 2014: Sirius – create your own graphical designers for IT and EA

3. November, 2014

Etienne Juliot from Obeo, demonstrated Sirius, a tool to create your own modeling tools in Eclipse. Under the hood, the new UI editors work on EMF models. If you struggled with EMF (and it’s … uh … “basic” set of default editors), you should definitely have a look.

The tools also work well with Xtext, so you have a mix of textual (detail) and graphical editors (overview) in your product. The magic sauce in EMF makes sure that updates on one side propagate to the other.