Tudor Girba gave a presentation at the Jazoon about a topic that is very dear to him: Software assessment. To quote:
What is assessment? The process of understanding a given situation to support decision-making. During software development, engineers spend as much as 50% of the overall effort on doing precisely that: they try to understand the current status of the system to know what to do next.
In other words: Assessment is a process and a set of tools to help developers to make decisions. They typical example is a bug shows up and you need to fix it. That raises the usual questions:
- What happened?
- Why did it happen?
- Where did it happen?
- How can I fix it?
As we all know, each of these steps can be difficult. As an extreme example, someone mentioned selling software to the NSA. It crashed. The NSA calls the developer:
NSA: “There is a problem with your software.”
You: “Who am I talking with?”
NSA: “Sorry, I can’t tell you that.”
You: “Well … okay. So what problem?”
NSA: “I can’t tell you that either.”
You: “… Can you give me a stack trace?”
NSA: “I’m afraid not.”
Unlikely but we all know similar situations. Even seasoned software developers are guilty of giving completely useless failure reports: “It didn’t work.” … “What are you talking about? What’s ‘it’?”
Tudor gave some nice examples how he used simple assessment tools that allow him to query log files and sources of some application to locate bugs, locate similar bugs and help to find out why some part doesn’t behave well. Examples:
- An application usually returns text in the user’s language but some rare error message is always in German. Cause: When the error message was created, the code called Locale.getDefault()
- Several other places could be found that showed the same behavior by searching the source code for places where Locale.getDefault() was called either directly or indirectly. A test case was added to prevent this from happening again.
- Some cache would have a hit ratio of less than 50%. Analyzing the logs showed that two components used the same cache. When each got their own cache, the hit ratios reached sane levels.
So assessments allow you to do strategic planning by showing you all the dependencies that some part of the code has (or the whole application).
In a spike assessment, you can analyze some small part to verify that a change would or could have the desired effect (think performance).
Did you know that developers spend about 50% of the time reading code? If tools can help them understand some piece of code faster, that makes them more productive. Unfortunately, today’s tools are pretty limited when it comes to this. Eclipse can show me who calls Locale.getDefault() but it can’t show me indirect calls.
Worse: If the developer makes the wrong decision because she couldn’t see all the important facts, then these often have a huge impact.
Another important aspect is how you use metrics. Metrics are generally useful but the same is not true for every metric. Just like you wouldn’t copy unit tests from one project to the next, you need to reevaluate the metrics that you extract from each project. Some will just be a waste of time for certain projects.
My comments:
We really, really need better tooling to chop data. IDEs should allow me to run queries against my source code, collect and aggregate data and check the results in unit tests to validate design constraints.
It was also interesting to see how Tudor works. He often uses simple words which can be misleading. But when you look at the slides, then there was this graph about some data points. Most graphs show a linear Y axis with the ticks evenly spread. He uses a different approach:
Usual diagram to the left, Tudor’s version to the right
Related links:
- http://humane-assessment.com/ “The missing method for making software engineering decisions”
- What do developers do all day?
Dark Views 