Jazoon 2013 Overview

25. October, 2013

Here is the list of blog posts about all the talks that I attended at Jazoon 2013:

Day 1

Day 2


Jazoon 2013 – Successful Collaboration in Agile Software Teams

25. October, 2013

Jazoon 2013 badgeIn his talk “Successful Collaboration in Agile Software Teams,” Martin Kropp lists the key factors for collaboration in agile software teams. (slides on slideshare)

For me the most important points were (slide 6):

“Agile collaboration is …

  • Face-to-face
  • Regular
  • Often
  • Informal
  • Openly
  • Under equals
  • Focused
  • Transparent
  • Respectful
  • Flexible

No is no “Agile by the Book.”

After that, he lists the usual pain points:

  • Hidden leaders
  • People sending assignments by email
  • Meetings without purpose and goals
  • Missing tools like chat, video conferencing
  • Physical vs. the digital whiteboard (slide 18)

Modern technologies like large touchscreens will make things better (slide 23).

A word about open-plan offices: They are the best tool to reduce productivity. See these articles to learn why:


Jazoon 2013 – The Economies of Scaling Software

25. October, 2013

Jazoon 2013 badgeIf you’re small, scalability is not an issue. If you’re big, you can plan for it. But what if you wake up one morning and you suddenly find you’ve become the next Google?

In his talk “The Economies of Scaling Software“, Abdelmonaim Remani talks about what scaling means, how you can plan for it (even if you don’t expect it to happen) and all the nasty details that stand between you and success (slides on slideshare).

Today, the ubiquity of the Internet has blurred the lines between consumers and enterprises. They start to ask the same questions which eventually all boil down to: “How can I find what I’m looking for in a universe of haystacks?”

One day, many of us will find themselves with the need to scale because the performance of the old solution has become unbearable and all low-hanging fruit (faster CPU, more RAM) have been picked.

To solve this, you can look at the CPU (slide 16-35) or you can start to build clusters (36-45).

Or your I/O might be the bottleneck (slides 46-60). You can solve this by looking at NoSQL databases and caching.

Is the network the problem? (slides 61-75) Start with asynchronous processing, batch jobs, content delivery networks (CDN), DNS sharding or use a different protocol to connect the various parts of your system.

But how do you know what part is the bottleneck? The answer here is monitoring. (slide 77)

Note that scaling often helps with disaster recovery, you still have to plan for it – if all nodes of your cluster are in the same room, it’s still a single point of failure (slide 79)

Software isn’t everything. Don’t forget your team (slide 82)


Jazoon 2013 – Join the Java Evolution; JCP & Adopt-a-JSR

25. October, 2013

Jazoon 2013 badgeHeather VanCura explained how the new JCP works in her talk “Join the Java Evolution; JCP & Adopt-a-JSR” (slides on slideshare)

Oracle spent the last years to make the JCP much more open and accessible. One example here are the JSRs which are much more lightweight than the complex JCPs. You can even adopt one.

Maybe you’re a long time joda-time user and want to make sure important features make it into JSR 310 – new Date and Time API for Java? Join them to discuss your need, share some code, help write, improve or translate documentation.

Some bug in the SDK nagging you? Contribute a fix to the OpenJDK. With Java 8, the OpenJDK build has been simplified tremendously. Sun ignored your bug report for years even though it contained a patch? Now is the time to change this.

I had the chance to chat with Heather after the talk which earned me a copy of “Java 7 Concurrency Cookbook”; thanks for that 🙂 We discusses a couple of ideas and she gave me points; maybe I’ll submit a few patches. If I do, I’ll blog about the experience here.


Jazoon 2013 – Real World Git Workflow

25. October, 2013

Jazoon 2013 badgeWhen using git, you really have to define a workflow that you want to use. In his talk “Real World Git Workflow” (slides on slideshare), Stefan Saasen explains some kinds of workflows and when they are appropriate.

The first step in any workflow design is asking questions. Some examples that will sound familiar:

  • “Can we fix a bug for a specific release?”
  • “Can we do a fast hotfix for the current release?”
  • “Can we build the current code?”
  • “Is the code for that feature complete?”
  • “Has everybody reviewed the code for this feature?”

Your questions will be different but make sure you ask them. There is no perfect workflow but there is your workflow. Omit this step at your own peril.

Collaboration Models

First, which collaboration model do you use?

“Anarchy” (anyone can push anywhere, slide 17), “Gatekeeper” (one person reviews all changes, often used by OSS projects, slide 18), “Dictator and Lieutenants” (Linux, slide 19) or “Centralised” (slide 20). See also “Git Workflows” on atlassian.com/git.

In the enterprise, the centralized model is often used. This approach makes it most simple to integrate all the tools (CI servers, code quality tools, deployment, …). (slide 23)

Branching Models

The two most common branching models are “continuous delivery” and “product releases.” (slide 25)

From slide 28:

Significant branches map to a concept in the outside world. It may be a past release, an environment or a role. Those branches are long-running and stable whereas feature branches are short lived and volatile.”
– Stefan Saasen

Slide 27 shows the branches for “continuous delivery”. PR is a “pull request.”

As you can see, development is consolidated in the staging branch and pushed into production (master branch) from there. If you make a hotfix in the master branch, it is cherry-picked back into staging.

Slide 30 shows how to handle “product releases.” You have a single, central repository with a master branch which consolidates the development (no staging). Each feature and bugfix happens in a short-lived “feature branch.” When a release is made (slide 31), then a new long living release branch is created. Bug fixes still happen in short lived branches.

When the bug is fixed, the fix is first merged back into the oldest affected release. From there in the next release until you get to the latest release from which it is then merged back into master (slide 33).

But what about changes you don’t want to merge like the changes made by the Maven release plugin? Maybe you will want to replace it with a better release workflow.

Or we use the correct git merge strategy: ours. This creates a new changeset with the merge information without actually merging anything. For git, it will look as if everything has been done and it won’t bother us with merging those changes ever again (slide 39).

Merge Protocols

While the above sounds reasonable, the question is why? What are the rules and forces which make this better? Stefan introduces “the merge protocol” to answer this. See slide 42 for this.

In a nutshell, you always try to merge more stable branches into less stable ones: Bug fix branch into branches where the bug hasn’t been fixed, yet. Features into branches which don’t have the feature.

That’s why you never merge master back into a release branch: Releases are most stable. You merge them into master. If you have fixed in master that you really need in a release, you cherry-pick them (slide 43).

Pull Requests

A lot of people understand why code reviews would be a good thing, “but …” Sounds familiar? Then pull requests are for you.

Pull requests are an easy, low-overhead tool to have as much “code review” as you feel comfortable with. You can merge with by clicking a button or you can review the changes line-by-line. Your choice.

Topology

Most projects will use a single canonical repository but remote forks are useful, too. Imagine you have fixed an important (for you at least) bug in a OSS project. You send them a pull request but it’s rejected! What do you do?

You fork the project. git allows you to still track the changes made by the original project while isolating your life as much as you want (slide 52)

A fork is nice if you want to do an innovation spike – code that might never be included in the product. Fork instead of polluting the project history with dead experiments (53).

Some department needs big changes to some component? Fork it until the feature stabilizes. You can still merge them if you want, but you don’t have to (54).

Reduce the noise (55). A fork allows you to rewrite history.

Hooks

You can use pre and post hooks to make everyone’s life easier. Use a local pre-commit or pre-push hooks to make sure some important tests have been run. For example, you could run FindBugs or checkstyle.

An interesting post-checkout hook would be to check whether the branch is green (66), i.e. code builds and all tests pass. Stop wasting time to search for bugs that were already there before you started your work. You can get this gem from bitly.com/green-builds (69).

Continuous Integration

The explosion of branches can quickly bog down your build server if you don’t come up with a strategy to handle this (71). Usually, it’s enough to build stable and master but developers will love it when they can manually trigger feature branch builds (72).


Jazoon 2013 – Kafka and Storm – event processing in realtime

25. October, 2013

Jazoon 2013 badgeGuido Schmutz introduces two frameworks in his talk “Kafka and Storm – event processing in realtime” (slides on slideshare)

Apache Kafka is a publish-subscribe messaging system like JMS but more simple. Messages are kept in files and never deleted. Together with the fact that subscribers have to tell the system which message they want next, this means you can recover from bugs that corrupted your data even if you notice them only after some time: Just process all the corrupted messages again.

Storm is a “distributed realtime computation system. ” It makes it easy to define topologies (= graphs) of bolts (= places where a computation takes place) flowing your real-time data through a complex network to process, filter and aggregate it. Just like Akka, it defines all kinds of operations (filters, switches, routers, …) so you can easily and quickly build the topology you need. Trident makes this set-up step even more simple.

Compared to Hadoop, Storm is meant for real-time processing. Some projects combine the two.

If you need a good framework for serializing data in Java, have a look at Apache Avro.


Jazoon 2013 – Rocket Propelled Java

25. October, 2013

Jazoon 2013 badgeRocket Propelled Java” is about being more productive by writing less boilerplate code (slides on slideshare).

In his talk, Andres Almiray showed some tools to make your code more compact while keeping or even improving readability.

The first one is lombok and lombok-pg which use Java annotations to generate a lot of boiler plate code (slides 3-6)

Similar, you can use Groovy AST transformations to create things like immutables or singletons correctly (“There are 5-6 ways to create singletons. Only two of them are correct”, slides 7-9)

Note: You might be worried to add Groovy to a pure Java project. You don’t have to. Just use it for the AST transformations to stop wasting time on writing code that only makes your compiler happy. Use Groovy as a “boilerplate code buster.”

If you ever wanted to use “Design by Contract“, GContracts is for you.


Jazoon 2013 – 33 things you want to do better

25. October, 2013

Jazoon 2013 badgeTom Bujok listed a lot of methods, technologies and frameworks that you should be aware about in his talk “33 things you want to do better” (slides on speakerdeck)

At the beginning he reminded us how quickly a well designed system goes bad due to hurried changes. We need to be aware of our technical debt and we need to allocate time to spend on reducing it (slides 3-12).

As an example, car batteries are easy to find. They are a replacement part, designers and engineers make it easy to find. Compare this to the configuration of your project. If you need to change it, how easy is it to find the file that needs to be changed and then the place in the file?

Another important point is skills. In most other professions, you have some mastery of a skill before you use it. You train hundreds of hours before you play your first football game. In Software, we show you a computer, we show you the programming language of the year (not necessarily this year’s). There is no time to master the tools you have to use from day one (slides 13-15).

“We are what we repeatedly do; excellence then, is not an act but a habit.” – Aristotle

Or as Wikipedia defines it:

“Habits […] are routines of behavior that are repeated regularly and tend to occur subconsciously.”

Stop wondering why you always make the same mistakes – they’re habits. Eliminate them ASAP (slide 19):

Bad Habits – Katherine Murdock “The Psychology of Habit”:

  • Recognize bad habits and eliminate them ASAP
  • The older you get the more difficult it is to remove a bad habit
  • Each repetition leaves its mark!

Turning bad habits into good ones – Dr. Michael Roussell, PhD.:

  • You can’t erase a habit, you can only overwrite one.
  • Insert the new habits into the current habit loops

Bad Habits

Configure your IDE properly and remove bad defaults. Replace “ex.printStackTrace();” with “throw new RuntimeException(ex.getMessage(), ex);” (slides 43-45).

One bad habit is empty catch blocks with “can never happen” comments. If you see one during a code review, replace it with “System.exit(-1);”. It can never happen, right? Right? (slides 46-47).

Note: I have create a “ShouldNotHappenException” for this case 🙂

Another one is to make every method in a static helper class public. Maybe some of them can be package private? (slide 48)

Learn about other good habits. Read books like “Effective Java” (Joshua Bloch) and “Clean Code” (Robert C. Martin) (slide 49)

Use code reviews to notice bad habits and to spread knowledge in your team but prevent blame games. (slide 73)

Learn the keyboard shortcuts of your IDE (slide 78)

Remember (slide 79):

“Any jackass can kick down a barn, but it takes a good carpenter to build one.” – Sam Ryburn

Projects You Should Know

Project lombok and lombok-pg – In a nutshell, these hook into the Java compiler and generate additional bytecode when certain annotations are present. Bored with getters, setters, hashCode() and equals() plus a nice toString()? Use @Data (slides 21-28).

Guava (slides 29-33)) is a great library with many tools that you have been missing in Java for years. You might also want to look at commons-lang.

Want to use lambda expressions but can’t upgrade to Java 8? Then lambdaj is for you (slides 34-39).

Logging? slf4j (40-42). Especially nice when combined with the @Slf4j annotation from lombok.

Bored to write all that boiler plate code to create all those services and managers that form your app? Look at Guice or Spring.

Use Spock to make tests more compact and easier to understand. (slides 50-52)

Unitils contains all the helper functions that we always missed in JUnit and Hamcrest (53-56).

JUnitParams will help you run tests with different parameters (57-59).

Need to wait for something during a test? Awaitility will help. (60-61)

When mocking isn’t enough and you need to inject code during a test, Byteman is the tool you want to look at (62-63)

Getting bored writing boiler plate code in Java to make a compiler happy? Have a look at Groovy. (64-67)

How about adding dependencies to your scripts? Try Grape. (68-69)

Is your build a mess? Do you feel Maven is too verbose or too limiting? Gradle might be for you. (70-72)

Version control is slowing you down? Have a look at Mercurial or Git (75)

Use bash or Python to automate man-/menial work. If you’re on Windows, look at Babun or Cygwin. (76-77).


Jazoon 2013 – Open Safety For Drones

25. October, 2013

Jazoon 2013 badgeDrones – fully or partially autonomous aircraft – are the second most wide-spread form of private-owned robots today only challenged by room-cleaners. The current estimation is that there are 500’000 drones currently in use. Parrot AR alone sold about 250’000 units. In his talk “Open Safety For Drones,” Lorenz Meier asks important questions about safety, regulation and how this area will develop in the future.

When it comes to safety, then there are two areas: Safety when developing drones and safety when using them. People have started to demonstrate the usual stupidity by flying drones in crowded areas without regard to safety. Lorenz showed a video where an inexperienced controller flew a drone in Manhattan until it crashed into skyscraper and dropped into the streets below. In another example, someone flew a drone through a campus. I say “through” because he went below footbridges that connect the first floor of buildings, eventually passing a bus on the way. The “pilot” also failed to catch the drone after the stunt, crashing it into a hedge.

It will be interesting to see how legal bodies like the FAA respond to things like this.

Despite this, autonomous drones are already useful. A nice example is the new 3D model of the Matterhorn.

A more related issue is the other one: Building safe aircraft. Bugs in a drone can really ruin your day (or life). How can we make sure that hundreds of thousands of lines of code have an acceptable number of bugs?

The issues here are many:

  • If you want to fly an autonomous aircraft, you need to license it.
  • Regulation for drones are non-existing or immature. They might try to handle you like a normal aircraft manufacturer (think Boeing or Airbus). Can you even afford to apply for such a license?
  • You’re using an image recognition software. It uses a math library. In that library, a single line of code has changed. What is the impact on the safety of your drone? How do we bring safety and the need for faster iterations together?
  • Regulators might insist that you license the whole thing again.
  • To get real engineers on your project for a new drone, your API must be really simple. As every seasoned developer knows, nothing is harder than “simple.”

A word on the safety of military drones: Not a single one of them is allowed in civilian airspace. But it’s probably very expensive. Germany spent €562 million to certify the EuroHawk drone. The project was cancelled and the defense minister “referred to the project as being ‘a horror without end'”.

So how can we make a DIY drone ever safe?

There are some approaches:

  1. Involve as many people as possible in a project. Everyone will find something that can be improved.
  2. Allow commercial use of your project. Companies have completely different views on safety than hobbyists.
  3. Collect logs of your flights and share them. Share your project on github and make sure you include the hash of the changeset that was used to run the drone.
    Over time, you can use statistical data to locate even “once in a million years” bugs.

But there are good news, too. With a plane, you have X people on the aircraft and 0 on the ground (most crash in empty areas). With a drone, you have 0 people on the aircraft and X on the ground. But using the laws of physics, it’s pretty simple to estimate the maximum radius that a drone can still travel if it’s “brain” shuts down right now. That means we can test them pretty safely in a virtual cage. Also, since all the components are open, we can run extensive simulations before bringing the thing into the wild.

But the OSS approach also has drawbacks. With OSS, a unit test is easy to write, the test case is known and if something goes horribly wrong, we might experience data loss.

With unmanned systems, the worst case is loss of life. And where do you get your test cases from? How do you plan to test for a broken coil in an actuator? An overheated battery which results in unstable power? Wind gusts? Collisions with birds? Interference with other electrical systems?

Maybe it’s time to open a new field:

Open Safety – Build a safety track record in confined test areas in an open, community-driven process, and only let statistically proven safe systems out of the sandbox. SHARE validation results on the library and version level across systems.

Open Safety Manifesto

  1. Dropping is Free – Ground impact must only damage vehicle
  2. Anytime Shutdown – A separate shutdown unit is present to enforce a drop
  3. Built to Drop – Battery protected against impact, parachutes, etc.
  4. Every Flight is Logged
  5. Every Log is Reviewed – Can be done on Droneshare

State of the Art

Sites like Droneshare have collected so many flight hours for some drones to satisfy normal regulation rules.

Keep in mind that certification alone doesn’t make things safer, they just help to catch the most common errors. Simulations and test protocols improve the situation further. Lastly, lots of people mean that every part is tested in thousands of ways. Software developers know how quickly someone else finds problems with their work (one reason why pairing is so unpopular).

A big problem is YouTube since it teases out the “I more stupid” behavior in humans.

Military (Ab)Use

Another issue is the (ab-)use of this work in military drones used to kill. The scene is aware of this but they currently feel that the good (like cheap access to aerial photography, autonomous mesh networks built by drones in disaster areas) to outweigh the risk. A more visible discussion about OSS and hardware in military projects would be welcome. Some projects forbid military use (Gnutella). It’s worth noting that the US Navy has started to buy Linux based drones. Personally, I find things like the Samsung SGR-A1 military robot sentry (with 5.56 mm robotic machine gun and a grenade launcher) disturbing. Especially when it’s being advertised to be able to “identify and shoot a target automatically from over two miles (3.2 km) away.