Jazoon 2013 – Kafka and Storm – event processing in realtime

25. October, 2013

Jazoon 2013 badgeGuido Schmutz introduces two frameworks in his talk “Kafka and Storm – event processing in realtime” (slides on slideshare)

Apache Kafka is a publish-subscribe messaging system like JMS but more simple. Messages are kept in files and never deleted. Together with the fact that subscribers have to tell the system which message they want next, this means you can recover from bugs that corrupted your data even if you notice them only after some time: Just process all the corrupted messages again.

Storm is a “distributed realtime computation system. ” It makes it easy to define topologies (= graphs) of bolts (= places where a computation takes place) flowing your real-time data through a complex network to process, filter and aggregate it. Just like Akka, it defines all kinds of operations (filters, switches, routers, …) so you can easily and quickly build the topology you need. Trident makes this set-up step even more simple.

Compared to Hadoop, Storm is meant for real-time processing. Some projects combine the two.

If you need a good framework for serializing data in Java, have a look at Apache Avro.


Jazoon 2012: Akka 2.0 – Scaling up and out with Actors

29. June, 2012

Concurrency is too hard but we need it. In his talk “Akka 2.0 – Scaling up and out with Actors,” Viktor Johan Klang showed new features of Akka 2.0.

The framework now uses Future to create pipes between actors and Promise to write data to, say, a stream (docs).

To make error handling more simple, there is now “parental supervision.”

Decoupling actors becomes even more with the Event Bus API.

There is support for ZeroMQ to create grids/meshes of actors (docs).

But every framework has its limitations. If you hit one of those, it’s usually either “Use the Source, Luke” or “You’re out of luck”. Akka 2.0 comes with a new extensions mechanism to hook into the framework.