Jazoon 2012: CQRS – Trauma treatment for architects

4. July, 2012

A few years ago, concurrency and scalability were a hype. Today, it’s a must. But how do you write applications that scale painlessly?

Command and Query Responsibility Segregation (CQRS) is an architectural pattern to address these problems. In his talk, Allard Buijze gave a good introduction. First, some of the problems of the standard approach. Your database, everyone says, must be normalized.

That can lead to a couple of problems:

  • Historic data changes
  • The data model is neither optimized for writes nor for queries

The first problem can result in a scenario like this. Imagine you have a report that tells you the annual turnover. You run the report for 2009 in January, 2010. You run the same report again in 2011 and 2012 and each time, the annual turnover of 2009 gets bigger. What is going on?

The data model is in third normal form. This is great, no data duplication. It’s not so great when data can change over time. So if your invoices point to the products and the products point to the prices, any change of a price will also change all the existing invoices. Or when customers move, all the addresses on the invoices change. There is no way to tell where you sent something.

The solution is to add “valid time range” to each price, address, …, which makes your SQL hideous and helps to keep your bug tracker filled.

It will also make your queries slow since you will need lots and lots of joins. These joins will eventually get in conflict with your updates. Deadlocks occur.

On the architectural side, some problems will be much easier to solve if you ignore the layer boundaries. You will end up business logic in the persistence layer.

Don’t get me wrong. All these problems can be solved but the question here is: Is this amount of pain really necessary?

CQRS to the rescue. The basic idea is to use two domain models instead of one. Sounds like more work? That depends.

With CQRS, you will have more code to maintain but the code will be much more simple. There will be more tables and data will be duplicated in the database but there will never be deadlocks, queries won’t need joins in the usual case (you could get rid of all joins if you wanted). So you trade bugs for code.

How does it work? Split your application into two main parts. One part takes user input and turns that into events which are published. Listeners will then process the events.

Some listeners will write the events into the database. If you need to, you will be able to replay these later. Imagine your customer calls you because of some bug. Instead of asking your customer to explain what happened, you go to the database, copy the events into a test system and replay them. It might take a few minutes but eventually, you will have a system which is in the exact same state as when the bug happened.

Some other listeners will process the events and generate more events (which will also be written to the database). Imagine the event “checkout”. It will contain the current content of the shopping cart. You write that into the database. You need to know what was in the shopping basket? Look for this event.

The trick here is that the event is “independent”. It doesn’t contain foreign keys but immutables or value objects. The value objects are written into a new table. That makes sure that when you come back 10 years later, you will see the exact same shopping cart as the customer saw when she ordered.

When you need to display the shopping cart, you won’t need to join 8 tables. Instead, you’ll need to query 1-2 tables for the ID of the shopping cart. One table will have the header with the customer address, the order number, the date, the total and the second table will contain the items. If you wanted, you could add the foreign keys to the product definition tables but you don’t have to. If that’s enough for you, those two tables could be completely independent of any other table in your database.

The code to fill the database gets the event as input (no database access to read anything from anywhere) and it will only write to those two tables. Minimum amount of dependencies.

The code to display the cart will only need to read those two tables. No deadlocks possible.

The code will be incredibly simple.

If you make a mistake somewhere, you can always replay all the events with the fixed code.

For tests, you can replay the events. No need to a human to click buttons in a web browser (not more than once, anyway).

Since you don’t need foreign keys unless you want to, you can spread the data model over different databases, computers, data centers. Some data would be better in a NoSQL repository? No problem.

Something crashes? Fix the problem, replay the events which got lost.

Instead of developing one huge monster model where each change possibly dirties some existing feature, you can imagine CQRS as developing thousands of mini-applications that work together.

And the best feature: It allows you to retroactively add features. Imagine you want to give users credits for some action. The idea is born one year after the action was added. In a traditional application, it will be hard to assign credit to the existing users. With CQRS, you simply implement the feature, set up the listeners, disable the listeners which already ran (so the action isn’t executed again) and replay the events. Presto, all the existing users will have their credit.


Jazoon 2012: Building Scalable, Highly Concurrent and Fault-Tolerant Systems: Lessons Learned

29. June, 2012

What do Cloud Computing, multi-core processors and Big Data have in common?


In his presentation, Jonas Bonér showed what you should care about:

  • Always prefer immutable
  • Separate concerns in different layers with the minimum amount of dependencies
  • Separate error handling from the business logic
  • There is no free lunch: For every feature, you will have to pay a price
  • Avoid using RPC/RMI. Try lure you into “convenience over correctness”
  • Make sure you handle timeouts correctly
  • Use CALM if you can
  • Not all your data needs ACID.
  • Know about CAP and BASEDrop ACID And Think About Data
  • Get rid of dependencies by using event sourcing/CQS/CQRS
  • Frameworks like Hibernate always leak in places where you can’t have it. KISS.

Longer explanation:

Immutables can always be shared between threads. Usually, they are also simple to share between processes, even when they run on different computers. Trying locks and clever concurrency will only get you more bugs, unmaintainable code and a heart attack.

Dependencies kill a project faster and more efficiently than almost any other technique. Avoid them. Split your projects into Maven modules. You can’t import what you don’t have on the classpath.

Error handling in your business logic (BL) will bloat the code and make it harder to maintain. Business logic can’t handle database failures. Parameters should have been validated before they were passed to business logic. Business logic should produce a result and the caller should then decide what to do with it (instead of mixing persistence code into your business layer). The BL shouldn’t be aware that the data comes from a database or that the result goes back into a database. What would your unit tests say? See also Akka 2.0 and “parental supervision.”

Obvious programming has a value: You can see what happens. It has a price: Boiler plate code. You can try to hide this but it will still leak. Hibernate is a prefect example for this. Yes, it hides the fact that getChildren() needs to run a query against the database – unless the entity leaks outside of your transaction. It does generate proxies to save you from seeing the query but that can break equals().

Same applies to RMI. When RMI decides that you can’t handle the message, then you won’t even see it. In many cases, a slightly “unusual” message (like one with additional fields) wouldn’t hurt.

As soon as you add RMI or clustering, you add an invisible network in your method calls. Make sure you have the correct timeouts (so your callers don’t block forever) and that you handle them correctly. New error sources that are caused adding the network:

  1. Failure to serialize the message
  2. Host unreachable
  3. Packet drops
  4. Network lag
  5. Destination doesn’t accept message because of configuration error
  6. Message is sent to the wrong destination
  7. Destination can’t read message
Claim checks allow to resend a message again after a timeout without having it processed twice by the consumer.

CALM and BASE refer to the fact that you can only have two of the tree CAP characteristics: Consistency, Availability and Partition Tolerance. Since Partition Tolerance (necessary for scaling) and Availability (what’s the point of having a consistent but dead database?) are most important, you have to sacrifice consistency. CALM and BASE show ways to eventually reach consistency, even without manual intervention. For all data related to money, you will want consistency as well but think about it: How many accounts are there in your database? And how many comments? Is ACID really necessary for each comment?

Solution: Put your important data (when money is involved) into an old school relational database. Single instance. Feed that database with queues, so it doesn’t hurt (much) when it goes down once in a while. Put comments, recommendations, shopping carts into a NoSQL database. So what if a shopping cart isn’t synchronized over all your partitions? Just make sure that users stay on one shard and they will only notice when the shard dies and you can’t restore the shopping cart quickly enough from the event stream.

Which event stream? The one which your CQRS design created. More on that in another post. You might also want to look at Akka 2.0 which comes with a new EventBus.

What’s Wrong With XA/Two Phase Commit

27. June, 2011

To recap, two phase commit (XA) means that you have two or more systems which take part in a single transaction. The first phase ask all system “are you 100% sure that you can commit?” (prepare) and the second phase is the actual commit.

Of course, the answer to the first question can be a lie. You can never be 100% sure that a commit will go through. The system might crash in the middle of the actual commit or the network my break between prepare and commit. Doom.

So XA doesn’t when it should. How can you solve this?

By using a less brittle protocol. Imagine you want to copy data from database A to B. You could use XA and clean up the mess every once in a while.

Or you could add a field to A which says “this has been copied to B” and when inserting data into B, you must ignore data that is already there. Here is the pseudo code:

  1. Create the connections
  2. Find all rows in A that don’t have the flag set
  3. Read all the rows and insert them into B. If a row already exists, skip it.
  4. Set the flags for all copied rows in A.

Note: You can’t use MAX(ID) or MAX(TIMESTAMP) here. Why not? Imagine:

  1. TX1 is created in A and a row is inserted. MAX(ID) == 2
  2. TX2 is created in A and a row is inserted. MAX(ID) == 3! At this time, you can’t read WHERE ID == 2 but you can conclude from the gap in the ID values that it will soon exist.
  3. TX2 is committed.
  4. TX3 is created to copy the data to B. TX1 is still running! MAX(ID) == 3 but row 2 will not be copied!
  5. TX3 ends without row 2. Since MAX(ID) is now 3, it won’t ever be copied.

If you’d rather avoid a “to copy” or “has been copied” flag in each row, create a “to transfer” table which contains IDs of the rows to copy. In step #4, delete the rows that have to be copied.


  1. Resilient. If the transfer fails in the middle for any reason, it can pick up where it left of. In the worst case, a lot of data will be copied again but data will never be lost. In the usual case, no data will be copied twice.
  2. If you make a mistake, chances are that it won’t have matter much. Say you forget to set the “has been copied” flag. Well, for every transfer, too much data will be copied but it will still work. Slower than expected but you won’t lose data. The database will always be consistent!
  3. Say something goes wrong in B and you need to transfer the data again. With my approach, you just reset the flags. It doesn’t matter if you can’t say for sure which flags to reset. If in doubt, reset all of them. The algorithm will heal itself.
  4. You can copy the data in chunk sizes of your choice. It doesn’t matter if you copy everything in one go or in blocks of 1’000 rows or row by row.
  5. It’s a simple algorithm, so it will be quick to implement and there is only a small chance for bugs. Even if there is a bug, in most cases, you will be able to resolve the situation with one or two simple SQL queries.

Disadvantages:  The XA sales guy won’t make a buck from it. Expect some resistance.


Jazoon 2011, Day 2 – NoSQL – Schemaless Data-stores Not Only for the Cloud – Thomas Schank

26. June, 2011

NoSQL – Schemaless Data-stores Not Only for the Cloud – Thomas Schank

Thomas gave an overview of some NoSQL databases and the theoretical background of it.

The main points are: SQL databases get inefficient as the data grows and if you need to split the data between instances (how do you join tables between two DB servers? Even if you can, performance is going to suffer).

But there are new problems: Data can be inconsistent for a while (keyword: MVCC).

OTOH, these databases don’t need locks and, as Amazon demonstrated, any kind of lock will eventually become a bottleneck:

Each node in a system should be able to make decisions purely based on local state. If you need to do something under high load with failures occurring and you need to reach agreement, you’re lost. If you’re concerned about scalability, any algorithm that forces you to run agreement will eventually become your bottleneck. Take that as a given.

Werner Vogels, Amazon CTO and Vice President

And since the servers can heal inconsistencies, broken connections don’t mean the end of the world. The acronym of the hour is BASE: Basically Available, Soft-state, Eventually consistent.

Interesting stuff, especially since every company owns a super-computer today. My own team has one with 64 cores, 64GB of RAM and 16TB of disk space sitting in 8 boxes spread under the desks of the developers. Not much but it only costs $8 000. And if we need more, we can simply buy another node. Unfortunately, it’s hard to leverage this power. I’ll come back to that in a later post.

One thing to take with you: If you can’t stand JavaScript but you need to write it, have a look at CoffeeScript.


Declaring Relations Between Model Instances

14. March, 2011

I’ve been playing with Xtext 1.0 the last three weeks. The editor support is most impressive. It’s one of those well balanced technologies that work “right” out of the box, offer lots of places where to hook in to tweak things.

The last few days, I’ve been chewing on a hard problem: Relations. I could to this:

class A {

class B {
    A parent 1:*;

Read: Instances of B have a field “parent” that references instances of A. There is a list in A with all instances of B where the field “parent” is that exact instance. In simple words, it’s a parent-child relation between A and B. For each B, there is always one A. Each A can have several B’s attached.

Or like this:

class A {

class B {

relation A children 1:* B parent;

The first solution avoids repetition but when looking at A, you will miss the fact that it has fields. It looks empty but the declaration in B adds a field. I don’t like it. While it might work in this case, how do you map N:M? Where do you declare a many-to-many relation? In A or in B? My guts say: Don’t do it.

The second solution is also flawed if less so. Here, both classes are empty. I have to look up the relations to see what else is going on. And I have to repeat information. I don’t like it.

Time to take a step back and sleep over it.

Fast forward over the weekend.

How about this:

class A {
parent of
class B {

One word: Intent. A construct in a programming language should clearly communicate intent. Which is one of the weak points in Java: It’s hard to express what you intend with Java. It’s not impossible but it takes years of experience to avoid the luring shortcuts.

So this new syntax clearly states what I want: A’s are parents of B’s. No repetition. No odd “1:*” which every reader has to decode. Which you can get wrong.

But what if A is parent of more classes? Simple:

class A { }
parent of {
    class B { } parent of class X { }
    class C { }

See? It nests. Let’s add tree-like structures to the mix:

tree class A { }
parent of {
    class B { } parent of class X { }
    class C { }

So A gets a parent-child relation with itself. One word says everything: Ownership. Cardinality. Field names.

My gut likes it. Listen to your guts. Especially when a simple solution tries to lure you into a shortcut.

TNBT: Persistence

19. February, 2011

In this issue of “The Next Big Thing”, I’ll talk about something that every software uses and which is always developed again from scratch for every application: Persistence.

Every application needs to load data from “somewhere” (user preferences, config settings, data to process) and after processing the data, it needs to save the results. Persistence is the most important feature of any software. Without it, the code would be useless.

Oddly, the most important area of the software isn’t a shiny skyscraper but a swamp: Muddy, boggy, suffocating.

Therefore, the next big thing in software development must make loading and saving data a bliss. Some features it needs to have:

  • Transaction contexts to define which data needs to be rolled back in case of an error. Changes to the data model must be atomic by default. Even if I add 5,000 elements at once, either all or none of them must be added when an error happens.
  • Persistence must be transparent. The language should support rules how to transform data for a specific storage (file, database) but these should be generic. I don’t want to poison my data model with thousands of annotations.
  • All types must support persistence by default; not being able to be persisted must be the exception.
  • Creating a binary file format must be as simple as defining the XML format.
  • It must have optimizers (which run in the background like garbage collection runs today) that determine how much of the model graph needs to be loaded from a storage.

Related Articles:

  • The Next Best Thing – Series in my blog where I dream about the future of software development

ORA-01461: can bind a LONG value only for insert into a LONG column tips

23. August, 2010

Should you ever encounter this message:

ORA-01461: can bind a LONG value only for insert into a LONG column tips

then chances are you tried to insert more than 4096 characters at once.

Again the price for useless error messages goes to … Oracle! 😉

To add insult to injury, I couldn’t find anything useful related to this error with Oracle’s own site search.