Of course, the answer to the first question can be a lie. You can never be 100% sure that a commit will go through. The system might crash in the middle of the actual commit or the network my break between prepare and commit. Doom.

So XA doesn’t when it should. How can you solve this?

By using a less brittle protocol. Imagine you want to copy data from database A to B. You could use XA and clean up the mess every once in a while.

Or you could add a field to A which says “this has been copied to B” and when inserting data into B, you must ignore data that is already there. Here is the pseudo code:

1. Create the connections
2. Find all rows in A that don’t have the flag set
3. Read all the rows and insert them into B. If a row already exists, skip it.
4. Set the flags for all copied rows in A.

Note: You can’t use MAX(ID) or MAX(TIMESTAMP) here. Why not? Imagine:

Thomas gave an overview of some NoSQL databases and the theoretical background of it.

The main points are: SQL databases get inefficient as the data grows and if you need to split the data between instances (how do you join tables between two DB servers? Even if you can, performance is going to suffer).

But there are new problems: Data can be inconsistent for a while (keyword: MVCC).

OTOH, these databases don’t need locks and, as Amazon demonstrated, any kind of lock will eventually become a bottleneck:

Each node in a system should be able to make decisions purely based on local state. If you need to do something under high load with failures occurring and you need to reach agreement, you’re lost. If you’re concerned about scalability, any algorithm that forces you to run agreement will eventually become your bottleneck. Take that as a given.

Werner Vogels, Amazon CTO and Vice President

And since the servers can heal inconsistencies, broken connections don’t mean the end of the world. The acronym of the hour is BASE: Basically Available, Soft-state, Eventually consistent.

Interesting stuff, especially since every company owns a super-computer today. My own team has one with 64 cores, 64GB of RAM and 16TB of disk space sitting in 8 boxes spread under the desks of the developers. Not much but it only costs $8 000. And if we need more, we can simply buy another node. Unfortunately, it’s hard to leverage this power. I’ll come back to that in a later post.

One thing to take with you: If you can’t stand JavaScript but you need to write it, have a look at CoffeeScript.

Links:

• Slides
• “Why NoSQL is bad for startups” (not completely serious, of course)

The last few days, I’ve been chewing on a hard problem: Relations. I could to this:

class A {
}

class B {
    A parent 1:*;
}

Read: Instances of B have a field “parent” that references instances of A. There is a list in A with all instances of B where the field “parent” is that exact instance. In simple words, it’s a parent-child relation between A and B. For each B, there is always one A. Each A can have several B’s attached.

Or like this:

class A {
}

class B {
}

relation A children 1:* B parent;

The first solution avoids repetition but when looking at A, you will miss the fact that it has fields. It looks empty but the declaration in B adds a field. I don’t like it. While it might work in this case, how do you map N:M? Where do you declare a many-to-many relation? In A or in B? My guts say: Don’t do it.

The second solution is also flawed if less so. Here, both classes are empty. I have to look up the relations to see what else is going on. And I have to repeat information. I don’t like it.

Time to take a step back and sleep over it.

Fast forward over the weekend.

How about this:

class A {
}
parent of
class B {
}

One word: Intent. A construct in a programming language should clearly communicate intent. Which is one of the weak points in Java: It’s hard to express what you intend with Java. It’s not impossible but it takes years of experience to avoid the luring shortcuts.

So this new syntax clearly states what I want: A’s are parents of B’s. No repetition. No odd “1:*” which every reader has to decode. Which you can get wrong.

But what if A is parent of more classes? Simple:

class A { }
parent of {
    class B { } parent of class X { }
    class C { }
    ...
}

See? It nests. Let’s add tree-like structures to the mix:

tree class A { }
parent of {
    class B { } parent of class X { }
    class C { }
    ...
}

So A gets a parent-child relation with itself. One word says everything: Ownership. Cardinality. Field names.

My gut likes it. Listen to your guts. Especially when a simple solution tries to lure you into a shortcut.

Every application needs to load data from “somewhere” (user preferences, config settings, data to process) and after processing the data, it needs to save the results. Persistence is the most important feature of any software. Without it, the code would be useless.

Oddly, the most important area of the software isn’t a shiny skyscraper but a swamp: Muddy, boggy, suffocating.

Therefore, the next big thing in software development must make loading and saving data a bliss. Some features it needs to have:

• Transaction contexts to define which data needs to be rolled back in case of an error. Changes to the data model must be atomic by default. Even if I add 5,000 elements at once, either all or none of them must be added when an error happens.
• Persistence must be transparent. The language should support rules how to transform data for a specific storage (file, database) but these should be generic. I don’t want to poison my data model with thousands of annotations.
• All types must support persistence by default; not being able to be persisted must be the exception.
• Creating a binary file format must be as simple as defining the XML format.
• It must have optimizers (which run in the background like garbage collection runs today) that determine how much of the model graph needs to be loaded from a storage.

“The Next Big Thing” is a series of blog posts in which I dream about features of a programming language which will replace OO just like OO replaced structured programming.

ORA-01461: can bind a LONG value only for insert into a LONG column tips

then chances are you tried to insert more than 4096 characters at once.

Again the price for useless error messages goes to … Oracle!

To add insult to injury, I couldn’t find anything useful related to this error with Oracle’s own site search.

During your first run, your DB returns some values, you copy them into a String. When you run the tests again, the tests fail because all the IDs have changed. Bummer. Now what? Just verify that the correct number of results was returned?

Don’t. assertEquals (3, list.size()) doesn’t give you a clue where to search for the error when the assert fails.

The solution is to give each ID a name. The name always stays the same and when you look at the test results, you’ll immediately know that the ID has been “fixated” (unlike when you’d replace the number with another, for example). Here is the code:

import java.util.*;

/** Helper class to fixate changing object IDs */
public class IDFixer
{
    private Map<String, String> replacements = new HashMap<String, String> ();
    private Map<String, Exception> usedNames = new HashMap<String, Exception> ();
    
    /** Assign a name to the number which follows the string <code>id=</code> in the object's
     * <code>toString()</code> method. */
    public void register (Object o, String name)
    {
        if (o == null)
            return;
        
        String s = o.toString ();
        int pos = s.indexOf ("id=");
        if (pos != -1)
            pos = s.indexOf (',', pos);
        if (pos == -1)
            throw new RuntimeException ("Can't find 'id=' in " + s);
        String search = s.substring (0, pos + 1);
        
        pos = search.lastIndexOf ('=');
        String replace = search.substring (0, pos + 1) + name + ",";
        
        add (name, search, replace);
        
        registerSpecial (o, name);
    }

    /** Add a search&replace pattern. */
    protected void add (String name, String search, String replace)
    {
        if (usedNames.containsKey (replace))
            throw new RuntimeException ("Name "+name+" is already in use", usedNames.get (replace));
        
        //System.out.println ("+++ ["+search+"] -> ["+replace+"]");
        usedNames.put (replace, new Exception ("Name was registered here"));
        replacements.put (search, replace);
    }
    
    /** Allow for special mappings */
    protected void registerSpecial (Object o, String name)
    {
        // NOP
    }

    /** Turn a <code>Collection</code> into a <code>String</code>, replacing all IDs with names. */
    public String toString (Collection c)
    {
        StringBuilder buffer = new StringBuilder (10240);
        String delim = "";
        for (Object o: c)
        {
            buffer.append (delim);
            delim = "\n";
            buffer.append (o);
        }
        return toString (buffer);
    }
    
    /** Turn a <code>Map</code> into a <code>String</code>, replacing all IDs with names. */
    public String toString (Map m)
    {
        StringBuilder buffer = new StringBuilder (10240);
        String delim = "";
        for (Iterator iter=m.entrySet ().iterator (); iter.hasNext (); )
        {
            Map.Entry e = (Map.Entry)iter.next ();
            
            buffer.append (delim);
            delim = "\n";
            buffer.append (e.getKey ());
            buffer.append ('=');
            buffer.append (e.getValue ());
        }
        return toString (buffer);
    }
    
    /** Turn an <code>Object</code> to a <code>String</code>, replacing all IDs with names.
     * 
     *  <p>If the object is a <code>Collection</code> or a <code>Map</code>, the special collection handling methods will be called. */
    public String toString (Object o)
    {
        if (o instanceof Collection)
        {
            return toString ((Collection)o);
        }
        else if (o instanceof Map)
        {
            return toString ((Map)o);
        }
        
        if (o == null)
            return "null";

        String s = o.toString ();
        for (Map.Entry<String, String> entry: replacements.entrySet ())
        {
            s = s.replace (entry.getKey (), entry.getValue ());
        }
        
        return s;
    }

    public boolean knowsAbout (String key)
    {
        return replacements.containsKey (key);
    }
}

To use the IDFixer, call register(object, name) to assign a name to whatever follows the number after id=. The method will search for this substring in the result of calling object.toString().

If you call IDFixer.toString(object) for a single object or a collection, it will replace id=number with id=name everywhere. If you have references between objects, you can register additional pattern to be replaced by overriding registerSpecial().

To make it easier to compare lists and maps in assertEquals(), each item gets its own line.

class Knowledge {
    Set tags;
    Knowledge parent;
    List children;
    String name;
    String content;
}
class Tag { String name; }
class Relation { String name; Knowledge from, to;

A most simple model but it contains everything you can encounter in an application: Parent-child/tree structure, 1:N and N:M mappings. Now the idea is to have a way to build a UI and a DB mapping from this code. The idea of traits is to implement real properties in Java.

So instead of fields with primitive types, you have real objects to work with:

    assert "name" == Knowledge.name.getName()

These objects exist partially at the class and at the instance level. There is static information at the class level (the name of the property) and there is instance information (the current value). But it should be possible to add more information at both levels. So a DB mapper can add necessary translation information to the class level and a Hibernate mapper can build on top of that.

Oh, I hear you cry “annotations!” But annotations can suck, too. You can’t have smart defaults with annotations. For example, you can’t say “I want all fields called ‘timestamp’ to be mapped with an java.sql.Timestamp“. You have to add the annotation to each timestamp field. That violates DRY. It quickly gets really bad when you have to do this for several mappers: Database, Hibernate, JPA, the UI, Swing, SWT, GWT. Suddenly, each property would need 10+ annotations!

I think I’ve found a solution which should need relatively few lines of code with Groovy. I’ll let that stew for a couple of days in by subconscious and post another article when it’s well done

Wikis basically allow to publish mostly unstructured data. I say “mostly” because wikis give them some structure but not (too) much: You can organize it in pages and put it into context (by linking between the pages). Often, this is enough. But recently, MediaWiki has started to add support for structured data. See that infobox in the top left corner? But that’s just half-hearted.

XWiki takes this one step further. XWiki, as I understand it, is a framework of loosely coupled components which allow you to create a wiki. The default one is pretty good, too, so most of the time, you won’t even get into this. The cool part about XWiki is that you can define a class inside of it. Let me repeat: You can create a page (like a normal text page) that XWiki will treat as a class definition. So this class gets versioned, etc. You can then add attributes as you like.

After that, you can create instances of this class. The instances are again wiki pages. You can even use more than a single instance on a page, for example, you can have several tag instances and a single person instance. Instances are versioned, too. Of course they are, this is a wiki!

Now you need to display that data. You can use Velocity or Groovy for that. And guess what, the view is … a wiki page. So your designers can create a beautiful look for your the boring raw data. With versions and comments and everything. While some other guys are adding data to the system.

In “normal” wiki pages, you can reference these instances and render them using such a template. The same is true for editors. With a few lines of code, you can create overview pages: All instances of a class or all instances with or without a certain property or you can use Groovy to do whatever you can think of.

Now imagine this: You have an existing database where your marketing guys can, say, plan the next campaign. They can use all the wiki features to collect ideas, filter and verify them, to come up with a really good plan. Some of that data needs to go into a corporate database. In former wikis, you’d have to use an external application, switch back and forth, curse a lot when they get out of sync.

With XWiki, you can finally annotate data in your corporate database with a wiki page, with all the power of a wiki and you can even display the data set in the wiki and edit it there. Granted, the data set won’t be versioned unless your corporate database allows that but it’s simple to do the versioning in the data access layer (for example, you can save all modifications in a log database).

Suddenly, possibilities open up.

My solution is to wrap the JDBC layer. Either use a mock JDBC interface like the one provided by mockrunner. Or write your own. With Java 5 and varargs, this is simple:

public int update (Connection conn, String sql, Object... params) throws SQLException {
    PreparedStatement stmt = null;
    try {
        stmt = conn.prepareStatement (sql);
        int i = 1;
        for (Object p: params) {
            stmt.setObject(i++, p);
        }
        return stmt.executeUpdate ();
    }
    finally {
        stmt.close ();
    }
}

Put all these methods into an object that you can pass around. In your tests, override this object with a mockup that simply collects the SQL strings and parameter arrays. You can even mix and match: By examining the SQL string, you can decide whether you want to run a query against the database or handle it internally.

This way, you can collect any newly created objects but still load some background data from the database (until you get bored and make the query methods return predefined results).

In the asserts, just collect all the results into a big String and compare them all at once.

Notes: The code above is a bit more complicated if you allow null values. In this case, you need to tell JDBC what the column type is. My solution is a NullParameter class which contains the type. If the loop encounters this class, then it calls setNull() instead of setObject().