Using Eclipse to parse Java code

5. November, 2010

Eclipse comes with its own Java compiler. You can use this compiler to generate an AST from Java code by adding plugins/org.eclipse.jdt.core_<version>.jar to the classpath (details):

import java.io.*;
import java.util.LinkedHashSet;
import java.util.Map;
import java.util.Set;
import org.apache.log4j.Logger;
import org.eclipse.jdt.core.JavaCore;
import org.eclipse.jdt.core.dom.AST;
import org.eclipse.jdt.core.dom.ASTParser;
import org.eclipse.jdt.core.dom.CompilationUnit;

public class EclipseAstParser {

    public static final String VERSION_1_4 = "1.4";
    public static final String VERSION_1_5 = "1.5";
    public static final String VERSION_1_6 = "1.6";

    private static final Set<String> ALLOWED_TARGET_JDKS = new LinkedHashSet<String>();
    static {
        ALLOWED_TARGET_JDKS.add(VERSION_1_4);
        ALLOWED_TARGET_JDKS.add(VERSION_1_5);
        ALLOWED_TARGET_JDKS.add(VERSION_1_6);
    }

    private static final Logger log = Logger.getLogger(EclipseAstParser.class);
    public static boolean DEBUG;

    private String targetJdk = VERSION_1_4;
    private String encoding = "UTF-8";

    public void setTargetJdk( String targetJdk ) {
        if(!ALLOWED_TARGET_JDKS.contains(targetJdk))
            throw new IllegalArgumentException("Invalid value for targetJdk: [" + targetJdk + "]. Allowed are "+ALLOWED_TARGET_JDKS);

        this.targetJdk = targetJdk;
    }

    public void setEncoding( String encoding ) {
        if( encoding == null )
            throw new IllegalArgumentException("encoding is null");
        if( encoding.trim().length() == 0 )
            throw new IllegalArgumentException("encoding is empty");
        this.encoding = encoding;
    }

    public AstVisitor visitFile( File file ) throws IOException {
        if(!file.exists())
            new IllegalArgumentException("File "+file.getAbsolutePath()+" doesn't exist");

        String source = readFileToString( file, encoding );

        return visitString( source );
    }

    public static String readFileToString( File file, String encoding ) throws IOException {
        FileInputStream stream = new FileInputStream( file );
        String result = null;
        try {
            result = readInputStreamToString( stream, encoding );
        } finally {
            try {
                stream.close();
            } catch (IOException e) {
                // ignore
            }
        }
        return result;
    }

    public AstVisitor visit( InputStream stream, String encoding ) throws IOException {
        if( stream == null )
            throw new IllegalArgumentException("stream is null");
        if( encoding == null )
            throw new IllegalArgumentException("encoding is null");
        if( encoding.trim().length() == 0 )
            throw new IllegalArgumentException("encoding is empty");

        String source = readInputStreamToString( stream, encoding );

        return visitString( source );
    }

    public static String readInputStreamToString( InputStream stream, String encoding ) throws IOException {

        Reader r = new BufferedReader( new InputStreamReader( stream, encoding ), 16384 );
        StringBuilder result = new StringBuilder(16384);
        char[] buffer = new char[16384];

        int len;
        while((len = r.read( buffer, 0, buffer.length )) >= 0) {
            result.append(buffer, 0, len);
        }

        return result.toString();
    }

    public AstVisitor visitString( String source ) {
        ASTParser parser = ASTParser.newParser(AST.JLS3);

        @SuppressWarnings( "unchecked" )
        Map<String,String> options = JavaCore.getOptions();
        if(VERSION_1_5.equals(targetJdk))
            JavaCore.setComplianceOptions(JavaCore.VERSION_1_5, options);
        else if(VERSION_1_6.equals(targetJdk))
            JavaCore.setComplianceOptions(JavaCore.VERSION_1_6, options);
        else {
            if(!VERSION_1_4.equals(targetJdk)) {
                log.warn("Unknown targetJdk ["+targetJdk+"]. Using "+VERSION_1_4+" for parsing. Supported values are: "
                        + VERSION_1_4 + ", "
                        + VERSION_1_5 + ", "
                        + VERSION_1_6
                );
            }
            JavaCore.setComplianceOptions(JavaCore.VERSION_1_4, options);
        }
        parser.setCompilerOptions(options);

        parser.setResolveBindings(false);
        parser.setStatementsRecovery(false);
        parser.setBindingsRecovery(false);
        parser.setSource(source.toCharArray());
        parser.setIgnoreMethodBodies(false);

        CompilationUnit ast = (CompilationUnit) parser.createAST(null);

        // AstVisitor extends org.eclipse.jdt.core.dom.ASTVisitor
        AstVisitor visitor = new AstVisitor();
        visitor.DEBUG = DEBUG;
        ast.accept( visitor );

        return visitor;
    }
}

Eclipse Modeling Day

29. October, 2010
Figure 4-3: Data Modelling Today

Image via Wikipedia

Yesterday was Eclipse Modeling Day here in Zürich. There were a couple of talks from people who were using modeling for projects and talks from project leaders of modeling projects like EMF and CDO.

Eclipse Modeling Platform for Enterprise Modeling

If you’ve used the Eclipse modeling projects, you’ll know the pain: Where to start? Which project is worth to spend time with? Caveats? Things like that. It seems that’s not a superficial problem. Eclipse Modeling is a big, unsolved jigsaw puzzle. The new project “Eclipse Modeling Platform” sets out to close the major gaps in the next two years. On the road map are things like authentication, large scale models, comparing models, etc.

For me, the list of topics looked more like an MBA’s wish-list than something that will make life easier for software developers. Their standpoint was that the funders call the shots. My standpoint is that the we need tools to help us solve the basic issues like good editors for (meta-)models and a useful debugging framework for code generators.

Interesting projects: Sphinx and Papyrus.

User Story: Models as First Class Citizens in the Enterprise

Since many people didn’t seem to be aware that modeling can do, Robert Blust (UBS AG) showed an example. Like most banks, the UBS has tons of legacy code. And tons of rules. Rules like: Any application A must access data of another application B via a well-defined interface. Their product would collect a couple of gigabytes of data from old COBOL code and use that to determine dependencies (like the DB tables it uses).

The next step would be to define which tables belong to which application and the end result is an application which can show and track you rule violations. Or which can show a Java developer which tables he must care for if he has to replace an old COBOL application.

There was the question of authentication: Who can see what of the model? This is going to be some work to solve in a way that it’s still manageable. For example, a part of the model could be accessible via a roles-based model. A software developer should be able to see all the data which is relevant to his project. But what about bug reports? Should a reporter be allowed to see all of them? What about the security related ones?

If we go to fraud tracking, individual instances in the model might be visible to just a very few people. So authentication is something which needs to scale extremely well. It must be as coarse of fine-grained as needed, sometimes the whole range in a single model.

Eclipse Modeling Framework for Data Modeling

Ed Merks introduced EMF. Not much new here for me. I tried to talk to him during the coffee break but he was occupied by Benjamin Ginsberg. Benjamin was interested to get a first rough view on modeling. Apparently, I made some impression on him, because he came back later to see me.

Textual Modeling with Xtext

Sven Efftinge showed some magic using Xtext: He had his meta-model open in two editors, a textual and a graphical one. When he changed something in the graphical view, it would show up in the text editor after save. Nice. I couldn’t ask him how much code it took to implement this.

Under the hood, Xtext uses Guice for dependency injection.

Graphical Modeling with Graphiti

Michael Wenz from SAP showcased Graphiti. It’s a graphical editor framework for models like GMF but I guess there is a reason why SAP invented the wheel again. Several people at the event mentioned GMF unfavorably. I’m not sure why that is but I remember that EMF generated huge, non-reusable blobs of Java code when I asked it to generate an editor for my models. Ed wasn’t exactly excited when I asked to change that.

Graphiti itself looks really promising. The current 0.8 is pretty stable and has a graphical editor for JPA models which allow to define relations between instances via drag-and-drop. No more wondering which side is the “opposite.” It also creates all the fields, gives them the right types, etc. From the back of the room, it looked like a great time-saver.

User Story: The Usage of Models in an Embedded Automotive IDE

A guy from Bosch showed some real-life problems with modeling, especially with performance. They have huge models. Since they didn’t look at CDO, their editors had to load the whole model into RAM. Since Java can only allocate 1.5GB of RAM on a 32-bit hardware, they are at the limit of what they can handle (some projects have 400MB sources).

It’s a good example how an existing technology could have made their lives easier if they only knew about it. Or maybe EMF is too simple a solution (as in “A scientific theory should be as simple as possible, but no simpler.” — AlbertEinstein).

Modeling Repository with CDO

Eike Stepper was glad, though. It gave him a perfect opportunity to present CDO which solves exactly this problem. CDO connects a client to a repository server. Any change to the model in the client is sent to the server, applied and then confirmed for all connected clients. So things like scalability, remote access and multi-client support come for free.

Over the years, CDO has collected a big number of connection modes like replication and an off-line mode. They even solve problems like processing lists with millions of elements. Promising.

One problem Eike mentioned are the default EMF editors. Not reusable, not exactly user-friendly. Since that didn’t change for the last four years, it’s probably something the modeling community doesn’t deem “important.” For some people, XML is apparently good enough.

Project Dawn is trying to improve the situation.

User Story: Successful Use of MDSD in the Energy Industry

RWE (one of the largest European energy companies) showed how they used model driven software development (MDSD) to create software to automatically handle all the use cases of their energy network. He stressed the fact that without strict rules and their application, MDSD will fail just like any other methodology. Do I hear moaning out of the agile corner? 😉

Anyway. My impression was that these guys don’t come up with stup…great new ideas every five minutes and expect that they are already implemented. Delivering electricity isn’t something that you entrust just on anybody. These people are careful to start with. So I see it that there are in fact industries where strict rules work. Anyway, MDSD is another arrow in the quiver. Use it wisely.

User Story: Nord/LB – Modeling of Banking Applications with Xtext and GMF

The last speaker was from Nord/LB, a German bank. He dropped a couple of remarks about GMF. Seems like he hit some of the gaps mentioned earlier.

Their solution included several DSLs which allowed them to describe the model, the UIs, the page flow in the web browser, etc. Having seen Enthought Traits, I’m wondering which approach is better: Keep everything in a separate model (well, Xtext can track cross-references between models just like the Java editor can) or put all the information in a single place.

If you keep everything in a single place (i.e. every part of the model also knows what to tell the UI framework when it wants to generate the editors), that makes the description of the model quite big and confusing. The information you want to see is drowned in a dozen lines. If you keep the information separate, you must store that in your memory when you switch editors.

I guess the solution is to create an editor which can display that part of the information which you need right now.

The Reception

After the talks, I had a long talk with Eike Stepper and Ed Merks. One of my main issues is that models are pretty static. You can’t add properties and methods to it at runtime. At least not to Ecore-based models. Or maybe you could but you shouldn’t. Which seems odd to me. We have plug-in based architectures like Eclipse. We have XML which stands for Extensible Markup Language. Why does modeling have to start in the stone age again without support for model life-cycle, migration, evolution?

When I presented my use case to Eike, he said “never heard that before.” So either the modeling community is going for the long hanging fruit or my use cases are exceptional. All I’m asking is a model which I can attribute with additional information at runtime. Oh, yes, I could use EMF annotations for this. Which EMF default editor supports that? Hm. So if my users want to extend the EClass “Person” with a middle name? Something that HyperCard could do, hands down, in 1987?

 


Maven 3.0 is here

18. October, 2010

Maven 3.0 has been released.Maven Logo


Hidden JUnit features: @Rules

8. October, 2010

@Rules seem a better solution than @RunWith to do some special work before/after a test. The release notes mention a couple of ideas:

  • Notification on tests
  • Setting up or tearing down resources, especially when they are used in multiple test classes
  • Special checks performed after every test, possibly causing a test to fail.
  • Making information about the test available inside the test

Related articles:


Logging JDBC with slf4j

7. October, 2010

 

This chart represents several constituent comp...

Image via Wikipedia

 

If you use slf4j in your project and need to log JDBC events, then have a look at log4jdbc. It has all the usual features: Timing statements, showing all arguments, mapping SQL to connections. It can even log stack traces if you need to know where a JDBC call was made. Sample output:

16:32:56.162 [INFO ] jdbc.connection - 1. Connection opened  java.sql.DriverManager.getConnection(DriverManager.java:525)
16:32:56.168 [DEBUG] jdbc.connection - open connections:  1 (1)
16:32:56.169 [DEBUG] jdbc.audit - 1. Connection.new Connection returned   java.sql.DriverManager.getConnection(DriverManager.java:525)
16:32:56.284 [DEBUG] jdbc.audit - 1. PreparedStatement.new PreparedStatement returned   com.avanon.basic.db.XPreparedStatement.prepare(XPreparedStatement.java:84)
16:32:56.292 [DEBUG] jdbc.audit - 1. Connection.prepareStatement(SELECT *
FROM V_RCSA_40_4_CTRL_RATING) returned net.sf.log4jdbc.PreparedStatementSpy@423606  com.avanon.basic.db.XPreparedStatement.prepare(XPreparedStatement.java:84)
16:32:56.162 [INFO ] jdbc.connection - 1. Connection opened  java.sql.DriverManager.getConnection(DriverManager.java:525)
16:32:56.168 [DEBUG] jdbc.connection - open connections:  1 (1)
16:32:56.169 [DEBUG] jdbc.audit - 1. Connection.new Connection returned   java.sql.DriverManager.getConnection(DriverManager.java:525)
16:32:56.284 [DEBUG] jdbc.audit - 1. PreparedStatement.new PreparedStatement returned   com.avanon.basic.db.XPreparedStatement.prepare(XPreparedStatement.java:84)
16:32:56.292 [DEBUG] jdbc.audit - 1. Connection.prepareStatement(SELECT *FROM V_RCSA_40_4_CTRL_RATING) returned net.sf.log4jdbc.PreparedStatementSpy@423606  com.avanon.basic.db.XPreparedStatement.prepare(XPreparedStatement.java:84)
16:32:56.342 [DEBUG] jdbc.audit - 1. PreparedStatement.clearParameters() returned   com.avanon.basic.db.SetParameters.clear(SetParameters.java:144)
16:32:56.343 [DEBUG] jdbc.sqlonly -  com.avanon.basic.db.XPreparedStatement.executeQuery(XPreparedStatement.java:71)
1. SELECT * FROM V_RCSA_40_4_CTRL_RATING
16:32:56.350 [INFO ] jdbc.sqltiming - SELECT * FROM V_RCSA_40_4_CTRL_RATING  {executed in 7 msec}
16:32:56.356 [DEBUG] jdbc.audit - 1. PreparedStatement.executeQuery() returned net.sf.log4jdbc.ResultSetSpy@2c5444  com.avanon.basic.db.XPreparedStatement.executeQuery(XPreparedStatement.java:71)
16:32:56.412 [DEBUG] jdbc.audit - 1. PreparedStatement.close() returned   com.avanon.basic.db.DBUtil.close(DBUtil.java:114)
16:32:56.418 [INFO ] jdbc.connection - 1. Connection closed  com.avanon.basic.db.DBUtil.close(DBUtil.java:129)
16:32:56.418 [DEBUG] jdbc.connection - open connections:  none

The strange “1.” is the connection number.

Alternatively, check out jdbcdslog.


Extremely complex queries in BIRT reports

30. September, 2010

Sometimes, SQL just isn’t enough. What do you do when you just can’t get the SQL to run in under an hour while a little piece of Java code does the same query in a few seconds? Put the result into a report table and then run the report against that table.


Google Relaunches Instantiations Developer Tools

29. September, 2010
Google Web Toolkit

Image via Wikipedia

From the website:

In early August, Google acquired Instantiations, a company known for its focus on Eclipse Java developer tools, including GWT Designer. We’re happy to announce today that we’re relaunching the following former Instantiations products under the Google name and making them available to all developers at no charge:

  • GWT Designer
    Powerful Eclipse-based development tools that enable Java developers to quickly create Ajax user interfaces using Google Web Toolkit (GWT)
  • CodePro AnalytiX
    Comprehensive automated software code quality and security analysis tools to improve software quality, reliability, and maintainability
  • WindowBuilder Pro
    Java graphical user interface designer for Swing, SWT, GWT, RCP, and XWT UI frameworks
  • WindowTester Pro
    Test GUI interactions within Java client rich applications for the SWT and Swing UI frameworks

I played a bit with CodePro. The tools look promising even through there were some glitches, namely:

  1. The JUnit editor looks cool but the table with the current unit results often hangs.
  2. It was more complicated than I liked to generate test cases
  3. I couldn’t get the code coverage tool to work
  4. The dependency works but didn’t play with it long enough to say for sure how useful it is
  5. The code analysis shows a lot of numbers but the workflow is clumsy. For example, it says that something has a cyclomatic complexity of 16 but I couldn’t find out what and where.

When inheritance is not enough

16. September, 2010

Ever had the problem that inheritance just wasn’t enough? For example in this case:

package core;
public class Main {
    public static void main(String[] args) {
        sayHello();
    }
    private static void sayHello() {
        System.out.println("Hello World!");
    }
}

Say you need to override sayHello(). What are your options?

You could edit the source. But if you only have the bytecode?

You could decompile the bytecode. But what if you need the same change in several places?

Eventually, you’ll have to use copy more or less code around because Java just doesn’t offer enough ways to combine code. The solution? Object Teams. See this article for a solution of the problem above.

What is Object Teams?

In a nutshell, a team is a group of players which have roles. Usually, the roles are distributed by way of the strengths of your players. Slow and sturdy ones will be defense. Quick and agile players will attack. The calm ones will do tactics.

OT/J (the Java implementation of Object Teams) does just that: You define teams and roles and then apply those to existing Java code. Let’s take the Observer pattern (you know that design patterns are workarounds for flaws in the language’s design, yes?).

The problem with this pattern is that you can’t apply it retroactively. You must modify the involved classes to support the pattern even if they couldn’t care less. That leads to a lot of problems. Classes which don’t need it must still allocate resources for it. It clutters the API. You can’t add it to classes which don’t support it. If a class only allows you to observe some behavior, you can’t change that.

OT/J fixes all that. The OTExample Observer shows three classes which could make use of the Observer pattern but don’t. Instead, the pattern is retrofitted in all the places where it’s necessary. This allows for many optimizations. For example, you can create a single store for all your observers (instead of one per instance). The API isn’t cluttered. You can observe almost any change.


Java Project Lambda

10. September, 2010
Java (programming language)

Image via Wikipedia

It seems that someone came to reason and proposed an implementation for closures in Java that would not only work but also be useful without adding too much complexity to the language: Project Lambda.


Java development through Excel

7. September, 2010

When developing software, one pain is always the same: Getting valuable input from the customer. One solution is to use a tool that your customer probably understands: Excel.

“Oh, great, CSV files,” you think. No. I mean Excel work sheets, with colors and formatting and formulas. Let me explain two approaches to save you a lot of time and effort, pain and tears.

Jakarta POI Logo

Image via Wikipedia

Apache POI

Apache POI is the Swiss Army Knife for Java developers who have to deal with Excel. It basically allows you to access the content of an Excel document in a humane way: By book, sheet and then indexed by row and column. It doesn’t matter if the file is in the old binary format (*.xls) or the new XML one (*.xlsx).

Especially the XML format is suited perfectly to allow your customers to send you master data. Just write a small tool that reads the Excel file directly and dumps it into your database. No CSV conversion! You can use colors to mark rows/columns to ignore. You can define styles to ignore (like “Section” or “Heading”). That allows your customer to keep the table readable.

Formula Compiler

But the data in the spreadsheets is only half of the value. The guys at Abacus Research AG created a new project: Abacus Formula Compiler (AFC). This gem gives you access to the formulas in the cells. POI does that, too, but you will just get a string. AFC will convert the formulas into Java Code. That means your customer can not only send you master data but also how to process it.

They can send you a spreadsheet with an example what they are talking about. They can use the Excel sheets they have been using to drive their business for the last five years. If something changes, they can send you an updated version, you run AFC on that and get a new set of classes which replace the old ones and which do exactly what your customer wants.