Spot the real Java class name

This is hilarious (if you’re a programmer). Some folks trained a Markov chain on the class names in Spring and then made a game out of it – three Java class names are presented, only one is real. I didn’t do so hot – 4/11.

Via Jeff Dean on Google+.

Go gotcha #0: Why taking the address of an iterated variable is wrong

Go mascot – by Renée French under Creative Commons Attribution-Share Alike 3.0 Unported from http://en.wikipedia.org/wiki/File:Golang.png

Disclaimer: Go is open source and developed by many Google employees. I work for Google, but the opinions expressed here are my own and do not necessarily represent that of Google.

Go is my new favorite programming language. It’s compact, garbage collected, terse, and very easy to read. There are some things that trip me up even now after I’ve been using it for awhile. Today I’m going to discuss the range construct and how it has a surprising feature that might violate your assumptions.

Range

First, the range keyword is a way to iterate through the various builtin data structures in Go. For instance,

a := map[string]int {
    "hello": 1,
    "world": 2,
}
// 2 element range gets key and value
for key, value := range a {
    fmt.Printf("key %s value %d\n", key, value)
}
// 1 element is just the key
for key := range a {
    fmt.Printf("key %s\n", key)
}

// Works for slices (think of them as vectors/lists) too
b := []string {"hello", "world"}
// 2 element range gets the index as well as the entry
for i, s := range b {
    fmt.Printf("entry %d: %s\n", i, s)
}
// 1 element gets just the index (notice the pattern?)
for i := range b {
    fmt.Printf("entry %d\n", i)
}

This outputs

key hello value 1
key world value 2
key hello
key world
entry 0: hello
entry 1: world
entry 0
entry 1

Try this code in the Go Playground

Solution search – pointers

Imagine the case where we have a struct as follows

type Solution struct {
    Name string
    Cost int
    Complete bool
}

Say that we’re doing some sort of optimization where we’re looking for the minimum cost solution that meets some criteria; for simplicity’s sake, I’ve put that as the ‘complete’ bool. It’s possible that no such Solution matches, in which case we return a nil solution.

A reasonable implementation would be as follows

func FindBestSolution(solutions []Solution) *Solution {
    var best *Solution
    for _, solution := range solutions {
        if solution.Complete {
            if best == nil || solution.Cost < best.Cost {
                best = &solution
            }
        }
    }
    return best
}

Do you see the bug? Don’t worry if you don’t – I’ve made this mistake a few times now.

Let’s add some tests to find the problem. This is an example of a table driven test, where the test cases are given as a slice of struct literals. This makes it very easy to add new test cases.

func TestFindBestSolution(t *testing.T) {
    tests := []struct {
        name      string
        solutions []Solution
        want      *Solution
    }{
        {
            name:      "Nil list",
            solutions: nil,
            want:      nil,
        },
        {
            name: "No complete solution",
            solutions: []Solution{
                {
                    Name:     "Foo",
                    Cost:     25,
                    Complete: false,
                },
            },
            want: nil,
        },
        {
            name: "Sole solution",
            solutions: []Solution{
                {
                    Name:     "Bar",
                    Cost:     12,
                    Complete: true,
                },
            },
            want: &Solution{
                Name:     "Bar",
                Cost:     12,
                Complete: true,
            },
        },
        {
            name: "Multiple complete solution",
            solutions: []Solution{
                {
                    Name:     "Foo",
                    Cost:     25,
                    Complete: false,
                },
                {
                    Name:     "Bar",
                    Cost:     12,
                    Complete: true,
                },
                {
                    Name:     "Baz",
                    Cost:     25,
                    Complete: true,
                },
            },
            want: &Solution{
                Name:     "Bar",
                Cost:     12,
                Complete: true,
            },
        },
    }
    for _, test := range tests {
        got := FindBestSolution(test.solutions)
        if got == nil && test.want != nil {
            t.Errorf("FindBestSolution(%q): got nil wanted %v", test.name, *test.want)
        } else if got != nil && test.want == nil {
            t.Errorf("FindBestSolution(%q): got %v wanted nil", test.name, *got)
        } else if got == nil && test.want == nil {
            // This is OK
        } else if *got != *test.want {
            t.Errorf("FindBestSolution(%q): got %v wanted %v", test.name, *got, *test.want)
        }
    }
}

If you run the tests you’ll find that the last test fails:

--- FAIL: TestFindBestSolution (0.00 seconds)
    prog.go:82: FindBestSolution("One complete solution"): got {Baz 25 true} wanted {Bar 12 true}
FAIL
 [process exited with non-zero status]

This is strange – it works fine in the single element case, but not with multiple values. Let’s try adding a case where the correct value is last in the list.

    {
        name: "Multiple - correct solution is last",
        solutions: []Solution{
            {
                Name:     "Baz",
                Cost:     25,
                Complete: true,
            },
            {
                Name:     "Bar",
                Cost:     12,
                Complete: true,
            },
        },
        want: &Solution{
            Name:     "Bar",
            Cost:     12,
            Complete: true,
        },
    },

Sure enough, this test passes. So somehow if the element is last the algorithm works. What’s going on?

From the go-wiki entry on Range:

When iterating over a slice or map of values, one might try this:

items := make([]map[int]int, 10)
for _, item := range items {
        item = make(map[int]int, 1) // Oops! item is only a copy of the slice element.
        item[1] = 2                 // This 'item' will be lost on the next iteration.
}

The make and assignment look like they might work, but the value property of range (stored here as item) is a copy of the value from items, not a pointer to the value in items.

This is exactly what’s happening in this case. The solution variable is getting a copy of each entry, not the entry itself. Thus when you take the address of the entry, you end up with a pointer pointing at the LAST element in the slice (since the iteration stops at that point). To illustrate:

package main

import "fmt"

func main() {
    strings := []string{"some","value"}
    for i, s := range strings {
        fmt.Printf("Element %d: %s Pointer %v\n", i, s, &s)
    }
}

Element 0: some Pointer 0x10500168
Element 1: value Pointer 0x10500168

Note that the same pointer is used in both cases. This explains why the Solution pointer ended up pointing at the last element of the slice.
Playground

So how do we work around this problem? The key is to introduce a new variable whose address it’s safe to take; its contents won’t change out from underneath you.

Broken:

if solution.Complete {
    if best == nil || solution.Cost < best.Cost {
        best = &solution
    }
}

Fixed:

if solution.Complete {
    if best == nil || solution.Cost < best.Cost {
        tmp := solution
        best = &tmp
    }
}

With this patch the tests pass:

PASS

Program exited.

Alternative design

A great feature of Go is that you can return multiple values from a single function. Here’s an alternative implementation that doesn’t suffer from the previous problem.

func FindBestSolution(solutions []Solution) (Solution, bool) {
    var best Solution
    found := false
    for _, solution := range solutions {
        if solution.Complete {
            if !found || solution.Cost < best.Cost {
                best = solution
                found = true
            }
        }
    }
    return best, found
}

Since best is copying the VALUE of the solution variable, this works correctly. You can play with this example and see how the tests change in the Playground.

This illustrates one other nice feature of Go – all types have a ‘zero’ value that is legal to use. For strings this is the empty string, for pointers it’s nil, for ints it’s 0, for structs all of types are set to zero values. The line var best Solution implicitly sets best to be the zero solution. If I wanted to I could get rid of the found bool altogether and just compare the returned solution with another zero valued Solution.

Conclusion

I introduced some basic features of Go, including maps, slices, range, structs, and functions. I provided links to the amazingly useful Go playground which lets you easily test out code, format it, and share it with others.

I showed two implementations of a function that searches through a slice of struct values, searching for a solution that meets some criteria.

The first example using pointers led to a subtle bug that’s hard to find and solve unless you know how range works. I showed how to write unit tests that exercise the function and helped flush out the bug. I also explained what the bug was and how to work around it.

Finally I showed a version of the same function that uses Go’s multiple return types to return a found boolean rather than using a nil pointer to signify that the value wasn’t found.

Glazed Lists – an essential Java library for lists and tables

Swing is the built in toolkit for creating user interfaces for Java programs. While these types of standalone desktop applications are becoming less prevalent, perhaps due to increasing functionality of webapps, there are still some industries which are highly reliant on them. If you find yourself creating a Java desktop application, you will probably have to learn Swing, and you will also probably have to learn to display information to the user in list or table form. In standard Java Swing applications, it is difficult, or at least annoying, to do the following three tasks:

1. Displaying domain specific models
2. Filtering
3. Sorting

Glazed Lists is an open source project that makes all three of these tasks trivial. Its primary author, Jesse Wilson, is a current Google employee. Let’s examine each of these aspects in turn.

Provides a simplified API for representing objects within a JTable

Swing uses the Model View Controller paradigm throughout. Thus the table or list merely presents a view for an underlying model data structure. Part of your job in displaying data in a Swing table is to define the TableModel implementation which provides the data for the JTable to display.

Swing provides an AbstractTableModel that does most of the work for you, requiring you only to implement the following methods:

public int getRowCount();
public int getColumnCount();
public Object getValueAt(int row, int column);

Here’s a simple domain model object we might want to visualize in a table:

public class Person {
    int age;
    String name;

    public Person(String name, int age) {
        this.name = name;
        this.age = age;
    }

    public int getAge() { return age; }

    public String getName() { return name; }
}

The logical way of doing that would be two have two columns, one for the age, one for the name. Let’s make a table model for this case:

public class PersonTableModel extends AbstractTableModel {
    private static final String[] columns = {"Name", "Age"};
    private final List people;

    public PersonTableModel(List people) {
        // Make a defensive copy
        this.people = new ArrayList(people);
    }

    public int getRowCount() {
        return people.size();
    }
    public int getColumnCount() {
        return columns.length;
    }
    public Object getValueAt(int row, int column) {
        Person p = people.get(row);
        if (column == 0) {
            return p.getName();
        } else {
            return p.getAge();
        }
    }
}

This certainly works, but it requires a fair bit of boilerplate. Furthermore, the code above does not provide any way of modifying the list of people after it is copied by the TableModel.

Glazed Lists simplifies your life by treating the table not as an arbitrary two dimensional grid, but instead as a collection of rows, where the rows are kept in sync with changes to the domain models that they represent. All you have to do is define how a row is laid out, and Glazed Lists takes care of the rest.

The interface you need to use in order to define how the table looks and which aspects of your model objects are exposed is called [TableFormat][].

The interface is as follows:

• int getColumnCount() – The number of columns to display.
• String getColumnName(int column) – Gets the title of the specified column.
• Object getColumnValue(E baseObject, int column) – Gets the value of the specified field for the specified object.

This should remind you of the TableModel interface presented previously, but note how the getColumnValue method is different – rather than getting a row and column, and forcing you to look up the object corresponding to that row, you are provided the object directly.

Here is a TableFormat which allows Person objects to be easily visible in a JTable:

public class PersonTableFormat implements TableFormat {

    String[] columnNames = {"Name", "Age"};
    private static final int NAME_INDEX = 0;
    private static final int AGE_INDEX = 1;

    public int getColumnCount() { return columnNames.length; }

    public String getColumnName(int column) { return columnNames[i]; }

    public Object getColumnValue(Person baseObject, int column) {
        switch (column) {
            case NAME_INDEX:
                return baseObject.getName();
            case AGE_INDEX:
                return baseObject.getAge();
            default:
                throw new IllegalArgumentException("Expected column 0 or 1, got " + column);
        }
    }
}

While this isn’t too hard to write, it’s still a lot of boilerplate (and not significantly different from the previous example). Glazed Lists makes it even easier than this. The entire class definition above can be replaced with three lines:

TableFormat personTableFormat = GlazedLists.tableFormat(Person.class,
    // Names of the properties to fetch
    new String[] {"name","age"},
    // Names for the columns
    new String[] {"Name", "Age"});

What’s this doing? And how can it do all that I had previously in one line of code? Well, it requires and takes advantage of JavaBeans naming convention. The static function uses reflection to find the methods mapping to properties named “name” and “age”. In this case, it looks for two methods, getName() and getAge(), both of which it finds. (If I didn’t name my methods appropriately, I would get a runtime exception). The second array defines the strings that should be used to identify the corresponding entry in the properties array. In other words, element 0 in the names column is used to identify the property name at index 0.

This TableFormat class alone is insufficient to display data in a table. To do that, you need a class which fulfills the TableModel interface I described previously. Fortunately, Glazed Lists makes this easy.

The fundamental building block of Glazed Lists is the EventList class. It is similar to the ArrayList class in Java, except that it has support for observers. If you’re not familiar with the Observer/Observable design pattern, it allows objects (observers) to register themselves and receive notifications whenever a different object (the observable) is changed. For instance, when a new item is added to the EventList, the UI element representing it on screen automatically refreshes itself.

The EventTableModel class fulfills the TableModel interface, making use of the EventList and TableFormat we described earlier. The EventList is the data provider, and the TableFormat determines how to extract the data from the EventList and display it in the table.

EventList people = new BasicEventList();
// Add all the elements
for (Person p : getPeople()) {
    personList.add(p);
}
TableFormat personTableFormat = GlazedLists.tableFormat(Person.class,
    // Names of the properties to fetch
    new String[] {"name","age"},
    // Names for the columns
    new String[] {"Name", "Age"});
EventTableModel tableModel = new EventTableModel(people, personTableFormat);
JTable table = new JTable(tableModel);
// Any modifications to the ‘people’ list is automatically reflected in the table

Provides a simplified means of filtering a table or list

Perhaps one of the most important features of any interactive table is the ability to filter out extraneous information. Glazed Lists makes this possible by chaining together EventList transformations; these transformations provide a different view of the underlying data. When the original model is modified, the filtered views automatically pick up the changes and update accordingly.

Say we want to provide the ability to filter the list based on people’s names. We will add a listener to a text field which listens for changes (new letters typed or deleted), and filters the list in real time. Once we have an EventList of some sort, it is easy to create a new “view” of that same list, filtering out entries you don’t want to see. You do this by wrapping the list in a FilterList, and then assigning some sort of filter criterion. Let’s start simple with a filtered list which only shows those users whose names start with the letter ‘A’.

EventList personList = new BasicEventList();
personList.add(new Person("Anthony Hopkins", 74));
personList.add(new Person("Barack Obama", 50));
personList.add(new Person("American McGee", 39));

Matcher personFilter = new Matcher() {
    public boolean matches(Person p) {
        return p.getName().startsWith("A");
    }
};
// Create a filtered list
FilterList filteredList = new FilterList(personList, personFilter);
// Displaying the people in a list as opposed to a table; could also create EventTableModel
// as in the last example.
EventListModel filteredListModel = new EventListModel(personList)
JList list = new JList(filteredListModel);
// At this point, shows Anthony Hopkins and American McGee

The filter I’ve defined above is static – once it’s instantiated, its filter condition never changes. Glazed Lists supports dynamic filters as well, through the MatcherEditor interface. We will see how to use a MatcherEditor instance for a text field, but first we need to tell Glazed Lists which strings to use when filtering for a given object. We do this with the TextFilterator interface.

Picture illustrating a FilterList which accepts only those people whose name starts with 'A'

 

public class PersonTextFilterator imlements TextFilterator {
    // Slightly strange interface, but done for efficiency reasons
    public getFilterStrings(List baseList, Person element) {
        baseList.add(element.getName());
        // Allow users to filter by age as well
        baseList.add(String.valueOf(element.getAge()));
    }
}

The MatcherEditor class to use in our case is TextComponentMatcherEditor. We provide it with the text field that it will use as the filter source, as well as an instance of the PersonTextFilterator class we just defined.

EventList personList = new BasicEventList();
personList.add(new Person("Anthony Hopkins", 74));
personList.add(new Person("Barack Obama", 50));
personList.add(new Person("American McGee", 39));

JTextField filterTextField = new JTextField();
// Add the text field to the UI - add to a JPanel

// Hook the text field up to a filter list
MatcherEditor filter = new TextComponentMatcherEditor(filterTextField, new PersonTextFilterator());

// Create a filtered list
FilterList filteredList = new FilterList(personList, filter);
EventListModel filteredListModel = new EventListModel(filteredList)
JList list = new JList(filteredListModel);
// List automatically updates in response to typing in the text field

Each transformed EventList is itself an EventList, meaning it can also be used as the basis of an EventListModel or EventTableModel. This chaining capability is extremely powerful.

Provides sorting capabilities

Finally, Glazed Lists makes it extremely easy to implement rich sorting capabilities in your tables or lists.

As we saw in the last example, it is possible to wrap a given EventList to provide a different view. In this case, we will have a sorted view of the data, which automatically updates whenever the underlying data changes.

To create the SortedList, you need to make your domain object implement Comparable, or create a Comparator. For instance,

public class PersonNameComparator implements Comparator {
    public int compare(Person p1, Person p2) {
        return p1.getName().compareTo(p2.getName());
    }
}

EventList personList = new BasicEventList();
personList.add(new Person("Anthony Hopkins", 74));
personList.add(new Person("Barack Obama", 50));
personList.add(new Person("American McGee", 39));

Comparator nameComparator = new PersonNameComparator();
// Create a sorted list decorator
SortedList sortedList = new SortedList(personList, nameComparator);
EventListModel sortedListModel = new EventListModel(sortedList)
JList list = new JList(filteredListModel);

A SortedList, wrapping a standard EventList

While the above example works for JLists, it’s nice to be able to sort a JTable as well. This is not too hard, either, as long as you have set up a TableFormat instance as described in the first section of this post. In essence, the TableFormat defines the type of each column, which is then used to sort the table whenever the corresponding column header is clicked. This behavior is defined in the TableComparatorChooser class, which exposes a static method to perform the installation on the target JTable. Here’s an example:

Comparator nameComparator = new PersonNameComparator();
// Create a sorted list decorator
SortedList sortedList = new SortedList(personList, nameComparator);
EventTableModel peopleTableModel = new EventTableModel(sortedList, new PersonTableFormat());
JTable peopleTable = new JTable(peopleTableModel);

// Use MULTIPLE_COLUMN_MOUSE to allow sorting by multiple columns, or SINGLE_COLUMN
// to sort by just a single column
TableComparatorChooser tableSorter = TableComparatorChooser.install(
    peopleTable, sortedList, TableComparatorChooser.MULTIPLE_COLUMN_MOUSE);

// At this point, clicking on the table headers will sort by this column

As the more detailed Glazed Lists tutorial warns,

By default, TableComparatorChooser sorts by casting column values to Comparable. If your column’s values are not Comparable, you’ll have to manually remove the default Comparator using TableComparatorChooser.getComparatorsForColumn(column).clear().

As long as your columns are represented by Comparable classes such as Number or String, you shouldn’t have to worry about this caveat.

Conclusion

Glazed Lists is one of the best Java Swing libraries I’ve ever used. It simplifies life for the programmer as well as the end user of the software project, since tables that allow sorting and filtering are far more useful than those which do not. If you do any sort of Swing programming, you owe it to yourself to try this library out. You can find much more information, including the aforementioned tutorial, on the Glazed List website.

 

Chaining together multiple list transformations makes it easy to create powerful programs

Python Gotcha: Word boundaries in regular expressions

TL;DR

Be careful trying to match word boundaries in Python using regular expressions.  You have to be sure to either escape the match sequence or use raw strings.

Word boundaries

Word boundaries are a great way of performing regular expression searches for whole words while avoiding partial matches.  For instance, a search for the regular expression “the” would match both the word “the” and the start of the word “thesaurus”.

>>> import re
>>> re.match("the", "the")
# matches
>>> re.match("the", "thesaurus")
# matches 

In some cases, you might want to match just the word “the” by itself, but not when it’s embedded within another word.

The way to match a word boundary is with ‘\b’, as described in the Python documentation.  I wasted a few minutes wrestling with trying to get this to work.

>>> re.match("\bthe\b", "the")
# no match

It turns out that \b is also used as the backspace control sequence.  Thus in order for the regular expression engine to interpret the word boundary correctly, you need to escape the sequence:

>>> re.match("\\bthe\\b", "the")
# match

You can also use raw string literals and avoid the double backslashes:

>>> re.match(r"\bthe\b", "the")
# match

In case you haven’t seen the raw string prefix before, here is the relevant documentation:

String literals may optionally be prefixed with a letter ‘r’ or ‘R’; such strings are called raw strings and use different rules for interpreting backslash escape sequences.

Conclusion

Make sure you are familiar with the escape sequences for strings in Python, especially if you are dealing with regular expressions whose special characters might conflict.  The Java documentation for regular expressions makes this warning a bit more explicit than Python’s:

The string literal “\b”, for example, matches a single backspace character when interpreted as a regular expression, while “\\b” matches a word boundary.

Hopefully this blog post will help others running into this issue.

Mule 3 Deployment Gotchas / Workarounds

Mule is an open source enterprise service bus written in Java. I’ve worked with Mule 2.2 quite a bit but only recently have started to work with Mule 3. This post details some of the pains involved with the transition, none of which are well documented or hinted at in the Migration guide.

Gotchas/Workarounds

Mule IDE specific

The Mule IDE is really a misnomer – it’s not a standalone product, but instead an Eclipse plugin. See the installation guide for more information.

XML validation warnings

By default, Eclipse 3.5 will flag all sorts of spurious errors in your XML configuration file. See the blog post for more details, but here’s the short version on how to solve it:

General

These issues exist whether you use the IDE to deploy the app or deploy the app via the command line.

Failure to launch / Timeouts

Mule is configured via XML. You must declare the namespaces and schema locations in order to make use of the built-in Mule constructs. For instance, here’s a snippet of one of my Mule configurations:

<mule xmlns="http://www.mulesoft.org/schema/mule/core"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      xmlns:spring="http://www.springframework.org/schema/beans"
      xmlns:vm="http://www.mulesoft.org/schema/mule/vm"
      xmlns:script="http://www.mulesoft.org/schema/mule/scripting"
      xmlns:http="http://www.mulesoft.org/schema/mule/http"
      xmlns:cxf="http://www.mulesoft.org/schema/mule/cxf"
      xmlns:xm="http://www.mulesoft.org/schema/mule/xml"
      xmlns:pattern="http://www.mulesoft.org/schema/mule/pattern"
      xmlns:servlet="http://www.mulesoft.org/schema/mule/servlet"
      xmlns:jetty="http://www.mulesoft.org/schema/mule/jetty"
      xmlns:test="http://www.mulesoft.org/schema/mule/test"
      xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
        http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/3.1/mule.xsd
        http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/3.1/mule-http.xsd
        http://www.mulesoft.org/schema/mule/cxf http://www.mulesoft.org/schema/mule/cxf/3.1/mule-cxf.xsd
        http://www.mulesoft.org/schema/mule/scripting http://www.mulesoft.org/schema/mule/scripting/3.1/mule-scripting.xsd
        http://www.mulesoft.org/schema/mule/pattern http://www.mulesoft.org/schema/mule/pattern/3.1/mule-pattern.xsd
        http://www.mulesoft.org/schema/mule/xml http://www.mulesoft.org/schema/mule/xml/3.1/mule-xml.xsd
        http://www.mulesoft.org/schema/mule/vm http://www.mulesoft.org/schema/mule/vm/3.1/mule-vm.xsd
        http://www.mulesoft.org/schema/mule/servlet http://www.mulesoft.org/schema/mule/servlet/3.1/mule-servlet.xsd
        http://www.mulesoft.org/schema/mule/test http://www.mulesoft.org/schema/mule/test/3.1/mule-test.xsd
        http://www.mulesoft.org/schema/mule/jetty http://www.mulesoft.org/schema/mule/jetty/3.1/mule-jetty.xsd"
        >

Make absolutely sure that the version of the xsd that you include matches the major version of mule that you’re using! If you accidentally place a 3.0 instead of a 3.1 in any of those entries, your app will mysteriously fail to launch and you’ll get a stack trace like the following:

INFO  2011-06-09 17:21:20,015 [main] org.mule.MuleServer: Mule Server initializing...
INFO  2011-06-09 17:21:20,298 [main] org.mule.lifecycle.AbstractLifecycleManager: Initialising RegistryBroker
INFO  2011-06-09 17:21:20,355 [main] org.mule.config.spring.MuleApplicationContext: Refreshing org.mule.config.spring.MuleApplicationContext@19bb5c09: startup date [Thu Jun 09 17:21:20 EDT 2011]; root of context hierarchy
WARN  2011-06-09 17:22:36,265 [main] org.springframework.beans.factory.xml.XmlBeanDefinitionReader: Ignored XML validation warning
java.net.ConnectException: Operation timed out
    at org.apache.xerces.util.ErrorHandlerWrapper.createSAXParseException(Unknown Source)
    at org.apache.xerces.util.ErrorHandlerWrapper.warning(Unknown Source)
    at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
    at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
    at org.apache.xerces.impl.xs.traversers.XSDHandler.reportSchemaWarning(Unknown Source)
    at org.apache.xerces.impl.xs.traversers.XSDHandler.getSchemaDocument1(Unknown Source)
    at org.apache.xerces.impl.xs.traversers.XSDHandler.getSchemaDocument(Unknown Source)
    at org.apache.xerces.impl.xs.traversers.XSDHandler.parseSchema(Unknown Source)
    at org.apache.xerces.impl.xs.XMLSchemaLoader.loadSchema(Unknown Source)

Deploying via command line

While it’s nice to be able to use an IDE to develop Mule applications, I prefer to deploy from the command line. This allows me to script the launch of the applications. Furthermore, this approach works in a headless (screenless) remote server, whereas the IDE approach will not. The basic way to deploy an app has changed from Mule 2.2 to Mule 3. It used to be that you would call mule -config /path/to/your/config.xml. Now you move your application to the $MULE_HOME/apps folder and run mule, which in turn will deploy all the apps in the apps folder. This can be very handy, especially when coupled with the Hot Deployment features of Mule; you no longer need to have one terminal per mule app you’re running. From the article, “Mule 3: A New Deployment Model”, here are the ostensible steps you must take to deploy your application in this manner:

• Create a directory under: $MULE_HOME/apps/foo
• Jar custom classes (if any), and put them under: $MULE_HOME/apps/foo/lib
• Put the master Mule config file at: $MULE_HOME/apps/foo/mule-config.xml (note that it has to be named: mule-config.xml
• Start your app with: mule -app foo

While these instructions are correct, there are a lot of gotchas involved. Let me detail them below.

Relative paths

There is often a need to make reference to resources within your configuration file. For instance, you might need to configure an embedded Jetty webserver and tell Jetty where its configuration file is located. When you do this, it’s crucial that you prepend relative paths in the XML configuration file with ${app.home}.

The reason for this is that the current working directory in which you launch the mule process becomes the current working directory for all of your application configuration files. So if you have mule-config.xml in the root of your folder, and conf/jetty.xml in that same folder, then your reference to the jetty.xml should be ${app.home}/conf/jetty.xml. Otherwise, if you just use conf/jetty.xml and launch mule from a folder that’s not the same as the root folder of your application, all of your paths will break.

Property files / Resources

As the step #2 above says, you must jar up all of your compiled classes and include them in the lib folder of your project. If you don’t do this, you’ll get an exception when your component / custom classes are attempted to be instantiated.

What should be emphasized is that all resources that you reference from within your code must end up in the jar as well. By default, that won’t happen. You can use something like the solution presented in Ant Build: copy properties file to jar file to get this to happen.

Unintentional Application Deletion

When you deploy an app by copying a zip or folder into the apps directory and then running mule, Mule will launch it and then create a text file called ‘$APP_NAME-anchor.text’. If you delete this file, Mule will “undeploy this app in a clean way”. What isn’t noted by this is that it will delete the corresponding zip/folder. So be careful not to accidentally delete your whole project. (Not that I did that or anything).

JDBC drivers problems

One nice feature of the hot deploy process is that Mule will automatically load all of the jars in the lib folder and ensure that they’re on the classpath. Unfortunately there is an extremely annoying problem with JDBC drivers, in which they corresponding jar will be loaded correctly, but then will fail to be found at runtime.

At startup:

Loading the following jars:
=============================
file:/opt/local/Mule/mule-standalone-3.1.1/apps/XMLPlayer/lib/mysql-connector-java-5.1.13-bin.jar
=============================
<!-- snip -->
WARN 2011-06-09 15:56:12,130 [http://XMLPlayer].connector.http.mule.default.receiver.2 org.hibernate.cfg.SettingsFactory: Could not obtain connection to query metadata
java.sql.SQLException: No suitable driver found for jdbc:mysql://localhost:3306/db

The exact same project works perfectly in the Mule IDE. The only solution I’ve found is to copy the mysql-connector-java-5.1.13-bin.jar into $MULE_HOME/lib/endorsed. There is a similar bug report but it was closed for some reason. It most certainly does not work the way you would intuitively expect.

Conclusion

Mule 3 has many improvements over Mule 2, particular with the introduction of Flows. Unfortunately, deployment is a much tricker problem than it was in Mule 2, and the resources online are woefully inadequate for the task at hand. I hope this blog post helps some poor soul going through the same frustration I went through to get a Mule 3 application deployed.

Embed a Jetty file server within Mule 3.1.1

This post details how to embed a Jetty webserver within Mule, such that static files hosted within your application are accessible to the outside world. The resources describing how to do this are few and far between; I also found them erroneous. For some reason, any time I include a test:component element in my Mule configuration files, I get a timeout. By eliminating that piece, I got things to work.

These config files assume that both jetty.xml and mule-config.xml are located in the same folder, namely conf.

mule-config.xml

<mule xmlns="http://www.mulesoft.org/schema/mule/core"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      xmlns:spring="http://www.springframework.org/schema/beans"
      xmlns:http="http://www.mulesoft.org/schema/mule/http"
      xmlns:xm="http://www.mulesoft.org/schema/mule/xml"
      xmlns:jetty="http://www.mulesoft.org/schema/mule/jetty"
      xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
        http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/3.1/mule.xsd
        http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/3.1/mule-http.xsd
scripting.xsd
        http://www.mulesoft.org/schema/mule/xml http://www.mulesoft.org/schema/mule/xml/3.1/mule-xml.xsd
        http://www.mulesoft.org/schema/mule/jetty http://www.mulesoft.org/schema/mule/jetty/3.1/mule-jetty.xsd"
        >
        
  <description>
  This configuration uses an embedded Jetty instance to serve static content.
 </description>


  <jetty:connector configFile="${app.home}/conf/jetty.xml" name="jetty_connector" ></jetty:connector>
  <!-- do not use localhost here or you will not be able to access the server except locally.-->
  <jetty:endpoint address="http://0.0.0.0:8080" 
              name="jettyEndpoint" 
              connector-ref="jetty_connector"
              path="/">
            
  </jetty:endpoint> 

  <model name="Jetty">
    <service name="jettyUMO">
      <inbound>
        <jetty:inbound-endpoint ref="jettyEndpoint" /> 
      </inbound>
    </service>
  </model>
</mule>

jetty.xml

Modified from Newbie Guide to Jetty, namely changing class names (the classes in question are bundled with Mule 3.1.1, in the Jar file found in $MULE_HOME/lib/opt/jetty-6.1.26.jar).

<?xml version="1.0"?>
<!DOCTYPE Configure PUBLIC "-//Jetty//Configure//EN" "http://www.eclipse.org/jetty/configure.dtd">

<Configure id="FileServer" class="org.mortbay.jetty.Server">
  <Set name="handler">
    <New class="org.mortbay.jetty.handler.HandlerList">
      <Set name="handlers">
        <Array type="org.mortbay.jetty.Handler">
          <Item>
            <New class="org.mortbay.jetty.handler.ResourceHandler">
              <!--  Jetty 6.1.26, which comes with Mule 3.1, does not have this method --> 
              <!--<Set name="directoriesListed">true</Set>-->
              <Set name="welcomeFiles">
                <Array type="String">
                  <Item>index.html</Item>
                </Array>
              </Set>
              <!-- This folder maps to the root URL configured for this Jetty endpoint.  If I wanted to start serving content from the a folder named "static", I would replace the . with "static".-->
              <Set name="resourceBase">.</Set>
            </New>
          </Item>
          <Item>
            <New class="org.mortbay.jetty.handler.DefaultHandler" />
          </Item>
        </Array>
      </Set>
    </New>
  </Set>
</Configure>

A gist with both of these code snippets can be found here.

Conclusion

With these two configuration files, you can launch an embedded instance of Jetty within your application, and use it to serve static content. Due to a limitation in the version of Jetty 6.1.26 which Mule 3.1.1 comes with, you cannot use the Jetty instance to list the contents of folders; instead the client must know the absolute path to the file. For my purposes this was not a problem.

Hibernate + MySQL + Mac = Foreign Key Nightmares. A painless solution to a painful problem

tl;dr summary: Avoid using mixed case table names when using MySQL on a Mac.  Use lowercase underscore separated table names instead.

I was using Hibernate to map my Java classes to MySQL tables and columns.  For most classes, inserts worked perfectly.  For other classes, I’d consistently get errors like

- SQL Error: 1452, SQLState: 23000
- Cannot add or update a child row: a foreign key constraint fails

By running the command

show engine innodb status

in my mysql window, I found following clue:

110520 14:26:09 Transaction:
TRANSACTION 85B76, ACTIVE 0 sec, OS thread id 4530606080 inserting
mysql tables in use 1, locked 1
1 lock struct(s), heap size 376, 0 row lock(s)
MySQL thread id 3, query id 2175 localhost root update
insert into TableName (pk_Pdu) values (10)
Foreign key constraint fails for table `myproj`.`tablename`:
,
  CONSTRAINT `FKEC7DE11817B41BEB` FOREIGN KEY (`pk_Pdu`) REFERENCES `ParentClass` (`pk_Pdu`)
Trying to add to index `PRIMARY` tuple:
DATA TUPLE: 3 fields;
 0: len 8; hex 800000000000000a; asc         ;;
 1: len 6; hex 000000085b76; asc     [v;;
 2: len 7; hex 00000000000000; asc        ;;

But the parent table `myproj`.`ParentClass`
or its .ibd file does not currently exist!

I knew for a fact the table existed; I was able to query it and it showed up fine. Something else must be going on.

I finally stumbled onto the answer by way of a StackOverflow post:

However, I did rename the tables all to lowercase and that did make a difference. A quick search indicates I should maybe setting lower_case_table_names = 1 since I am using InnoDB. On Mac OS/X it is 2 by default (and I failed to mention I’m using a new box which may be why it isn’t working locally).

Sure enough, as soon as I renamed the table names to be all lowercase underscore separated, things worked perfectly. The default naming strategy in Hibernate names the tables in exactly the same way as the class names (e.g. in CamelCase as opposed to lower_case_underscore_separated). Fortunately the designers saw fit to make this naming convention overridable. All I had to do was add one line of code to fix my entire problem:


Configuration config = new Configuration();
// Name tables with lowercase_underscore_separated
config.setNamingStrategy(new ImprovedNamingStrategy());

Thanks to this blog post on ImprovedNamingStrategy for pointing the way. This post also helped me find the problem.

Conclusion

If you’re using Hibernate and a MySQL database running on MacOSX, make sure that your table names are all in lowercase.  This can be accomplished by using the ImprovedNamingStrategy class when configuring Hibernate.

This experience taught me a valuable lesson.  The first is, sometimes a problem can be caused by something that’s not directly your fault per se (i.e. I hadn’t incorrectly structured my Hibernate annotations, as I initially suspected), but rather due some quirk in the operating system or external tools you’re using.  The second is it’s crucial for cross platform libraries like Hibernate to provide the hooks for you to be able to swap out default behavior, precisely to be able to work around problems like these.  Thankfully Hibernate had built in just the hooks I needed to solve the problem.

New lines in XML attributes

If you have an attribute in xml that spans multiple lines, e.g.

 q2="2
B"

you might expect the newline literal to be encoded in the resulting string when the attribute is parsed. Instead, the above example will be parsed as “2 B”, at least with Java’s SAX parser implementation. In order to have the new line literal included, you should insert the entity & #10; instead (this entity keeps getting eaten by wordpress, so ignore the space) This StackOverflow answer by Tomalak gives some more insight:

Bottom line is, the value string is saved verbatim. You get out what you put in, no need to interfere.
However… some implementations are not compliant. For example, they will encode & characters in attribute values, but forget about newline characters or tabs. This puts you in a losing position since you can’t simply replace newlines with
beforehand.
…
Upon parsing such a document, literal newlines in attributes are normalized into a single space (again, in accordance to the spec) – and thus they are lost.
Saving (and retaining!) newlines in attributes is impossible in these implementations.

Datetimes in Python – gotchas and workarounds

I’ve written previously about working with dates in Java; as I mentioned there it’s very easy to get dates/times incorrect. I feel like I have a fairly good handle on how things work in Java, but today I was faced with learning how to deal with dates/times in Python. It wasn’t an altogether pleasant experience, but I’m going to show what I learned, so hopefully this is of use to you.

The task I was trying to accomplish was to convert between Unix timestamps (seconds/milliseconds since the Epoch) and more user friendly data objects. Creating a datetime is easy:

>>> from datetime import *
>>> datetime.now()
datetime.datetime(2011, 4, 5, 19, 36, 18, 894325)
>>> datetime(2004, 1, 24)
datetime.datetime(2004, 1, 24, 0, 0)

One gotcha to note is that both the month and day fields are 1 based (1 <= month <= 12, 1 <= day <= number of days in the given month and year), whereas in Java, the month field is 0 indexed.

By default, these datetime objects will use a naïve time zone understanding that ignores offsets from UTC/day light savings time. To fix this, you need to implement your own subclass of tzinfo. It’s kind of unfortunate that they don’t make this easier. Here is an example implementation representing the UTC timezone from the previously linked page:

from datetime import tzinfo, timedelta, datetime

ZERO = timedelta(0)
HOUR = timedelta(hours=1)

class UTC(tzinfo):

    def utcoffset(self, dt):
        return ZERO

    def tzname(self, dt):
        return "UTC"

    def dst(self, dt):
        return ZERO

OK that’s not the end of the world. Now assuming we’ve created out datetime object correctly, how do we retrieve its corresponding Unix style timestamp? Let’s look at the available methods.

>>> [x for x in dir(datetime) if not x.startswith("__")]
["astimezone", "combine", "ctime", "date", "day", "dst", "fromordinal", "fromtimestamp", "hour", "isocalendar", "isoformat", "isoweekday", "max", "microsecond", 
"min", "minute", "month", "now", "replace", "resolution", "second", "strftime", "strptime", "time", "timetuple", "timetz", "today", "toordinal", "tzinfo", "tznam
e", "utcfromtimestamp", "utcnow", "utcoffset", "utctimetuple", "weekday", "year"]

Well, there’s a bunch of methods that convert from a timestamp to a datetime object. But going back the other direction is a little harder. After digging, I found a way to do so:

>>> from datetime import datetime
>>> from time import mktime
>>> dt = datetime(2008, 5, 1, 13, 35, 41, 567777)
>>> seconds = mktime(dt.timetuple())
>>> seconds += (dt.microsecond / 1000000.0)
>>> seconds
1209663341.5677769
>>> 
>>> dt2 = datetime.fromtimestamp(seconds)
>>> dt == dt2
True

Well, there you have it. To convert from a unix timestamp to a datetime object, use datetime.fromtimestamp. To convert the other direction, use time.mktime(datetime_instance.timetuple()). I wish that the library authors had seen fit to maintain symmetry (i.e. datetime should implement a totimestamp method), but fortunately there is an easy workaround. The last thing to note if you’re used to Java is that the timestamps in Python measure seconds from the epoch, as opposed to Java which deals in milliseconds from the epoch.

NetBeans Platform – Duplicate pair in treePair error

I have previously written about the NetBeans Platform
I’ve been using a lot recently at work and so have found a lot of pain points when using the APIs. I’ll try to document some of the more vexing problems here for other programmers who might be facing the same problem.

One problem you will probably face while using the NetBeans Platform is an inscrutable error message when you start up your application, something to the effect of:

java.lang.IllegalStateException: Duplicate pair in treePair1: java.lang.Object pair2: java.lang.Object index1: 55 index2: 55 item1: null item2: null id1: 16309239 id2: 4ecfe790

    at org.openide.util.lookup.ALPairComparator.compare(ALPairComparator.java:83)
    at org.openide.util.lookup.ALPairComparator.compare(ALPairComparator.java:54)
    at java.util.TreeMap.put(TreeMap.java:530)
    at java.util.TreeSet.add(TreeSet.java:238)
    at org.openide.util.lookup.AbstractLookup.getPairsAsLHS(AbstractLookup.java:322)
    at org.openide.util.lookup.MetaInfServicesLookup.beforeLookup(Met…

While it’s impossible to know from this error message, this really indicates that you had some sort of exception in the initializer of one or more of your TopComponents, leading to duplicate null entries in some internal set that NetBeans Platform maintains, leading to this error message. Fortunately, there is usually a “Previous” button on the exception window so you can see what caused the real problem.