Archive

Posts Tagged ‘workaround’

EventBus – how to switch EventService implementations for unit testing

February 23, 2011 2 comments

I’ve written previously about EventBus, a great open source Java library for pub-sub (publish subscribe). It’s a truly excellent way to write loosely coupled systems, and much preferable to having to make your domain models extends Observable and your listeners implement Observer. I’m writing today to describe some difficulties in incorporating EventBus into unit tests, and how to overcome that problem.

Test setup

I was attempting to test that certain messages were being published by a domain model object when they were supposed to. In order to test this, I wrote a simple class that did nothing more than listen to the topics I knew that my model object was supposed to publish to, and then increment a counter when these methods were called. It looked something like this:

class EventBusListener {
    private int numTimesTopicOneCalled = 0;
    private int numTimesTopicTwoCalled = 0;

    public EventBusListener() {
        AnnotationProcessor.process(this);
    }

    @EventTopicSubscriber(topic="topic_one")
    public void topicOneCalled(String topic, Object arg) {
        this.numTimesTopicOneCalled++;
    }

    @EventTopicSubscriber(topic="topic_two")
    public void topicTwoCalled(String topic, Object arg) {
        this.numTimesTopicTwoCalled++;
    }

    public int getNumTimesTopicOneCalled() {
        return this.numTimesTopicOneCalled;
    }

    public int getNumTimesTopicOneCalled() {
        return this.numTimesTopicTwoCalled;
    }
}

The basic test routine looked something like this:

@Test
public void testTopicsFired() {


    // Uses EventBus internally
    DomainObject obj = new DomainObject();

    int count = 10;
    EventBusListener listener = new EventBusListener();
    for (int i = 0; i < count; i++) {
        obj.doSomethingThatShouldFireEventBusPublishing();
    }

    assertEquals(count, listener.getNumTimesTopicOneCalled());
    assertEquals(count, listener.getNumTimesTopicTwoCalled());
}

This code kept failing, but in nondeterministic ways – sometimes the listener would report having its topic one called 4 times instead of 10, sometimes 7, but never the same issue twice. Stepping through the code in debug mode I saw that the calls to EventBus.publish were in place, and sometimes they worked. Nondeterminism like this made me think of a threading issue, so I began to investigate.

Problem

After reading through the EventBus javadoc, I came upon the root of the problem:

The EventBus is really just a convenience class that provides a static wrapper around a global EventService instance. This class exists solely for simplicity. Calling EventBus.subscribeXXX/publishXXX is equivalent to EventServiceLocator.getEventBusService().subscribeXXX/publishXXX, it is just shorter to type. See EventServiceLocator for details on how to customize the global EventService in place of the default SwingEventService.

And from the SwingEventService javadoc (emphasis mine):

This class is Swing thread-safe. All publish() calls NOT on the Swing EventDispatchThread thread are queued onto the EDT. If the calling thread is the EDT, then this is a simple pass-through (i.e the subscribers are notified on the same stack frame, just like they would be had they added themselves via Swing addXXListener methods).

Here’s the crux of the issue: the EventBus.publish calls are not occurring on the EventDispatchThread, since the Unit testing environment is headless and this domain object is similarly not graphical. Thus these calls are being queued up using SwingUtilities.invokeLater, and they have no executed by the time the unit test has completed. This leads to the non-deterministic behavior, as a certain number of the queued up messages are able to be processed before the end of execution of the unit test, but not all of them.

Solutions

Sleep Hack

One solution, albeit a terrible one, would be to put a hack in:

@Test
public void testTopicsFired() {
    // same as before

    // Let the messages get dequeued
    try {
        Thread.sleep(3000);
    }
    catch (InterruptedException e) {}

    assertEquals(count, listener.getNumTimesTopicOneCalled());
    assertEquals(count, listener.getNumTimesTopicTwoCalled());
}

This is an awful solution because it involves an absolute hack. Furthermore, it makes that unit test always take at least 3 seconds, which is going to slow the whole test suite down.

ThreadSafeEventService

The real key is to ensure that whatever we call for EventBus within our unit testing code is using a ThreadSafeEventService. This EventService implementation does not use the invokeLater method, so you can be assured that the messages will be delivered in a deterministic manner. As I previously described, the EventBus static methods are convenience wrappers around a certain implementation of the EventService interface. We are able to modify what the default implementations will be by the EventServiceLocator class. From the docs:

By default will lazily hold a SwingEventService, which is mapped to SERVICE_NAME_SWING_EVENT_SERVICE and returned by getSwingEventService(). Also by default this same instance is returned by getEventBusService(), is mapped to SERVICE_NAME_EVENT_BUS and wrapped by the EventBus.

To change the default implementation class for the EventBus’ EventService, use the API:

EventServiceLocator.setEventService(EventServiceLocator.SERVICE_NAME_EVENT_BUS, new SomeEventServiceImpl());

Or use system properties by:

System.setProperty(EventServiceLocator.SERVICE_NAME_EVENT_BUS,
 YourEventServiceImpl.class.getName());

In other words, you can replace the SwingEventService implementation with the ThreadSafeEventService by calling

EventServiceLocator.setEventService(EventServiceLocator.SERVICE_NAME_EVENT_BUS, 
new ThreadSafeEventService());

An alternative solution is use an EventService instance to publish to rather than the EventBus singleton, and expose getters/setters to that EventService. It can start initialized to the same value that the EventBus would be wrapping, and then the ThreadSafeEventService can be injected for testing. For instance:


public class ClassToTest{
    // Use the default EventBus implementation
    private EventService eventService = EventServiceLocator.getEventBusService();

    public void setEventService(EventService service) {
        this.eventService = service;
    }
    public EventService getEventService() {
        return this.eventService;
    }

    public void doSomethingThatNotifiesOthers() {
        // as opposed to EventBus.publish, use an instance of EventService explicitly
        eventService.publish(...);
    }
}

Conclusion

I have explained how EventBus static method calls map directly to a singleton implementation of the EventService interface. The default interface works well for Swing applications, due to its queuing of messages via the SwingUtilities.invokeLater method. Unfortunately, it does not work for unit tests that listen for these EventBus publish events, since the behavior is nondeterministic and the listener might not be notified by the end of the unit test. I presented a solution for replacing the default SwingEventService implementation with a ThreadSafeEventService, which will work perfectly for unit tests.

Java annoyances

January 28, 2011 Leave a comment
Having had Java as the programming language of the vast majority of my undergraduate courses, as well as the language I program in every day, I am most comfortable and fluent in it.  When I return to Java after using different languages such as AWKPython, or Ruby, I’m always left with a bitter taste in my mouth.  There are some things Java just makes way too hard, verbose, and painful to accomplish.  It’s for that reason that I’m learning Scala, what could be (simplistically) described as a cleaned up, more succint version of Java.

Asymmetry in standard libraries

Symmetry is an important feature of a library; it basically means that methods come in pairs.  For instance, you’d expect that a class with a read method has a write method, or one with a set method has a get method.  (That’s not always the case, certainly, but API writers often strive for symmetry.  See Practical API Design: Confessions of a Java Framework Architect) As another example, there’s both a method to convert from an array to a list (Arrays.asList) and there is a method to go the other direction (List.toArray()).  Unfortunately, not all of the Java library APIs adhere to this convention.  The one that bothers me the most is in the String library.  There is a String split method that breaks a String up around a given regular expression, but no corresponding method to reconstitute a String from a collection of other String objects, with a specified separator between them.  This leads to code like the following to comma separate a collection of strings:
String[] strings = ...;
StringBuilder b = new StringBuilder();
for (int i = 0; i < strings.length; i++) {
 b.append(strings[i]);
 if (i != strings.length -1) {
 b.append(",");
 }
}
System.out.println(b.toString());
This whole mess could be replaced with one line of Python code
print(",".join(strings))
or in Scala:
println(strings.mkString(","))
It’s pretty sad that you have to either write that ugly mess, or turn to something like Apache StringUtils.

Different treatment of primitives and objects

It is a lot harder to deal with variable length collections of primitive types than it should be.  This is because you cannot create collections out of things that are not objects.  You can create them out of the boxed primitive type wrappers, but then you have to iterate through and convert back into the primitive types.

In other words, you can create a List<Double> but you cannot create a List<double>.  This leads to code like the following:
// Need a double[] but don't know how long it's going to be
List<Double> doubles = new LinkedList<Double>();
for (...) {
 doubles.append(theComputedValue);
}
// Option 1: Use for loop and iterate over Double list, converting to primitive values through
// auto unboxing
// Bad: leads to O(n^2) running time with LinkedList
double[] doubleArray = new double[doubles.size()];
for (int i = 0; i < doubleArray.length; i++) {
 doubleArray[i] = doubles.get(i);
}
// Option 2: Use enhanced for loop syntax (described below), along with
// additional index variable.
// Better performance but extraneous index value hanging around
int index = 0;
// Automatic unboxing
for (double d : doubles) {
 doubleArray[index++] = d;
}
...
b[index] // Oops
As I blogged about previously, there is a library called Apache Commons Primitives that can be used for variable sized lists for primitive types, but it is a shame one has to turn to third party libraries for such a common task.

Patchwork iteration support

Java 5 introduced the “Enhanced for loop” syntax which allows you to replace
Collection<String> strings = new ArrayList<String>();
Iterator<String> it = strings.iterator();
while (it.hasNext()) {
 String theString = it.next();
}
with the much simpler
for (String s : strings) {
 // deal with the String
}
Here’s the rub: this syntax is supported for arrays and Iterable objects.  But guess what?  Iterators are not Iterable.  Why is this a problem?  Well, you might want to return read-only iterators to your data.  If you do this, the client code cannot use the enhanced for loop syntax, and is stuck with the earlier hasNext() code.  If you want to use the enhanced for loop syntax to work for Iterators, you need to introduce a wrapper around the Iterator which implements the Iterable interface.  From the previously linked blog post:
class IterableIterator<T> implements  Iterable<T> {
    private Iterator<T> iter;
    public IterableIterator(Iterator<T> iter) {
        this.iter = iter;
    }
    // Fulfill the Iterable interface
    public Iterator<T> iterator() {
        return iter;
    }
}
I hope this strikes you as inelegant as well.
Furthermore, arrays are not iterable either, despite the fact that you can use the enhanced for loop syntax with them.
What this all boils down to is that there’s no great way to accept an Iterable collection of objects.  If you accept an Iterable<E>, you close yourself off to arrays and iterators.  You’d have to convert the arrays to a suitable collection type by using the Arrays.asList method.  It would be great if we could treat arrays, collections, etc., agnostically when all we want to do is iterate over their elements.

Lack of type inference for constructors with generics

Yes, we all know we should program to an interface rather than to a specific implementation; doing so will allow our code to be much more flexible and easily changed later.  Furthermore, we also know we should use generics for our collections rather than raw collections of objects; this allows us to catch typing errors before they occur.  So in other words

// BAD: Raw hashmap and programming to the implementation!
HashMap b = new HashMap();
// Good
Map<String, Integer> wordCounts = new HashMap<String, Integer>();
In fact, this lack of type inference is one reason why Joshua Bloch suggests that static factory methods can be better that constructors – it is possible to have a static factory method that can infer the correct types and instantiate the object, without making you explicitly repeat the type parameters.  For instance, Google Guava provides many static methods to instantiate maps:
Map<String, Integer> wordCounts = Maps.newHashMap();
Fortunately, the problem of having to repeat type parameters twice for constructors is being fixed in JDK 7 with something called the Diamond Operator.  It will allow you to replace
Map<String, Integer> wordCounts = new HashMap<String, Integer>();
with
Map<String, Integer> wordCounts = new HashMap<>();
This improvement to the language can’t come fast enough.

Conclusion

I use Java on a daily basis for about 90% of the work I need to do.  I’m comfortable with it, I understand its syntax, it’s fast, it’s powerful.  After being exposed to languages like python and scala, certain issues in Java stand out in stark contrast, and thus I’ve enumerated a few of the reasons that Java annoys me on a daily basis.  Fortunately excellent libraries exist to correct many of the annoyances, but it’s painful to have to use them to do such basic things as joining a list of Strings with a given separator character, or creating a variable sized list of primitive types. Fortunately Java continues to evolve, and at least some of my irritations will be fixed in JDK 7.
Post in the comments if you have better workarounds than those that I’ve suggested, you have other languages that make these tasks easy and would like to highlight them, or any other reason you can think of.

NetBeans Platform – ModuleInstall ClassNotFoundException

January 3, 2011 Leave a comment

In the NetBeans Platform, you can create an ModuleInstall class which handles the lifecycle of your module, and provides methods you can override to handle when your module is loaded and unloaded.  The standard way of creating this class is to use the wizard that NetBeans provides for this purpose, accessible by right clicking on the project and choosing New -> Other -> Module Development -> Module Installer

If you decide that you do not want to use the ModuleInstall mechanism any longer (the NetBeans Platform folks suggest you do NOT use it, as it will increase the startup time of the application), you might think you can just delete the Installer.java file that the wizard created.  Unfortunately, that’s not the case.  When you run the project you’ll get an exception like the following

org.netbeans.InvalidException:  
StandardModule:net.developmentality.moduleexample jarFile: ...
  java.lang.ClassNotFoundException:  net.developmentality.moduleexample.Installer starting from  ModuleCL@482...

The problem is that the wizard modified the manifest.mf file as well, and you need to manually clean up the file before your project will work again.

Open the manifest.mf file and you’ll see a line in the file like the following:

OpenIDE-Module-Install: net/developmentality/moduleexample/Installer.class

delete that line, and rebuild.  You should be ready to go.
In general, you need to be very careful about using any of the wizards that NetBeans provides.  They are extremely useful, but they end up changing XML files that you will need to manually edit later if you decide to refactor.  For instance, if you create an Action instance using the New Action Wizard, the action will be registered in the layer.xml file.  If you rename the action using the refactor command, the entries in the xml file are NOT modified, and you will get an exception at runtime unless you remember to edit that file.
Fortunately, the wizards are smart enough to at least tell you which files they are modifying.  It’s just a matter of remembering this further down the road when you need to refactor, move, or delete files.  Look for the modified files section in the wizard dialog:

 

Excel 2008 for Mac’s CSV export bug

December 6, 2010 7 comments
I ran into this at work a few weeks ago and thought I’d share.

Excel 2008’s CSV export feature is broken.  For instance, enter the following fake data into Excel:

Row Name Age
0 Nick 23
1 Bill 48
Save as -> CSV file

Full list of choices

When you use standard unix commands to view the output, the results are all garbled.

[Documents]$ cat Workbook1.csv
1,Bill,48[Documents]$
$ wc -l Workbook1.csv
0 Workbook1.csv
What is the issue?  The file command reveals the problem:
$ file Workbook1.csv
Workbook1.csv: ASCII text, with CR line terminators
CR stands for Carriage return, the ‘\r’ control sequence which, along with the newline character (‘\n’), is used to break up lines on Windows.  Unix OSes like Mac OS expect a single ‘\n’ new line character to terminate lines.
How can we fix this?

dos2unix.

# convert the Workbook1.csv file into a Unix appropriate file
dos2unix Workbook1.csv WithUnixLineEndings.csv
If you don’t have dos2unix on your Mac, and you don’t want to install it, you can fake it with the tr command:
tr '\15' '\n' < Workbook1.csv # remove the carriage returns, replace with a newline
Row,Name,Age
0,Nick,23
1,Bill,48
Very annoying that the Mac Excel doesn’t respect Unix line terminators.  Interestingly, I found a post that talks about ensuring that you choose a CSV file encoded for Mac, but that option seems missing from the Mac version itself.
If I’m missing something obvious, please correct me.

NetBeans Platform – How to handle the working directory problem

November 9, 2010 7 comments

I have written previously about NetBeans Platform, but as a refresher or an introduction to my new readers, it’s a Java based framework for developing rich desktop applications.  A NetBeans Project is comprised of one or more modules.  The code in Module A cannot be accessed by Module B, unless Module B declares a dependency on Module A, and Module A explicitly declares that its code is public.  In this way, code is segmented and encapsulated into logical blocks.

In this example, I’ve created an example project, with three modules – one for the View, Model, and Controller.  (See Wikipedia if that’s not ringing a bell, and then my post about a Java Solar system for a concrete example implementation).

The project can be run either from the main project node,

or by any of the modules:

While in most cases, the behavior will be identical, there is one way in which the manner in which you launch the application matters enormously: relative paths.
The working directory is directly influenced by whether you choose the project or the module to launch with:
# Application launched from the ExampleApplication node:
Current Directory       = /NetbeansExample/ExampleApplication

# Application launched from the View module:

Current Directory       = /NetbeansExample/ExampleApplication/View

Why is this a problem?

Say my View code has a properties file it needs to read in to initialize some code.  Let’s say I store it at
/NetbeansExample/ExampleApplication/View/resources/example.properties
From within my sourcecode I might attempt to do the following:
File f = new File("resources/example.properties");
If I have started the application from the View module, then this will work fine; the current working directory is /NetbeansExample/ExampleApplication/View, as I previously said; thus by tacking on resources/example.properties, we come to a valid file.  If you launch from the main project, you will find that the file doesn’t exist, because /NetbeansExample/ExampleApplication/resources/example.properties does not exist.

How do can we handle this?

The best way I’ve found is through the use of the InstalledFileLocator class, provided by the Netbeans Platform Module APIs.

First, we need to move the .properties file from View/resources to View/release.

Once we have done that, we need to declare a dependency on the Module API in order to use this InstalledFileLocator class.  Right click on the View node and choose Properties,
In the libraries pane, click Add Dependency, and then start typing InstalledFileLocator.
The Module API should appear in the list.  Choose it and click OK.  You should see
From within your view code, you now replace code that looked like
File f = new File("resources/example.properties");

with

File f = InstalledFileLocator.getDefault().locate("example.properties", "net.developmentality.view", false);

where “example.properties” is the path to the file you’re trying to load, relative to the “release” folder, and where “net.developmentality.view” is the package name of the file in which you’re working.  The last argument is described in the Javadocs as:

localized – true to perform a localized and branded lookup (useful for documentation etc.)

I have had no problem leaving it false for my purposes.

Conclusion

I found this workaround after scouring the forums and being led to an FAQ about the LocalizedFileLocator.  It makes the code slightly more messy, but at least it works regardless of how the user or developer launches the application, which cannot be said for the standard way of specifying relative paths.  It is a very hard thing to search for, so hopefully someone running into this problem using NetBeans Platform finds this post.

How to remove “smart” quotes from a text file

October 11, 2010 4 comments

If you’ve copied and pasted text from Microsoft Word, chances are there will be the so-called smart quotes in that text. Some programs don’t handle these characters very well. You can turn them off in Word but if you’re trying to remedy the problem after the fact, sed is your old friend.  I’ll show you how to replace these curly quotes with the traditional straight quote.

Recall that you can do global find/replace by using sed.

sed s/[”“]/'"'/g File.txt

This won’t actually change the contents of the File, but you can save the results to a new file

sed s/[”“]/'"'/g File.txt > WithoutSmartQuotes.txt

If you wish to save the files in place, overwriting the original contents, you would do

sed -i ".bk" s/[”“]/'"'/g File.txt

This tells the sed command to make the change “in place”, while backing up the original file to File.txt.bk in case anything goes wrong.

To fix the smart quotes in all the text files in a directory, do the following:

for i in *.txt; do sed -i ".bk" s/[”“]/'"'/g $i; done

At the conclusion of the command, you will have double the number of text files in the directory, due to all the backup files. When you’ve concluded that the changes are correct (do a diff File.txt File.txt.bk to see the difference), you can delete all the backup files with rm *.bk.

Java gotcha: Splitting a string

September 22, 2010 Leave a comment

CSV, or comma separated value, files are the workhorse of flat file processing. They’re human readable, you can edit them in any spreadsheet editor worth its salt, they’re portable, and they’re easy to parse and read into your programs.  Unfortunately, text you might want to store in the columns might also have commas in it, which complicates the parsing significantly – instead of splitting on all of the commas, you have to know whether the comma is in a quoted string, in which case it doesn’t signify the end of a field … it gets messy, and you’re probably going to get it wrong if you roll your own.  For a quick and dirty script, sometimes it’s better to delimit the columns with a different character, one that comes up much less often in text.  One good choice is the pipe character, |.

Foolishly I processed lines of text like the following in Scala:


val FIELD_DELIMITER = "|";
val FILE_NAME_INDEX = 0
val AVG_COLOR_INDEX = 1
val THUMBNAIL_IMAGE_INDEX = 2


def parseLine(indexRow:String) = {
 Console.println("Row: " + indexRow)

 val entries = indexRow.split(FIELD_DELIMITER)
 // extract the file name, average color, thumbnail

}
 

Unfortunately, this code is subtly broken.  The problem is that the String split method does not accept a plain string on which to split – rather, it treats the string as a regular expression.  Since the pipe character has special meaning in regular expressions (namely “OR”), this has the effect of splitting the string on every character, rather than just at the pipe delimiters.

To get around this, you need to tell the regular expression engine that you want to treat the pipe as a literal, which you do by backslash escaping the pipe.  Unfortunately “\|” is not a valid string in Java, because it will attempt to interpret it as a control character (like the newline \n).  Instead, you need to backslash escape the backslash, leaving “\\|”

Conclusion:
Be careful when you’re using String.split in Java/Scala.  You have to be aware that it’s treating your string as a regular expression.  Read this StackOverflow discussion for a better understanding of the pros and cons of Java using regular expressions in many of its core String libraries.