Posts Tagged ‘bug’

rstripping Simon Pegg: Don’t use rstrip for file extension removal

March 12, 2018 Leave a comment

In Python, what does '/path/to/my/simon_pegg.jpeg'.rstrip('.jpeg') yield?

If you guessed '/path/to/my/simon_pegg' – good try, but not quite right. The real answer is '/path/to/my/simon_'.

Despite what you might intuitively think, rstrip does NOT strip off a substring from the end of a string. Instead, it eliminates all of the characters from the end of the string that are in the argument. (Note that this is not mutating the string; it returns a copy of the string.)

Since pegg contains the characters that are in .jpeg, it is eliminated as well.

While this behavior is documented, it may be surprising.

Why does it matter? There are many instances of people attempting to use this rstrip approach to strip off a file extension. For instance, you might be converting an image from one filetype to another, and need to construct the final path. Or you might want to rename a bunch of files to have consistent extensions (jpegjpg).

This buggy implementation of stripping a file extension is tricky because most of the time it works – but it works by coincidence (the file name happens not to end with the characters in the file extension).

Github is rife with examples of people making this same mistake. For instance, name = str(name).rstrip(".tif"),
i.path.rstrip('.gzip').rstrip('.gz') + '.gzip') = [i.rstrip(".jpg") for i in k]

The Go language has the same semantics for its TrimRight function. This leads to the same sort of mistakes when people use it to trim file names. For instance,

filehelper.go: filename := strings.TrimRight(f.Name( ".pdf")

latest_images.go: idStr := strings.TrimRight(f.Name(), ".jpg")

The lessons to be learned from this are,

  1. Read the documentation of the library functions you use.
  2. Test your code, and not just of the happy paths. Good tests should try to break your implementation and exercise edge cases.


(Hat tip to my colleague Fredrik Lundh who alerted me to this problem and inspired this post)

Mule 3 Deployment Gotchas / Workarounds

June 10, 2011 1 comment

Mule is an open source enterprise service bus written in Java. I’ve worked with Mule 2.2 quite a bit but only recently have started to work with Mule 3. This post details some of the pains involved with the transition, none of which are well documented or hinted at in the Migration guide.


Mule IDE specific

The Mule IDE is really a misnomer – it’s not a standalone product, but instead an Eclipse plugin. See the installation guide for more information.

XML validation warnings

By default, Eclipse 3.5 will flag all sorts of spurious errors in your XML configuration file. See the blog post for more details, but here’s the short version on how to solve it:


These issues exist whether you use the IDE to deploy the app or deploy the app via the command line.

Failure to launch / Timeouts

Mule is configured via XML. You must declare the namespaces and schema locations in order to make use of the built-in Mule constructs. For instance, here’s a snippet of one of my Mule configurations:

<mule xmlns=""

Make absolutely sure that the version of the xsd that you include matches the major version of mule that you’re using! If you accidentally place a 3.0 instead of a 3.1 in any of those entries, your app will mysteriously fail to launch and you’ll get a stack trace like the following:

INFO  2011-06-09 17:21:20,015 [main] org.mule.MuleServer: Mule Server initializing...
INFO  2011-06-09 17:21:20,298 [main] org.mule.lifecycle.AbstractLifecycleManager: Initialising RegistryBroker
INFO  2011-06-09 17:21:20,355 [main] org.mule.config.spring.MuleApplicationContext: Refreshing org.mule.config.spring.MuleApplicationContext@19bb5c09: startup date [Thu Jun 09 17:21:20 EDT 2011]; root of context hierarchy
WARN  2011-06-09 17:22:36,265 [main] org.springframework.beans.factory.xml.XmlBeanDefinitionReader: Ignored XML validation warning Operation timed out
    at org.apache.xerces.util.ErrorHandlerWrapper.createSAXParseException(Unknown Source)
    at org.apache.xerces.util.ErrorHandlerWrapper.warning(Unknown Source)
    at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
    at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
    at org.apache.xerces.impl.xs.traversers.XSDHandler.reportSchemaWarning(Unknown Source)
    at org.apache.xerces.impl.xs.traversers.XSDHandler.getSchemaDocument1(Unknown Source)
    at org.apache.xerces.impl.xs.traversers.XSDHandler.getSchemaDocument(Unknown Source)
    at org.apache.xerces.impl.xs.traversers.XSDHandler.parseSchema(Unknown Source)
    at org.apache.xerces.impl.xs.XMLSchemaLoader.loadSchema(Unknown Source)

Deploying via command line

While it’s nice to be able to use an IDE to develop Mule applications, I prefer to deploy from the command line. This allows me to script the launch of the applications. Furthermore, this approach works in a headless (screenless) remote server, whereas the IDE approach will not. The basic way to deploy an app has changed from Mule 2.2 to Mule 3. It used to be that you would call mule -config /path/to/your/config.xml. Now you move your application to the $MULE_HOME/apps folder and run mule, which in turn will deploy all the apps in the apps folder. This can be very handy, especially when coupled with the Hot Deployment features of Mule; you no longer need to have one terminal per mule app you’re running. From the article, “Mule 3: A New Deployment Model”, here are the ostensible steps you must take to deploy your application in this manner:

  • Create a directory under: $MULE_HOME/apps/foo
  • Jar custom classes (if any), and put them under: $MULE_HOME/apps/foo/lib
  • Put the master Mule config file at: $MULE_HOME/apps/foo/mule-config.xml (note that it has to be named: mule-config.xml
  • Start your app with: mule -app foo

While these instructions are correct, there are a lot of gotchas involved. Let me detail them below.

Relative paths

There is often a need to make reference to resources within your configuration file. For instance, you might need to configure an embedded Jetty webserver and tell Jetty where its configuration file is located. When you do this, it’s crucial that you prepend relative paths in the XML configuration file with ${app.home}.

The reason for this is that the current working directory in which you launch the mule process becomes the current working directory for all of your application configuration files. So if you have mule-config.xml in the root of your folder, and conf/jetty.xml in that same folder, then your reference to the jetty.xml should be ${app.home}/conf/jetty.xml. Otherwise, if you just use conf/jetty.xml and launch mule from a folder that’s not the same as the root folder of your application, all of your paths will break.

Property files / Resources

As the step #2 above says, you must jar up all of your compiled classes and include them in the lib folder of your project. If you don’t do this, you’ll get an exception when your component / custom classes are attempted to be instantiated.

What should be emphasized is that all resources that you reference from within your code must end up in the jar as well. By default, that won’t happen. You can use something like the solution presented in Ant Build: copy properties file to jar file to get this to happen.

Unintentional Application Deletion

When you deploy an app by copying a zip or folder into the apps directory and then running mule, Mule will launch it and then create a text file called ‘$APP_NAME-anchor.text’. If you delete this file, Mule will “undeploy this app in a clean way”. What isn’t noted by this is that it will delete the corresponding zip/folder. So be careful not to accidentally delete your whole project. (Not that I did that or anything).

JDBC drivers problems

One nice feature of the hot deploy process is that Mule will automatically load all of the jars in the lib folder and ensure that they’re on the classpath. Unfortunately there is an extremely annoying problem with JDBC drivers, in which they corresponding jar will be loaded correctly, but then will fail to be found at runtime.

At startup:

Loading the following jars:
<!-- snip -->
WARN 2011-06-09 15:56:12,130 [http://XMLPlayer].connector.http.mule.default.receiver.2 org.hibernate.cfg.SettingsFactory: Could not obtain connection to query metadata
java.sql.SQLException: No suitable driver found for jdbc:mysql://localhost:3306/db

The exact same project works perfectly in the Mule IDE. The only solution I’ve found is to copy the mysql-connector-java-5.1.13-bin.jar into $MULE_HOME/lib/endorsed. There is a similar bug report but it was closed for some reason. It most certainly does not work the way you would intuitively expect.


Mule 3 has many improvements over Mule 2, particular with the introduction of Flows. Unfortunately, deployment is a much tricker problem than it was in Mule 2, and the resources online are woefully inadequate for the task at hand. I hope this blog post helps some poor soul going through the same frustration I went through to get a Mule 3 application deployed.

Hibernate + MySQL + Mac = Foreign Key Nightmares. A painless solution to a painful problem

May 23, 2011 5 comments

tl;dr summary: Avoid using mixed case table names when using MySQL on a Mac.  Use lowercase underscore separated table names instead.

I was using Hibernate to map my Java classes to MySQL tables and columns.  For most classes, inserts worked perfectly.  For other classes, I’d consistently get errors like

- SQL Error: 1452, SQLState: 23000
- Cannot add or update a child row: a foreign key constraint fails

By running the command

show engine innodb status

in my mysql window, I found following clue:

110520 14:26:09 Transaction:
TRANSACTION 85B76, ACTIVE 0 sec, OS thread id 4530606080 inserting
mysql tables in use 1, locked 1
1 lock struct(s), heap size 376, 0 row lock(s)
MySQL thread id 3, query id 2175 localhost root update
insert into TableName (pk_Pdu) values (10)
Foreign key constraint fails for table `myproj`.`tablename`:
  CONSTRAINT `FKEC7DE11817B41BEB` FOREIGN KEY (`pk_Pdu`) REFERENCES `ParentClass` (`pk_Pdu`)
Trying to add to index `PRIMARY` tuple:
DATA TUPLE: 3 fields;
 0: len 8; hex 800000000000000a; asc         ;;
 1: len 6; hex 000000085b76; asc     [v;;
 2: len 7; hex 00000000000000; asc        ;;

But the parent table `myproj`.`ParentClass`
or its .ibd file does not currently exist!

I knew for a fact the table existed; I was able to query it and it showed up fine. Something else must be going on.

I finally stumbled onto the answer by way of a StackOverflow post:

However, I did rename the tables all to lowercase and that did make a difference. A quick search indicates I should maybe setting lower_case_table_names = 1 since I am using InnoDB. On Mac OS/X it is 2 by default (and I failed to mention I’m using a new box which may be why it isn’t working locally).

Sure enough, as soon as I renamed the table names to be all lowercase underscore separated, things worked perfectly. The default naming strategy in Hibernate names the tables in exactly the same way as the class names (e.g. in CamelCase as opposed to lower_case_underscore_separated). Fortunately the designers saw fit to make this naming convention overridable. All I had to do was add one line of code to fix my entire problem:

Configuration config = new Configuration();
// Name tables with lowercase_underscore_separated
config.setNamingStrategy(new ImprovedNamingStrategy());

Thanks to this blog post on ImprovedNamingStrategy for pointing the way. This post also helped me find the problem.


If you’re using Hibernate and a MySQL database running on MacOSX, make sure that your table names are all in lowercase.  This can be accomplished by using the ImprovedNamingStrategy class when configuring Hibernate.

This experience taught me a valuable lesson.  The first is, sometimes a problem can be caused by something that’s not directly your fault per se (i.e. I hadn’t incorrectly structured my Hibernate annotations, as I initially suspected), but rather due some quirk in the operating system or external tools you’re using.  The second is it’s crucial for cross platform libraries like Hibernate to provide the hooks for you to be able to swap out default behavior, precisely to be able to work around problems like these.  Thankfully Hibernate had built in just the hooks I needed to solve the problem.

Cut/Paste bug in Google Docs Spreadsheets

February 1, 2011 4 comments
Video illustrating the bug

Video illustrating the bug

I use Google Docs Spreadsheets frequently and love the ability to collaborate on the same spreadsheet with multiple people currently.  The automatic versioning is excellent as well.

There is an extremely annoying bug that plagues at least the Chrome/Firefox versions of the application running on Mac OSX 10.6.6; I am posting here with the hopes that more people up-vote my support ticket and the bug is resolved sooner.  See the full bug ticket here.

In a nutshell, the problem is that the cut keyboard shortcut (⌘X) acts differently than the cut context menu.  When you fill the cut buffer via the keyboard shortcut and paste it via the paste keyboard shortcut (⌘V), things work fine.  When you later cut a different row/cells via the Cut context menu and paste with the keyboard shortcut, the original text you cut via the ⌘X shortcut is pasted instead of the text you just chose to cut.

It’s easier to see what I’m talking about via screencast; see this video for a minimal example of how to reproduce the bug.

I’m interested if this plagues Windows users and those using different browsers.  Please post in the comments with your results.

Excel 2008 for Mac’s CSV export bug

December 6, 2010 7 comments
I ran into this at work a few weeks ago and thought I’d share.

Excel 2008’s CSV export feature is broken.  For instance, enter the following fake data into Excel:

Row Name Age
0 Nick 23
1 Bill 48
Save as -> CSV file

Full list of choices

When you use standard unix commands to view the output, the results are all garbled.

[Documents]$ cat Workbook1.csv
$ wc -l Workbook1.csv
0 Workbook1.csv
What is the issue?  The file command reveals the problem:
$ file Workbook1.csv
Workbook1.csv: ASCII text, with CR line terminators
CR stands for Carriage return, the ‘\r’ control sequence which, along with the newline character (‘\n’), is used to break up lines on Windows.  Unix OSes like Mac OS expect a single ‘\n’ new line character to terminate lines.
How can we fix this?


# convert the Workbook1.csv file into a Unix appropriate file
dos2unix Workbook1.csv WithUnixLineEndings.csv
If you don’t have dos2unix on your Mac, and you don’t want to install it, you can fake it with the tr command:
tr '\15' '\n' < Workbook1.csv # remove the carriage returns, replace with a newline
Very annoying that the Mac Excel doesn’t respect Unix line terminators.  Interestingly, I found a post that talks about ensuring that you choose a CSV file encoded for Mac, but that option seems missing from the Mac version itself.
If I’m missing something obvious, please correct me.

Android – disappearing emulator ? Restart adb server

September 13, 2010 13 comments
While developing for the Android platform has gotten a lot better in the past year, there are still some rough edges.  In particular, emulators sometimes disappear from the list of attached devices (accessed via adb devices from the terminal, or within the Devices tab when in DDMS view in the Eclipse plugin).  When this happens, you cannot interact with it via the terminal or Eclipse.  That’s a pretty big problem.

A screen shot illustrating a running emulator that does not appear in the list of attached devices

An emulator within the devices window

To solve this, you should take the following steps:

# Device is running but not showing up
[497][nicholasdunn: /Users/nicholasdunn]$ adb devices
List of devices attached

# Kill and restart
[498][nicholasdunn: /Users/nicholasdunn]$ adb kill-server
[499][nicholasdunn: /Users/nicholasdunn]$ adb start-server
* daemon not running. starting it now *
* daemon started successfully *

# Device appears, but is listed as offline
[500][nicholasdunn: /Users/nicholasdunn]$ adb devices
List of devices attached
emulator-5554    offline

# One more invocation of adb devices should get it recognized
[501][nicholasdunn: /Users/nicholasdunn]$ adb devices
List of devices attached
emulator-5554    device

If this happens to you frequently (it does to me), you can create an alias within your .bash_profile file (~/.bash_profile):

alias adb-restart='adb kill-server; adb start-server; adb devices; adb devices'

Reload your .bash_profile file:

source ~/.bash_profile

You can then invoke it from the terminal by typing adb-restart.  Sometimes one invocation of adb devices is enough to have the emulator show up as a device; others requires two.  Not sure why that is.  To be safe I’m including two in the script.

Android: Principle of least surprise violation in GPS emulation

July 16, 2010 1 comment

Some of the most viewed posts on this blog have been about Android development, namely some workarounds for bizarre bugs that Google *still* has not fixed, months later.  I haven’t touched Android in a few months, but recently have started to dive back in.  Here’s yet another instance where an Android related piece of software performs completely counter-intuitively.

When developing an Android application, I often use the emulator rather than an actual hardware device.  In order to test GPS related functionality, you can either send simulated GPS events through the Dalvik Debug Monitor Service (DDMS) via Eclipse or through the Android terminal via telnet.  I’ll detail surprising bugs/defects in both ways.


The DDMS Eclipse view is available after installing the ADT plugin, and it’s absolutely crucial in order to see logging messages produced from your application (“LogCat”).    It also allows you to send GPS coordinates, as the following screenshot shows.

Unfortunately there is a bug in this software that drove me up the wall, and I never would have figured it out myself.  I was running into the problem where the first GPS coordinate sent would be received by the emulator, but all subsequent ones would not.  It turns out the problem has to do with locales and decimals being treated as commas, or vice versa.  See the following Google bug report for more details.  The bug has been open over a year; fortunately it looks like it’s slated to be fixed in the next SDK release.


If you telnet into your emulator instance, you have access to the Android console environment.

telnet localhost 5554

Android Console: type ‘help’ for a list of commands

One of the commands you can send is fix, which send a latitude longitude pair to the emulator.  Well, that’s what I assumed it did.  After wondering why all my locations were were coming out incorrectly, I read the documentation a little more carefully.

fix <longitude> <latitude> [<altitude>] Send a simple GPS fix to the emulator instance. Specify longitude and latitude in decimal degrees. Specify altitude in meters.

There is a VERY strong convention to specify latitude before longitude.  (“latitude and longitude” returns 4,240,000 results as opposed to 542,000 for “longitude and latitude”)  Having this command take arguments in the opposite order is very surprising and counter-intuitive.   I notice that the DDMS screen has it longitude, latitude as well, but even within the Android APIs, you specify the latitude first!

I realize I should have read the documentation first, but it still is a flaw that the software behaves in unexpected manners (it violates the Principle of Least Surprise/Astonishment), and inconsistent with how API calls in the rest of the OS work.  If someone has an explanation for why it’s written this way, I’d love to hear it.

Edit: Later I learned that KML has the same convention, longitude followed by latitude. This corresponds with the x,y ordering conventional in math, so it’s perhaps not so surprising after all. This ordering problem is fairly minor in the grand scheme of things, and reflects more on my own inexperience with geospatial APIs than any fault of Android itself.

NetBeans: Hang after debugging session

July 9, 2010 Leave a comment

I’ve run into the issue a lot where I will use the debugger, stop the debugger, attempt to run the project again, and it just hangs.  I always forget how to solve the problem so I figured I’d write it down for posterity.

In the build folder of your project, look for “testuserdir/lock”.  If it exists, delete it.

I find the problem manifests itself when debugging Netbeans Platform modules; when you clean and build the module no errors or warnings are reported, but when you clean and build the main project, you get an error:

/Applications/NetBeans/NetBeans Will not delete /Users/nicholasdunn/Desktop/project/build/testuserdir because /Users/nicholasdunn/Desktop/project/build/testuserdir/lock still exists; kill any running process and delete lock file if necessary
BUILD FAILED (total time: 0 seconds)