Archive
rstripping Simon Pegg: Don’t use rstrip for file extension removal
In Python, what does '/path/to/my/simon_pegg.jpeg'.rstrip('.jpeg') yield?
If you guessed '/path/to/my/simon_pegg' – good try, but not quite right. The real answer is '/path/to/my/simon_'.
Despite what you might intuitively think, rstrip does NOT strip off a substring from the end of a string. Instead, it eliminates all of the characters from the end of the string that are in the argument. (Note that this is not mutating the string; it returns a copy of the string.)
Since pegg contains the characters that are in .jpeg, it is eliminated as well.
While this behavior is documented, it may be surprising.
Why does it matter? There are many instances of people attempting to use this rstrip approach to strip off a file extension. For instance, you might be converting an image from one filetype to another, and need to construct the final path. Or you might want to rename a bunch of files to have consistent extensions (jpeg → jpg).
This buggy implementation of stripping a file extension is tricky because most of the time it works – but it works by coincidence (the file name happens not to end with the characters in the file extension).
Github is rife with examples of people making this same mistake. For instance,
main.py: name = str(name).rstrip(".tif")
fixname.py:os.rename(i.path,
i.path.rstrip('.gzip').rstrip('.gz') + '.gzip')
selector.py:l = [i.rstrip(".jpg") for i in k]
The Go language has the same semantics for its TrimRight function. This leads to the same sort of mistakes when people use it to trim file names. For instance,
filehelper.go: filename := strings.TrimRight(f.Name( ".pdf")
latest_images.go: idStr := strings.TrimRight(f.Name(), ".jpg")
The lessons to be learned from this are,
- Read the documentation of the library functions you use.
- Test your code, and not just of the happy paths. Good tests should try to break your implementation and exercise edge cases.
(Hat tip to my colleague Fredrik Lundh who alerted me to this problem and inspired this post)
Mule 3 Deployment Gotchas / Workarounds
Mule is an open source enterprise service bus written in Java. I’ve worked with Mule 2.2 quite a bit but only recently have started to work with Mule 3. This post details some of the pains involved with the transition, none of which are well documented or hinted at in the Migration guide.
Gotchas/Workarounds
Mule IDE specific
The Mule IDE is really a misnomer – it’s not a standalone product, but instead an Eclipse plugin. See the installation guide for more information.
XML validation warnings
By default, Eclipse 3.5 will flag all sorts of spurious errors in your XML configuration file. See the blog post for more details, but here’s the short version on how to solve it:
General
These issues exist whether you use the IDE to deploy the app or deploy the app via the command line.
Failure to launch / Timeouts
Mule is configured via XML. You must declare the namespaces and schema locations in order to make use of the built-in Mule constructs. For instance, here’s a snippet of one of my Mule configurations:
<mule xmlns="http://www.mulesoft.org/schema/mule/core"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:spring="http://www.springframework.org/schema/beans"
xmlns:vm="http://www.mulesoft.org/schema/mule/vm"
xmlns:script="http://www.mulesoft.org/schema/mule/scripting"
xmlns:http="http://www.mulesoft.org/schema/mule/http"
xmlns:cxf="http://www.mulesoft.org/schema/mule/cxf"
xmlns:xm="http://www.mulesoft.org/schema/mule/xml"
xmlns:pattern="http://www.mulesoft.org/schema/mule/pattern"
xmlns:servlet="http://www.mulesoft.org/schema/mule/servlet"
xmlns:jetty="http://www.mulesoft.org/schema/mule/jetty"
xmlns:test="http://www.mulesoft.org/schema/mule/test"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/3.1/mule.xsd
http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/3.1/mule-http.xsd
http://www.mulesoft.org/schema/mule/cxf http://www.mulesoft.org/schema/mule/cxf/3.1/mule-cxf.xsd
http://www.mulesoft.org/schema/mule/scripting http://www.mulesoft.org/schema/mule/scripting/3.1/mule-scripting.xsd
http://www.mulesoft.org/schema/mule/pattern http://www.mulesoft.org/schema/mule/pattern/3.1/mule-pattern.xsd
http://www.mulesoft.org/schema/mule/xml http://www.mulesoft.org/schema/mule/xml/3.1/mule-xml.xsd
http://www.mulesoft.org/schema/mule/vm http://www.mulesoft.org/schema/mule/vm/3.1/mule-vm.xsd
http://www.mulesoft.org/schema/mule/servlet http://www.mulesoft.org/schema/mule/servlet/3.1/mule-servlet.xsd
http://www.mulesoft.org/schema/mule/test http://www.mulesoft.org/schema/mule/test/3.1/mule-test.xsd
http://www.mulesoft.org/schema/mule/jetty http://www.mulesoft.org/schema/mule/jetty/3.1/mule-jetty.xsd"
>
Make absolutely sure that the version of the xsd that you include matches the major version of mule that you’re using! If you accidentally place a 3.0 instead of a 3.1 in any of those entries, your app will mysteriously fail to launch and you’ll get a stack trace like the following:
INFO 2011-06-09 17:21:20,015 [main] org.mule.MuleServer: Mule Server initializing...
INFO 2011-06-09 17:21:20,298 [main] org.mule.lifecycle.AbstractLifecycleManager: Initialising RegistryBroker
INFO 2011-06-09 17:21:20,355 [main] org.mule.config.spring.MuleApplicationContext: Refreshing org.mule.config.spring.MuleApplicationContext@19bb5c09: startup date [Thu Jun 09 17:21:20 EDT 2011]; root of context hierarchy
WARN 2011-06-09 17:22:36,265 [main] org.springframework.beans.factory.xml.XmlBeanDefinitionReader: Ignored XML validation warning
java.net.ConnectException: Operation timed out
at org.apache.xerces.util.ErrorHandlerWrapper.createSAXParseException(Unknown Source)
at org.apache.xerces.util.ErrorHandlerWrapper.warning(Unknown Source)
at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
at org.apache.xerces.impl.xs.traversers.XSDHandler.reportSchemaWarning(Unknown Source)
at org.apache.xerces.impl.xs.traversers.XSDHandler.getSchemaDocument1(Unknown Source)
at org.apache.xerces.impl.xs.traversers.XSDHandler.getSchemaDocument(Unknown Source)
at org.apache.xerces.impl.xs.traversers.XSDHandler.parseSchema(Unknown Source)
at org.apache.xerces.impl.xs.XMLSchemaLoader.loadSchema(Unknown Source)
Deploying via command line
While it’s nice to be able to use an IDE to develop Mule applications, I prefer to deploy from the command line. This allows me to script the launch of the applications. Furthermore, this approach works in a headless (screenless) remote server, whereas the IDE approach will not. The basic way to deploy an app has changed from Mule 2.2 to Mule 3. It used to be that you would call mule -config /path/to/your/config.xml. Now you move your application to the $MULE_HOME/apps folder and run mule, which in turn will deploy all the apps in the apps folder. This can be very handy, especially when coupled with the Hot Deployment features of Mule; you no longer need to have one terminal per mule app you’re running. From the article, “Mule 3: A New Deployment Model”, here are the ostensible steps you must take to deploy your application in this manner:
- Create a directory under: $MULE_HOME/apps/foo
- Jar custom classes (if any), and put them under: $MULE_HOME/apps/foo/lib
- Put the master Mule config file at: $MULE_HOME/apps/foo/mule-config.xml (note that it has to be named: mule-config.xml
- Start your app with: mule -app foo
While these instructions are correct, there are a lot of gotchas involved. Let me detail them below.
Relative paths
There is often a need to make reference to resources within your configuration file. For instance, you might need to configure an embedded Jetty webserver and tell Jetty where its configuration file is located. When you do this, it’s crucial that you prepend relative paths in the XML configuration file with ${app.home}.
The reason for this is that the current working directory in which you launch the mule process becomes the current working directory for all of your application configuration files. So if you have mule-config.xml in the root of your folder, and conf/jetty.xml in that same folder, then your reference to the jetty.xml should be ${app.home}/conf/jetty.xml. Otherwise, if you just use conf/jetty.xml and launch mule from a folder that’s not the same as the root folder of your application, all of your paths will break.
Property files / Resources
As the step #2 above says, you must jar up all of your compiled classes and include them in the lib folder of your project. If you don’t do this, you’ll get an exception when your component / custom classes are attempted to be instantiated.
What should be emphasized is that all resources that you reference from within your code must end up in the jar as well. By default, that won’t happen. You can use something like the solution presented in Ant Build: copy properties file to jar file to get this to happen.
Unintentional Application Deletion
When you deploy an app by copying a zip or folder into the apps directory and then running mule, Mule will launch it and then create a text file called ‘$APP_NAME-anchor.text’. If you delete this file, Mule will “undeploy this app in a clean way”. What isn’t noted by this is that it will delete the corresponding zip/folder. So be careful not to accidentally delete your whole project. (Not that I did that or anything).
JDBC drivers problems
One nice feature of the hot deploy process is that Mule will automatically load all of the jars in the lib folder and ensure that they’re on the classpath. Unfortunately there is an extremely annoying problem with JDBC drivers, in which they corresponding jar will be loaded correctly, but then will fail to be found at runtime.
At startup:
Loading the following jars:
=============================
file:/opt/local/Mule/mule-standalone-3.1.1/apps/XMLPlayer/lib/mysql-connector-java-5.1.13-bin.jar
=============================
<!-- snip -->
WARN 2011-06-09 15:56:12,130 [http://XMLPlayer].connector.http.mule.default.receiver.2 org.hibernate.cfg.SettingsFactory: Could not obtain connection to query metadata
java.sql.SQLException: No suitable driver found for jdbc:mysql://localhost:3306/db
The exact same project works perfectly in the Mule IDE. The only solution I’ve found is to copy the mysql-connector-java-5.1.13-bin.jar into $MULE_HOME/lib/endorsed. There is a similar bug report but it was closed for some reason. It most certainly does not work the way you would intuitively expect.
Conclusion
Mule 3 has many improvements over Mule 2, particular with the introduction of Flows. Unfortunately, deployment is a much tricker problem than it was in Mule 2, and the resources online are woefully inadequate for the task at hand. I hope this blog post helps some poor soul going through the same frustration I went through to get a Mule 3 application deployed.
Hibernate + MySQL + Mac = Foreign Key Nightmares. A painless solution to a painful problem
tl;dr summary: Avoid using mixed case table names when using MySQL on a Mac. Use lowercase underscore separated table names instead.
I was using Hibernate to map my Java classes to MySQL tables and columns. For most classes, inserts worked perfectly. For other classes, I’d consistently get errors like
- SQL Error: 1452, SQLState: 23000 - Cannot add or update a child row: a foreign key constraint fails
By running the command
show engine innodb status
in my mysql window, I found following clue:
110520 14:26:09 Transaction: TRANSACTION 85B76, ACTIVE 0 sec, OS thread id 4530606080 inserting mysql tables in use 1, locked 1 1 lock struct(s), heap size 376, 0 row lock(s) MySQL thread id 3, query id 2175 localhost root update insert into TableName (pk_Pdu) values (10) Foreign key constraint fails for table `myproj`.`tablename`: , CONSTRAINT `FKEC7DE11817B41BEB` FOREIGN KEY (`pk_Pdu`) REFERENCES `ParentClass` (`pk_Pdu`) Trying to add to index `PRIMARY` tuple: DATA TUPLE: 3 fields; 0: len 8; hex 800000000000000a; asc ;; 1: len 6; hex 000000085b76; asc [v;; 2: len 7; hex 00000000000000; asc ;; But the parent table `myproj`.`ParentClass` or its .ibd file does not currently exist!
I knew for a fact the table existed; I was able to query it and it showed up fine. Something else must be going on.
I finally stumbled onto the answer by way of a StackOverflow post:
However, I did rename the tables all to lowercase and that did make a difference. A quick search indicates I should maybe setting lower_case_table_names = 1 since I am using InnoDB. On Mac OS/X it is 2 by default (and I failed to mention I’m using a new box which may be why it isn’t working locally).
Sure enough, as soon as I renamed the table names to be all lowercase underscore separated, things worked perfectly. The default naming strategy in Hibernate names the tables in exactly the same way as the class names (e.g. in CamelCase as opposed to lower_case_underscore_separated). Fortunately the designers saw fit to make this naming convention overridable. All I had to do was add one line of code to fix my entire problem:
Configuration config = new Configuration();
// Name tables with lowercase_underscore_separated
config.setNamingStrategy(new ImprovedNamingStrategy());
Thanks to this blog post on ImprovedNamingStrategy for pointing the way. This post also helped me find the problem.
Conclusion
If you’re using Hibernate and a MySQL database running on MacOSX, make sure that your table names are all in lowercase. This can be accomplished by using the ImprovedNamingStrategy class when configuring Hibernate.
This experience taught me a valuable lesson. The first is, sometimes a problem can be caused by something that’s not directly your fault per se (i.e. I hadn’t incorrectly structured my Hibernate annotations, as I initially suspected), but rather due some quirk in the operating system or external tools you’re using. The second is it’s crucial for cross platform libraries like Hibernate to provide the hooks for you to be able to swap out default behavior, precisely to be able to work around problems like these. Thankfully Hibernate had built in just the hooks I needed to solve the problem.
Cut/Paste bug in Google Docs Spreadsheets
Video illustrating the bug
I use Google Docs Spreadsheets frequently and love the ability to collaborate on the same spreadsheet with multiple people currently. The automatic versioning is excellent as well.
There is an extremely annoying bug that plagues at least the Chrome/Firefox versions of the application running on Mac OSX 10.6.6; I am posting here with the hopes that more people up-vote my support ticket and the bug is resolved sooner. See the full bug ticket here.
In a nutshell, the problem is that the cut keyboard shortcut (⌘X) acts differently than the cut context menu. When you fill the cut buffer via the keyboard shortcut and paste it via the paste keyboard shortcut (⌘V), things work fine. When you later cut a different row/cells via the Cut context menu and paste with the keyboard shortcut, the original text you cut via the ⌘X shortcut is pasted instead of the text you just chose to cut.
It’s easier to see what I’m talking about via screencast; see this video for a minimal example of how to reproduce the bug.
I’m interested if this plagues Windows users and those using different browsers. Please post in the comments with your results.
Excel 2008 for Mac’s CSV export bug
Excel 2008’s CSV export feature is broken. For instance, enter the following fake data into Excel:
| Row | Name | Age |
| 0 | Nick | 23 |
| 1 | Bill | 48 |
Full list of choices
When you use standard unix commands to view the output, the results are all garbled.
[Documents]$ cat Workbook1.csv 1,Bill,48[Documents]$ $ wc -l Workbook1.csv 0 Workbook1.csv
$ file Workbook1.csv Workbook1.csv: ASCII text, with CR line terminators
# convert the Workbook1.csv file into a Unix appropriate file dos2unix Workbook1.csv WithUnixLineEndings.csv
tr '\15' '\n' < Workbook1.csv # remove the carriage returns, replace with a newline Row,Name,Age 0,Nick,23 1,Bill,48
Android – disappearing emulator ? Restart adb server
A screen shot illustrating a running emulator that does not appear in the list of attached devices
An emulator within the devices window
To solve this, you should take the following steps:
# Device is running but not showing up [497][nicholasdunn: /Users/nicholasdunn]$ adb devices List of devices attached # Kill and restart [498][nicholasdunn: /Users/nicholasdunn]$ adb kill-server [499][nicholasdunn: /Users/nicholasdunn]$ adb start-server * daemon not running. starting it now * * daemon started successfully * # Device appears, but is listed as offline [500][nicholasdunn: /Users/nicholasdunn]$ adb devices List of devices attached emulator-5554 offline # One more invocation of adb devices should get it recognized [501][nicholasdunn: /Users/nicholasdunn]$ adb devices List of devices attached emulator-5554 device
If this happens to you frequently (it does to me), you can create an alias within your .bash_profile file (~/.bash_profile):
alias adb-restart='adb kill-server; adb start-server; adb devices; adb devices'
Reload your .bash_profile file:
source ~/.bash_profile
You can then invoke it from the terminal by typing adb-restart. Sometimes one invocation of adb devices is enough to have the emulator show up as a device; others requires two. Not sure why that is. To be safe I’m including two in the script.
Android: Principle of least surprise violation in GPS emulation
Some of the most viewed posts on this blog have been about Android development, namely some workarounds for bizarre bugs that Google *still* has not fixed, months later. I haven’t touched Android in a few months, but recently have started to dive back in. Here’s yet another instance where an Android related piece of software performs completely counter-intuitively.
When developing an Android application, I often use the emulator rather than an actual hardware device. In order to test GPS related functionality, you can either send simulated GPS events through the Dalvik Debug Monitor Service (DDMS) via Eclipse or through the Android terminal via telnet. I’ll detail surprising bugs/defects in both ways.
DDMS
The DDMS Eclipse view is available after installing the ADT plugin, and it’s absolutely crucial in order to see logging messages produced from your application (“LogCat”). It also allows you to send GPS coordinates, as the following screenshot shows.
Unfortunately there is a bug in this software that drove me up the wall, and I never would have figured it out myself. I was running into the problem where the first GPS coordinate sent would be received by the emulator, but all subsequent ones would not. It turns out the problem has to do with locales and decimals being treated as commas, or vice versa. See the following Google bug report for more details. The bug has been open over a year; fortunately it looks like it’s slated to be fixed in the next SDK release.
Telnet
If you telnet into your emulator instance, you have access to the Android console environment.
telnet localhost 5554
Android Console: type ‘help’ for a list of commands
OK
One of the commands you can send is fix, which send a latitude longitude pair to the emulator. Well, that’s what I assumed it did. After wondering why all my locations were were coming out incorrectly, I read the documentation a little more carefully.
| fix <longitude> <latitude> [<altitude>] | Send a simple GPS fix to the emulator instance. | Specify longitude and latitude in decimal degrees. Specify altitude in meters. |
There is a VERY strong convention to specify latitude before longitude. (“latitude and longitude” returns 4,240,000 results as opposed to 542,000 for “longitude and latitude”) Having this command take arguments in the opposite order is very surprising and counter-intuitive. I notice that the DDMS screen has it longitude, latitude as well, but even within the Android APIs, you specify the latitude first!
I realize I should have read the documentation first, but it still is a flaw that the software behaves in unexpected manners (it violates the Principle of Least Surprise/Astonishment), and inconsistent with how API calls in the rest of the OS work. If someone has an explanation for why it’s written this way, I’d love to hear it.
Edit: Later I learned that KML has the same convention, longitude followed by latitude. This corresponds with the x,y ordering conventional in math, so it’s perhaps not so surprising after all. This ordering problem is fairly minor in the grand scheme of things, and reflects more on my own inexperience with geospatial APIs than any fault of Android itself.
NetBeans: Hang after debugging session
I’ve run into the issue a lot where I will use the debugger, stop the debugger, attempt to run the project again, and it just hangs. I always forget how to solve the problem so I figured I’d write it down for posterity.
In the build folder of your project, look for “testuserdir/lock”. If it exists, delete it.
I find the problem manifests itself when debugging Netbeans Platform modules; when you clean and build the module no errors or warnings are reported, but when you clean and build the main project, you get an error:
/Applications/NetBeans/NetBeans 6.8.app/Contents/Resources/NetBeans/harness/suite.xml:451: Will not delete /Users/nicholasdunn/Desktop/project/build/testuserdir because /Users/nicholasdunn/Desktop/project/build/testuserdir/lock still exists; kill any running process and delete lock file if necessary
BUILD FAILED (total time: 0 seconds)
Stack Overflow profile
Developmentality 