So here we go, this time it’s about a “web application” used by the Croatian National Bank to collect balance of payments data. I won’t bother you with too many details, although there is a million details done wrong.

The purpose of the “application” is to collect quarterly information about import&export transactions. In a senseful environment, it would probably be implemented by a regular web application where companies would log in, enter some simple data (all the data is essentially key-value pairs like this: country,export/import) and hit the “submit” button. And probably get an email reminder when the report is due.

Now back to Croatian reality

Upon registration, you get the username/password to access the portal. Then you download one XLS file (in the beginning, they supported Microsoft Excel only, of course). When all the relevant sheets of the XLS are filled in with transaction data, you press the big “generate TXT file” button.

The magic starts in this moment. A macro command iterates all sheets, collects the data and /:drumroll:/ GENERATES A TXT FILE At least it’s polite enough to point you where the file has been generated.

Then you go back to the web portal. Upload the TXT file, and hope for the best.

Besides the whole concept being broken, the actual implementation of the system is so amateurish you can hardly believe it comes from an institution so important. Have look at the official User Guide, and judge for yourself how serious this looks.

Anyway, if anyone from the Croatian National Bank gets to read this – Calyx would gladly do this for free, just to stop using this abomination.

This post won’t be about the advantages and disadvantages of distributed source control systems. It will be about setting up a source control environment based on Mercurial.

We wanted two things – integration with Eclipse and a simple method of creating/using remote Mercurial repositories which would be considered central repositories for our projects. The integration with Eclipse part wasn’t a problem – we use MercurialEclipse which works fine. Since the plug-in doesn’t enable you to do everything Mercurial-related we use it together with the standard Mercurial command-line utility.

The other thing we wanted was a practical way of creating remote Mercurial repositories and managing access rights. Since we have an online Ubuntu server we chose to use mercurial-server. Mercurial-server is a set of scripts which enables using and creating remote repositories, SSH public key authentication and fine grained per-user-per-repository access control. All communication goes through SSH.

Since we mainly use Windows for development, the first thought was to use PuTTY (actually Plink) for the communication between Mercurial and mercurial-server. The instructions on how to configure mercurial to use PuTTY can be found on the official site. After trying everything we couldn’t get it to work. After quite a lot of hours of fiddling – we found the problem. PuTTY’s saved sessions. If you’re like me, you save the connection setting for myserver.com under the session name myserver.com Big mistake, my friend… It turns out the saved sessions are known throughout the PuTTY suite, Plink included. This is normally a very convenient thing – you can type ‘plink sessionname‘ and plink will use all the connection parameters from PuTTY’s saved session – no need to specify everything again. The problem with mercurial-server is that you have to connect via SSH as ‘hg’. If the server name is the same as the session name, Plink will override the connection parameters provided by Mercurial and will use those from the saved session (which is probably configured to use another username for connection and thus mercurial-server won’t respond). Having sorted this out – Mercurial worked fine.

And by fine I mean really slow The initial cloning of a 60-something megabyte repository took more than 20 minutes. The problem turned out to be the SSH client – PuTTY in particular. We switched to cygwin + OpenSSH and the initial cloning time went down to 5 or 6 minutes which is much better. Of course this is nowhere near as fast as cloning the repository on a Linux system – it takes about a minute there

If you have some ideas on how to increase the communication speed on a Windows machine further, I would love to hear.

Besides making it hard for non-intruders to type something like “BAC34DAAAB56B7CADA336B23DA” twice, correctly

The TTS system was built on the Festival framework and is still updated. It already provides a satisfying synthesis – the intelligibility of the synthesis is quite high though we aim to improve the naturalness of the produced speech (which right now sounds quite robotic and non-enthusiastic ). We aim to release the synthesizer as free for personal use.

You can try out the synthesizer here. This blog post and the synthesizer web page will probably be updated in the following couple of days with more information. For more info you can send an e-mail to vladimir.prenner@calyx.hr.

Usually, the culprits are either some action listeners that are left hanging, or some references in some long forgotten collections (Singletons really are evil sometimes!).

In our case, it was neither. Here are the symptoms: the application does a whole lot of calculation and uses some tree-like data structures to do the math. We feed it with a lot of data and the memory footprint at one moment in time is ~ 700megs (we’ve got ~ 1G of heap). Then we’re done. And it’s not yet freed. But that’s OK, since GC tends to be lazy. The problem occurs when you want to do the same operation twice – the second time time, JVM runs out of memory.

So, after a couple of days of tracing the leak (my first suspicion was that we had some sort of a leak in our tree-like structure), it can all boil down to this small code snippet:

public class MemoryLeakTest {
    public static void main(String[] args) {
        for (int i = 0; i < 1000; i++) {
            JDialog dialog = new JDialog() {
                int[] take4Megs = new int[1000000];
            };
           dialog.setDefaultCloseOperation(JDialog.DISPOSE_ON_CLOSE);
           dialog.setVisible(true);
           dialog.setVisible(false);
        }
    }
}

Here, we create a JDialog instance that has one neat attachment to it – a 4meg worth of empty integer. Set up -Xmx64m for this test case and only after a few runs, you’ll run out of memory. How is it possible that I run out of memory here? There are no collections, all variables are locally scoped – GC should be picking JDialog up after each loop iteration (or at least after a few). However, it seems that this does not occur. The only thing that seems to fix the issue is calling dispose manually like this:

public class MemoryLeakTest {
    public static void main(String[] args) {
        for (int i = 0; i < 1000; i++) {
            JDialog dialog = new JDialog() {
                int[] take4Megs = new int[1000000];
            };
           dialog.setDefaultCloseOperation(JDialog.DISPOSE_ON_CLOSE);
           dialog.setVisible(true);
           dialog.setVisible(false);
           dialog.dispose();
        }
    }
}

Please note that the tested platform is JVM 1.6.0_19, Windows7 Ultimate (it also happens on XP). Here is what the Java documentation says about dispose:

“Releases all of the native screen resources used by this Window, its subcomponents, and all of its owned children. That is, the resources for these Components will be destroyed, any memory they consume will be returned to the OS, and they will be marked as undisplayable.”

Inspecting heap dumps of both cases revealed that if we don’t explicitly call dispose(), that JDialog (actually, MemoryLeakTest$1 instances) instances remain in memory (I guess Swing has some references to them – we obviously don’t have them) and are not garbage collected. However, doing dispose() seems to have cleaned up the system resources and removed those references in Swing because we can’t see a single dialog instance on the heap anymore. The worst thing here is that it seems that this behaviour is not documented in the documentation (dispose() docs don’t say anything about this – see http://java.sun.com/javase/6/docs/api/java/awt/Window.html#dispose() ).

The shortcut does a simple thing – it reopens the last closed tab. If you press it more than once it will reopen the tab you closed before the last one etc. Since I’m quite a tab juggler while browsing the web (meaning I usually open 7 tabs at once, reorder them, then read them and close them) I often close a tab prematurely. Also, sometimes I realize there’s something I wanted to check on a tab that I just closed.

Before I used to go through the recently closed tabs menu option, or even the history one if I couldn’t find it in the recently closed tabs (and it could happen). Now all I need to do is click 3 buttons and the magic happens! Things like these really make me happy

While the previously mentioned shortcut might not be revolutionary and not be considered “important” for one’s daily work, there’s another example – the Eclipse IDE. I’ve been using Eclipse on a daily basis for almost 3 years now. And only a few months ago I became aware of the beautiful shortcuts ctrl+shift+R and ctrl+O. For those of you who aren’t aware of them – ctrl+shif+R opens a wildcard-search window for finding any resource (for example a class file) in your workspace. Similarly ctrl+O opens a wildcard search for the members/fields of the currently open class.

Finding a class in a project was a frustrating task before – I’d wildly scroll through the project tree trying to remember which package the class belongs to. Finding a method inside a class was usually easier, but frustrating nonetheless. Now I feel happy each time I use those shortcuts (which is probably a few hundred times a day). Happy and stupid because I didn’t use them before. I wonder how much time I’ve wasted on searching for classes manually – time which could have been spent in a more productive way. Nowadays I spend a few seconds on what took me a minute or more before – the most basic task during software development – finding the code you’re working at.

The unawareness of the capabilities of the tools we use daily is truly an amazing thing. I recall I tried to find such a functionality in Eclipse, but couldn’t find it in the menus so I assumed it wasn’t there. I even considered writing my own Eclipse plugin to help me find classes. I failed to realize, at the time, that I’m not the only one with that problem and that someone else probably already solved it – what’s more, the solution was already there.

The bottom line would be, I guess, that learning about software development doesn’t start with learning the programming language we use. It starts with learning about the tools we use for coding it – something we can forget and lose valuable time. For starters – try pressing ctrl+shift+L in Eclipse. I’m sure you’ll find something you didn’t know

After the process is done, your boss wants some kind of reporting – how many entries were processed, how many were processed sucessful, how many errors were there and which errors they were. Not much of a problem when working on a UNIX machine:

A:    cat out.txt | grep 'COMMAND SUCCESS' | wc -l
B:    cat out.txt | grep 'COMMAND FAILED' | wc -l

… and just to make sure, lets check if A+B = C:

C:    cat in.txt | wc -l

Now, lets report on errors:

cat out.txt | grep 'ERROR_CODE:' | sort | uniq

returns a list of errors:

ERROR_CODE: 10065
ERROR_CODE: 11245
ERROR_CODE: 19543


and now just lets find out how many of each we got:

cat out.txt | grep 'ERROR_CODE: 10065' | wc -l
cat out.txt | grep 'ERROR_CODE: 11245' | wc -l
cat out.txt | grep 'ERROR_CODE: 19543' | wc -l


Email to the boss, and we’re done. All good? Great. But, another email comes in: “Could you please send me a list of all the IDs that caused an error #11245″. Sure, no problem:

cat out.txt | grep -B 7 'ERROR_CODE: 11245' | grep 'REQUEST_ID' | awk '{ print $2; }' | sed 's/REQUEST_ID:\([0-9*]\)/\1/g' > ids.txt

Lets explain this one a bit:

• the initial request that was sent to the system was logged 7 lines before the ERROR_CODE (therefore the “-B 7″)
• the line with the request had the following format:
  START REQUEST_ID:XXXXX SOME_OTHER_STUFF
  (therefore the awk part)
• with sed we just extracted the number from the request_id column

Can it get any more powerful than this?

Of course, I like to at least try to code stuff up, instead of just writing to Santa. However, I usually go and try to generalize a bit. Let’s say that I want to monitor a few files or outputs of various commands at the same time. Moreover, let’s say I want to add these things as I go. Enter
remotespy.

Remotespy is a script that parses a list of commands to run and compiles their output into a HTML webpage. The page is then saved to a local folder, e.g. public_html. Additionally, it can be scp-ed to a destination of your choice. This is useful if your “workhorse” server doesn’t have a http server setup, but you do have some other place to post your output to.

The syntax is simple:
remotespy COMMANDFILE HTMLFILE"
The spirit of the command file is similar to the one of crontab scripts: every line represents a command that is run. Syntax of every
line is:
TITLE ## COMMAND ## COMMENT

A sample script looks like this:
Hostname ## hostname
OS ## uname -orpi ## OS version
Disk ## df -h ## Free disk space
Top ## top -b -n 1 -i ## Process state
Vmstat ## vmstat ## Memory state

Every command is run as-is and its output captured and formatted into HTML. The HTML code is hardcoded in the script, but, script being a script, it can be easily edited. To make remotespy into a tool satisfying the initial desire of remote monitoring, it can be paired with cron and ssh to periodically run commands from the script and send the files to a (remote) public_html folder which is available to the internets. For the SCP upload to be useful, you should set up a ssh key pair between the source server (one running remotespy) and destination http server. These two articles can help if you don’t know how to do it yet:

• http://www.linuxjournal.com/article/8600
• http://mindspill.net/computing/linux-notes/ssh-or-scp-without-password.html

The refresh meta tag is also set for the page so you don’t have to refresh the page manually to check if cron ran the command.

Remotespy was written in Python and tested on versions 2.4 – 2.6. I don’t know how far back the compatibility actually extends. I like to use it as it allows me to edit the command script without having to reinsert anything into cron. The HTML output is simple enough to be quickly read from mobile devices so I can check on my simulations state whenever I feel like it. You can find it in my Mercurial repository among other scripts I’ve written:

http://bitbucket.org/mbudisic/cl-goodies/src/tip/remotespy

(The repo name was taken from el-goodies collection for Emacs).

Let me know if you have suggestions for improvements. Future versions should probably migrate from “commands” to “subprocess” Python module. Also, I might make remotespy into a daemon to make it cron-independent, but that’s a low-priority for me. This does the job well. If you would like to extend remotespy, please let me know, I’d love to hear from you. If once wasn’t enough, you can read this post at http://mbudisic.wordpress.com/2009/04/03/monitoring-from-afar/ too.

BTW, did you know that the next two suggestions for the company name were InCumulo and FooBar? Whoever can guess the explanation for those two names has a beer