Ola Bini: Programming Language Synchronicity

Moving away from blogspot

2008-08-26T17:14:00.002+02:00

This will be my last post on this blog. For several reasons I like the idea of keeping more in control over my blog and the environment surrounding it. I also have some things I'd like to publish that isn't well suited for the blog format, and moving to another location means that I can keep all my content in the same place. More long term I'm planning on migrating information about my open source projects there to.

But what you need to know is this: This blog ends. A new blog is born. All my old entries have been migrated. The important addresses for the new blog is:

Blog: http://olabini.com/blog
Feed: http://feeds.feedburner.com/olabini

And that's it. The new content will obviously be available at http://olabini.com, but right now this site just redirects to the blog.

The blog is dead, long live the blog.

ThoughtWorks Sweden is available

2008-08-25T13:02:00.001+02:00

I would like to announce that ThoughtWorks Sweden is now in motion. We have business cards and an office. Everyone is returning from their long lovely Swedish summer vacations.

This means that ThoughtWorks Sweden is ready, and available for work. If you or your business have a project you need help with, don't hesitate to contact me (at obini@thoughtworks.com) or Marcus Ahnve (at mahnve@thoughtworks.com).

We are located in Stockholm, but we are open for work anywhere in the Nordic regions.

So what kind of work are we most suited for? Our sweet spot is in delivery and technical advisory regarding Java, Ruby and JRuby. And if you're interested in understanding how our Agile approach can change your company, we can do organizational transformation projects and also coaching and advisory.

Don't hesitate to get in touch!

JtestR 0.3.1 Released

2008-08-17T13:40:00.001+02:00

JtestR allows you to test your Java code with Ruby frameworks.

Homepage: http://jtestr.codehaus.org
Download: http://dist.codehaus.org/jtestr

JtestR 0.3.1 is the current release of the JtestR testing tool. JtestR integrates JRuby with several Ruby frameworks to allow painless testing of Java code, using RSpec, Test/Unit, Expectations, dust and Mocha.

Features:
- Integrates with Ant, Maven and JUnit
- Includes JRuby 1.1, Test/Unit, RSpec, Expectations, dust, Mocha and ActiveSupport
- Customizes Mocha so that mocking of any Java class is possible
- Background testing server for quick startup of tests
- Automatically runs your JUnit and TestNG codebase as part of the build

Getting started: http://jtestr.codehaus.org/Getting+Started

New in the 0.3.1 release is upgrade of JRuby to revision r7479 which includes several new Java Integration features, upgrading of ActiveSupport to 2.1.0, fixing a severe memory leak in the background server and some minor usability features.

New and fixed in this release:
JTESTR-50 Difference in functionality when stubbing a method on a Java class vs a Ruby class using mocha
JTESTR-51 Mocking of classes lacking default constructors results in a NameError
JTESTR-53 Push the JtestR JRuby builds to maven repos
JTESTR-56 Upgrade ActiveSupport
JTESTR-57 Make it possible to use local versions of libraries.
JTESTR-59 No output when no tests found.
JTESTR-60 OutOfMemoryError
JTESTR-61 Documentation improvments - ant test-server
JTESTR-62 Having the jtestr.jar in the base directory doesn't work
JTESTR-63 Update JRuby version

Where is the Net::SSH bug

2008-08-14T11:01:00.003+02:00

Yesterday I spent several hours trying to find the problem with our implementation of OpenSSL Cipher, that caused the Net::SSH gem to fail miserable during negotiation and password verification. After various false leads I finally found the reason for the strange behavior. But I really can't decide if it's a bug, and if it's a bug where the bug is. Is it in Ruby's interface to OpenSSL, or is it in Net::SSH?

No matter what cipher suite you use for SSH, you generally end up using a block cipher, mostly something like CBC. That means an IV (initialization vector) is needed, together with a key. The relevant parts of OpenSSL used is the EVP_CipherInit, EVP_CipherUpdate and EVP_CipherFinal family of methods. Nothing really strange there. The Ruby interface matches these methods quite closely; every time you set a key, or an IV, or some other parameter, the CipherInit method is called with the relevant data. When CipherUpdate is called, the actual enciphering or deciphering starts happening, and CipherFinal takes care of the final block.

At the point EVP_CipherFinal is called, nothing more should be done using the specific Cipher context. Specifically, no more Update operations should be used. The man page has this to say about the Final-methods:

After this function is called the encryption operation is finished and no further calls to EVP_EncryptUpdate() should be made.

Now, what I found was that same documentation is not part of the Ruby interface. And Net::SSH is actually reusing the same Cipher object after final has been called on it. Specifically, it continues the conversation, calling update a few times and then final. The general flow for a specific Cipher object in Net::SSH is basically init->update->update->final->update->update->final.

So what is so bad about this then? Well, the question is really this: what IV will the operations after the first final call be using? The assumption I made is that obviously it will use the original IV set on the object. Something else would seem absurd. But indeed, the IV used is actually the last IV-length bytes of encrypted data returned. Is this an obvious or intended effect at some level? Probably not, since the OpenSSL documentation says you shouldn't do it. The reason it works that way is because the temporary buffer used in the Cipher context isn't cleared out at the end of the call to final.

In contrast, the Java Cipher object will call reset() as part of the call to doFinal(). Where reset() will actually reset the internal buffers to use the original IV. So the solution is simple for encryption. Just save away 8 or 16 bytes of the last generated crypto text and set that manually as the IV after the call to doFinal. And what about decryption? Well, here the IV needs to be the last crypto text sent in for deciphering, not the result of the last operation.

So Net::SSH seems to work fine with JRuby now. I'm about to release a new version of JRuby-OpenSSL including these and many other things.

But the question remains. Is it a bug? If it is, is it in the Ruby OpenSSL integration, or in the Net::SSH usages of Ciphers? If it's in the Net::SSH code, why does it actually work correctly when communicating with an SSH server? Or is this behavior of using the last crypto text as IV something documented in the SSH spec?

Enlightenment would be welcome.

Security vs Convenience

2008-08-10T16:16:00.003+02:00

I really like Cryptogram and read every issue. It's interesting stuff that talks a lot about how our minds work in conjunction with risk and reward. Today I had a typical example of how security versus convenience is a part of day to day life.

I had just checked out from my hotel, and wanted to store all my luggage (including my laptop bag) in the hotel until my ride out of town arrived. I asked about this, and it was fine, they had a room for this. The person in the reception pointed me to an open room and said it was open and that I could put my stuff there. Feeling uneasy I asked how secure it was, and she answered that the door was usually locked. OK, I said, but can someone take any bag from inside of there? Yes, was the answer. I decided I couldn't store my stuff there. Even if the risk was small, losing my work laptop would be way to bad to risk. But I also decided I couldn't drag my two heavy bags and laptop bag around.

I ended up putting the large bags in the room, and just taking my laptop bag around. I didn't have as much to lose with the large bags, and the price of inconvenience in taking them along was just to high. These considerations go into everything we do in programming and systems engineering. A totally secure system is generally quite inconvenient to use, while an insecure system can be very pleasant to use. The trick is to get the balance right, I guess.

JtestR doesn't start up.

2008-08-10T16:13:00.003+02:00

Justin Smestad uncovered an issue with JtestR that can cause some quite unintuitive output, and be hard to debug. Some info can be found here: http://www.evalcode.com/2008/08/jtestr-woes/ and here: http://jira.codehaus.org/browse/JTESTR-62. The issue has been fixed on trunk, but hasn't been released yet. The issue is very simple - just make sure you don't have the jtestr.jar file in the base directory where your project lives (this is usually the same place as the build.xml file). There are two ways to achieve this, either move the file into a directory or rename the file to something else.

Java and mocking

2008-07-04T22:08:00.002+02:00

I've just spent my first three days on a project in Leeds. It's a pretty common Java project, RESTful services and some MVC screens. We have been using Mockito for testing which is a first for me. My immediate impression is quite good. It's a nice tool and it allows some very clean testing of stuff that generally becomes quite messy. One of the things I like is how it uses generics and the static typing of Java to make it really easy to make mocks that are actually type checked; like this for example:

Iterator iter = mock(Iterator.class);
stub(iter.hasNext()).toReturn(false);

// Call stuff that starts interaction

verify(iter).hasNext();

These are generally the only things you need to stub stuff out and verify that it was called. The things you don't care about you don't verify. This is pretty good for being Java, but there are some problems with it too. One of the first things I noticed I don't like is that interactions that isn't verified can't be disallowed in an easy way. Optimally this would happen at the creation of the mock, instead of actually calling the verifyNoMoreInteractions() afterwards instead. It's way to easy to forget. Another problem that quite often comes up is that you want to mock out or stub some methods but retain the original behavior of others. This doesn't seem possible, and the alternative is to manually create a new subclass for this. Annoying.

Contrast this to testing the same interaction with Mocha, using JtestR, the difference isn't that much, but there is some missing cruft:

iter = mock(Iterator)
iter.expects(:hasNext).returns(false)

# Call stuff that starts interaction

Ruby makes the checking of interactions happen automatically afterwards, and so you don't have any types you don't need to care about most stuff the way you do in Java. This also shows a few of the inconsistencies in Mockito, that is necessary because of the type system. For example, with the verify method you send the mock as argument and the return value of the verify-method is what you call the actual method on, to verify that it's actually called. Verify is a generic method that returns the same type as the argument you give to it. But this doesn't work for the stub method. Since it needs to return a value that you can call toReturn on, that means it can't actually return the type of the mock, which in turn means that you need to call the method to stub before the actual stub call happens. This dichotomy gets me every time since it's a core inconsistency in the way the library works.

Contrast that to how a Mockito like library might look for the same interaction:

iter = mock(Iterator)
stub(iter).hasNext.toReturn(false)

# Do stuff

verify(iter).hasNext

The lack of typing makes it possible to create a cleaner, more readable API. Of course, these interactions are all based on how the Java code looked. You could quite easily imagine a more free form DSL for mocking that is easier to read and write.

Conclusion? Mockito is nice, but Ruby mocking is definitely nicer. I'm wondering why the current mocking approaches doesn't use the method call way of defining expectations and stubs though, since these are much easier to work with in Ruby.

Also, it was kinda annoying to upgrade from Mockito 1.3 to 1.4 and see half our tests starting to fail for unknown reasons. Upgrade cancelled.

JtestR, RubyGems, and external code

2008-06-27T16:06:00.001+02:00

One question I've gotten a few times now that people are starting to use JtestR, is how to make it work with external libraries. This is actually two different questions, masquerading as one. The first one regard the libraries that are already included with JtestR, such as JRuby, RSpec or ActiveSupport. There is an open bug in JIRA for this, called JTESTR-57, but the reason I've been a bit hesitant to add this functionality until now, is because JtestR actually does some pretty hairy things in places. Especially the JRuby integration does ClassLoader magic that can potentially be quite version dependent. The RSpec and Mocha integration is the same. I don't actually modify these libraries, but the code using them is a bit brittle at the moment. I've worked on fixing this by providing patches to the framework maintainers to include the hook functionality I need. This has worked with great success for both Expectations and RSpec.

That said, I will provide something that allows you to use local versions of these libraries, at your own risk. It will probably be part of 0.4, and if you're interested JTESTR-57 is the one to follow.

The second problem is a bit more complicated. You will have seen this problem if you try to do "require 'rubygems'". JtestR does not include RubyGems. There are both tecnnical and non-technical reasons for this. Simply, the technical problem is that RubyGems is coded in such a way that it doesn't interact well with loading things from JAR-packaged files. That means I can't distribute the full JtestR in one JAR-file if I wanted RubyGems, and that's just unacceptable. I need to be able to bundle everything in a way that makes it easy to use.

The non-technical reason is a bit more subtle. If RubyGems can be used in your tests, it encourages locally installed gems. It's a bit less pain to do it that way initially, but remember that as soon as you check the tests in to version control (you are using version control, right?) it will break in unexpected ways if other persons using the code doesn't have the same gems installed, with the same versions.

Luckily, it's quite simple to work provide functionality to JtestR, even if no gems are used. The first step is to create a directory that contains all the third party code. I will call it test_lib and place it in the root of the project. After you have done that you must first unpack your gems:

mkdir test_lib
cd test_lib
jruby -S gem unpack activerecord

When you have the gems you want unpacked in this directory, you can add something like this to your jtestr_config.rb:

Dir["test_lib/*/lib"].each do |dir|
  $LOAD_PATH << dir
end

And finally you can load the libraries you need:

require 'active_record'

TheServerSide Java Symposium Europe is over

2008-06-21T12:20:00.002+02:00

Well, I'm home from Prague, from another edition of TheServerSide Java Symposium. This year was definitely a few notches up from last year in Barcelona in my opinion. And being in beautiful Prague didn't really cause any trouble either. =)

I landed on Tuesday, and worked quite heavily on my talks. Due to the ThoughtWorks AwayDay I was really out in the last second with my two slide decks. But I still got to see parts of the city in the evening. Very nice.

I managed to sleep over the opening keynote, but dragged myself down to the main room to watch the session on Spring Dynamic Modules. This ended up being more about OSGi style things than really dynamic things, so I felt a bit cheated, and kept on working on my slides instead. Before lunch I sat in on Alex Popescu's talk about scripting databases with Groovy. Over all a very good overview of the database landscape from a Groovy point of view, including just using the language to make the JDBC API's more flexible, building a builder style DSL for working with SQL, to the full blown GORM framework. All in all quite nice. But the funniest part was definitely peoples reaction to the SQL DSL, where most in the room preferred the real SQL to the Groovy version.

After lunch I had planned to see the session that compared different dependency injection frameworks, but the speaker never showed up, so I found myself listening to info about JSR-275, that provides support for units in a monetary system. Quite useful if you're working in that domain, but at the same time it felt like this would look so much cleaner in Ruby. Of course, that's how I react to most Java code nowadays.

Holly Cummins gave a very good talk about Java Performance Tooling. Of course it was coming with a slight IBM slant, but that's fair. The tools built around their JVM is actually really good for identifying several kinds of performance problems. So I'm actually in a mind to try JRuby on the IBM JVM and see if we can glean some more interesting information from that.

Geert gave his Terracotta talk about JVM clustering, and it's really interesting if you haven't seen it before. In this case I took the opportunity to listen while working on my slides.

And that was the end of day one.

Day two I was a good boy and was actually up in time for the keynote. This might have something to do with the fact that it was Neal Ford giving it, and he talked about Language-Oriented Programming. This is one of my favorite topics, and I'd only seen his slides to this talk before, not heard him give it. If you've been following the discussions about polyglot programming, the content made lots of sense. If you don't believe in polyglot programming, you might have been convinced.

After the keynote, it was time for breakfast, so I didn't see the sessions in that slot. After breakfast I sat in on Guillaume's Groovy in the Enterprise: Case Studies. While the presentation were good, he spent more than half of it just giving an introduction to Groovy. I'm not one to throw stones in glass houses, though, so I have to admit that this is something I can be found guilty of too. I'm trying to improve on this though. It makes a disservice to the audience - if they have to sit through the same kind of intro they might already have seen to get to the actual meat. That's one of the reasons I tried to minimize introductionary material in my testing session.

It was also in this session that a slide with the words "Groovy is the fastest dynamic language on the JVM" showed up. That's based only on the Alioth benchmarks, and it doesn't actually matter if it's true or not. It's a disservice to the audience. Especially in this case where even if Groovy actually on average is faster than JRuby, we are talking maximum 1-2% in average. The speed differences aren't really why you would be interested in using such a language, and in my opinion Groovy has got lots of other interesting features you can use to sell and market it. In summary, it felt a bit unnecessary.

Directly after that session, me, Ted Neward, and Guillaume was featured in a panel on the languages of the next generation. Eugene Ciurana who was supposed to moderate didn't really show up, so John Davies and Kirk Pepperdine had to jump in instead. It ended up being quite fun, but no real heat in the discussion. In something like this, I think it would be useful to have someone with different views to spice it up. Me, Ted and Guillaume just agree about these things way too much. But we got some nice Czeckian vodka. That was good. =)

After lunch I spent more time prepping my talk, and then it was finally time to give it. This was the JRuby on Rails introduction, and it ended up being quite nice. I had a good turn-up, and interestingly enough, many in the audience had actually tried Ruby already.

After my session was up, I could relax, so I went to Kirks talk about Concurrency and High Performance, which included many things to think about while working on the performance of an enterprise scale application. Very useful material.

Finally, at the end of the day it was time for the fireside chats, which is basically another word for BOFs. I sat in on the Zero Turnaround in Java Development session, which ended up not being as much discussion as I had expected, and more talking about the three principals different approachs (RIFE, Grails and JavaRebel).

The Fireside Performance Clinic was good fun, and some useful material. In particular, knowing whether JRuby startup time is CPU or IO bound is something I have never thought about, and might yield some interesting insights.

Day three felt a bit slower, as the last day usually does. The first session for me was Ted's Scala talk. I've seen it a few times before, but the most interesting part is actually the audience questions. As usual I wasn't disappointed. And Ted did his regular thing and weaved me into the examples. One of the more funny bits were when he was explaining the differences between var and val in Scala, and he decided that it might be good to be able to switch my surname. Then came the killer, where he said something like this: "well, and you might want to change the surname of Ola. Since Ola was just married, congratulations by the way, and he's from Sweden where the husband generally takes the surname of the wife, so we need to change his surname". At that point I had a hard time keeping it together.

The session on what's new and exciting in JPA 2 ended up not exciting me at all, so frankly I don't remember anything at all about that. I have vague blurry images of many at-signs.

Shashank Tiwari gave a presentation on how to choose your web framework, and this generated some discussion that were quite interesting. At this point I still wasn't finished with the examples for my testing session though, so I had to work on them. And I finally managed to finish it. Because lo, at that time I did the presentation on testing with JRuby. I spent some time on the different Ruby testing frameworks, first showing off how you can test Ruby code with them. Then I switched the model to a Java class, and used basically the same tests again. The cutest example is probably my story about a Stack. Not a literary master piece, but it's still prose.

People seemed to like the session and get something out of it, and that feels great since this was the first time I showed JtestR to a larger group of people. My mocking domain consisting of Primates, Food and Factories also seemed to go home. I got the expected laughs at the source code line where a Chimpanzee tries to eat Tuna and "throw new Up();".

Typesafe Embedded Java DSLs basically talked about how you can use the standard generic builder patterns to create DSLs that your IDE can help you quite much with. Sadly, my computer decided to give me a heart attach during this presentation, so I had to run out and give it CPR instead of sitting in on the rest of the session.

And that was TSSJS-E. For me, the first day was quite weak, but the content of the other two days were definitely extremely good. I can recommend it to anyone next year.

Testing programming language implementations

2008-06-18T15:23:00.000+02:00

While writing the post yesterday about testing regular expressions, I realized that this problem is not really specific to regular expressions. I got a very good comment noting that testing any place that uses some kind of DSL is definitely prudent. SQL is another example.

But these examples are both about actually testing the usage of them, and the problem becomes that you have two languages, but you're mostly only testing the code written in the outer language. This is due to several reasons. One of the most obvious ones is that our tools really doesn't make it that easy to do.

Thinking about these issues made me start thinking about how we generally test languages. Having worked on several language implementations and worked on both new languages, and implementations of existing languages, I've come to the conclusion that the whole area of testing languages are actually quite complicated, and also there are no real best practices for doing it.

First, there is a problem of terminology. Many implementations of languages that are really executable specifications of how the language should work. What's the difference? Well, testing the language according to such a spec, you are really only doing functional, black-box testing. I've looked at several of the open source language implementations, and I don't really see much usage of anything else than such language spec tests. This means basically that some parts of the implementation can be implemented wrongly, and by some freak chance it still works correctly in all the cases you have tests for, but it might fail in other ways.

Unit tests for the actual implementation would help with this - it helps since you will be doing TDD on the unit level, it helps because you make a conscious decision about the implementation and what it should be doing in these cases. It still doesn't make everything clear cut and simple, but it absolutely would help. So why don't most implementations do unit testing of the internals? I don't really know. Maybe it's because implementations can be extremely complicated. But that should be a reason for testing more, not testing less. One reason I feel a bit about is that it makes larger changes quite hard. Large refactorings are one of the ways JRuby has used to get incredible performance improvements and new subsystems, but unit tests can sometimes act as inertia for these.

I'm totally disregarding the academic approaches here. Yeah, in soem cases for simple languages, you can actually prove that it does what you want it to do, and for small enough implementations using a suitable language, you can actually prove the same things about the implementation. The problem is that this approach doesn't scale.

And since a language almost always is turing complete, that means that you can't exhaustively test it. There is no way of testing all permutations - either manually or automatically. So what should a language spec do? The first thing that many languages do are to specify that whole areas of functionality result in undefined behavior. That makes it easier. But the real problems exist when you start combining different features which can interact in different ways.

At the end of the day, I have no idea how to actually do this well. I would like to know though - how should I test the implementation, and how should I write an executable language specification? And these questions doesn't even touch on the question of testing the core libraries. Many of the some problems apply, but it gets even more complicated.

Local things in Emacs

2008-06-18T12:28:00.003+02:00

This is just a small note, since this have bugged me for a while. Basically, I have lots of extra key bindings running around in my Emacs configuration. Now, I use local-set-key for many of these. The problem is I hadn't actually read the documentation for local-set-key enough.

One example that annoyed me was this: I had some local key bindings for RSpec buffers, that differed from the regular Ruby buffers. My RSpec minor mode still uses the ruby-mode-map though. My assumption was that local-set-key did things exactly as all other things with "local" in their name, namely doing a buffer local modification only. I finally found out that this wasn't the case. Instead, when the RSpec minor mode was loaded for the first time, it ended up modifying the ruby-mode-map with its key bindings, which were then visible for all other Ruby buffers. Ouch.

So, if you use local-set-key, make sure you actually want to set that key in the current mode map, instead of only for the current buffer.

As far as I know, there is no way to set a real buffer local key binding without some acrobatics that unsets and resets the keys manually. I ended up solving my problem with the RSpec minor mode to having it clone the Ruby mode map and have its own mode map. Not an ideal solution, but it works for now.

Testing Regular Expressions

2008-06-17T15:44:00.002+02:00

Something has been worrying me a bit lately. Being test infected and all, and working for ThoughtWorks, where testing is part of the life blood, I think more and more about these issues. And one thing I've started noticing is that regular expressions seems to be a total blind spot in many cases. I first started thinking about it when I changed a quite complicated regular expression in RSpec. Now RSpec has coverage tests as part of their build, and if the test coverage is less than a 100%, the build will fail. Now, since I had changed something to add new functionality, but hadn't added any tests for it, I instinctively assumed that it would be caught be the coverage tool.

Guess what? It wasn't. Of course, if I had changed the regexp to do something that the surrounding code couldn't support, one of the tests for surrounding lines of code would have caught it, but I got no mention from the coverage tool that I needed more tests to fully handle the regular expressions. This is logical if you think about it. There is no way that a coverage tool could find all the regular expressions in your source code, and then make sure that all branches and alternatives of that particular regular expression was exercised. So that means that the coverage tool doesn't do anything with them at all.

OK, I can live with that, but it's still one of those points that would be very good to keep in mind. Every time you write a regular expression in your code, you need to take special care to actually exercise that part of the code with many inputs. What is many in this case? That's another part of the problem - it depends on the regular expression. It depends on how complicated it is, how long it is, how many special operators are used, and so on. There is no real way around it. To test a regular expression, you really need to understand how they work. The corollary is obvious - to use a regular expression in your code, you need to know how to test it. Conclusion - you need to understand regular expressions.

In many code bases I haven't seen any tests for regular expressions at all. In most cases these have been crafted by writing them outside the code, testing them by hand, and then putting them in the code. This is brittle to say the least. In the cases where there are tests, it's much more common that they only test positives, and not negatives. And I've seldom heard of code bases with enough tests for regular expressions. One of the problems is that in a language like Ruby, they are so easy to use, so you stick them in all over the place. A standard refactoring could help here, by extracting all literal regular expressions to constants. But then the problem becomes another - as soon as you use regular expressions to extract values from a string, it's a pain to not have the regular expression at the same place as the extracted groups are used. Example:

PhoneRegexp = /(\d{3})-?(\d{4})-?(\d{4})/
# 200 lines of code
if phone_number =~ PhoneRegexp
puts "phone number is: #$1-#$2-#$3"
end

If the regular expression had been at the same place as the usage of the $1, $2 and $3 it would have been easy to tie them to the parts of the string. In this case it would be easy anyway, but in more complicated cases it's more complicated. The solution to this is easy - the dollar numbers are evil: don't use them. Instead use an idiom like this:

area, number, extension = PhoneRegexp.match(phone_number).captures

In Ruby 1.9 you will be able to use named captures, and that will make it even easier to make readable usage of the extracted parts of a string. But fact is, the difference between the usage point and the definition point can still cause trouble. A way of getting around this would be to take any complicated regular expression and putting it inside of a specific class for only that purpose. The class would then encapsulate the usage, and would also allow you to test the regular expression more or less in isolation. In the example above, maybe creating a PhoneNumberParser would be a good idea.

At the end of the day, regular expressions are an extremely complicated feature, and in general we don't test the usage of them enough. So you should start. Begin by first creating both positive and negative tests for them. Figure out the boundaries, and see where they can go wrong. Know regular expressions well enough to know what happens in these strange circumstances. Think about unicode characters. Think about whitespace. Think about greedy and lazy matching. As an example of something that took a long time to cause trouble; what's wrong with this regexp that tries to discern if a string is a select statement or not?

/^\s*\(*\s*SELECT\W+/i

And this example actually covers most of the ground, already. It checks case insensitive. It checks for white space before any optional parenthesis, and for any white space after. It makes sure that the word SELECT isn't continued by checking for at least one non word character. So what's wrong with it? Well... It's the caret. Imagine if we had a string like this:

"INSERT INTO foo(a,b,c)\nSELECT * FROM bar"

The regular expression will in fact match this, even though it's not a select statement. Why? Well, it just so happens that the caret matches the beginning of lines, not the beginning of strings. The dollar sign works the same way, matching the end of lines. How do you solve it? Change the caret to \A and the dollar sign to \Z and it will work as expected. A similar problem can show up with the "." to match any character. Depending on which language you are using, the dot might or might not match a newline. Always make sure you know which one you want, and what you don't want.

Finally, these are just some thoughts I had while writing it. There is much more advice to give, but it can be condensed to this: understand regular expressions, and test them. The dot isn't as simple as it seem. Regular expressions are a full blown language, even though it's not turing complete (in most implementations). That means that you can't test it completely, in the general case. This doesn't mean you shouldn't try to cover all eventualities.

How are you testing your regular expressions? How much?

Applications and libraries

2008-06-17T15:43:00.000+02:00

In a recent discussion around one of Steve Yegge's blog post, an incidental remark was that it's OK that a language makes it harder for a library creator than for an application developer. This point was made by David Pollak and Martin Odersky in relation to some of the complications that you need to handle when creating a Scala library that you can intuitively use without a full understanding of the Scala type system. Make no mistake, I have lots of respect for both Martin and David, it's just that in this case I think it's actually a quite damaging assumption to make. And they are not the only ones who reason like that either. Joshua Bloch's book Effective Java includes this assumption too, in many places.

So what's wrong with it then? Isn't there a difference between developing an application and a library. Yes, there is a difference, but it's definitely not as large as people make it out to be. And even more importantly: it _shouldn't_ be that much of a difference. The argument from David was that when creating a library in Scala, he needs to focus and work with quite complicated parts of the type system so that the consumer gets a nice API to use the library through. This process is much harder than just using the library would be.

Effective Java contains much good advice, but most of them are from the perspective of someone who creates libraries for a living, and there are a few places where Josh explicitly says that his advice isn't necessarily applicable when writing an application, since he doesn't have that point of view.

Let's take a look at a fundamental question then. What is actually a library, and what is an application? In my opinion, a library is a module providing functionality of some kind, restricted to a specific domain. This can be a horizontal or vertical domain, that doesn't matter, but it's usually something that is usable in more than one circumstance. It's not uncommon that libraries use other libraries to implements its functionality. An application is usually a collection of libraries that provide functionality to an end user. That end user can be either a person, a program or another computer - that doesn't matter. But wait, isn't libraries usually also created to provide functionality to other pieces of code? And even though libraries have a tendency to contain more specific code, and less usage of other libraries, the line is extremely fuzzy.

The way most applications seems to be built now, most of the work is done to collect libraries, provide the missing functionality and glue them together in some way. But that doesn't mean that the code you write in the application won't be used as a library by another consumer. In fact, it's more and more common to try to reuse as much as possible, and especially when you extend an existing application, it's extremely important that you can consume the existing functionality in a sane way.

So why make the distinction? Doing that seems to me to be an excuse for writing bad code if it's in an application. Why won't we as programmers admit that we don't know if someone else will need to consume the code later, and write the best code we can, including creating usable ad well thought out public APIs? Yes, the cost and time will be higher, but that's true for writing tests too. I don't see any value in arguing that libraries should be designed with more care than application code. In fact, I think that attitude is actively detrimental to the industry. And adding a language feature to a language that is complicated, and then arguing that only "library developers" will need to understand it is definitely not the right way to go. A responsible developer using a language needs to understand how that language works. Otherwise that developer will sooner or later cause a great mess. It's just a matter of time.

JtestR 0.3 Released

2008-06-15T15:37:00.000+02:00

JtestR allows you to test your Java code with Ruby frameworks.

Homepage: http://jtestr.codehaus.org
Download: http://dist.codehaus.org/jtestr

JtestR 0.3 is the current release of the JtestR testing tool. JtestR integrates JRuby with several Ruby frameworks to allow painless testing of Java code, using RSpec, Test/Unit, Expectations, dust and Mocha.

Features:
- Integrates with Ant, Maven and JUnit
- Includes JRuby 1.1, Test/Unit, RSpec, Expectations, dust, Mocha and ActiveSupport
- Customizes Mocha so that mocking of any Java class is possible
- Background testing server for quick startup of tests
- Automatically runs your JUnit and TestNG codebase as part of the build

Getting started: http://jtestr.codehaus.org/Getting+Started

The 0.3 release has focused on stabilizing Maven support, and adding new capabilities for JUnit integration.

New and fixed in this release:
JTESTR-47 Maven with subprojects should work intuitively
JTESTR-42 Maven dependencies should be automatically picked up by the test run
JTESTR-41 Driver jtestr from junit
JTESTR-37 Can't expect a specific Java exception correctly
JTESTR-36 IDE integration, possibility to run single tests
JTESTR-35 Support XML output of test reports

Team:
Ola Bini - ola.bini@gmail.com
Anda Abramovici - anda.abramovici@gmail.com

Ruby can't be good since I won't bother learning it...

2008-06-10T23:02:00.002+02:00

Best quote this whole day, found in http://www.codinghorror.com/blog/archives/001131.html#comments.

If Ruby offered something new I would have learned it fine tbh... its just difficult enough to not be able to "pick up and run with" like almost everything else out there... but honestly, it wouldn't let me do anything I can't already do.

My brain almost exploded reading that.

Git completion in tcsh

2008-06-04T13:49:00.002+02:00

So I've been a bit envious at the lovely git completion bash users have - but obviously I can't just switch to bash. Anyone who is in the same kind of situation might like the fact that I've started a project to provide this functionality for tcsh.

The first thing you need to do is download the source for tcsh 6.15, and apply the patch you can find here: http://bugs.gw.com/bug_view_advanced_page.php?bug_id=60. Without it it won't work. Compile and install the new tcsh version. The next step is to check out the project from github, at http://github.com/olabini/git_complete_tcsh. Make sure that git_complete is executable and on your path. You need to have Ruby installed for this, btw.

The final step is to modify your .cshrc to add something like this: complete git{,-*} 'p/*/`git_complete`/'.

Now git completion should work, although most of the commands aren't implemented yet. I'll get to them in time. The whole project is a port of the bash completion for git

Fractal Programming

2008-06-03T23:26:00.004+02:00

This is a continuation of my previous posts describing layers of code written in different programming languages. I have thought about the things involved for a while, and had several discussions with people about it. There were some parts that I didn't describe as well as I thought in my posts, and I will try to do better in this one.

The core of these ideas are based on polyglot programming, the thinking that you should use several different languages in a project, based on which languages are better suited for different parts of it. Another term for this concept is Language-oriented programming. So how do you organize a polyglot system? The most natural way for me is to divide it into layers. In most cases you will find that different categories of languages will be better suited to different layers of the application.

In my original post I identified three layers that can be used to organize polyglot systems. These layers are the stable layer, the dynamic layer, and the domain layer. There are several reasons for organizing them this way, and I'll take a harder look at each of the layers further down. But first let me note that these layers are usually depicted in the form of a pyramid, with the stable layer being that base. That is definitely not how I think about it. In fact, I see it as an inverted pyramid, where the stable layer is the tip of the pyramid, providing the base. The Dynamic layer is the middle part. The domain layer should be the largest part and will very often include more than one dynamic language. So in my mind I represent the different domain languages as smaller pyramids standing upside down, covering the base area. Now, the dynamic layer can also be divided into smaller parts like this, based on language or functionality. This is a bounded fractal representation, which is the reason for the title of this blog post.

This diagram shows how I think about it:Of course, the smaller pyramids can be all the same language and system, or several different ones. It all depends on the application or system you are building. So you can for example use a combination of Ruby, Java and external or internal DSLs:Or you could use Clojure, Scala and JavaScript:
Or any other combination you can imagine. As long as the combination is what's best suited for the problem.

Let's take a look at the definitions of the different layers. There have been some discussion about the names I've chosen for them, so let me describe a little more what the responsibility of each part is, and why it's in that part of the system.

The Domain Layer
This layer is the simplest. This is where all the actual domain rules are defined. In general that means one or more domain specific languages. It doesn't really matter if they are internal or external. This model see them as the same layer. This part of the system is what needs to be malleable enough that it should be possible to change rules in production, allow domain experts to do things with it, or just plain a very complicated configuration. The languages used in this layer are mostly external DSLs, but can also include extremely DSL-friendly languages like Ruby, Python or Groovy.

The Dynamic Layer
Neal Ford argues that this layer isn't so uch about dynamic, as it is about essence. That was never my intention. The problem is that even if you take a language like Scala, which is usually classified as an essential language, Scala requires compilation. To me, compilation is ceremony, which means that it's one extra thing you don't want to care about when writing most of your application code. That's why this layer needs to be dynamic. This is where languages like Ruby, Groovy, Python, JavaScript, Clojure and others live.

The Stable Layer
I view the stable layer as the core set of axioms, the hard kernel or the thin foundation that you can build the rest of your system in. There is definitely advantages to having this layer be written in an expressive language, but performance and static type checking is most interesting here. There is always a tradeof in giving up static typing, and the point of having this layer is to make that tradeof smaller. The dynamic layer runs on top of the stable layer, utilizing resources and services provided.

Another important feature of this layer is that this is where all interfaces are defined. By interfaces I mean external API's. They need to be hard for other clients to be able to trust them. But the implementations for them lives in the dynamic layer, not in the stable. By doing it this way you can take advantage of static type information for your API's while still retaining full flexbility in implementation of them. Languages in the stable layer can be Java, Scala or F#. It should be fairly small compared to the rest of the application, and just provide the base necessary services needed for everything to function.

The most common objection I hear from people about this strategy is the same as for the general polyglot programming idea: if we have a proliferation of languages in a system, it will be harder to find skilled programmers who can work with it.

This objection is true to a degree, but there are several ways around it. First, I have to say that I don't believe this is such a big problem as many others think. Programmers nowadays depend on their tool chains quite heavily, all of them including many advanced features that takes lots of time to learn. But most programmers doesn't even view their languages as tools. In my mind, the programming language is the most important tool. And once we start using better languages for systems, many of the things we need other tools for will disappear or become less of a problem.

I tend to believe that programming languages are quite easy to learn as soon as you understand the fundamental building blocks of programming languages. And if you don't have a fair understanding of these building blocks, I would say that you probably aren't using your current language as well as you should either. I see this as part of being responsible programmers.

I also believe quite strongly that if we used better languages for our code, many code bases would be smaller, easier to understand, easier to maintain and cost less - which means you could afford to find a more skilled programmer to do the work for you. This would mean that both parties win - the programmer gets more interesting work and better code, while the client gets more worth for his money in less time.

RailsConf 2008

2008-06-03T17:38:00.003+02:00

I've landed, gotten mostly back in the right timezone without too many incidents (except running through SFO to board very badly scheduled connection).

After allowing the impressions from the last 6-7 days to sink in a little, it's time to summarize RailsConf. I'll go through the sessions I saw and then do some concluding remarks.

The first day was tutorials. I had a good time in Neal Fords and Pat Farleys tutorial on Metaprogramming. I can't say I learned much from the sessions, but it was very good content, extremely well presented, and I got the impression that many in the room learned lots of crucial things. The kind of knowledge about internals you get from a talk like this allows you to understand how metaprogramming in Ruby actually works, which makes it easier to achieve the effects you want.

After that I sat around hacking in the Community Code Drive for the rest of the day, with lots of other people. I wasn't involved in gitjour (which by the way is incredibly cool), but I did manage to find a memory leak in iTerms Bonjour handling due to gitjour. Neat. Me and David Chelimsky paired on getting support for multiline plain text story arguments into RSpec, and by the end of the afternoon it was in.

Finally, we headed out to the JRuby hackfest, which ended up being over full with people. That's a good problem to have. We had a great time, hacking on different things, helping people to get started and debugging various problems. All in all it was a very productive day.

I began the Friday with Joel Spolsky's keynote. In contrast to many other people I didn't like it. There wasn't really any content at all, just some humorous content and lots of jokes about naked women. I expect something a bit more profound for the first keynote of the conference, since they have a tendency to actually set the standard for the rest of the days.

After the keynote, John Lam showed off IronRuby running a few simple Rails requests. This is a great achievement, and I'm very impressed with their results. I have argued that IronRuby would probably never reach this point, and I'm very happy to admit I was wrong and offer my apologies to John Lam and the IronRuby team. That said, the fact that IronRuby runs a few different Rails requests is not the same thing as saying that IronRuby runs Rails. My personal definition of running Rails is more about having the Rails test suite run at a high percentage of success (something like 96-98% would be good enough for almost all Rails apps to work, provided they are the right 98%). (ED: Evan Phoenix just told me that MRI doesn't run the Rails test suite totally clean either, because of the way the Rails development process works. So a 100% is probably not a good measure of Rails compatibility.) I assume that this is going to be the next goal for the IronRuby team, and I wish them good luck.

I saw the Hosting talk after that, but I have to admit I was wrapped up in a seriously annoying JRuby bug at the moment so I didn't really pay attention.

The DataMapper talk was very full and gave a good overview of why DataMapper might be a better choice than AR in many cases. The presentation style could possibly have been a bit less dry, but the content was definitely delicious.

If the next two days were the JRuby days, the Friday was the day for all other alternative implementations. I sat in on the Rubinius talk by Evan Phoenix and friends, and then the much talked about MagLev presentation.

I first want to congratulate Rubinius on running several different Rails requests. It's very cool and a great milestone. The same caveats as for IronRuby applies of course. But wow, the debugging features is awesome. First class meta objects are extremely powerful, and will provide many capabilities to the platform. The presentation was also extremely entertaining. One of the best presentations for the sheer fun everyone seemed to have. Props to Evan, Brian and Wilson for this.

So. The MagLev talk. First, there seems to be some misunderstandings about what MagLev actually is. It is not a hosting service. Gemstone might offer a hosting service around MagLev in the future, but that's not what is going on here. MagLev is a new virtual machine for Ruby, based on Gemstone/S. Basing it on a Smalltalk machine makes it very easy for Gemstone to implement a large subset of Ruby and having it running cleanly and with good performance. Exactly how much has been implemented at this point is not really clear, since no major applications run, and the RubySpecs have not been used on it yet. I assume that the implementation doesn't handle enough Ruby features yet to be able to run the mspec runner and other important machinery.

Was this presentation important? Yeah, sure. To a degree. It was a cool presentation, whetting peoples appetite by showing something that might some day become a real Ruby platform with built in support for an incredible OODB. But it's still early days.

The Saturday began with Jeremy's keynote. He talked about the new things in Rails 2.1 and also showed the same app running in Ruby 1.8, 1.9, Rubinius and JRuby. Very cool.

I ended up in Nathaniel Talbotts 23 Hacks session which was fun. Good stuff.

After that the JRuby day began in earnest with Nick's talk about deploying JRuby on Rails. This was mostly the same talk as given at JavaOne, but more geared towards Ruby programmers. Useful information.

Dan Manges and Zak Tamsen gave an extremely useful talk about how to test Rails applications correctly. Very good material. Exactly the strong kind of deep technical knowledge, gained by experience, that people go to conferences to get.

My talk about JRuby on Rails was generally well received. I had a fun time, and of course I managed to run out of time as usual. I wonder why I'm always afraid of running out of material. That has never happened when I'm talking bout JRuby.

The final technical session of the day ended up being a walk-around to all the different presentations going on and taking a peek, and then ending up hacking in the speakers room.

The evening keynote was by Kent Beck, and as usual he is fantastic to listen to.

The Sunday started with the CS nerds anonymous session, held by Evan Phoenix. It ended up being a kind of lightning talk session, and had some nice points.

After that Ezra gave his talk - that had nothing to do with the session title. He presented Vertebra, which is a cloud computing control system, based on XMPP, Erlang and the actors model. Very cool stuff, although it might not be that useful for people who aren't in charge of a quite large number of computers. But if you have your own botnet, this might be the best way to control them all. =)

The final session of the day was the JRuby Q&A session, which basically flew by. The first ten minutes went in normal time, and then suddenly the session was over. I think we had good attendance, and the right level of questions. You can see all the points covered in Nicks blog, here.

And then it was over.

So, what was good? The technical level was definitely deeper and more rooted in experience. I have to say that this was probably the best Ruby conference I've been to, based on the depth and level of the presentations. Kudos to the scheduling people.

And what was bad? A little bit too much hype about MagLev, and everyone's tendency to use dark colors on black backgrounds in their presentations. Hey, they look good on your computer screen, but it's really not readable!

Ruby doesn't have meta classes

2008-05-29T19:00:00.004+02:00

OK. It's time to get rid of this terminology problem. Ruby does NOT have meta classes. You can define them yourself, but it's not the same thing as what is commonly called the meta class. That is more correctly called the eigen class. The singleton class is also better than meta class, but eigen class is definitely the most correct term.

So what is a meta class then? Well, it's a class that defines the behavior of other classes. You can define meta classes in Ruby if you want too by defining a subclass of Class. Those classes would be metaclasses.

Edit: Of course, if you actually try to define a subclass of Class you will find that Ruby doesn't allow you to do that, which means that you don't have any meta classes in Ruby. Period.

Ruby closures addendum - yield

2008-05-29T18:13:00.002+02:00

This should probably have been part of one of my posts on closures or defining methods, but I'm just going to write this separately, because it's a very common mistake.

So, say that I want to have a class, and I want to send a block when creating the instance of this class, and then be able to invoke that block later, by calling a method. The idiomatic way of doing this would be to use the ampersand and save away the block in an instance variable. But maybe you don't want to do this for some reason. An alternative would be to create a new singleton method that yields to the block. A first implementation might look like this:

class DoSomething
def initialize
  def self.call
    yield
  end
end
end

d = DoSomething.new do
puts "hello world"
end

d.call
d.call

But this code will not work. Why not? Because as I mentioned in my post about defining methods, "def" will never create a closure. Why is this important? Well, because the current block is actually part of the closure. The yield keyword will use the current frames block, and if the method is not defined as a closure, the block invoked by the yield keyword will actually be the block sent to the "call" method. Since we don't provide a block to "call", things will fail.

To fix this is quite simple. Use define_method instead:

class DoSomething
 def initialize
   (class << self; self; end).send :define_method, :call do
     yield
   end
 end
end

d = DoSomething.new do
 puts "hello world"
end

d.call
d.call

As usual with define_method we need to open the singleton class, and use send. This will work, since the block sent to define_method is a real closure, that closes over the block sent to initialize.

JRuby RailsConf hackfest next Thursday

2008-05-22T13:40:00.002+02:00

LinkedIn, Joyent and Sun Microsystems is sponsoring a JRuby hackfest in conjunction with RailsConf. It will happen next Thursday from 6:30 PM in Portland, there will be some food and beer and so on. Oh, Charles, Nick, Tom and me will be there - bring your laptops and any and all questions/patches/bugs/ideas with regards to JRuby.

Read more in Charles blog, here: http://headius.blogspot.com/2008/05/jruby-pre-railsconf-hackfest-on.html

Remember to RSVP to Charles if you're coming. Space is limited so RSVP as soon as possible.

How large is your .emacs?

2008-05-21T18:20:00.003+02:00

I've been reading lots of blogs and opinions about emacs the last few days. What strikes me is all of these people who brag about how large their .emacs files have become. So let me make this very clear:

If your .emacs file is longer than a page YOU ARE DOING IT WRONG.

Why? Well. Unless you are a casual Emacs user, your .emacs should not be regarded as a configuration file. Rather, the .emacs file is actually the entry point to the source repository of your own version of Emacs. In effect, when you configure Emacs you create a fork, which has it's own source that you need to maintain. This is not configuration. This is programming and you should approach it like you do all programming. What does that mean? Modularization. Clean code. Code comments. Source control. Tests. But modularization and source control are the ones that are most important for my Emacs configuration. I have loads of files in ~/emacs and every kind of extension I do has it's own kind of file or directory to put it in. The ~/emacs directory is checked out from source control, and has got customizations for different platforms. That's why my .emacs file is 4-5 lines long. Two for setting customizations that are specific to this computer, and the rest to load the stuff inside of ~/emacs. And that's all.

So how do you handle modularization of Emacs Lisp code? This won't be a tutorial. Just a few advices that might make things easier.

In no specific order:

(load "file.el") will allow you to just load another file.
(require 'cl) will give you lots of nice functionality from Common Lisp

I recommend you have one place where you add all your load paths. Mine look something like this:

(labels ((add-path (p)
        (add-to-list 'load-path
                     (concat emacs-root p))))
(add-path "emacs/jde/lisp")
(add-path "emacs/nxml")
(add-path "emacs/own")             ;; Personal elisp code
 (add-path "emacs/lisp")            ;; Various elisp code, just dumped here
)

Why do it like this? Well, it gives you an easier way to add full paths to your load path without repeating lots of stuff. This depends on you defining emacs-root somewhere - do define it, it can be highly useful.
Set custom-file. (The custom-file is the file where Emacs saves customizations. If you don't set a specific file for this, you will end up getting all customizations saved into .emacs which you really don't want.) The code for this is simple. Just do (setq custom-file "the-file-name.el")
Use hooks and advice liberally. They allow you to attach new functionality without monkey patching.
If you ever edit XML, NEVER use Emacs builtin XML editor. Instead download the excellent NXML package.
Learn how to use Info and customizations
Use Ido mode

Feel free to add other good advice in the comments. These were just a small smattering of stuff I like and which helps your environment quite seriously. But the most important part is the whole thing about keeping your .emacs extremely small!

Addendum: As Phil just pointed out (and which was part of my plan from the beginning) is that Autoloads should be used as much as possible. Also, make sure to bytecompile as much as possible.

Break Java!

2008-05-19T16:36:00.002+02:00

As some of you might have noticed I am not extremely fond of everything the Java language. I have spent some time lately trying to figure out how I would change the language if I could. These changes are of course breaking, and would never be included in regular Java. I've had several names for it, but my current favorite is unJava. You can call it Java .314 or minijava if you want. Anyway, here's a quick breakdown of what I'd like to see done to make a better language out of Java without straying to far away from the current language:

No primitives. No ints, bytes, chars, shorts, floats, booleans, doubles or longs. They are all evil and should not be in the language.
No primitive arrays. Javas primitive arrays are not typesafe and are evil. With generics there is no real point in having them, especially since they interact so badly with generic collections. This point would mean that certain primitive collection types can't be implemented in the language itself. This is a price I'm willing to pay.
Scala style generics, with usage-defined contra/co-variance and more flexible type bounds.
No anonymous inner classes. There is no need for them with the next points.
First class methods.
Anonymous methods (these obviously need to be closures).
Interfaces that carries optional implementation.
No abstract classes - since you don't need them with the above.
Limited type inference, to avoid some typing. Scala or C# style is fine.
Annotations for many of the current keywords - accessibility specifically, but also things like transient, volatile and synchronized.
No checked exceptions.
No angle brackets for generics (they really hurt my eyes. are there XML induced illnesses? XII?). Square brackets look so much better.
Explicit separation of nullable from non-nullable values.

These points are probably quite substantial together, but I still don't think the language would be that difference from Java in syntax and semantics. The usage patterns would be extremely different though. You wouldn't sacrifice any performance with these kinds of things - they wouldn't change the characteristics of the output that much, and I believe these things could make the language smaller, cleaner, and easier to work with.

ThoughtWorks comes to Sweden

2008-05-17T00:32:00.003+02:00

I few months back I blogged about the possibility that ThoughtWorks would come to Sweden. Well, this is now reality. I have the extreme honor to be a part of this initiative together with Marcus Ahnve (who blogged about it here). If you read that blog you will know that Marcus will head the Swedish operation. I am immensely happy about having him as my new colleague and also boss. =)

People might ask what my role in this new office will be. That's a valid question. My main goal is to stay out of trouble - and trying my very best to not scare potential customers away. Marcus is extremely capable and will handle all challenges, which means that I'll do my best to bask in the glory of opening an new office. I might also have a hand in any billable work we do, and help out with recruitment and possibly even (shudder) marketing.

Not sure if you catched the meaning in that last sentence, but let me spell two points out. We will be selling work in Sweden from day one. I will be one of the consultants sold and that means that if you have an Ruby or JRuby work you want to start up, this might be an excellent time to call your local ThoughtWorks office... =)

The recruitment point is simply this. We plan on accumulating the best people we can find - as we aim to do in every country we enter. If you feel like you could fit this bill, mail me and we can talk.

It's important to note that this operation will initially be very low profile. Don't expect center folds in DN's Economy pages. We will work mostly with word-of-mouth. So if you hear of someone that might need our help, don't hesitate to mention our name. And even though we are low profile, we will still have the resources of the whole company to draw on. A 1000 ThoughtWorkers. That feels rather good, and it should feel even better for any prospective clients.

One thing I have been a bit worried about is my commitment to JRuby, and other open source projects. I assure everyone I'll do my best to live up to these commitments. Sleep be damned!

These are exciting times. I for one is looking forward to it very much. Me and Marcus will officially start on this from June this year. Get in touch if you have any questions. It's my name separated with dots at gmail, or obini at the official thoughtworks domain.

Dynamically created methods in Ruby

2008-05-15T14:35:00.003+02:00

There seems to be some confusion with regards to dynamically defining methods in Ruby. I thought I'd take a look at the three available methods for doing this and just quickly note why you'd use one method in favor of another.

Let's begin by a quick enumeration of the available ways of defining a method after the fact:

Using a def
Using define_method
Using def inside of an eval

There are several things to consider when you dynamically define a method in Ruby. Most importantly you need to consider performance, memory leaks and lexical closure. So, the first, and simplest way of defining a method after the fact is def. You can do a def basically anywhere, but it needs to be qualified if you're not immediately in the context of a module-like object. So say that you want to create a method that returns a lazily initialized value, you can do it like this:

class Obj
  def something
  puts "calling simple"
    @abc = 3*42
    def something
    puts "calling memoized"
    @abc
    end
  something
  end
end

o = Obj.new
o.something
o.something
o.something

As you can see, we can use the def keyword inside of any context. Something that bites most Ruby programmers at least once - and more than once if they used to be Scheme programmers - is that the second def of "something" will not do a lexically scoped definition inside the scope of the first "something" method. Instead it will define a "something" method on the metaclass of the currently executing self. This means that in the example of the local variable "o", the first call to "something" will first calculate the value and then define a new "something" method on the metaclass of the "o" local variable. This pattern can be highly useful.

Another variation is quite common. In this case you define a new method on a specific object, without that object being the self. The syntax is simple:

def o.something
puts "singleton method"
end

This is deceptively simple, but also powerful. It will define a new method on the metaclass of the "o" local variable, constant, or result of method call. You can use the same syntax for defining class methods:

def String.something
puts "also singleton method"
end

And in fact, this does exactly the same thing, since String is an instance of the Class class, this will define a method "something" on the metaclass of the String object. There are two other idioms you will see. The first one:

class << o
  def something
  puts "another singleton method"
  end
end

does exactly the same thing as

def o.something
puts "another singleton method"
end

This idiom is generally preferred in two cases - first, when defining on the metaclass of self. In this case, using this syntax makes what is happening much more explicit. The other common usage of this idiom is when you're defining more than one singleton method. In that case this syntax provide a nice grouping.

The final way of defining methods with def is using module_eval. The main difference here is that module_eval allows you to define new instance methods for a module like object:

String.module_eval do
  def something
  puts "instance method something"
  end
end

"foo".something

This syntax is more or less equivalent to using the module or class keyword, but the difference is that you can send in a block which gives you some more flexibility. For example, say that you want to define the same method on three different classes. The idiomatic way of doing it would be to define a new module and include that in all the classes. But another alternative would be doing it like this:

block = proc do
  def something
  puts "Shared something definition"
  end
end

String.module_eval &block
Hash.module_eval &block
Binding.module_eval &block

The method class_eval is an alias for module_eval - it does exactly the same thing.

OK, so now you know when the def method can be used. Some important notes about it to remember is this: def does _not_ use any enclosing scope. The method defined by def will not be a lexical closure, which means that you can only use instance variables from the enclosing running environment, and even those will be the instance variables of the object executing the method, not the object defining the method. My main rule is this: use def whenever you can. If you don't need lexical closures or a dynamically defined name, def should be your default option. The reason: performance. All the other versions are much harder - and in some cases impossible - for the runtimes to improve. In JRuby, using def instead of define_method will give you a large performance boost. The difference isn't that large with MRI, but that is because MRI doesn't really optimize the performance of general def either, so you get bad performance for both.

Use def unless you can't.

The next version is define_method. It's just a regular method that takes a block that defines that implementation of the method. There are some drawbacks to using define_method - the largest is probably that the defined method can't use blocks, although this is fixed in 1.9. Define_method gives you two important benefits, though. You can use a name that you only know at runtime, and since the method definition is a block this means that it's a closure. That means you can do something like this:

class Obj
def something
 puts "calling simple"
 abc = 3*42
 (class <<self; self; end).send :define_method, :something do
   puts "calling memoized"
   abc
 end
 something
end
end

o = Obj.new
o.something
o.something
o.something

OK, let this code sample sink in for a while. It's actually several things rolled into one. They are all necessary though. First, note that abc is no longer an instance variable. It's instead a local variable to the first "something" method. Secondly, the funky looking thing(class <<self; self; end) is the easiest way to get the metaclass of the current object. Unlike def, define_method will not implicitly define something on the metaclass if you don't specify where to put it. Instead you need to do it manually, so the syntax to get the metaclass is necessary. Third, define_method happens to be a private method on Module, so we need to use send to get around this. But wait, why don't we just open up the metaclass and call define_method inside of that? Like this:

class Obj
def something
 puts "calling simple"
 abc = 3*42
 class << self
   define_method :something do
     puts "calling memoized"
     abc
   end
 end
 something
end
end

o = Obj.new
o.something
o.something
o.something

Well, it's a good thought. The problem is that it won't work. See, there are a few keywords in Ruby that kills lexical closure. The class, module and def keywords are the most obvious ones. So, the reference to abc inside of the define_method block will actually not be a lexical closure to the abc defined outside, but instead actually cause a runtime error since there is no such local variable in scope. This means that using define_method in this way is a bit cumbersome in places, but there are situations where you really need it.

The second feature of define_method is less interesting - it allows you to have any name for the method you define, including something random you come up with at runtime. This can be useful too, of course.

Let's summarize. The method define_method is a private method so it's a bit problematic to call, but it allows you to define methods that are real closures, thus providing some needed functionality. You can use whatever name you want for the method, but this shouldn't be the deciding reason to use it.

There are two problems with define_method. The first one is performance. It's extremely hard to generally optimize the performance of invocation of a define_method method. Specifically, define_method invocations will usually be a bit slower than activating a block, since define_method also needs to change the self for the block in question. Since it's a closure it is harder to optimize for other reasons too, namely we can never be exactly sure about what local variables are referred to inside of the block. We can of course guess and hope and do optimistic improvements based on that, but you can never get define_method invocations are fast as invoking a regular Ruby method.

Since the block sent to define_method is a closure, it means it might be a potential memory leak, as I documented in an older blog post. It's important to note that most Ruby implementations keep around the original self of the block definition, as well as the lexical context, even though the original self is never accessible inside the block, and thus shouldn't be part of the closed environment. Basically, this means that methods defined with define_method could potentially leak much more than you'd expect.

The final way of defining a method dynamically in Ruby is using def or define_method inside of an eval. There are actually interesting reasons for doing both. In the first case, doing a def inside of an eval allows you to dynamically determine the name of the method, it allows you to insert any code before or after the actual functioning code, and most importantly, defining a method with def inside of eval will usually have all the same performance characteristics as a regular def method. This applies for invocation of the method, not definition of it. Obviously eval is slower than just using def directly. The reason that def inside of an eval can be made fast is that at runtime it will be represented in exactly the same way as a regular def-method. There is no real difference as far as the Ruby runtime sees it. In fact, if you want to, you can model the whole Ruby file as running inside of an eval. Not much difference there. In particular, JRuby will JIT compile the method if it's defined like that. And actually, this is exactly how Rails handles potentially slow code that needs to be dynamically defined. Take a look at the rendering of compiled views in ActionPack, or the route recognition. Both of these places uses this trick, for good reasons.

The other one I haven't actually seen, and to be fair I just made it up. =) That's using define_method inside of an eval. The one thing you would gain from doing such a thing is that you have perfect control over the closure inside of the method defined. That means you could do something like this:

class BinderCreator
def get
  abc = 123
  binding
end
end

eval(<<EV, BinderCreator.new.get)
  Object.send :define_method, :something do
  abc
end
EV

In this code we create a new method "something" on Object. This method is actually a closure, but it's an extremely controller closure since we create a specific binding where we want it, and then use that binding as the context in which the define_method runs. That means we can return the value of abc from inside of the block. This solution will have the same performance problems as regular define_method methods, but it will let you control how much you close over at least.

So what's the lesson? Defining methods can be complicated in Ruby, and you absolutely need to know when to use which one of these variations. Try to avoid define_method unless you absolutely have to, and remember that def is available in more places than you might think.

Would Type Inference help Java

2008-05-15T14:28:00.002+02:00

My former colleague Lars Westergren recently posted a blog (here) about type inferencing, posing the question whether type inference would actually be good for Java, and if it would provide any benefits outside of just "less typing".

In short: no. Type inferencing would probably not do much more than save you some typing. But how much typing it would save you could definitely vary depending on the type of type inference you added. The one version I would probably prefer is just a very simple hack to avoid writing out the generic type arguments. One simple way of doing that would be to allow an equals sign inside of the angle brackets. In that case you could do this:

List<=>      l  = new ArrayList<String>();
List<String> l2 = new ArrayList<=>();

Of course, you can do it on more complicated expressions:

List<Set<Map<Class<?>, List<String>>>> l = new ArrayList<=>();

This would save us some real pain in the definition of genericized types, and it wouldn't strip away much stuff you need for readability. In the above examples it would just strip away one duplication, and you don't need that duplication to read it correctly. The one case where it might be a little bit harder to read would be if you defined a variable and assigned it somewhere else. In that case the definition would need to carry the type information, so the instantiation would use the <=> syntax. I think that would be an acceptable price to reduce the verbosity of Java generics.

Another kind of generics that would be somewhat useful is the kind added to C#, which is only local to a scope. That means there will be no type inferencing of member variables, method parameters or return values. Of course, that's the crux of Lars question, since this kind of type inference potentially removes ALL type information in the current text, since you can do:

var x = someValue.DoSomething();

At this point there is no easy way for you to know what the type of x actually is. Reading it like this, it looks a bit frightening if you're used to Java type tags, but in fact this is not what you would see. In most cases you have a small method - maybe 5-15 lines of code, where x is being used in some way or another. In many cases you will see methods called on x, or x used as argument to method calls. Both of these usages gives you clues about what it might be, but in fact you don't always need to know what type it is. You just need to know what you can do with it. And that's exactly what Java interfaces represent. So for example, do you know what class you get back from Collections.synchronizedMap()? No, and you shouldn't need to know. What you do know is that it's something that implements Map, and the documentation says that it is synchronized, but that is it. The only thing you know about it is that you can use it as a map.

So in practice, the kind of type inference C# adds is actually quite useful, clean, and doesn't cause too much trouble - especially if you have one of those fancy ideas that do method completion... =)

From another angle, there are some things that type inference could possible do, but that you will never see in Java. For example, say that you assign a variable to something, and later you assign that variable to some other value. If these two values are distinct types that doesn't overlap in the inheritence chain, you will usually get an error. But if you have an advanced type system, it will do unification for you. The basic versions will just find the most common supertype (the disjunction), but you can also imagine the compiler injecting a new type into your program that is the union of the two types in use. This will provide something similar to duck typing while still retaining some static type safety. If your type system allows multiple inheritence, the synthetic union type might even be a subclass of both the types in question.

So yeah. The long answer is that you can actually do some funky stuff with type inference that doesn't immediately translate to less typing. Although less typing and better abstractions is what programming languages are all about, right? Otherwise assembler provides everything we want.

A New Hope: Polyglotism

2008-05-14T01:35:00.003+02:00

OK, so this isn't necessarily anything new, but I had to go with the running joke of the two blog posts this post is more or less a follow up to. If you haven't already read them, go read Yegge's Dynamic Languages Strikes Back, and Beust's Return Of The Statically Typed Languages.

So let's see. Distilled, Steve thinks that static languages have reached the ceiling for what's possible to do, and that dynamic languages offer more flexibility and power without actually sacrificing performance and maintainability. He backs this up with several research papers that point to very interesting runtime performance improvement techniques that really can help dynamic languages perform exceptionally well.

On the other hand Cedric believes that Scala is bad because of implicits and pattern matching, that it's common sense to not allow people to use the languages they like, that tools for dynamic languages will never be as good as the ones for static ones, that Java generics isn't really a problem, that dynamic language performance will improve but that this doesn't matter, that static languages really hasn't failed at all and that Java is still the best language of choice, and will continue to be for a long time.

Now, these two bloggers obviously have different opinions, and it's really hard to actually see which parts are facts and which are opinions. So let me try to sort out some facts first:

Dynamic language have been around for a long time. As long as statically typed languages in fact. Lisp was the first one.

There have been extremely efficient dynamic language implementations. Some of the Common Lisp implementations are on par with C performance, and Strongtalk also achieved incredible numbers. As several commenters have noted, Strongtalks performance did not come from the optional type tags.

All dynamic languages in large use today are not even on the same map with regards to performance. There are several approaches to fixing these, but we can't know how well they will work out in practice.

Java's type system is not very strong, and not very static, as these definitions go. From a type theoretic stand point Java does not offer neither static type safety nor any complete guarantees.

There is a good reason for these holes in Java. In particular, Java was created to give lots of hints to the compiler so the compiler can catch errors where the programmer is insoncistent. This is one of the reasons that you very often find yourself writing the same type name twice, including the type name arguments (generics). If the programmer makes a mistake at one side, the compiler will be able to catch this error very easily. It is a redundancy in the syntax that makes Java programs very verbose, but helps against certain kinds of mistakes.

Really strong type systems like those Haskell and OCaML use provide extremely strong compile time guarantees. This means that if the compiler accepts your program, you will never see any runtime errors from the type system. This allows these compilers to generate very efficient code, because they know more about the state of the application at most points in time, compared to the compiler for Java, which knows some things, but not nearly as much as Haskell or OCaML.

The downside of really strong type systems is that they disallow some extremely common expressions - these are things you intuitively can imagine, but it can't be expressed within the constraints of such a type system. One solution to these problems is to add higher kinds, but these have a tendency to create more complexity and also suffer from some of the same problems.

So, we have three categories of languages here. The strongly statically checked ones, like Haskell. The weakly statically checked ones, like Java. And the dynamically checked ones, like Ruby. The way I look at these, they are good at very different things. They don't even compete in the same leagues. And comparing them is not really a valid point of reasoning. The one thing that I am totally sure if is that we need better tools. And the most important tool in my book is the language. It's interesting, many Java programmers talk so much about tools, but they never seem to think about their language as a tool. For me, the language is what shapes my thinking, and thus it's definitely much more important than which editor I'm using.

I think Cedric have a point in that dynamic language tool support will never be as good as those for statically typed languages - at least not when you're defining "good" to be the things that current Java tools are good at. Steve thinks that the tools will be just as good, but different. I'm not sure. To a degree I know that no tool can ever be completely safe and complete, as long as the language include things like external configuration, reflection and so on. There is no way to include all dynamic aspects of Java, but using the common mainstream parts of the language will give you most of these. As always this is a tradeoff. You might get better IDE support for Java right now, but you will be able to express things in Ruby that you just can't express in Java because the abstractions will become too large.

This is the point where I'm going to do a copout. These discussions are good, to the degree that we are working on improving our languages (our tools). But there is a fuzzy line in these discussions, where you end up comparing apples and oranges. These languages are all useful, for different things. A good programmer uses his common sense to provide the best value possible. That includes choosing the best language for the job. If Ruby allows you to provide functionality 5 times faster than the equivalent functionality with Java, you need to think about whether this is acceptable or not. On the one hand, Java has IDEs that make maintainability easier, but with the Ruby codebase you will end up maintaining a fifth of the size of the Java code base. Is that trade off acceptable? In some cases yes, in some cases no.

In many cases the best solution is a hybrid one. There is a reason that Google allows more than one language (C++, Java, Python and JavaScript). This is because the languages are good at different things. They have different characteristics, and you can get a synergistic effect by combining them. A polyglot system can be greater than the sum of it's parts.

I guess that's the message of this post. Compare languages, understand your most important tools. Have several different tools for different tasks, and understand the failings of your current tools. Reason about these failings in comparison to the tasks they should do well, instead of just comparing languages to languages.

Be good polyglot programmers. The world will not have a new big language again, and you need to rewire your head to work in this environment.

JavaOne 2008, the other half

2008-05-12T01:09:00.002+02:00

So, the Thursday got a late start. For some strange reason I didn't feel motivated to go see the Intel General Session, so I showed up for Nick's session about JRuby on Rails deployment instead. Nick did a good job of outlining both the problem and the solution, and I have to say that this presentation was a good end cap for the JRuby week at JavaOne.

I had planned to go to some sessions after that, but I ended up hacking on JRuby instead. An interesting parser bug reared its head.

So the next session I went to was the Filthy Rich Clients one. Quite entertaining, although my interest in Swing is not what it used to be.

The final session of the day was the BOF about writing great technology books. This proved highly enjoyable since joining Josh Bloch and Brian Goetz as panelists were Burt Bates and Kathy Sierra. They did a wonderful job talking about how to write books that captures the readers attention and how to correctly use the brains weaknesses against it. I am tempted to say that this was the best session of the whole JavaOne. Brilliant.

On the Friday I was up early and sat in on Goslings Toy Story. Always funny, and some cool things there. For a geek like me, the CERN stuff and jMars was especially cool.

I managed to see quite a lot of sessions during the rest of the day. More Effective Java was useful as always, the Maxine Virtual Machine looks really cool, Neal Ford did an excellent job of comparing JRuby and Groovy, highlighting both the differences and similarities between the two. Finally the Jython session talked about some of the implementation challenges we in the JRuby team have wrestled with too, implementing a highly dynamic language on top of the JVM.

All in all, this JavaOne definitely stood out as the first non-Java-language JavaOne for me. And I didn't even attend a single one of the gazillions of JavaFX presentations. It was a good year in general, and specifically for dynamic languages.

Oh, and my book is the 5th bestseller in the conference bookstore. Yay!

JavaOne halfway point

2008-05-08T18:58:00.002+02:00

To say that I am seriously tired of JavaFX at this point, would be a gross understatement. So let's not even go there.

CommunityOne was a nice event. I like the feeling of it more and more, and the new open spaces approach seemed to be really successful. The alternative languages presentations were well attended and good. I recommend anyone in the Bay Area, or anyone attending JavaOne, to make the effort to go to CommunityOne next year. It's definitely worth it.

Tuesday was, for undisclosed reasons, a day where I didn't attend so many presentations. In fact, I missed both the keynote and Tom&Charlies JRuby talk. Bad on me. I did manage to go to both the technical session and the BOF about upcoming language features in Java 7. Let me say immediately that I don't want pluggable type systems as a part of Java. Yes, they are useful in certain settings, but they don't fit Java. Not at all. I'm all for having it possible to have annotations in more places, but not for type systems.

Wednesday was "my" day of the conference. Started out early with the Script Bowl, where Groovy, JRuby, Jython and Scala faced off in three different challenges. I was one of the three judges. It was a actually a great fun. Some people have gotten annoyed at it, but I feel that they are taking it too seriously. This format was a good way of introducing the audience to the capabilities of four languages that are sometimes very much alike and sometimes very different.

It's also interesting to note that Charles was the only one of the panel who solved his challenges by asking the community for help with it. In fact, Charles didn't do any of them himself, while the other three did their code in isolation. My opinion is that this shows a difference in attitude between the communities, and it's very interesting.

After that I took it easy for some time, and then it was time for my JRuby on Rails presentation. It went fairly well and was also well attended. I did a serious mistake in my database configuration, but Tom helped me out and the rest of the demonstrations was good.

I did a book signing session after that and then headed back to the office for a while before coming back and holding my JRuby at ThoughtWorks BOF which was also nice.

Today I'm going to take it easy and relax. I'll go to Nick's session in an hour and then I'll go to Josh and Brian's BOF tonight. After that we host a party which I'm going to attend for a while. It's going to be a nice day.

Faulty routes on MacOS X

2008-05-05T00:48:00.002+02:00

This has been driving me insane, but I haven't had time to actually try to fix the problem until now, so this is a small writeup about the symptom and what you can do about it. Sadly, it's kinda a hard problem to search for, unless you already knows whats wrong.

This was my problem: At work, accessing LiveJournal worked fine, but at home it didn't. Basically I got a Server Not Found from Firefox when trying to access from home. After a while I realized that the distinction wasn't between home/work, but wireless or wirebound network. See, at work I usually use a wire to connect, but I only use wireless at home.

My first shot at a solution was something that solved a similar problem a while back - namely flushing the DNS cache. Of course, doing an nslookup and verifying it told me that the cache wasn't the problem, but I tried anyway, because MacOSX sometimes plays funny tricks with this cache. So, to do this, just do "sudo lookupd -flushcache" in a console window, and then restart the network interface.

I tried to figure out what was wrong by pinging, and doing traceroute. Both of these gave clues that something was wrong. Traceroute gave me "Can't assign requested address" from the bind system call. That finally made me realize that the route tables for the network interface was screwed. Lo and behold, doing a "sudo route flush" and restarting the interface solved my problems. Apparently there were about 10 faulty routes stuck in the cache, and for some reason MacOS X didn't flush this cache automatically when connecting to a different wireless network. Highly annoying, but it's finally solved.

Just add scaling!

2008-05-04T01:16:00.000+02:00

(To protect the innocent, all names of programming languages have been changed. All likeness with existing names is purely for dramatic effect.)

I have heard many times now that LRM doesn't scale. People have been telling it to me in all these different circumstances. They tell me that languages like Deep Throttle and Moulder have scaling but LRM doesn't. Even the Scally language has scaling - you could understand that from the name I guess. So why doesn't LRM has this feature?

Since I'm rather fond of LRM, and really don't want to go back to either Deep Throttle or Moulder, I decided to add scaling to LRM. I mean, how hard could it be if all these other languages have it? I have implemented langauges before, so this should just be a case of finding the correct implementation of the feature, and grafting it onto LRM's code base, and then provide a patch to the mailing lists.

Since I knew LRM didn't have scaling, I decided to investigate other languages that have it. I began with the original language, the one people write operating systems in. Obviously a language like Deep Throttle has scaling. So I started reading the books, looking through the syntax. But I couldn't find anything there. What was I missing? Was there some obscure syntactic element that wasn't documented, that provided scaling? Or was it hidden somewhere? Maybe the pointers were in reality another word for scaling? But no, it didn't seem that way. I decided to break open GDPC, the Gnu Deep Throttle Compiler, and see if this scaling thing was actually something the compiler did autoamtically for you - a bit like register allocation. But I didn't find any scaling allocation. There was lots of other neat stuff there, but all of it was quite mundane translations from the Deep Throttle code into machine code.

Feeling utterly dejected I turned my attention to Moulder. I assume that you all are familiar with the language - it's used in many places around the world and provides lots of really advanced features such as garbage collection, a virtual machine, and yes - scaling! Of course, this is where it would be. Maybe Deep Throttle had hidden the feature somewhere obscure, but I was sure Moulder would make it easier to find. I started out with the syntax again, looking through the things you could do with the language. Interestingly, Moulder is a really small language, when you get down to it. It's small but you need to say lots of things to get anything done. I couldn't find the syntax for scaling, so I started entertaining other ideas. Maybe all those extra words "protected final static Foo" was actually necessary. Maybe the scaling happened in the spaces between the words, or in some strange interaction between them? The only way to find out was to crack open the compiler and look what was happening. In this case it was actually easier, because the compiler did even less than Deep Throttle's compiler. Frankly, the Moulder compiler just output something bytecodes. There were no hidden scaling inlining in the Moulder compiler.

You see where this is heading? Right, I had to investigate the bytecodes that Moulder use. Where is that precious scaling bytecode? I assume it would be called something like invoke_scaling or maybe ldscaled. But nothing. There was no scaling bytecode. All of the existing bytecodes just did the same mundane things that I know LRM already can do. I was now feeling ready to give up, but I came up with one final idea. Scally is famous for being a language with scaling. It's creators even named it after that feature. So if both Moulder and Scally has scaling, maybe it's somewhere in Moulders gigantic standard library?

But guess what? I didn't find it. I still haven't found anyone who knows how you implement Scaling in a language, so I guess that LRM will never have it... Anyone who care to enlighten me, please send me a detailed email with an implementation of Scaling. I really feel the need to know how this thing works.

The Twitter Conspiracy

2008-05-03T11:12:00.001+02:00

First, let me warn you. This is a conspiracy theory. It's got all the usual logical fallacies and problems of a conspiracy theory. Add to that the fact that it's almost guaranteed to not be true, you might ask why I'm writing about it. Well, the thing is, if I was in charge of Twitter, I would do something like this. Of course, I'm not in charge, and the people in charge are probably more sane and well adjusted persons than me. But still, who knows?

What is the idea then? Well, it's actually really simple. Think about the Twitter network, the kind of people who connect there, and the way things spread. What is the difference between Twitter and mobile texting for example? First, everything is by default multicast. It's not reciprocal - you don't need to know how many hundreds or thousands read what you write. And you don't care how many others read the persons you read. You are restricted in length. And, the whole thing is open enough that you can follow all the tweets going on in the system.

The characteristics I've described means that Twitter is more or less the ideal memetic engine. What I mean by this is that it's a wonderful way to spread your ideas, if you can express them in a concise and readable way. This means that certain memes doesn't work well in this setting, but most do. And you can convince more people to join, because if your tweets are interesting enough, someone will notice them in the all-tweet. Also, you can see who the people you are interested in follows, which means that you can spread your network selectively, but really quickly.

These are not really part of the theory. They are just the axioms. So what's the theory then? Well, what are Twitter doing with all this data? If I would have been them, I would have used it to do research on memetic spread and viral marketing. I would use it to try out ideas based on how good uptake they have. Finding this information is not really hard when you have control over all the messages happening. In fact, you could actually do it even outside of Twitter, by using the published tools correctly.

What got me thinking along these lines? Well, the whole TechCrunch debacle was the thing that triggered the idea. How would it work in practice? Well, first, the Twitter gang couldn't necessarily know what kind of people would take up Twitter the most, so the cultural fit of Twitter is actually mostly self organizing. The people and groups taking part of twitter select themself for this experiment. Now, of course there are lots of overlapping groups, and that gives even more interesting possibilities for the sociodynamics of meme transfer between non-overlapping social circles.

Take the Ruby people, who have a quite significant presence on Twitter. They are one of the test groups in my theory, and the TechCrunch article was a very directed way of inserting a meme and see how fast and to how many it propagated. It was very easy to insert this into the blog-sphere, since Twitter could have had any amount of people "leak" the information in the TechCrunch article. Once it was out, they just needed to set up some suitable filters and follow the spread. They also inserted a couonter meme, through one of their employees, to see if it work out as an "antidote" to the first meme, or which one of them was stronger. All in all, I think they got enough material for several research articles out of this stunt.

OK, so really, you don't need to grasp for conspiracy theories to explain the TC debacle. It's not necessary, so Occam's razor demands that we choose the simplest available conclusion that explains all the facts. This theory does not fall into this category. But it's still an entertaining notion.

And I predict that sooner or later, someone will use Twitter, or another network like it, to do this kind of research. It's a question of time. This kind of information is way to valuable for marketing purposes and also for the understanding of the human mind, that it will happen. The question is: will you know about it, when you're participating in their research?

Let's not forget the lovely recursive interpretation that this blog post is a way of doing the same kind of research I've just described.

RbYAML in Google Summer of Code

2008-04-23T13:26:00.004+02:00

Great news for all Ruby implementations around. A project to bring RbYAML up-to-date and perform better has been accepted for Google Summer of Code. Long Sun is the name of the student, and me and Xue Yong Zhi will jointly mentor this effort.

In fact, I'm very excited about this news. RbYAML was an incredibly important piece of the puzzle to get JRuby to finally work with RubyGems, and that kickstarted our possibilities to start testing numerous other applications. I soon ported RbYAML to Java, and created the JvYAML and JvYAMLb projects, to get better efficiency. Sadly, this left RbYAML without any TLC. That changed a while back when Rubinius picked up the project to get their YAML support going, and now that Long Sun will work on it, hopefully we will finally get an extremely compliant and bug free YAML implementation for Ruby.

This will obviously benefit Rubinius, but it will also be very good for both JRuby and IronRuby. The work will be test-driven which means a more complete test suite will be built around YAML in Ruby.

If you're interested in following the project, it's now hosted at Google Code (due to problems with RubyForge from China) at http://code.google.com/p/rbyaml/. Long Sun will also blog about his progress here: http://rbyaml.blogspot.com/.

Exciting news indeed.

Ruby Design meeting

2008-04-22T12:16:00.002+02:00

Yesterday marked the first of the Ruby design meetings, where most of the Ruby implementers got together on IRC and started hashing out the solutions to several current concerns, and worked on getting more cooperation in the design of Ruby features.

This is slated to become a weekly meeting, and it's a huge deal. This will make the lives of all Ruby implementations much easier, and the meeting yesterday actually accomplished some very nice things.

There were representatives from JRuby, Rubinius and macruby present, and of course also Matz, Koichi, Nobu and Tanaka from the Ruby core team.

Some of the highlights was a decision to start working on a common API for MultiVM, initial acceptance to add the RubySpecs (coming from Rubinius originally) into the 1.8 and 1.9 build process, meaning that regression testing will be much better from now on, and also having most implementations using the same specs for compliance testing. In reality, this takes us one step closer to a real executable specification that everyone agrees on and has official blessing from Matz.

In conjunction with this, a decision to set up continuous integration for 1.8 and 1.9 was made. The exact practicalities is still to be decided, but the decision to get it done is also very important.

All in all, these are excellent news, and I'm feeling extremely hopeful about more cooperation between the Ruby implementors.

If you're interested in exactly what happened, you can find the agenda, action items and log here: http://ruby-design.pbwiki.com/Design20080421.

JtestR 0.2 released!

2008-04-17T15:08:00.002+02:00

And so, JtestR 0.2 has finally been released. The highlights include support for Expectations and TestNG, RSpec stories and lots of other goodies.

Here is the release announcement:

JtestR allows you to test your Java code with Ruby frameworks.

Homepage: http://jtestr.codehaus.org
Download: http://dist.codehaus.org/jtestr

JtestR 0.2 is the current release of the JtestR testing tool. JtestR integrates JRuby with several Ruby frameworks to allow painless testing of Java code, using RSpec, Test/Unit, Expectations, dust and Mocha.

Features:
- Integrates with Ant and Maven
- Includes JRuby 1.1, Test/Unit, RSpec, Expectations, dust, Mocha and ActiveSupport
- Customizes Mocha so that mocking of any Java class is possible
- Background testing server for quick startup of tests
- Automatically runs your JUnit and TestNG codebase as part of the build

Getting started: http://jtestr.codehaus.org/Getting+Started

New and fixed in this release:
JTESTR-10 It should be possible to run TestNG tests
JTESTR-12 Buildr support
JTESTR-13 CC.rb should be able to run JtestR tests
JTESTR-17 Tests should be groupable and runnable per groups
JTESTR-21 Support RSpec stories
JTESTR-28 JtestR should include expectations
JTESTR-30 code coverage support
JTESTR-31 Autoloading of Java constants
JTESTR-32 Can't load IA 32-bit .so on a IA 32-bit platform
JTESTR-33 JtestR should use latest version of JRuby
JTESTR-34 Errors when project is in path with spaces on Windows XP
JTESTR-37 Can't expect a specific Java exception correctly
JTESTR-38 Problem with mocking Java classes
JTESTR-39 RSpec story runner seems to require rubygems
JTESTR-40 Package missing or.jruby.exceptions.RaiseException
JTESTR-43 It should be possible to get the generated mock class without instantiation
JTESTR-44 New output files start with a whitespace
JTESTR-45 RSpec raise_error and Test/Unit assert_raise and assert_nothing_raised handled JRuby NativeException stuff correctly.

Team:
Ola Bini - ola.bini@gmail.com
Anda Abramovici - anda.abramovici@gmail.com

Connecting languages (or polyglot programming example 1)

2008-04-15T21:57:00.003+02:00

Today I spent some time connecting two languages that are finding themselves popular for solving wildly different kinds of problems. I decided I wanted to see how easy it was and if it was a workable solution if you would want to take advantage of the strengths of both languages. The result is really up to you. My 15 minutes experiment is what I'll discuss here.

If you'd like, you can see this as a practical example of the sticky part where two languages meet, in language-oriented programming.

The languages under consideration is Ruby and Erlang. The prerequisite reading is this eminent article by my colleague Dennis Byrne: Integrating Java and Erlang.

The only important part is in fact the mathserver.erl code, which you can see here:

-module(mathserver).
-export([start/0, add/2]).

start() ->
Pid = spawn(fun() -> loop() end),
register(mathserver, Pid).

loop() ->
receive
  {From, {add, First, Second}} ->
      From ! {mathserver, First + Second},
      loop()
end.

add(First, Second) ->
mathserver ! {self(), {add, First, Second}},
receive
  {mathserver, Reply} -> Reply
end.

Follow Dennis' instructions to compile this code and start the server in an Erlang console, and then leave it there.

Now, to use this service is really easy from Erlang. You can really just use the mathserver:add/2 operation directly or remotely. But doing it from another language, in this case Ruby is a little bit more complicated. I will make use of JRuby to solve the problem.

So, the client file for using this code will look like this:

require 'erlang'

Erlang::client("clientnode", "cookie") do |client_node|
server_node = Erlang::OtpPeer.new("servernode@127.0.0.1")
connection = client_node.connect(server_node)

connection.sendRPC("mathserver", "add", Erlang::list(Erlang::num(42), Erlang::num(1)))

sum = connection.receiveRPC

p sum.int_value
end

OK, I confess. There is no erlang.rb yet, so I made one. It includes some very small things that make the interfacing with erlang a bit easier. But it's actually still quite straight forward what's going on. We're creating a named node with a specific cookie, connecting to the server node, and then using sendRPC and receiveRPC to do the actual operation. The missing code for the erlang.rb file should look something like this (I did the minimal amount here):

require 'java'
require '/opt/local/lib/erlang/lib/jinterface/priv/OtpErlang.jar'

module Erlang
import com.ericsson.otp.erlang.OtpSelf
import com.ericsson.otp.erlang.OtpPeer
import com.ericsson.otp.erlang.OtpErlangLong
import com.ericsson.otp.erlang.OtpErlangObject
import com.ericsson.otp.erlang.OtpErlangList
import com.ericsson.otp.erlang.OtpErlangTuple

class << self
def tuple(*args)
  OtpErlangTuple.new(args.to_java(OtpErlangObject))
end

def list(*args)
  OtpErlangList.new(args.to_java(OtpErlangObject))
end

def client(name, cookie)
  yield OtpSelf.new(name, cookie)
end

def num(value)
  OtpErlangLong.new(value)
end

def server(name, cookie)
  server = OtpSelf.new(name, cookie)
  server.publish_port

  while true
    yield server, server.accept
  end
end
end
end

As you can see, this is regular simple code to interface with a Java library. Note that you need to find where JInterface is located in your Erlang installation and point to that (and if you're on MacOS X, the JInterface that comes with ports doesn't work. Download and build a new one instead).

There are many things I could have done to make the api MUCH easier to use. For example, I might add some methods to OtpErlangPid, so you could do something like:

pid << [:call, :mathserver, :add, [1, 2]]

where the left arrows sends a message after transforming the arguments.

In fact, it would be exceedingly simple to make the JInterface API downright nice to use, getting the goodies of Erlang while retaining the Ruby language. And oh yeah, this could work on MRI too. There is an equivalent C library for interacting with Erlang, and there could either be a native extension for doing this, or you could just wire it up with DL.

If you read the erlang.rb code carefully, you might have noticed that there are several methods not in use currently. Say, why are they there?

Well, it just so happens that we don't actually have to use any Erlang code in this example at all. We could just use the Erlang runtime system as a large messaging bus (with fault tolerance and error handling and all that jazz of course). Which means we can create a server too:

require 'erlang'

Erlang::server("servernode", "cookie") do |server, connection|
 terms = connection.receive
 arguments = terms.element_at(1).element_at(3)
 first = arguments.element_at(0)
 second = arguments.element_at(1)

 sum = first.long_value + second.long_value
 connection.send(connection.peer.node, Erlang::tuple(server.pid, Erlang::num(sum)))
end

The way I created the server method, it will accept connections and invoke the block for every time it accepts a connection. This connection is yielded to the block together with the actual node object representing the server. The reason the terms are a little bit convoluted is because the sendRPC call actually adds some things that we can just ignore in this case. But if we wanted, we could check the first atoms and do different operations based on these.

You can run the above code in server, and use the exact same math code if you want. For ease of testing, switch the name to servernode2 in both server and client, and then run them. You have just sent Erlang messages from Ruby to Ruby, passing along Java on the way.

Getting different languages working together doesn't need to be hard at all. In fact, it can be downright easy to switch to another language for a few operations that doesn't suit the current language that well. Try it out. You might be surprised.

Pragmatic Static Typing

2008-04-12T21:14:00.001+02:00

I have been involved in several discussions about programming languages lately and people have assumed that since I spend lots of time in the Ruby world and with dynamic languages in general I don't like static typing. Many people in the dynamic language communities definitely expresses opinions that sound like they dislike static typing quite a lot.

I'm trying to not sound defensive here, but I feel the need to clarify my position on the whole discussion. Partly because I think that people are being extremely dogmatic and short-sighted by having an attitude like that.

Most of my time I spend coding in Java and in Ruby. My personal preference are to languages such as Common Lisp and Io, but there is no real chance to use them in my day-to-day work. Ruby neatly fits the purpose of a dynamic language that is close to Lisp for my taste. And I'm involved in JRuby because I believe that there is great worth in the Java platform, but also that many Java programmers would benefit from staying less in the Java language.

I have done my time with Haskell, ML, Scala and several other quite statically typed languages. In general, the fact that I don't speak that much about those languages is that I have less exposure to them in my day-to-day life.

But this is the thing. I don't dislike static typing. Absolutely not. It's extremely useful. In many circumstances it gives me things that really can't have in a dynamic language.

Interesting thought: Smalltalk is generally called a dynamic language with very late binding. There are no static type tags and no type inference happening. The only type checking that happens will happen at runtime. In this regard, Smalltalk is exactly like Ruby. The main difference is that when you're working with Smalltalk, it is _always_ runtime. Because of the image based system, the type checking actually happens when you do the programming. There is no real difference between coding time and runtime. Interestingly, this means that Smalltalk tools and environments have most of the same features as a static programming language, while still being dynamic. So a runtime based image system in a dynamic, late bound programming language will actually give you many of the benefits of static typing at compile time.

So the main take away is that I really believe that static typing is extremely important. It's very, very useful, but not in all circumstances. The fact that we reach for a statically typed programming language by default is something we really need to work with though, because it's not always the right choice. I'm going to say this in even stronger words. In most cases a statically typed language is a premature optimization that gets extremely much in the way of productivity. That doesn't mean you shouldn't choose a statically typed language when it's the best solution for the problem. But this should be a conscious choice and not by fiat, just because Java is one of the dominant languages right now. And if you need a statically typed language, make sure that you choose one that doesn't revel in unnecessary type tags. (Java, C#, C++, I'm looking at you.) My current choice for a static language is Scala - it strikes a good balance in most cases.

A statically typed language with type inference will give you some of the same benefits as a good dynamic language, but definitely not all of them. In particular, you get different benefits and a larger degree of flexibility from a dynamic language that can't be achieved in a static language. Neal Ford and others have been talking about the distinction between dynamic and static typing as being incorrect. The real question is between essence and ceremony. Java is a ceremonious language because it needs you to do several dances to the rain gods to declare even the simplest form of method. In an essential language you will say what you need to say, but nothing else. This is one of the reasons dynamic languages and type inferenced static languages sometimes look quite alike - it's the absence of ceremony that people react to. That doesn't mean any essential language can be replaced by another. And with regads to ceremony - don't use a ceremonious language at all. Please. There is no reason and there are many alternatives that are better.

My three level architecture of layers should probably be updated to say that the stable layer should be an essential, statically typed language. The soft/dynamic layer should almost always be a strongly typed dynamic essential language, and the DSL layers stays the same.

Basically, I believe that you should be extremely pragmatic with regards to both static and dynamic typing. They are tools that solve different problems. But out industry today have a tendency to be very dogmatic about these issues, and that's the real danger I think. I'm happy to see language-oriented programming and polyglot programming get more traction, because they improve a programmers pragmatic sensibilities.

JvYAMLb finally released as separate project

2008-04-11T18:22:00.003+02:00

So I've finally made the time to extract JvYAMLb from JRuby. That means that JvYAML is mildly deprecated. Of course, since JvYAMLb only uses ByteLists it might not be the best solution for everyone.

If you're interested in downloading the 0.1 release, you can do it at the Google Code download site http://code.google.com/p/jvyamlb/downloads/list.

Presentations this and next week

2008-04-08T12:05:00.002+02:00

Wow. JRuby 1.1 is out! Go get! (And incidentally, wouldn't JRuby be the perfect way for Google to support Ruby in Google Apps Engine?)

I'll be doing a talk tonight at .NETAkademien about Ruby and Rails. It's mostly an introduction.
You can find more info here.

On Thursday I'll speak about JRuby at Developer Summit in Stockholm. I'll be in the Dynamic Languages track and you can read the abstract and more information here.

And finally, next Monday I'll talk about JRuby at the Stockholm Ruby User Group. You can find out more information about that event here.

Overall, it feels like the conference season is starting up again. I'll be presenting at JavaOne in a month too.

I've been extremely quiet here lately. It's been a bit slow in JRuby land, actually, and my focus has been elsewhere. Hopefully I'll be able to start posting some really cool stuff soon - I have something up my sleeve that should be possible to talk about soon.

Meta-level thinking

2008-03-29T17:46:00.002+01:00

I have been trying to figure out what characterizes some of the best programmers I know, or know about. It's a frustrating endeavor of course, since most of these people are very different from each other, and have different experiences and ways to think and learn about stuff. Not to mention the fact that programmers tend to be highly individualistic.

But I think that I'm finally zeroing in on something that is general enough but also specific enough to categorize most of these people, and the general mind needed to be a really good programmer. In this blog post I'll call that "meta-level thinking", and I'll explain more about what I mean as we go along.

When people try to become better programmers there are generally a few different kinds of advices you can get. The ones that seem prevalent right now is (among others):

Learn - and understand - Lisp
More generally, learn a new language that is sufficiently different from your current knowledge
Work with domain specific languages
Understand metaprogramming
Read up on the available data structures (Knuth anyone)?
Implement a compiler and runtime system for a language

Of course, these are only a few examples, and the categories are a bit fuzzy. These items are based on most of my experiences so they tend to be a bit language heavy. But I believe that what you practice by choosing any of these routes is an ability to abstract your thinking. In short, I believe that being able to abstract and understand what goes on in a programming language is one way to become more proficient in that language, but not only that - by changing your thinking to see this part of your environment you generally end up programming differently in all languages.

This is why Lisp has a tendency to change the way people write Java code. Lisp teaches you metaprogramming and DSL's but they don't really do it in a way that need words. DSLs and metaprogramming are just part of Lisp. It's so much a part of the structure that you don't see it. But when you turn to Java you'll need to start working with these abstractions on a different level. Ruby programmers embrace metaprogramming and this changes the way these Ruby programmers think about things.

I'm really happy to see metaprogramming and DSLs getting more and more focus, because I really believe that understanding them is a really good way to make programmers better. Of course, you can get the same effect by writing a copmiler and runtime system - as Steve Yegge proposes. But I disagree that you really need that experience. There are other ways you can get the same way of looking at the world.

I call this meta-level thinking. I think it's mostly a learned ability, but also that there is an aptitude component. Some of the best programmers I've met just have a mind that fits very well. It's interesting to note that this kind of meta-level thinking is not an ability that is only applicable to programming. In fact, that's probably just a matter of genetic luck that the same ability works very well for programming as for many other things. I think that there is a connection between certain abilities and a capacity for meta-level thinking too. Like music - it's interesting to see how many programmers that have artistic leanings, and specifically towards music. It would be interesting to see some statistics about this.

In conclusion, this is my somewhat fuzzy view about what is one of the most important abilities that contributes to create brilliant programmers. If you have any interesting ideas about other ways you can reliably practice this ability, please improve on my list.

The contract of IO#read

2008-03-20T13:26:00.002+01:00

It's interesting. After Charlie made an immense effort and rewrote our IO system, basically from scratch, I have started to find bugs. But these are generally not bugs in the IO code, but bugs in Ruby libraries that depend on the way MRI usually works. One of the more annoying ones are IO#read(n), where n is the length you want to read.

This method is not guaranteed to return a string of length n, even if we haven't hit EOF yet. You can NEVER be sure that what you get back is the length you requested. Ever. If you have code that doesn't check the length of the returned string from read, you are almost guaranteed to have a bug just waiting to happen.

Of course, it might work perfectly on your machine and every other machine you test it on. The reason for this is that read(n) will usually return n bytes, but that depends on the socket implementation or file reading implementation of the operating system, it depends on the size of the cache in the network interface, it depends on network latency, and many other things. Please, just make sure to check the return values length before going ahead and using it.

Case in point: net/ldap has this exact problem. Look in net/ber.rb and you will see. There are also two - possibly three - bugs (couple of years old too) that reports different failures because of this.

One thing that makes this problem hard to find is the fact that if you insert debug statement, you will affect the timing in such a way that the code might actually work with debug statement but not without them.

Oh, did I mention that StringIO#read works the same way? It has exactly the same guarantees as IO#read.

After QCon London 2008

2008-03-15T17:21:00.002+01:00

This week has mostly been taken up with QCon London. I spent most of Monday, Wednesday, Thursday and Friday here, and I thought that I'd take the opportunity to write up some of my impressions and thoughts about the sessions I attended.

First, in general the conference definitely didn't disappoint me. It held at least as high standard as I had expected from earlier QCon and JAOO conferences. Solid speakers, a wide range of exciting topics and lots of interesting people made for a grade A conference.

I started out on the Monday with listening to my colleagues Neal Ford, Rebecca J Parsons and Martin Fowler give a tutorial on domain specific languages. I've seen bits and parts of this tutorial before, but seeing as the three speakers are working on evolving it to a full and coherent "pedagogical framework" for teaching DSLs, the current presentation had changed quite a bit since the last time. I really liked it and I recommend it to anyone interested in getting a firm grasp about what DSLs are. Having Rebecca talk about external DSLs in the context of parsers and grammars makes total sense, and the only real thing I would say was a problem was the time allotted to it. Her part of the subject was large enough that 75 minutes felt a bit rushed. Of course, I don't see how Martins or Neals parts could be compressed much more either, so maybe the subject actually is too large for a one day tutorial? Anyway, great stuff.

For several reasons I decided to spend the Tuesday working from the office instead of attending tutorials again.

During the Wednesday I mostly spent my time in the exhibition hall, talking to people and doing general networking. For some reason the tracks I was least interested in had all been scheduled on the same day, so I was lazy and worked on other stuff in the ThoughtWorks booth.

The evening keynote on Wednesday by Martin Fowler and Jim Webber was hilarious, and also managed to get a quite important message across. I had a good time.

Thursday started the session attending for me, beginning with Markus Völters presentation of XText in the DSL track. Highly informative and something that I'll keep in mind if I see something that would be benefit from it. The approach is definitely not for all problem domains, of course.

After that. Venkat Subramaniam gave a talk about how to blend dynamic languages with Java. This talk was useful in explaining why you'd want to do something like this, and why it's such a powerful technique. It also served to set up my talk - which was next in that track - about evolving the Java platform. My talk went well, but I had the timing for it really messed up, so I ran out of material 10 minutes earlier than I expected. Neal Gafter was in the audience and helped out with some corrections during the talk. =)

Finally I headed back to the DSL track and saw Avi Bryant talk about DSLs in Smalltalk and then Magnus Christerson and Henk Kolk talk about the Intentional Workbench. Lots of neat stuff in both of these presentations.

Then there was the speakers dinner... Lots of interesting discussions with lovely people. =)

And then, more quickly than I had expected, the final day of QCon arrived. Me as a Ruby person and programming language nerd had quite a good selection of tracks. I ended up seeing Ted's presentation on F#, which made me feel: wow! Microsoft took ML and did exactly what they've done to all languages on the CLR - added support for .NET objects in the mix. The talk ended with a quite strange discussion about whether F# actually helps with concurrent programming or not, and why a functional programming language has primitives that allow you to have mutable state.

After that I did my talk in the Ruby track, talking about more advanced things regard JRuby. It ended up being great fun, and I spent lots of time in the talk answering questions and showing how seamlessly things work with JRuby. I ended up eating up 10 minutes of everyone's lunch time, but I had a great time and I thing most in the audience had too.

Feeling happy and finished with my contributions, I ended up in the Erlang talk by Joe Armstrong. It gave a quite good overview of why Erlang was created and how it solves some of the problems in that particular problem domain. There is no doubt that Armstrong is an entertaining talker, but his buffoon image gets a little tiring and repetitive after a while. Some of the things that interested me in the talk was missing too. He started out saying that Erlang solves a particular problem, but then expanded that into something that sounded like "Erlang should be used for everything, everywhere". I tried to ask a question related to that, but the answer didn't really go in the direction I was interested in.

I stayed in the languages track and saw the introduction to Scala, which is always fun, except that I'd already learned most of the things showcased. The most interesting about the presentation was the audience interest and questions.

Finally I realized that my contributions were not over at all, since I'd agreed to be part of the closing Ruby panel. This ended up revolving quite a lot around the question whether Ruby and Windows ever will be a good match, if this is important, and if we really want to push Ruby into all kinds of environments.

The closing panel were OK, but nothing special. It ended the day on a good note, but at that time I was tired enough to fall asleep in my chair. For some reason this always happens that last day of conferencing.

Anyway. I had a great time and I look forward to being back the next time. I can definitely recommend QCon as one of the best conferences around in this industry.

I'm at QCon

2008-03-10T14:02:00.001+01:00

I'm attending QCon in London this whole week. I'll give a talk on Thursday called "Evolving the Java platform", talking about different things that might be a part of the next generation Java platform. On Friday I give the talk "JRuby: Power on the JVM", which is a more advanced JRuby talk than the regular introduction talks I usually give.

Hope to see you around!

JRuby Google Tech talk online

2008-03-03T19:30:00.001+01:00

The talk I gave at Google last week is now online: http://youtube.com/watch?v=PfnP-8XbJao

Language design philosophy: more than one way?

2008-02-25T23:12:00.002+01:00

When talking about how dynamic scripting languages are designed, people have a tendency to divide them into "There is more than one way to do it" and "There is one way to do it". Perl is the quintessential example of "more than one way", and Python is the opposite, going so far as to make it impossible to have your own indentation.

These different ways of doing things really divide programmers. Some hate the way Python gives you a bondage strait jacket and no scissors, while some programmers love that they don't need to make any choices and that everyone's code will be equally readable.

From a pure perspective, the Python philosophy seems to be the right one, but it just doesn't work for me. In the same way, I agree with Perl's way of doing things, but the problems that cause in most Perl code is just amazing. Sometimes it feels like the Python way is actually directly a result of someone reacting extremely badly to a Perl code base and decided to never allow that to happen in Python.

So what's the point? Well, the point is Ruby. In fact, Ruby has almost all of the flexibility of Perl to do things in different ways. But at the end of the day, none of the Perl problems tend to show up. Why is this? And why do I feel so comfortable in Ruby's "There is more than one way to do it" philosophy, while the Perl one scares me?

I think it comes down to language design. The Python approach is impossible for the simple reason that what the language designer chooses is going to be the "one way", by fiat. Some people will agree, and some will not. But what I'm seeing in Ruby is that the many ways have been transformed into idioms and guidelines. There are no hard rules, but the community have evolutionary evolved idioms that work and found out many of the ways that doesn't work. This seems to be the right way - if you do the choice as a language designer, you have actually chosen the people that will use your language: that's going to be the persons who doesn't dislike the language designers choices. But if you leave it open enough for evolutionary community design to happen you can actually get the best of both world: both a best way to do things, and something that works for a much larger percentage of the programmer world.

I have come to believe that this is one of the major reasons that Ruby feels so good to me, and is such a good language. And it's a lesson for language designers. When you choose philosophy, make sure to take the possibility of communities and idiom evolution into account.

SF presence

2008-02-20T11:00:00.001+01:00

So, next week and the week after that I'll be in San Francisco. I'll participate in the ThoughtWorks code jam on Tuesday and Wednesday, and do a tech talk at Google sometime by the end of the week.

If someone is interested in meeting up, talking about JRuby, Ruby or programming languages in general, just get in touch.

Is Groovy like Java?

2008-02-16T00:00:00.000+01:00

I usually don't mention the G word in my blog - the results have a tendency to be less than palatable. This time I'm going to make an exception though. The reason is that this meme has been floating around as an argument for Groovy, and I really don't agree with it. In fact, the reason I'm writing about it is because I think the meme in itself generally can be bad for Groovy.

The thing I'm talking about is the saying that Groovy is just like Java. That you can start using Groovy directly without learning anything about Groovy and that basically all Java syntax is valid Groovy.

Now, the last part has never been really true, since there are a few valid Java constructs that are not supported in Groovy (anonymous classes, I'm looking at you). This might be fixed in the 1.5 version. I don't really know, but that's not the point. Since Groovy isn't using the Java parser there will always be a few small corner cases that are not valid in Groovy but Java accepts. This is not a problem, by the way. The only way it can be a problem is when you're using Groovy the wrong way, or you absolutely need to do something that Groovy doesn't support. Since one of the Groovy catch calls is that you shouldn't implement everything in Groovy, this can't really be a concern either. If you're using Groovy correctly, you should just drop down to the Java layer for that particular feature.

Now that that part is out of the way, let's take a look at the other parts of this idea. Namely that you should choose Groovy because it's just like Java, and you can start writing Java code directly in Java. These statements are definitely true, since you can absolutely pick up a language that way. In my mind, though, this misses the whole point of using another language on top of the JVM. If you're building an application and chooses to use Groovy for this instead of Java, isn't the reason that you're doing that the fact that you expect Groovy to give you something Java does not? And if that's the case, doesn't it sound like a disservice to people picking up Groovy to say that they can begin by just programming Java. There is no gain here. Really. In fact, the equivalent program in Groovy and Java will perform much worse on Groovy since there are several layers of dispatching in Groovy that you don't have in Java. The result? An application with exactly the same source code, but running 10 times slower.

Since this message is mostly used in marketing and introduction facilities, the people it's geared at are by definition newbies to the Groovy world. Many of them are totally new to dynamic languages. And the first thing they do is write Java in Groovy, find it slow and obviously, from that point don't understand why you should use Groovy. This is bad for Groovy. Groovy has some extremely powerful features and libraries that can really make your life on the JVM blissful (compared to using Java, for example). Groovy's integration with Java is top notch, which means that you can always fall back to Java when you need Java specific features. If you're using Groovy, I think you should use the full power of the language so you can actually get the full benefit of Groovy.

Soft and hard layers

2008-02-14T23:59:00.002+01:00

I hadn't read this specific c2 post, but it's funny how much of what I write is just a rehash of stuff Cunningham and company came up with 15 years ago. Well. I guess that's why they're the masters and I'm not.

Incidentally, this is my 300th blog post, since starting this blog a little bit more than 2 years ago. Time rushes by, don't it?

Scala exploration status

2008-02-14T23:53:00.002+01:00

Someone asked in a comment where my Scala exploration were going. It's a fair question.

I did a first implementation of the core of the system I was using. Most of if was tested using Specs (which incidentally is very good, and even more importantly - have a responsive creator). Now, I realized a while back that the implementation weren't really using Scala properly. I had mostly used the imperative way of doing things. So I'm planning a rewrite of these parts, using pattern matchin, case classes and more recursive algorithms correctly. It's kinda funny. I didn't expect this to happen since I have so much experience with functional languages. But apparently, the fact that Scala looks like Java in many regards makes it very easy to slip into the Java imperative mode. Not good.

Right now I'm spending some time working on the next release of JtestR, writing on a book, and doing some pesky day time work. =) Also, JRuby needs to have its 1.1 release. I expect to get back to Scala within one or two weeks, though.

Silly Io experiment

2008-02-14T15:32:00.002+01:00

I have known for some time that Io is quite powerful as a language. I wanted to check if it was also suited for creating DSLs. Since I'm lazy I didn't want to create a new DSL just for this, so I decided to appropriate the associations language from ActiveRecord.

The process I followed was this - first come up with something that looks passably nice, then try to implement it. The only requirements was that the names of the associations were saved away somewhere, once per prototype for each model. This is the syntax I came up with (well OK, I didn't work very long on it...):

Post := IoRecord {
has many authors
belongs to blog
belongs to isp
}

Author := IoRecord {
has many blogs
has many posts
has one name
}

Actually, this looks almost readable, right? It's all valid Io code. The question is, how do we get it to do what we want? Without further ado, here's an implementation that gives us enough:

IoRecord := Object clone do(
 init := method(
   self belongings := list
   self hasOnes := list
   self hasManies := list
   self hasFews := list
 )

 appender := method(msg,
   blk := block(
     call sender doMessage(msg) append(call message next name)
     call message setNext(call message next next)
   )
   blk setIsActivatable(true)
 )

 collector := method(
   meths := call argCount / 2
   waiter := Object clone
   for(index, 0, meths-1,
     waiter setSlot(call argAt(index*2) name,
       appender(call argAt(index*2+1)))
   )
   waiter
 )

 belongs := collector(
   to, belongings
 )

 has := collector(
   many, hasManies,
   one, hasOnes
 )

 curlyBrackets := method(
   current := self clone
   call message setName("do")
   current doMessage(call message)
 )
)

Since I'm a bad person and really enjoys metaprogramming, I had to get rid of most of the duplication the first implementation contained. Let's say it like this: the first version didn't have the collector and appender methods. And boy do they make a difference. This is real metaprogramming. Actually, these two methods are actually macros, almost as powerful as Common Lisps. Notice that the words we send in to collector doesn't actually get evaluated. This is one of the reasons we don't need to use symbols - we can just take the unevaluated messages and take their name. In the appender macro we're doing something really funky where we use setNext.

The final effect of that setNext call is that in something like "has many foobar", Io will not try to evaluate foobar at all. In fact, even before Io tries this, we remove the foobar message from the chain and inserts the next message instead.

Oh, right, and the Io lexer actually inserts a synthetic message called "curlyBrackets" when it finds curly brackets. The brackets themselves actually act exactly like parenthesis. You can try this out in Io by changing the closing parenthesis to a closed curly bracket instead - Io doesn't care. Your brackets doesn't have to match up in type, just in cardinality.

Of course, this is just the beginning. Io will blow your mind.

The thing I'm finding myself missing is a good literal syntax for hashes and lists. I'm thinking about implementing something for that. Also, using atPut is starting to get annoying. I want square brackets access. Shouldn't be too hard, since Io does the same thing with square brackets as it does with curly brackets. Almost, at least.

And btw, the Io standard library... Not sure what I think about it. I miss many things from the Ruby core library actually. Now, Io combined with the Ruby libraries, that might be fun?

Java Annotations

2008-02-13T22:16:00.002+01:00

I feel that a few rants are about to burst in me. Let me take this first; this is not a rant, though. Just a very large exclamation:

Java Annotations are NOT a way to implement DSLs, and will never be.

Java annotations are really nice, but seriously, they are NOT manna from heaven.

The upcoming JtestR release

2008-02-09T16:36:00.000+01:00

I'm spending some time working on version 0.2 of JtestR, which has the goal of rounding out the framework, adding some needed things like RSpec story support, test coverage metrics, TestNG running, groupability of tests, support for CC.rb and Expectations.

Now is the time to tell me if there's anything you are missing from JtestR that stops you from using it! Either by a comment here or by the JtestR JIRA at http://jira.codehaus.org/browse/JTESTR.

Thanks.

Ruby.NET closing down

2008-02-05T11:05:00.000+01:00

A letter on the Ruby.NET mailing list just announced that Dr. Wayne Kelly have decided to shut down the Ruby.NET project and instead asks everyone to contribute to IronRuby.

More information can be found on the mailing list thread here.

I don't like these news at all. In many ways having a strong competitor is something that will improve the ecosystem for everyone. Now IronRuby will become the only player on the field - unless other people (like Ted Neward and David Peterson) decide to pick up Ruby.NET. I hope someone does. The .NET world will be better of for it.

The crucial question is not whether we trust John Lam about IronRuby. The question is if we trust Microsoft to do the right thing. Do we?

JNA is really, really good

2008-02-05T10:32:00.001+01:00

The last few months JRuby have used something called JNA to introduce some functionality we couldn't do before. This mainly includes POSIX file operations of different kinds. I've just spent a few days getting a basis for UNIX Domain Sockets working in JRuby, using JNA, and I've gotta say that I'm incredibly impressed by it. For simpler function calls, it just works. For more complicated stuff you might have to set up structures and so on, but it's still incredibly slick. The best part: no native compilation needed.

Of course, this has some performance considerations, but it's still better than nothing.

Denmark, QCon and JavaOne

2008-02-01T21:40:00.000+01:00

Today is my last day in Sweden for a while. It's been an incredible time, and I think that my coworkers from ThoughtWorks have had a wonderful experience in Sweden. What will happen regarding a Swedish office is still up in the air, but I'm cautiously optimistic about it all. In fact, if someone knows a possible client please get in touch. =)

Next week I'm going to Denmark, visiting the castle Hindsgavl where the Danish Java User Group (JavaGruppen) is holding a conference. I will do one presentation on JRuby and one tutorial on JRuby on Rails.

The next planned event will be QCon in London where I will present about some of the more exciting upcoming features in the JVM. That's going to be great fun.

The last week in February I'll be in San Francisco, possibly helping out with some really cool ThoughtWorks events. I'll post more about that when I have the information.

I will also present at Developer Summit in April, in Stockholm.

Finally, expect me to do one presentation and one BoF at JavaOne. I might also do a talk at CommunityOne.

It's going to be a busy spring time.

Polyglot, not panglot or omniglot

2008-01-23T12:31:00.000+01:00

It's interesting. One of the common reactions I've heard to my recent writings about polyglot programming is something I really don't understand. Actually, I've heard the same objection to other persons writing about polyglot.

The objection is that just because I propose polyglot programming - using several different programming languages for different purposes in the same system - I can use whichever language
and as such should not try to find better languages or say that certain languages are bad.

But that's really a confounding of the issue. Just because I can use any language in the dynamic layer doesn't mean I should. In fact, just because polyglot programming as a strategy means you will use more than one language, it is even more important to be careful and use the best languages available for the task. Which is why I'm working to improve JRuby, why I'm evaluating Scala as a replacement for Java, why I'm working on a language based on Io. It's all about using the best languages. I may be a polyglot, but I'm definitely not a panglot or omniglot.

Distributed version control and SVN

2008-01-18T12:57:00.000+01:00

As everyone else has discovered, I've just realized how nice DVCS can be. Of course, I'm part of several open source projects that all happen to use Subversion. I've tried Bazaar, Mercurial, Git and SVK. Of all these, Mercurial is the one I really, really like. But here is the problem: hgsvn is really not up to scratch right now. I tried to pull some stuff from SVN at one project and it immediately errored out. git-svn on the other hand works like a charm. It seems that right now I have to use git, unless someone has a good solution for getting Mercurial really working with SVN. (Also, the fact that hgsvn is pull-only is a bit problematic.)

Advice appreciated.

Ruby antipattern: Using eval without positioning information

2008-01-18T10:23:00.000+01:00

I have noticed that the default way eval, instance_eval(String) and module_eval(String) almost never does what you want unless you supply positioning information. Oh, it will execute all right, and provided you have no bugs in your code, everything is fine. But sooner or later you or someone else will need to debug the code in that eval. In those cases it's highly annoying when the full error message is "(eval):1: redefining constant Foo". Sure, in most cases you can still get all the information. But why not just make sure that everything needed is there to trace the call?

I would recommend changing all places you have eval, or where using instance_eval or module_eval that takes a string argument, into the version that takes a file and line number:

eval("puts 'hello world'")
# becomes
eval("puts 'hello world'", binding, __FILE__, __LINE__)

"str".instance_eval("puts self")
# becomes
"str".instance_eval("puts self", __FILE__, __LINE__)

String.module_eval("A=1")
# becomes
String.module_eval("A=1", __FILE__, __LINE__)

Viability of Java and the stable layer

2008-01-16T14:12:00.000+01:00

This post is basically a follow up to Language explorations. That post had several statements about what I believe, and some of them have been a bit misunderstood, so I'll try to substantiate them a bit more here. Specifically, I want to talk about the viability of the Java language and whether "the stable layer" should be a static or dynamic language.

Lets begin with the stable layer. First, notice that I didn't have any specific definitions for the different layers when I wrote the post. In fact, I think that they are inherently a bit fuzzy. That's OK. DSL's are also quite fuzzy. The attributes I put at the stable layer is first of all that it shouldn't need to change much. In a smaller application, you can actually look at supporting structures as the stable layer, things like application servers or web servers. But for larger applications I find that you usually need your own stable kernel.

The reasons I choose static typing for stable layer is for several reasons. Performance is absolutely one of these, since everything will run on this base it's an advantage if that part is as performant as possible. Secondly, I am no dynamic language, TDD purist that says that static typing is totally unnecessary. Static type checking can be very helpful, but it stops you from having a malleable language. So the kernel should be stable, this means that as much checking as possible is necessary. None of these requirements explicitly preclude a dynamic language, but the specific features of dynamic languages doesn't really come to their rights at this level, and also leading to performance and static type checking suffering.

The more contentious part about my post seems to have been my off hand comment that I don't believe Java is really good for anything. I didn't give much reason in that post, and people have reacted in different ways, from saying that "Java is really good" to "Why are you involved in JRuby if you think Java is so bad?". So lets start with why Java is a bad language. (I am only talking about the language here, not the platform). Also, that Java is a bad doesn't say anything about the worse alternatives, so no comments along the lines of "if you think Java is so bad, why don't you use X instead".

In short:

Java is extremely verbose. This is really a few different problems in one:

No type inference
Verbose generic type signatures
Anonymous class implementations

There is no way to create new kinds of abstractions:

No closures
Anonymous classes are a really clunky way to handle "block" functionality

Broken generics implementation
Language is becoming bloated

A few notes. That Java is becoming bloated is one of those things that people have been blogging about lately. After 1.5 came out, the complexity of the language got very much larger, without actually adding much punch for it. The improvements of generics were mostly negated by the verbosity of them. Fixing all the problems above will bloat the language even more. And at the same time, programming in Java is needlessly painful right now.

So, lets look at the question about why I'm working on JRuby. Well, first, I believe the JVM is a very good place to be at. It's the best platform out there, in my opinion. The libraries are great, the runtime is awesome, and it's available basically everywhere. The bytecodes of the JVM spec is good enough for most purposes. There is some tweaking that can be done (which we are looking at in JSR292), but mostly it's a very nice place. And working on JRuby is really one of the ways I've seen how bad Java as a language is. We are employing several different tricks to get away from the worst parts of it, though. Code generation is used in several places. We are generating byte codes at runtime. We are using annotations to centralize handling of JRuby methods. And we are moving parts of the implementation to Ruby. I believe that JRuby is important because it can run in the same environment as Java, but without the problems of Java.

What are the solutions to the problem with Java? There are basically two different ways. Either define a subset of Java, not necessarily totally compatible, that takes the best parts of Java syntax, does away with the problems and so on. That should be possible, but I don't know anyone who has done it. Or, you can go with an existing static language on the JVM. Here you have two choices - either one of the ports of existing extremely static languages (like ML or Haskell), or you can go with something like Scala. I haven't decided on the best solution here yet. The only thing I'm sure of is that Java in itself is a problem. I'm investigating Scala, but maybe Scala isn't the right answer either. We'll see.

The final, objective truth of Rails versus Grails.

2008-01-15T19:20:00.000+01:00

So it's time to bring out the guns, boxing gloves and whatever martial arts knowledge you possess. If you want the full story, start with Graeme's post here, and drill your way down from there.

I finally lost my patience with this discussion and decided that I should tell you all the truth. The final, totally objective, totally realistic answer to the question of Rails, Grails and Everything.

No, wait. I won't!

And that's it. This discussion is becoming a bit silly. I don't expect someone will see me recommending Grails, and I'm sure we all would be very surprised seeing Graeme recommend JRuby on Rails. So what's the point?

Are you using ResultSetMetaData.getColumnName?

2008-01-08T00:28:00.000+01:00

As the title says, are you using java.sql.ResultSetMetaData.getColumnName in your code? It's interesting, I have done for years, and I didn't know that it was just a bug, waiting to happen.

Until I tried MySQL's 5.1-branch of their JDBC code, I'd always assumed that getColumnName was the right one for generic SQL code. Turns out it isn't. Specifically, it isn't the right one if you're using aliasing in your code. Say you have SELECT Host AS h FROM Host. Now, until the 5.1 branch MySQL JDBC, you would get "h" if you did getColumnName(1) on this result sets metadata. Not so anymore. Now you get "Host". So what should you use? getColumnLabel. It's on the same interface. Until tonight I'd never seen a difference between them. But now there is one - so go through all your JDBC code and make sure you're using the right one here.

Oh, that's right. MySQL 5.0.5 seems to have a bug in multibyte aliasing. So if you alias Host to be a Chinese character, for example, you will not get the same value back from getColumnName or getColumnLabel. I assume this is a bug, since the 5.1-branch seems good.

Emacs Inferior mode for the Io language

2008-01-06T21:51:00.000+01:00

For anyone who like the Io language and work with Emacs, here is a small Inferior mode for Emacs and Io.
It's adapted from the Ruby Inferior mode. Download it here. To use it, just do "(require 'inf-io)" and then M-x run-io.

Enjoy.

Scala magic

2008-01-04T14:43:00.001+01:00

Oh, right. This is a perfect example of Scala magic that I REALLY don't understand yet. If there are any Scala wizards reading, please explain!

This line of code is from a test cases written in specs:

      runtime.ev("\"hello world\"")._with(interpreter)
           must be_==(runtime.stringFor("hello world"))

Runtime is a Scala class and so is interpreter. I added println()'s before and after, and at different methods in this statement. Now, the baffling thing, the really strange thing (at least in my mind), is that the left hand side of the must, gets evaluated THREE times. But this code only run once. How can that happen? What does Scala do here? (And must doesn't seem to be a special method. I looked at the implementation, and it's just a regular method on a trait, mixed into the Scala Object.

Someone please explain this! =)

Keywords in languages

2008-01-04T14:29:00.000+01:00

It's nice to see how the amount of people looking into Scala has really exploded lately, based on the rush of blog posts an discussion.

One of the things I find a bit annoying about Scala is the proliferation of keywords. Actually, this is something I really don't like in any languages. A language should be as keywordless as possible. Of course, such a vision goes against ease of implementation for language implementers, so there always needs to be a balance here. Coming from languages such as Lisp and Io, it's amazing how clear a language can be with a well chosen message passing or invocation model. In fact, both of those languages have zero keywords. That makes it incredibly nice to implement whatever you want.

Actually, Java has been quite good at not taking much more keywords than they had from the beginning, so I found it a bit annoying when I tried to build a fluent interface in Scala, and found out that the word "with" is a keyword. And it's a keyword in the strictest sense, meaning you can't use it in places where you can't use the "with" keyword anyway. So there is no way to implement a method named "with" in Scala. Annoying. It's just, the English connection words are so much more useful for method names, especially when you can use the methods in "operator" position. Then you just want to be able to use all these words.

So. If you design a language, make sure that you take care to actually add every keyword extremely carefully. If you can, make sure that keywords can actually be used for some things when there is no ambiguity. Of course, I'm not proposing the kind of madness you can do in some languages, where a statement such as "IF IF THEN THEN" is valid, since the first IF is a keyword, the next is a variable name, etc. But be reasonable about keywords. They are sometimes necessary, but not always as often as people believe.

Scala testing with specs

2008-01-03T12:46:00.000+01:00

So. The story about unit testing is over for now. I will use specs. Eric made an incredible job and got the JUnit support working with JUnit 4 too very quickly. Today I reintegrated it, and everything works fine.

So, if you want working Ant integration for your Scala testing, I recommend specs.

Of course, this still doesn't explain while all the other alternatives failed so miserably. Hopefully there will be a bit more testing in the community soon. Testing frameworks need competition to evolve well.

It's funny. One of the points to my original post about Scala unit testing was this quote: "Now some lovely Ruby people are looking at Scala, and the very first thing they must do (of course) is write the sacred unit tests:".

I'm not sure about you, but I would say that that's a good statement about Ruby people in general, if that's the way people view us. =)

Sweden visit

2008-01-03T00:33:00.000+01:00

I will be coming to Sweden on Sunday, and I will stay a full 4 weeks. So if anyone feels like grabbing a beer and talking about anything remotely geeky, I'm up for it. Ping me or leave a comment on this post. I'll mostly be in Stockholm, but some time will be spent in Gothenburg too.

Language explorations

2008-01-02T15:07:00.000+01:00

I blogged about looking at languages a while back. At that point I didn't know what my next language to explore would be. I got lots of excellent suggestions. In the end I decided to try OCaML, but gave that up quickly when I found out that half of the type system exists to cover up deficiencies in the other half of it. So I went back and decided to learn Scala. I haven't really had time to start with it though. Until now, that is.

So let's get back to the motivation here? Why do I want to learn another language? Aren't I happy with Ruby? Well, yes and no. But that's not really the point. You can always point to the Prags one-language-a-year, but that's not it either. I mean, it's really good advice, but there is a more urgent reason for me to learn Scala.

I know many people have said this before, but it bears repeating. Everyone doesn't share this opinion, but have a firm belief that the end of big languages is very close. There won't be a next big language. There might be some that are more popular than others, but the way development will happen will be much more divided into using different languages in the same project, where the different languages are suited for different things. This is the whole Polyglot idea. And my take on it is this: the JVM is the best platform there is for Polyglot platform, and I think we will see three language layers emerge in larger applications. Now, the languages won't necessarily be built on top of each other, but they will all run on the JVM.

The first layer is what I called the stable layer. It's not a very large part of the application in terms of functionality. But it's the part that everything else builds on top off, and is as such a very important part of it. This layer is the layer where static type safety will really help. Currently, Java is really the only choice for this layer. More about that later, though.

The second layer is the dynamic layer. This is where maybe half the application code resides. The language types here are predominantly dynamic, strongly typed languages running on the JVM, like JRuby, Rhino and Jython. This is also the layer where I have spent most of my time lately, with JRuby and so on. It's a nice and productive place to be, and obviously, with my fascination for JVM languages, I believe that it's the interplay between this layer and the stable layer that is really powerful.

The third layer is the domain layer. It should be implemented in DSL's, one or many depending on the needs of the system. In most cases it's probably enough to implement it as an internal DSL within the dynamic layer, and in those cases the second and third layer are not as easily distinguishable. But in some cases it's warranted to have an external DSL that can be interacted with. A typical example might be something like a rules engine (like Drools).

I think I realized a long time ago that Java is not a good enough language to implement applications. So I came up with the idea that a dynamic language on top of Java might be enough. But I'm starting to see that Java is not good enough for the stable layer either. In fact, I'm not sure if Java the language is good enough for anything, anymore. So that's what my language exploration is about. I have a suspicion that Scala might be a good language at the stable layer, but at this point the problem is there aren't any other potential languages for that layer. So what I'm doing is trying to investigate if Scala is good enough for that.

But I need to make one thing clear - I don't believe there will be a winner at any of these layers. In fact, I think it would be a clearly bad thing if any one language won at any layer. That means, I'm seeing a future where we have Jython and JRuby and Rhino and several other languages coexisting at the same layer. There doesn't need to be any rivalry or language wars. Similarly, I see even less point in Scala and Ruby being viewed as competing. In my point of view they aren't even on the same continent. And even if they were, I see no point in competing.

I got accused of being "religious" about languages yesterday. That was an interesting way of putting it, since I have always been incredibly motivated to see lots of languages coexisting, but coexisting on the JVM in a productive way.

Does established tools matter or - Is Ant support important?

2008-01-02T13:54:00.000+01:00

This post is a bit of a follow up to my rant about Unit testing in Scala two days back. First let me tell you that that story actually has a happy ending. Eric Torreborne (creator of the Specs framework) immediately stepped up and helped me, and the problem seemed to be a lack of support for JUnit4 test running, which he subsequently implemented. I'm going to reintegrate
specs into my test suite later today. So that makes me happy. I might still retain JtestR. Actually, it would be a bit interesting to see the differences in writing tests in Ruby and Scala.

I spent some time on #scala on FreeNode yesterday. Overall it was an interesting experience. We ended up talking a bit about the unit testing bit, and Ant integration in particular. I'll get back to that conversation later. But this sparked in me the question why I felt that it was really important to have Ant integration. Does it actually matter?

I have kind of assumed that for a tool running on the JVM, that people might need during their build process, integration with Ant and Maven is more or less a must. It doesn't really matter what I think of these tools. If I want anyone to actually use the tool in question, this needs to work. In many cases that means it's not enough to just have a Java class that can be called with the Java-task in Ant. The integration part is a quite small step, but important enough. Or at least I think it is.

I am thinking that no matter what you think of Ant, it's one of those established tools that are here to stay for a long time. I know that I reach for Ant by default when I need something built. I know there are technologically better choices out there. Being a Ruby person, I know that Rake is totally superior, and that there is Raven and Buildr who both provide lots of support for building my project with Ruby. So why do I still reach for Ant whenever I start a new project?

I guess one of the reasons is that I'm almost always building open source projects. I want people to use my stuff, to do things with it. That means the barrier to entry need to be as low as possible. For Java, the de facto standard build system is still Ant, so the chance of people having it installed is good. Ant is easy enough to work with. Sure, it can be painful for larger things, but for smaller projects there really is no problem with Ant.

What do you think? Does it matter if you use established tools, that might be technologically inferior? Or should you always go for the best solution?

Sometimes I'm considering using other tools for build my own personal projects, but I never know enough to say with certainty I will never release it. That means I have two choices - either I use something else first and then convert it if I release it, or I just go with Ant directly.

Now, heading back to that conversation. It started with a comment about "an ant task for failing unit tests is severely overrated". Then it went rapidly downhill to "Haskell shits all over Ant for a build script for example", at which point I totally tuned out. Haskell might be extremely well suited for build scripts, but it's not an established tool that anyone can use from their tool chain. And further, I got two examples of how these build scripts would look, later in the conversation, and both of them were barely better than shell scripts for building and testing. Now there is a very good reason people aren't using shell scripts for their standard building and testing tool chain.

Or am I being totally unreasonable about this? (Note, I haven't in any way defended the technological superiority of Ant in this post, just to make it clear.)

Antlr lexing problem

2008-01-02T13:24:00.000+01:00

I should probably post this on a mailing list instead, but for now I want to document my problem here. If anyone has any good suggestions I'd appreciate it.

I'm using Antlr to lex a language. The language is fixed and has some cumbersome features. One in particular is being really annoying and giving me some trouble to handle neatly with Antlr 3.

This problem is about sorting out Identifiers. Now, to make things really, really simple, an identifier can consist of the letter "s" and the character ":" in any order, in any quantity. An identifier can also be the three operators "=", ":=" and "::=". That is the whole language. It's really easy to handle with whitespace separation and so on. But these are the requirements that give me trouble. The first three are simple baseline examples:

"s" should lex into "s"
"s:" should lex into "s:"
"s::::" should lex into "s::::"
"s:=" should lex into "s" and ":="
"s::=" should lex into "s:" and ":="
etc.

Now, the problem is obviously that any sane way of lexing this will end up eating the last colon too. I can of course use a semantic predicate to make sure this isn't allowed when the last character is a colon and the next is "=". This helps for the 4th case, but not for the 5th.

Anyone care to help? =)

Scala unit testing

2007-12-31T22:53:00.000+01:00

I wish this could be a happy story. But it's really not. If I have made any factual errors in this little rant, don't hesitate to correct me - I would love to be proved wrong about this.

Actually, I wrote this introduction before I went out to celebrate New Years. Now I'm back to finish the story, and the picture have changed a bit. Not enough yet, but we'll see.

Let's tell it from the start. I have this project I've just started working on. It seemed like a fun and quite large thing that I can tinker on in my own time. It also seemed like a perfect match to implement in Scala. I haven't done anything real in Scala yet, and wanted to have a chance to do it. I like everything I've seen about the language itself. I've said so before and I'll say it again. So I decided to use it.

As you all probably know, the first step in a new project is to set up your basic structure and getting all the simple stuff working together. Right, for me that means a simple Ant script that can compile Java and Scala, package it into a jar file, and run unit tests on the code. This was simple. ... Well, except for the testing bit, that is.

It seems there are a few options for testing in Scala. The ones I found was SUnit (included in the Scala distribution), ScUnit, Rehersal and specs (which is based on ScalaCheck, another framework). So these are our contestants.

First, take SUnit -- this is a very small project, no real support for anything spectacular. The syntax kinda stinks. One class for each test? No way. Also, no integration with Ant. I haven't even tried to run this. Testing should be painless, and in this case I feel that using Java would have been an improvement.

ScUnit looked really, really promising. Quite nice syntax, lots of smarts in the framework. I liked what the documentation showed me. It had a custom Ant task and so on. Very nice. It even worked for a simple Hello, World test case. I thought that this was it. So I started writing the starting points for the first test. For some reasons I needed 20 random numbers for this test. Scala has so many ways of achieving this... I think I've tried almost all of them. But all failed with this nice class loading exception just saying InstantiationException on an anonymous function. Lovely. Through some trial and error, I found out that basically any syntax that causes Scala to generate extra classes, ScUnit will fail to run. I have no idea why.

So I gave up and started on the next framework. Rehersal. I have no idea what the misspelling is about. Anyway, this was a no show quite quickly since the Ant test task didn't even load (it referenced scala.CaseClass, which doesn't seem to be in the distribution anymore). Well then.

Finally I found specs and ScalaCheck. Now, these frameworks look mighty good, but they need better Google numbers. Specs also has the problem of being the plural of a quite common word. Not a good recipe for success. So I tried to get it working. Specs is built on top of ScalaCheck, and I much preferred the specs way of doing things (being an RSpec fan boy and all). Now specs doesn't have Ant integration at all, but it does have a JUnit compatibility layer. So I followed the documentation exactly and tried to run it with the Ant JUnit task. KABOOM. "No runnable methods". This is an error message from JUnit4. But as far as I know, I have been able to run JUnit3 classes as good as JUnit4 classes with the same classpath. Hell, JRuby uses JUnit3 syntax. So obviously I have JUnit4 somewhere on my classpath. For the world of me I cannot find it though.

It doesn't really matter. At that point I had spent several hours getting simple unit testing working. I gave up and integrated JtestR. Lovely. Half of my project will now not be Scala. I imagine I would have learned more Scala by writing tests in it, than with the implementation. Apparently not. JtestR took less than a minute to get up and working.

I am not saying anything about Scala the language here. What I am saying is that things like this need to work. The integration points need to be there, especially with Ant. Testing is the most important thing a software developer does. I mean, seriously, no matter what code you write, how do you know it works correctly unless you test it in a repeatable way? It's the only responsible way of coding.

I'm not saying I'm the worlds best coder in any way. I know the Java and Ruby worlds quite well, and I've seen lots of other stuff. But the fact that I can't get any sane testing framework in Scala up and running in several hours, with Ant, tells me that the Scala ecosystem might not be ready for some time.

Now, we'll see what happens with specs. If I get it working I'll use it to test my code. I would love that to happen. I would love to help make it happen - except I haven't learned enough Scala to actually do it yet. Any way or another, I'm not giving up on using Scala for this project. I will see where this leads me. And you can probably expect a series of these first impression posts from me about Scala, since I have a tendency to rant or rave about my experiences.

Happy New Years, people!

JtestR 0.1 released

2007-12-28T22:36:00.000+01:00

If people have wondered, this is what I have been working on in my spare time the last few weeks. But now it's finally released! The first version of JtestR.

So what is it? A library that allows you to easily test your Java code with Ruby libraries.

Homepage: http://jtestr.codehaus.org
Download: http://dist.codehaus.org/jtestr

JtestR 0.1 is the first public release of the JtestR testing tool. JtestR integrates JRuby with several Ruby frameworks to allow painless testing of Java code, using RSpec, Test/Unit, dust and Mocha.

Features:

Integrates with Ant and Maven
Includes JRuby 1.1, Test/Unit, RSpec, dust, Mocha and ActiveSupport
Customizes Mocha so that mocking of any Java class is possible
Background testing server for quick startup of tests
Automatically runs your JUnit codebase as part of the build

Getting started: http://jtestr.codehaus.org/Getting+Started

Team:
Ola Bini - ola.bini@gmail.com
Anda Abramovici - anda.abramovici@gmail.com

Code size and dynamic languages

2007-12-24T15:11:00.000+01:00

I've had a fun time the last week noting the reactions to Steve Yegge's latest post (Code's Worst Enemy). Now, Yegge always manages to write stuff that generate interesting - and in some cases insane - comments. This time, the results are actually quite a bit more aligned. I'm seeing several trends, the largest being that having generated a 500K LOC code base in the first case is a sin against mankind. The second one being that you should never have one code base that's so large, it should be modularized into several hundreds of smaller projects/modules. The third reaction is that Yegge should be using Scala for the rewrite.

Now, from my perspective I don't really care that he managed to generate that large of a code base. I think any programmer could fall down the same tar pit, especially if it's over a large amount of time. Secondly, you don't need to be one programmer to get this problem. I would wager that there are millions of heinous code bases like this, all over the place. So my reaction is rather the pragmatic one: how do you actually handle the situation if you find yourself in it? Provided you understand the whole project and have the time to rewrite it, how should it be done? The first step in my opinion, would probably be to not do it alone. The second step would be to do it in small steps, replacing small parts of the system while writing unit tests while going.

But at the end of the day, maybe a totally new approach is needed. So that's where Yegge chooses to go with Rhino for implementation language. Now, if I would have tackled the same problem, I would never reimplement the whole application in Rhino - rather, it would be more interesting to try to find the obvious place where the system needs to be dynamic and split it there, keep those parts in Java and then implement the new functionality on top of the stable Java layer. Emacs comes to mind as a typical example, where the base parts are implemented in C, but most of the actual functionality is implemented in Emacs Lisp.

The choice of language is something that Stevey gets a lot of comments about. People just can't seem to understand why it has to be a dynamic language. (This is another rant, but people who comment on Stevey's blog seems to have a real hard time distinguishing between static typing and strong typing. Interesting that.) So, one reason is obviously that Stevey prefers dynamic typing. Another is that hotswapping code is one of those intrinsic features of dynamic languages that are really useful, especially in a game. The compilation stage just gets in the way at that level, especially if we're talking something that's going to live for a long time, and hopefully not have any down time. I understand why Scala doesn't cut it in this case. As good as Scala is, it's good exactly because it has a fair amount of static features. These are things that are extremely nice for certain applications, but it doesn't fit the top level of a system that needs to be malleable. In fact, I'm getting more and more certain that Scala needs to replace Java, as the semi stable layer beneath a dynamic language, but that's yet another rant. At the end of it, something like Java needs to be there - so why not make that thing be a better Java?

I didn't see too many comments about Stevey's ideas about refactoring and design patterns. Now, refactoring is a highly useful technique in dynamic languages too. And I believe Stevey is wrong saying that refactorings almost always increase the code size. The standard refactorings tend to cause that in a language like Java, but that's more because of the language. Refactoring in itself is really just a systematic way of making small, safe changes to a code base. The end result of refactoring is usually a cleaner code base, better understanding of that code base, and easier code to read. As such, they are as applicable to dynamic languages as to static ones.

Design patterns are another matter. I believe they serve two purposes - the first and more important being communication. Patterns make it easier to to understand and communicate high level features of a code base. But the second purpose is to make up for deficiencies in the language, and that's mostly what people see when talking about design patterns. When you're moving in a language like Lisp, where most design patterns are already in the language, you tend to not need them for communication as much either. Since the language itself provides ways of creating new abstractions, you can use those directly, instead of using design patterns to create "artificial abstractions".

As a typical example of a case where a design pattern is totally invisible due to language design, take a look at the Factory. Now, Ruby has factories. In fact, they are all over the place. Lets take a very typical example. The Class.new method that you use to create new instances of a class. New is just a factory method. In fact, you can reimplement new yourself:

class Class
def new(*args)
  object = self.allocate
  object.send :initialize, *args
  object
end
end

You could drop this code into any Ruby project, and everything would continue to work like before. That's because the new-method is just a regular method. The behavior of it can be changed. You can create a custom new method that returns different objects based on something:

class Werewolf;end
class Wolf;end
class Man;end

class << Werewolf
 def new(*args)
   object = if $phase_of_the_moon == :full
              Wolf.allocate
            else
              Man.allocate
            end
   object.send :initialize, *args
   object
 end
end

$phase_of_the_moon = :half
p Werewolf.new

$phase_of_the_moon = :full
p Werewolf.new

Here, creating a new Werewolf will give you either an instance of Man or Wolf depending on the phase of the moon. So in this case we are actually creating and returning something from new that is not even sub classes of Werewolf. So new is just a factory method. Of course, the one lesson we should all take from Factory, is that if you can, you should name your things better than "new". And since there is no difference between new and other methods in Ruby, you should definitely make sure that creating objects uses the right name.

Ruby closures and memory usage

2007-12-21T17:05:00.001+01:00

You might have seen the trend - I've been spending time looking at memory usage in situations with larger applications. Specifically the things I've been looking at is mostly about deployments where a large number of JRuby runtimes is needed - but don't let that scare you. This information is exactly as applicable for regular Ruby as for JRuby.

One of the things that can really cause unintended high memory usage in Ruby programs is long lived blocks that close over things you might not intend. Remember, a closure actually has to close over all local variables, the surrounding blocks and also the living self at that moment.

Say that you have an object of some kind that has a method that returns a Proc. This proc will get saved somewhere and live for a long time - maybe even becoming a method with define_method:

class Factory
def create_something
 proc { puts "Hello World" }
end
end

block = Factory.new.create_something

Notice that this block doesn't even care about the actual environment it's created in. But as long as the variable block is still live, or something else points to the same Proc instance, the Factory instance will also stay alive. Think about a situation where you have an ActiveRecord instance of some kind that returns a Proc. Not an uncommon situation in medium to large applications. But the side effect will be that all the instance variables (and ActiveRecord objects usually have a few) and local variables will never disappear. No matter what you do in the block. Now, as I see it, there are really three different kinds of blocks in Ruby code:

Blocks that process something without needing access to variables outside. (Stuff like [1,2,3,4,5].select {|n| n%2 == 0} doesn't need closure at all)
Blocks that process or does something based on living variables.
Blocks that need to change variables on the outside.

What's interesting is that 1 and 2 are much more common than 3. I would imagine that this is because number 3 is really bad design in many cases. There are situations where it's really useful, but you can get really far with the first two alternatives.

So, if you're seeing yourself using long lived blocks that might leak memory, consider isolating the creation of them in as small of a scope as possible. The best way to do that is something like this:

o = Object.new
class << o
def create_something
  proc { puts "Hello World" }
end
end
block = o.create_something

Obviously, this is overkill if you don't know that the block needs to be long lived and it will capture things it shouldn't. The way it works is simple - just define a new clean Object instance, define a singleton method in that instance, and use that singleton method to create the block. The only things that will be captured will be the "o" instance. Since "o" doesn't have any instance variables that's fine, and the only local variables captured will be the one in the scope of the create_something method - which in this case doesn't have any.

Of course, if you actually need values from the outside, you can be selective and onle scope the values you actually need - unless you have to change them, of course:

o = Object.new
class << o
 def create_something(v, v2)
   proc { puts "#{v} #{v2}" }
 end
end
v = "hello"
v2 = "world"
v3 = "foobar" #will not be captured by the block
block = o.create_something(v, v2)

In this case, only "v" and "v2" will be available to the block, through the usage of regular method arguments.

This way of defining blocks is a bit heavy weight, but absolutely necessary in some cases. It's also the best way to get a blank slate binding, if you need that. Actually, to get a blank slate, you also need to remove all the Object methods from the "o" instance, and ActiveSupport have a library for blank slates. But this is the idea behind it.

It might seem stupid to care about memory at all in these days, but higher memory usage is one of the prices we pay for higher language abstractions. It's wasteful to take it too far though.

ThoughtWorks is looking at Sweden

2007-12-19T19:46:00.000+01:00

I am not sure how well it comes across in my blog posts, but joining ThoughtWorks have been the best move of my life. I can't really describe what a wonderful place this is to be (for me at least). I sometimes try - in person, after a few beers - but I always end up not being able to capture the real feeling of working for a company that is more than a company.

I'm happy at being with ThoughtWorks. It's as simple as that - I feel like I've found my home.

So imagine how happy I am to tell you that ThoughtWorks is exploring opportunities for an office in Sweden!

Now, I am one of the persons involved in this effort, and we have been talking about it for a while (actually, we started talking about it for real not long after I joined). But now it's reality. The first trips to Sweden will be in January. ThoughtWorks will be sponsoring JFokus (which is shaping up to be a really good conference, by the way. I'm happy to have been presenting there the first year). We will have a few representatives at JFokus, of course. I will be there, for example. =)

Of course, exploring Sweden is not the same thing as saying that an office will actually happen. But we think there are good reasons to at least consider it. I personally think it would be a perfect fit, but I am a bit biased about it.

So what are we doing for exploration? Well, of course we have started to look into business opportunities and possible clients. We are looking at partnerships and collaboration. We are looking at potential recruits. But really, the most important thing at this stage is to talk to people, get a feeling for the lay of the land, get to know interesting folks that can give us advice and so on. And that is what our travels in January will be about.

So. Do you feel you might fit any of the categories of people above? We'd love to meet you and talk - very informally. So get in touch.

This is exciting times for us!

Your Ruby tests are memory leaks

2007-12-19T14:51:00.000+01:00

The title says it all. The only reason you haven't noticed, is that you probably aren't working on a large enough application, or have enough tests. But the fact is, Test::Unit leaks memory. Of course, that's to be expected if it is going to be able to report results. But that leak should be more or less linear.

That is not the case. So. To make this concrete, lets take a look at a test that exhibits the problem:

class LargeTest < Test::Unit::TestCase
def setup
  @large1 = ["foobar"] * 1000
  @large2 = ["fruxy"] * 1000
end

1000_000.times do |n|
  define_method :"test_abc#{n}" do
    assert true
  end
end
end

This is obviously fabricated. The important details are these: the setup method will create two semi large objects and assign them to instance variables. This is a common pattern in many test suites - you want to have the same objects created, so you assign them to instance variables. In most cases there is some tear down associated, but I rarely see teardown that includes assigning nil to the instance variables. Now, this will run one million tests, with one million setup calls. Not only that - the way Test::Unit works, it will actually create one million LargeTest instances. Each of those instances will have those two instance variables defined. Now, if you take a look at your test suites, you probably have less than one million tests all over. You also probably don't have that large objects all over the place. But remember, it's the object graph that counts. If you have a small object that refers to something else, the whole referral chain will be stopped from garbage collection.

... Or God forbid - if you have a closure somewhere inside of that stuff. Closures are a good way to leak lots of memory, if they aren't collected. The way the structures work, they refer to many things all over the place. Leaking closures will kill your application.

What's the solution? Well, the good one would be for test unit to change it's implementation of TestCase.run to remove all instance variables after teardown. Lacking that, something like this will do it:

class Test::Unit::TestCase
 NEEDED_INSTANCE_VARIABLES = %w(@loaded_fixtures @_assertion_wrapped @fixture_cache @test_passed @method_name)

 def teardown_instance_variables
   teardown_real
   instance_variables.each do |name|
     unless NEEDED_INSTANCE_VARIABLES.include?(name)
       instance_variable_set name, nil
     end
   end
 end

 def teardown_real; end
 alias teardown teardown_instance_variables

 def self.method_added(name)
   if name == :teardown && !@__inside
     alias_method :teardown_real, :teardown
     @__inside = true
     alias_method :teardown, :teardown_instance_variables
     @__inside = false
   end
 end
end

This code will make sure that all instance variables except for those that Test::Unit needs will be removed at teardown time. That means the instances will still be there, but no memory will be leaked for the things you're using. Much better, but at the end of the day, I feel that the approach Test::Unit uses is dangerous. At some point, this probably needs to be fixed for real.

Joda Time

2007-12-18T15:12:00.000+01:00

I spent a few hours this weekend converting RubyTime in JRuby to use Joda Time instead of Calendar or Date. That was a very nice experience actually. I'm incredibly impressed by Joda, and overall I think it was worth adding a new dependency to JRuby for this. The API is very nice, and immutability in these classes make things so much easier.

There were a few things I got a bit annoyed at though. First, that Joda is ISO 8601 compliant is a really good thing, but I missed the functionality to tune a few things. Stuff like saying which weekday a week should start on, for the calculation of current week would be very nice. As it is right now, that functionality has to use Calendar. It might be in Joda, but I couldn't find it.

The other thing I had a problem with - and this actually made me a bit annoyed - was how Joda handles GMT and UTC. Now, it says clearly in the documentation that Joda works with the UTC concept, and that GMT is not exactly the same thing. So why is it this code passes (if assertNotEquals is assumed):

    public void testJodaStrangeNess() {
       assertEquals(DateTimeZone.UTC, DateTimeZone.forID("UTC"));
       assertEquals(DateTimeZone.UTC, DateTimeZone.forID("GMT"));
       assertEquals(DateTimeZone.UTC, DateTimeZone.forOffsetHours(0));
       assertNotEquals(DateTimeZone.UTC, DateTimeZone.forID("Etc/GMT"));
       assertNotEquals(DateTimeZone.forID("GMT"), DateTimeZone.forID("Etc/GMT"));
   }

Yeah, you're reading it right - UTC and GMT is the same time zone. +00:00 is the same as UTC too. But Etc/GMT is not the same as UTC or GMT or +00:00. Isn't that a bit strange?

JavaPolis report

2007-12-14T17:26:00.000+01:00

Earlier today I attended the last sessions for this years JavaPolis. This was the first time I attended, and I've been incredibly impressed by it. The whole conference have been very good.

I arrived on Monday, sneaking in on Brian Leonard and Charlies JRuby tutorial. I didn't see much of it though, and after that me and Charles had to prepare our session a bit, so no BOFs.

Tuesday I slept late (being sick and all), and then saw Jim Weavers JavaFX tutorial, which was very adept. I feel I have a fairly good grasp of the capabilities of Java FX Script now, at least. There were a few BOFs I wanted to go to that evening, but since the speaker dinner/open bar was that night, I obviously choose that. Cue getting to bed at 3am, after getting home to the hotel from... uhm. somewhere in or around Antwerpen.

On Wednesday, the real conference started. My first session was the Groovy Update. I always enjoy seeing presentations of other language implementations, partly because I'm a language geek, but also because everyone has a very different presentation style that I like to contrast with each other. One thing I noticed about the Groovy presentation was that much of it was spent comparing Groovy to "other" languages.

Right, after that I saw two quickies - the first one about IntelliJ's new support for JRuby. And yes, this is support for JRuby, not just Ruby. You can use IntelliJ to navigate from Ruby code to Java code, where you have used that Java code in your Ruby. It looks really promising actually, and I spent some time showing the presenter a few things more that could be included. I don't know of any IDE that supports JRuby specific things like that, actually.

After that I saw Dick Wall's presentation on GWT. Since I have actually managed to avoid any knowledge about GWT, it was kinda interesting.

The next sessions didn't seem too interesting, so I worked a bit more on my presentation, and walked around talking to people.

Charles and my presentation went quite well, even though I managed to tank all the demonstrations quite heavily. For some reason I actually locked JIRB in comment mode, and couldn't get out of it, and then I fell upon the block coercion bug that happens when you call a Java method that is overloaded so that one takes no arguments and another overload takes the interface you want to coerce into. Charles didn't stop me until afterwards... =)

But yes, it went well. Lots of people in the audience, and lots of interest.

The final session of the day was the future of computing panel, with Gosling, Bloch, Gafter and Odersky. To be honest, I found it boring - Quinn was moderator, but didn't really manage to get the panel as enthused about anything.

After that, it was BOF time, I sat in on the Adobe one to pass the time, but didn't learn anything spectacular. The Groovy BOF was nice - it's always fun to see lots of code.

I started Thursday with the Scala presentation. Now, I didn't learn anything I didn't know here, but it was still a very good presentation. And oh, I found out that there is a Scala book on the way. (It's actually available as a Rough Cut from Artima. Very nice.)

The next session was supposed to be Blochs Effective Java, but he used to spot to rant about the BGGA closures proposal instead. Of course, Joshua Bloch always rants in a very entertaining way, and he had chosen insidiously good examples for his point of view - but I'm still not convinced.

The Java Posse live show was good fun. After that I managed to see Bob Lee's Web Beans presentation, and then the one on JAX-RS. Doesn't really have much to say about those two. Except... am I the only one who starts getting bored by annotations all over the place?

The day was nearly over, and then it was time for BOF's. The main difference being that it was time for the JRuby BOF. All went well, except that Charles didn't show up, I didn't have a projector the first half of the BOF, Tom introduced a bug on Wednesday that made all my examples fail, and so on. A huge thanks to Damian Steer who saved me by keeping the audience entertained while I fixed the bug in front of everyone.

I sat through Chet Haase's talk about Update N, but didn't pay that much attention since I was hacking on JRuby.

Finally, it was time for the BOF on other new language features in Java, with Gafter and Bloch. This was actually very interesting stuff. It ended up being almost 2 hours. But I think most people got their fill of new language syntax in it. The question is, which parts are good? I particularly didn't like method extensions. All the proposals seems to lose the runtime component of it, and in that case it just stops being interesting. I would much rather see the language add real categories or something like that.

Friday was a lazy day. I sat in on the OGIi presentation and the TDD one, but nothing really exciting there either.

So that's my JavaPolis week. It's been a good time. And now I think it's time to have some more beers with JRuby people before moving out from here.

JDBC and DDL

2007-12-14T17:07:00.000+01:00

It would really be time for JDBC to add support for database agnostic DDL. This is still one of the more gross areas of many database libraries (just look at dialects in Hibernate). Most of it is actually caused by DDL. But at the end of the day, most of the operations supported are actually exactly the same. Am I the only one who thinks it would be nice to have programmatic access to DDL operations?

JavaPolis

2007-12-09T13:06:00.001+01:00

Tomorrow I'm going to JavaPolis - me and Charles are presenting on JRuby on Rails on Wednesday. We'll be there the whole week so if you wanna get in touch, don't hesitate. We're aiming to taste many nice Belgian beers.

Except for that, it's actually quite quiet right now. There are some things in the works which will soon be announced, though.

AspectJ and JRuby?

2007-12-06T15:34:00.000+01:00

This is one of those idea-posts. There is no implementation and no code. But if someone wants to take the idea and do something with it, go ahead.

The gist of it is this: what if you could implement the actions for AspectJ in Ruby? You could define an on-load pointcut that matches everything and dispatches to Ruby. From there on you could do basically anything from the Ruby side - including dynamically changing the stuff happening. Of course, there would be a performance cost, but it could be incredibly useful for debugging, when you don't really want to restart your application and recompile every time you want to change the implementation of the aspect code.

ThoughtWorks calling Ruby developers in San Francisco

2007-11-27T18:49:00.000+01:00

Friends! ThoughtWorks is hiring Ruby developers all over the world, but right now San Francisco is the hottest place to be. So if you're located in the Bay Area and want to work with Ruby and Rails, don't hesitate to make contact.

The time since I joined ThoughtWorks about 6 months ago have been the best of my life, and of all our offices around the world, I like the San Francisco one best. ThoughtWorks is really the home for people passionate about development and people who love Ruby.

If you have been following my blog, you know about Oracle Mix and other interesting things we have been doing out of the SF office. And there's more to come - exciting times!

Could ThoughtWorks in San Francisco be your home? Take contact with recruiting here: http://www.thoughtworks.com/work-for-us/apply-online.html, or email me directly and I'll see to it that your information gets to the right place!

Joni merged to JRuby trunk

2007-11-27T12:47:00.000+01:00

This is a glorious day! Joni (Marcin's incredible Java port of the Oniguruma regexp engine) has been merged to JRuby trunk. It seems to work really well right now.

I did some initial testing, and the Petstore numbers are more or less the same as before, actually. This is explained by the fact that I did the integration quite quick and tried to get stuff working without concern for performance. We will go through the implementations and tune them for Joni soon, and this will absolutely give JRuby a valuable boost.

Marcin is also continuing to improve Joni performance, so over all this is a very nice approach.

Happy merge day!

JRuby regular expression update

2007-11-25T20:45:00.000+01:00

It's been some time since I wrote about what's happening in JRuby trunk right now, and what we're working on. The reason is I've been really boring. All my time I've spent on Regular Expressions and the REJ implementation. Well, that's ended now. After Marcin got the Oniguruma port close enough, we are both focusing on that instead. REJ's implementation had some fundamental problems that would make it really hard to get better performance. In this regard, Joni is a better implementation. Also, Marcin is incredible at optimization so if everything goes as planned, we're looking at better general Regular Expression performance, better compatibility and a much more competent implementation.

And boy am I bored by this now. =) I'd really like to get back to fixing bugs and get JRuby ready for the next release. That might happen soon, though - I've spent the weekend getting Joni integrated with JRuby inside a branch and today reached the goal of getting everything to compile. Also, easier programs run, like jirb. Our test suite fails, though, so there are still things to do. But getting everything compiling and ditching JRegex is a major point on the way of replacing JRegex in JRuby core. It shouldn't be too far off, and I think it will be fair to say we will have Joni in JRuby 1.1. Actually, 1.1 is really going to be an awesome release.

The development of Oracle Mix

2007-11-22T15:19:00.000+01:00

Rich Manalang just posted a very nice entry on the Oracle AppsLab about the technology behind Oracle Mix, how we developed it and so on. Read it here.

Accumulators in Ruby

2007-11-20T20:51:00.000+01:00

So, me and Ben Butler-Cole discussed the fact that accumulators in Ruby isn't really done in the obvious way. This is due to the somewhat annoying feature of Ruby that nested method definitions with the def-keyword isn't lexically scoped, so you can't implement an internal accumulator like you would in Python, Lisp, Haskell or any other languages like that.

I've seen three different ways to handle this in Ruby code. To illustrate, let's take the classic example of reversing a list. The functional way of doing this is to define an internal accumulator, this takes care of making the implementation tail recursive, and very efficient on linked list.

So, the task is to reverse a list in a functional, recursive approach. First version, using optional arguments:

class Reverser
def reverse(list, index = list.length-1, result = [])
 return result if index == -1
 result << list[index]
 reverse(list, index - 1, result)
end
end

So, this one uses two default arguments, which makes it very easy to reuse the same method in the recursive case. The problem here is that the optional arguments expose an implementation detail which the caller really has no need of knowing. The implementation is simple but it puts more burden on the caller. This is also the pattern I see in most places in Ruby code. From a design perspective it's not really that great.

So, the next solution is to just define a private accumulator method:

class Reverser
def reverse(list)
  reverse_accumulator(list, list.length-1, [])
end

private
def reverse_accumulator(list, index, result)
  return result if index == -1
  result << list[index]
  reverse_accumulator(list, index - 1, result)
end
end

This is probably in many cases the preferable solution. It makes the interface easier, but adds to the class namespace. To be sure, the responsibility for the implementation of an algorithm should ideally belong at the same place. With this solution you might have it spread out all over the place. Which brings us to the original problem - you can't define lexically scoped methods within another method. So, in the third solution I make use of the fact that you can actually have recursive block invocations:

class Reverser
 def reverse(list)
   (rec = lambda do |index, result|
      return result if index == -1
      result << list[index]
      rec[index - 1, result]
    end)[list.length-1, []]
 end
end

The good thing about this implementation is that we avoid the added burden of both a divided implementation and an exposed implementation. It might seem a bit more complex to read if you're not familiar with the pattern. Remember that [] is an alias for the call method on Procs. Also, since the assignment to rec happens in a static scope we can actually refer to it from inside the block and get the right value. Finally, all assignments return the assigned value which means that we can just enclose everything in parens and apply it directly. Another neat aspect of this is that since the block is a closure, we don't need to pass the list variable around anymore.

Does anyone have a better solution on how to handle this? Accumulators aren't really that common in Ruby - is this a result of Ruby making functional programming unneat, or is it just don't needed?

A new language

2007-11-19T20:22:00.000+01:00

So, it's that time of the year again. The restlessness flows over me. I feel cold and numb. And no, it's not because I live in London - it's because I need the warmth of learning a new language.

Now, I want something I can actually get into and learn. I've tried to get into OCaml, but I gotta admit I hate the type system. I have no problem with bondage static typed languages (Haskell's type system is really nice, for example) but OCaml's really feels like half of it exists just to cover up holes in the other half. There seems to be a large overlap in functionality, and lots of workarounds for handling things that should be simple.

I'm half way into Erlang, but for several reasons the language feels very primitive.

I've kinda thought about maybe getting serious with Scala. I like many of the language features, it's a nicely designed language and so on. But - hear this, people - I would love to get away from the JVM for a while, just for the sake of it. I can do Scala later. I actually have a medium sized project lined up for my Scala learning. But not right now.

So, what do I want? Something I haven't touched before. I would love something that involves radically new language features, if there are any left to discover. I have no need for it to be static or dynamic specifically. Doesn't really matter. It would be fun if it's new, but if it's old, good and still in use in some sectors that would be fun too. Specifically something that's not mainly run on the JVM or CLR. And of course, not any of the "mainstream" languages, who I actually tend to know fairly well (and yeah, to my sorrow that includes the whole W-family...).

Please help me! Give this December new meaning for me. I promise, if someone comes up with a nice language to try out, I'll be very fair to it when I evaluate and learn it. =)

Ruby memory leaks

2007-11-19T19:24:00.001+01:00

They aren't really common, but they do exist. As with any other garbage collected language, you can still be susceptible to memory leaks. In many cases they can also be very insidious. Say that you have a really large Rails application. After some time it grinds to a halt, CPU bound in GC. It may not even be a leak, it could just be something that creates so much garbage that the collector cannot take care of it.

I gotta admit, I'm not sure how to find such a problem. After getting histograms of objects, and trying to profile it, maybe run with ruby-debug, I would be out of options. Maybe some kind of shotgun technique - shutting down parts of the application, trying to pinpoint the location of the problem.

Now, ordinarily, that would have been the end of my search. A failure. Or maybe several weeks of trying to read through the sources.

The alternative? Run the application in JRuby. See if the same memory leak shows up (remember, it might be a bad interaction with MRI's runtime system that gives you grief. Or maybe even a bug in MRI Garbage Collector). But if it doesn't go away, you're in luck. Wait until the CPU starts chugging for real, and then take a heap dump using the jmap Java SDK tool. Once that's done, you'll be sitting with a large honking binary file that you can't do much with. The standard way of reading it is through jhat, but that don't give much to go on.

But then I found this wonderful tool called SAP Memory Analyzer. Google it and download it. It's marvelous. Easily the best heap analyzer I've run across in a long time. It's only flaw is that it runs in Eclipse... But well, it can't be everything, right?

Once you've opened up the file in SAP, you can do pretty much everything. It's quite self explanatory. The way I usually go about things is to use the core option, and then choose "find_leak". That almost always gives me some good suspects that I can continue investigating. From there on it's just to drill down and find out exactly what's going on.

Tell me if you can do that in any way as easy as that with MRI. I would love to know. But right now, JRuby is kicking butt in this regard.

Oracle developers on OSX unite!

2007-11-18T21:46:00.000+01:00

All my ranting aside, Oracle RDBMS is pretty good. It's got good performance, and lots of features you really need in a database. I shan't proclaim it my favorite database, but it's definitely something I have no problem working with. Except for that one small detail...

Yeah, you guessed it. Oracle support on Mac OS X is kinda... nonexistent. The best solution I've come up with is to run Parallels with a Windows or Linux instance and run Oracle XE inside of that. But that only works if I want to use the JDBC thing driver. OCI development? You're screwed. And the Parallels route isn't exactly painless either. Especially from a performance point of view.

So what do we need? OCI8 precompiled binaries would be a good start. But in the end, the only workable solution for all developers on OSX in the world who wants to be able to use Oracle is a compatible Oracle XE for Intel OS X. It shouldn't really be to hard, right? It's just a BSD beneath the covers...

Anyway, it's kinda interesting. If you're a consultant or a developer, OS X is definitely the superior platform. That's a fact (well, except for Java 6...). The lack of Oracle support forces people to develop their application against Postgres and then let continuous integration - you are using CI, right? - tell you if you made any Oracle-unfriendly mistakes. That doesn't really sound to professional.

So, go on and vote for this in Oracle Mix. The links are here: https://mix.oracle.com/ideas/we-need-the-oracle-clients-oci-jdbc-for-the-apple-intel-osx-platform,
https://mix.oracle.com/ideas/compile-oracle-xe-for-intel-os-x

Oracle Mix has launched

2007-11-12T08:12:00.000+01:00

The last 5 weeks, a team consisting of me, Alexey Verkhovsky, Matt Wastrodowski and Toby Tripp from ThoughtWorks, and Rich Manalang from Oracle have created a new application based on an internal Oracle application. This site is called Oracle Mix, and is aimed to be the way Oracles customers communicate with Oracle and each other, suggesting ideas, answering each others questions and generally networking.

Why is this a huge deal? Well, for me personally it's really kinda cool... It's the first public JRuby on Rails site in existance. It's deployed on the "red stack": Oracle Enterprise Linux, Oracle Application Server, Oracle Database, Oracle SSO, Oracle Internet Directory. And JRuby on Rails.

It's cool. Go check it out: http://mix.oracle.com.

QCon San Francisco recap

2007-11-12T04:22:00.000+01:00

Last week I attended QCon San Francisco, a conference organized by InfoQ and Trifork (the company behind JAOO). It must admit that I was very positively surprised. I had expected it to be good, but I was blown away by the quality of most presentations. The conference had a system where you rated sessions by handing in a green, yellow or red card - I think I handed in two yellow cards, and the rest was green.

Everything started out with tutorials. I didn't go to the first tutorial day, but the second day tutorial was my colleagues Martin Fowler and Neal Ford talking about Domain Specific Languages, so I decided to attend that. All in all it was lots of very interesting material. Sadly, I managed to get slightly food poisoned from the lunch, so I didn't stay the whole day out.

On Wednesday, Kent Beck started the conference proper with a really good keynote on why Agile development really isn't anything else than the way the world expects software development to happen nowadays. It's clear to see that the Agile way provides many of the ilities that we have a responsibility to deliver. A very good talk.

After that Richard Gabriel delivered an extremely interesting presentation on how to think about ultralarge, self sustaining systems, and how we must shift the way we think about software to be able to handle large challenges like this.

The afternoons sessions was dominated by Brian Goetz extremely accomplished presentation on concurrency. I really liked seeing most of the knowledge available right now into a 45 minute presentation, discussion most of the things we as programmers need to think about regarding concurrency. I am so glad other people are concentrating on these hard problems, though - concurrency scares me.

The panel on the future of Java was interesting, albeit I didn't really agree with some of the conclusions Rod Johnson and Josh Bloch arrived at.

The day was capped by Richard Gabriel doing a keynote called 50 in 50. I'm not sure keynote is the right word. A poem, maybe? Or just a performance. It was very memorable, though. And beautiful. It's interesting that you can apply that word to something that discusses different programming languages, but there you have it.

During the Thursday I was lazy and didn't attend as many sessions as I did on the Wednesday. I saw Charles doing the JRuby presentation, Neal Ford discussing DSLs again, and my coworker Jim Webber rant about REST, SOA and WDSL. (Highly amusing, but beneath the hilarious surface Jim definitely had something very important to say about how we build Internet applications. I totally agree. Read his blog for more info.)

The Friday was also very good, but I missed the session about Second Life architecture which seemed very interesting. Justin Gehtland talked about CAS and OpenID in Rails, both solutions that I think is really important, and have their place in basically any organization. Something he said that rang especially true with me is that a Single Sign-On architecture isn't just about security - it's a way to make it easier to refactor your applications, giving you the possibility to combine or separate applications at will. Very good. Although it was scary to see the code the Ruby CAS server uses to generate token IDs. (Hint, it's very easy to attack that part of the server.

Just to strike a balance I had to satisfy my language geekery by attending Erik Meijer's presentation on C#. It was real good fun, and Erik didn't get annoyed at the fact that me and Josh Graham interrupted him after more or less every sentence, with new questions.

Finally, I saw half of Obie's talk about the new REST support in Rails 2.0 (and he gave me a preview copy of his book - review forthcoming). There is lots of stuff there that can really make your application so much easier to code. Nice.

The day ended with two panels, first me, Charles, Josh Susser, Obie and James Cox talking about Rails, the future of the framework and some about the FUD that inevitably happens.

The final panel was Martin Fowler moderating me, Erik Meijer, Aino Vonge Corry and Dan Pritchett, talking about the things we had seen at the conference. The discussion ranged from large scale architecture down to concurrency implementations. Hopefully the audience were satisfied.

All in all, an incredibly good time.

JRuby 1.0.2 released

2007-11-01T17:45:00.000+01:00

The JRuby community is pleased to announce the release of JRuby 1.0.2.

Homepage: http://www.jruby.org/
Download: http://dist.codehaus.org/jruby/


JRuby 1.0.2 is a minor release of our stable 1.0 branch. The fixes in this
release include primarily obvious compatibility issues that we felt were
low risk. We periodically push out point releases to continue supporting
production users of JRuby 1.0.x.

Highlights:
- Fixed several nasty issues for users on Windows
- Fixed a number of network compatibility issues
- Includes support for Rails 1.2.5
- Reduced memory footprint
- Improved File IO performance
- trap() fix
- 99 total issues resolved since JRuby 1.0.1

Special thanks to the new JRuby contributors who rose to Charlie's challenge
to write patches for some outstanding bugs: Riley Lynch, Mathias Biilmann
Christensen, Peter Brant, and Niels Bech Nielsen.  Welcome aboard...

An interesting memory leak in JRuby

2007-10-31T19:50:00.000+01:00

The last two days I had lots of fun with the interesting task of finding a major memory leak in JRuby. The only way I could reliably reproduce it was by running Mingle's test suite and see memory being eaten. I tried several approaches, the first being using jhat to analyze the heap dumps. That didn't really help me much, since all the interesting queries I tried to run with OQL had a tendency to just cause out of memory errors. Not nice.

Next step was to install SAP Memory Analyzer which actually worked really well, even though it's built on top of Eclipse. After several false starts, including one where I thought we had found the memory leak I finally got somewhere. Actually, I did find a memory leak in our method cache implementation. But alas, after fixing that it was obvious there was another leak in there.

I finally got SAP to tell me that RubyClasses was being retained. But when I tried to find the root chain to see how that happened I couldn't see anything strange. In fact, what I saw what the normal chaining of frames, blocks, classes and other interesting parts. And this is really the problem when debugging this kind of problem in JRuby. Since a leak almost always be leaking several different objects, it can be hard to pinpoint the exact problem. In this case I guess that the problem was in a large branch that Bill merged a few weeks back, so I tried going back to it and checking. Alas, the branch was good. In fact, since I went back 200 revisions I finally knew within which range the problem had to be. Since I couldn't find anything more from the heap dumps I resorted to the venerable tradition of binary search. Namely going through the revisions and finding the faulty one. According to log2, I would find the bad revision in less than 8 tries, so I started out.

After I while I actually found the problem. Let me show it to you here:

def __jtrap(*args, &block)
 sig = args.first
 sig = SIGNALS[sig] if sig.kind_of?(Fixnum)
 sig = sig.to_s.sub(/^SIG(.+)/,'\1')
 signal_class = Java::sun.misc.Signal
 signal_class.send :attr_accessor, :prev_handler
 signal_object = signal_class.new(sig) rescue nil
 return unless signal_object
 signal_handler = Java::sun.misc.SignalHandler.impl do
   begin
     block.call
   rescue Exception => e
     Thread.main.raise(e) rescue nil
   ensure
     # re-register the handler
      signal_class.handle(signal_object, signal_handler)
   end
 end
 signal_object.prev_handler = signal_class.handle(signal_object, signal_handler)
end

This is part of our signal handling code. Interestingly enough, I was nonplussed. How could trap leak? I mean, noone actually calls trap enough times to make it leak, right?

Well, wrong. Actually, it seems that ActiveRecord traps abort in transactions and then restore the original handler. So each transaction created new trap handlers. That would have been fine, except for the last line. In effect, in the current signal handler we save a reference to the previous signal handler. After a few iterations we will have a long chain of signal handlers, all pointing back, all holding a hard reference from one of the single static root sets in the JVM (namely, the list of all signal handlers). That isn't so bad though. Except a saved block has references to dynamic scopes (which reference variables). It has a reference to the Frame, and the Frame has references to RubyClass. RubyClass has references to method objects, and method objects have in some cases references to RubyProcs, which in turn have more references to Blocks. At the end, we have a massive leak.

The solution? To simple remove saving of the previous handler and simplify the signal handler.

QCon and OpenWorld

2007-10-31T19:46:00.000+01:00

As mentioned before I will be in San Francisco next week for QCon, and the week after that for Oracle OpenWorld. I will be part of a panel debate at QCon and man a booth on Oracle OpenWorld. In fact, if you're attending OpenWorld you should visit ThoughtWorks booth at 343 Moscone South. Looking forward to seeing you there.

Michael Gira and Tim Bray

2007-10-28T00:49:00.000+02:00

Doesn't Tim Bray and Michael Gira look very much alike? See for yourself:

I'm not sure if Tim has the voice of Michael, though. Anyway, I gotta say, warming up with preparty with Bill Hicks, and then a 90 minutes Michael Gira show was awesome.

Is this the wrong blog for this? Yeah, probably, but the Tim Bray inclusion warrants it. =)

Current state of Regular Expressions

2007-10-26T14:36:00.000+02:00

As I've made clear earlier, the current regular expression situation has once again become impractical. To reiterate the history: We began with regular Java regex support. This started to cave in when we found out that the algorithm used is actually recursive, and fails for some common regexps used inside Rails among others. To fix that, we integrated JRegex instead. That's the engine 1.0 was released with and is still the engine in use. It works fairly well, and is fast for a Java engine. But not fast enough. In particular, there is no support for searching for exact strings and failing fast, and the engine requires us to transform our byte[]-strings to char[] or String. Not exactly optimal. Another problem is that compatibility with MRI suffers, especially in the multi byte support.

There are two solutions currently on the way. Core developer Marcin are working on a port of the 1.9 regexp engine Oniguruma. This port still has some way to go, and is not integrated with JRuby. The other effort is called REJ, and is a port of the MRI engine I did a few months back. I've freshened up the work and integrated it with JRuby in a branch. At the moment this work actually seems to go quite well, but there are some snags.

First of all, let me point out that this approach gives us more or less total multibyte compatibility for 1.8, which is quite nice.

When doing benchmarking, I'm generally using Rails as the bar. I have a series of regular expressions that Petstore uses for each requst, and I'm using these to check performance. As a first datapoint, JRuby+REJ is faster at parsing regexps than JRuby trunk for basically all regexps. This ranges from slightly faster to twice as fast.

Most of the Rails regexen are actually faster in REJ than in JRuby+trunk, but the problem is that some of them are actually quite a bit slower. 4 of the 22 Rails regexps are slower, by between 20 and 250% percent. There are also this one: /.*_f/ =~ "_fxxxxxxxxxxxxxxxxxxxxxxx" which basically runs about 10x slower than JRuby trunk. Not nice at all.

In the end, the problem is backtracking. Since REJ is a straight port of the MRI code, the backtracking is also ported. But it seems that Java is unusually bad at handling that specific algorithm, and it performs quite badly. At the moment I'm continuing to look at it and trying to improve performance in all ways possible, so we'll see what happens. Charles Nutter have also started to look at it.

But what's really interesting is that I reran my Petstore benchmarks with the current REJ code. To rehash, my last results with JRuby trunk looked like this:

controller :   1.804000   0.000000   1.804000 (  1.804000)
view       :   5.510000   0.000000   5.510000 (  5.510000)
full action:  13.876000   0.000000  13.876000 ( 13.876000)

But the results from rerunning with REJ was interesting, to say the least. I expected bad results because of the bad backtracking performance, but it seems the other speed improvements weigh up:

controller :   1.782000   0.000000   1.782000 (  1.782000)
view       :   4.735000   0.000000   4.735000 (  4.735000)
full action:  12.727000   0.000000  12.727000 ( 12.727000)

As you can see, the improvement is quite large in the view numbers. It is also almost there compared to MRI which had 4.57. Finally, the full action is better by a full second too. Again, MRI is 9.57s and JRuby 12.72. It's getting closer. I am quite optimistic right now, provided that we manage to fix the remaining problems with backtracking, our regexp engine might well be a great boon to performance.

Interesting times in JRuby land

2007-10-26T13:24:00.001+02:00

The last week or two have been quite interesting. We are finally starting to see performance numbers that seem good enough. Take a look at Nick's posts here and here for more information on this. The second post contains some valuable tips. In particular, make sure to turn off ObjectSpace and run with -server. Both of these will improve your performance and scalability quite much.

Secondly, JRuby on Rails on Oracle Application Server. Nice, huh? I would imagine more interesting things coming out of all this.

But the end message seems to be that JRuby is really ready now. The 1.1 release looks like it's going to be something really amazing. I can't wait!

Updated JRuby on Rails performance numbers

2007-10-16T13:25:00.000+02:00

So, after my last post, several of us have spent time looking at different parts of Rails and JRuby performance. We have managed to improve things quite nicely since my last performance note. Some of the things changed have been JRuby's each_line implementation, JRuby's split implementation, a few other improvements, and some small fixes to AR-JDBC. After that, here are some new numbers for Petstore. Remember, the MRI numbers are for MRI 1.8.6 with native C MySQL. The JRuby numbers is with ActiveRecord-JDBC trunk and MySQL. (I'm only showing the best numbers of each). The number in bold is the most important one for comparison.

MRI   controller :   1.000000   0.070000   1.070000 (  1.430260)
JRuby controller :   1.804000   0.000000   1.804000 (  1.804000)

MRI   view       :   4.410000   0.150000   4.560000 (  4.575399)
JRuby view       :   5.510000   0.000000   5.510000 (  5.510000)

MRI   full action:   8.260000   0.410000   8.670000 (  9.574460)
JRuby full action:  13.876000   0.000000  13.876000 ( 13.876000)

As you can see, we are talking about 9.5s MRI to 13.8s for JRuby, which I find is a quite nice achievement if you look at the numbers from Friday. We are inching closer and closer. Both the view and the controller numbers are looking very nice. This is actually indicative of a nice trend - since general JRuby primitive performance is really good, the slowness in our Regular Expression engine is weighed up by much faster execution speed.

Once the port of Oniguruma lands, this story will almost certainly look very different. But even so, this is looking good.

JRuby discovery number one

2007-10-14T17:24:00.000+02:00

After my last entry I've spent lots of time checking different parts of JRuby, trying to find the one true bottleneck for Rails. Of course, I still haven't found it (otherwise I would have said YAY in the subject for this blog). But I have found a few things - for example, symbols are slow right now, but Bill's work will make them better. And it doesn't affect Rails performance at all.

But the discovery I made was when I looked at the performance of the regular expressions used in Rails. There are exactly 50 of them for each request, so I did a script that checked the performance of each of them against MRI. And I found that there was one in particular that had really interesting performance when comparing MRI to JRuby. In fact, it was between 200 and a 1000 times slower. What's worse, the performance wasn't linear.

So which regular expression was the culprit? Well, /.*?\n/m. That doesn't look to bad. And in fact, this expression displayed not one, but two problems with JRuby. The first one is that any regular expression engine should be able to fail fast on something like this, simply because there is a string that always needs to be part of a string for this expression to match. In MRI, that part of the engine is called bm_search, and is a very fast way to fail. JRuby doesn't have that. Marcin is working on a port of Oniguruma though, so that will fix that part of the problem.

But wait, if you grep for this regexp in the Rails sources you won't find it. So where was it actually used? Here is the kicker: it was used in JRuby's implementation of String#each_line. So, let's take some time to look at a quick benchmark for each_line:

require 'benchmark'

str = "Content-Type: text/html; charset=utf-8\r\nSet-Cookie: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa "

TIMES=100_000

puts "each_line on small string with several lines"
10.times do
puts(Benchmark.measure{TIMES.times { str.each_line{} }})
end

str = "abc" * 15

puts "each_line on short string with no line divisions"
10.times do
puts(Benchmark.measure{TIMES.times { str.each_line{} }})
end

str = "abc" * 4000

puts "each_line on large string with no line divisions"
10.times do
puts(Benchmark.measure{TIMES.times { str.each_line{} }})
end

As you can see, we simple measure the performance of doing a 100 000 each_line calls on three different strings. The first one is a short string with several newlines, the second is a short string with no newlines, and the last is a long string with no newlines. How does MRI run this benchmark?

each_line on small string with several lines
 0.160000   0.000000   0.160000 (  0.157664)
 0.150000   0.000000   0.150000 (  0.160450)
 0.160000   0.000000   0.160000 (  0.171563)
 0.150000   0.000000   0.150000 (  0.157854)
 0.150000   0.000000   0.150000 (  0.154578)
 0.150000   0.000000   0.150000 (  0.154547)
 0.160000   0.000000   0.160000 (  0.158894)
 0.150000   0.000000   0.150000 (  0.158064)
 0.150000   0.010000   0.160000 (  0.156975)
 0.160000   0.000000   0.160000 (  0.156857)
each_line on short string with no line divisions
 0.080000   0.000000   0.080000 (  0.086789)
 0.090000   0.000000   0.090000 (  0.084559)
 0.080000   0.000000   0.080000 (  0.093477)
 0.090000   0.000000   0.090000 (  0.084700)
 0.080000   0.000000   0.080000 (  0.089917)
 0.090000   0.000000   0.090000 (  0.084176)
 0.080000   0.000000   0.080000 (  0.086735)
 0.090000   0.000000   0.090000 (  0.085536)
 0.080000   0.000000   0.080000 (  0.084668)
 0.090000   0.000000   0.090000 (  0.090176)
each_line on large string with no line divisions
 3.350000   0.020000   3.370000 (  3.404514)
 3.330000   0.020000   3.350000 (  3.690576)
 3.320000   0.040000   3.360000 (  3.851804)
 3.320000   0.020000   3.340000 (  3.651748)
 3.340000   0.020000   3.360000 (  3.478186)
 3.340000   0.020000   3.360000 (  3.447704)
 3.330000   0.020000   3.350000 (  3.448651)
 3.350000   0.010000   3.360000 (  3.489842)
 3.350000   0.020000   3.370000 (  3.429135)
 3.350000   0.010000   3.360000 (  3.372925)

OK, this looks reasonable. The large string is obviously taking more time to search, but not incredibly much time. What about trunk JRuby?

each_line on small string with several lines
32.668000   0.000000  32.668000 ( 32.668000)
30.785000   0.000000  30.785000 ( 30.785000)
30.824000   0.000000  30.824000 ( 30.824000)
30.878000   0.000000  30.878000 ( 30.877000)
30.904000   0.000000  30.904000 ( 30.904000)
30.826000   0.000000  30.826000 ( 30.826000)
30.550000   0.000000  30.550000 ( 30.550000)
32.331000   0.000000  32.331000 ( 32.331000)
30.971000   0.000000  30.971000 ( 30.971000)
30.537000   0.000000  30.537000 ( 30.537000)
each_line on short string with no line divisions
 7.472000   0.000000   7.472000 (  7.472000)
 7.350000   0.000000   7.350000 (  7.350000)
 7.516000   0.000000   7.516000 (  7.516000)
 7.252000   0.000000   7.252000 (  7.252000)
 7.313000   0.000000   7.313000 (  7.313000)
 7.262000   0.000000   7.262000 (  7.262000)
 7.383000   0.000000   7.383000 (  7.383000)
 7.786000   0.000000   7.786000 (  7.786000)
 7.583000   0.000000   7.583000 (  7.583000)
 7.529000   0.000000   7.529000 (  7.529000)
each_line on large string with no line divisions

Ooops. That doesn't look so good... And also, where is the last ten lines? Eh... It's still running. It's been running for two hours to produce the first line. That means that it's taking at least 7200 seconds which is more than 2400 times slower than MRI. But in fact, since the matching of the regular expression above is not linear, but exponential in performance, I don't expect this to ever finish.

There are a few interesting lessons to take away from this exercise:

There may still be implementation problems like this in many parts of JRuby - performance will improve by quite much every time we find something like this. I haven't measured Rails performance after this is fixed, and I don't expect it to actually fix the whole problem, but I think I'll see better numbers.
Understand regular expressions. Why is /.*?\n/ so incredibly bad for strings over a certain length? In this case it's the combination of .* and ?. What would be a better implementation in almost all cases? /[^\n]\n/. Notice that there is no backtracking in this implementation, and because of that, this regexp will have performance O(n) while the earlier one was O(n^2). Learn and know these things. They are the difference between usage and expertise.

Mystery: An exposé on JRuby performance

2007-10-12T11:40:00.000+02:00

So, after Charlies awesome work yesterday (documented here), I felt it was time I put the story straight on general JRuby performance. It's something of a mystery. I'll begin by showing you the results of lots of different benchmarks and comparison to MRI. All these benchmarks have been run on my MacBook Pro, dual 2.33Ghz cores with 2Gb memory. JRuby revision is 4578, Java is Apple 1.6 beta, and Ruby version is 1.8.6 (2007-03-13 patchlevel 0). JRuby was ran with -J-server and -O.

Now, lets begin with the YARV benchmarks. The first values is always MRI, the second i JRuby. I've made the benchmark red when MRI is faster, and green when JRuby is faster.

bm_app_answer.rb:
0.480000   0.010000   0.490000 (  0.511641)
0.479000   0.000000   0.479000 (  0.478000)
bm_app_factorial.rb:
ERROR stack level too deep
2.687000   0.000000   2.687000 (  2.687000)
bm_app_fib.rb:
5.880000   0.030000   5.910000 (  6.151324)
2.687000   0.000000   2.687000 (  2.687000)
bm_app_mandelbrot.rb:
1.930000   0.010000   1.940000 (  1.992984)
2.752000   0.000000   2.752000 (  2.876000)
bm_app_pentomino.rb:
84.160000   0.450000  84.610000 ( 88.205519)
77.117000   0.000000  77.117000 ( 77.117000)
bm_app_raise.rb:
2.600000   0.430000   3.030000 (  3.169156)
2.162000   0.000000   2.162000 (  2.162000)
bm_app_strconcat.rb:
1.390000   0.010000   1.400000 (  1.427766)
1.003000   0.000000   1.003000 (  1.003000)
bm_app_tak.rb:
7.570000   0.060000   7.630000 (  7.888381)
2.676000   0.000000   2.676000 (  2.676000)
bm_app_tarai.rb:
6.020000   0.030000   6.050000 (  6.186971)
2.236000   0.000000   2.236000 (  2.236000)
bm_loop_times.rb:
4.240000   0.020000   4.260000 (  4.404826)
3.354000   0.000000   3.354000 (  3.354000)
bm_loop_whileloop.rb:
9.450000   0.050000   9.500000 (  9.678552)
5.037000   0.000000   5.037000 (  5.037000)
bm_loop_whileloop2.rb:
1.890000   0.010000   1.900000 (  1.936502)
1.039000   0.000000   1.039000 (  1.039000)
bm_so_ackermann.rb:
ERROR stack level too deep
4.928000   0.000000   4.928000 (  4.927000)
bm_so_array.rb:
5.580000   0.020000   5.600000 (  5.709101)
5.552000   0.000000   5.552000 (  5.552000)
bm_so_concatenate.rb:
1.580000   0.040000   1.620000 (  1.647592)
1.602000   0.000000   1.602000 (  1.602000)
bm_so_exception.rb:
4.170000   0.390000   4.560000 (  4.597234)
4.683000   0.000000   4.683000 (  4.683000)
bm_so_lists.rb:
0.970000   0.030000   1.000000 (  1.036678)
0.814000   0.000000   0.814000 (  0.814000)
bm_so_matrix.rb:
1.700000   0.010000   1.710000 (  1.765739)
1.878000   0.000000   1.878000 (  1.879000)
bm_so_nested_loop.rb:
5.130000   0.020000   5.150000 (  5.258066)
4.661000   0.000000   4.661000 (  4.661000)
bm_so_object.rb:
5.480000   0.030000   5.510000 (  5.615154)
3.095000   0.000000   3.095000 (  3.095000)
bm_so_random.rb:
1.760000   0.010000   1.770000 (  1.806116)
1.495000   0.000000   1.495000 (  1.495000)
bm_so_sieve.rb:
0.680000   0.010000   0.690000 (  0.705296)
0.853000   0.000000   0.853000 (  0.853000)
bm_vm1_block.rb:
19.920000   0.110000  20.030000 ( 20.547236)
12.698000   0.000000  12.698000 ( 12.698000)
bm_vm1_const.rb:
15.720000   0.080000  15.800000 ( 16.426734)
7.654000   0.000000   7.654000 (  7.654000)
bm_vm1_ensure.rb:
14.530000   0.070000  14.600000 ( 15.137106)
7.588000   0.000000   7.588000 (  7.589000)
bm_vm1_length.rb:
17.230000   0.090000  17.320000 ( 20.406438)
6.415000   0.000000   6.415000 (  6.416000)
bm_vm1_rescue.rb:
11.520000   0.040000  11.560000 ( 11.736435)
5.604000   0.000000   5.604000 (  5.603000)
bm_vm1_simplereturn.rb:
17.560000   0.080000  17.640000 ( 18.178065)
5.413000   0.000000   5.413000 (  5.413000)
bm_vm1_swap.rb:
22.160000   0.110000  22.270000 ( 22.698746)
6.836000   0.000000   6.836000 (  6.835000)
bm_vm2_array.rb:
5.600000   0.020000   5.620000 (  5.675354)
1.844000   0.000000   1.844000 (  1.844000)
bm_vm2_method.rb:
9.800000   0.030000   9.830000 (  9.918884)
5.152000   0.000000   5.152000 (  5.151000)
bm_vm2_poly_method.rb:
13.570000   0.050000  13.620000 ( 13.803066)
10.289000   0.000000  10.289000 ( 10.289000)
bm_vm2_poly_method_ov.rb:
3.990000   0.010000   4.000000 (  4.071277)
1.750000   0.000000   1.750000 (  1.749000)
bm_vm2_proc.rb:
5.670000   0.020000   5.690000 (  5.723124)
3.267000   0.000000   3.267000 (  3.267000)
bm_vm2_regexp.rb:
0.380000   0.000000   0.380000 (  0.387671)
0.961000   0.000000   0.961000 (  0.961000)
bm_vm2_send.rb:
3.720000   0.010000   3.730000 (  3.748266)
2.135000   0.000000   2.135000 (  2.136000)
bm_vm2_super.rb:
4.100000   0.010000   4.110000 (  4.138355)
1.781000   0.000000   1.781000 (  1.781000)
bm_vm2_unif1.rb:
3.320000   0.010000   3.330000 (  3.348069)
1.385000   0.000000   1.385000 (  1.385000)
bm_vm2_zsuper.rb:
4.810000   0.020000   4.830000 (  4.856368)
2.920000   0.000000   2.920000 (  2.921000)
bm_vm3_thread_create_join.rb:
0.000000   0.000000   0.000000 (  0.006621)
0.368000   0.000000   0.368000 (  0.368000)

What is interesting about these numbers is that almost all of them are way faster, and the ones that are slower are so by a quite narrow margin (except for the regexp and thread tests).

So, this is the baseline against MRI. Let's take a look at a few other benchmarks. These can all be found in JRuby, in the test/bench directory. When you run them, all of them generate 5 or 10 runs of all measures, but I've simply taken the best one for each. The repetition is to allow Java to warm up. Here are a few different benchmarks, same convention and running parameters as above:

bench_fib_recursive.rb
----------------------
  1.390000   0.000000   1.390000 (  1.412710)
0.532000   0.000000   0.532000 (  0.532000)

bench_method_dispatch.rb
------------------------
Control: 1m loops accessing a local variable 100 times
2.830000   0.010000   2.840000 (  2.864822)
0.105000   0.000000   0.105000 (  0.105000)
Test STI: 1m loops accessing a fixnum var and calling to_i 100 times
10.000000   0.030000  10.030000 ( 10.100846)
2.111000   0.000000   2.111000 (  2.111000)
Test ruby method: 1m loops calling self's foo 100 times
16.130000   0.060000  16.190000 ( 16.359876)
7.971000   0.000000   7.971000 (  7.971000)

bench_method_dispatch_only.rb
-----------------------------
Test ruby method: 100k loops calling self's foo 100 times
1.570000   0.000000   1.570000 (  1.588715)
0.587000   0.000000   0.587000 (  0.587000)

bench_block_invocation.rb
-------------------------
1m loops yielding a fixnum 10 times to a block that just retrieves dvar
2.800000   0.010000   2.810000 (  2.822425)
1.194000   0.000000   1.194000 (  1.194000)
1m loops yielding two fixnums 10 times to block accessing one
6.550000   0.030000   6.580000 (  6.623452)
2.064000   0.000000   2.064000 (  2.064000)
1m loops yielding three fixnums 10 times to block accessing one
7.390000   0.020000   7.410000 (  7.467841)
2.120000   0.000000   2.120000 (  2.120000)
1m loops yielding three fixnums 10 times to block splatting and accessing them
9.250000   0.040000   9.290000 (  9.339131)
2.451000   0.000000   2.451000 (  2.451000)
1m loops yielding a fixnums 10 times to block with just a fixnum (no vars)
1.890000   0.000000   1.890000 (  1.908501)
1.278000   0.000000   1.278000 (  1.277000)
1m loops calling a method with a fixnum that just returns it
2.740000   0.010000   2.750000 (  2.766255)
1.426000   0.000000   1.426000 (  1.426000)

bench_string_ops.rb
----
Measure string array sort time
5.950000   0.060000   6.010000 (  6.055483)
8.061000   0.000000   8.061000 (  8.061000)
Measure hash put time
0.390000   0.010000   0.400000 (  0.398593)
0.208000   0.000000   0.208000 (  0.209000)
Measure hash get time (note: not same scale as put test)
1.620000   0.000000   1.620000 (  1.646155)
0.740000   0.000000   0.740000 (  0.740000)
Measure string == comparison time
2.340000   0.010000   2.350000 (  2.368579)
0.812000   0.000000   0.812000 (  0.812000)
Measure string == comparison time, different last pos
2.690000   0.000000   2.690000 (  2.724772)
0.860000   0.000000   0.860000 (  0.860000)
Measure string <=> comparison time
2.340000   0.010000   2.350000 (  2.369915)
0.824000   0.000000   0.824000 (  0.824000)
Measure 'string'.index(fixnum) time
0.790000   0.010000   0.800000 (  0.808189)
1.113000   0.000000   1.113000 (  1.113000)
Measure 'string'.index(str) time
2.860000   0.010000   2.870000 (  2.892730)
0.956000   0.000000   0.956000 (  0.956000)
Measure 'string'.rindex(fixnum) time
0.800000   0.000000   0.800000 (  0.817300)
0.631000   0.000000   0.631000 (  0.631000)
Measure 'string'.rindex(str) time
12.190000   0.040000  12.230000 ( 12.310492)
1.247000   0.000000   1.247000 (  1.247000)

bench_ivar_access.rb
----
100k * 100 ivar gets, 1 ivar
0.500000   0.000000   0.500000 (  0.582682)
0.340000   0.000000   0.340000 (  0.340000)
100k * 100 ivar sets, 1 ivar
0.700000   0.010000   0.710000 (  0.816724)
0.402000   0.000000   0.402000 (  0.401000)
100k * 100 attr gets, 1 ivar
0.970000   0.000000   0.970000 (  0.988212)
0.875000   0.000000   0.875000 (  0.874000)
100k * 100 attr sets, 1 ivar
1.390000   0.010000   1.400000 (  1.406535)
1.114000   0.000000   1.114000 (  1.114000)
100k * 100 ivar gets, 2 ivars
0.490000   0.000000   0.490000 (  0.506206)
0.344000   0.000000   0.344000 (  0.344000)
100k * 100 ivar sets, 2 ivars
0.680000   0.000000   0.680000 (  0.693064)
0.388000   0.000000   0.388000 (  0.388000)
100k * 100 attr gets, 2 ivars
0.970000   0.000000   0.970000 (  0.989313)
0.878000   0.000000   0.878000 (  0.878000)
100k * 100 attr sets, 2 ivars
1.400000   0.000000   1.400000 (  1.434206)
1.129000   0.000000   1.129000 (  1.128000)
100k * 100 ivar gets, 4 ivars
0.490000   0.000000   0.490000 (  0.502097)
0.340000   0.000000   0.340000 (  0.340000)
100k * 100 ivar sets, 4 ivars
0.690000   0.000000   0.690000 (  0.696852)
0.389000   0.000000   0.389000 (  0.389000)
100k * 100 attr gets, 4 ivars
0.970000   0.010000   0.980000 (  0.986163)
0.872000   0.000000   0.872000 (  0.872000)
100k * 100 attr sets, 4 ivars
1.370000   0.010000   1.380000 (  1.394921)
1.128000   0.000000   1.128000 (  1.128000)
100k * 100 ivar gets, 8 ivars
0.500000   0.000000   0.500000 (  0.519511)
0.344000   0.000000   0.344000 (  0.344000)
100k * 100 ivar sets, 8 ivars
0.690000   0.000000   0.690000 (  0.710896)
0.389000   0.000000   0.389000 (  0.389000)
100k * 100 attr gets, 8 ivars
0.970000   0.000000   0.970000 (  0.987582)
0.870000   0.000000   0.870000 (  0.870000)
100k * 100 attr sets, 8 ivars
1.380000   0.000000   1.380000 (  1.400542)
1.132000   0.000000   1.132000 (  1.132000)
100k * 100 ivar gets, 16 ivars
0.500000   0.000000   0.500000 (  0.523690)
0.342000   0.000000   0.342000 (  0.343000)
100k * 100 ivar sets, 16 ivars
0.680000   0.000000   0.680000 (  0.707385)
0.391000   0.000000   0.391000 (  0.391000)
100k * 100 attr gets, 16 ivars
0.970000   0.010000   0.980000 (  1.017880)
0.879000   0.000000   0.879000 (  0.879000)
100k * 100 attr sets, 16 ivars
1.370000   0.010000   1.380000 (  1.387713)
1.128000   0.000000   1.128000 (  1.128000)

bench_for_loop.rb
----
100k calls to a method containing 5x a for loop over a 10-element range
0.890000   0.000000   0.890000 (  0.917563)
0.654000   0.000000   0.654000 (  0.654000)

All of this is really good, of course. Lots of green. And the red parts are not that bad either.

But here is the mystery. General Rails performance sucks. This test case can be ran by checking out petstore from tw-commons at RubyForge. There is a file called script/console_bench that will run the benchmarks. The two commands ran was: ruby script/console_bench production, and jruby -J-server -O script/console_bench ar_jdbc. With further ado, here are the numbers:

controller :   1.000000   0.070000   1.070000 (  1.472701)
controller :   2.144000   0.000000   2.144000 (  2.144000)
view       :   4.470000   0.160000   4.630000 (  4.794336)
view       :   6.335000   0.000000   6.335000 (  6.335000)
full action:   8.300000   0.400000   8.700000 (  9.597351)
full action:  16.055000   0.000000  16.055000 ( 16.055000)

These numbers plainly stink. And we can't find the reason for it. As you can see, most benchmarks are much better, and there is nothing we can think of that would add up to a general degradation like this. It can't be IO since these tests use app.get directly. So what is it? There is very likely to be one or two bottlenecks causing this problem, and when we have found it, Rails will most likely blaze along. But profiling haven't helped yet. We found a few things, but there is still stuff missing. It's an interesting problem.

My current thesis is that either symbols or regexps are responsible. I'll spend the day checking that.

Recent interviews

2007-10-12T10:08:00.000+02:00

I have been doing several interviews recently, mostly in connection with my book. Here are the ones I know of right now: