Me at Øredev

I was fortunate to get to go to Øredev 2009 last week, and wanted to share a couple of impressions with you. The picture on the left is me with my Øredev passport, after a very large espresso from one of the Oracle-sponsored baristas.

Øredev is a developer conference held each year in Malmö. This years’ event took place between November 4^th and 6^th. It is not a technology-specific event, but covers a wide range of topics such as .Net, Java, Agile ways, testing, project management, and web/cloud/mobile development. Talks range from concrete hands-on topics such as how to work with a specific technology, to more theoretical subjects on software development and computer science. Speakers are brought in from around the world, and include prominent figures such as Jeff Sutherland (“father” of Scrum), Cameron Purdy (Vice President of Development at Oracle), Dan North (agile thinker and consultant at ThoughtWorks), James Coplien (agile development and consulting guru) and Trygve Reenskaug (OO guru and “father” of the MVC pattern).

Øredev 2009 conference map showing tracks

I think this mix is what makes the conference so interesting – you can discuss the nitty gritty details of your favorite framework with likeminded peers over a cup of coffee  in the morning and discuss paradigms of software development with someone else with your afternoon tea, regardless of what technology they usually work with.

Over the course of 3 days I attended 18 talks and 4 keynotes, which again was only a small part of what the conference offered. The following are some of the things that caught my attention, in words and images, and admittedly not an attempt at “fair” coverage. If your favorite topic is missing, wait for the videos to come online at http://www.oredev.org/, or even better – go see for yourself next year!

Efficiency vs. effectiveness

Maybe the best coffee at any conference?

The theme this year was efficiency, and several of the keynotes and talks pointed out why it’s important to differentiate between efficiency and effectiveness. While efficiency is concerned with how fast or well you can perform any specific task or sub-goal, effectiveness is concerned with how that contributes to reaching an overall goal. They argued that while many of the practices used in agile methods such as Scrum and XP can make us good at “local” efficiency, it doesn’t help if we’re busy building the wrong things, or have a bottleneck somewhere else in the value chain.

It seems part of the current trend among agile thinkers is finding ways to supplement the software development practices from Scrum, XP and other agile methods with core concepts from Lean development, thereby emphasizing not just doing, but also thinking, and not just local efficiency but effectiveness towards the overall goal and value chain. As part of his “Why your agile rollout is failing”, Dan North from ThoughtWorks had a really good phrase summarizing why we need to introduce some systems thinking/theory into software development to create lasting change, instead of focusing narrowly on individual practices: “A bad system will beat a good person every time”, meaning that no matter how much any single person tries, he will always fail if the system around him is bad. So much to learn – so few hours in the day

DCI – the paradigm to replace Object Oriented Programming?

One of the most interesting sessions at the conference, was what may turn out to be the biggest change in the foundations of software development since OOP (object oriented programming) was introduced. It was a series of talks by Trygve Reenskaug and James O. Coplien, discussing a concept they call DCI Architecture (DCI stands for Data-Context-Interaction, but we’ll get back to that in a minute). I will try to explain it as well as I can, but if you’re interested, or just don’t understand what all the fuss is about, I suggest you read a couple of the articles I link to in the end of this section.

Trygve Reenskaug, from the University of Oslo, is a veteran in software development. As visiting scientist at the Xerox Palo Alto Research Laboratory, he was involved in the development of the Smalltalk programming language, and among other things coined the Model-View-Controller (MVC) pattern in 1978. He has spent the last several years trying to change the way we do model and program software.

Trygve Reenskaug and James Coplien argue that what we have come to know as OOP is not actually object oriented programming, but class oriented programming (a distinction you may start hearing around the Internet), and that it is a poor way to model the users’ mental model of the domain.

Classic OOP is all about objects, but what we’re actually writing are classes. Classes are good at describing the structure of data, but this is not the only thing interesting to the users of the software services we develop: We’re also interested in interactions, such as algorithms and business rules.

Most main-stream programming languages offer no special language construct for interactions, that is, collaborations between objects. They don’t capture algorithms that flow over those collaborations. Instead the interactions of a “use case” or algorithm often get spread across a number of classes, so that you cannot find a cohesive representation of them in code. There is no natural place for the algorithms and business rules to “live” – they are not a first-class citizen – and so they easily become fragmented, and hard to communicate, review and verify.

It turns out that the inability to express interactions severely hinders our ability to reason about the quality and effectiveness of the software we develop:

“Back in the 1960s, we could take the behavioral requirements for a program, and the FORTRAN code that implemented them, and give both of them to an office mate—together with a big red pen—to review whether the code matched the requirements. The overall form of the code reflected the form of the requirements. In 1967, software engineering took away my ability to do this: the algorithm had to be distributed across the objects, because to have a large method that represented an entire algorithm was believed to not be a “pure” object-oriented design. […] That works fine when an algorithm lives within a single object, as might be true for changing the color of a circle on the screen, or adding a typed character to a word processor’s text buffer. However, interesting business functionality often cuts across objects. […] Object-orientation pushed us into a world where we had to split up the algorithm and distribute it across several objects, doing the best piecemeal job that we could.” (from http://www.artima.com/articles/dci_vision.html)

So in order to better express and communicate interactions in code, DCI aims to make interactions a first-class citizen, just like classes. Like the MVC pattern separates concerns between components when building applications with user interfaces, the DCI architecture separates concerns between data and business rules. Another way to describe it is that DCI tries to close the gap between the end users mental model and the way the programmer describes it in code, in order to more easily build the software services that the users need, and facilitate communication and reasoning about them.

The key concepts in DCI are:

Data – this is expressed in terms of classes, but these classes should be dumb. They should express structure, but not behavior. Classes capture what objects are.

Context – can be thought of as a use case. It sets up the pieces we need for the interaction, meaning live objects based on our classes, and assigns them the roles (behavior) that we want them to perform. The roles are assigned on a per-use-case basis, so the same object can participate in different roles and use cases during its lifetime.

Interaction – describes end-user algorithms in terms of roles. Roles capture what objects do. At runtime, classes and roles are combined into objects as needed by the context (use cases). This idea takes a little getting used to, because most mainstream programming languages today are not good at expressing roles separately from classes.

The important thing is that the roles objects “play” are expressed separately from the classes defining data. This makes it easier to reason about an algorithm or interaction, because it is expressed as a unit, separate from the classes of participating objects, and separate from other interactions that the objects may take part in. In other words: Interactions are expressed cohesively, not fragmented across a number of classes.

As mentioned, most mainstream programming languages do not make it easy to express roles, and inject them into objects at runtime. In some languages such as Ruby and Scala, however, it is straightforward. Check out the references for examples in C++, Scala and Smalltalk.

The inventors of DCI, Trygve Reenskaug and James Coplien, together with Rickard Öberg working on related concepts in Qi4J in Java

References

Trygve Reenskaug and James Coplien can explain the concepts better than me, so if you think this was interesting, I encourage you to read more. These links should be a good start:

http://www.gertrudandcope.com-a.googlepages.com/thedciarchitecture

http://www.artima.com/articles/dci_vision.html

http://www.infoq.com/news/2009/05/dci-coplien-reenskau

UPDATE: There’s also a google group dedicated to DCI, with discussions and source code examples in various languages, such as Ruby, Javascript and Python:

http://groups.google.com/group/object-composition

ASP.Net 4 and MVC

There were a lot of talks on the .Net track about Microsoft’s newest web-framework “ASP.NET MVC”. ASP.NET MVC is a fully-supported alternative to ASP.NET WebForms (what people typically think of when they say .Net), and have generated a lot of interest in the developer community from people who didn’t like the programming model that WebForms built on top of the basic web application technologies such as HTTP and HTML. The framework is built on the model-view-controller pattern, and Microsoft makes no attempt to hide that it is heavily inspired by Ruby on Rails.

I’ve had the opportunity to work with ASP.NET MVC for the past 4-5 months, and I much prefer it to WebForms for a number of reasons: It is much easier to test, it provides clear separation of concerns (courtesy of the MVC pattern), and instead of abstracting away the details of HTML and HTTP that a developer needs to build modern web apps with AJAX, it provides a set of useful conventions, but with full developer control over page flow and markup.

Microsoft is working hard on the next version of their MVC framework, version 2, and appears to be really committed to it. What’s interesting is also that the next version of ASP.NET, ASP.NET 4, will include a number of changes to WebForms inspired by the feedback they have gotten on MVC. It will now be possible to regain full control over the markup and element id’s generated by WebForms (finally!), and it will be possible to disable ViewState on a page-wide basis. ViewState genocide is now an attainable option… ready….set….GO!

Oh yeah, and they’re now bundling the jQuery javascript library by default, instead of their own proprietary version. Three cheers for OpenSource!

Other notes

Scott Hanselman speaking on effectiveness

Neal Ford, author of The Productive Programmer had a lot of good tips for individual-developer-level-productivity. If you’re interested in that kind of stuff, you can check out his website where there are a lot of slides, tools and links. On a similar note, Scott Hanselman, Principal Program Manager at Microsoft, gave a talk about personal effectiveness, and recommended a couple of tools including Evernote for keeping notes synchronized across multiple platforms and with the ability to search within images files (it works!). He also mentioned a tool for knowledge workers called RescueTime, which automatically monitors what you do on your computer, allowing you to see when you are spending time on distracting activities. You can find more info on Scott’s blog.

Neal Ford also gave a talk comparing JRuby (a Ruby interpreter running on the JVM) and Groovy (a dynamic language similar to Ruby but with stronger ties to Java). His main message was that they were more similar than different, and that the community should try to avoid another unconstructive “Emacs vs. vi”-style war.

Thanks for reading

Well, that’s it. Hope you found it interesting. Sharing is caring

The application

I thought it might be useful to give you a basic idea of what the application does: The application TidTilMere is a highly specialized administration app for Danish folk high-schools (højskoler). It is build around the idea of having a single place to store information on everyone the school has had contact with, and it therefore spans many different areas:

• Enrollment: …for courses (such as “fall ’09″) and subjects (ex. “painting”)
• Accommodation: Room booking and availability
• Orders: Ordering of brochures and newsletter signup
• Search: Find people based on complex criteria
• Mass-mailing: Send template-based emails to people in a search result
• Mail-merge: Documents can be produced from people/enrollments in a search result, based on uploaded templates in Word and RTF format.
• CRM: A single place to store everyone the school has been in contact with, their contact information, and their relation to the school (including all of the above)

The application is used on a daily basis by the schools staff (1-4 people), and has to be very responsive. It is not likely to win any awards for front-end design, but it gets the job done. If you want to know more (and happen to speak danish), you can visit TidTilMere.net.

With that out of the way, lets get to the setup.

The server

I’ve been hosting the application with HEXONET for the past two years, and have been very happy with them. The reason I went with them, was that they were hosting out of Germany, which meant lower round-trip-times than a server farm in the US (~20ms vs. ~150ms), and we wanted the application to be very responsive.

HEXONET only sell to companies (not individuals), but have a wide range of virtual server offerings. The few times I’ve needed to contact support, they’ve been quick in getting back to me. The old server was a “vServer Platinum Pro”, whereas the new server is a “dServer VNS Premium”, which translates into 28€ for a Debian-etch with 1500MHz CPU, 45GB HD, 1000GB traffic and 1024MB RAM (1536 MB burstable). You can see more options on their feature/price list.

If you need to service customers in the EU, I think HEXONET is a good option.

Phusion Passenger (mod_rails)

A couple of weeks ago i switched all my applications from Apache 2 + Mongrel to Apache 2 + Passenger, and I’m glad I did, because it is so much easier to install and maintain. I haven’t noticed any performance gains/hits, but the application is much easier to deploy and maintain now than it was before, and installation was simpler than getting Mongrel to work with Apache 2 through mod_proxy_balancer.

Passenger will automatically launch additional workers according to load, and will also relieve them of duty when they are no longer needed. A sideeffect of this, is that if no one is accessing your site for a period of 5 minutes (default), all the workers will be killed, and Passenger will have to instantiate a new worker + the rails environment when the next visitor comes by. This can take some time, and proved to be a little annoying for my users, because they were often doing other tasks for 5 minute periods. Fortunately, it is very easy to set up a cron job (or something similar) to poll your application every once in a while, to make sure a least one worker is alway ready:

# m h  dom mon dow   command
*/3 *  * * * wget -O /dev/null http://localhost:80/admin/login --no-check-certificate 2>/dev/null

I highly recommend giving Passenger a try.

Ruby Enterprise Edition (REE)

I use Ruby Enterprise Edition together with Passenger, to improve performance and decrease memory consumption.

Recently though, comparisons here and here indicate that the tides may be changing, and that Ruby 1.9 or JRuby might be faster. For now, I’m sticking with REE, since Passenger doesn’t support JRuby yet, and Ruby 1.8.x is still what the Rails people recommends. I’m keeping my eye out though.

Monit

Monit is a monitoring tool for Unix systems, that can monitor processes, files etc. It can both alert the administrator when something is wrong, and in many cases automatically correct the problem.

I found Monit to be crucial when I had my application on Mongrel, because a Mongrel instance would crash once in a while. Monit would then automatically restart the instance and notify me, without the users experiencing any down-time. Now that Passenger takes care of always having running workers, Monit rarely has to restart anything, but I still count on it for notifying me if anything out of the ordinary happens.

I use monit for monitoring my system for abnormal load, checking that apache, mysql and my mailer deamon are running (and automatically restart them if they are not), and for making sure that I don’t accidentally run out of disk space (this has happened once or twice, and took me longer-than-I-liked to pinpoint). This is an example from my monit configuration file, which also shows some of Monits power:

set daemon  60 # Poll at 1-minute intervals

set alert erik@underbjerg.com

# Check for abnormal load
check system localhost
    if loadavg (1min) > 4 for 3 cycles then alert
    if loadavg (5min) > 3 for 3 cycles then alert
    if memory usage > 90% then alert
    if cpu usage (user) > 70% for 5 cycles then alert
    if cpu usage (system) > 30% for 5 cycles then alert
    if cpu usage (wait) > 20% for 5 cycles then alert

# Check apache2
check process apache with pidfile /var/run/apache2.pid
  start program = "/etc/init.d/apache2 start"
  stop program  = "/etc/init.d/apache2 stop"
  if cpu > 60% for 2 cycles then alert
  if cpu > 80% for 5 cycles then restart
  if totalmem > 200.0 MB for 5 cycles then restart
  if children > 250 then restart
  if loadavg(5min) greater than 10 for 8 cycles then stop
  if failed host localhost port 80 protocol http
     with timeout 15 seconds
     then restart
  if 3 restarts within 5 cycles then timeout
  group server

# Check that mysql is running
check process mysql with pidfile /var/run/mysqld/mysqld.pid
  start program = "/etc/init.d/mysql start"
  stop program = "/etc/init.d/mysql stop"

# Check that the ar_sendmail mailer daemon is running
check process ar_sendmail
  with pidfile /home/erik/apps/hojskole_sys/shared/log/ar_sendmail.pid
  start program = "/etc/init.d/ar_sendmail start" with uid erik and gid erik
  stop program = "/etc/init.d/ar_sendmail stop" with uid erik and gid erik
  if totalmem is greater than 65.0 MB for 2 cycles then restart      # eating up memory?
  if loadavg(5min) greater than 10 for 8 cycles then restart          # bad, bad, bad
  if 20 restarts within 20 cycles then timeout                        # something is wrong, call the sys-admin
  group ar_sendmail

# Check disk space
check device vzfs with path /
   if space usage > 85% then alert
   group server

So if you don’t already have a way to automatically check the health of server, I recommend giving Monit a try. Note that Monit is in no way specific to rails, so you should be able to use it for any kind of Unix server.

New Relic

The New Relic RPM performance monitoring tool is a real gem, that I wish I had when I was trying to optimize performance two years ago. It is an amazing piece of software, that plugs right in to your rails app and gives you detailed breakdowns, for each browser request, of where time is spent. It’s incredibly easy to install, and you can start using it in development mode right away.

It was never been this easy to figure out exactly where to optimize your rails app.

Screenshot of New Relic Lite in production

New Relic is free to use in development mode on your local machine, where you can get extremely detailed info, and in production, where the Lite-edition gives you basic metrics for the last 30 minutes. If you want more metrics or longer time periods in production, you will have to buy a licence, which I think is fair, since the Lite edition is already so useful. So far, I’ve gotten tremendous results out of using the Lite edition on my local machine, where I have been able to find exactly the places that were slow, and in production, where I’ve been able to see how the system was performing under real load.

If you are running a Rails application in production, and if you are just the slightest bit interested in how it performs (why wouldn’t you be?), download New Relic right now.

Thanks for reading

I hope you have found it useful. Please feel free to leave any comments, suggestions or questions you might have, and I’ll try to answer them.

For example, a page can have multiple content boxes, and possibly a sidebar with a number of info boxes. What I used to do, was something like this:

<div id="sidebar">
  <div class="infobox">
    Here is some help
  </div>
</div>
<div class="content_box">
	<%= render :partial => "table", :locals => {:people => @people} %>
</div>

Instead, by looking at the implementaton of the content_tag_for ActionView helper, I was able to find a way to do this:

<% sidebar do %>
  <% infobox do %>
    Here is some help
  <% end %>
<% end %>
<% content_box do %>
  <%= render :partial => "table", :locals => {:people => @people} %>
<% end %>

The “raw DIV tag approach” works, but there is a lot of repeated text strings (the ids and classes of the DIV tags), and if I some day want to add another stylesheet class to the content boxes, or maybe rename the content_box class to something else, I have to go though all of my view templates.

The second was made possible by adding methods similar to the following to my application_helper.rb:

def content_box(&block)
  concat content_tag(:div, capture(&block), :class => "content_box"), block.binding
end

What is does is essentially capturing the html output within the block given, and wrapping it in a DIV tag with the given parameters. That way, the ids and classes I want to use are defined in one place, and easily changed. I can even decide to wrap the contents of the content box in a different tag, such as a SPAN, if I want to at some point.

To me this approach is DRY(er), and looks more like the rails form helpers (which I think is a plus). I don’t know, however, if there are any performance-disadvantages to using this approach. Do you?