Ugly Perl: A lesson in the importance of API design

Posted Sunday, April 19th at 6:33p.m. under Perl

One of the most challenging aspects of my role as a Software Architect has been trying to standardise our development practices in a language as flexible as Perl. Powered by the culture of There is More Than One Way to Do It and a previous lack of technical leadership, we have ended up in a situation where almost every conceivable style of code has been utilised across all of our different mod_perl web applications. It's a maintenance nightmare.

Last year I started to turn that around by advocating object oriented programming and model view controller architecture using modern Perl techniques powered by Moose. Something I noticed though when we sat down and evaluated the state of our code was that some of the easiest and most reliable systems we had were the procedural ones, and by far the most painful systems to work with on a daily basis were those where the lead developer had went on an architectural expedition and built a tightly-coupled OOP monstrosity.

What we took away from that evaluation process is that, more than choice of programming language, more than choice of development paradigm and more than choice of development methodology, good API design is the single most important factor in how developers perceive the quality of code. Introducing the code review process to our team only confirmed this: almost every review I've taken part in has revolved around naming conventions and style rather than performance or implementation.

And it's no coincidence that by focussing less on concepts like OOP and MVC and more on creating good APIs, we've really started to turn the corner in the land of maintenance programming.

API in action: jQuery

One of the best examples of the impact a good API on a language is in the Javascript framework community. When jQuery launched, it was a textbook case of reinventing the wheel. It was slower than most of the competitors, had less features and there was no community that compared with the following that YUI and Prototype had built up. Yet somehow it still managed to muscle its way into becoming synonymous with web 2.0.

$('#my-link').click(function(e) { 
    $('#content').addClass('active').slideUp();
});

The secret to its success was in the only area where it introduced something new: the slick and concise CSS selector API. No new functionality, no performance benefits, the selling point from day one of jQuery has been "designed to change the way that you write Javascript." And people really bought into it.

jQuery really is a great example of how a solid framework API can turn around the whole perception of working with a language. Three years ago Javascript had a nasty reputation of being frustrating and difficult to work with, these days whether it's jQuery or Dojo or MooTools, the elegant APIs of the most popular frameworks more often than not make it a pleasure.

The Perl ORM problem

The motivation for this post came when I started to look for a Perl ORM solution to adopt for our MVC development. Moving away from Perl would have been too extreme and unrealistic a step, as we were going for slow and gradual refactoring of each system. So when I looked at the two most frequently discussed Perl solutions, DBIx::Class and Rose::DB::Object, what I saw was more than a little disappointing.

As a team we quickly came to the conclusion that it might be best to roll our own. Most of us had used the likes of ActiveRecord, the Django ORM and SQL Alchemy from the Ruby and Python communities, so we definitely felt like the syntax of an ORM could be a lot better than what we were seeing. When I mentioned in #moose a few months back that I was thinking of writing my own ORM, I was mocked quite heavily and pointed in the direction of Fey::ORM. Again, looking at the code all I could think of was that the standard was nowhere near what I had seen from other solutions.

In particular, I really liked using the Django ORM. I used Django to build this blog, as well as a few other personal projects and the syntax seemed to gracefully scale and handle almost all of the nasty edge cases. Trying to describe how it felt moving from the Django ORM to DBIx::Class is best illustrated with an example. The following models are directly from the DBIx::Class tutorial on CPAN.

package MyDatabase::Main;
use base qw/DBIx::Class::Schema/;
__PACKAGE__->load_namespaces;

1;

package MyDatabase::Main::Result::Artist;
use base qw/DBIx::Class/;
__PACKAGE__->load_components(qw/PK::Auto Core/);
__PACKAGE__->table('artist');
__PACKAGE__->add_columns(qw/ artistid name /);
__PACKAGE__->set_primary_key('artistid');
__PACKAGE__->has_many('cds' => 'MyDatabase::Main::Result::Cd');
1;

package MyDatabase::Main::Result::Cd;
use base qw/DBIx::Class/;
__PACKAGE__->load_components(qw/PK::Auto Core/);
__PACKAGE__->table('cd');
__PACKAGE__->add_columns(qw/ cdid artist title/);
__PACKAGE__->set_primary_key('cdid');
__PACKAGE__->belongs_to('artist' => 'MyDatabase::Main::Result::Artist');
__PACKAGE__->has_many('tracks' => 'MyDatabase::Main::Result::Track');
1;

package MyDatabase::Main::Result::Track;
use base qw/DBIx::Class/;
__PACKAGE__->load_components(qw/PK::Auto Core/);
__PACKAGE__->table('track');
__PACKAGE__->add_columns(qw/ trackid cd title/);
__PACKAGE__->set_primary_key('trackid');
__PACKAGE__->belongs_to('cd' => 'MyDatabase::Main::Result::Cd');
1;

The repetition of the Perl special variable __PACKAGE__ immediately stands out, but the rest isn't too bad. For me though, the django equivalent is far superior:

from django.db import models

class Artist(models.Model):
  artistid = models.AutoField(primary_key = True)
  name = models.CharField()

class Cd(models.Model):
  cdid   = models.AutoField(primary_key = True)
  title  = models.CharField()
  year   = models.YearField()
  artist = models.ForeignKey(Artist)

class Track(models.Model):
  trackid = models.AutoField(primary_key = True)
  title   = models.CharField()
  cd      = models.ForeignKey(Cd)

The first thing that stands out about the django syntax is that it very similar to what it represents: the definition of a SQL table. In fact, the nature of the django model definition is that you provide all the information required to actually run a CREATE TABLE command if you need to. In one lean Python class we have everything we need to know about our model.

A notable difference between the two packages is in the way they handle the bidirectional ForeignKey relationship: you only have to define them once in Django, whereas in DBIx::Class you need to define both directions separately. You could argue that in DBIx::Class this is intentional for readability – when you load up each model you see all of that model's relationships inside its own package – but in real, complex systems this would actually lead to major model definition bloat.

When it comes to querying these models, the design gap is even more apparent. The DBIx::Class tutorial has this as example CRUD code:

  my $schema = MyDatabase::Main->connect('dbi:SQLite:db/example.db');

  my @artists = (['Michael Jackson'], ['Eminem']);
  $schema->populate('Artist', [
     [qw/name/],
     @artists,
  ]);

  my %albums = (
    'Thriller' => 'Michael Jackson',
    'Bad' => 'Michael Jackson',
    'The Marshall Mathers LP' => 'Eminem',
  );

  my @cds;
  foreach my $lp (keys %albums) {
    my $artist = $schema->resultset('Artist')->search({
      name => $albums{$lp}
    });
    push @cds, [$lp, $artist->first];
  }

  $schema->populate('Cd', [
    [qw/title artist/],
    @cds,
  ]);


  my %tracks = (
    'Beat It'         => 'Thriller',
    'Billie Jean'     => 'Thriller',
    'Dirty Diana'     => 'Bad',
    'Smooth Criminal' => 'Bad',
    'Leave Me Alone'  => 'Bad',
    'Stan'            => 'The Marshall Mathers LP',
    'The Way I Am'    => 'The Marshall Mathers LP',
  );

  my @tracks;
  foreach my $track (keys %tracks) {
    my $cdname = $schema->resultset('Cd')->search({
      title => $tracks{$track},
    });
    push @tracks, [$cdname->first, $track];
  }

  $schema->populate('Track',[
    [qw/cd title/],
    @tracks,
  ]);

Even though I know what they're trying to achieve here, looking at the code it still confuses me as to why they need to search a result set to connect these models together. Once again, when compared to the equivalent django API, the difference speaks for itself:

  from myapp.models import Artists, Albums, Tracks;
  
  mj = Artists.create(name = 'Michael Jackson);
  eminem = Artists.create(name = 'Eminem')

  thriller = Albums.create(title = 'Thriller', artist = mj)
  bad = Albums.create(title = 'Bad', artist = mj)
  themmlp = Albums.create(title = 'The Marshall Mathers LP', artist = eminem)
  Tracks.create(title = 'Beat It', album = thriller)
  Tracks.create(title = 'Billie Jean', album = thriller)
  Tracks.create(title = 'Dirty Diana', album = bad)
  Tracks.create(title = 'Smooth Criminal', album = bad)
  Tracks.create(title = 'Leave Me Alone', album = bad)
  Tracks.create(title = 'Stan', album = themmlp)
  Tracks.create(title = 'The Way I Am', album = themmlp)

Simple, clean and self-documenting.

Singling out DBIx::Class might also seem a little snide, but I really don't like sitting here criticising open source code. DBIx::Class solves a difficult problem and the authors have contributed far more to the Perl and open source communities than I have. The library is also over four years old and has had to maintain backwards compatability throughout that time. The original designer might have solved the ORM problem differently today.

Still, the state of affairs is that the elegant and evolving solutions in other languages are exactly what the Perl community wants to compete with. When I read posts like this one claiming that Catalyst and DBIx::Class are a demonstration of what Perl can do in 2009 I can only think that a lot of these guys are missing the point. Competing against feature sets just isn't enough anymore: frameworks like jQuery, Rails and Django have set the expectation - developers want clean APIs. And the frustrating thing is that Perl is the perfect language to deliver them.

Modern Perl Case Study: Sugar ORM

To illustrate the point (and so those whose code I've criticised can tear me a new one back!), I'll show you what my own solution to the ORM problem has been. Let's reuse the same tables as before and create a new interface with the help of Moose sugar.

Model definitions:

package models::base;
use base 'ORM::Table';

__PACKAGE__->add_connection(username => 'david', password => 'thdnsn484');

package models::artists;
use Sugar::ORM;
extends 'models::base';

field 'artistid' => (type => 'Int', primary_key => 1);
field 'name'     => (type => 'Int');
1;

package models::cds;
use Sugar::ORM;
extends 'models::base';

field 'cdid'  => (type => 'Int', primary_key => 1);
field 'title' => (type => 'Char');
foreign_key 'artist' => (through => 'models::artist', reverse => 'albums');
1;

package models::tracks;
use Sugar::ORM;
extends 'models::base';

field 'trackid' => (type => 'Int', primary_key => 1);
field 'title'   => (type => 'Char');
foreign_key 'album' => (through => 'models::cd');
1;

CRUD operations:

use aliased 'models::artists' => 'Artist';
use aliased 'models::cds' => 'Album';
use aliased 'models::tracks' => 'Song';

my $mj = Artist->create(name => 'Michael Jackson');
my $eminem = Artist->create(name => 'Eminem');

my $thriller = Album->create(title => 'Thriller', artist => $mj);
my $bad = Album->create(title => 'Bad', artist => $mj);
my $themmlp = Album->create(title => 'The Marshall Mathers LP', artist => $eminem);

Song->create(title =>'Beat It', album => $thriller);
Song->create(title => 'Billie Jean', album => $thriller);
Song->create(title => 'Dirty Diana', album => $bad);
Song->create(title => 'Smooth Criminal', album => $bad);
Song->create(title => 'Leave Me Alone', album => $bad);
Song->create(title => 'Stan', album => $themmlp);
Song->create(title => 'The Way I Am', album => $themmlp);

And, as a bonus some extra code to traverse the relationships:

for my $artist ($mj, $eminem)
{
    say $artist->name;
    for my $album ($artist->albums)
    {
        say $album->title;
        say join ',', $album->tracks;
    }
}

Design Explanation

One of the reasons the Django model definition is so much trimmer than DBIx::Class is because Python supports shared class data natively and Perl doesn't. This is important for our models' performance because you only want to load up the fields and relationships once at compile time rather than at each object instantiation. To compensate, Sugar::ORM models are just Moose classes that use Moose::Exporter to add table, field, foreign_key and many_to_many to the package namespace. Each one of those subroutines in turn stores information about the model as metadata at compile time.

One key difference from django is that you have to define a base class to provide database connectivity, this could just as easily be refactored using roles. This decision was made because of how tightly coupled to the Django web environment the ORM is. If you want to access your models outside of a Django web request, like in a cron script, then you have a bit of leg work to do to set up the django environment. To avoid this and decouple the ORM entirely from the web, all configuration for the ORM talking to the database is completely self-contained.

Since the sample models here are pretty basic, here is a more complex one to illustrate how Sugar::ORM handles some common edge cases (of package names not matching the table name or foreign key field names being different from the primary key of the table they relate to, etc.):

package blog::models::posts;
use Sugar::ORM;
extends 'model::base';

table 'blogposts'; 
field 'id'               => (type => 'Int', primary_key => 1);
field 'title'            => (type => 'Char', max_length => 128, required => 1);
field 'slug'             => (type => 'Slug', from => 'title');
field 'datetime_posted'  => (type => 'DateTime', auto_on_add => 1);
field 'datetime_updated' => (type => 'DateTime', auto_now => 1);
field 'post'             => (type => 'Text', required => 1);
foreign_key 'author'     => (from => 'added_by_id', through => 'blog::models::user');
many_to_many 'tags'      => (through => 'blog::models::tags', using => 'blogposts_tags');
has 'something_extra'    => (is  => 'rw', isa => 'Bool', default => sub { 0; });

sub permalink 
{
    my $self = shift;
    return sprint("/post/%s", $self->slug);
}

The equivalent model in Django would be:

class Posts(models.Model):
    title            = models.CharField(max_length = 128, required = 1)
    slug             = models.SlugField(max_length = 128)
    datetime_posted  = models.DateTimeField(auto_on_add = 1)
    datetime_updated = models.DateTimeField(auto_now = 1)
    post             = models.TextField(required = 1)
    added_by         = models.ForeignKey(User, rel_name = 'author')
    tags             = models.ManyToManyField(Tag)
    something_extra  = None

    def permalink(self):
        return "/post/%s" % self.slug
    class Meta:
        db_table = 'blogposts'

When you compare the two, there is very little difference in terms of readability except perhaps the symbol bloat in the Perl version. But what is interesting is that the DBIx::Class version is actually much trimmer because of how little information is stored about columns:

package MyDatabase::Main::Result::Posts;
use base qw/DBIx::Class/;
__PACKAGE__->load_components(qw/PK::Auto Core/);
__PACKAGE__->table('blogposts');
__PACKAGE__->add_columns(qw/ id title slug datetime_posted datetime_updated post /);
__PACKAGE__->set_primary_key('id');
__PACKAGE__->belongs_to('author' => 'MyDatabase::Main::Result::User');
__PACKAGE__->many_to_many('tags' => 'blogposts_tags', 'post');

This is a good example of why conciseness isn't everything.  With Django and Sugar::ORM, everything you'd need to know about the SQL table is stored in the model – that way to understand the models all you need to do is read the code. With DBIx::Class I suspect there would be a whole load of DESC `table` commands at the RDBMS command line.  Plus, by storing the field types in the model code you can also do useful things later, like add data integrity checks during CRUD operations.

Of course neither the django ORM nor Sugar::ORM are perfect, and with something non-deterministic like this there will always be some people who prefer the DBIx code. Putting that to one side, what I've tried to demonstrate with my API design is that Perl is incredibly flexible. If it has a reputation as being ugly, then all that says is that the Perl community has been writing ugly APIs.

Enter the beauty contest

The good news for Perl is that one of the major causes of all the bad code out there, its flexibility, also makes it one of the most powerful languages around for creating amazing APIs with. Armed with Moose and tools like Moose::Exporter and MooseX::Declare, there is no limit to the elegant and clean Perl code that we could all be writing.

The community definitely has the talent, which is why it would be such a shame if Perl lost out in the beauty contest. But if the Perl community is really serious about a modern Perl rennaisance, then there is a very real need to get serious about API design.

Update:

Jess Robinson pointed out that the DBIx::Class tutorial I used was not the most elegant way to use the API. Her updated code adds field types to the model definition as well as provides a much cleaner insertion API:

package MyDatabase::Main::Result::Artist;
use base qw/DBIx::Class/;
__PACKAGE__->load_components(qw/PK::Auto Core/);
__PACKAGE__->table('artist');
__PACKAGE__  ->add_columns(
    artistid => {
       data_type => 'integer',
       is_auto_increment => 1,
    },
    name => {
       data_type => 'varchar',
       size => 255,
    }
);
__PACKAGE__->set_primary_key('artistid');
__PACKAGE__->has_many('cds' => 'MyDatabase::Main::Result::Cd');
1;
 
my $schema = MyDatabase::Main->connect('dbi:SQLite:db/example.db');

my $mj = $schema->create({ name => 'Michael Jackson' });
my $eminem = $schema->resultset('Artist')->create({ name => 'Eninem' });

my $thriller = $schema->resultset('Album')->create({
    artist => $mj,
    title => 'Thriller',
});
my $bad = $schema->resultset('Album')->create({
    artist => $mj,
    title => 'Bad',
});
my $themmlp = $schema->resultset('Album')->create({
    artist => $eninem,
    title => 'The Marshall Mathers LP',
});

$schema->resultset('Track')->create({ title => 'Beat Id', album => $thriller })

Clearly the original comparison I made was a case of apples and oranges as populate in the DBIx::Class API is actually a multi-insert interface which django doesn't provide. It's also clear from this example now that resultset selects the name of the model to work with. Perhaps this illustrates that good documentation is just as important as a sexy API.

25 Responses

colophon