David McLaughlin Posts Feed

SSL Client Authentication with Python and pycurl

If you are using Python to make requests over SSL then most likely you'll have run into the limitations of urllib2 if you are using a web service that requires client authentication. I found some recipes that "kind of" solved the problem but didn't bother to check server certificates, which kind of defeats the point if you are using two-way SSL to prevent man-in-the-middle attacks.

It was frustrating since most simple things are relatively simple to do in Python. In cURL, two-way SSL verification can be as easy as:

curl -cacert CA.crt -E client.pem https://myservice.com:443/secure/page

Obviously here we are doing verification with a Certificate Authority we created ourselves. When I vented to a colleague about the difficulty of implementing similar functionality in Python, he wondered why I hadn't bothered using pycurl. pycurl is a C-binding and the documentation is not the greatest, nor do the examples explain how to do simple SSL never mind two-way. But after digging around the source, here is one recipe that works:

import pycurl

class Response(object):
    """ utility class to collect the response """
    def __init__(self):
        self.chunks = []
    def callback(self, chunk):
        self.chunks.append(chunk)
    def content(self):
        return ''.join(self.chunks)

res = Response()

curl = pycurl.Curl()
curl.setopt(curl.URL, "https://yourservice/path")
curl.setopt(curl.WRITEFUNCTION, res.callback)
curl.setopt(curl.CAINFO, "/path/to/CA.crt")
curl.setopt(curl.SSLCERT, "/path/to/client.pem")
curl.perform()

print res.content()

And that's it. One other option you might use when testing your SSL implementation with curl is the -k flag, which skips host verification of the server. The option to set in pycurl to do the same is:

curl.setopt(curl.SSL_VERIFYHOST, 0)

Hopefully this saved you digging around the pycurl mailing list and source code like I had to.

Looking forward to Berlin Buzzwords

Just a quick post to point out that a really exciting high-scalability conference is coming to Berlin in June. Berlin Buzzwords is due to take place on June 7th and 8th and features a variety of talks focusing on free open source software solutions to scalability problems.

It is particularly interesting for me because for the last few months I've been experiencing first hand just how many open source technologies have stepped up to the plate in the last few years to make the task of creating distributed and scalable systems accessible, even to mere mortals like myself.

Some of the technologies that will be covered are:

1) Apache Hadoop - an open source version of Google's MapReduce technology, there is almost an entire day of talks dedicated to showcasing the kind of problems Hadoop is a perfect fit for.

2) Apache Lucene and Solr - Lucene and Solr are open source search engines built for enterprise-level performance. Solr in particular gives you a whole bunch of distribution and scalability features out of the box and also lets you play with Lucene without needing to get your hands dirty with Java. In the schedule there are talks on doing geosearch and how Lucene aims to make fuzzy searching scalable in the future.

3) NoSQL technologies - there are talks on a variety of key-value stores such as Apache Cassandra, CouchDB and MongoDB and how to use these technologies for real problems rather than to simply drink the kool-aid.

The full schedule is online, so if you're in the Berlin area or are just interested in building scalable systems, check out the Berlin Buzzwords ticket page. I will be attending the conference with some of my colleagues at Nokia so if you're coming through for the conference and fancy having a coffee or beer, feel free to drop me a message at david<at>dmclaughlin.com or just send me a message on twitter.

Chained accessors in Moose

I've created my second repository on Github - MooseX::ChainedAccessors. The code is extremely simple, but very powerful: it provide a Moose attribute trait that allows you to method chain via write operations on your accessor methods. If you already know what this means and just want to grab the code, then you can do so from the github download page (I'm still waiting on my CPAN account being set up before I can upload it there).

Method chaining

Method chaining is a simple yet powerful way of creating concise APIs for objects which have lots of small, individual operations where we don't need to know about the return value. One of the most popular examples of method chaining in action is in the jQuery API:

var $mydiv = $('#my-div');
$mydiv.html('hello, world!');
$mydiv.slideUp();

// becomes...

$('#my-div').html('hello, world!').slideUp();

To achieve this, a new jQuery object is created and returned by the selector query and then each subsequent method call returns that same object. For example, a crude version of html might look like:

function html(html_as_str)
{
    this.innerHTML = html_as_str;
    return this;
}

And that one line - return this - is all there is to it. Regardless of the language, OO method chaining is achieved in the same way: in PHP it would be return $this, in Python return self, in Perl return $self, etc.

Accessor chaining

If method chaining is as simple as that, then adding chaining to accessors (methods which abstract read/write operations on class attributes) will be as simple as adding return $self to the end of write accessors. This is easy if you're manually writing your accessors, but Moose handles accessor creation for you using the has sugar subroutine. If you want method chaining on accessors, you'll need to write those accessors yourself - or extend Moose.

Why go to all the trouble? Well, the driving case for me was that I had created a Moose role which allowed me to display debug info on a per-instance basis. A simplified version of this Role is:

package MyApp::Roles::Debug;
use Moose::Role;

has 'debug' => (
    is  => 'rw',
    isa => 'Bool',
    default => sub { return 0; },
);

sub debug_message
{
    my ($self, $message) = @_;
    print $message . "\n" if $self->debug;
}

1;

This allowed me to attach my role to any complex class which required debugging when tests failed or strange things happened and I wanted an overview of what was happening. Typical usage was:

package MyApp::Model;
use Moose::Role;

with 'MyApp:Roles::Debug';

sub complex_method
{
    my $self = shift;

    # complex stuff in here.. lots of potential to go wrong
    $self->debug_message("Here is some info about the current state");
    # more complex stuff
    return;
}


my $model = MyApp::Model->new(debug => 1);
$model->complex_method();

This worked fine, until I started trying to use this role with some of the APIs I had designed that took full advantage of method chaining and composition. In a previous post I showed some code from the Sugar::ORM project I'm currently working on. Well, the query interface (which is heavily based on the Django ORM) for that ORM looks like this:

my @model_objs = Model->query->filter(name__like => 'Example%');

The call to Model->query returns a Sugar::ORM::ResultSet instance which has a bunch of methods like filter, exclude, order_by and load_related which return $self unless called in list context. This allows us to build up complex filters based on criteria (like a query string from a GET request) or apply sorting and pagination all in one nice line:

my @bands = Band->query->filter(genre__name => 'Rock')->order_by(name => 'ASC');

# OR

my $query = Band->query->filter(events__date__gte => $now);
if(my @genres = CGI->param('genre'))
{
    $query->filter(genre__id__in => [@genres]); 
}
if(my $location = CGI->param('location'))
{
    $query->filter(location__name => $location);
}
if(my $order_by = CGI->param('sort'))
{
    $query->order_by(name => 'ASC');
}
my @bands = $query->limit(10);

In practice this has work really well, but an ORM is a fairly complex code base so there have been many edge cases that have led to subtle bugs. This has meant a lot of debugging to be done deep down in the ORM engine classes. I can add my debugging role to my ResultSet class but to turn on debugging on has meant splitting up the chained filters in test scripts:

my @results = Band->query->filter(genre__name => 'Rock')->order_by(name => 'ASC');

# becomes...

my $query = Band->query->filter(genre__name => 'Rock');
$query->debug(1);
my @results = $query->order_by(name => 'ASC');

This quickly became tedious, what I really wanted to do was set my attribute without breaking the chain.

MooseX::ChainedAccessors

Now, there are a couple of other ways I could have solved this problem: debug could have been passed into the call to Model->query or the debug attribute in my Role could have been changed to a method which set up chaining. But both solutions would have to be repeated any time I wanted chained accessors in the future. In the spirit of DRY, I wanted a more elegant and easier to implement solution. After a quick glance at the Moose docs and long look at how has works in Moose, the Chained trait was born. The debug attribute in my Role now needed one extra line:

package MyApp::Roles::Debug;
use Moose::Role;

has 'debug' =>
(
    traits => ['Chained'],
    is => 'rw',
    isa => 'Bool',
    default => sub { 0; },
);

# .. etc.

And I could now debug my ORM queries with one simple change:

Band->query->load_related->get(name => 'Mesa Verde');

# becomes ...

Band->query->debug(1)->load_related->get(name => 'Mesa Verde');

And that's all there is to it.

Installing

The package isn't on CPAN yet, so in the meantime you can install it manually by downloading from github, unpacking the tar into a temporary directory and running these commands:

perl Makefile.PL
make test
make install

That's it. The only dependency is, of course, Moose. Oh, and Module::Install (thanks Tim).

Progressive enhancement with PerlTemplates

I've just created my first repository on github: PerlTemplates. PerlTemplates is a JavaScript template engine which uses the same syntax as the Perl template engine HTML::Template. Why would you need something like this?

Well hopefully you are aware of the importance of progessive enhancement; and if, like me, you work on web applications where you just can't ignore those who have disabled JavaScript in their browsers then you'll also be aware of the banality of creating both JavaScript and non-JavaScript versions of your slick AJAX interfaces.

Some of the common ways of getting round this that I've seen from other developers have been to return HTML (as opposed to returning XML or JSON) and inject that directly into the DOM, or to create the DOM elements in the Javascript code itself. Both methods have their drawbacks: as soon as you've got HTML creation in the Javascript code you now need to maintain markup in two seperare locations. Returning HTML doesn't have this problem, but for complex situations you rarely want just presentation back - you need data.

Using PerlTemplates has allowed us to follow the principles of DRY, and it has made progressive enhancement quick and simple. To illustrate, the 'output' of a search results script (which is a prime candidate for AJAXification) might look like this:

        
my $template = HTML::Template->new('/path/to/search/results.tmpl');
$template->param(%values); # where values contains the template data structure
print CGI->header, $template->output();

Well, to make it PerlTemplates-ready, we simply add a condition which returns JSON in the event of an AJAX request (which can be detected in various different ways):

if($ajax_request)
{
    print CGI->header, JSON->new->encode(%values);        
}
else
{
    my $template = HTML::Template->new('/path/to/search/results.tmpl');
    $template->param(%values);
    print CGI->header, $template->output();
}

Then in our Javascript, we have code that looks like this (using jQuery):

$.getJSON('/search/results', function(json_data) {
    var tmpl = new PerlTemplates({url:'results.tmpl', data: json_data, target: 'search-results'});
    tmpl.render();
}

That's all there is to it. The one drawback to this approach is that your templates must be made available over HTTP, which might be tricky if your templates are stored outside of your root web directory (or you're a fan of security by obscurity).

Download

PerlTemplates has no dependencies and the minified version comes in at just over 5kb. You can see it in action here and download the package from here. PerlTemplates has been tested in all major browsers, and is currently in use on sites serving millions of pages per month.

Ugly Perl: A lesson in the importance of API design

One of the most challenging aspects of my role as a Software Architect has been trying to standardise our development practices in a language as flexible as Perl. Powered by the culture of There is More Than One Way to Do It and a previous lack of technical leadership, we have ended up in a situation where almost every conceivable style of code has been utilised across all of our different mod_perl web applications. It's a maintenance nightmare.

Last year I started to turn that around by advocating object oriented programming and model view controller architecture using modern Perl techniques powered by Moose. Something I noticed though when we sat down and evaluated the state of our code was that some of the easiest and most reliable systems we had were the procedural ones, and by far the most painful systems to work with on a daily basis were those where the lead developer had went on an architectural expedition and built a tightly-coupled OOP monstrosity.

What we took away from that evaluation process is that, more than choice of programming language, more than choice of development paradigm and more than choice of development methodology, good API design is the single most important factor in how developers perceive the quality of code. Introducing the code review process to our team only confirmed this: almost every review I've taken part in has revolved around naming conventions and style rather than performance or implementation.

And it's no coincidence that by focussing less on concepts like OOP and MVC and more on creating good APIs, we've really started to turn the corner in the land of maintenance programming.

API in action: jQuery

One of the best examples of the impact a good API on a language is in the Javascript framework community. When jQuery launched, it was a textbook case of reinventing the wheel. It was slower than most of the competitors, had less features and there was no community that compared with the following that YUI and Prototype had built up. Yet somehow it still managed to muscle its way into becoming synonymous with web 2.0.

$('#my-link').click(function(e) { 
    $('#content').addClass('active').slideUp();
});

The secret to its success was in the only area where it introduced something new: the slick and concise CSS selector API. No new functionality, no performance benefits, the selling point from day one of jQuery has been "designed to change the way that you write Javascript." And people really bought into it.

jQuery really is a great example of how a solid framework API can turn around the whole perception of working with a language. Three years ago Javascript had a nasty reputation of being frustrating and difficult to work with, these days whether it's jQuery or Dojo or MooTools, the elegant APIs of the most popular frameworks more often than not make it a pleasure.

The Perl ORM problem

The motivation for this post came when I started to look for a Perl ORM solution to adopt for our MVC development. Moving away from Perl would have been too extreme and unrealistic a step, as we were going for slow and gradual refactoring of each system. So when I looked at the two most frequently discussed Perl solutions, DBIx::Class and Rose::DB::Object, what I saw was more than a little disappointing.

As a team we quickly came to the conclusion that it might be best to roll our own. Most of us had used the likes of ActiveRecord, the Django ORM and SQL Alchemy from the Ruby and Python communities, so we definitely felt like the syntax of an ORM could be a lot better than what we were seeing. When I mentioned in #moose a few months back that I was thinking of writing my own ORM, I was mocked quite heavily and pointed in the direction of Fey::ORM. Again, looking at the code all I could think of was that the standard was nowhere near what I had seen from other solutions.

In particular, I really liked using the Django ORM. I used Django to build this blog, as well as a few other personal projects and the syntax seemed to gracefully scale and handle almost all of the nasty edge cases. Trying to describe how it felt moving from the Django ORM to DBIx::Class is best illustrated with an example. The following models are directly from the DBIx::Class tutorial on CPAN.

package MyDatabase::Main;
use base qw/DBIx::Class::Schema/;
__PACKAGE__->load_namespaces;

1;

package MyDatabase::Main::Result::Artist;
use base qw/DBIx::Class/;
__PACKAGE__->load_components(qw/PK::Auto Core/);
__PACKAGE__->table('artist');
__PACKAGE__->add_columns(qw/ artistid name /);
__PACKAGE__->set_primary_key('artistid');
__PACKAGE__->has_many('cds' => 'MyDatabase::Main::Result::Cd');
1;

package MyDatabase::Main::Result::Cd;
use base qw/DBIx::Class/;
__PACKAGE__->load_components(qw/PK::Auto Core/);
__PACKAGE__->table('cd');
__PACKAGE__->add_columns(qw/ cdid artist title/);
__PACKAGE__->set_primary_key('cdid');
__PACKAGE__->belongs_to('artist' => 'MyDatabase::Main::Result::Artist');
__PACKAGE__->has_many('tracks' => 'MyDatabase::Main::Result::Track');
1;

package MyDatabase::Main::Result::Track;
use base qw/DBIx::Class/;
__PACKAGE__->load_components(qw/PK::Auto Core/);
__PACKAGE__->table('track');
__PACKAGE__->add_columns(qw/ trackid cd title/);
__PACKAGE__->set_primary_key('trackid');
__PACKAGE__->belongs_to('cd' => 'MyDatabase::Main::Result::Cd');
1;

The repetition of the Perl special variable __PACKAGE__ immediately stands out, but the rest isn't too bad. For me though, the django equivalent is far superior:

from django.db import models

class Artist(models.Model):
  artistid = models.AutoField(primary_key = True)
  name = models.CharField()

class Cd(models.Model):
  cdid   = models.AutoField(primary_key = True)
  title  = models.CharField()
  year   = models.YearField()
  artist = models.ForeignKey(Artist)

class Track(models.Model):
  trackid = models.AutoField(primary_key = True)
  title   = models.CharField()
  cd      = models.ForeignKey(Cd)

The first thing that stands out about the django syntax is that it very similar to what it represents: the definition of a SQL table. In fact, the nature of the django model definition is that you provide all the information required to actually run a CREATE TABLE command if you need to. In one lean Python class we have everything we need to know about our model.

A notable difference between the two packages is in the way they handle the bidirectional ForeignKey relationship: you only have to define them once in Django, whereas in DBIx::Class you need to define both directions separately. You could argue that in DBIx::Class this is intentional for readability – when you load up each model you see all of that model's relationships inside its own package – but in real, complex systems this would actually lead to major model definition bloat.

When it comes to querying these models, the design gap is even more apparent. The DBIx::Class tutorial has this as example CRUD code:

  my $schema = MyDatabase::Main->connect('dbi:SQLite:db/example.db');

  my @artists = (['Michael Jackson'], ['Eminem']);
  $schema->populate('Artist', [
     [qw/name/],
     @artists,
  ]);

  my %albums = (
    'Thriller' => 'Michael Jackson',
    'Bad' => 'Michael Jackson',
    'The Marshall Mathers LP' => 'Eminem',
  );

  my @cds;
  foreach my $lp (keys %albums) {
    my $artist = $schema->resultset('Artist')->search({
      name => $albums{$lp}
    });
    push @cds, [$lp, $artist->first];
  }

  $schema->populate('Cd', [
    [qw/title artist/],
    @cds,
  ]);


  my %tracks = (
    'Beat It'         => 'Thriller',
    'Billie Jean'     => 'Thriller',
    'Dirty Diana'     => 'Bad',
    'Smooth Criminal' => 'Bad',
    'Leave Me Alone'  => 'Bad',
    'Stan'            => 'The Marshall Mathers LP',
    'The Way I Am'    => 'The Marshall Mathers LP',
  );

  my @tracks;
  foreach my $track (keys %tracks) {
    my $cdname = $schema->resultset('Cd')->search({
      title => $tracks{$track},
    });
    push @tracks, [$cdname->first, $track];
  }

  $schema->populate('Track',[
    [qw/cd title/],
    @tracks,
  ]);

Even though I know what they're trying to achieve here, looking at the code it still confuses me as to why they need to search a result set to connect these models together. Once again, when compared to the equivalent django API, the difference speaks for itself:

  from myapp.models import Artists, Albums, Tracks;
  
  mj = Artists.create(name = 'Michael Jackson);
  eminem = Artists.create(name = 'Eminem')

  thriller = Albums.create(title = 'Thriller', artist = mj)
  bad = Albums.create(title = 'Bad', artist = mj)
  themmlp = Albums.create(title = 'The Marshall Mathers LP', artist = eminem)
  Tracks.create(title = 'Beat It', album = thriller)
  Tracks.create(title = 'Billie Jean', album = thriller)
  Tracks.create(title = 'Dirty Diana', album = bad)
  Tracks.create(title = 'Smooth Criminal', album = bad)
  Tracks.create(title = 'Leave Me Alone', album = bad)
  Tracks.create(title = 'Stan', album = themmlp)
  Tracks.create(title = 'The Way I Am', album = themmlp)

Simple, clean and self-documenting.

Singling out DBIx::Class might also seem a little snide, but I really don't like sitting here criticising open source code. DBIx::Class solves a difficult problem and the authors have contributed far more to the Perl and open source communities than I have. The library is also over four years old and has had to maintain backwards compatability throughout that time. The original designer might have solved the ORM problem differently today.

Still, the state of affairs is that the elegant and evolving solutions in other languages are exactly what the Perl community wants to compete with. When I read posts like this one claiming that Catalyst and DBIx::Class are a demonstration of what Perl can do in 2009 I can only think that a lot of these guys are missing the point. Competing against feature sets just isn't enough anymore: frameworks like jQuery, Rails and Django have set the expectation - developers want clean APIs. And the frustrating thing is that Perl is the perfect language to deliver them.

Modern Perl Case Study: Sugar ORM

To illustrate the point (and so those whose code I've criticised can tear me a new one back!), I'll show you what my own solution to the ORM problem has been. Let's reuse the same tables as before and create a new interface with the help of Moose sugar.

Model definitions:

package models::base;
use base 'ORM::Table';

__PACKAGE__->add_connection(username => 'david', password => 'thdnsn484');

package models::artists;
use Sugar::ORM;
extends 'models::base';

field 'artistid' => (type => 'Int', primary_key => 1);
field 'name'     => (type => 'Int');
1;

package models::cds;
use Sugar::ORM;
extends 'models::base';

field 'cdid'  => (type => 'Int', primary_key => 1);
field 'title' => (type => 'Char');
foreign_key 'artist' => (through => 'models::artist', reverse => 'albums');
1;

package models::tracks;
use Sugar::ORM;
extends 'models::base';

field 'trackid' => (type => 'Int', primary_key => 1);
field 'title'   => (type => 'Char');
foreign_key 'album' => (through => 'models::cd');
1;

CRUD operations:

use aliased 'models::artists' => 'Artist';
use aliased 'models::cds' => 'Album';
use aliased 'models::tracks' => 'Song';

my $mj = Artist->create(name => 'Michael Jackson');
my $eminem = Artist->create(name => 'Eminem');

my $thriller = Album->create(title => 'Thriller', artist => $mj);
my $bad = Album->create(title => 'Bad', artist => $mj);
my $themmlp = Album->create(title => 'The Marshall Mathers LP', artist => $eminem);

Song->create(title =>'Beat It', album => $thriller);
Song->create(title => 'Billie Jean', album => $thriller);
Song->create(title => 'Dirty Diana', album => $bad);
Song->create(title => 'Smooth Criminal', album => $bad);
Song->create(title => 'Leave Me Alone', album => $bad);
Song->create(title => 'Stan', album => $themmlp);
Song->create(title => 'The Way I Am', album => $themmlp);

And, as a bonus some extra code to traverse the relationships:

for my $artist ($mj, $eminem)
{
    say $artist->name;
    for my $album ($artist->albums)
    {
        say $album->title;
        say join ',', $album->tracks;
    }
}

Design Explanation

One of the reasons the Django model definition is so much trimmer than DBIx::Class is because Python supports shared class data natively and Perl doesn't. This is important for our models' performance because you only want to load up the fields and relationships once at compile time rather than at each object instantiation. To compensate, Sugar::ORM models are just Moose classes that use Moose::Exporter to add table, field, foreign_key and many_to_many to the package namespace. Each one of those subroutines in turn stores information about the model as metadata at compile time.

One key difference from django is that you have to define a base class to provide database connectivity, this could just as easily be refactored using roles. This decision was made because of how tightly coupled to the Django web environment the ORM is. If you want to access your models outside of a Django web request, like in a cron script, then you have a bit of leg work to do to set up the django environment. To avoid this and decouple the ORM entirely from the web, all configuration for the ORM talking to the database is completely self-contained.

Since the sample models here are pretty basic, here is a more complex one to illustrate how Sugar::ORM handles some common edge cases (of package names not matching the table name or foreign key field names being different from the primary key of the table they relate to, etc.):

package blog::models::posts;
use Sugar::ORM;
extends 'model::base';

table 'blogposts'; 
field 'id'               => (type => 'Int', primary_key => 1);
field 'title'            => (type => 'Char', max_length => 128, required => 1);
field 'slug'             => (type => 'Slug', from => 'title');
field 'datetime_posted'  => (type => 'DateTime', auto_on_add => 1);
field 'datetime_updated' => (type => 'DateTime', auto_now => 1);
field 'post'             => (type => 'Text', required => 1);
foreign_key 'author'     => (from => 'added_by_id', through => 'blog::models::user');
many_to_many 'tags'      => (through => 'blog::models::tags', using => 'blogposts_tags');
has 'something_extra'    => (is  => 'rw', isa => 'Bool', default => sub { 0; });

sub permalink 
{
    my $self = shift;
    return sprint("/post/%s", $self->slug);
}

The equivalent model in Django would be:

class Posts(models.Model):
    title            = models.CharField(max_length = 128, required = 1)
    slug             = models.SlugField(max_length = 128)
    datetime_posted  = models.DateTimeField(auto_on_add = 1)
    datetime_updated = models.DateTimeField(auto_now = 1)
    post             = models.TextField(required = 1)
    added_by         = models.ForeignKey(User, rel_name = 'author')
    tags             = models.ManyToManyField(Tag)
    something_extra  = None

    def permalink(self):
        return "/post/%s" % self.slug
    class Meta:
        db_table = 'blogposts'

When you compare the two, there is very little difference in terms of readability except perhaps the symbol bloat in the Perl version. But what is interesting is that the DBIx::Class version is actually much trimmer because of how little information is stored about columns:

package MyDatabase::Main::Result::Posts;
use base qw/DBIx::Class/;
__PACKAGE__->load_components(qw/PK::Auto Core/);
__PACKAGE__->table('blogposts');
__PACKAGE__->add_columns(qw/ id title slug datetime_posted datetime_updated post /);
__PACKAGE__->set_primary_key('id');
__PACKAGE__->belongs_to('author' => 'MyDatabase::Main::Result::User');
__PACKAGE__->many_to_many('tags' => 'blogposts_tags', 'post');

This is a good example of why conciseness isn't everything. With Django and Sugar::ORM, everything you'd need to know about the SQL table is stored in the model – that way to understand the models all you need to do is read the code. With DBIx::Class I suspect there would be a whole load of DESC `table` commands at the RDBMS command line. Plus, by storing the field types in the model code you can also do useful things later, like add data integrity checks during CRUD operations.

Of course neither the django ORM nor Sugar::ORM are perfect, and with something non-deterministic like this there will always be some people who prefer the DBIx code. Putting that to one side, what I've tried to demonstrate with my API design is that Perl is incredibly flexible. If it has a reputation as being ugly, then all that says is that the Perl community has been writing ugly APIs.

Enter the beauty contest

The good news for Perl is that one of the major causes of all the bad code out there, its flexibility, also makes it one of the most powerful languages around for creating amazing APIs with. Armed with Moose and tools like Moose::Exporter and MooseX::Declare, there is no limit to the elegant and clean Perl code that we could all be writing.

The community definitely has the talent, which is why it would be such a shame if Perl lost out in the beauty contest. But if the Perl community is really serious about a modern Perl rennaisance, then there is a very real need to get serious about API design.

Update:

Jess Robinson pointed out that the DBIx::Class tutorial I used was not the most elegant way to use the API. Her updated code adds field types to the model definition as well as provides a much cleaner insertion API:

package MyDatabase::Main::Result::Artist;
use base qw/DBIx::Class/;
__PACKAGE__->load_components(qw/PK::Auto Core/);
__PACKAGE__->table('artist');
__PACKAGE__  ->add_columns(
    artistid => {
       data_type => 'integer',
       is_auto_increment => 1,
    },
    name => {
       data_type => 'varchar',
       size => 255,
    }
);
__PACKAGE__->set_primary_key('artistid');
__PACKAGE__->has_many('cds' => 'MyDatabase::Main::Result::Cd');
1;
 
my $schema = MyDatabase::Main->connect('dbi:SQLite:db/example.db');

my $mj = $schema->create({ name => 'Michael Jackson' });
my $eminem = $schema->resultset('Artist')->create({ name => 'Eninem' });

my $thriller = $schema->resultset('Album')->create({
    artist => $mj,
    title => 'Thriller',
});
my $bad = $schema->resultset('Album')->create({
    artist => $mj,
    title => 'Bad',
});
my $themmlp = $schema->resultset('Album')->create({
    artist => $eninem,
    title => 'The Marshall Mathers LP',
});

$schema->resultset('Track')->create({ title => 'Beat Id', album => $thriller })

Clearly the original comparison I made was a case of apples and oranges as populate in the DBIx::Class API is actually a multi-insert interface which django doesn't provide. It's also clear from this example now that resultset selects the name of the model to work with. Perhaps this illustrates that good documentation is just as important as a sexy API.

How I learnt to love Perl

I love the programming reddit. This week was an especially good one in terms of relevant content for me, with discussions on the death of Perl 5, the difficulties of hiring Perl programmers and how stupid it is to write your own framework. By strange coincidence, I just finished the alpha of a new in-house Perl 5 framework for my development team at work. And we're hiring!

The two Perl articles were interesting because for three years I have worked with the language and up until earlier this year, I loathed it. This was mostly because of its awful support for Object Oriented Programming, but the fact that anyone I've worked with that called themselves a Perl programmer turned out to be a total amateur hack played a part too. After seeing no less than four guys come and go from our team this year after being 'found out' on their first big project, this quote from use.perl.org particularly resonated with me:

We had no trouble finding Perl programmers. They were a dime a dozen. People write a couple of admin scripts in Perl and they put Perl on their CV. Now don't get me wrong, I have Java on my CV, but I would never dream of sitting down for a Java interview without their knowing up front that my knowledge is pre 1.5 and I don't know any of the modern tools. It's only there because I want employers to know that I have some exposure to it, not that I think I know it.

Now, maybe I'm just being naive, but somehow people with Perl on their CV think they can do this job, but they can't. Not even close.

My disdain for Perl and the quality of code that was being written was so great that I wrote a proposal for us to move our entire code base over to PHP. The scary thing is that the technical management accepted it, and I even got as far as completing an alpha PHP port of our (basic) in house framework before the whole project was abruptly cancelled by upper management for reasons unknown. When the suits put the kybosh on my PHP framework, I decided to go covert and just start doing things right without them knowing. What I've never really discussed is what, exactly, this involved.

Object Oriented Perl

The initial reaction was obviously to start looking for alternative work, but quitting a difficult situation isn't really in my make up. Plus, it's amazing what coming out of University and spending the next two years writing procedural Perl can do for your job prospects! So, with renewed determination, I sat down and asked myself - what is it that I can do with PHP that I can't do with Perl? The whole thing boiled down to the syntax of Classes - PHP and Java both had natural constructs for building your classes whereas in Perl it seemed like a nasty hack. Still, it was clear that moving from Perl wasn't an option and I wasn't going to let minute details stop me from writing good code any longer.

The first thing I did was take Damien Conway's book on Object Oriented Perl and read it from cover to cover. What occurred to me whilst I was reading the introductory chapter on the basic concepts of Perl, was that this was the first time I had read anything at all about the language - every piece of code I had written in Perl previously was done using our in house framework which I had "learnt by doing." My experiences had always been at a very high level using a haphazard collection of Perl modules to do the very common tasks of web development - DB access, request details and parsing templates. Rather than truly understanding the underlying concepts of hashes, arrays, references, map, grep and regular expressions I just kinda lumped stuff together and hoped it work. Then I had the nerve to say the language was useless based on nothing more than the quality of the code I was writing.

A few chapters into the book, which was actually a little out of date, I began to realise that Perl doesn't just offer everything that PHP does but, in the right hands, it's actually a really nice language. I wrote a few classes in Perl and decided that writing a new subroutine just to create a class wasn't worse or better than writing a Class and constructor in Java or PHP - it was just different. Within a few weeks of utilising my new found knowledge of Perl, I wasn't just writing code as clean as what I could write in PHP, but it was actually much more elegant. This was thanks to the way Perl handles named arguments to method calls and constructors - optional parameters were easy and my interfaces became self-documenting as a result.

When I finished my first major OOP Perl project, it was something of a humbling experience - after two and a bit years of arrogantly proclaiming Perl to be a nothing language compared to PHP, the first real satisfaction I had felt after a job done came courtesy of language I spent so long trashing. This realisation that I really didn't know as much as I thought about programming led me to searching for "PHP sucks" and seeing that people had clued into that sentiment a lot sooner than I had; and with much better arguments than "the syntax for anonymous arrays sucks." After all my claims that PHP was the way forward, yeah, it was a pretty humbling experience.

Enter the Moose

Whenever I see a 'Perl guy' defend his language, the first thing they'll say is the CPAN. I had always known of it but had shied away from it for so long because, again, it didn't follow the same rules as my comfortable PHP shared hosting comfort zone. With me getting all philosophical over how wrong I was about Perl, I gave CPAN another shot and soon had modules like Template (aka Template Toolkit) and Test::Class up and running on our servers. I also started to look through the standard modules and was introduced to Devel::DProf too - all of a sudden I had this language with an easy to use profiling and unit test ecosystem and an extremely powerful templating system. For the first time I felt like I was working in an environment that was the equal of the ones I read all those enthusiastic PHP, Python and Ruby developers blog about. And it had been there the whole time that I was complaining that Perl was out dated.

The only remaining gripe I had with Perl was the Class definition syntax. I had gotten past the point where I blindly held the Java and PHP model for Classes as the Right Way, but every time I had to write my own "new" I couldn't help but feel like there was a better way. But never in a million years did I imagine something like Moose existed.

Moose isn't just an equaliser to the native OOP systems of the other big dynamic languages - it's the killer feature. People say that Catalyst is the Ruby on Rails or Django of Perl, but I disagree - Moose is the module bundle that makes Perl relevant again. There are plenty of modules on CPAN that try to make writing classes comparable to the way it's done in other languages, but the guys behind Moose had bigger ambitions and created something that provides features that no other language matches out of the box. If you had to install something like Moose just to get what other languages offer natively, then why not just use those other languages? That's why it's so important that Moose offers so much more.

Getting into specifics is for future articles, so for now I'll leave you with a link to the Cookbook and a sample of my Django-inspired CRUD model system for my new framework (courtesy of Moose sugar):

package Feature;
use Snow::Model::Sugar;

extends 'Snow::Model::Class';

table 'features';
field 'feature_id'  => (type => 'Int',  primary_key => 1);
field 'title'       => (type => 'Char', length      => 255, required => 1);
field 'slug'        => (type => 'Slug', from        => 'title');
field 'reporter_id' => (type => 'Int',  required    => 1);
field 'assigned_to' => (type => 'Int');
field 'status'      => (type => 'Enum', choices     => ['open', 'development', 'complete']);
field 'posted'      => (type => 'Datetime', auto_on_add => 1);
field 'description' => (type => 'Text', required    => 1);

# essentials
__PACKAGE__->meta->make_immutable;
1;

This code provides all the basic CRUD functionality with validation for each model defined. Of course this could have been done with the native has/isa functionality provided by Moose, but I liked this interface better and so did the guys who are going to be using this - the point is that Moose gave me the power to do this and so much more.

Life after PHP

I've read a few times that you learn more from a defeat than you do from a victory, and that certainly has been true in my case. I often think back and wonder what would have happened if the original attempts to "go PHP" had worked out. That project being cancelled has had such a dramatic effect on my approach, understanding and dedication to my profession. This blog, my recent promotion, learning Python and django, getting a VPS and learning more about the server stack - all stuff I wouldn't have dreamt of doing when knocking something up in PHP in a couple of days would have got the job done. I feel like I've learnt more in the last year than I have in the six that came before it.

Could my post-PHP epiphany have come in Ruby or Python? Having played with Ruby and used Python (django) to rebuild my band website I can safely say that it could, but the point of this post is that it didn't come in those languages - it came in Perl. And after all the negativity lately I felt like it would be a good idea to share the positive experience I've had using Perl 5 and the great progress we're making at work to modernise our development practices without changing languages.

Perl going forward

What I've taken from all of this is that a language is only ever going to be what you make of it. Moving past my own anecdote: with projects like Moose, Catalyst and Mojolicious out there, there is no doubt that Perl is a powerful language that lets good developers write good code. So why is Perl dying? I agree with the general sentiment of both articles - Perl is losing ground to Python, PHP and Ruby and it IS getting harder and harder to find good Perl guys. What concerns me is the recommended actions for fighting the rut.

The consensus in the Perl community seems to be that the areas of marketing, web design and their community organisation are where they are struggling, but these are all superficial next to the damage done by all the absolutely atrocious Perl code out there. I mean, surely the fact that PHP developers can sneer at the ugliness of most Perl code is a major warning sign?

Rather than tart up use.perl.org with some gradients, what really needs to be done is for more examples of elegant, maintainable and tested code like the kind found on the Mojo project to be put out there for people to read and aspire to. Rails is a popular framework but that has very little to do with Ruby or the nice user interfaces on 37signals websites. Design is extremely important, but it starts at the code level. Get that right, then worry about the CSS.

Obama's maintenance nightmare

On the 1st of this month, my promotion at work became official and I am now an official "Software Architect." It follows three full years of advocating the upgrading and modernising of the way we write code on our "corporate" development team. In my first post in this blog back in July, I wrote about how I was trying to save my career through a process called guerilla refactoring -- basically going rogue and fixing code and making it right without telling anyone. In the four months since that post, I became so confident that my work spoke for itself that I adopted a much simpler position - back me completely on the changes I want to make or I'm leaving.

Thankfully, my employers chose to get on board the change train.

For me I saw it as a personal victory. I don't like to quit something just because it's tough and my decision to stick it out and fight for change has been justified. Now I have to repay their faith by making the right changes and being patient through the long road ahead. I have enough experience in trying to implement small changes to know that trying to revamp the fundamentals of the way people work will be by far the most difficult thing I will ever embark on, but I wouldn't have demanded the promotion if I didn't believe I was man enough for the job.

We have several hundreds of thousands of lines of legacy code, mostly procedural Perl, that powers all of our critical products which bring in millions of pounds a month. These systems started small but grew in complexity to become bloated and monolithic, bug-ridden products with severe dependencies that limited what could be done to improve them. But I believe I can change that and bring us into the 21st century. This means Object Oriented Programming. Code reviews. Unit Testing. Refactoring. Model View Controller and all manners of proven techniques for writing maintainable code.

Over the next few months, I'll try to post regularly about the changes I attempt to make and the experiences of my most notable successes and failures. I'm sure there will be many of both.

With all this talk of change, I'd just like to spare a thought as well to this American guy that I know who also won a promotion recently, although I doubt he had to fight as hard to get it as I did mine. Still, on the 20th of January 2009 Barack Obama will become the 44th President of the United States of America, and he's taking up probably the worst maintenance job in history. The last guy in charge made a complete mess of the existing system and it's going to be this guy's job to go in there and clean it up. Personally I wouldn't touch the job for love nor money, but he seems pretty confident that he can turn things around.

And I wish him all the luck in the world.

The Django hype - it's not just its features

I just finished relaunching my band website. In the spirit of most of my personal projects recently, I have been determined to make a conscious effort to practice programming. To that end, my main goal was to use an MVC framework for the first time. As it turns out, I ended up going a lot further than that. Here is a list of the technologies or projects I've gained experience with through this pretty small project:

Django Framework
Python
Blueprint CSS framework
Setting up a server stack on a Virtual Private Server
General UNIX systems administration
nginx lightweight server platform

All in, some good experience and a great advert for making a conscious effort to pick up new things. By far the most exciting 'new thing' for me was the Django Framework, and most of the UNIX and server experience came as a result of needing to create a suitable deployment environment for the django applications I wrote.

Now, I have attempted to use many of other MVC frameworks (Zend Framework, CakePHP, Catalyst and Ruby on Rails to be precise) before I tried django but ultimately found them an overly complicated or frustrating experience. Why then did django hook me in to the point where I went from a cheap shared hosting solution to a fresh Ubuntu install costing more than double in monthly costs?

Well, there's the obvious killer feature - the automatic admin module. You really have to see this in action to find out how unbelievable it is - from the automatic validation on the forms to the slick JavaScript interfaces for handling Many to Many relationships. They really thought of everything. The model and ORM system is fantastic too, as is template inheritance and the comment/RSS frameworks. All of these features (and more) are fantastic, but in my opinion the widespread adoption of django owes a lot to two of the other killer features of django - the documentation and the development server.

Documentation

Having to write documentation is the worst part of creating reusable modules of code that you want other people to use, but it's also one of the most important parts when it comes to adopting a framework. When I tried to do a small project in Ruby on Rails, I went ahead and downloaded all the bits I needed from links on the official site. Then when I went to the "quick start" tutorials from that same site, they were all for RoR 1.3 rather than the backwards incompatible 2.0 release I had downloaded from the site. I eventually found a tutorial for 2.0 on a brazilian website that helped me create my small app but the whole experience was extremely disappointing.

With Zend Framework 1.5, the situation was the same - the documentation was there but the sheer wealth of it was overwhelming. There was documentation for each compoment in the framework and constant stressing that everything was decoupled if you needed it to be - but no quick tutorial on what to do if you just wanted to use the framework as a whole to see what it offered. In their defence, as of 1.6 this has now been fixed and a quick start link on the framework homepage now links to a quick start tutorial.

Django's documentation, by contrast, is by far the best I've ever used. Not only has the framework had the (regularly updated) standard documentation to supplement the pre 1.0 releases, but you also have the free django book if that's how you like to do your learning. With the wealth of features django offers, it would be extremely easy for the documentation to become bloated and overwhelming, but the reality is that it does a great job of presenting the information you need, when you need it. The examples they use to show off features in the documentation almost double up as an FAQ, every time I wanted to do something a bit different I typed django and the feature name into a google and a page from this documentation came up as a result. As an absolute beginner, the introductory tutorial is right there with plenty of supplementary information about related technologies and anticipated problems, and it takes you through the basics gradually - introducing complex parts only when there is a good reason.

Everyone involved in writing and editing this information should pat themselves on the back, it has been superbly well written and organised and sets a new benchmark that puts the competition to shame.

Development Server

The built-in development server was by far the most useful feature in getting me on board using django. Even when I tried to set up a "mock live" environment on my Windows Vista machine after doing the brunt of my "learning" on the development server, I had great difficulties with environmental paths and the general leap in difficulty deploying django over Apache compared to one-click installer PHP development solutions like xampp and Wampserver. And this is after I was sold on the framework!

Whoever came up with the idea of this development server, clearly understood that when you download a framework you just want to learn the framework, not how to set up a server stack that lets you run python web applications. The latter should become a problem only once you're sold on the framework and have created something you want to deploy. For Windows users with zero to limited UNIX experience, this really will reel in a whole bunch of people who would have otherwise had to wait until Wampserver for django came along.

Conclusion

The Django Project is a fantastic example of how the little things in your project can make all the difference. It would have been so easy for the team to just target open source enthusiasts who are already extremely familiar with UNIX and for whom setting up a django server platform would have been pretty straight forward, but putting the development server right in there completely eliminates the major barrier to entry for a whole other demographic of native Windows/shared hosting users.

Likewise, the documentation for django is ridiculously good. For such a young project (that only recently hit 1.0) it would also have been so easy for them to neglect the mundane part of writing about an ever-evolving code base and just concentrate on getting things right on that side of things. But the documentation from the very early versions has always been great and it paid off big.

When you see the effort they've put in to getting the details right, you can only imagine the quality of the code underneath.

Complete guide to deploying django on Ubuntu with nginx, FastCGI and MySQL

I recently deployed my first django project and it was an eye-opening experience to say the least. Coming from Windows, PHP, Wampserver and the cPanel shared hosting culture, starting with a fresh Ubuntu install and having to build the stack myself was something of a culture shock. There were of course a wealth of tutorials out there to help me get it up and running, but the reason I'm writing this is because the information I needed to get from brand new VPS to fully functional django app was split across many different tutorials, some of which were better than others. For my own reference (and for when I inevitably break something and need to start from scratch!), here is how I did it in ten simple steps:

Step 1: Get a slice

It's important to realise that you don't actually have to abandon the shared hosting model to deploy your django application, many shared hosts specialise in django hosting since the work needed to do this yourself can be quite intimidating. This thread over at Stack Overflow has a good outline of the available options.

I chose the route of getting a VPS (Virtual Private Server) because I liked the idea of having total control over the platforms my hosting service provided as well as the desire to get some systems administration experience under my belt. It's hard to get the chance to play with this stuff in a commercial environment when you have live systems with millions of users relying on everything working. So with that in mind, I started looking for a good VPS provider and after overwhelming recommendation, quickly settled on slicehost. Since my django project wouldn't be too high-traffic, I chose the 256MB slice with the default Linux distribution of Ubuntu 8.04.1 (Hardy).

Step 2: SSH into the slice

Windows users use this guide if you're having problems.

Step 3: Set up a non-root user account

I can't stress enough that you follow the two guides on slicehost for best practices for setting up your slice. The most important part is that you do not do anything under the root account. I put that in bold because in my haste I ignored all the stuff about creating a seperate user account for myself as I planned to be the only person logging into the server. This is fine, but when you create files and directories with root the default permissions are extremely strict and that will cause major problems when your server tries to read and/or execute these files later.

Step 4: Install the stack

You may have heard of the term LAMP before - it usually refers to the common server stack consisting of Linux, Apache, MySQL and PHP. Since we're starting with a nice fresh slice, we're going to have to set this up ourselves (with Python and Django replacing PHP, of course).

The only notable thing here is that I've went with nginx over Apache. This is because on a 256MB slice, I want to make the most of my limited RAM. The two major alternatives to Apache in the lightweight category are lighttpd and nginx. After doing a bit of research, I installed both, played with the virtual hosts and config and decided I liked nginx better. This article and others like it provide a good overview of all the options.

The stack:

Linux distro - Ubuntu 8.04.1
Server - nginx 0.5.33
Database - MySQL 5.0.51
Language - Python 2.5.2 (default with Ubuntu)

Step 4a: Installing nginx

Getting nginx up and running to serve static files is quite straight forward, and the guides on slicehost articles were extremely easy to follow so I'm not going to repeat them here:

By this point you should have at least the default nginx page displaying under your slice IP address. Getting a domain set up on your slice is quite tricky too but luckily there's an easy guide for that as well.

Step 4b: Installing MySQL

Again, aptitude makes this easy. Run the following command:

sudo apt-get install mysql-server mysql-client

You will be taken through a wizard to create a root account, make sure you do this then test the install by entering the mysql command line client:

mysql -u root -p

Just being prompted for your password tells you that MySQL was installed successfully. You should go ahead and create a database for your django app to use later. I recommend at some point you create a user with restricted permissions for your web client to connect as.

Step 4c: Confirm Python install

Ubuntu comes with Python pre-installed, you can type 'python' into the shell to make sure.

Step 5: Install django

Install django from the latest development branch. We will need to install subversion first:

sudo aptitude install subversion
mkdir /home/username/installers
cd /home/username/installers
sudo svn co http://code.djangoproject.com/svn/django/trunk/
sudo python trunk/setup.py install

This should be straight forward. We install subversion, check out the code somewhere (I recommended checking out installers and projects to one location to keep your slice organised) and then run the set up script. You should get clear feedback from the prompt to let you know how everything went.

Step 6: Install django dependencies

Our goal is to run django under FastCGI with a MySQL database. If you go ahead and try to do this, you'll be told there are missing packages when you run syncdb and later when you try to run django as a FastCGI: we need to install both flup and python-mysqldb.

Installing python-mysqldb is easy:

sudo apt-get install python-mysqldb

To install flup we first need to install "easy_install" - a Python utility for installing packages. Here are the commands:

cd /home/username/installers
sudo wget http://pypi.python.org/packages/2.5/s/setuptools/setuptools-0.6c9-py2.5.egg
sudo sh setuptools-0.6c9-py2.5.egg
sudo wget http://www.saddi.com/software/flup/dist/flup-0.5-py2.5.egg
sudo easy_install flup-0.5-py2.5.egg

Once again, pay attention to the feedback at the command prompt to make sure everything is (hopefully) installing successfully.

Step 7: Create a test project

Create a directory somewhere on your file system and start a django project with a small default app, then go in and change the settings to look for your MySQL database (you will need to create a new database for django to use if you haven't already done so). If you used the slicehost method of organising your code, you would run commands like this:

cd /home/username/public_html/mydomain.com/private/
django-admin.py startproject myproject
cd myproject
python manage.py startapp myapp
nano settings.py

You then need to enter your database details and paths. Here is a sample for a MySQL connection:

DATABASE_ENGINE = 'mysql'      #
DATABASE_NAME = 'mysite'       #
DATABASE_USER = 'websites'     #
DATABASE_PASSWORD = 'mypasswd' #
DATABASE_HOST = ''             #
DATABASE_PORT = ''             #

Don't worry too much about the other path details (like media and templates) for now, all we're doing here is trying to get that default "welcome to django" page to display so that we can confirm everything else is working.

Step 8: Set up nginx conf for django

When you installed nginx, hopefully you took the trouble to get yourdomain.com running and displaying static content on your nginx install - because we're about to edit the config file to make it serve django content. If you didn't, then follow these steps:

Change your domains nameserver records to the slicehost settings. Whoever you registered your domain with will provide the tools to do this. See your slicehost manager control panel for the nameserver domains to point to.
Use the DNS wizard in slicehost manager to handle the domain
Set up a nginx virtual host which will handle requests to this domain

If you don't have a domain yet and just want to use the IP for the time being, use the default config file located at (/etc/nginx/sites-available/default) in the following examples.

Open your domain config file:

sudo nano /etc/nginx/sites-available/mydomain.com

And change it to look like this, substituting in your project path:

server
{
    listen   80;

    server_name www.yourdomain.com;
    access_log /path/to/yoursite.com/log/access.log;
    error_log /path/to/yoursite.com/log/error.log;
    root /path/to/yoursite.com/public;

    location /site_media
    {
        root /path/to/yoursite.com/public;
    }

    location /
    {
        # host and port to fastcgi server
        fastcgi_pass 127.0.0.1:8081;
        fastcgi_param PATH_INFO $fastcgi_script_name;
        fastcgi_param REQUEST_METHOD $request_method;
        fastcgi_param QUERY_STRING $query_string;
        fastcgi_param CONTENT_TYPE $content_type;
        fastcgi_param CONTENT_LENGTH $content_length;
        fastcgi_pass_header Authorization;
        fastcgi_intercept_errors off;
    }
}

This will direct all web requests to http://www.yoursite.com/ to a FastCGI process (i.e django) except those under http://www.yoursite.com/site_media/. The port you use here has to be different for each instance of django you want to run, so take a note of it here as we'll need it when we run the FastCGI process later.

I had a number of legacy links on my band website that I had to transfer over and I also used a slightly different directory structure from the slicehost articles but it all follows the same pattern. For reference, here are my settings:

server
{
    # redirect all mesaverde.co.uk requests to www.mesaverde.co.uk
    listen 80;
    server_name mesaverde.co.uk;
    rewrite ^/(.*) http://www.mesaverde.co.uk/$1 permanent;
}

server
{
    listen   80;
    server_name www.mesaverde.co.uk;

    access_log /home/david/websites/mesaverde.co.uk/log/access.log;
    error_log /home/david/websites/mesaverde.co.uk/log/error.log;

    root /home/david/websites/mesaverde.co.uk/public;

    location /ui
    {
        root /home/david/websites/mesaverde.co.uk/public;
    }

    location /mp3
    {
        root /home/david/websites/mesaverde.co.uk/public;

    }

    location ~* ^.+\.(jpg|css|jpeg|gif|png|ico|zip|pdf|txt|wav|js|mov|mp3)
    {

       access_log   off;
       expires      30d;
    }

    location /

    {

        # host and port to fastcgi server
        fastcgi_pass 127.0.0.1:8081;
        fastcgi_param PATH_INFO $fastcgi_script_name;
        fastcgi_param REQUEST_METHOD $request_method;
        fastcgi_param QUERY_STRING $query_string;
        fastcgi_param CONTENT_TYPE $content_type;
        fastcgi_param CONTENT_LENGTH $content_length;
        fastcgi_pass_header Authorization;
        fastcgi_intercept_errors off;

    }

}

All my CSS, JavaScript and image files are located inside www.mesaverde.co.uk/ui/ and I also have a page of mp3s that I host at http://www.mesaverde.co.uk/mp3/, using this set up I don't involve django in any static file serving.

Step 9: Run django as a FastCGI

The final step is starting up django as a FastCGI. Remember in the development mode when you had to use runserver to test your code? Well it's the same idea here, with a couple of extra options:

cd /path/to/your/project
python manage.py runfcgi host=127.0.0.1 port=8081 --settings=settings

Note that the port you specify here should match the one you chose in your site conf in step 8. Once you have run this command, you can go to http://www.yourdomain.com and if all is well, you will get the default 'welcome to django' page. Congratulations, you have deployed django and the worst is over!

Step 10: Settings, uploading projects and other configuration

Setting up an FTP server

Now that django is working, you'll probably want to upload your django app and attempt to get that working. If you come from a shared hosting background like I did, you'll probably take FTP access for granted. Unfortunately it doesn't quite work like that. You'll need to install an FTP server on your slice, but this is a lot easier than it sounds. I chose VSFTP for this:

sudo aptitude install vsftpd

Once it's installed (and automatically started) you'll now need to open up the conf and change a couple of key settings:

sudo nano /etc/vsftpd.conf

Most importantly, you should disable anonymous access and set up access for local users:

anonymous_enable=NO
local_enable=YES
local_umask=022

The two local settings allow you to FTP in as the same user you SSH in as and make sure that the permissions are reasonable. Once you change these settings, restart your FTP server for them to take effect:

sudo /etc/init.d/vsftpd restart

Now you can log into your slice with your usual FTP client and upload your django project files. Use your slice IP, and the username and password you've been using to SSH to your slice. Remember to change settings.py to reflect the new templates path.

Admin Media

If your project is using the automatic admin then you might notice it looks a bit bare. That's because django doesn't automatically load the media for you like it does on the development server. We'll need to set this up.

The admin media is located in a restricted part of your system that you need superuser permissions to access. So first we grant it more liberal permissions:

sudo chmod +rx /usr/lib/python2.5/site-packages/django/contrib/admin/media

Here we've given it read and executable permissions for all users. Next we need to make this directory available inside the directory we chose for our static files when we set up our nginx conf (your site_media directory or whatever you named it). The typical Windows way to do this is just to make a copy, but in UNIX we can use a symbolic link to achieve this.

ln -s /usr/lib/python2.5/site-packages/django/contrib/admin/media /home/username/public_html/yoursite.com/public/site_media/admin

The idea is that you can now access these files through a URL like www.yoursite.com/site_media/admin/dashboard.css. By default, admin media is served through www.yoursite.com/media/ so we need to change the ADMIN_MEDIA_PREFIX setting in settings.py:

ADMIN_MEDIA_PREFIX = '/site_media/admin/'

Restart django and the admin should now look more familiar.

Restarting Django

When we make changes to nginx, there is a handy init script that takes care of it for us:

/etc/init.d/nginx restart

So using information from the official docs let's create a similar one for django. Create the script in your /usr/local/bin directory.

sudo nano /usr/local/bin/yoursite-restart

Then save into this file the following script:

#!/bin/bash
PROJDIR="/home/username/public_html/yoursite.com/private/project"
PIDFILE="$PROJDIR/yoursite.pid"
cd $PROJDIR
if [ -f $PIDFILE ]; then
   kill `cat -- $PIDFILE`
   rm -f -- $PIDFILE
fi
exec python ./manage.py runfcgi host=127.0.0.1 port=8081 pidfile=$PIDFILE --settings=settings

Obviously you need to change the path to point to your project directory and make sure the port setting matches the one in your nginx config. Save the file, give it executable permissions and then make sure django isn't currently running:

sudo chmod +rx /usr/local/bin/yoursite-restart
ps -ef | grep "fcgi"

This will give an output of currently running processes which contain "fcgi" in them. It should look like:

root     12085     1  0 Nov02 ?        00:00:00 python ./manage.py runfcgi host=127.0.0.1 port=8081 --settings=settings
root     12086 12085  0 Nov02 ?        00:00:02 python ./manage.py runfcgi host=127.0.0.1 port=8081 --settings=settings
root     12087 12085  0 Nov02 ?        00:00:02 python ./manage.py runfcgi host=127.0.0.1 port=8081 --settings=settings
root     14818 12085  0 10:29 ?        00:00:00 python ./manage.py runfcgi host=127.0.0.1 port=8081 --settings=settings
root     14869 12085  0 12:23 ?        00:00:00 python ./manage.py runfcgi host=127.0.0.1 port=8081 --settings=settings
root     16544 12085  0 15:54 ?        00:00:00 python ./manage.py runfcgi host=127.0.0.1 port=8081 --settings=settings

The key process we're interested in is the first in this list. We want to take note of the first number - 12085. This is the process ID. The other runfcgi processes are just children of this process and killing the parent will kill the children processes too:

kill 12085

Django is now stopped. We can test the bash script we just created by running:

sudo yoursite-restart

If you created the file in /usr/local/bin you don't need to specify the full path to the file, which is why we can just type the script name directly. Django should now be running again and each time you make changes to your django settings or models, etc. you can just run this command to restart the django processes.

Congratulations, you're done!

Conclusion

Although there are a lot of small steps, deploying django is simple and a good exercise in beginning to understand the most popular software stack that supports the web. For Windows user in particular, this kind of basic systems administration will be a good introductory exercise to UNIX and set you apart from many applications developers who avoid this kind of thing.

I Got 99 Problems But a Manager Ain't One

There's a few predictable ways people deal with unhappiness in life. They can dwell on it and make everyone around them miserable. They can suck it up and say "that's life." Or they can actually do something about it.

Me? I tend to go through a cycle of all three. One thing I realised from going on a few job interviews and looking at my list of complaints about my current work environment was that the problems I had were a veritable checklist of every working environment out there. Legacy code? Which established software company doesn't have that? A standard set of tools and practices for common problems? I'd hope so! Changing something that isn't broke?

There's nothing worse than reinventing the wheel for no reason.

Except when you're being asked to make major changes to a bunch of spaghetti procedural code full of inconsistent architectural decisions and quick-fix hacks for the numerous bugs that appeared since launch. Oh and could you improve performance whilst you're at it? And by the way we're already behind schedule. No manager wants to hear "complete rewrite" when they're adding something that took the guy before you two or three weeks to hack in there.

So I hacked it in just that one time. And the next. And again. And in no time at all that horrible legacy code I used to blame on the other guy is now 50% mine. Oops! And those archaic practices and standards now dominate my CV .

Not my managers CV. Or their managers CV. My CV.

To top it all off, there has been a steady rise in web development jobs looking for expert OOP programmers and experience with MVC frameworks. It's all Zend Framework this, Rails that and Django another thing. In my office we use Perl. And not even cool Perl like Catalyst or Template::Toolkit. We're talking HTML::Template Perl here. I'm not even relevant in modern Perl jobs (all 2 openings per year).

Guerilla Refactoring

Guerilla Refactoring is a phrase I thought I had coined but was in fact soundly beaten. Essentially it's the act of covertly fighting the good fight agains the almighty resistance of middle management, developing your own skills and getting better at your profession while you're at it.

See, the biggest problem I had six months ago was that even if I could start from scratch and do a total rewrite of the code I was having to maintain, I didn't have the real world experience to put all the theory into practice. Sure I knew what the acronyms MVC and DRY meant, but it's not until you start trying to write truly reusable code that YAGNI and KISS suddenly become the acronyms of choice. The code behind a good API and simple interface can often be extremely complex, and it takes a lot of experience to get it right.

Things changed for me during the last major project I worked on. The existing codebase was so huge that no business case in the world could justify a complete rewrite of the existing code to achieve the stated goals of the project, on that issue I had admitted defeat. So there I was, hacking away on a OOP/Procedural mash-up with some hundreds of thousands of lines of legacy code when I found myself about a week ahead of the schedule with some time to burn. What I did next was something I did for the first time in my professional career - rather than alert my manager to my heroic ability to beat the schedule, I went back and improved the design of a piece of fully functional and tested code I had written. I believe they call it refactoring?

It was hardly a pioneering moment, refactoring is a fundamental concept to agile methodologies and OOP programming books across the land. But I work in an Enterprise environment and when a manager asks me what I've done and I show them a fully-functional solution, unfortunately their first thought isn't "that's great but I wonder if david could improve the code and incorporate some advanced techniques so he can outgrow us and get a better job". In an enterprise environment, to improve the maintainability or scalability of a piece of working code you generally have a fight on your hands. If they know you're doing it, anyway.

The refactoring I did was extremely successful, I turned 300 lines of procedural code into a completely reusable class. When the same problem inevitably came up a few days later at another part of the application, I trimmed yet more time off the schedule by plugging my class in there. The actual process of reusing the code raised some flaws in the initial design as well as some areas that needed improvement, so I went ahead and used the time saved to make those to my class. Coding at work actually started getting interesting again.

Soon I had the Guerilla Refactoring process down to an artform. I would solve whatever problem I was facing as fast as possible, then I would spend the rest of the budgeted time refactoring the design of my code and any surrounding code. My refactorings were subtle and isolated and I burnt my fingers on some misapplication of a lot of theory without having to commit to any major architectural decisions. Once a lot of those rookie mistakes were out of my system, I was having to do less and less refactoring because I was getting it right sooner. I became so confident in the long-term benefits of Guerilla Refactoring that it became less "guerilla" and more "public displays of affection for".

We can't all work for Google

If you're lucky enough to be working in some small, dynamic and modern web development team with great seed engineers and all the latest and greatest agile practices then you probably scoff at the idea of having to go behind your managers back to do things right. But it's the reality for most people working in the Enterprise environment.

Guerilla Refactoring works because you don't break budget or blow a deadline by trying to rewrite an entire application when you were asked to solve a simple problem. Nor do you become responsible for an overblown and complex mess of an OOP architecture when you're still wet behind the ears. You simply work within the time you're allowed and go dark for a period of that time.

If the situation I was in is anything like yours, give it a try. It might just save your sanity. And your career.