Author Archives: Jack Slingerland

About Jack Slingerland

Hi! My name is Jack Slingerland. I'm a software craftsman working and living in Raleigh, NC. I work in Python, Django, Node.js, AngularJS, and PHP. I like to work out with Kettlebells, run, and spend my free time with my wife, cat, and dog.

Shuttering Side Projects

Over the past few years I’ve slowly accumulated some big side projects. They weren’t done for clients, but just for myself. At some point maintenance of these side projects isn’t fun anymore and hinders the creative juices. I have other things I want to work on, but having these other zombie side projects feels too much like an albatross around my neck.

After much deliberation, I’ve decided to shut down two of my large side projects: BookCheaply and Smooth Bulletin. I really believe both of these projects could do someone some good, but they were both learning projects for me and I don’t see them moving forward anymore. Effective immediately I’m disabling their Apache configs, backing up their DBs, TARing it all together, and putting it somewhere safe. I’ll keep access to the Git repos, but eventually I’ll clone a copy of those out too and archive them. If I don’t get them out of the way completely, I feel like I’ll want to work on them too much.

Shuttering these projects marks a transition for me, where I move from using Python and Django on side projects to Node.js, Express, and Angular. While apprehensive about abandoning my go-to stack for side projects, I’m excited to learn the nooks and crannies of Node (and I still use Python/Django for my day job anyways).

Here’s to the future!

Smooth Bulletin

Restoring a 14 year old website

If you want to skip the restoration stuff, the final product can be found at http://re-cycledair.com/starwars/Entrance.html.

A long time ago in an era just before the first dot-com bust, I was a bright-eyed child with aspirations to make the greatest Star Wars web site of all time. With only Microsoft Front Page Express and a limited knowledge of HTML in hand, the 1999 version of me created the best Star Wars web site the world had ever seen.

Unfortunately I was hosting it on my ISPs free web hosting area. When that ISP was later gobbled up by another company, all traces of my website were gone… or so I thought. Recently I discovered that most of it still exists on the Way Back machine, however it’s in poor shape. Many of the assets don’t exist anymore, the linking is broke, and the frames just don’t work right in modern browsers. This is my journey to save this little piece of history while using minimal modern techniques. My goal is to have a legitimate 1999 website when I’m done and have it render nicely on most modern web browsers.

The Internet Archive

The Wayback machine at the Internet Archive has long been one of my favorite websites. I love getting nostalgic for the web of the past, my childhood, and history in general. It was awhile ago that I realized it still had some of my original Star Wars website. If you want, check out the capture from 2001. It doesn’t work much at all, but it gives you a good idea of what we’re working with here.
Star Wars Restoration - Before

In addition to the main page I also had an “Entrance” page, because you know, every cool web site had an entrance page back then. This one is in slightly better shape, meaning the star field background survived intact.
Star Wars Restoration - Entrance Before

The Game Plan

My goal for this restoration is to make the website display great in modern web browsers with minimal code changes and without using too many modern techniques. At that point in time, CSS wasn’t really a thing that a ton of people used yet, and even if it was it was beyond my understanding, so I’m going to try and avoid it if at all possible. Javascript was available in most browsers by that time, but I didn’t understand it, so I didn’t use it. However, we do have FRAMES(!), which is going to make things super exciting for everyone still reading.

Now that we’ve laid down some ground rules, here is the order of attack:

  1. Entrance: Fix / replace broken images.
  2. Entrance: Clean up the HTML (alignment) and remove anything that the Internet Archive crawler added.
  3. Main site: Fix / replace broken images
  4. Main site: Remove dead links. Sadly the Internet Archive couldn’t capture everything.
  5. Main site: Fix any styling issues that remain.
  6. Main Site: Clean up HTML (alignment) and remove anything that the Internet Archive crawler added.

Lets get started!

Entrance: Replacing Broken Images

While it’s fantastic that the background star field has survived, the rest of the images didn’t. This is sort of a problem because I have to reach pretty far back into my memory to figure out what was there originally. Luckily, 1999 me wasn’t completely terrible at naming things.

Star Wars Restoration - Entrance File Names 1

From these file names and memory, I’m going to say that `tie.gif` was a rotating TIE fighter gif, and `starwars.gif` was the Star Wars logo that survived on the main section of the site.

Star Wars Restoration - Entrance File Names 2

And from this file name we get nothing. Luckily I remember this being an image of Tatooine, roughly 350px x 100px. This site came out a bit before Episode I, so I’m going to assume it was an image from one of the Episode I trailers. At least, thats how I remember it.

After a bit of digging, I found replacement images for everything but the Tatooine banner. In it’s place, I found a light saber gif that I definitely used on this site somewhere in the past. I also updated the star field to be a little more 1999.

Star Wars Restoration: Entrance Complete

Entrance: HTML Clean Up

Now that the entrance is starting to look like a real 1999 web site again, we can start to clean up the HTML and fix the broken link to enter the site. The entrance area contains 3 pages, 1 to bring the frames together, and 1 for each frame. They currently look like this:
entrance_html_before

entrance_banner_before

entrance_intro_before

The code isn’t too bad. With a bit of alignment help and some nesting, things are going to look pretty good. Now check out the final product:

entrance_banner_after

entrance_html_after

entrance_intro_after

Main Site: Fix Broken Images

Now that we have the entrance out of the way, I’ve have a pretty good idea of how this whole restoration will work for the rest of the site. Let’s start by fixing the background on the main site so things are a bit more workable.

Star Wars Restoration: Main site with background

Now we’re talking! There are few things that I noticed immediately after getting the background set.

  • 1999 me had some pretty sweet design skills
  • I should use more animated gifs on my websites
  • The frame content on the left isn’t staying inside of it correctly, so I had to scroll to the center to get it to work.
  • I had a Yahoo! club!
  • I wasn’t so great with words back then. (I was a kid, give me a break)

Looking at the source, there are a few images that aren’t linking quite right.

  • Left side: starwars.gif – I think this was just a smaller version of the big logo.
  • Left side: deathstar.gif – Definitely a gif of the Death Star. It might have been animated, but I think it was just transparent so that it looked awesome.
  • Right side: lettersabove.gif – I honestly don’t know, but it might have been “A long time ago in a galaxy far far away…”
  • Right side: pulselightsaber.gif – A pulsing light saber page break. I might just use the static one from the entrance instead.
  • Right side: xwing.gif – A rotating X-Wing fighter. Because nothing says “Email me!” like a rotating X-Wing.

After tracking down many of the original images that I used here, the home page now looks like this.

Star Wars Restoration: Home Page Finished

Looking good! Next up, code clean up.

Main Site: Code Clean Up

The code for the main site is in pretty poor condition. There is a ton of code in there from the Internet Archive, it has poor indentation, and in general just isn’t anywhere near the quality of the entrance. Given the amount of code here, I thought using a Github Gist would be better.

Before Code: https://gist.github.com/vital101/56b2783b8e75b00fdac9

After Code: https://gist.github.com/vital101/2d1e6a44ac3ddbb1789d

As you can see I went ahead and re-aligned everything so that it’s more readable, took out all of the javascript and css added by the Internet Archive, and condensed the code a little bit so there wasn’t large bits of whitespace between elements.

The next step in the code cleanup is fixing URLs. If you look at the code, you’ll notice that all the links still point to the Internet Archive. That obviously isn’t going to work for us, so I simply need to replace “https://web.archive.org/web/20010811043323/http://my.voyager.net/~hands/star-wars/” with an empty string, making the link relative.

After fixing links, I found that many of the sub-pages have been lost to time, but there are still a few for us to look at including:

  • Luke Skywalker
  • Darth Vader
  • Princess Leia
  • Chewbacca
  • G.Moff Willguf Tarkin
  • Star Wars: Rogue Squadron
  • Weapons
  • B-Wing
  • Snowspeeder
  • Corellian Corvette
  • Mon Calamari Cruiser
  • At-AT
  • Death Star
  • Chat
  • Movie

Fixing up these pages follows a similar pattern to the entrance page and the main site, so I’ll spare you the details. There were a few interesting things that popped up while I was doing the sub-page restoration though.

  • Plagiarism – 1999 me definitely ripped off somebody else with most of the encyclopedic data on characters, weapons, and ships. The change in voice is a dead giveaway.
  • Front Page Versions – It appears that I made the entrance and the menu/main page of the main site in FrontPage 3.0, but all of the sub-pages in FrontPage 2.0 (express). FrontPage Express was shareware that came bundled with IE 4 so I know how I got that. I believe I had access to FrontPage 3.0 at my school at that time, which I probably used to generate the frames.
  • Xoom – I had completely forgotten about it, but Xoom used to be a free unlimited web host back in the early dot-com days. I used it to host videos, games, and other things. Unfortunately my Xoom site wasn’t crawled by the Internet Archive.
  • Java Chat – Back in 1999, there weren’t a whole lot of options for getting a chat room going on your website. Using a Java applet was basically it, so thats what I did. It looks like Xoom offered it’s members a free chat tool, so how could I resist?
  • Game Demo – Ah, the good old days. The one game page that survived has some pretty hefty system requirements: Windows 95/98, 32 MB of RAM, Pentium 166. Direct3D card, DirectX 6. Oh, and I also made sure to let people know the estimated download time on 28.8 kbps dial-up modem.

Final Thoughts

The final product can be found at http://re-cycledair.com/starwars/Entrance.html.

While going through this restoration I realized that this is when I knew what I wanted to do with my life. My parents had no idea what I was doing, but they realized that it stimulated my mind so they let me continue to do it anyways. Without that kind of encouragement and freedom to create, I probably wouldn’t be where I am today. This restoration has also re-kindled my respect for the old web. It was simple, but I’m extremely happy that the web has progressed to where it is today.

HealthCare.gov: It almost worked

Ever since HealthCare.gov launched all I’ve been hearing about is how horrible the experience is. It never affected me directly, so I chalked it to a few vocal detractors trying to sway public opinion. I even went so far as to go to HealthCare.gov and use the logged-out experience to compare some plans. It looked pretty good actually, and I tweeted as such.

HealthCar.gov Tweet

That was all about to change though. I recently made the move to work at a small startup that doesn’t yet have an employer-sponsored health plan. They do however offer a stipend of sorts to help you pay for insurance costs on HealthCare.gov. And with that, my journey began.

Initial Sign Up

After discussing the healthcare situation with my wife, I ran the numbers and decided that it would be best for her to use her employer-sponsored healthcare and for me to use the exchange separately. At this point everything went as well as it could. Even though the UX on HealthCare.gov is pretty bad, I was able to figure out how to fill out an application, select health coverage, select dental coverage, and get things rolling. This all happened before December 23rd, which means I’d be getting coverage by January 1st! All was well… until I had to make a change.

The Change

As it turns out, I math’d wrong. When I was calculating the cost of healthcare for my wife at her job, I was off by something like $200 per month, which pushed her healthcare costs well into the “unreasonable for what I’m going to receive” range. With that in mind, I thought I could make a quick change on HealthCare.gov to get her on the same plan that I was. Boy was I wrong.

The process I had to take to get both my wife and myself health and dental insurance was ridiculous, and still hasn’t worked correctly. To add my wife to my plan, I had to delete the entire thing. Not only that, but I needed to delete my entire application and start over from scratch. Once I did that, I kept constantly running into javascript errors in their application which would force me to re-authenticate.

After finally getting through the application process, I get to the last step where I need to confirm. I press the button and… nothing. Not a goddamn thing. Being a software engineer, I do a little investigating and find out that the server is returning 500 errors to the client (in non-developer speak, this means that the server couldn’t process my request because of some error on their side). The error that is returned says that I should try to log-out, then resume the application. I do this, but when I try to resume my application I get:

Thats right, my application is locked. It doesn’t give any reason, which is especially confusing considering that I was asked to re-authenticate. But there is a little link that can “explain this task”, so I click it, and it naturally didn’t work.

At this point, I was beginning to realize that I would need to contact their support to get things resolved.

Support and Next Steps

The first time I tried to use support, I clicked the helpful little “Live Chat” button. I then proceeded to wait for 25 minutes with no response. This was over the holidays, so I gave it another try after Christmas. This time I was connected to somebody, but no matter what question I asked I was given a canned response to contact the call center. My question is: Why have the “Live Chat” at all if you’re just going to tell me to contact the call center? It’s ridiculous and a waste of my time.

After my initial experience with HealthCare.gov, I’m not really excited to contact the call center. I honestly don’t have a ton of time to do it, and given the number of people that will be signing up for healthcare I’m sure the wait will be long. My likely next steps will be to cancel what remains of my application and start over. If it doesn’t work this time, then I’m going to cancel that application and contact my insurance provider of choice directly. In my case thats Blue Cross Blue Shield of Michigan, which happens to have a walk-in office about a block from where I work.

I create things all the time on the web. Its my chosen profession, so I know how hard it can be to make a good website when you need to integrate with a lot of different 3rd parties. However this sort of experience isn’t going to cut it. If I’m federally mandated to have health insurance, then the experience should be as painless as possible.

Because the system is broke I won’t have health coverage until February 1st now; a month long gap in coverage for my family. This just isn’t acceptable.

Why I Still Have Faith in Hacker News

If you browse Hacker News enough you’ll start to see article trends.

  • Startup X is Startup Y for Dogs
  • Why you should use framework Z over formerly hot framework B
  • Cool Javascript demo is N lines of code
  • How I failed at A and it made me better at B.

Don’t get me wrong, some of this stuff is interesting, but most of it is just noise to me. The reason I come to HN is for the comments. Thats where the good stuff is. HN is one of the brightest internet communities I’ve ever had the pleasure to interact with. For instance, today I posted to HN asking for the community to review my startup Smooth Bulletin. Within an hour I had two very thoughtful comments about business and market ideas that I hadn’t even thought of. I expect I’ll probably get even more feedback as the day goes on, and that I’ll benefit from those just as much as the first comments.

Thats why I still have faith in Hacker News. Not because of the content, but because of the people. The people there are incredibly smart and willing to give their advice just so that maybe you can succeed one day. I’ve been a member of HN for several years now and this hasn’t changed at all since the time I joined, and I hope that it never does.

And yes, I posted this to Hacker news, so maybe I should add “Why I still have faith in X” to the list.

Using Restangular with Django TastyPie

TastyPie’s JSON responses split the response into two sections: meta data and actual data. The meta is really nice, because it helps you data pagination, result counts, and the like, but it kind of gets in the way of Restangular. Integrating with Restangular is easy though!

Add the following to your app.js file.

yourApp.config(function(RestangularProvider) {
    RestangularProvider.setBaseUrl("/api");
    RestangularProvider.setResponseExtractor(function(response, operation, what, url) {
        var newResponse;
        if (operation === "getList") {
            newResponse = response.objects;
            newResponse.metadata = response.meta;
        } else {
            newResponse = response;
        }
        return newResponse;
    });
    RestangularProvider.setRequestSuffix('/?');
});

Done.

Virtualenv Error: Could not find the …site.py element of the Setuptools distribution

For a few hours I was running into a problem whenever I would try to install my PIP requirements file. The install would go alright until it got to Distribute, at which point I would get an that ended up in a stack trace with:

Could not find the /lib/python2.7/site-packages/site.py element of the Setuptools distribution

After much searching and digging, the issue was that my virtual environment needed to be instantiated with the –distribute flag.

virtualenv venv --distribute

Problem solved.

A Case for WordPress

I recently made the case for using the open-source WordPress CMS instead of a custom hand-rolled CMS to a manager. What follows is the email (with names and companies changed) that was sent. I thought that it might be useful to other people who are trying to convince their organization to take the leap to WordPress, Drupal, or any other open-source CMS. It’s also worth noting that I’m on very friendly terms with my manager, hence the informality and straight-forwardness. As for the Django references, we develop most of our data-driven sites in Python/Django.

Hey [Manager],

As you already know, I completely loathe the [Custom] CMS. My recent and ongoing experiences with the [Client] website have only served to reinforce these feelings. Unbridled hate rarely serves any useful purpose though, so I’ve decided to see what I can do to make the problem better. My way of doing that is by making a case for using WordPress on large, non-trivial, content driven websites. As of right now WordPress has been lasso’d into small-size websites at [Company], but I believe it is capable of far more then that. What follows is a breakdown of all the feedback, questions, and concerns that were raised while I was gathering information. I’ve provided solutions to almost all of the problems, and its worth noting that some of the problems with WordPress are not unique to WordPress. I also want to make it clear that I’m not trying to get rid of the [Custom] CMS, especially since we have a lot of clients with a lot of money invested in it. I’m simply making a case for using WordPress on large sites instead of the [Custom] CMS.

Content Creation / Management

For as long as I can remember, [Other Manager] always raised a red flag with WordPress because it was inferior in content creation to the CMS. In vanilla form this is arguably true, however WordPress has an amazing community of plugin developers surrounding it who have solved all of these problems.

  • Pods – I have a love-hate relationship with pods. I like that they provide nearly infinite flexibility to the content creator, but that comes with the price of nearly infinite complexity. While I disagree with giving the content-creator that much control over how content is displayed, I can see the usefulness of it. To solve that problem, there are many visual composition plugins available for WordPress. My favorite is Visual Composer (http://vc.wpbakery.com/). Extending Visual Composer (akin to adding “module functions”) is as simple as adding a hook in your WordPress widget.
  • Categories / Taxonomies for Media (assets) – WordPress has brilliant support for categories and custom taxonomies out of the box. Beyond that, it also has support for tags. One thing that doesn’t work out of the box is media categories. Luckily there is a large handful of community plugins that easily solve this problem for us.
  • Media – The media library in WordPress is extremely slick, ajax-y (asynchronous uploads, etc), and easily extendable. Multiple thumbnails and image sizes are generated out of the box. Adding new image sizes is easy too.
  • Videos – WordPress supports videos out of the box. Adding Youtube/Hulu/Vimeo videos is as easy as adding one of the numerous plugins. There are even plugins out there that interface with FFMPEG to get you a nice selection of thumbnails to choose from.
  • Form Builder – The new hotness of [Custom] CMS is Form Builder. WordPress has a large amount of awesome form builders available, and most of them are cheap and come with support. Gravity Forms(http://www.gravityforms.com) is my favorite. Although Ninja forms is looking like a valid alternative.
  • User Management – WordPress comes with excellent user management out of the box. It has 4 default roles (Administrator, Editor, Contributor, Subscriber), but no way to change what those roles are capable of. They generally have pretty sane defaults, but sometimes you want to tweak things. User Role Editor(http://wordpress.org/plugins/user-role-editor) provides an easy way to modify group and user permissions. If you need to add new roles, there are plugins for that as well.
  • Content Deployment – We’ve often struggled with how to deploy new content to sites. Generally you don’t want to create the content on live, because it will need to be hidden. There is a straight ballin’ plugin for WordPress called Ramp(http://crowdfavorite.com/wordpress/ramp/) that solves this problem. Seriously, check this out. It will blow your mind.

General Notes

I have a handful of notes that don’t fit well into Content Creation / Management or Development, so here we go.

  • Update Cycle – WordPress has a release cycle (http://en.wikipedia.org/wiki/WordPress#Releases). It has feature releases, bug releases, and HOLY HELL THATS A BIG RELEASE releases. WordPress updates are vetted by an insanely large community of professional developers who catch bugs/regressions and fix them before release.
  • CMS “Upgrades” – Somehow its become normal and accepted at [Company] that a CMS Upgrade costs north of $25,000, takes a month, and almost always ends up braking some part of the site that wasn’t broken before. WordPress upgrades take 10 seconds, don’t break changes, and have deprecation warnings just in case you’re using a method/hook that won’t be supported in a future release. We might make some money off of [Custom] CMS upgrades, but my experience has been that they end up costing more time/effort/good will then they are worth. Having a one-click upgrade option allows us to keep our sites secure, up to date, and gets us back to making things awesome.
  • Community – WordPress.org has 1,998 base themes that can be used and extended via theme inheritance. It also has over 26,000 plugins that extend WordPress’ functionality. This doesn’t account for the numerous premium plugins that are available throughout the internet. Help is always just a Google search away. If that doesn’t work, Stack Overflow most certainly will. Having the backing of a community while you are developing is invaluable. Did I mention the WordPress IRC channel?
  • Documentation – One of the most frustrating things about the [Custom] CMS isn’t the lack of code documentation, but the lack of code usage documentation. WordPress has a ludicrous amount of documentation on pretty much anything you can think of. It stays up to date with best practices and is updated for every release. If the WordPress.org documentation isn’t good enough, the source code is very well documented. If the source doesn’t suit your fancy, there is probably a blogger that wrote about your problem.
  • Hiring – Hiring is hard in this economic climate. Selling a PHP developer to come work for [Company] with [Custom] CMS is even harder. If the person is a WordPress expert, all of that domain knowledge stagnates and becomes useless here. It also makes the on-boarding process take 2-4 months. And even then, they still aren’t all that useful in the [Custom] CMS. Having WordPress as a core tool allows us to hire people who are already experts. The on-boarding becomes a few weeks, and everyone is happy. It also makes it a lot less fatal if we lose a developer, because it’s possible to hire another one that already has all the domain knowledge.
  • Focus – Using WordPress is similar to using Django in that it gets out of your way at lets you focus on what’s important: making awesome stuff. You don’t have to piddle around with random regressions, bugs, or odds and ends. Those are already worked out for you. You get to focus on making custom functionality the best you can. Not having to mess around with bugs and regressions allows you to focus on quality instead of just “getting it done”.
  • Security – WordPress can be a security nightmare. With an install footprint of over 60 million websites (as of 2012), everybody is trying to hack it. Luckily most plugin authors and the WordPress security team are always right on top of it. There is a plugin called WP Security Scan(http://wordpress.org/plugins/wp-security-scan/) that takes care of making sure all the directories have the proper permissions, and in general gives good advice to harden your WordPress installation. Taking 10 minutes a week to update all your WordPress sites is also the best way to prevent security issues.
  • Who? – WordPress.com has a nice list of notable WordPress users (http://en.wordpress.com/notable-users/). The list includes, CNN, NYTimes, Forbes, Reuters, Sony, Jay-Z, Rolling Stones, GM, and others.
  • Cost – We keep upgrading the CMS. It needs it, but it cost lots of money. WordPress and the open-source community keep things up-to-date for us, so the cost is nil.

Development

In the past, a lot of misinformation has been spread about WordPress and how developing with it is awful. A lot of that comes from lack of knowledge, so I thought I’d address a lot of the pain points that people have experienced.

  • Plugin Creation – [Other developer] brought up the point that WordPress plugins are created in a procedural fashion, and are therefore a huge PITA to work with. As it turns out, the correct way to make WordPress plugins is via OOP. This NetTuts tutorial covers most of the basics (http://net.tutsplus.com/tutorials/wordpress/create-wordpress-plugins-with-oop-techniques/).
  • Content Deployment – WordPress hard-codes URLs into the database content. This is obviously a bad idea for many reasons. To rectify, if you move content from one server to another, you just need to execute 3 sql queries as part of your deployment. Usage of Ramp (mentioned above) solves this issue for 99% of use cases.
  • Versioning – WordPress shouldn’t be versioned. Any plugin you install from WordPress.org should not be versioned. If you are making a custom theme, that should get it’s own repository (hey, look at Stash and all it’s repos we can create). If you are making a custom plugin, that should also get it’s own repo. This promotes re-usability and good coding practices. Deployment of theme/plugin updates is as easy as checking out the latest version or git-pulling.
  • Deployment – Initial deployment of a WordPress site is just like moving a staging site to the live server for the [Custom] CMS. After that, deploying code updates to custom plugins and themes is just a git pull/checkout of a tag. This could easily be streamlined with the use of Fabric or Capistrano.
  • Modules / Custom Post Types – The [Custom] CMS has a pretty nice way of creating basic custom content via modules and the scaffolding. WordPress has custom post types, and several different plugins for creating them on the fly. My current favorite is Advanced Custom Fields (http://www.advancedcustomfields.com/)
  • Theme Inheritance – As mentioned before, WordPress has LOTS of themes. Nearly any one of them can be used as a base theme, saving front-enders a ton of time getting base CSS and templates set up.
  • ORM – WordPress has a weak ORM. Hopefully you can use WordPress’ features enough that you can write minimal SQL. On the bright side, you’ll have some nicely optimized queries ;) This is also why you should use the right tool for the job. If you have to interact with the database so much that you need an ORM, you should probably be using a framework.
  • Testing – WordPress core is fully unit and integration tested. The majority of premium plugins and popular plugins are tested as well. Plugins can easily be tested using PHPUnit, and their interfaces can be tested using Behat.
  • Developing as a Team – WIth a sane Git strategy (custom plugins and themes versioned only) developing as a team is a non-issue. In the past, having the whole project under version control caused serious issues when there was more then one developer.
  • Database Migrations – Just as with [Custom] CMS, if you have complex migration that destroys data you’ll need to handle the migration yourself. http://codex.wordpress.org/Creating_Tables_with_Plugins has more info. If the migration isn’t destructive, you can use the wp_delta method to handle it for you.
  • Environment Abstraction – WordPress does a great job of not caring about what version of PHP you’re on. So long as you are PHP >= 5.2.4 and MySQL >= 5.0.0, you’re all set

I know we’ve talked in the past about vetting a Django CMS, but I think that using WordPress is the best choice. We already have people who know it, the front-enders know how to develop for it, the community is massive, and learning it is easy. I look forward to talking with you about this.

Integration Testing With Django and Lettuce: Getting Started

The Ruby on Rails community has long been a proponent of Behavior Driven Development(BDD) and has a great ecosystem around it supporting that testing methodology. From Cucumber to Capybara, RoR developers have it made when it comes to BDD. But what about Django? What about Python? Django and Python don’t have access to Cucumber or Capybara, but what we do have is a fantastic port of Cucumber called Lettuce.

What is Behavior Driven Development

Before we can get started talking about Lettuce and all the cool things you can do with it, we first need to talk about BDD.

Behavior-driven development combines the general techniques and principles of TDD with ideas from domain-driven design and object-oriented analysis and design to provide software developers and business analysts with shared tools and a shared process to collaborate on software development. (Wikipedia)

BDD arose out of the need for the business side of software and the engineering side of software to communicate more effectively. Prior to BDD, it was a lot more difficult to communicate the business requirements of a project to developers. Sure there were spec documents, but those still needed to be translated into a language the computer can understand. With BDD, tests and acceptance criteria are more accessible to everybody involved. Dan North suggested a few guidelines for BDD, and then the development community took it from there.

  • Tests should be grouped into user stories. Essentially narratives about the expected functionality.
  • Stories should have a title. The title should be clear and explicit.
  • There should be a short narrative at the beginning of the story, that explains who the primary stakeholder of the story is, what effect the story should have, and what business value the stakeholder derives from this from this effect.
  • Scenarios(tests) should follow the format of first describing the initial conditions for the scenario, then which event(s) triggers the start of the scenario, and finally what the expected outcome of the scenario should be.
  • All of these steps should be written out in natural language, preferably using the Gherkin syntax.

An example feature using Gherkin.

Feature: Authentication
    In order to protect private information
    As a registered user
    I want to log in to the admin portal
 
    Scenario: I enter my password correctly
        Given the user "Jack" exists with password "password"
        And I am at "/login/"
        When I fill in "Login" with "Jack"
        And I fill in "Password" with "password"
        And I press "Login"
        Then I should be at "/portal/"
        And I should see "Welcome to the admin portal"

So now that we know the gist of BDD, why would you want to use it? There are probably more reasons than the 3 I’m going to list, but I found these to justify my use of BDD in most cases.

  1. It’s easy for business minded people to understand what you’re trying to test.
  2. It’s easier to translate complicated business requirements into tests.
  3. Some things are easier to explain in natural language.

Alright, now we’re done with the background information. Let’s get rolling on some testing.

Getting Started

To follow the rest of this article, you’re going to need the following:

  • A little Python experience
  • A little Django experience
  • Extremely basic knowledge of regular expressions
  • Knowledge of how to set up a virtual environment using virtualenv (I also use virtualenvwrapper to make my life a bit easier)
  • Firefox – Yes, I know you don’t need Firefox to do this, but its probably the easiest to use with Selenium.

On the bright side, no previous testing experience required!

The best place to start with all this getting the virtual environment set up.

jacks$ mkvirtualenv learning_lettuce

After that, lets get Django installed.

(learning_lettuce)jack:repos jacks$ pip install django

And now we’ll need to create a new Django project.

(learning_lettuce)jack:repos jacks$ django-admin.py startproject learning_lettuce
(learning_lettuce)jack:repos jacks$ cd learning_lettuce/
(learning_lettuce)jack:learning_lettuce jacks$ ls -la
total 8
drwxr-xr-x   4 jacks  staff  136 Jun 19 07:50 .
drwxrwxrwx  28 jacks  staff  952 Jun 19 07:50 ..
drwxr-xr-x   6 jacks  staff  204 Jun 19 07:50 learning_lettuce
-rw-r--r--   1 jacks  staff  259 Jun 19 07:50 manage.py

At this point, I also like to CHMOD manage.py so I can execute it without calling Python directly.

(learning_lettuce)jack:learning_lettuce jacks$ chmod +x manage.py
(learning_lettuce)jack:learning_lettuce jacks$ ./manage.py runserver
Validating models...
 
0 errors found
June 19, 2013 - 06:53:29
Django version 1.5.1, using settings 'learning_lettuce.settings'
Development server is running at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

If you can run the server and see the image below, then we can proceed.
Django Powered!

Now that we have Django set up, lets go ahead and create the app we’ll be testing in.

(learning_lettuce)jack:learning_lettuce jacks$ ./manage.py startapp blog

And then go ahead and add the blog app to INSTALLED_APPS in your settings.py file.

INSTALLED_APPS = (
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.sites',
    'django.contrib.messages',
    'django.contrib.staticfiles',
 
    # Authored
    'blog',
)

Also, lets configure our project to use SQLite3.

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': 'learning_lettuce.db',
    }
}

Now that we have a Django project and one app set up, its time to take a break and talk about Lettuce.

Lettuce

No, not the vegetable you add to salads. I’m talking about the the BDD testing framework for Python (http://www.lettuce.it). Lettuce is basically a port of a BDD testing framework from the RoR community called Cucumber. The Lettuce website contains extensive documentation and is a great source for learning best practices with it. It’s worth noting however, that at the time of this writing the Lettuce website is undergoing some design changes. They’re incomplete and have made it pretty hard to extract information from the site. Hopefully by the time you need it for reference it’s back to being usable again.

Alright, back to work. Lets install Lettuce, Selenium, and Nose and then freeze a requirements file so we can replicate this environment if we ever need to.

(learning_lettuce)jack:learning_lettuce jacks$ pip install lettuce selenium django-nose
(learning_lettuce)jack:learning_lettuce jacks$ pip freeze > requirements.txt
(learning_lettuce)jack:learning_lettuce jacks$ cat requirements.txt 
Django==1.5.1
django-nose==1.1
fuzzywuzzy==0.2
ipdb==0.7
ipython==0.13.2
lettuce==0.2.18
nose==1.3.0
selenium==2.33.0
sure==1.2.2
wsgiref==0.1.2

You’ll also need to add Lettuce to INSTALLED_APPS in your settings.py file.

INSTALLED_APPS = (
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.sites',
    'django.contrib.messages',
    'django.contrib.staticfiles',
 
    # 3rd party
    'lettuce.django',
 
    # Authored
    'blog',
)

So now that you have Lettuce installed, lets see that it actually works.

(learning_lettuce)jack:learning_lettuce jacks$ ./manage.py harvest
Django's builtin server is running at 0.0.0.0:8000
Oops!
could not find features at ./blog/features

Great, Lettuce worked! It didn’t find any tests to run, but thats ok. At least we’ve verified that we installed everything correctly.

Your First Test

Before you can test anything, you should probably have some content to test on. So let’s quickly wire up a simple view in the blog app.

# blog/views.py
from django.http import HttpResponse
 
def quick_test(request):
	return HttpResponse("Hello testing world!");
# learning_lettuce/urls.py
from django.conf.urls import patterns, include, url
 
from blog.views import quick_test
 
urlpatterns = patterns('',
    url(r'^quick-test/$', quick_test),
)

Great! Now when you go to http://127.0.0.1:8000/quick-test/ you should see “Hello testing world!”.

The next step is to create a folder inside of the blog app called “features”. And inside of that create a file called “test.feature”. It’s worth noting that Lettuce doesn’t actually care what your file is named, so long as the extension is “.feature”. In “test.feature”, add the following:

Feature: Test
	As someone new to testing
	So I can learn behavior driven development
	I want to write some scenarios
 
	Scenario: I can view the test page
		Given I am at "/quick-test/"
		Then I should see "Hello testing world!"

Look at all that plain english! Even without me telling you anything, you can probably figure out what we’re trying to test. But let me break it down for you.

  • Line 1: This loosely describes what all of the scenarios below are testing. Think of it as a way to logically group tests together.
  • Lines 2-4: This is the narrative. It explains why you’re testing in the first place.
  • Line 6: The title of your scenario. This describes what you are specifically testing in this instance.
  • Lines 7-8: These are called “steps”. Steps are how you test your scenario. Each step maps to a method in your code.

Alright, so now that you have your first test written, run it using “./manage.py harvest”. You should see the following:
Learning Lettuce - Unimplemented Steps
Look at all that beautiful output! But what does it all mean?! It’s telling you that Lettuce attempted to run one scenario, and that the two steps within that scenario aren’t implemented yet (remember, each step maps to a method in your code). And because Lettuce is great, it gives you some code to help you implement those two steps.

The Terrain File

Lettuce keeps all of it’s settings and configuration is a file called terrain.py in the root of your Django project. It’s here that we’re going to configure the test database, Firefox, and Selenium. Go ahead and create a terrain.py file in the root of your Django project, and drop the following in it.

from django.core.management import call_command
from django.test.simple import DjangoTestSuiteRunner
 
from lettuce import before, after, world
from logging import getLogger
from selenium import webdriver
 
try:
	from south.management.commands import patch_for_test_db_setup
except:
	pass
 
logger = getLogger(__name__)
logger.info("Loading the terrain file...")
 
@before.runserver
def setup_database(actual_server):
	'''
	This will setup your database, sync it, and run migrations if you are using South.
	It does this before the Test Django server is set up.
	'''
	logger.info("Setting up a test database...")
 
	# Uncomment if you are using South
	# patch_for_test_db_setup()
 
	world.test_runner = DjangoTestSuiteRunner(interactive=False)
	DjangoTestSuiteRunner.setup_test_environment(world.test_runner)
	world.created_db = DjangoTestSuiteRunner.setup_databases(world.test_runner)
 
	call_command('syncdb', interactive=False, verbosity=0)
 
	# Uncomment if you are using South
	# call_command('migrate', interactive=False, verbosity=0)
 
@after.runserver
def teardown_database(actual_server):
	'''
	This will destroy your test database after all of your tests have executed.
	'''
	logger.info("Destroying the test database ...")
 
	DjangoTestSuiteRunner.teardown_databases(world.test_runner, world.created_db)
 
@before.all
def setup_browser():
	world.browser = webdriver.Firefox()
 
@after.all
def teardown_browser(total):
	world.browser.quit()

In your settings.py file, you’re going to need some additions too.

# Nose
TEST_RUNNER = 'django_nose.NoseTestSuiteRunner'
 
# Lettuce
LETTUCE_SERVER_PORT = 9000

We use the Nose test runner because it’s faster than Django’s default test runner, and we change the server port for running tests so it doesn’t collide with our development server. At this point if you run `./manage.py harvest` again, you’ll still get notices for unimplemented steps, but you’ll also see Firefox open and close real quick. That means we’ve done our job correctly.

Your First Step Definition

Alright, lets make something happen. If you look at the output from the harvest command, you’ll see that it gave you some code to help you implement the new steps that you wrote. Go ahead and copy that code into the bottom of the terrain.py file (and make sure to import ‘step’ at the top). Now, re-run ./manage.py harvest. You should get the following output.
Lettuce - Failing ouput
So why did our steps fail? If you look that the code that was generated for you, there is a line that essentially says “False is equal to some string”. This is obviously not true, so our step fails. So why don’t we make the test pass? We’re going to change a few things:

  • Change the decorator – We want this step to match even if we use other Gherkin keywords like “when”, “and”, and “then”.
  • Change the function name and args – “group1″ isn’t very descriptive
  • Write the code – We need this to do something, and right now it doesn’t!
@step(u'I am at "([^"]*)"')
def i_am_at_url(step, url):
    world.browser.get(url)

Now if you run ./manage.py harvest command again your tests will still fail, but this time for a different reason. The reason is the url that we’re passing into the step definition isn’t well formed. We were hoping to be able to pass relative urls in, but we can’t. So go ahead and modify the step in your scenario to look like this.

Given I am at "http://127.0.0.1:9000/quick-test/"

Run ./manage.py harvest again. You’ll see one passing test and one failing test!
Lettuce - Passing and failing

To make the next step pass, we need to make our web page a bit more formal. Go ahead and create a folder called “templates” inside of the “blog” app. Inside that folder, add a file called “base.html” and populate it with:

<html>
	<head>
		<title>Learning Lettuce!</title>
	</head>
	<body id='content'>
		{% block content %}{% endblock %}
	</body>
</html>

Now create a file called “blog.html” inside the same folder. Give it the following content:

{% extends "base.html" %}
 
{% block content %}
Hello testing world!
{% endblock %}

You’ll also need to update the view:

from django.shortcuts import render_to_response
 
def quick_test(request):
	return render_to_response("blog.html", {})

And you’ll need to update your settings file.

## Add this at the top of settings.py
import os.path
root = os.path.dirname(__file__).replace('\\','/')
 
## Make your TEMPLATE_DIRS variable look like this
TEMPLATE_DIRS = (
    root + "/../blog/templates/",
)

Now that our template is more formalized, lets update the step definition in “terrain.py”.

@step(u'I should see "([^"]*)"')
def i_should_see_content(step, content):
	if content not in world.browser.find_element_by_id("content").text:
		raise Exception("Content not found.")

This code explains itself pretty easily. We check to see if the content that is passed in via the step exists inside the body of the page. This has a few drawbacks:

  • What if we don’t want to check the body? What if we want to check a different element?
  • What if the content isn’t visible? (CSS hidden)

Since this is a simple example, we’re going to ignore these issues for now and just run our tests.
Lettuce - Passing tests!
Passing tests!

Next Steps

Now that you have passing steps, you’re well on your way to writing serious integration tests for your code. But there is still a lot more to learn. The next article in this series will cover using Lettuce Webdriver to handle common step definitions, tables, scenario outlines, and much much more.

All code related to this post can be found at https://github.com/vital101/Learning-Lettuce