What’s New With Kernl – May 2016

Since last month I’ve been working hard on getting a few features out the door.  They are:

  • All new files are now hosted on S3 – Part of the work to make Kernl highly available is to get files hosted elsewhere.  When you upload a new version to Kernl, or push a change via webhooks, the deliverable now lives on S3.  Existing versions were a bit complicated, so thats going to be a task for May.
  • SSL and domain renewed – The SSL cert and domain for Kernl were renewed this month.  This should have been a completely transparent change.
  • Editable version fields – For plugins, you can now edit a few fields on a version once it has been created.  This was a pre-requisite for getting changelogs implemented nicely.
  • Plugin changelog API – You can now programmatically add, get, and remove changelog entries from you plugins.   Documentation on this feature is available at https://kernl.us/documentation/api#changelog-api and full examples are available at https://github.com/vital101/Kernl-API-Examples
  • Plugin changelog tab – The changelog tab in the plugin detail update window is now populated automatically and looks like the wordpress.org version.

So whats on backlog for May?

  • Moving the legacy version files to S3.
  • Moving the database to its own server + adding a replica.
  • Moving Memcache to its own server.
  • Analytics

Getting around memory limitations with Django and multi-processing

I’ve spent the last few weeks writing a data migration for a large high traffic website and have had a lot of fun trying to squeeze every bit of processing power out of my machine. While playing around locally I can cluster the migration so it executes on fractions of the queryset. For instance.

./manage.py run_my_migration --cluster=1/10
./manage.py run_my_migration --cluster=2/10
./manage.py run_my_migration --cluster=3/10
./manage.py run_my_migration --cluster=4/10

All this does is take the queryset that is generated in the migration and chop it up into tenths. No big deal. The part that is a big deal is that the queryset contains 30,000 rows. In itself that isn’t a bad thing, but there are a lot of memory and cpu heavy operations that happen on each row. I was finding that when I tried to run the migration on our Rackspace Cloud servers the machine would exhaust its memory and terminate my processes. This was a bit frustrating because presumably the operating system should be able to make use of the swap and just deal with it. I tried to make the clusters smaller, but was still running into issues. Even more frustrating was that this happened at irregular intervals. Sometimes it took 20 minutes and sometimes it took 4 hours.

Threading & Multi-processing

My solution to the problem utilized the clustering ability I already had built into the program. If I could break the migration down into 10,000 small migrations, then I should be able to get around any memory limitations. My plan was as follows:

  1. Break down the migration into 10,000 clusters of roughly 3 rows a piece.
  2. Execute 3 clustered migrations concurrently.
  3. Start the next migration after one has finished.
  4. Log the state of the migration so we know where to start if things go poorly.

One of the issues with doing concurrency work with Python is the global interpreter lock (GIL). It makes writing code a lot easier, but doesn’t allow Python to spawn proper threads. However, its easy to skirt around if you just spawn new processes like I did.

Borrowing some thread pooling code here, I was able to get pretty sweet script running in no time at all.

import sys
import os.path
from util import ThreadPool
def launch_import(cluster_start, cluster_size, python_path, command_path):
    import subprocess
    command = python_path
    command += " " + command_path
    command += "{0}/{1}".format(cluster_start, cluster_size)
    # Open completed list.
    completed = []
    with open("clusterlog.txt") as f:
        completed = f.readlines()
    # Check to see if we should be running this command.
    if command+"\n" in completed:
        print "lowmem.py ==> Skipping {0}".format(command)
        print "lowmem.py ==> Executing {0}".format(command)
        proc = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
        output = proc.stdout.read() # Capture the output, don't print it.
        # Log completed cluster
        logfile = open('clusterlog.txt', 'a+')
if __name__ == '__main__':
    # Simple command line args checking
        lowmem, clusters, pool_size, python_path, command_path = sys.argv
        print "Usage: python lowmem.py <clusters> <pool_size> <path/to/python> <path/to/manage.py>"
    # Initiate log file.
    if not os.path.isfile("clusterlog.txt"):
        logfile = open('clusterlog.txt', 'w+')
    # Build in some extra space.
    print "\n\n"
    # Initiate the thread pool
    pool = ThreadPool(int(pool_size))
    # Start adding tasks
    for i in range(1, int(clusters)):
        pool.add_task(launch_import, i, clusters, python_path, command_path)

Utilizing the code above, I can now run a command like:

python lowmem.py 10000 3 /srv/www/project/bin/python "/srv/www/project/src/manage.py import --cluster=" &

Which breaks the queryset up into 10,000 parts and runs the import 3 sets at a time. This has done a great job of keeping the memory footprint of the import low, while still getting some concurrency so it doesn’t take forever.

Shuttering Side Projects

Over the past few years I’ve slowly accumulated some big side projects. They weren’t done for clients, but just for myself. At some point maintenance of these side projects isn’t fun anymore and hinders the creative juices. I have other things I want to work on, but having these other zombie side projects feels too much like an albatross around my neck.

After much deliberation, I’ve decided to shut down two of my large side projects: BookCheaply and Smooth Bulletin. I really believe both of these projects could do someone some good, but they were both learning projects for me and I don’t see them moving forward anymore. Effective immediately I’m disabling their Apache configs, backing up their DBs, TARing it all together, and putting it somewhere safe. I’ll keep access to the Git repos, but eventually I’ll clone a copy of those out too and archive them. If I don’t get them out of the way completely, I feel like I’ll want to work on them too much.

Shuttering these projects marks a transition for me, where I move from using Python and Django on side projects to Node.js, Express, and Angular. While apprehensive about abandoning my go-to stack for side projects, I’m excited to learn the nooks and crannies of Node (and I still use Python/Django for my day job anyways).

Here’s to the future!

Smooth Bulletin

Restoring a 14 year old website

If you want to skip the restoration stuff, the final product can be found at http://re-cycledair.com/starwars/Entrance.html.

A long time ago in an era just before the first dot-com bust, I was a bright-eyed child with aspirations to make the greatest Star Wars web site of all time. With only Microsoft Front Page Express and a limited knowledge of HTML in hand, the 1999 version of me created the best Star Wars web site the world had ever seen.

Unfortunately I was hosting it on my ISPs free web hosting area. When that ISP was later gobbled up by another company, all traces of my website were gone… or so I thought. Recently I discovered that most of it still exists on the Way Back machine, however it’s in poor shape. Many of the assets don’t exist anymore, the linking is broke, and the frames just don’t work right in modern browsers. This is my journey to save this little piece of history while using minimal modern techniques. My goal is to have a legitimate 1999 website when I’m done and have it render nicely on most modern web browsers.

The Internet Archive

The Wayback machine at the Internet Archive has long been one of my favorite websites. I love getting nostalgic for the web of the past, my childhood, and history in general. It was awhile ago that I realized it still had some of my original Star Wars website. If you want, check out the capture from 2001. It doesn’t work much at all, but it gives you a good idea of what we’re working with here.
Star Wars Restoration - Before

In addition to the main page I also had an “Entrance” page, because you know, every cool web site had an entrance page back then. This one is in slightly better shape, meaning the star field background survived intact.
Star Wars Restoration - Entrance Before

The Game Plan

My goal for this restoration is to make the website display great in modern web browsers with minimal code changes and without using too many modern techniques. At that point in time, CSS wasn’t really a thing that a ton of people used yet, and even if it was it was beyond my understanding, so I’m going to try and avoid it if at all possible. Javascript was available in most browsers by that time, but I didn’t understand it, so I didn’t use it. However, we do have FRAMES(!), which is going to make things super exciting for everyone still reading.

Now that we’ve laid down some ground rules, here is the order of attack:

  1. Entrance: Fix / replace broken images.
  2. Entrance: Clean up the HTML (alignment) and remove anything that the Internet Archive crawler added.
  3. Main site: Fix / replace broken images
  4. Main site: Remove dead links. Sadly the Internet Archive couldn’t capture everything.
  5. Main site: Fix any styling issues that remain.
  6. Main Site: Clean up HTML (alignment) and remove anything that the Internet Archive crawler added.

Lets get started!

Entrance: Replacing Broken Images

While it’s fantastic that the background star field has survived, the rest of the images didn’t. This is sort of a problem because I have to reach pretty far back into my memory to figure out what was there originally. Luckily, 1999 me wasn’t completely terrible at naming things.

Star Wars Restoration - Entrance File Names 1

From these file names and memory, I’m going to say that `tie.gif` was a rotating TIE fighter gif, and `starwars.gif` was the Star Wars logo that survived on the main section of the site.

Star Wars Restoration - Entrance File Names 2

And from this file name we get nothing. Luckily I remember this being an image of Tatooine, roughly 350px x 100px. This site came out a bit before Episode I, so I’m going to assume it was an image from one of the Episode I trailers. At least, thats how I remember it.

After a bit of digging, I found replacement images for everything but the Tatooine banner. In it’s place, I found a light saber gif that I definitely used on this site somewhere in the past. I also updated the star field to be a little more 1999.

Star Wars Restoration: Entrance Complete

Entrance: HTML Clean Up

Now that the entrance is starting to look like a real 1999 web site again, we can start to clean up the HTML and fix the broken link to enter the site. The entrance area contains 3 pages, 1 to bring the frames together, and 1 for each frame. They currently look like this:



The code isn’t too bad. With a bit of alignment help and some nesting, things are going to look pretty good. Now check out the final product:




Main Site: Fix Broken Images

Now that we have the entrance out of the way, I’ve have a pretty good idea of how this whole restoration will work for the rest of the site. Let’s start by fixing the background on the main site so things are a bit more workable.

Star Wars Restoration: Main site with background

Now we’re talking! There are few things that I noticed immediately after getting the background set.

  • 1999 me had some pretty sweet design skills
  • I should use more animated gifs on my websites
  • The frame content on the left isn’t staying inside of it correctly, so I had to scroll to the center to get it to work.
  • I had a Yahoo! club!
  • I wasn’t so great with words back then. (I was a kid, give me a break)

Looking at the source, there are a few images that aren’t linking quite right.

  • Left side: starwars.gif – I think this was just a smaller version of the big logo.
  • Left side: deathstar.gif – Definitely a gif of the Death Star. It might have been animated, but I think it was just transparent so that it looked awesome.
  • Right side: lettersabove.gif – I honestly don’t know, but it might have been “A long time ago in a galaxy far far away…”
  • Right side: pulselightsaber.gif – A pulsing light saber page break. I might just use the static one from the entrance instead.
  • Right side: xwing.gif – A rotating X-Wing fighter. Because nothing says “Email me!” like a rotating X-Wing.

After tracking down many of the original images that I used here, the home page now looks like this.

Star Wars Restoration: Home Page Finished

Looking good! Next up, code clean up.

Main Site: Code Clean Up

The code for the main site is in pretty poor condition. There is a ton of code in there from the Internet Archive, it has poor indentation, and in general just isn’t anywhere near the quality of the entrance. Given the amount of code here, I thought using a Github Gist would be better.

Before Code: https://gist.github.com/vital101/56b2783b8e75b00fdac9

After Code: https://gist.github.com/vital101/2d1e6a44ac3ddbb1789d

As you can see I went ahead and re-aligned everything so that it’s more readable, took out all of the javascript and css added by the Internet Archive, and condensed the code a little bit so there wasn’t large bits of whitespace between elements.

The next step in the code cleanup is fixing URLs. If you look at the code, you’ll notice that all the links still point to the Internet Archive. That obviously isn’t going to work for us, so I simply need to replace “https://web.archive.org/web/20010811043323/http://my.voyager.net/~hands/star-wars/” with an empty string, making the link relative.

After fixing links, I found that many of the sub-pages have been lost to time, but there are still a few for us to look at including:

  • Luke Skywalker
  • Darth Vader
  • Princess Leia
  • Chewbacca
  • G.Moff Willguf Tarkin
  • Star Wars: Rogue Squadron
  • Weapons
  • B-Wing
  • Snowspeeder
  • Corellian Corvette
  • Mon Calamari Cruiser
  • At-AT
  • Death Star
  • Chat
  • Movie

Fixing up these pages follows a similar pattern to the entrance page and the main site, so I’ll spare you the details. There were a few interesting things that popped up while I was doing the sub-page restoration though.

  • Plagiarism – 1999 me definitely ripped off somebody else with most of the encyclopedic data on characters, weapons, and ships. The change in voice is a dead giveaway.
  • Front Page Versions – It appears that I made the entrance and the menu/main page of the main site in FrontPage 3.0, but all of the sub-pages in FrontPage 2.0 (express). FrontPage Express was shareware that came bundled with IE 4 so I know how I got that. I believe I had access to FrontPage 3.0 at my school at that time, which I probably used to generate the frames.
  • Xoom – I had completely forgotten about it, but Xoom used to be a free unlimited web host back in the early dot-com days. I used it to host videos, games, and other things. Unfortunately my Xoom site wasn’t crawled by the Internet Archive.
  • Java Chat – Back in 1999, there weren’t a whole lot of options for getting a chat room going on your website. Using a Java applet was basically it, so thats what I did. It looks like Xoom offered it’s members a free chat tool, so how could I resist?
  • Game Demo – Ah, the good old days. The one game page that survived has some pretty hefty system requirements: Windows 95/98, 32 MB of RAM, Pentium 166. Direct3D card, DirectX 6. Oh, and I also made sure to let people know the estimated download time on 28.8 kbps dial-up modem.

Final Thoughts

The final product can be found at http://re-cycledair.com/starwars/Entrance.html.

While going through this restoration I realized that this is when I knew what I wanted to do with my life. My parents had no idea what I was doing, but they realized that it stimulated my mind so they let me continue to do it anyways. Without that kind of encouragement and freedom to create, I probably wouldn’t be where I am today. This restoration has also re-kindled my respect for the old web. It was simple, but I’m extremely happy that the web has progressed to where it is today.

HealthCare.gov: It almost worked

Ever since HealthCare.gov launched all I’ve been hearing about is how horrible the experience is. It never affected me directly, so I chalked it to a few vocal detractors trying to sway public opinion. I even went so far as to go to HealthCare.gov and use the logged-out experience to compare some plans. It looked pretty good actually, and I tweeted as such.

HealthCar.gov Tweet

That was all about to change though. I recently made the move to work at a small startup that doesn’t yet have an employer-sponsored health plan. They do however offer a stipend of sorts to help you pay for insurance costs on HealthCare.gov. And with that, my journey began.

Initial Sign Up

After discussing the healthcare situation with my wife, I ran the numbers and decided that it would be best for her to use her employer-sponsored healthcare and for me to use the exchange separately. At this point everything went as well as it could. Even though the UX on HealthCare.gov is pretty bad, I was able to figure out how to fill out an application, select health coverage, select dental coverage, and get things rolling. This all happened before December 23rd, which means I’d be getting coverage by January 1st! All was well… until I had to make a change.

The Change

As it turns out, I math’d wrong. When I was calculating the cost of healthcare for my wife at her job, I was off by something like $200 per month, which pushed her healthcare costs well into the “unreasonable for what I’m going to receive” range. With that in mind, I thought I could make a quick change on HealthCare.gov to get her on the same plan that I was. Boy was I wrong.

The process I had to take to get both my wife and myself health and dental insurance was ridiculous, and still hasn’t worked correctly. To add my wife to my plan, I had to delete the entire thing. Not only that, but I needed to delete my entire application and start over from scratch. Once I did that, I kept constantly running into javascript errors in their application which would force me to re-authenticate.

After finally getting through the application process, I get to the last step where I need to confirm. I press the button and… nothing. Not a goddamn thing. Being a software engineer, I do a little investigating and find out that the server is returning 500 errors to the client (in non-developer speak, this means that the server couldn’t process my request because of some error on their side). The error that is returned says that I should try to log-out, then resume the application. I do this, but when I try to resume my application I get:

Thats right, my application is locked. It doesn’t give any reason, which is especially confusing considering that I was asked to re-authenticate. But there is a little link that can “explain this task”, so I click it, and it naturally didn’t work.

At this point, I was beginning to realize that I would need to contact their support to get things resolved.

Support and Next Steps

The first time I tried to use support, I clicked the helpful little “Live Chat” button. I then proceeded to wait for 25 minutes with no response. This was over the holidays, so I gave it another try after Christmas. This time I was connected to somebody, but no matter what question I asked I was given a canned response to contact the call center. My question is: Why have the “Live Chat” at all if you’re just going to tell me to contact the call center? It’s ridiculous and a waste of my time.

After my initial experience with HealthCare.gov, I’m not really excited to contact the call center. I honestly don’t have a ton of time to do it, and given the number of people that will be signing up for healthcare I’m sure the wait will be long. My likely next steps will be to cancel what remains of my application and start over. If it doesn’t work this time, then I’m going to cancel that application and contact my insurance provider of choice directly. In my case thats Blue Cross Blue Shield of Michigan, which happens to have a walk-in office about a block from where I work.

I create things all the time on the web. Its my chosen profession, so I know how hard it can be to make a good website when you need to integrate with a lot of different 3rd parties. However this sort of experience isn’t going to cut it. If I’m federally mandated to have health insurance, then the experience should be as painless as possible.

Because the system is broke I won’t have health coverage until February 1st now; a month long gap in coverage for my family. This just isn’t acceptable.

Why I Still Have Faith in Hacker News

If you browse Hacker News enough you’ll start to see article trends.

  • Startup X is Startup Y for Dogs
  • Why you should use framework Z over formerly hot framework B
  • Cool Javascript demo is N lines of code
  • How I failed at A and it made me better at B.

Don’t get me wrong, some of this stuff is interesting, but most of it is just noise to me. The reason I come to HN is for the comments. Thats where the good stuff is. HN is one of the brightest internet communities I’ve ever had the pleasure to interact with. For instance, today I posted to HN asking for the community to review my startup Smooth Bulletin. Within an hour I had two very thoughtful comments about business and market ideas that I hadn’t even thought of. I expect I’ll probably get even more feedback as the day goes on, and that I’ll benefit from those just as much as the first comments.

Thats why I still have faith in Hacker News. Not because of the content, but because of the people. The people there are incredibly smart and willing to give their advice just so that maybe you can succeed one day. I’ve been a member of HN for several years now and this hasn’t changed at all since the time I joined, and I hope that it never does.

And yes, I posted this to Hacker news, so maybe I should add “Why I still have faith in X” to the list.

28 Questions

Over the past few years I’ve tried my hand at launching a couple of different start ups. Over this time period, I’ve collected a set of questions from many different sources (HN, YCombinator, TechStars, etc) that I ask myself before proceeding. They help be decided if this is a real or manufactured problem, if it can be profitable, and if people will buy what I’m selling. My latest project idea is the only idea I’ve had that passes all of these questions to my satisfaction. How do yours hold up?

  1. What are you going to build?
  2. What is the actual problem?
  3. How will you sell your product/service?
  4. What are some potential obstacles?
  5. What are some existing options that solve this problem? How are you different?
  6. Who needs what you’re making?
  7. How do you know that they need it?
  8. How is the problem being solved now? Is it being solved now?
  9. Why isn’t this being done your way already?
  10. How will customers find out about you?
  11. What resistance will they have to trying your product?
  12. What are some key things about your project that outsiders don’t understand?
  13. Who will your first paying customer be?
  14. How might you expand if your initial idea succeeds?
  15. Why did you choose to work on this idea?
  16. Six months from now, what will be you biggest problem?
  17. What are the hard parts of this idea?
  18. Who would you hire/add to your team?
  19. What is the next step in product evolution?
  20. How does your product work?
  21. How big is the opportunity? [market]
  22. How do you know customers need what you’re making?
  23. What domain expertise do you have? Why should you be the one to do this?
  24. What part of your project will you build first? (could be business connections, hardware, software, etc)
  25. How much money could you make per year?
  26. How will you make money?
  27. What have you built in the past?
  28. How would you spend $5,000, how would you use it?

Using Git with Subversion(SVN) on a Non-Standard Repository Layout

For the longest time I was a loyal Subversion(SVN) user. I know, it’s crazy, but I was. When I found out about Git, I was hooked immediately and used it for all of my personal projects. The problem was that at work we use SVN, and getting everyone to migrate to Git just wasn’t in the cards. Much to my surprise, Git has the ability to interact with a SVN repository, so I could still use it anyways.

The issue with my work’s current SVN layout is that it is non-standard. By that I mean all projects exist in one big happy repository. Something like:

Repository -> Project -> Trunk/Branches/Tags

Unfortunately, Git+SVN isn’t really all that excited about working with non-standard repositories, so I had to do some experimentation and Googling to figure it all out. Eventually, I came up with the following steps:

Step 1: Clone the Repo
The first step in this process is actually cloning your SVN repository. By clone, we mean make a full copy of it and all of the revision history.

git svn clone svn://the.svn.server/allEncompassingRepo/project -trunk=trunk/ .

After the initial clone is complete, we move to fixing where git should look for things.

Step 2: Set up fetch, branches, and tags
The initial setup for fetch, branches, and tags gets screwed up at this point if you have a non-standard layout like my employer does, so we need to do some cleanup. Open .git/config and set the following:

url = svn://the.svn.server/allEncompassingRepo
fetch = project/trunk:refs/remotes/trunk
branches = project/branches/*:refs/remotes/*
tags = project/tags/*:refs/remotes/tags/*

Now that the config file is all set, go ahead and save.

Step 3: Pull down your files
The config file is all set, so you’re ready to do your first pull.
git svn fetch svn
That’s all there is to it. If you’d like to know how to use Git+SVN, I suggest reading the fine article over at Viget.