What’s New With Kernl – September 2017

It’s been great month of development for Kernl! We saw continually increasing traffic, a new cohort of customers, and some great work around making continuous delivery easier to use.

Features

Dropdown Branch Selection for Plugins & Themes – As part of the push to make continuous deployment easier for our customers, you no longer need to type branch names into Kernl. When you select a repository to build the branch selector automatically populates with the branches that are available in the source system. This will hopefully cut down on confusion and instances of typing the branch name wrong.

Github & Gitlab: Deploy Keys & Webhooks – Easily the most exciting feature from this last month of development is the ability for Kernl to fully manage your deploy keys and webhooks on Gitlab and Github. What this means is that when you create, update, or delete a plugin/theme, we automatically manage the deploy keys and webhooks in the source system. The goal is that you don’t have to touch any of the settings in your repository to make Kernl work. The process is currently opt-in (you’ll see the checkbox). We had hoped to get Bitbucket working this month as well, but it didn’t make the cut. Our goal is to have this working sometime in September.

Documentation Updates – As hard as we try to keep Kernl’s documentation up to date sometimes we miss things. Pictures get out of date, 3rd party interfaces change, and any number of other issues crop up. It took some time but we went through and checked the documentation for accuracy and up to date pictures.

Bug Fixes & Other

  • A bug in our Memcached client was causing degraded performance occasionally. It appears that the client would sometimes drop it’s connection and be unable to re-connect. This has been resolved.
  • The plugin/theme latest-version endpoint now includes a “downloadUrl” field.
  • Minor updates to the marketing page to make pricing clearer.

What’s New With Kernl – April 2017

March was a light month for Kernl development for a couple of different reasons. The first is that I was on vacation for 10 days, and the second is that I bought and moved into a new house. Now that things are settling down development is ramping up again. There were still a few things that got completed though:

– Lots of great stuff accomplished on Feature Flags. They are not ready for prime time yet, but I’ve started dog-fooding them and so far things are working out great.

– There was a bug in the changelog.json code where the last version would always be one behind in certain situations.

– A regression was fixed where you couldn’t deploy from branches.

– I wrote a blog post about my experience building Kernl for the Freemius blog.

Kernl: Important BitBucket Changes

It came to my attention that the way BitBucket handles deployment keys has changed. Until recently the same deployment key could be shared across multiple repositories. That rule has been changed and now each repository requires a unique deployment key. So what does this mean for you? You’ll need to take a few steps to make sure that your “push to build” functionality continues to work as you expect it to.

  1. I’ve deployed changes that allow you to add unique deployment keys to all of your repositories. For those of you with a lot of repositories this is going to be pretty tedious, but in the end it will give you greater access control to your repositories. Documentation for adding deployment keys can be found at https://kernl.us/documentation#deploy-key , but you likely won’t need it. Just go to “Continuous Deployment” and then click “Manage Deployment Keys” (if you don’t see that button, hard refresh).
  2. Starting tomorrow (February 21, 2017) at 7pm EST, access with the old Kernl deployment key will be cut off. From this point forward only the new deployment keys will be able to access your repository.
  3. After February 21, 2017 @ 7pm EST you can delete the old Kernl deployment key from your repositories. If you do it before then your builds will fail.

Sorry for the short notice and inconvienience of this change, but it’s necessary to make sure that all customers are able to deploy continuously with Kernl. If you have any questions or concerns about this change, please reach out. And once again, sorry for this inconvience!

What’s New With Kernl – February 2017

It’s been a little while since my update, so let’s dive right in to what’s new.

  • Kernl now has an enhanced billing area. The mechanism for paying an expired invoice was pretty confusing and it wasn’t possible to see your past invoice amounts. With these changes you can now see the last 10 invoices from Kernl and if you ever need to pay an out-of-date invoice that process is much simpler.
  • Purchase codes can now be limited to a domain. This is handy if you don’t want your customers buying a single license and using it on many sites.
  • Work has started on feature flags! Feature flags are a software development best practice of gating functionality. Functionality can be deployed “off”, then turned on via the feature flag, separate from deployment. With feature flags, you can manage the entire lifecycle of a feature. This is _super_ useful for the WordPress community because it allows you to turn functionality on/off without creating and deploying a new version. You can roll out flags based on a boolean on/off, percentage of users, or just to specific users. If you’d like to be part of the beta, let us know.
  • Kernl’s 3 Node.js app servers were upgraded. They now have 1GB of RAM per server instead of 512MB.

If you have any questions, thoughts, or concerns, feel free to reach out on Twitter or comment here.  Cheers!

0 to 1 Million: Scaling my side project to 1 million requests a day

In the Beginning

In late 2014 I decided that I needed a side project.  There were some technologies that I wanted to learn, and in my experience building an actual project was the best way to do that.  As I sat on my couch trying to figure out what to build, I remembered an idea I had back when I was still a junior dev doing WordPress development.  The idea was that people building commercial plugins and themes should be able to use the automated update system that WordPress provides.  There were a few self-managed solutions out there for this, but I thought building a SaaS product would be a good way to learn some new tech.

Getting Started

My programming history in 2014 looked something like: LAMP (PHP, MySQL, Apache) -> Ruby on Rails -> Django.  In 2014 Node.js was becoming extremely popular and MongoDB had started to become mature.  Both of these technologies interested me, so I decided to use them on this new project.  As to not get too overwhelmed with learning things, I decided to use Angular for fronted since I was already familiar with it.

A few months after getting started, I finally deployed https://kernl.us for the world to see.  To give you an idea of the expectations I had for this project, I deployed it to a $5/month Digital Ocean droplet.  That means everything (Mongo, Nginx, Node) was on a single $5 machine.  For the next month or two, this sufficed since my traffic was very low.

The First Wave

In December of 2014 things started to get interesting with Kernl.  I had moved Kernl out of a closed alpha and into beta, which led to a rise in sign ups.  Traffic steadily started to climb, but not so high that it couldn’t be handled by a single $5 droplet.

Around December 5th I had a customer with a large install base start to use Kernl.  As you can see the graph scale completely changes.  Kernl went from ~2500 requests per day, to over 2000 requests per hour.  That seems like a lot (or it did at the time), but it was still well within what a single $5 droplet could handle.  After all, thats less that 1 request per second.

Scaling Up

Through the first 3 months of 2015 Kernl experienced steady growth.  I started charging for it in February, which helped fuel further growth as it made customers feel more comfortable trusting it with something as important as updates.  Starting in March, I noticed that resource consumption on my $5 droplet was getting a bit out of hand.  Wanting to keep costs low (both in my development time and actual money) I opted to scale Kernl vertically to a $20 per month droplet.  It had 2GB of RAM and 2 cores, which seemed like plenty.  I knew that this wasn’t a permanent solution, but it was the lowest friction one at the time.

During the ‘Scaling Up’ period that Kernl went through, I also ran into issues with Apache.  I started out by using Apache as a reverse proxy because I was familiar with it, but it started to fall over on me when I would occasionally receive requests rates of about 20/s.  Instead of tweaking Apache, I switched to using Nginx and have yet to run in to any issues with it.  I’m sure Apache can handle far more that 20 requests/s, but I simply don’t know enough about tweaking it’s settings to make that happen.

SCaling Out & Increasing Availability

For the rest of 2015 Kernl saw continued steady growth.  As Kernl grew and customers started to rely on it for more than just updates (Bitbucket / Github push-to-build), I knew that it was time to make things far more reliable and resilient than they currently were.  Over the course of 6 months, I made the following changes:

  • Moved file storage to AWS S3 – One thing that occasionally brought Kernl down or resulted in dropped connections was when a large customer would push an update out.  Lots of connections would stay open while the files were being download, which made it hard for other requests to get through without timing out.  Moving uploaded files to S3 was a no-brainer, as it makes scaling file downloads stupid-simple.
  • Moved Mongo to Compose.io – One thing I learned about Mongo was that managing a cluster is a huge pain in the ass.  I tried to run my own Mongo cluster for a month, but it was just too much work to do correctly.  In the end, paying Compose.io $18/month was the best choice.  They’re also awesome at what they do and I highly recommend them.
  • Moved Nginx to it’s own server – In the very beginning, Nginx lived on the same box as the Node application.  For better scaling (and separation of concerns) I moved Nginx to it’s own $5 droplet.  Eventually I would end up with 2 Nginx servers when I implemented a floating ip address.
  • Added more Node servers – With Nginx living on it’s own server, Mongo living on Compose.io, and files being served off of S3, I was able to finally scale out the Node side of things.  Kernl currently has 3 Node app servers, which handle requests rates of up to 170/second.

Final Thoughts

Over the past year I’ve wondered if taking the time to build things right the first time through would have been worth it.  I’ve come to the conclusion that optimizing for simplicity is probably what kept me interested in Kernl long enough to make it profitable.  I deal with enough complication in my day job, so having to deal with it in a “fun” side project feels like a great way to kill passion.

What’s New With Kernl – November 2016

It’s been a long time since the last Kernl update blog, so lets get right into it.

Big Features

  • GitLab CI Support – You can now build your plugins and themes automatically on Kernl using GitLab.com!  We’ve had support for GitHub and BitBucket for a long time, and finally figured out a good way to make things work for GitLab.  See the documentation on how to get started.
  • Slack Build Integration – If you are a slack user, you can now tell Kernl where to publish build status messages.
  • Replay Last Webhook – Sometimes when you’re running a CI service with Kernl it would be useful to re-try that last push that Kernl received.  You can now do that on the “Continuous Integration” page.

Minor Features

  • Repository Caching – We now do some minor caching of your git repositories on the Kernl front end.  The first load will still reach out to the different git providers, but subsequent loads during your sessions will read an in-memory cache instead.
  • Better Webhook Log Links – Instead of displaying a UUID, the webhook build log now displays the name of the plugin or theme.

Other

  • Miscellaneous Upgrades – Underlying OS packages and Node.js packages were upgraded.
  • Payment Bug Fixes – There were a few minor bugs that kept showing up if someone’s credit card expired.  This fix hopefully allows for a more self-service approach.
  • Minor copy changes – A few changes were made to the wording on the Kernl landing page.

What’s next?

  • It’s been a few months since Ubuntu 16.04 LTS came out, so I’ll be spending significant amounts of time upgrading our infrastructure to the latest LTS version.
  • If our load balancer goes down right now, everything goes under.  A floating IP address between two load balancers will solve that issue and provide high(er) availability.
  • Better insights into purchase code usage and activity.

What’s New With Kernl – July 2016

With summer in full-swing here in the United States, development on Kernl has been slowing down to accommodate much busier schedules than during the rest of the year.  This doesn’t mean we haven’t been busy though.

Features

Infrastructure, Bugs, and Miscellaneous

  • When the server throws a 500 error, it renders the correct template.  Prior to this fix Kernl would render a 404 page, which made it very hard to tell when you encountered an actual problem.
  • We now have a robots.txt file!
  • Kernl’s Mongo infrastructure has been moved to Compose.io.  Having a professional DBA manage Kernl’s database helps me sleep easier at night and provides customers with a more performant and stable backend.
  • The landing page for Kernl was taking over 1 second to load for many people.  Caching was added, and we now have the number down to under 100ms on average.

What’s next?

July is a busy month outside of Kernl, so I don’t expect much to get done.  The current plan is to take it easy in July and then come back with renewed vigor in August.

What’s New With Kernl – June 2016

The past month of work on Kernl has seen a lot of great infrastructure improvements as well as a few customer facing features that I’m pretty excited about.

Customer Facing Features

  • Direct Uploads to AWS S3 – When Kernl was originally created all file uploads were stored directly on Kernl’s servers.  As we grew, this became an unsustainable solution, so the process changed to just use Kernl’s servers as temporary holding space before putting the file on S3.  This month we made this process even better by having files upload directly to S3. For you, this means faster uploads and less time waiting to get updates out to your customers.
  • Expiring Purchase Codes – You can now create purchase codes that expire on a specific date.  This allows you to sell your updates over time, instead of having to give them away for free for the life of the plugin or theme.
  • Max Download Purchase Code Flag – You can configure a purchase code to only allow a certain number of update downloads.  This will help resolve any issues with customers sharing purchase codes amongst themselves or across multiple installations.
  • JS Cache Busting – As customer facing features get rolled out Kernl automatically busts the client-side javascript cache for https://kernl.us.  This should help prevent confusion and remove the need for any sort of “hard refresh” when new features are released.
  • plugin_update_check.php Bug Fixes – There was an edge-case bug where some code in this file would collide with an old version of WP-Updates plugin update check file.  This happens when a customer has your plugin and also has a really old version of somebody else’s plugin installed.  This update takes care of that collision permanently.
  • Client-side JS Errors – A few minor miscellaneous bug fixes were performed on the front-end of Kernl.

Infrastructure

  • MongoDB – The month started off with Kernl’s database moving to it’s own server.  This was a temporary step that aimed to make the move to a highly available setup easier.
  • Mongo Replica Sets – After the first MongoDB move, the next step was to make the setup highly available.  Kernl now has 3 Mongo databases (1 master + 2 replicas).  In the event that the master database goes down, Kernl automatically fails over to one of the replicas with no downtime.
  • Memcache – Memcache was moved to it’s own server to make it easier to increase the number of items that Kernl caches over time.  This piece of the setup doesn’t need to be highly available.  If for some reason it goes down, Kernl will continue to operate fine.
  • Nginx – Nginx is used by Kernl both as a front-door to the application as well as load balancer between the app servers.  This was moved to it’s own server which allows it scale up when we need additional capacity.  In the future (hopefully soon), we’ll use a floating IP address to give this portion of the infrastructure the ability to fail over to a backup Nginx server.
  • Multiple App Servers – Kernl’s app servers can now scale horizontally.  We’re currently running 3 app servers which Nginx load balances traffic to.  This setup allows us to add app servers easily as our traffic grows.
  • Automated Deployment – Kernl can now be deployed with a single command.
A rough drawing of how Kernl is architected now.
A rough drawing of how Kernl is architected now.

What’s Next?

  • Caching the repository list that you see when you set up CI builds.
  • Get a rich text editor set up on the installation and description fields.
  • Theme change logs.
  • Wrap up infrastructure work.
  • Sign in / Sign up with BitBucket & GitHub.
  • Slack Integration.
  • HipChat Integration.