blog, jake.ai-importin

A long time ago when I was working on NASA Nebula I was introduced to the idea of a CloudTop. CloudTop is the simple idea of using a VM hosted in a private or public cloud as your development environment, but also having tools to make bootstrapping your environment easy in ephemeral scenerios. The benefits of using this type of environment as a development environment is pretty clear:

  • Crazy fast 10Gbit internet
  • Regardless of the internet connection you’re currently using, as long as its fast enough to smoothly run an SSH session you will be able to download things to your development environment and very high speeds.
  • Access to 16core+ machines
  • Lots of ram, more ram that will fit in a macbook air

Over time this pattern has become much more common with the price of public cloud services dropping, and better tools for keeping your development environment in sync and working as you expect it.

Recently while trying to get back into the habbit of using a CloudTop I worked through how to use Amazon EC2’s spot instances as a CloudTop. This type of environment is a little quirky, and it requires a few tools to make it actually work well.

Getting Started

I’m going to go over the basics of how I setup a CloudTop environment on Amazon EC2 using a persistent spot instance request.

The Instance Type

For the last month I’ve been using the hi1.4xlarge (16 cores, 60GB ram, and 2x1TB ephemeral SSDs which I stripe together as 1 disk using RAID0). At the time of writing the average bid for this type of machine on us-west-2 is ~$0.14/hr. In an attempt to ensure that the spot instance doesn’t get reaped regularly I bid $0.2/hr for this instance type. This means that in a worst case scenerio I would pay ~$145/month for my development environment.

This may sound like a lot of money, but having pre-provisioned resources with all the tools I need to be effective is worth it.

If you’re interested in something a little more standard but still want the benefits of fast internet I can recommend the c3.xlarge instance type (4cores, 7.5GB ram, 2x40GB ephemeral SSDs). At the time of writing the c3.xlarge type runs at ~$0.03/hr on the spot market, which means you end up paying a little over $20/month for it.

The intricacies of spot instances

Spot instances are a little strange in that they can be terminated at any moment, and all of your data will be wiped. This presents two problems:

  1. If my development gets wiped every time spot prices increase, it’s a pain in the ass to reprovision everything when prices come back down. I address this with a few things. First: my spot requests are “persistent” meaning if the instance gets reaped because of price increase, it will come back when the prices return to normal. So that makes sure my CloudTop is up when it is financially possible. The other thing that I do is create an AMI image using Packer.io that contains my standard development environment on the root disk. This includes services like docker, and general tools like vim and tmux. I may publish a tutorial on how to use Packer for this, but the summary is that I just run my dotfiles repository to create the base environment I want. Using persistent spot requests means that whenever my development environment is terminated because spot prices increased, it will bounce back once prices are low enough, and all of my tools and services will be presnet.
  2. If I write code and do not push it back upstream to github, and my VM gets reaped I can potentially lose code. I address this by attaching a persistent EBS volume to the spot instance at /dev/xvdj which I mount to /home/ubuntu/persistent. When I’m working on code that I do not want to lose between cloudtop reboots I place it in this persistent directory. In the spot request you can map the device to a specific EBS volume so it will always come back with your data after a new spot instance has spawned.

The next time I get into the weeds with rebuilding an ami for cloudtops I might publish some of the scripts that I used to provision the cloudtop that I use and also tutorials on how to use packer with dotfiles and stuff. This post is meant to get the point across and the next post will be a technical how-to.