Month: October 2014

Map

Go is all of the rage, all my friends are doing it so I feel like I have to do it too… So I’ve been playing with Go and Docker a bit recently to try and learn more about go and how I can use it for work and side projects.

I’m enjoying things so far, and I’m finding interesting tidbits as I go. This is one of them.

With go it is easy to build staticly linked binaries which can be distributed. This is awesome by itself, but with docker it is even more awesome.

You can build your go binaries like so:

CGO_ENABLED=0 go build -a ../src/worker.go

Once you’ve built your staticly linked binary you can build a docker image on top of the scratch base image. You can read about the Scratch Image on the docker site, but basically its an empty image that you can drop your binary into that will run as a very minimal container.

FROM scratch
MAINTAINER Jake Dahn <[email protected]>
ADD bin/worker /worker
ENTRYPOINT ["/worker"]

Below is output of some shell output from building and running a docker container that contains a go binary. Notice the image size is 5MB. If I use ubuntu or other distros as a base image my docker image is significanly larger:

[email protected]:/vagrant# docker pull scratch
Pulling repository scratch
511136ea3c5a: Download complete
Status: Image is up to date for scratch:latest

[email protected]:/vagrant$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
scratch             latest              511136ea3c5a        16 months ago       0 B

[email protected]:/vagrant$ docker build -t jake .
Sending build context to Docker daemon 53.73 MB
Sending build context to Docker daemon
Step 0 : FROM scratch
 ---> 511136ea3c5a
Step 1 : MAINTAINER Jake Dahn <[email protected]>
 ---> Running in 27f068468755
 ---> a063241634df
Removing intermediate container 27f068468755
Step 2 : ADD bin/worker /worker
 ---> 5053eed5787c
Removing intermediate container 3a36a6c5e19b
Step 3 : ENTRYPOINT /worker
 ---> Running in 1f8014d919b5
 ---> ae0bd28d3e2e
Removing intermediate container 1f8014d919b5
Successfully built ae0bd28d3e2e

[email protected]:/vagrant$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
jake                latest              ae0bd28d3e2e        4 seconds ago       5.191 MB
scratch             latest              511136ea3c5a        16 months ago       0 B

[email protected]:/vagrant$ docker run -t -i jake
2014/10/24 05:40:02  [*] Waiting for messages. To exit press CTRL+C # this is a running go program that is a rabbitmq consumer

This is an interesting way to ship and deploy software.

In the early days of chrome extensions (january 2010) Jesse wrote an extension called Chromr. The general idea was simple and awesome — every time you open a new tab in chrome the page is a full-screen photo that comes from Flickr’s interestingness API.

I worked with him on it to improve style and the project was renamed More Interestingness and moved to our shared github organization.

I think More Interestingness is my alltime favorite chrome extension — it has led to many great conversations with coworkers and friends. Unfortunately on June 27 2014 Flickr made their API only support HTTPS/SSL encrtypted requests. This change silently broke the extension and I’ve spent the summer staring at blank newtab windows in chrome 🙁

Today it came up in conversation and I decided to take a look and fix things. Since the last updates in May of 2012 a lot of things have changed in the chrome extension environment, and after a few updates to the manifest.json file I was able to get things back in working condition.

more interestingness

You can download the More Interestingness chrome extension from the Chrome WebStore: https://chrome.google.com/webstore/detail/more-interestingness/ngddmdmkjnnefgggjnnnepijkcighifa

A long time ago when I was working on NASA Nebula I was introduced to the idea of a CloudTop. CloudTop is the simple idea of using a VM hosted in a private or public cloud as your development environment, but also having tools to make bootstrapping your environment easy in ephemeral scenerios. The benefits of using this type of environment as a development environment is pretty clear:

  • Crazy fast 10Gbit internet
  • Regardless of the internet connection you’re currently using, as long as its fast enough to smoothly run an SSH session you will be able to download things to your development environment and very high speeds.
  • Access to 16core+ machines
  • Lots of ram, more ram that will fit in a macbook air

Over time this pattern has become much more common with the price of public cloud services dropping, and better tools for keeping your development environment in sync and working as you expect it.

Recently while trying to get back into the habbit of using a CloudTop I worked through how to use Amazon EC2’s spot instances as a CloudTop. This type of environment is a little quirky, and it requires a few tools to make it actually work well.

Getting Started

I’m going to go over the basics of how I setup a CloudTop environment on Amazon EC2 using a persistent spot instance request.

The Instance Type

For the last month I’ve been using the hi1.4xlarge (16 cores, 60GB ram, and 2x1TB ephemeral SSDs which I stripe together as 1 disk using RAID0). At the time of writing the average bid for this type of machine on us-west-2 is ~$0.14/hr. In an attempt to ensure that the spot instance doesn’t get reaped regularly I bid $0.2/hr for this instance type. This means that in a worst case scenerio I would pay ~$145/month for my development environment.

This may sound like a lot of money, but having pre-provisioned resources with all the tools I need to be effective is worth it.

If you’re interested in something a little more standard but still want the benefits of fast internet I can recommend the c3.xlarge instance type (4cores, 7.5GB ram, 2x40GB ephemeral SSDs). At the time of writing the c3.xlarge type runs at ~$0.03/hr on the spot market, which means you end up paying a little over $20/month for it.

The intricacies of spot instances

Spot instances are a little strange in that they can be terminated at any moment, and all of your data will be wiped. This presents two problems:

  1. If my development gets wiped every time spot prices increase, it’s a pain in the ass to reprovision everything when prices come back down. I address this with a few things. First: my spot requests are “persistent” meaning if the instance gets reaped because of price increase, it will come back when the prices return to normal. So that makes sure my CloudTop is up when it is financially possible. The other thing that I do is create an AMI image using Packer.io that contains my standard development environment on the root disk. This includes services like docker, and general tools like vim and tmux. I may publish a tutorial on how to use Packer for this, but the summary is that I just run my dotfiles repository to create the base environment I want. Using persistent spot requests means that whenever my development environment is terminated because spot prices increased, it will bounce back once prices are low enough, and all of my tools and services will be presnet.
  2. If I write code and do not push it back upstream to github, and my VM gets reaped I can potentially lose code. I address this by attaching a persistent EBS volume to the spot instance at /dev/xvdj which I mount to /home/ubuntu/persistent. When I’m working on code that I do not want to lose between cloudtop reboots I place it in this persistent directory. In the spot request you can map the device to a specific EBS volume so it will always come back with your data after a new spot instance has spawned.

The next time I get into the weeds with rebuilding an ami for cloudtops I might publish some of the scripts that I used to provision the cloudtop that I use and also tutorials on how to use packer with dotfiles and stuff. This post is meant to get the point across and the next post will be a technical how-to.

Over the last year I’ve grown quite fond of the idea of spot instances on EC2. The idea that you can spin up a relatively large cluster for almost no money to play around with new technology and tools is amazing.

I’ve been playing with CoreOS and the various cloud_config options for the last few hours, and I was getting sick of having to click through the EC2 console every time I wanted to spin a new cluster based on my new cloud_config. So I made a quick (read hacky/janky) script to spawn CoreOS clusters on EC2 as spot instances.

spawn_coreos_cluster.py
#!/usr/bin/env python

import argparse
import os
import sys
import time

from boto import ec2
from boto.ec2.blockdevicemapping import BlockDeviceMapping, BlockDeviceType



if not os.environ.get('AWS_SECRET_KEY'):
    err = 'No AWS credentials present in the environment, try again...'
    raise SystemExit(err)


INSTANCE_TYPE = 'c3.xlarge'
INSTANCE_BID = '0.05'

COREOS_AMI = 'ami-31222974'

AWS_ACCESS_KEY = os.environ.get('AWS_ACCESS_KEY')
AWS_SECRET_KEY = os.environ.get('AWS_SECRET_KEY')
EC2_KEY_NAME = 'jake'
SECURITY_GROUPS = ['sg-1234']


def parse_args():
    ap = argparse.ArgumentParser(description='Spawn a CoreOS cluster')
    ap.add_argument('-r', '--region', default='us-west-1')
    ap.add_argument('-n', '--node-count',
                    type=int,
                    default=3,
                    help='How many nodes should be in the cluster?')
    args = ap.parse_args()

    return args


def _get_cloudconfig():
    base_path = os.path.dirname(os.path.realpath(__file__))
    cloud_config = open(os.path.join(base_path, 'cloud_config.yml'))
    return cloud_config.read()

def spawn_cluster(count, region):
    conn = ec2.connect_to_region(region,
                                 aws_access_key_id=AWS_ACCESS_KEY,
                                 aws_secret_access_key=AWS_SECRET_KEY)

    mapping = BlockDeviceMapping()
    eph0 = BlockDeviceType(ephemeral_name='ephemeral0')
    eph1 = BlockDeviceType(ephemeral_name='ephemeral1')
    mapping['/dev/xvdb'] = eph0
    mapping['/dev/xvdc'] = eph1

    instance_params = {
        'count': count,
        'key_name': EC2_KEY_NAME,
        'user_data': _get_cloudconfig(),
        'instance_type': INSTANCE_TYPE,
        'block_device_map': mapping,
        'security_group_ids': SECURITY_GROUPS
    }

    spot_reqs = conn.request_spot_instances(INSTANCE_BID, COREOS_AMI, **instance_params)
    for req in spot_reqs:
        req.add_tags({'Name': 'coreos-cluster', 'coreos': True})
    spot_ids = [s.id for s in spot_reqs]

    for x in xrange(50):
        print 'Waiting for instances to spawn...'
        spot_reqs = conn.get_all_spot_instance_requests(request_ids=spot_ids)
        instance_ids = [s.instance_id for s in spot_reqs if s.instance_id != None]
        if len(instance_ids) == len(spot_reqs):
            print 'Instances all spawned'
            print '====================='
            for i in conn.get_only_instances(instance_ids=instance_ids):
                print 'CoreOS Node:'
                print '    - spot req id: %s' % i.spot_instance_request_id
                print '    - instance id: %s' % i.id
                print '    - Public IP: %s' % i.ip_address
                print '    - Public DNS: %s' % i.public_dns_name
            break

        time.sleep(10)


if __name__ == '__main__':
    args = parse_args()
    spawn_cluster(args.node_count, args.region)
cloud_config.yml
#cloud-config
coreos:
  etcd:
    discovery: https://discovery.etcd.io/fancy
    addr: $public_ipv4:4001
    peer-addr: $private_ipv4:7001
  fleet:
      public-ip: $public_ipv4
  units:
    - name: etcd.service
      command: start
    - name: fleet.service
      command: start

    - name: format-ephemeral.service
      command: start
      content: |
        [Unit]
        Description=Stripes the ephemeral instance disks to one btrfs volume
        [Service]
        Type=oneshot
        RemainAfterExit=yes
        ExecStart=/usr/sbin/wipefs -f /dev/xvdb /dev/xvdc
        ExecStart=/usr/sbin/mkfs.btrfs -f -d raid0 /dev/xvdb /dev/xvdc

    - name: var-lib-docker.mount
      command: start
      content: |
        [Unit]
        Description=Mount ephemeral to /var/lib/docker
        Requires=format-ephemeral.service
        After=format-ephemeral.service
        Before=docker.service
        [Mount]
        What=/dev/xvdb
        Where=/var/lib/docker
        Type=btrfs
STDOUT
➜  core  ./spawn_coreos_cluster.py -n 3
Waiting for instances to spawn...
Waiting for instances to spawn...

Instances all spawned
=====================
CoreOS Node:
    - spot req id: sir-03rt1m
    - instance id: i-ead754
    - Public IP: 54.183.220.1
    - Public DNS: ec2-54-183-220-1.us-west-1.compute.amazonaws.com
CoreOS Node:
    - spot req id: sir-03rw5q
    - instance id: i-cfd053
    - Public IP: 54.183.178.2
    - Public DNS: ec2-54-183-178-2.us-west-1.compute.amazonaws.com
CoreOS Node:
    - spot req id: sir-03rwp8
    - instance id: i-45d053
    - Public IP: 54.183.218.3
    - Public DNS: ec2-54-183-218-3.us-west-1.compute.amazonaws.com

I’ve posted everything as a gist here: https://gist.github.com/jakedahn/374e2e54fdcef711bf2a