Posted by on September 19, 2012

Opscode Chef is a powerful tool for automating the configuration of new servers, and indeed, entire clouds, multi-server architectures etc. We now use it to deploy MyTardis. It’s quite daunting at first, so here’s a quick guide to the setup I use. You’ll probably need to refer to more complete tutorials to cover gaps (eg  OpsCode’s Fast Start Guide, the MyTardis chef documentation and this random other one.) We’ll use installing MyTardis as the goal.

1. Sign up to Hosted Chef. That gives you a place where Chef cookbooks, environments, roles etc will live.
2. Install Chef (client) locally.
3. Get the MyTardis Chef cookbook:

   cd ~/mytardis
   git clone https://github.com/mytardis/mytardis-chef

4. Make a directory, say ~/chef, and download the “…-validator.pem” and knife.rb files that Opscode gives you to there. The knife.rb file contains settings for Knife, like directories. Modify it so it points to ~/mytardis/mytardis-chef. Mine looks like this

   current_dir = File.dirname(__FILE__)
   log_level :info
   log_location STDOUT
   node_name "stevage"
   client_key "#{current_dir}/stevage.pem"
   validation_client_name "versi-validator"
   validation_key "#{current_dir}/versi-validator.pem"
   chef_server_url "https://api.opscode.com/organizations/versi"
   cache_type 'BasicFile'
   cache_options( :path => "#{ENV['HOME']}/.chef/checksums" )
   cookbook_path ["#{current_dir}/../mytardis/mytardis-chef/site-cookbooks", "#{current_dir}/../mytardis/mytardis-chef/cookbooks"]

5. You now need to upload these cookbooks and assorted stuff to Chef Server. This part of Chef is really dumb. (Why isn’t there a single command to do this – or why can’t knife automatically synchronise either a local directory tree or git repo with the Chef server. Why is one command ‘upload’ and another ‘from file’…? Why are cookbooks looked for in the ‘cookbooks path’ but roles are sought under ./roles/?)

   knife cookbook upload -a
   knife role from file ../mytardis/mytardis-chef/roles/mytardis.json

6. Get a NeCTAR Research Cloud VM. Launch an instance of “Ubuntu 12.04 LTS (Precise) amd64 UEC”. Call it “mytardisdemo”, give it your NeCTAR public key, and do what you have to (security groups…) to open port 80.
7. Download your NeCTAR private key to your ~/chef directory.
8. Now, to make your life a bit easier, here are three scripts that I’ve made to simplify working with Knife:bootstrap.sh:

   #!/bin/bash -x
   # Bootstraps Chef on the remote host, then runs its runlist.
   source ./settings$1.sh
   knife bootstrap $CHEF_IP -x $CHEF_ACCOUNT --no-host-key-verify -i nectar.pem --sudo $CHEF_BOOTSTRAP_ARGS

   update.sh:

   #!/bin/bash -x
   # Runs Chef-client on the remote host, to run the latest version of any applicable cookbooks.
   source settingsi$1.sh
   knife ssh name:$NECTAR_HOST -a ipaddress -x $NECTAR_ACCOUNT -i nectar.pem -o "VerifyHostKeyDNS no" "sudo chef-client"

   connect.sh:

   #!/bin/bash -x
   # Opens an SSH session to the remote host.
   source ./settings$1.sh
   ssh $CHEF_ACCOUNT@$CHEF_IP -i nectar.pem -o "VerifyHostKeyDNS no"

   (We disable host key checking because otherwise SSH baulks every time you allocate a new VM with an IP that it’s seen before.)With these, you can have several “settings…sh” files, each one describing a different remote host. (Or just a single settings.sh). So, create a “settings-mytardisdemo.sh” file, like this:

   # Settings for Ubuntu Precise NeCTAR VM running mytardis as a demo
   export CHEF_HOST=mytardisdemo
   export CHEF_IP=115.146.94.53
   export CHEF_ACCOUNT=ubuntu
   export CHEF_BOOTSTRAP_ARGS="-d ubuntu12.04-gems -r 'role[mytardis]'"

9. Now you have it all in place: A chef server with the cookbook you’re going to run, a server to run it on, and the local tools to do it with. So, bootstrap your instance:

   ./bootstrap.sh -mytardisdemo

   If all goes well (it won’t), that will churn away for 40 minutes or so, then MyTardis will be up and running.

10. If you change a cookbook or something (don’t forget to re-upload) and want to run chef again:

    ./update.sh -mytardisdemo

11. And to SSH into your box:

    ./connect.sh -mytardisdemo

System administration chef, cloud, configuration management, cookbook, deployment, knife, mytardis, nectar, nectar research cloud, tutorial, virtual machine