Time to read: 9 minutes

Recently had a conversation with a few engineers from a large scale tech company. While talking it was brought to my attention that they use Terraform to make AWS less miserable. As someone who as always been miserable using AWS I got really excited.

This is a rough draft and will be edited in the near future. The main ideas are still helpful, until then!

This is meant to be a guide to:

  • why you might consider Terraform
  • how to think about what Terraform does
  • how to get started
  • mistakes I made

Terraform is an infrastructure as code provider. Its website lists “Write, Plan and Create infrastructure as Code” as its main benefit. Let us unpack this a little and think about what this might actually mean.

When I first had this topic explained to me I compared Terraform to a config.yml file. If that is where you started as well, great, that makes me feel a bit better! I am going to make this more concrete by sharing an actual architecture of a project I have in AWS right now, and how I would have done the same using Terraform.

I currently have a project called Yancy, You can go to its basic website here.

Yancy is an API that eliminates spam. You can use it to stop bad actors in chat apps, on blogs or in Shopify stores. To use Yancy you sign up for an account, get an API key and run your comments through the API.

It follows a basic standard I won’t get into much here but rough synopsis is attach your api key and the content and get back a yes its spam or no its not answer.

Yancy was built so I could get comfortable with working on scikit-learn and have a basic understanding of ML models (and their trade offs).

The architecture of yancy is as follows:

  • python app
  • dockerized container
  • deployed on ECS
  • Behind an ELB
  • Uses RDS as its database.

I set the project up that way because that lay out is the one I have the most experience with. I was not trying to innovate on the infrastructure layer and just went with what I know. In doing so, I actually ended up with a few problems I will discuss below.

When I was deploying my resources I initially got my VPC config all sorts of messed up. This led me to spending a day trying to figure out why nothing would connect and at times made me think about pulling my hair out. I learned about VPCs and why they are so important because of that (IN DEPTH) but would have been happy not doing that.

I also felt the pain of how slow getting new things up is. It took me almost as long to write the project as it did to make it publicly available. At times I thought about scrapping it because I was so frustrated with AWS. Once comfortable with Terraform I should never have this same issue again. And that is what mainly had me hooked.

I already do a TON from the command line (generally avoid doing as much as possible with a GUI) and hated having to click around to 100 different places and confirm if this load balancer was set up right and check the health status and confirm the container was alive and blah blah blah.

INSERT OUR SAVIOR, TERRAFORM!

Terraform lets you write (in your favorite code editor) a script that handles everything above for you. The language and syntax is approachable and easy to digest. In my experience you can get up and running in less than an hour (including reading/install/first deployment).

You need to install the Terraform binary. You can do so with this link. Once you do that you will do 2 things. 1-click the binary to unzip it. I was stuck on this for awhile. 2 - put the binary in your $PATH.

I have a mac so I will detail below how I was able to do this.

foo@bar: echo $PATH

At this point I discovered I have a $PATH /usr/local/bin and decided to store my sys link there. You may decide to do differently at your own peril.

foo@bar: ln -s /Downloads/terraform /usr/local/bin

The above worked for me (the binary was in my Downloads folder). You can adapt as you need based on where yours was stored. Once I did this, and restarted my terminal I had the terraform command available.

Once you get the terraform binary installed and linked the command should be available. If it is not, you can click here. All joking aside just tweet me and I will help.

After getting terraform loaded up you will need to run the following command:

foo@bar: terraform init

This should download the other modules you might need but don’t have locally. This is similar to a npm init or pip install -r requirements.txt.

One key point brought up to me recently was the discrepancy between great software engineers and great software engineers who are great at AWS. In the south east that gap is large and there are a few ways you can address it.

Understanding AWS in my case, is an afternoon or two of clicking around and taking diligent notes as I go. In a large software organization this is not possible. If you have 100, 200, 500+ engineers needing to deploy mission critical services 24/7 you should approach the problem very different than I did. And that is where the return on investment in Terraform is even more pronounced.

Once you have terraform init set up, you can get to the meat and potatoes of the product.

To keep our example as simple as possible you should do 2 things.

  • Get your AWS credentials (preferably from an IAM user, not root access)
  • create an example.tf file in a side project repo

Once you have your example.tf file you will want to paste the following in it:

provider "aws" {
  access_key = "YOUR ACCESS KEY HERE"
  secret_key = "YOUR SECRET KEY HERE"
  region     = "us-east-1"
}

resource "aws_instance" "example" {
  ami           = "ami-b374d5a5"
  instance_type = "t2.micro"
}

After you create this, hit save and you should now be able to run terraform plan. If things go as planned you will get the following result.

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  + aws_instance.example
      id:                           <computed>
      ami:                          "ami-b374d5a5"
      arn:                          <computed>
      associate_public_ip_address:  <computed>
      availability_zone:            <computed>
      cpu_core_count:               <computed>
      cpu_threads_per_core:         <computed>
      ebs_block_device.#:           <computed>
      ephemeral_block_device.#:     <computed>
      get_password_data:            "false"
      host_id:                      <computed>
      instance_state:               <computed>
      instance_type:                "t2.micro"
      ipv6_address_count:           <computed>
      ipv6_addresses.#:             <computed>
      key_name:                     <computed>
      network_interface.#:          <computed>
      network_interface_id:         <computed>
      password_data:                <computed>
      placement_group:              <computed>
      primary_network_interface_id: <computed>
      private_dns:                  <computed>
      private_ip:                   <computed>
      public_dns:                   <computed>
      public_ip:                    <computed>
      root_block_device.#:          <computed>
      security_groups.#:            <computed>
      source_dest_check:            "true"
      subnet_id:                    <computed>
      tenancy:                      <computed>
      volume_tags.%:                <computed>
      vpc_security_group_ids.#:     <computed>


Plan: 1 to add, 0 to change, 0 to destroy.

If you got this, congrats, you now have access to create basic resources in AWS for your account. In this simple example all we are doing is provisioning a new EC2 but in the future you will be able to improve this to your hearts desire.

You may now run terraform apply to enact this plan. After that completes, check your AWS account and like magic, you now have new resources provisioned.

Ok, so now that we have roughly accomplished the equivalent to a hello world server lets level up, add a few necessary improvements and keep this blog post interesting.

You should run terraform destroy at any point you’d like to kill the resources you created above. Do this before you send me an email saying my blog caused you to get billed $x for hardware. You have been warned!

As you can tell above we hard coded our credentials. You should not do this. Especially if you are working in a large software organization and/or are checking your code into version control.

The next logical step we will take is using variables as our credentials. This requires 2 updates to the existing code we have.

The first is the example.tf file we created. Update that file to look like this:

provider "aws" {
  access_key = "${var.access_key}"
  secret_key = "${var.secret_key}"
  region     = "${var.region}"
}

resource "aws_instance" "example" {
  ami           = "ami-b374d5a5"
  instance_type = "t2.micro"
}

This file now will look in a variables.tf file to get the information. You currently do not have a variables.tf file, so lets create one together. Create the file in the directory you created your example.tf and paste in the following:

variable "access_key" {}
variable "secret_key" {}
variable "region" {
  default = "us-east-1"
}

This sets variables for the access key and secret key, of which you will paste in on the command line. It also sets a default for where to deploy your resources. In this case into us-east 1. You can change that if you would like.

Hit save and run terraform plan. This should prompt you to put in your credentials and allow to re-deploy the previous instances we deleted above. Small improvement, but we’re working on it.