Terraform Cloud Series – Part 2

Reading Time: 5 minutes

So, Let’s continue from where we have left off at, In this blog, I will focus on the same build process of AWS VPC, but this time code will reside in a git repository (Gitlab for this demos). Assuming the audience is familiar with what is GitLab/GitHub – otherwise, I would recommend understanding the basics of Git before continuing with the rest of the demo.

For Part 2 of this series, I will be creating a new workspace for simplicity purposes.

And I will break the blog into the following areas:

  • Connect to a version control provider
    • Setup/Configure application
    • Generate Tokens
  • Integrate Terraform Cloud with Gitlab
  • Create AWS VPC network using git repo
  • Setup Cloud provider tokens/run time vars
  • Update the code base in git

Connect to a version control provider

Once signed into Terraform Cloud, click create “New workspace” you would be asked to set up the backend repository and cloud provider token:

For the purpose of this lab, I will be using Gitlab to setup my backend.

Note: If you planning on using GitHub or GitLab, one thing is to keep in mind is that each environment lifecycle should be a repo/project. If you combine your code with the root repo, it will be very difficult to manage the stack deployment and organization.

Setup/Configure application

Note: You will need to get properties from both GitLab and terraform cloud, hence I suggest that you open two windows/tabs to work in parallel.

Once you have signed into Gitlab, goto your account sections and applications:

Here we will create a new application that will integrate with Terraform Cloud, I am going to call my application “TerraformCloudisFun

Notice the Redirect URL is garbage value; that is on purpose. We will come back to this and fix it later. go ahead and save the application.

Now, let’s configure the Terraform Cloud section:

  • You should be already on “VCS Providers” section under your organization:
  • If not, you will need to get VCS Provider by clicking –> org –> New Workspace –> VCS provider
  • Again, I am calling my provider “TerraformCloudisFun” to keep the naming consistent.
  • We will need to provide application ID & Secret generated in step above
  • Add VCS provider and application is created.

Integrate Terraform Cloud with Gitlab

Locate the call-back URL and copy the URL – we need to modify with the Gitlab application we created in an earlier step.

  • If you are still on application created page, click the edit button and update callback URL with Terraform Cloud Callback URL:
  • Save & update the application.
  • Now, back to Terraform Cloud and click “connect organization
  • Terraform will try to access GitLab.com and authorize the application.

That’s it – Backend is configured and ready to be used.

Create AWS VPC network using git repo

Now that we have our backend ready, let us try to create the AWS VPC by pulling the code directly from version control.

Terraform application we created will fetch the repos/projects from Gitlab.com:

Select your repository or working project for provisioning and create workspace:

You might have to wait a bit before the workspace is ready to be configured.

Hit the configure button and provide the required properties for the cloud provider:

Setup Cloud provider tokens/run time vars

I will add my AWS IAM user Access Key and Secret which is needed to create the stack in AWS.

  1. AWS Access Key
  2. AWS Access Secret
  3. Additional tag values

Notice that TF Cloud allows you to encrypt the secret values, but this information may appear in TF outputs/debug logs.

  • Select the “Sensitive” checkbox & save the variables.

Now we are ready to create the stack using TF Cloud.

Hit the “Queue Plan” button and stack creation will generate plan and if there any errors, it will stop:

If all looks good, TF Cloud will ask the user to verify and apply the changes:

Apply the changes and provide comments.

While it is creating the stack, you can look at the raw logs:

If everything goes as planned, job will change the status to success:

Update the code base in git

For the final piece, I will update one of the subnet CIDR range in TF code block from 10.10.104.0/24 to 10.10.105.0/24 – Push the changes to Gitlab.

From:

module "vpc" {
  source = "terraform-aws-modules/vpc/aws"
  version = "2.29.0"

  # insert the 12 required variables here
  name = "poc-vpc-${var.prefix}"
  cidr = "10.10.0.0/16"
   azs             = ["us-east-1a", "us-east-1b", "us-east-1c"]
  private_subnets = ["10.10.1.0/24"]
  public_subnets  = ["10.10.104.0/24"]

  enable_nat_gateway = true
  enable_vpn_gateway = true
  enable_s3_endpoint = true

  tags = var.default_tags
}

To:

module "vpc" {
  source = "terraform-aws-modules/vpc/aws"
  version = "2.29.0"

  # insert the 12 required variables here
  name = "poc-vpc-${var.prefix}"
  cidr = "10.10.0.0/16"
   azs             = ["us-east-1a", "us-east-1b", "us-east-1c"]
  private_subnets = ["10.10.1.0/24"]
  public_subnets  = ["10.10.105.0/24"]

  enable_nat_gateway = true
  enable_vpn_gateway = true
  enable_s3_endpoint = true

  tags = var.default_tags
}


Terraform detected the changes from backend and generated the new infra plan:

Changes are detected as we can see the subnet will be re-created.

It is obvious that pushing this change impacts the network, I have the ability to discard the run with the provided comment:

In the next series, we will discuss organizations of the projects/workspace, state file, and advance features.

Hope this post helped to get you started with Terraform Cloud.

Terraform Cloud Series – Part 1

Reading Time: 4 minutes

What is a terraform cloud? Terraform cloud is a managed platform for teams/enterprises to create a TF stack via a managed platform. With Terraform cloud, tfstate file & stack state is stored in the Terrafrom Cloud platform. I won’t go into how the application works, but you can read up on it at the link below:

https://www.terraform.io/docs/cloud/index.html

TF Cloud is currently free for now, so go sign-up and start hands-on with it. Once you have created your account, you will see the following:

In order to create cloud resources, you need to create a workspace. Think a workspace as where you will add/modify/change cloud resources i.e. vpc, subnet, compute, etc, etc. For part 1, we will start small and work our way up to a bit more complex setup.

To create a working workspace, we will need to have the following tools:

  • TF Cloud workspace
  • TF with a remote backend
  • Backend repo control provider
  • Stack management & lifecycle management with Terraform Cloud
  • Terraform code for building stack

For this part 1 of the series, I will limit it to creating TF workspace and setup my terraform templates with remote backend. Also, If you need sample terraform templates you can get it from my git repo here.

Creating Workspace

Once signed into Terraform Cloud, click create “New workspace” you would be asked to set up the backend repository or use “no VCS connection”:

For the purpose of this lab, I will be using no VCS connection to setup my backend. Before you begin, stage local directory and download sample code from here.

TF Cloud backend

In order to use TF cloud, you need to create a remote backend. Let’s create a new file called backend.tf in a location where we have staged terraform code and copy-paste contents from below and update organization and workspace name:

terraform {
  backend "remote" {
    organization = "example-demo-org"

    workspaces {
      name = "example-demo-org-sandbox"
    }
  }
}

Note: You will need to update terraform to => .12.19 version to work with TF login.

Verify all information for backend.tf and cloud provider access key & token has been updated. After that execute the following command:

terraform login 

The above command should ask you to generate a token or if you already have token created, you can provide the token.

If everything goes as planned, you can execute the following commands next:

terrafrom init
terraform plan

If no errors are indicated, Terraform should spit out a plan with TF backend:

Wait! shouldn’t I see something in my workspace? no, not yet! with remote backend, only tfstate file & plan is stored on Terraform cloud. As soon as you apply the changes, you see the queued plan created for the stack asking for conformation.

Also, You can see my AWS account and my custom VPC is not created yet.

After confirming TF Plan – accept the changes and let’s see what happens.

Status changes from confirmation to applying.

Explore the TF Cloud stack, notice we can see the output as it is captured during the execution:

Congratulations! You have successfully created a stack using Terraform Cloud and stack statefile is managed by Terraform Cloud. In the next series, I will show you how to use Gitlab or Github with a remote repository.

If you have questions or stuck somewhere in this tutorial, please contact me or leave a comment.

AWS Pricing Calculator **NEW

Reading Time: 3 minutes

Recently I had a need to create a quote for AWS infrastructure and I noticed that AWS is switching from “AWS simple calculator” to “AWS pricing calculator” – So, let’s give it a try.

The process is pretty straight forward, you punch in some input and AWS generates TCO for AWS kit. It is a bit of a learning curve to get around, but not bad.

https://calculator.aws/#/addService


Once you click the URL, you will start with a blank pricing sheet which will allow the user to add by service and you simply input your requirements.

For instance, let’s say we need to provision 10 ec2 instances, simply click configure and add your inputs.

There are two methods:

  • Quick estimate
  • Advance estimate

For this demo, I am sticking with a quick estimate!

Check out the nice feature where I just plug in my numbers for “vCPUs” and “Memory” and AWS automatically suggested that I should use “r5a.8xlarge” – this is pretty nice since I don’t have to scramble with figuring out what shape I need to select for my use case.

Next, I need to define how many ec2 instances I need to add.


Great what about the pricing model, not to worry! The new pricing calculator allows us to select the model for pricing:

Another example with “Standard Reserved Instances”:

Next, we can add storage for EBS block volume:

Finally, we add the ec2 estimate to the overall pricing estimate and continue to work with additional resources.

Give it a try! it is free!

Attach is an example exported output from Pricing calculator:

Terraform registry AWS module

Reading Time: 2 minutes

I am starting to transition from TF .11 to TF .12, recently I started to work on AWS terraform registry module and start to run into the following issue.

module "vpc" {
  source = "terraform-aws-modules/vpc/aws"
  version = "2.29.0"

  # insert the 12 required variables here
  name = "my-vpc"
  cidr = "10.0.0.0/16"
   azs             = ["us-east-1a", "us-east-1b", "us-east-1c"]
  private_subnets = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
  public_subnets  = ["10.0.101.0/24", "10.0.102.0/24", "10.0.103.0/24"]

  enable_nat_gateway = true
  enable_vpn_gateway = true

  tags = {
    Terraform = "true"
    Environment = "dev"
  }
}

(dev-tools) ➜  sandbox-vpc  terraform plan       

Error: Call to unknown function

  on .terraform/modules/vpc/main.tf line 288, in resource "aws_subnet" "public":
 288:   availability_zone               = length(regexall("^[a-z]{2}-", element(var.azs, count.index))) > 0 ? element(var.azs, count.index) : null

There is no function named "regexall".


Error: Call to unknown function

  on .terraform/modules/vpc/main.tf line 289, in resource "aws_subnet" "public":
 289:   availability_zone_id            = length(regexall("^[a-z]{2}-", element(var.azs, count.index))) == 0 ? element(var.azs, count.index) : null

There is no function named "regexall".


Error: Call to unknown function

  on .terraform/modules/vpc/main.tf line 316, in resource "aws_subnet" "private":
 316:   availability_zone               = length(regexall("^[a-z]{2}-", element(var.azs, count.index))) > 0 ? element(var.azs, count.index) : null

There is no function named "regexall".


Error: Call to unknown function

  on .terraform/modules/vpc/main.tf line 317, in resource "aws_subnet" "private":
 317:   availability_zone_id            = length(regexall("^[a-z]{2}-", element(var.azs, count.index))) == 0 ? element(var.azs, count.index) : null

There is no function named "regexall".


Error: Call to unknown function

  on .terraform/modules/vpc/main.tf line 343, in resource "aws_subnet" "database":
 343:   availability_zone               = length(regexall("^[a-z]{2}-", element(var.azs, count.index))) > 0 ? element(var.azs, count.index) : null

There is no function named "regexall".

Turns out this is an issue with TF version .12.06

(dev-tools) ➜  sandbox-vpc  terraform --version
Terraform v0.12.6
+ provider.aws v2.55.0

Fix

Upgrade TF to .12.24

(dev-tools) ➜  sandbox-vpc  brew upgrade terraform
Updating Homebrew...
==> Auto-updated Homebrew!
Updated Homebrew from 8d3aa49ae to c1708ff6b.
Updated 2 taps (homebrew/core and homebrew/cask).
==> Updated Formulae
openssl@1.1 ✔
==> Updated Casks
loginputmac                                 openttd                                     wacom-inkspace

==> Upgrading 1 outdated package:
terraform 0.12.6 -> 0.12.24
==> Upgrading terraform 0.12.6 -> 0.12.24 
==> Downloading https://homebrew.bintray.com/bottles/terraform-0.12.24.mojave.bottle.tar.gz
==> Downloading from https://akamai.bintray.com/2a/2a21a77589673b2064c9fa7587a79a0375d69a8e02f824e5dc22dc960bf2d78b?__gda__=exp=1585
######################################################################## 100.0%
==> Pouring terraform-0.12.24.mojave.bottle.tar.gz
🍺  /usr/local/Cellar/terraform/0.12.24: 6 files, 51.2MB
==> `brew cleanup` has not been run in 30 days, running now...
/usr/local/share/ghostscript/9.19/Resource/CIDFSubst/ipaexg.ttf
(dev-tools) ➜  sandbox-vpc  terraform --version   
Terraform v0.12.24
+ provider.aws v2.55.0
(dev-tools) ➜  sandbox-vpc  terraform plan        
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.


------------------------------------------------------------------------

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # module.vpc.aws_eip.nat[0] will be created
  + resource "aws_eip" "nat" {
      + allocation_id     = (known after apply)
      + association_id    = (known after apply)
      + domain            = (known after apply)
      + id                = (known after apply)
      + instance          = (known after apply)
      + network_interface = (known after apply)
      + private_dns       = (known after apply)
      + private_ip        = (known after apply)
      + public_dns        = (known after apply)
      + public_ip         = (known after apply)
      + public_ipv4_pool  = (known after apply)
      + tags              = {
          + "Environment" = "dev"
          + "Name"        = "my-vpc-us-east-1a"
          + "Terraform"   = "true"
        }
      + vpc               = true
    }