Polyglot Cloud Blog

Terraform and Gitlab make an excellent combination to develop and deploy immutable infrastructure as code. This post explains an opinionated DevOps workflow and provides a CI/CD pipeline template using Gitlab and Terraform to deploy multiple cloud environments.

Introduction

If you have not read my previous post about Gitflow and working with multiple AWS accounts, please take look as this blog post builds on the concepts explained there.

Why Gitlab?

In case you don’t know Gitlab or don’t have an account yet, check it out here. The hosted Gitlab SaaS is free to use and offers a collection of seamlessly integrated services which enable a highly productive DevOps workflow. Relevant for this post are:

• Git Repository
  including Merge Requests for code reviews and approvals
• CI/CD Pipeline
  with hosted or private job runners

Gitlab has proven invaluable for my professional work maintaining repos in Kotlin, Python, Go, Typescript, Terraform and many more.

Pipeline #1: Local File (simple, stateless)

For sake of simplicity, we’ll use the local_file Terraform resource from the Local Provider to create a file containing the name of the environment we’re running in.

Code

main.tf in the root directory contains the following:

// main.tf
resource "local_file" "foo" {
    content  = var.environment
    filename = "/tmp/bar"
}

variable "environment" {}

We keep the infrastructure code DRY and create configuration files in a the subdirectory config/; one file per environment:

// config/production.tfvars
environment = "production"

// config/staging.tfvars
environment = "staging"

// config/sandbox.tfvars
environment = "sandbox"

And finally the Gitlab CI/CD pipeline .gitlab-ci.yml in the root directory:

# .gitlab-ci.yml
stages:
  - infrastructure

.terraform:
  stage: infrastructure
  image:
    name: hashicorp/terraform:light
    entrypoint:
      - '/usr/bin/env'
      - 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'    
  before_script:
    - terraform init

# ~~~~~~~~~~~~~~~~ Apply ~~~~~~~~~~~~~~~~
.terraform_apply:
  extends: .terraform
  script:
    - terraform apply -auto-approve -var-file=config/${ENVIRONMENT}.tfvars

terraform_apply_production:
  extends: .terraform_apply
  variables:
    ENVIRONMENT: production
  only:
    refs:
      - master
        
terraform_apply_staging:
  extends: .terraform_apply
  variables:
    ENVIRONMENT: staging
  only:
    refs:
      - develop
        
terraform_apply_sandbox:
  extends: .terraform_apply
  variables:
    ENVIRONMENT: sandbox
  except:
    refs:
      - master
      - develop

Explanation

This pipeline defines an abstract (= not executed) job .terraform from which all other jobs extend. The before_script executes the compulsory terraform init to download providers and initialize the state.

.terraform_apply defines another abstract job, from which all environment specific jobs inherit and set the config file to use via gitlab variables.

The Gitlab strategies defined with only and except constrain the Git branches, for which the specific job is executed.

Following the Gitflow workflow of merging branches, the assigned environment is now automatically deployed.

Pipeline #2: AWS Docker Registry (advanced)

As next step, we create one AWS Docker registry (ECR) per environment in a single AWS account using S3 + DynamoDB to store and lock the Terraform backend state.

Preparation - Terraform state

If you don’t have an S3 bucket and/or DynamoDB table yet, visit this article or use the following Terraform snippet with randomized bucket and table names:

resource "aws_s3_bucket" "terraform_state" {
    bucket = "terraform-state-8ab67ec2"
 
    versioning {
      enabled = true
    }
 
    lifecycle {
      prevent_destroy = true
    }    
}

resource "aws_dynamodb_table" "terraform_lock" {
  name = "terraform-state-8ab67ec2-lock"
  hash_key = "LockID"
  read_capacity = 5
  write_capacity = 5
 
  attribute {
    name = "LockID"
    type = "S"
  }
}

Preparation - Access key

The Terraform AWS provider requires AWS access keys to modify AWS infrastructure on behalf of the user the access keys belong to.

Either create access keys for an existing user or create a dedicated terraform IAM user. On later option, select Programmatic access on creation, under Permissions select Attach existing policies directly and choose a policy which has the rights to create the infrastructure you intend to maintain. The policy AdministratorAccess has full access to all resources.

Preparation - Environment Variables

The Terraform AWS provider can be configured via HCL arguments or environment variables. To avoid storing credentials in the Git repository, we set the following pipeline environment variables for the gitlab-runner:

• AWS_DEFAULT_REGION
• AWS_ACCESS_KEY_ID
• AWS_SECRET_ACCESS_KEY

These variables can be set in the Gitlab UI at: Settings -> CI / CD -> Variables

Code

main.tf in the root directory contains the following:

// main.tf
provider "aws" { }

resource "aws_ecr_repository" "nginx" {
  name = "${var.environment}-nginx"
}

To comply with Terraform best-practices, we create variables.tf in the root directory:

// variables.tf
variable "environment" {}

The Terraform backend state is kept in a separate HCL file in the root directory:

// backend.tf
terraform {
  backend "s3" {
    encrypt = true
  }
}

A config per environment in the sub-directory config/:

// config/production.tfvars
environment = "production"

// config/staging.tfvars
environment = "staging"

// config/sandbox.tfvars
environment = "sandbox"

A backend config per environment in the sub-directory config/:

// config/production_backend.tfvars
region         = "us-east-1"
bucket         = "terraform-state-8ab67ec2"
dynamodb_table = "terraform-state-8ab67ec2-lock"
key            = "production/nginx.state"

// config/staging_backend.tfvars
region         = "us-east-1"
bucket         = "terraform-state-8ab67ec2"
dynamodb_table = "terraform-state-8ab67ec2-lock"
key            = "staging/nginx.state"

// config/sandbox_backend.tfvars
region         = "us-east-1"
bucket         = "terraform-state-8ab67ec2"
dynamodb_table = "terraform-state-8ab67ec2-lock"
key            = "sandbox/nginx.state"

And finally the Gitlab CI/CD pipeline .gitlab-ci.yml in the root directory:

# .gitlab-ci.yml
stages:
  - infrastructure

.terraform:
  stage: infrastructure
  image:
    name: hashicorp/terraform:light
    entrypoint:
      - '/usr/bin/env'
      - 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'    
  before_script:
    - terraform init -backend-config=config/${ENVIRONMENT}_backend.tfvars

# ~~~~~~~~~~~~~~~~ Apply ~~~~~~~~~~~~~~~~
.terraform_apply:
  extends: .terraform
  script:
    - terraform apply -auto-approve -var-file=config/${ENVIRONMENT}.tfvars

terraform_apply_production:
  extends: .terraform_apply
  variables:
    ENVIRONMENT: production
  only:
    refs:
      - master
        
terraform_apply_staging:
  extends: .terraform_apply
  variables:
    ENVIRONMENT: staging
  only:
    refs:
      - develop
        
terraform_apply_sandbox:
  extends: .terraform_apply
  variables:
    ENVIRONMENT: sandbox
  except:
    refs:
      - master
      - develop

# ~~~~~~~~~~~~~~~~ Destroy ~~~~~~~~~~~~~~~~
terraform_destroy:
  extends: .terraform
  script:
    - terraform destroy -auto-approve -var-file=config/${ENVIRONMENT}.tfvars
  only:
    variables:
      - $ENVIRONMENT
      - $DESTROY

Explanation

The command terraform init is executed with the backend config for the environment.

An additional job terraform_destroy allows removing the infrastructure created by Terraform. To enable this job, the pipeline variable ENVIRONMENT must be set to the environment name and DESTROY to true, manually via Gitlab UI at: CI / CD -> Run Pipeline

Pipeline #3: AWS multiple accounts (final)

The previous pipeline deploys all Docker registries in the same AWS account, which is feasible for small setups. Larger cloud architectures should contain multiple environments in dedicated AWS accounts. Please refer to my previous post for more details.

So for the third and final pipeline, we create one AWS Docker registry (ECR) per environment, each in a separate AWS sub-account.

Preparation - AWS accounts

If you don’t have sub-accounts for your workloads environments yet, create them using the AWS Organization in the AWS console or deploy the following Terraform snippet in your AWS master account:

resource "aws_organizations_account" "production" {
  name  = "production"
  email = "production+admin@foobar.io"
}

resource "aws_organizations_account" "staging" {
  name  = "staging"
  email = "staging+admin@foobar.io"
}

resource "aws_organizations_account" "sandbox" {
  name  = "sandobx"
  email = "sandbox+admin@foobar.io"
}

Make sure the AWS user (to which the access keys belong) has the rights to assume the role OrganizationAccountAccessRole in the target account. This is given if the policy AdministratorAccess was attached. The AWS docs provide more information how to access sub-accounts.

Code

main.tf in the root directory contains the following:

// main.tf
provider "aws" { 
  assume_role {
    role_arn = "arn:aws:iam::${var.account}:role/OrganizationAccountAccessRole"
  }
}

resource "aws_ecr_repository" "nginx_hello" {
  name = "${var.environment}-nginx-hello"
}

Similar to Pipeline #2, we create variables.tf in the root directory:

// variables.tf
variable "environment" {}
variable "account" {}

A config per environment with the new account variable in the sub-directory config/:

// config/production.tfvars
environment = "production"
account     = "123456789"

// config/staging.tfvars
environment = "staging"
account     = "987654321"

// config/sandbox.tfvars
environment = "sandbox"
account     = "313374985"

The backend config in the sub-directory config/and Gitlab CI/CD pipeline .gitlab-ci.yml in the root directory are identical to the ones in Pipeline #2.

Explanation

The assume_role attribute of the AWS provider takes a role_arn which is assumed on access to your AWS account. We assemble the role_arn with account id provided via config variable ${var.account} and the role OrganizationAccountAccessRole which is created by default in all sub-accounts (but missing in the master account).

Setting role_arn effects in switching to the target account before creating the AWS resources.

The result of running this pipeline are three ECR registries, one in each sub-account production, staging and sandbox.

Optimization: Deploy production manually

The Gitlab pipeline explained above follows the continuous deployment principle and automatically deploys infrastructure changes to production on merge to the master branch. But for some use-cases or setups, an additional verification step is desired.

The following .gitlab-ci.yml jobs execute terraform plan for production and turn terraform apply into a manual job:

terraform_plan_production:
  extends: .terraform
  variables:
    ENVIRONMENT: production
  script:
    - terraform plan -var-file=config/${ENVIRONMENT}.tfvars
  except:
    variables:
      - $DESTROY
  only:
    refs:
      - master


terraform_apply_production:
  extends: .terraform_apply
  when: manual
  variables:
    ENVIRONMENT: production
  only:
    refs:
      - master

Conclusion

Applying the Gitflow concept to Terraform on Gitlab is a powerful and highly productive workflow to enable CI/CD for instrastructure-as-code and deploy immutable cloud infrastructure.

FAQ

1. Why are the steps terraform validate and terraform plan skipped?
   validate and plan are executed as part of apply.

2. There is a lot of redundant code in the pipeline. Why not use YAML anchors?
   YAML anchor decrease readability and make it harder for other engineers to follow the workflow.