On a recent project, my team identified the need to transition from having a directory in our main DevOps repository that contains all of our Terraform modules to having a discrete repository for each module.
The benefits of a multi-repo arrangement are:
- Modules can be individually versioned
- Modules can be easily modified without affecting existing infrastructure
- Organization of modules will be easier
- Development of modules can be done without pull requests on the main DevOps repository
I decided to make a template repository to facilitate this process and perform the following automated actions:
- Lint/Format the module using
terraform fmt - Validate the module syntax using
terraform validate - Auto-generate documentation for the module
- Perform a GitHub release for each meaningful merge to the default branch
We’re using CircleCI for these repositories, so this will be specific to that. However, you should be able to adapt these CI actions to any CI platform. In CircleCI, we configure our workflows and jobs in a .circleci/config.yml file. The specifics of this file are documented here. We first configure our executors, the Docker containers that will run the various parts of our workflow. I’ve chosen the CircleCI convenience images for python, node, and golang for the various tasks I want to execute.
version: 2.1
executors:
docker-terraform:
docker:
- image: circleci/python:latest
node:
docker:
- image: circleci/node:latest
doc:
docker:
- image: circleci/golang:latest
Next, we configure our job definitions. First, the job that will test our module in CI for proper format and syntax. If there are errors in Terraform syntax or format, the pull request author will be notified, and the pull request will not be able to be merged into the codebase. For this process, we need to ensure that we’re using the proper version of Terraform, so we use tfenv to control the version. This requires us to define the version of Terraform we’re writing our module for in the main.tf. We also define our provider and default region.
terraform {
required_version = "0.12.29"
}
provider "aws" {
version = "~>2.70.0"
region = "us-east-1"
}
Testing
The test job definition in CircleCI checks out the module, and then installs tfenv, which we’ll use to install the minimum required version of Terraform as defined in our module, and then we can use the built-in commands to format and validate the module.
jobs:
test:
executor: docker-terraform
steps:
- checkout
- setup_remote_docker:
docker_layer_caching: true
- run:
name: Install tfenv
command: |
git clone https://github.com/tfutils/tfenv.git /tmp/.tfenv
chmod -R +rx /tmp/.tfenv/bin
- run:
name: Install terraform
command: /tmp/.tfenv/bin/tfenv install min-required
- run:
name: Set terraform version
command: /tmp/.tfenv/bin/tfenv use min-required
- run:
name: Run terraform fmt
command: /tmp/.tfenv/bin/terraform fmt
- run:
name: Run terraform init
command: /tmp/.tfenv/bin/terraform init
- run:
name: Run Terraform validate
command: /tmp/.tfenv/bin/terraform validate
If any of these steps fails, the test job will fail and GitHub and CircleCI will notify us of the failure.
Releasing
In order to take as much manual work out of the module development as we can, we want to automate the creation of GitHub releases. Rather than a naive approach, where we increment the module’s version number by .1 on every merge to the default branch, we want to take a more nuanced approach and use Semantic Release.
Semantic Release will analyze the commit messages that we’re merging to master. Using the Angular Commit Message Guidelines, it will determine if the release should be a bugfix, minor, major, or breaking release, write a release note, and increment the version number of the module. This means that the developer needs to remember to include at least one commit message that follows the format, but also means that merges to the default branch will not always trigger a new release, but only when a new feature, bugfix, or breaking change is added. If developers have trouble remembering to use the proper commit message format, you can add a git commit hook to remind them to do so, but hopefully this won’t be necessary!
To use Semantic Release, the simplest way is to use NodeJS’s npx tool. Adding this step to the pipeline is simple. In .circleci/config.yml we add:
release:
executor: node
steps:
- checkout
- run: npx semantic-release
And we also need to add a configuration file for semantic-release, .releaserc:
branch: main
branches: ["main"]
plugins:
[
"@semantic-release/commit-analyzer",
"@semantic-release/release-notes-generator",
"@semantic-release/github",
]
This instructs Semantic Release to analyze the commits, generate release notes, and make a new release on GitHub. Semantic Release has other plugins that run by default but we need to make sure they don’t run.
In addition, you’ll need to provide an environment variable for your project in CircleCI called GH_TOKEN or GITHUB_TOKEN that contains a token with write privileges to your repository.
Documentation
Writing documentation for Terraform modules can be difficult, and many modules don’t have documentation for that reason. We want to generate documentation for our Terraform module automatically, so that we can be assured that the model is easy to use for future teams. Terraform-Docs is the only tool I am aware of that performs this task. Here’s how you add the Terraform-Docs configuration to .circleci/config.yml:
documentation:
executor: doc
steps:
- checkout
- setup_remote_docker:
docker_layer_caching: true
- run:
name: Install terraform-docs
command: GO111MODULE="on" go get github.com/terraform-docs/terraform-docs@v0.10.0-rc.1
- run:
name: Configure github
command: |
git config user.email "your_automation_user@your_company.com"
git config user.name "automation_user"
- run:
name: Run terraform docs
command: terraform-docs -c .terraform-docs.yml . > README_MODULE.md
- run:
name: Check in documentation
command: |
git add README_MODULE.md
git commit -m "[skip ci] Add auto-gen documentation."
git push https://automation_user:${GH_TOKEN}@github.com/${CIRCLE_PROJECT_USERNAME}/${CIRCLE_PROJECT_REPONAME}.git
And we place a configuration file for terraform-docs; .terraform-docs.yml:
formatter: markdown table
header-from: main.tf
settings:
indent: 4
It’s important to note that you should skip the CI run for this commit/push, so you don’t create a loop. You must also use the same GH_TOKEN or GITHUB_TOKEN environment variable you used for the release authentication here.
The pieces are almost complete! Now, we just need a workflow definition for CircleCI, and we can start automatically testing and releasing our Terraform modules.
workflows:
version: 2
test-ci:
jobs:
- test:
filters:
branches:
ignore: main
release:
jobs:
- test:
filters:
branches:
only: main
- release:
requires:
- test
filters:
branches:
only: main
- documentation:
filters:
branches:
only: main
Here, we’ve configured CircleCI to run tests on every push of a branch not named main, our default branch, and then tests and documentation on every merge to the main branch, and on a successful test run, to run a release. Even though we should not be merging something that did not pass tests in a developer’s branch, it’s still possible for a branch with an error to get merged to the default branch, so we want to ensure we don’t release in the event that the tests don’t pass.
That’s it! Now each time code is committed to the repository, you can enjoy using a syntactically correct, properly formatted module and auto-generated documentation. Releases will be automated, and not sentimental or romantic. Your team can spend the time they would have used writing this documentation, making manual changes to code formats (indenting, outdenting, etc), and authoring releases on other tasks.
