Use custom Terraform provider builds in your pipeline

I work a lot with Terraform, due to my job especially with the AzureRM provider. The AzureRM provider is used to deploy and configure nearly all available resources in Microsoft Azure. The team is doing an amazing job and is releasing an updated version by the end of nearly every week.

The regular release cadence keeps the time window short it takes till new features, bugfixes and other contributions are available in the official releases.

But, nevertheless there are cases where you might want to use your own, custom build of a given provider to test features before they make it into the official release. In my case this happens a lot when developing new features for the AzureRM provider.

I usually develop in separate branches in my own fork of the terraform-provider-azurerm repository and test my changes locally on my machine. This works well in most cases, but especially for larger changes, completly new resources with tens or even hundreds of arguments, this does not scale well and additional help might be required.

Asking others to clone my fork, build the provider locally and test my changes does also not scale well. The overhead of explaining all the details and setting up the environment is just too high.

A more suitable option we have recently tested is using custom builds of the AzureRM provider in one of our projects by embedding them into our build processes.

Install golang and other tools in your build agent

Before we can start using custom builds of the AzureRM provider, we need to install some prerequisites. This includes the right version of golang, the language the AzureRM provider is written in. But which version? And how?

The AzureRM provider repo contains a file called .go-version that contains the version currently used to build the provider.

Content of the .go-version file in the AzureRM repository

This is the most important prerequisite we need in our build agent to build the provider binaries. How do we install these prerequisites? This is usually a bit challenging, most Linux distros already contain a, usually outdated, version of golang. You can either install it via apt, yum or other mechanisms, making sure that the PATH variable is properly set etc.

Another handy way for Azure DevOps is using the GoTool task:

GoTool@0 Task in Azure DevOps to install golang in a specific version

Clone source code and build provider binaries

Assuming that we have all prerequisites in place, we can now proceed with getting our source code into the build agent. I’m therefore now cloning my fork via git clone:

# define custom terraform azurerm provider repo and branch
SOURCE_REPO=https://github.com/heoelri/terraform-provider-azurerm.git
SOURCE_BRANCH=feature/newafdsku
# making sure that GOPATH is set
export GOPATH=~/go
# create and enter the required directories
mkdir -p $GOPATH/src/github.com/hashicorp
cd $GOPATH/src/github.com/hashicorp
# git clone the source repository
git clone $SOURCE_REPO

Output of git clone

After we have successfully cloned our source repository, in my case my fork of the terraform-provider-azurerm repository, we have to checkout the right branch that contains our changes.

cd $GOPATH/src/github.com/hashicorp/terraform-provider-azurerm
git checkout $SOURCE_BRANCH

Output of git checkout

In the right directory we can now build our source code:

make build

That is it. This will now build new binaries based on the selected repository and branch. These binaries will land in $GOPATH/bin. This is how it looks like in my Azure DevOps pipeline:

Output of make build in Azure DevOps Pipeline

Use custom, local binaries with terraform

To let Terraform know that we want to use our custom build instead of the official release, we can use dev_overrides. These overrides can be set for example in the ~/.terraformrc file.

Here is an example from my local machine how these overrides look like:

Content of the ~/.terraformrc file containing a dev_override for hashicorp/azurerm

These overrides can easily be set via some bash magic:

echo "provider_installation {
dev_overrides {
\"hashicorp/azurerm\" = \"$GOPATH/bin\"
}
direct {}
}" > ~/.terraformrc

From now on, Terraform will let you know that overrides are in place everytime you use the terraform cli:

dev_overrides warning when using terraform cli

To revert back to the official release, you can either update the ~/.terraformrc file or use terraform init -upgrade.