The advent of DevOps helped minimise the dependence on sysadmins who used to set up infrastructure manually, while seated at some unknown corner of the office. Managing servers and services manually is not a very complicated task in a data centre. But when we move to the cloud, scale and start working with many resources from multiple providers (AWS, GCP, Azure, etc), manually setting up and configuring to achieve on-demand capacity slows things down. Being repetitive, the manual process is most likely error-prone. And it cannot be managed and automated when working with resources from different service providers together.
Infrastructure-as-Code (IaC)
This approach is the management and configuration of infrastructure (virtual machines, databases, load balancers and connection topology) in a descriptive cloud operating model. Infrastructure can be maintained just like application source code under the same version control. This lets the engineers maintain, review, test, modify and reuse their infrastructure and avoid direct dependence on the IT team. Systems can be deployed, managed and delivered fast, and automatically, through the IaC. There are many tools available for IaC such as CloudFormation (only for AWS), Terraform, Chef, Ansible and Puppet.
What is Terraform?
Terraform is an open source provisioning tool from HashiCorp (more can be read at http://terraform.io/) written in the Go language. It is used for building, changing and versioning infrastructure, safely and efficiently. Provisioning tools are responsible for the creation of server and associated services rather than configuration management (installation and management of software) on existing servers. Terraform acts as a provisioner, and focuses on the higher abstraction level of setting up the servers and associated services.
The infrastructure Terraform can manage includes low level components such as compute instances, storage, and networking, as well as high level components such as DNS entries, SaaS features, etc. It leaves the configuration management (CM) to tools such as Chef that do the job better. It lays the foundation for automation in infrastructure (both cloud and on-premise) using IaC. Its governance policy makes the cloud operating model system-compliant, which otherwise is only known internally to the IT team.
Terraform is cloud-agnostic and uses a high level declarative style language called HashiCorp Configuration Language (HCL) for defining infrastructure in ‘simple for humans to read’ configuration files. Organisations can use public templates and can have a unique private registry. Templates are a maintained repository containing pre-made modules for infrastructure components needed under version control systems (like Git).
Installation
Installing Terraform is very simple; just follow the steps mentioned below.
1. Download the archive
(https://releases.hashicorp.com/terraform/${VER}/terraform_${VER}_linux_amd64.zip):export VER=”0.12.9” wgethttps://releases.hashicorp.com/terraform/${VER}/terraform_${VER}_linux_amd64.zip |
2. Once downloaded, extract the archive:
unzip terraform_${VER}_linux_amd64.zip |
3. The last step created a Terraform bin file in the working directory. For Terraform to be accessible everywhere, move it to usr/local/bin:
sudo mv terraform /usr/local/bin |
4. Confirm the Terraform installation:
terraform –v //v0.12.9 |
Life cycle
Terraform template (code) is written in HCL language and stored as a configuration file with a .tf extension. HCL is a declarative language, which means our goal is just to provide the end state of the infrastructure and Terraform will figure out how to create it. Terraform can be used to create and manage infrastructure across all major cloud platforms. These platforms are referred to as ‘providers’ in Terraform jargon, and cover AWS, Google Cloud, Azure, Digital Ocean, Open Stack and many others.
Let us now discuss each stage in the IaC lifecycle (Figure 1), which is managed using Terraform templates.
Code
Figure 2 is sample code for starting an EC2 or t2.micro instance on AWS. As visible, only a few lines will instantiate an instance on AWS. It’s also implicit that the same code can be maintained under VCS and be used to instantiate instances in various regions with different resource configurations, removing error-prone and time-consuming manual work.
- Provider: All major cloud players such as AWS, Azure, GCP and OpenStack have their APIs for Terraform. These APIs are maintained by the community.
Username: Key given by provider
Password: Key given by provider
Region: Specify region of deployment
- Resource: There are many kinds of resources such as an open-stack basic instance, AWS EC2, Droplet and Azure VM, which can be created as follows:
resource <provider_instance_type> <identifier>
- Image ID: This is machine image specific, a tag for an image we need to install (Ubuntu and Windows).
- Flavour type: This is the type of instance governing the CPU, memory and disk space.
Here, template defines the provider as AWS, and provides the access key, secret key and region for being able to connect to AWS. After that, resource to be create is specified ie. an aws_instance here and is named as “example”. Also, the count and instance type (size of instance) is mentioned as code and can be seen in Figure 2.
The same code can be used to set up an instance in another region. Also, Terraform offers users the power of using variables and other logical statements such as if-else and the for loop, which can optimise the setting up of infrastructure even more.
Maintain and reuse
Configuration templates, i.e., pre-made modules for infrastructure components are written, reviewed, managed, and reused after storing them under VCS. Organisations can use public templates contributed by the community as well as have unique private templates stored in a central registry.
Init
The Terraform binary contains the basic functionality and everything else is downloaded as and when required. The ‘terraform init’ step analyses the code, figures out the provider and downloads all the plugins (code) needed by the provider (here, it’s AWS). Provider plugins are responsible for interacting over APIs provided by the cloud platforms using the corresponding CLI tools. They are responsible for the life cycle of the resource, i.e., create, read, update and delete. Figure 3 shows the checking and downloading of the provider ‘aws’ plugins after scanning the configuration file.
Plan
The ‘terraform plan’ is a dry run for our changes. It builds the topology of all the resources and services needed and, in parallel, handles the creation of dependent and non-dependent resources. It efficiently analyses the previously running state and resources, using a resource graph to calculate the required modifications. It provides the flexibility of validating and scanning infrastructure resources before provisioning, which otherwise would have been risky. The ‘+’ symbol signifies the new resources that will be added to the already existing ones, if any. Figure 4 shows the generation of the plan and shows the changes in resources, with ‘+’ indicating addition and ‘-’ indicating deletion.
Validation
Administrators can validate and approve significant changes produced in ‘terraform planning’s’ dry run. This completely prevents specific workspaces from exceeding predetermined thresholds, lessening the costs and increasing productivity. Also, any standards or governing policy for the cloud operating model have not been put in place yet. A policy has not been codified yet; rather, certain practices are internally known among teams and organisations. Terraform enforces sentinel policies as code before provisioning the workflow to minimise the risks through active policy enforcement.
Apply
The ‘terraform apply’ executes the exact same provisioning plan defined in the last step, after being reviewed. Terraform transforms configuration files (.tf) based on appropriate API calls to cloud provider(s) automating resource creation (using the provider’s CLI) seamlessly. This will create the resources (here, an EC2 server) on AWS in a flexible and straightforward manner. Figure 5 shows the resources that will be created, and Figure 6 confirms the changes that took place.
If we proceed to the AWS console to verify the instantiation, a new EC2 instance will be up and running, as shown in Figure 7.
Destroy
After resources are created, there may be a need to terminate them. As Terraform tracks all the resources, terminating them is also simple. All that is needed is to run ‘terraform destroy’. Again, Terraform will evaluate the changes and execute them after you give permission.
Additional features of Terraform
1. It provides a GUI to manage all the running services. It also provides an access control model based on the organisation, teams and users. Its audit logging emits logs whenever a change (here, change signifies sensitive write to existing IaC) happens in the infrastructure.
2. Existing pipeline integration: Terraform can be triggered from within most continuous integration/continuous deployment (CI/CD) DevOps pipelines such as Travis, Circle, Jenkins and Gitlab. This enables a plugin provisioning workflow and sentinel policies into the CI/CD pipeline.
3. Terraform supports many providers (more at https://www.terraform.io/docs/providers/index.html), allowing users to easily manage resources no matter where they are located. Instances can be provisioned on cloud platforms such as AWS, Azure, GCP and OpenStack using APIs provided by the cloud service providers.
4. Terraform uses a declarative style when the desired end state is written directly. Here, the tool is responsible for figuring out and achieving that end state by itself.
Let’s say, we want to deploy five Elastic instances on AWS using Chef (Figure 8a) and Terraform (Figure 8b).
Observing the script given in Figure 8, one can see that both are equivalent and will produce the same results.
But let’s assume a festive sale is coming up. The expected traffic will increase, and infrastructure must scale our application. Let’s say, five more instances are required to handle the predicted traffic.]
As language is procedural, setting count as 10 will start additional 10 instances rather than adding extra five, thus initiating a total of 15 instances. We must manually remember the previous count, as shown in Figure 9a. Hence, we must write a completely new script adding one more redundant syntax code file.
As language is declarative, setting the count as 10 (as can be seen in Figure 9b) will start an additional five instances. We do not have to manually remember the previous count. We have provided the end state and everything else will be handled by Terraform itself. This demonstrates all possible scenarios when manual interventions occur, such as slow speed, prone to human error, etc.
Advantages of incorporating IaC using Terraform
1. Prevents configuration drift: Terraform, being provisioning software, binds you to make changes in your container and only then deploy the new ones across every server. This separates server configuration from any dependency, resulting in identical instances across our infrastructures.
2. Easy collaboration: The terraform registry (Terraform’s central registry version control) enables teams to collaborate on infrastructure.
3. No separate documentation needed: The code written for infrastructure will become your documentation. By looking at the script, thanks to its procedural nature, we can figure out what’s currently deployed and its configuration.
4. Flexibility: Terraform not only handles IaaS (AWS, Azure, etc) but also PaaS (SQL, NodeJS). It can also store local variables such as cloud tokens and passwords in encrypted form on the terraform registry.
5. Masterless: Terraform is masterless, by default, i.e., it does not need a master node to keep track of all configuration and distributing updates. This saves the extra infrastructure and maintenance costs we’d have to incur in maintaining an extra master node. Terraform directly uses the cloud providers’ API, thus saving extra infrastructure costs and other overheads.
Terraform is an open source tool that helps teams manage infrastructure in an efficient, automated and reusable manner. It has a simple modular syntax and supports multi-cloud infrastructure configuration. Enterprises can use Terraform in their DevOps methodology to construct, modify, manage and deliver infrastructure at a faster pace with less manual intervention.
