Posted Updated 4 minutes read (About 660 words)

Orchestrate Multiple Environments with GCP

The purpose of this series of posts on Terraform with GCP is to accomplish more with less. Here we try to optimize our templates for bringing up multiple environments across multiple projects in GCP. Below approach will help spin multiple instances with minimal efforts by introducing .tfvars files into our templates.

Use case: I have 2 projects gcp-homecompany-qa and gcp-homecompany-dev for this purpose and we will have to create compute instances with terraform on GCP. Lets get on with it.

The folder structure goes as below

1
2
3
4
5
6
7
8
9
---/gce/`
	-- firewall.tf
	-- httpd_install.sh
	-- main.tf
	-- output.tf
	-- provider.tf
	-- variables.tf
	-- app-dev.tfvars
    -- app-qa.tfvars

The varaibles.tf file will be used to declare the variables and to assign few default values.

1
2
3
4
5
6
7
8
9
10
11
12
variable "test_servers" { 

  type = list(any) 
}

variable "disk_zone" { default = "" }
variable "disk_type" { default = "" }
variable "disk_name" { default = "" }
variable "disk_size" { default = "" }
variable "project_id" { default = "" }
variable "credentials_file" { default = "" }
variable "path" { default = "/home/admin/gcp_credentials/keys" }

The values to these variables are assigned in the respective .tfvars files, so here we create 2 .tfvars files to lets say spin up 2 environments Dev and QA environments. And the two files are defined as below:

dev.tfvars 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
credentials_file = "gcp-homecompany-dev-key.json"

project_id = "gcp-homecompany-dev"



test_servers = [{

  id = 1

  compute_instance_name = "demo1"

  compute_machine_type = "e2-standard-2"

  compute_image = "centos-8"

  compute_network = "home-network"

  compute_subnet = "home-sub-subnetwork"

  compute_zone = "us-central1-a"

  compute_size = "100"

},

{

  id = 2

  compute_instance_name = "demo2"

  compute_machine_type = "e2-standard-2"

  compute_image = "centos-8"

  compute_network = "home-network"

  compute_subnet = "home-sub-subnetwork"

  compute_zone = "us-central1-a"

  compute_size = "100"

},

{

  id = 3

  compute_instance_name = "demo3"

  compute_machine_type = "e2-standard-2"

  compute_image = "centos-8"

  compute_network = "home-network"

  compute_subnet = "home-sub-subnetwork"

  compute_zone = "us-central1-a"

  compute_size = "100"

}]



disk_zone = "us-east1-b"

disk_type = "pd-ssd"

disk_name = "additional volume disk"

disk_size = "150"

app-qa.tfvars

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
credentials_file = "gcp-homecompany-qa-key.json"

project_id = "gcp-homecompany-qa"



test_servers = [{

  id = 1

  compute_instance_name = "demo1"

  compute_machine_type = "e2-standard-2"

  compute_image = "centos-8"

  compute_network = "home-network"

  compute_subnet = "home-sub-subnetwork"

  compute_zone = "us-central1-a"

  compute_size = "100"

},

{

  id = 2

  compute_instance_name = "demo2"

  compute_machine_type = "e2-standard-2"

  compute_image = "centos-8"

  compute_network = "home-network"

  compute_subnet = "home-sub-subnetwork"

  compute_zone = "us-central1-a"

  compute_size = "100"

},

{

  id = 3

  compute_instance_name = "demo3"

  compute_machine_type = "e2-standard-2"

  compute_image = "centos-8"

  compute_network = "home-network"

  compute_subnet = "home-sub-subnetwork"

  compute_zone = "us-central1-a"

  compute_size = "100"

}]

disk_zone = "us-east1-b"

disk_type = "pd-ssd"

disk_name = "additional volume disk"

disk_size = "150"

In the above .tfvars files we tried to populate the list test_servers with 3 google compute instances. In order to iterate through this list with key-value pairs we try to implement for loop with a for_each meta-arguments in the below template. Hence following changes are to be done to our main.tf file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
resource "google_compute_instance" "test_instance" {

  for_each = { for test_instance in var.test_servers : test_instance.id => test_instance }

  name = each.value.compute_instance_name

  machine_type = each.value.compute_machine_type

  zone = each.value.compute_zone

  metadata_startup_script = "${file("httpd_install.sh")}" 

  can_ip_forward = "false"



//  tags = ["",""]

  description = "This is our virtual machines"
  tags = ["allow-http","allow-https"]
  boot_disk {
    initialize_params {
      image = each.value.compute_image
      size = each.value.compute_size
    }
  }


  network_interface {
    network = each.value.compute_network
    subnetwork = each.value.compute_subnet
    access_config {
   // Ephemeral IP
    }
  }

 service_account {
  scopes = ["userinfo-email", "compute-ro", "storage-ro"]
  }

The for_each meta argument will assign the values to the arguments from the list with key-value pair. While we can now test the above template. For generalizing the network, subnet and load balancer related stuffs, I will post in the future articles.

terraform apply -var-file=app-<env>.tfvars

And the above command create compute instances depending on the .tfvars files passed while applying .

Posted Updated 4 minutes read (About 534 words)

Making Terraform Dynamic with Interpolation

Continuing from the previous post we will try to introduce interpolation, flow control and looping. We will split the main.tf to different chunks of files that hold specific definitions to create the resources in GCP. We will create the provider.tf file which holds the provider configurations.

provider.tf

1
2
3
4
5
6
7
8
9
10
variable "path" {  default = "/home/vagrant/gcp_credentials/keys" }

provider "google" {
    project = "triple-virtue-271517"
    version = "~> 3.38.0"
    region = "us-central1"
    zone = "us-central1-a"
    credentials = "${file("${var.path}/triple-virtue.json")}"
  
}

Firewall rules can be defined in a separate file as firewall.tf as below:

firewall.tf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
resource "google_compute_firewall" "allow-http-port" {
    name = "allow-http-port"
    network = "default"

    allow {
        protocol = "tcp"
        ports = ["80"]
    }

    target_tags = ["allow-http"]
      
}

resource "google_compute_firewall" "allow-https-port" {
    name = "allow-https-port"
    network = "default"

    allow {
        protocol = "tcp"
        ports = ["443"]
    }

    target_tags = ["allow-https"]
      
}

Interpolation in Terraform helps to assign values to variables, this way we can dynamically manage the provisioning of resources in the cloud environments. Here we create variables.tf file with defines the variables that can be used in the script.

variable.tf

1
2
3
4
5
6
7
variable "image" {  default = "centos-8" }
variable "machine_type" { default = "n1-standard-2" }
variable "name_count" { default = ["server-1","server-2","server-3"]}
variable "environment" { default = "production" }
variable "machine_type_dev" { default = "n1-standard-1" }
variable "machine_count" { default = "1" }
variable "machine_size" { default = "20" }

We will then create a seperate file httpd_install.sh where we install the web servers into the compute instances.

httpd_install.sh

1
2
3
4
5
sudo yum update 
sudo yum install httpd -y
sudo systemctl start httpd
sudo systemctl start httpd
sudo systemctl enable httpd

Now lets define the main.tf that reffers to the interpolation, firewall rules and the script to install the apache webservers.

main.tf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
resource "google_compute_instance" "default" {
    
    count = length(var.name_count)
    name = "list-${count.index+1}"
    machine_type = var.environment != "production" ? var.machine_type : var.machine_type_dev
    metadata_startup_script = "${file("httpd_install.sh")}" 
    

    can_ip_forward = "false"
    description = "This is our virtual machines"

        tags = ["allow-http","allow-https"]

  


    boot_disk {
        initialize_params {
            image = var.image
            size = var.machine_size
        }
    }


    network_interface {
        network = "default"
        access_config {
      // Ephemeral IP
        }
    }

    metadata = {
        size = "20"
        foo = "bar"
    }

}

Now by carefully observing main.tf, we see that the lines refer to the variables defined in the variables.tf 

1
2
3
count = length(var.name_count)
name = "list-${count.index+1}"
machine_type = var.environment != "production" ? var.machine_type : var.machine_type_dev

Further the above lines also shows the looping and flow control. Here we are looping to create 3 compute instances of type production grade. Below we see clear interpolation the terraform which refers the image and machine_size defined in the variables.tf

boot_disk {
    initialize_params {
        image = var.image
        size = var.machine_size
    }
}

The below line initializes the installation of apache webservers with httpd_install.sh script.

metadata_startup_script = "${file("httpd_install.sh")}"

Hence the output.tf will look like below:

output.tf

1
2
3
4
5
6
7
output "machine_type" {
  value = "${google_compute_instance.test_instance[*].machine_type}"
}

output "name" {
  value = "${google_compute_instance.test_instance[*].name}"
}

the overall files created in this regard is as below:

1
2
3
4
5
6
7
---/gce/
	-- firewall.tf
	-- httpd_install.sh
	-- main.tf
	-- output.tf
	-- provider.tf
	-- variables.tf

The results of the above experiments are as below:

Posted Updated 4 minutes read (About 540 words)

Beginning Terraform with GCP

This and the next series of posts will demonstrate the simplification of introducing complexity in IaC best practices. But first a simple Terraform script to provision resources on a GCP cloud. We dive into getting a VM instance with Apache web server with in Google Cloud Platform public in public cloud. We start with one main.tf which has all the configurations and the resources to provision and orchestrate in GCP.

Lets first define the provider configurations:

1
2
3
4
5
6
7
8
9
provider "google" {

project = "triple-virtue-271517"
version = "~> 3.38.0"
region = "us-central1"
zone = "us-central1-a"
credentials = "${file("${var.path}/cloud-access.json")}"

}

The path variable refers to the access tokens to GCP cloud project as below:

1
variable "path" { default = "/home/vagrant/gcp_credentials/keys" }

Lets define the firewall rules with default network resources

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
resource "google_compute_firewall" "allow-http-port" {
    name = "allow-http-port"
    network = "default"

    allow {
        protocol = "tcp"
        ports = ["80"]
    }

    target_tags = ["allow-http"]
      
}

resource "google_compute_firewall" "allow-https-port" {
    name = "allow-https-port"
    network = "default"

    allow {
        protocol = "tcp"
        ports = ["443"]
    }

    target_tags = ["allow-https"]
      
}

The target_tags defined shall then be referred in the resources (VM instances) that may require the firewall rules to enable http and https ports.

Further we will define code to provision a VM instance and map it to the default network with above mentioned firewall rule

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
resource "google_compute_instance" "test_instance" {
    
    name            = "demo-01"
    machine_type    = "e2-standard-2"
    zone            = "us-central1-a"
    metadata_startup_script = <<-EOF
    sudo yum update 
    sudo yum install httpd -y
    sudo systemctl start httpd
    sudo systemctl start httpd
    sudo systemctl enable httpd
    EOF

    tags = ["allow-http","allow-https"]

    boot_disk {
      
    initialize_params{

          image           = "centos-8"
          size            = "100"
            

        }
    }

    network_interface {
        network = "default"
    
            access_config {
      // Ephemeral IP
        }
    }

    service_account {
        scopes = ["userinfo-email", "compute-ro", "storage-ro"]
    }

}

The metadata_startup_script also tries to install webserver while provisioning the vm instance. The network_interface section assigns a public ip to the same instance.

Now putting all together

main.tf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
variable "path" {  default = "/home/vagrant/gcp_credentials/keys" }

provider "google" {
    project = "triple-virtue-271517"
    version = "~> 3.38.0"
    region = "us-central1"
    zone = "us-central1-a"
    credentials = "${file("${var.path}/cloud-access.json")}"
  
}

resource "google_compute_firewall" "allow-http-port" {
    name = "allow-http-port"
    network = "default"

    allow {
        protocol = "tcp"
        ports = ["80"]
    }

    target_tags = ["allow-http"]
      
}

resource "google_compute_firewall" "allow-https-port" {
    name = "allow-https-port"
    network = "default"

    allow {
        protocol = "tcp"
        ports = ["443"]
    }

    target_tags = ["allow-https"]
      
}


resource "google_compute_instance" "test_instance" {
    
    name            = "demo-01"
    machine_type    = "e2-standard-2"
    zone            = "us-central1-a"
    metadata_startup_script = <<-EOF
    sudo yum update 
    sudo yum install httpd -y
    sudo systemctl start httpd
    sudo systemctl start httpd
    sudo systemctl enable httpd
    EOF

    tags = ["allow-http","allow-https"]

    boot_disk {
      
    initialize_params{

          image           = "centos-8"
          size            = "100"
            

        }
    }

    network_interface {
        network = "default"
    
            access_config {
      // Ephemeral IP
        }
    }

    service_account {
        scopes = ["userinfo-email", "compute-ro", "storage-ro"]
    }

}

output "machine_type" {
  value = "${google_compute_instance.test_instance.machine_type}"
}

output "name" {
  value = "${google_compute_instance.test_instance.name}"
}

Here below are the results of the above resources created in GCP.

The server instance created in the vm console

The Apache webserver running in that instance

In the next part we will further refine the above script by splitting the script into different files and terraform interpolation.

Follow

Recents

Archives

Tags

Alexa1
Alexa Skills Development1
GCP3
Terraform3
Terraform GCP Series3
jenkins2
vagrant2

Subscribe for updates

Links

Advertisement

×