Terraform Modules for Amazon (RDS) Aurora MySQL

Introduction

The aim of this post is to give information on Terraform modules for Aurora MySQL and how to reuse these modules while launching multiple Aurora MySQL clusters. The audience for this topic is anyone who is aware of terraform and basics of MySQL.

Aurora

 Amazon Aurora is Relational Database engine and has the following features. Aurora is:

• designed to offer greater than 99.99% availability by integrating the database engine with a SSD-backed storage, increasing performance.
• fault-tolerant and self-healing and disk failures are repaired in the background without loss of database availability. 
• designed to automatically detect database crashes and restart without the need for crash recovery or to rebuild the database cache. If the entire instance fails, Amazon Aurora will automatically fail over to one of up to 15 read replicas.
• scales storage automatically, growing storage and rebalancing I/Os to provide consistent performance without the need for over-provisioning freeing you from time-consuming administrative tasks such as provisioning, patching, backup, recovery, failure detection and repair.

Create AWS Aurora Clusters using AWS console

You can quickly launch an Amazon Aurora database instance with a few clicks in the RDS Management Console.

Please follow this step by step instructions to create AWS Aurora Cluster - https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_GettingStarted.CreatingConnecting.Aurora.html

Aurora Terraform Modules

 Terraform modules provide an easy way to abstract common blocks of configuration into reusable infrastructure elements. Modules can help with to significantly reduce code duplication. instead of copying and pasting this code all over the place, you can turn it into a reusable module by putting it in a folder. These modules support versioning, so you can safely try out different versions of a module in different environments and roll back to previous versions if you hit a problem. Here is the simple  directory structures for the Aurora Terraform modules:

 They are hosted in this git repo

As a part of creating Aurora MySQL cluster, we should have few more resources before actually creating Aurora Cluster. we need 4 main resources before we start aurora cluster:

1. DB Option Group
2. DB Parameter Group
3. DB Subnet Group
4. Cluster Parameter Group

Once these basic resources are ready, then

1. Create Aurora Cluster
2. Aurora Cluster Instances

if you look at the module directory structure, it is same for everyone. each and every module is having

1. main.tf
2. variable.tf
3. outputs.tf 
4.  TERRAFORM.md or README.md

With this terraform module you can create 2 types of Aurora clusters:

1. Aurora Cluster without data (Empty Cluster) [Module - rds_cluster]
2. Aurora Cluster with Data (Restore from backup) [Module - rds_cluster_restore_data_from_s3]
   1. This is done by creating a backup using the Percona XtraBackup tool and uploading the resulting files to an Amazon S3 bucket.
      
      1. Install Percona XtraBackup tool 

         yum install http://www.percona.com/downloads/percona-release/redhat/0.1-4/percona-release-0.1-4.noarch.rpm yum install percona-xtrabackup-2.3.6

         
         
      2. Take MySQL backup using this command 

         /usr/bin/innobackupex --ibbackup=/usr/bin/xtrabackup --open-files-limit=65000 --slave-info --with-timestamp --backup --stream=tar | gzip - >

Note:

    1. Take MySQL backup using percona XtraDB backup tool and store in S3 bucket. this S3 bucket should be pass as one of the parameters while launching Aurora cluster with data.

    2. we have written our own Python tool which creates Aurora Cluster by restoring data from S3.

    3. By default Terraform doesn't support creating Aurora Cluster by "Restoring From backup" - we have already added feature request with Terraform - https://github.com/terraform-providers/terraform-provider-aws/issues/2440

Flowchart Diagram

This is how the Aurora terraform modules works.

Terraform module README links

1. DB Parameter Group
2. DB Subnet Group
3. Cluster Parameter Group
4. Aurora Cluster without data
5. Aurora Cluster with data
6. Aurora Cluster instances

How to use Modules

Create Aurora clusters using these modules, each README file has the details on how to call each module in code

please refer changelog for all new features - changelog 

For example - if you want to create DB parameter group please call modules like this 

module "aurora_db_parameter_group" {
  source                  = "git::https://github.com/sanjeet-deshpande//tf-aws-aurora.git//db_parameter_group?ref=v0.0.1"
  comp_vars               = "${var.temp_vars}"
  comp_product_service         = "${var.product_service}"
  role                    = "${var.role}"
  environment             = "${var.environment}"
  db_parameter_group_name = "${var.db_parameter_group_name}"
  db_parameter_family     = "${var.db_parameter_family}"
  db_parameters           = ["${var.db_parameters}"]
}