AWS Cloudformation and Boto an alternative approach to stacks creation. AWS CloudFormation gives developers and systems administrators an easy way to create and manage a collection of related AWS resources. We can make an entire AWS Infrastructure starting from a template. A template has the following high level JSON structure:

{
"Parameters": {   },
"Resources" : {   },
"Outputs" : {   }
}

You can deploy and update a template and its associated collection of resources (called a stack) by using the AWS Management Console, AWS Command Line Interface, or SDKs as boto (AWS SDK for Python).

Using Cloudformation we are able to create our AWS infrastructure, starting by a single template where we describe all the resources (EC2 instances, RDS Instances, bucket S3, Cloudfront distribution, etc) of our infrastructure. Usually the infrastructure is described by a single monolithic template. This is a good solution in all those cases in which there is a strong dependency between the "tiers" of the infrastructure and almost never the user needs to create a single tier independently of the others. For example in case of Blue/Green deployment the user would like to create only the WEB Tier resources (EC2 instances, Elastic Load balancing and Autoscaling group) keeping the others tiers as DBMS and CDN unchanged or in those cases where the user want to test a single tier configuration. In case like this is a good practice to decompose the template to smaller template, one for each tier and make each tier independently. In these cases we can use the command line at runtime indicating the template to start

aws cloudformation create-stack --stack-name web-tier --region eu-west-1 --template-body file://./cloudformation/web.json --parameters file://./cloudformation/web/production.json

This practice, however, has a limit: a output of a Tier may be an input of other tier. In this case the user needs to retrieve the output of the first tier and then add it as a parameter of the second tier. This can be done manually, but in a logic to automate our infrastructure is better to delegate this task to a script. In this post i would like to introduce an alternative approach to this practice: decompose a monolithic cloudformation template in to smaller template, one for each tier, and add a file (JSON) that describe which tier should be active/created, the relative template and parameters file, what are its relations with the other tiers and eventually script to execute for fix missing action on Cloudformation Engine. We have a JSON file that describe the acrhitecture (the tiers of our architecture and relative dependencies and fix) plus a python script that parse this file and resolve denpendencie and launch fix modules.

Infrastructure.json

In this file we have a list of stacks to create (if stack is note defined will be not created). Each stack has a name, a template file, a parameters file and an array of dependencies and fix. The array of dependencies is a set of dependency from other stacks/tier, for example the security group in output created during web stack creation is an input for demo-preproduction-cache, because the security group of Elasticache cluster accept connection only from security group of EC2 instances on WEB Tier. The array of fix is a set di python modules that will be executed for fix missing action on Cloudformation service. For example with fix module we can tag the elasticache cluster, infact at this moment the tags on elasticache cluster are not supported by Cloudformation.

This is an example of stack with dependencies and fix

For example output of Web Stack

{
      "name": "demo-preproduction-cache",
      "template_file": "./cloudformation/elasticache/elasticache.json",
      "parameters_file": "./cloudformation/elasticache/preproduction.json",
      "dependencies": [
        {
          "stack": "demo-preproduction-web",
          "mapping": [
            {
              "output": "SecurityGroup",
              "input": "BackendInstanceSecurityGroup"
            }
          ]
        }
      ],
      "fix" : [
        {
          "fix_file" : "fix/fixredis.py"
        }
      ]
}

Wrapper.py

The wrapper.py script is the root of our logic. The script parse infrastructure.json file retrieve the stacks that must to be create, resolve dependencies

def resolve_dependencies(parameters_file, dependencies, connection):
  parameters_string = read_conf(parameters_file)
  parameters_list = json.loads(parameters_string)
  for parameter in parameters_list:
    for depend in dependencies:
      for mapping in depend['mapping']:
        if parameter['ParameterKey'] == mapping['input']:
          for output in get_output_of_stack(depend['stack'],connection):
            if output.key == mapping['output']: 
              parameter['ParameterValue'] = output.value
 return json.dumps(parameters_list)

and execute fix.

def resolve_fix(fixes, stack):
 for fix in fixes:
 command = "python %s %s" % (fix['fix_file'], stack)
 return os.system(command)

Cloudformation is able to implement a similar solutions by a conditions and using a parameter for able or disable a resource. But this is an alternative approach particularly in those cases where the script is required to overcome some "missing" features of Cloudformation.  For example the script Wrapper.py is able to resolve dependency and launch a python script for fix missing action on Cloudformation engine.For example with fix module we can tag the elasticache cluster, infact at this moment the tags on elasticache cluster are not supported by Cloudformation.

All Scripts are available on my GitHub repository and i would like to thanks the MailUP team for support on this experimental task in particular Fabio e Daniele