Organize AWS Object using Resource Group¶
When there are many resources in your template, it becomes difficult to manage and debug. You probably want to put resources into different logic “Resource Group”, and you want to test them separately, even though in most of the case they depend on each other. For debugging, it is always better to deploy resources gradually rather than deploy all of them at once.
Let’s walk through an example to demonstrate how and the benefit of doing this.
Step 1. Declare cottonformation style parameter¶
At begin, we simply declare a CloudFormation stack object. And declared two simple cottonformation style parameters project_name and stage. Stack is not associated with a CloudFormation template object yet.
# -*- coding: utf-8 -*-
import attr
import cottonformation as cf
@attr.s
class IamStack(cf.Stack):
project_name: str = attr.ib()
stage: str = attr.ib()
@property
def env_name(self):
"""
A prefix for most of naming convention. Isolate resource from each other.
"""
return f"{self.project_name}-{self.stage}"
@property
def stack_name(self):
"""
CloudFormation stack name.
"""
return f"{self.env_name}-iam-stack"
Step 2. Declare abstract resource group logic¶
Now we believe that we will need two resource groups. For each resource group, we define a method to put into creation logics. And we have a high level method post_hook to explicitly call resource group creation methods. Of course, you can easily comment them in-and-out to enable/disable those resources. Right now we have nothing in those resource group creation methods.
# -*- coding: utf-8 -*-
import attr
import cottonformation as cf
@attr.s
class IamStack(cf.Stack):
project_name: str = attr.ib()
stage: str = attr.ib()
@property
def env_name(self):
"""
A prefix for most of naming convention. Isolate resource from each other.
"""
return f"{self.project_name}-{self.stage}"
@property
def stack_name(self):
"""
CloudFormation stack name.
"""
return f"{self.env_name}-iam-stack"
#=============================================================================
# New Code starts here
#=============================================================================
def mk_rg1(self):
"""
Make resource group 1
"""
pass
def mk_rg2(self):
"""
Make resource group 2
"""
pass
def mk_rg3(self):
"""
Make resource group 3
"""
pass
def post_hook(self):
"""
A user custom post stack initialization hook function. Will be executed
after object initialization.
We will put all resources in two different resource group.
And there will be a factory method for each resource group. Of course
we have to explicitly call it to create those resources.
"""
self.mk_rg1()
self.mk_rg2()
self.mk_rg3()
Step 3. Declare AWS Resources and Resource Group¶
We declare some AWS Resource for each resource group. Here we use IAM.Group, because it is fast to create and basically has no effect to the AWS Account.
Resource group itself is similar to other AWS Object, you have to pass a unique logic id. Basically you can think that it is just a AWS Object container (Parameter, Resource, Output, Mapping, Condition, Rule …), even for another Resource Group. You can use cottonformation.core.model.ResourceGroup.add or cottonformation.core.model.ResourceGroup.add_many API to add other resource to a resource group.
At the end, we create a instance of this stack and a template object for future use. At this moment, stack and template doesn’t know each other yet.
# -*- coding: utf-8 -*-
import attr
import cottonformation as cf
from cottonformation.res import iam
@attr.s
class IamStack(cf.Stack):
project_name: str = attr.ib()
stage: str = attr.ib()
@property
def env_name(self):
"""
A prefix for most of naming convention. Isolate resource from each other.
"""
return f"{self.project_name}-{self.stage}"
@property
def stack_name(self):
"""
CloudFormation stack name.
"""
return f"{self.env_name}-iam-stack"
#=============================================================================
# New Code starts here
#=============================================================================
def mk_rg1(self):
"""
Make resource group 1
"""
# declare a resource group, you can use Stack.rg1 to access it later.
self.rg1 = cf.ResourceGroup("RG1")
# declare a resource
self.iam_group1 = iam.Group(
"IamGroup1",
p_GroupName=f"{self.env_name}-group1",
)
# add resource to resource group
# internally it use "Depends On" mechanism. In this example, we can say
# rg1 depends on iam_group1
self.rg1.add(self.iam_group1)
def mk_rg2(self):
"""
Make resource group 2
"""
self.rg2 = cf.ResourceGroup("RG2")
# you can even add another resource group to it
self.rg2.add(self.rg1)
self.iam_group2 = iam.Group(
"IamGroup2",
p_GroupName=f"{self.env_name}-group2",
)
self.rg2.add(self.iam_group2)
def mk_rg3(self):
"""
Make resource group 3
"""
self.rg3 = cf.ResourceGroup("RG3")
self.rg3.add(self.rg2)
self.iam_group3 = iam.Group(
"IamGroup3",
p_GroupName=f"{self.env_name}-group3",
)
self.rg3.add(self.iam_group3)
def post_hook(self):
"""
A user custom post stack initialization hook function. Will be executed
after object initialization.
We will put all resources in two different resource group.
And there will be a factory method for each resource group. Of course
we have to explicitly call it to create those resources.
"""
self.mk_rg1()
self.mk_rg2()
self.mk_rg3()
#=============================================================================
# New Code starts here
#=============================================================================
iam_stack = IamStack(
project_name="ctf-best-prac-res-group",
stage="dev",
)
tpl = cf.Template(Description="Demo: Resource Group best practice")
Step 4. Manage Resource Group in Template¶
To add resource declared in a stack to a cloudformation Template is easy. You can add resource group in ascending order. Or you can just add the last resource group in dependency chain. All dependency resource will be automatically added. We can easily enable/disable many resources by comment in/out.
For debug, sometime you want to remove resource in descending order.
# -*- coding: utf-8 -*-
import attr
import cottonformation as cf
from cottonformation.res import iam
@attr.s
class IamStack(cf.Stack):
project_name: str = attr.ib()
stage: str = attr.ib()
@property
def env_name(self):
"""
A prefix for most of naming convention. Isolate resource from each other.
"""
return f"{self.project_name}-{self.stage}"
@property
def stack_name(self):
"""
CloudFormation stack name.
"""
return f"{self.env_name}-iam-stack"
def mk_rg1(self):
"""
Make resource group 1
"""
# declare a resource group, you can use Stack.rg1 to access it later.
self.rg1 = cf.ResourceGroup("RG1")
# declare a resource
self.iam_group1 = iam.Group(
"IamGroup1",
p_GroupName=f"{self.env_name}-group1",
)
# add resource to resource group
# internally it use "Depends On" mechanism. In this example, we can say
# rg1 depends on iam_group1
self.rg1.add(self.iam_group1)
def mk_rg2(self):
"""
Make resource group 2
"""
self.rg2 = cf.ResourceGroup("RG2")
# you can even add another resource group to it
self.rg2.add(self.rg1)
self.iam_group2 = iam.Group(
"IamGroup2",
p_GroupName=f"{self.env_name}-group2",
)
self.rg2.add(self.iam_group2)
def mk_rg3(self):
"""
Make resource group 3
"""
self.rg3 = cf.ResourceGroup("RG3")
self.rg3.add(self.rg2)
self.iam_group3 = iam.Group(
"IamGroup3",
p_GroupName=f"{self.env_name}-group3",
)
self.rg3.add(self.iam_group3)
def post_hook(self):
"""
A user custom post stack initialization hook function. Will be executed
after object initialization.
We will put all resources in two different resource group.
And there will be a factory method for each resource group. Of course
we have to explicitly call it to create those resources.
"""
self.mk_rg1()
self.mk_rg2()
self.mk_rg3()
iam_stack = IamStack(
project_name="ctf-best-prac-res-group",
stage="dev",
)
tpl = cf.Template(Description="Demo: Resource Group best practice")
#=============================================================================
# New Code starts here
#=============================================================================
# add resource group from stack to template, in ascending order.
tpl.add(iam_stack.rg1)
tpl.add(iam_stack.rg2)
tpl.add(iam_stack.rg3)
# or you can just do: add(rg3). since we declared that rg3 depends on rg2
# and rg2 depends on rg1, all resource declared in rg2 and rg1 will be
# automatically added.
# tpl.add(iam_stack.rg3) # uncomment this for testing
# For debugging, you could just remove some resource group and comment them
# in and out one by one between cloudformation deployment.
tpl.remove(iam_stack.rg3)
tpl.remove(iam_stack.rg2)
# or you can just do: remove(rg2). since rg3 depends on rg2
# if we remove rg2, rg3 will be automatically removed
# tpl.add(iam_stack.rg2) # uncomment this for testing
Step 5. Play with Deployment¶
Now we can play with the deployment. And feel free to comment in and out either in the template.add(resource_group) code block, either in the template.remove(resource_group) code block.
# -*- coding: utf-8 -*-
import attr
import cottonformation as cf
from cottonformation.res import iam
dummy_policy_document = {
"Version": "2012-10-17",
"Statement": [
]
}
@attr.s
class IamStack(cf.Stack):
project_name: str = attr.ib()
stage: str = attr.ib()
@property
def env_name(self):
"""
A prefix for most of naming convention. Isolate resource from each other.
"""
return f"{self.project_name}-{self.stage}"
@property
def stack_name(self):
"""
CloudFormation stack name.
"""
return f"{self.env_name}-iam-stack"
def mk_rg1(self):
"""
Make resource group 1
"""
# declare a resource group, you can use Stack.rg1 to access it later.
self.rg1 = cf.ResourceGroup("RG1")
# declare a resource
self.iam_group1 = iam.Group(
"IamGroup1",
p_GroupName=f"{self.env_name}-group1",
)
# add resource to resource group
# internally it use "Depends On" mechanism. In this example, we can say
# rg1 depends on iam_group1
self.rg1.add(self.iam_group1)
def mk_rg2(self):
"""
Make resource group 2
"""
self.rg2 = cf.ResourceGroup("RG2")
# you can even add another resource group to it
self.rg2.add(self.rg1)
self.iam_group2 = iam.Group(
"IamGroup2",
p_GroupName=f"{self.env_name}-group2",
)
self.rg2.add(self.iam_group2)
def mk_rg3(self):
"""
Make resource group 3
"""
self.rg3 = cf.ResourceGroup("RG3")
self.rg3.add(self.rg2)
self.iam_group3 = iam.Group(
"IamGroup3",
p_GroupName=f"{self.env_name}-group3",
)
self.rg3.add(self.iam_group3)
def post_hook(self):
"""
A user custom post stack initialization hook function. Will be executed
after object initialization.
We will put all resources in two different resource group.
And there will be a factory method for each resource group. Of course
we have to explicitly call it to create those resources.
"""
self.mk_rg1()
self.mk_rg2()
self.mk_rg3()
iam_stack = IamStack(
project_name="ctf-best-prac-res-group",
stage="dev",
)
tpl = cf.Template(Description="Demo: Resource Group best practice")
# add resource group from stack to template, in ascending order.
tpl.add(iam_stack.rg1)
tpl.add(iam_stack.rg2)
tpl.add(iam_stack.rg3)
# or you can just do: add(rg3). since we declared that rg3 depends on rg2
# and rg2 depends on rg1, all resource declared in rg2 and rg1 will be
# automatically added.
# tpl.add(iam_stack.rg3) # uncomment this for testing
# For debugging, you could just remove some resource group and comment them
# in and out one by one between cloudformation deployment.
tpl.remove(iam_stack.rg3)
tpl.remove(iam_stack.rg2)
# or you can just do: remove(rg2). since rg3 depends on rg2
# if we remove rg2, rg3 will be automatically removed
# tpl.add(iam_stack.rg2) # uncomment this for testing
#=============================================================================
# New Code starts here
#=============================================================================
tpl.batch_tagging(dict(ProjectName=iam_stack.project_name, Stage=iam_stack.stage))
if __name__ == "__main__":
# my private aws account session and bucket for testing
from cottonformation.tests.boto_ses import bsm, bucket
env = cf.Env(bsm=bsm)
env.deploy(
template=tpl,
stack_name=iam_stack.stack_name,
bucket_name=bucket,
include_iam=True,
)