What am I going to talk about today.

Why hard coding AWS ARNs and IDs is a bad idea.
Writing a filter using RDS as an example.
How to use an Ansible filter.
Benefits of using an Ansible filter.

Hard coding ARNs and IDs. Life without filters. :(

To deploy a new VPC and all of it’s resources (Subnets, Routes, IGWs, NAT Gateways, Peers).

Steps to Deploy a VPC (hard coding).

Run vpc_deploy.yml playbook. (Wait for playbook to finish running.)
Hard code IDs and ARNs in either your host_vars, group_vars, or vars folder in the role you used to deploy the VPC.
Commit the changes and push to GitHub or what ever VCS you use.

Example of hard coding IDs in group_vars. “group_vars/environments/test/network.yml“

Now you still have to deploy the AWS resources your services depend on (RDS, Security Groups, ELB, ASG, Kinesis, SQS, etc…).

Steps to Deploy AWS Services (hard coding).

Run aws_services.yml playbook (Wait for playbook to finish running.)
Hard code IDs and ARNs in either your host_vars, group_vars, or vars folder in the role you used to deploy the AWS services.
Commit the changes and push to GitHub or what ever VCS you use.

*Example of not using filters in a role “roles/services/vars/webapp.yml*”

Now if I had to do that for every VPC we manage, I would lose my mind. Also what if I want to deploy a new VPC to test out a new feature. This is not only time consuming but a real pain in the ass. Would you not rather come up with a name scheme for your AWS infrastructure and based on that name scheme. Now you will no longer have to worry about hard coding IDs or ARNs any more.

Filters and the end of hard coding ARN’s and IDs.

I see Ansible filters as easy to write Python functions. Anything you can write in a function can be used as a filter. If you have been writing scripts in Python for a while, you will find that writing Ansible plugins is such a breeze.

Due to the awesomeness of filters, I no longer have to hard code any ARN or ID in any of my playbooks. Instead I have a filter that grabs the ARN for me based on the name of the resource.

In order for these AWS filters to work, you will need to use the Name tag for all of your services.

A look into the get_rds_endpoint filter.

My decision to use the filter plugin instead of the lookup plugin, is purely a choice based on taste. I do not want to have a Python file for each lookup that I require. I rather write a module for each type of filtering that I need to do. For instance, my aws calls are in the filter_plugins/aws.py.

The get_rds_endpoint filter will query the AWS API using the RDS instance name and return back the endpoint address.

The get_rds_endpoint function is 15 lines of code and 13 lines of documentation. Such a small script and yet it has saved me from hard coding.

Example of using the get_rds_endpoint filter in a role. “roles/services/vars/webapp.yml”

---
aws_region: us-west-2
mysql_server: "{{ aws_region | get_rds_endpoint(rds_instance_name) }}"
memcached_server: "{{ aws_region | get_elasticache_endpoint(memcached_instance_name) }}"
redshift_server: "{{ aws_region | get_redshift_endpoint(redshift_name) }}"

The benefits of using the filters above.

You do not need to know the RDS instance address, SQS URL, or RedShift Cluster Address in advance.
You do not need to hard code it for each VPC or environment or app. Name space everything appropriately, and you will always get the right ARN, ID, Address, URL, etc..
AWS filters should be used in the task or role and not in group_vars/, host_vars, or vars. If the resource does not exist when you call the filter. The filter will raise an exception

Wrap up

Do not hard code ARNs or IDs.
Since you are no longer hard coding ARNs or IDs, you will now spend less time commiting to your VCS.
Use AWS filters in your roles or in tasks.
Do not use AWS filters in group_vars, host_vars, or vars. If the resource does not exist when you call the filter, the filter will raise an exception.
You can see all of the AWS filters I have written
An example Playbook that uses filters

AnsibleFest 2016 San Francisco Talk Deploying to AWS using Ansible and Magic

AnsibleFest2016-Deploying-To-AWS

Deploying to AWS using Ansible and Magic