Managing a multi-environment serverless architecture in AWS

Objective

Our pipeline contains three different environments — develop, staging and production. Source code commits merged into master should be deployed automatically to the develop environment, whereas transitioning to the other two environments requires manual approval. This strategy was easy to achieve with the single app backend we used until now (a single compiled Go binary) but poses a few challenges with a serverless architecture as we now need to deploy a number of different resources.

The three environments in our pipeline: develop, staging and production.

Serverless web applications on AWS

Before we get into the details of deployment let’s briefly look at the components involved in a typical serverless setup and what concepts AWS provides to handle multiple environments.

Client requests are routed and validated by AWS Gateway before being handled by AWS Lambda.

API Gateway

AWS offers powerful ways to handle different versions of the same API Gateway resource through the use of ‘stages’. Here’s what the documentation has to say:

Lambda

Lambda functions themselves have their own concept of versioning. Another look at the official documentation reveals the following:

CloudFormation

CloudFormation is an infrastructure management tool and accompanying spec that allow you to define a group of AWS resources grouped into a ‘stack’. Resources are defined in templates written in json or yaml and the cloudformation tool allows the infrastructure to be created, modified via change sets and destroyed.

  • package will parse the template file (yaml or json), find the functions with a codeUri that points to a local filesystem handler, package and upload them to S3, and then output a packaged template where the codeUris now point to S3 artefacts.
  • deploy uploads your packaged template to CloudFormation, creates a change set and executes it. This will attempt to migrate your target stack into a state that matches the template that you provided, including the latest version of your Lambda handlers which have been packaged as S3 artifacts.

SAM

Serverless Application Model is an attempt to simplify the definition of serverless applications by extending the CloudFormation specification. It adds three new resource types:

  • AWS::Serverless::Function
  • AWS::Serverless::Api
  • AWS::Serverless::SimpleTable

Local development environment

To run and test SAM based applications locally awslabs released sam local, a CLI that can be used to invoke Lambda functions directly or start a local mock API Gateway from a SAM template which will invoke your functions to handle incoming requests. It does this by executing your local handlers in Docker containers which mimic the real Lambda execution environment. In case you were wondering, it does also come with support for the recently announced official support for Go on AWS Lambda.

sam local start-api --template api-template.yaml
  • missing support for custom authorisers, hopefully that will change after SAM introduces first class support for authorisers.
  • bug preventing external OpenAPI yaml files from working (although JSON seems to be working according to multiple user reports)

Single stack vs multi stack

People are still finding out how to best use these tools in the real world, as evidenced by the issues where users ask for infrastructure modelling advice. One example of a contentious area is the management of different environments, such as develop, staging, production, etc.

API Gateway and Lambda require different configurations in single and multi stack setups.

Putting it all together

Having decided on a multi stack approach our CI setup remained pretty straightforward: we use CodePipeline to automatically take the latest commit on master, run unit tests, compile handlers and deploy the new versions to the develop stack before manually approving deployments to staging and production stacks.

aws cloudformation package --template-file stack-template.yaml --s3-bucket <s3-bucket> --output-template-file packaged-template.yaml
aws cloudformation deploy --template-file packaged-template.yaml --stack-name <StackDev|StackStaging|StackProd> --capabilities CAPABILITY_NAMED_IAM --parameter-overrides StageName=<Dev|Staging|Prod>

Conclusion

After spending some time familiarising ourselves with more AWS concepts than we could ever have wished for, attempting a single stack approach with SAM, and browsing numerous GitHub issues, blog posts and talks, we eventually decided that the multi stack approach was the best way for us to reach our multi environment target. We hope this write up will help others facing similar questions. In the meantime we’ll keep an eye on the issues mentioned above.

What we didn’t cover

  • Integration testing via a dedicated stack, as part of our CI pipeline.
  • Traffic shifting for deployments via Lambda aliases. Note that aliases are still a good way to control deployments within an environment, just not so much for environment separation.

References

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Sergio Garcez

Sergio Garcez

London based freelance software developer, interested in way too many things.