Dmitriy Drenkalyuk , DevOps / Senior Software Engineer
In this article, I will describe how to move an AWS Lambda function deployed as a Serverless application into a Terraform module. The article may be useful for:
- Those who are not too familiar with Lambda function development and deployment process and want to get a quick overview of it and compare how this could be done using Serverless and Terraform.
- Those who already use Serverless Framework to deploy a Lambda function in AWS, but want to use Terraform to manage it. This might be the case when everything else is already managed by Terraform, except for the Lambda function, which was historically developed as a Serverless application, so you want to consolidate your infrastructure and use Terraform for everything.
- Those who are going to create a new Lambda function and don’t want to use Serverless Framework for the same reasons as above.
The implementation also includes a special case of running
npm install from Terraform when deploying.
Serverless Framework implementation
So let’s assume we’re given a Serverless application deployed in AWS, this could be simply an HTTP(s) endpoint that redirects requests somewhere else. Let’s also assume that the function has a layer with Node.js runtime dependencies.
The project structure could look like this:
1 2 3 4 5 6 7
So we have:
- serverless.yml where our Lambda function is described
- index.js where we have an HTTP request handler
- layers/nodejs directory where our layer with Node.js dependencies is placed (note that the
nodejsdirectory name isn’t random, it’s restricted by the Node.js runtime, see here)
Our serverless.yml could look as follows:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
And we deploy it with the following commands:
1 2 3 4 5
For those who are not familiar with how Serverless Framework performs deployments, I will briefly describe the process, so that you can get an idea of how it works in general and which AWS resources are involved:
- An AWS CloudFormation template is created from our serverless.yml.
- An S3 bucket is created (if this is the first deployment), this is where zip files of the Lambda function code will be stored.
- The code of the function is packaged into zip files (in our case this would be 2 files - one for the function itself and another one for the Node.js layer).
- The hashes of the files are compared against the previous deployment (if any) and the deployment process is terminated if all file hashes are the same (this would mean we’re trying to deploy the same code).
- Zip files are uploaded to the S3 bucket.
- Any IAM Roles, Functions, Events and Resources are added to the CloudFormation template.
- The CloudFormation stack is updated with the new CloudFormation template.
- A new version is published for our Lambda function.
This process is described here.
This would create the following AWS resources:
- CloudFormation template used to provision the stack
- S3 bucket where zip files of the function are stored
- Lambda serverless application
- Lambda function, belonging to the application
- Lambda layer with Node.js dependencies, belonging to the function
- CloudWatch log group with log streams for each instance of the function
- Role attached to the function with the following policies:
- An “assume role” policy with a permission for AWS Lambda service to assume this role
- A policy allowing the CloudWatch log group to create log streams and put log events to these streams
- REST API Gateway with:
/event/pushPOST endpoint integrated with the function
- a permission to invoke the function
After we implement the Lambda function with Terraform, we will have pretty much the same set of resources, except for the:
- CloudFormation template
- S3 bucket
- Lambda serverless application
So you can remove them after you’ve moved your function.
Creating Terraform module
So, let’s start to create a Terraform module for deploying our Lambda function. The module we’ll implement will work for Terraform v0.11.13 (may work for v0.12 too, but I didn’t test that) and AWS plugin v2.56.0 (some of the AWS resources are not supported in some earlier plugin versions). In this article, I will put everything into a single module, but you might want to split the code to different reusable modules, for instance, you could create modules for CloudWatch log group and API Gateway related resources and include them to the main module.
The main Terraform module will be organized as follows:
1 2 3 4 5 6 7 8 9 10
IAM role and logging
Let’s first create resources that the Lambda function is going to depend on - CloudWatch log group and IAM role with policies.
Note: In Terraform code, you can place resources in any order, because Terraform will define the order of creation according to the references of the resources within other resources in the module and the
depends_on directives. But I personally like when resources are placed in the code in as much as possible the same order, this is why we will add these resources first.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53
Nothing special here, just note that we will pass
test_lambda_function_stage as a variable when deploying, this will allow us to deploy the function for different environments, e.g. dev/prod. Various resource names in the code below will include the STAGE as well. Also, I have declared a local variable for the Lambda function name as it will be reused further.
Our Lambda function requires two Terraform resources -
aws_lambda_layer_version (the Node.js layer) and
aws_lambda_function (the function itself with the layer attached to it). Both resources require the
filename argument with a path to a deployment file.
Note: An alternative to this argument is a group of
s3_* arguments that would specify a location of the deployment file in an S3 bucket, but we will not use this for our function.
We will create deployment packages using the
archive_file data source. For the Lambda function itself, we can simply zip index.js, but for the Node.js layer, we need to run
npm install first, and we will use
local-exec provisioner for this. The resource takes no action and the provisioner invokes a local executable after the resource is “created”.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
Note that the null resource has a trigger that generates a different string every time, so that the resource is “replaced” (in fact, just the local command is invoked) on every run. Also, the
archive_file data source depends on the null resource, so that we create the deployment file only after
npm install. You may want to add
rm -rf node_modules before
npm install if you are going to run Terraform in an environment which does not clean up its workspace.
Now, add the Lambda function:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
Here, we have attached the role to the function and added an explicit dependency on the log group. You also may want to add
timeout arguments or
aws_lambda_function_event_invoke_config resource if you want to use a non-default configuration for the Lambda function.
For the REST API Gateway, we will need to create the following Terraform resources:
aws_api_gateway_rest_api- the API itself
aws_lambda_permission- permission for the API Gateway service to invoke the Lambda function
aws_api_gateway_resource- represents a path part, for the
/event/pushpath, you need to create
push, specifying the
eventas a parent for the
pushand the root API resource as a parent for the
aws_api_gateway_method- POST method for the
aws_api_gateway_integration- to integrate the Lambda function as the target backend
aws_api_gateway_deployment- adds a deployment for the API (you can switch between deployments in AWS management console on the
Stagesview for your API stage)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51
A couple of notes:
- for Lambda integration, you have to specify the
aws_api_gateway_deploymentexplicitly depends on the
aws_api_gateway_integrationto avoid race conditions
- for the same reasons,
aws_lambda_permissiondepends on the
- the API stage is created automatically, because we’ve tagged our API with the
STAGEtag, but you can add more stages with
- the default
Errormodels are created automatically, but you can add more models with
Additionally, you may want to add a custom domain name for the API Gateway. Domain name and certificate creation/importing with Terraform is out of scope of this article, I will only show how to register it for use with our REST API Gateway.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
We have added an explicit dependency for the
aws_api_gateway_base_path_mapping, because we don’t have any references of the
aws_api_gateway_domain_name in it.
Importing resources to the Terraform state
Since all the AWS resources have been created by Serverless Framework, we don’t have them in the Terraform state file, so we need to import them. In the comments below you can see what you need to use (i.e. name, id, ARN, etc.) to import the resources:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
You apparently don’t need to import the null resource (as this is just a helper) and the
aws_api_gateway_deployment, because it doesn’t make sense to import previous deployments, and we want a new deployment to be created when we run Terraform. Terraform actually doesn’t even have a functionality to import API deployments.
If you run
terraform plan after you’ve imported everything, you should see that Terraform wants to do the following:
- Update some of the resources with some minor changes:
- Create a new API deployment (
aws_api_gateway_deployment), which will be basically the same as the previous one
- Create a new
aws_lambda_functionwith it - because we run
npm installand “replace” the related null resource everytime, and the Lambda layer resource depends on it.
You can run
terraform apply now, so that the state file is fully synchronized with the existing infrastructure. If you don’t want it to create a new Lambda layer version and API Gateway deployment, use the
-target=<resource_address> parameter to limit the operation to a subset of resources.
Our Lambda function is now completely managed by Terraform and the state file is updated with all the related AWS resources. We can now update it when we need using Terraform and use it with other Terraform resources that we have in our infrastructure. Thank you for reading and feel free to let me know in the comments below if you have found a mistake or have a question.