Rust, AWS Lambda, and too many Office quotes

Nov 2, 2023 aws

Back from a hard fought battle against writer's block, I've been looking for a way to convince my boss to let me use Rust at work. Most of our infrastructure is on AWS (:surprised-pikachu-face:) and I've been writing a lot of new system features designed to run serverless-ly with things like Lambda, Step Functions, SQS, SNS, and all the other band members we know and love. We're a .NET shop, so moving to Rust wouldn't exactly be an overnight transition, nor would I want to force my zealotry upon my fellow developers.

My days are mostly spent raising a newly added member to my family, and I needed a reason to stay up late staring at my laptop while watching The Office reruns. Then it dawned on me... what if I could combine my two favorite things in Rust and The Office? It might sound crazy, but we're all about technological experimentation around these parts.

The outcome of that experiment was a Lambda deployed to AWS fronted by an API Gateway available to make requests to and get random quotes from The Office out. A request might look like:

$ curl -l "https://{{gateway URL}}/quotes" | jq .
{
    "author": "Prison Mike",
    "quote": "The worst thing about prison was the Dementors."
}

With the help cargo lambda, I was surprised at how easy it was to get up and running with Lambdas that were even more easily deployed to AWS. As an added bonus, I sprinkled in some Terraform because I'm lazy and don't know which buttons to click in AWS most of the time. If you're following along, it'll help to have the following installed:

Cargo and cargo lambda installed (a quick cargo install cargo-lambda should do the trick)
Terraform CLI
An AWS account (I'm still on the free tier, luckily)

We'll touch the surface of a few things here, but won't be going into depth necessarily on any one topic. There's people a lot smarter than myself that are ackshually qualified to talk about Rust, AWS, and Terraform.

Getting started

First thing's first, we're gonna need some rust code to deploy. Let's spin up a new project with cargo lambda:

$ cargo lambda new office-quotes
> Is this function an HTTP function? Yes

We're prompted about the compute context of our Rust-based Lambda, which in our case, will be from an API Gateway request. Lambdas are compute services that can be triggered from any number of things in AWS like events from SNS. I plan to eventually display some random Office quotes for anyone visiting my website, so I'll make it available over the network for my Svelte frontend to call.

Cracking open our main.rs file, we'll see a pretty bare bones scaffolded Rust application:

use lambda_http::{run, service_fn, Body, Error, Request, RequestExt, Response};
 
/// This is the main body for the function.
/// Write your code inside it.
/// There are some code example in the following URLs:
/// - https://github.com/awslabs/aws-lambda-rust-runtime/tree/main/examples
async fn function_handler(event: Request) -> Result<Response<Body>, Error> {
    // Extract some useful information from the request
    let who = event
        .query_string_parameters_ref()
        .and_then(|params| params.first("name"))
        .unwrap_or("world");
    let message = format!("Hello {who}, this is an AWS Lambda HTTP request");
 
    // Return something that implements IntoResponse.
    // It will be serialized to the right response event automatically by the runtime
    let resp = Response::builder()
        .status(200)
        .header("content-type", "text/html")
        .body(message.into())
        .map_err(Box::new)?;
    Ok(resp)
}
 
#[tokio::main]
async fn main() -> Result<(), Error> {
    tracing_subscriber::fmt()
        .with_max_level(tracing::Level::INFO)
        // disable printing the name of the module in every log line.
        .with_target(false)
        // disabling time is handy because CloudWatch will add the ingestion time.
        .without_time()
        .init();
 
    run(service_fn(function_handler)).await
}

The event our function receives is an HTTP request from the API gateway that has a bunch of metadata about the request, like query strings, path parameters, where the request came from, body of the request, etc. Luckily, I ain't got time for all that noise - I want to simply return some quotes through a GET with an optional query parameter author to get character specific quotes.

Next up, we're gonna need some quote data. Through the power of ChatGPT, I was able to generate a quotes.json file that'll serve as our data source for quotes. When we eventually (inevitably?) choose to exercise our ability to prematurely optimize our solution, we'll add in connectors to a plethora of different data sources on the off chance we need to support multiple databases, caches, flat files, etc.

Our quotes file is pretty standard JSON:

quotes.json

{
  "quotes": [
    {
      "quote": "Would I rather be feared or loved? Easy. Both. I want people to be afraid of how much they love me.",
      "author": "Michael Scott"
    },
    {
      "quote": "Whenever I'm about to do something, I think, 'Would an idiot do that?' and if they would, I do not do that thing.",
      "author": "Dwight Schrute"
    }
    // And many more...
  ]
}

Placing that at the root of our project directly next to Cargo.toml should do the trick so we can read it in, parse it into a struct of sorts, and spit out some data on the other side when a request comes in. I'm gonna add a few crates to help me out:

$ cargo add anyhow # To make error handling a little easier
$ cargo add rand # To help us pick random quotes if no author is provided via query param
$ cargo add serde --features macros # To help us read in JSON quotes to Rust structs
$ cargo add serde_json # To help us write data out in the response the AWS Rust runtime expects

With our crates in place, next let's add a file for parsing quote data from the quotes file:

src/quotes.rs

use std::{env::current_dir, fs::File, io::Read};
 
use anyhow::Context;
use rand::Rng;
use serde::{Deserialize, Serialize};
 
#[derive(Debug, Clone, Deserialize, Serialize)]
pub struct Quote {
    pub author: String,
    quote: String,
}
 
#[derive(Debug, Deserialize)]
pub struct QuotesData {
    quotes: Vec<Quote>,
}
 
impl QuotesData {
    pub fn get_random_quote(&self) -> Quote {
        let random_index = rand::thread_rng().gen_range(0..self.quotes.len());
        self.quotes[random_index].clone()
    }
 
    pub fn get_quote_by(&self, author: &str) -> Option<Quote> {
        self.quotes
            .clone()
            .into_iter()
            .find(|q| q.author.to_lowercase().contains(&author.to_lowercase()))
    }
}
 
pub fn get_quotes() -> anyhow::Result<QuotesData> {
    let quotes_file_path = current_dir().context("unable to determine current directory")?;
 
    let mut file = File::open(format!(
        "{}/quotes.json",
        quotes_file_path.to_str().unwrap()
    ))
    .context("unable to read quotes file")?;
 
    let mut file_contents = String::new();
 
    file.read_to_string(&mut file_contents)
        .context("unable to read the file contents into buffer")?;
 
    serde_json::from_str::<QuotesData>(&file_contents).context("unable to parse quotes")
}

Nothing too fancy here. We're defining a few structs to hold our quote data in Quote and QuoteData, while impl' ing some functions on QuoteData to get us a quote when asked for one either from a specific author, or a random quote.

Finally, we export a function for parsing the quotes file into our QuotesData stuct so we can do some logic with it later. I should note that this isn't exactly the most exciting data, nor the most practical. You're probably already asking yourself "wait... so we're parsing JSON data into Rust structs only to... return JSON data in the response?" Yes, that's exactly what we're doing. Don't ask me why.

Okay, so we have the ability to read the quotes file, now let's update our entrypoint into the function that will determine the context of the request and grab a quote:

mod quotes;
 
use anyhow::Context;
use lambda_http::{run, service_fn, Body, Error, Request, RequestExt, Response};
use quotes::get_quotes;
use serde_json::json;
use tracing::info;
 
/// This is the main body for the function.
/// Write your code inside it.
/// There are some code example in the following URLs:
/// - https://github.com/awslabs/aws-lambda-rust-runtime/tree/main/examples
async fn function_handler(event: Request) -> Result<Response<Body>, Error> {
    info!("received request to get office quotes, loading quote data");
 
    // Load the quotes from our JSON file
    let quotes = get_quotes()?;
 
    // Grab a quote if an author name was sent along in the query params
    // If no author is sent, we'll grab a random quote from the JSON
    let quote = match event
        .query_string_parameters_ref()
        .and_then(|params| params.first("author"))
    {
        Some(author) => {
            info!("requested to retrieve quotes by author {author}");
            quotes.get_quote_by(author)
        }
        None => {
            info!("no author specified, retrieving a random quote");
            Some(quotes.get_random_quote())
        }
    };
 
    // Finally, determine the response based on the authored quote we generated
    // If we successfully generated a quote, wrap it up in a nice JSON response
    // In the case an author was passed in via query param but no quote was found,
    // return an error response in JSON format with the help of serde_json's `json!()` macro
    match quote {
        Some(authored_quote) => {
            info!("quote retrieved by author {}", &authored_quote.author);
 
            let resp = Response::builder()
                .status(200)
                .header("content-type", "application/json")
                .body(
                    serde_json::to_string(&authored_quote)
                        .context("unable to serialize the hilarious quote")?
                        .into(),
                )
                .context("error attempting to build response body")?;
 
            Ok(resp)
        }
        None => {
            let resp = Response::builder()
                .status(404)
                .header("content-type", "application/json")
                .body(
                    json!({
                        "error": "Quote by that author does not exist."
                    })
                    .to_string()
                    .into(),
                )
                .context("error attempting build the error response")?;
 
            Ok(resp)
        }
    }
}
 
#[tokio::main]
async fn main() -> Result<(), Error> {
    tracing_subscriber::fmt()
        .with_max_level(tracing::Level::INFO)
        // disable printing the name of the module in every log line.
        .with_target(false)
        // disabling time is handy because CloudWatch will add the ingestion time.
        .without_time()
        .init();
 
    info!("bootstrapping lambda");
 
    run(service_fn(function_handler)).await
}

Again, nothing too fancy here. At the top level, we're inspecting the request for query parameters and if one was sent, find a quote by that author from the JSON we parsed. If no author sent, generate a random quote. Finally, we'll return JSON the API gateway based on the result of the request, falling back to an error if an author was provided but no quote was found. I've sprinkled in some .context()? utilities to help us early return from unexpected errors with the help of anyhow. In a more robust application, we'd probably want to do some more fine-grained error handling.

Now that we've got our function in place, let's test it out. cargo lambda has some sweet utilities to help us out, including a watch command:

$ cargo lambda watch
INFO invoke server listening on [::]:9000
INFO starting lambda function function="_" manifest="Cargo.toml"
Compiling office-quotes v0.1.0 (/home/jmckenzie/typescript/joey-mckenzie-tech/examples/rust/with-aws-lambda)
Finished dev [unoptimized + debuginfo] target(s) in 1.40s
    Running `target/debug/office-quotes`
INFO bootstrapping lambda

And if we ping localhost:9000:

$ curl -l "localhost:9000" | jq .
{
  "author": "Michael Scott",
  "quote": "Would I rather be feared or loved? Easy. Both. I want people to be afraid of how much they love me."
}

Let's verify the query parameters are making it into the request as well:

$ curl -l "localhost:9000?author=kelly" | jq .
{
  "author": "Kelly Kapoor",
  "quote": "I talk a lot, so I've learned to just tune myself out..."
}

Lastly, let's check the error case where no author is found

$ curl -l "localhost:9000?author=ron" | jq .
{
  "error": "Quote by that author does not exist."
}

Nice! We've got ourselves an MVP, time to ship to production.

Deploying to AWS

With our deployment approach, we'll do something akin to the following:

Build the output artifact with the help of cargo lambda
Package up the output into a zip file to store in S3
Upload the zip file into a bucket
Setup an Lambda function using the zip file as the source executable
Setup an API Gateway instance that proxies request through to our Lambda function

Now doing all that stuff manually is not too tedious, but I've been writing a lot Terraform lately and thought it would fun to Terraform-erize this process. If you're not familiar with Terraform, it's a Hashicorp product with the goal of making provisioned infrastructure easier to main through infrastructure as code, or IaC. Terraform using a configuration language called Hashicorp Configuration Language, or HCL, to define the who/what/when/where/why/how of our AWS infrastructure.

I like to think of Terraform as a recipe for what our AWS infrastructure, while also having the ability to plan and apply those infrastructure changes for us, saving us an uncountable amount of mouse clicks navigating through the AWS console.

An example piece of TF configuration might look like:

resource "aws_lambda_function" "office_quotes" {
  function_name = "office-quotes"

  s3_bucket = aws_s3_bucket.lambda_bucket.id
  s3_key    = aws_s3_object.lambda_office_quotes.key

  handler = "rust.handler"
  runtime = "provided.al2"

  source_code_hash = data.archive_file.lambda_office_quotes.output_base64sha256

  role = aws_iam_role.lambda_execution_policy.arn
}

Here, we're defining a resource that happens to be a Lambda function called office_quotes. That Lambda has a function name of office-quotes, has its source files located in an S3 bucket (which we'll provision in just a minute), and runs on an EC2 instance with the provided.al2 runtime. There's some other stuff in there like the role, which defines the execution policy invokers of the function should have, and an MD5 hash of the zip file output. I mentioned earlier that this isn't necessarily a blog post about Terraform, so I'll leave a link to the example code here.

Following the plan above, first thing we need is an S3 bucket we can store our zipped up function code in. I'll create a bucket.tf configuration file that will do just that:

bucket.tf

resource "random_pet" "lambda_bucket_name" {
  prefix = "rust-lambda"
}

resource "aws_s3_bucket" "lambda_bucket" {
  bucket = random_pet.lambda_bucket_name.id
}

resource "aws_s3_bucket_ownership_controls" "lambda_bucket" {
  bucket = aws_s3_bucket.lambda_bucket.id
  rule {
    object_ownership = "BucketOwnerPreferred"
  }
}

resource "aws_s3_bucket_acl" "lambda_bucket" {
  depends_on = [aws_s3_bucket_ownership_controls.lambda_bucket]

  bucket = aws_s3_bucket.lambda_bucket.id
  acl    = "private"
}

data "archive_file" "lambda_office_quotes" {
  type = "zip"

  source_dir  = "${path.module}/../target/lambda/office-quotes"
  output_path = "${path.module}/bootstrap.zip"
}

resource "aws_s3_object" "lambda_office_quotes" {
  bucket = aws_s3_bucket.lambda_bucket.id

  key    = "bootstrap.zip"
  source = data.archive_file.lambda_office_quotes.output_path

  etag = filemd5(data.archive_file.lambda_office_quotes.output_path)
}

The first few resources defined above describe the bucket name with the help of a couple randomly generated names and permissions on the bucket. The last few pieces of configuration define some data we're going to work with that happens to be an archive file and an object that will exist in that bucket that's just the zip file of our function code.

You may have noticed that the source_dir of our archive_file data that we'll need doesn't actually exist yet - let's build it! Within our parent directory (I usually stick all my Terraform specific stuff in a subdirectory of the project I'm working in) let's run a quick cargo lambda build --release to build the output we need.

Once the build finishes, you should notice an exectuable file named bootstrap should be present in your target/lambda/office-quotes folder. cargo lambda offers different build configurations as well, allowing output formats to also be specified - we could also run the build with the --output-format zip flag to get a ready-to-upload file with bootstrap.zip. Since we're leaning on Terraform to do the file zipping for us though, we'll take the default generated executable instead.

With our bucket configuration in place, let's define our Lambda function configuration:

lambda.tf

resource "aws_lambda_function" "office_quotes" {
  function_name = "office-quotes"

  s3_bucket = aws_s3_bucket.lambda_bucket.id
  s3_key    = aws_s3_object.lambda_office_quotes.key

  handler = "rust.handler"
  runtime = "provided.al2"

  source_code_hash = data.archive_file.lambda_office_quotes.output_base64sha256

  role = aws_iam_role.lambda_execution_policy.arn
}

resource "aws_cloudwatch_log_group" "office_quotes" {
  name = "/aws/lambda/${aws_lambda_function.office_quotes.function_name}"

  retention_in_days = 1
}

resource "aws_iam_role" "lambda_execution_policy" {
  name               = "office-lambda-execution-role"
  assume_role_policy = jsonencode({
    Version   = "2012-10-17"
    Statement = [
      {
        Action    = "sts:AssumeRole"
        Effect    = "Allow"
        Sid       = ""
        Principal = {
          Service = "lambda.amazonaws.com"
        }
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "lambda_policy_attachment" {
  role = aws_iam_role.lambda_execution_policy.name

  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
}

As we saw earlier, we'll use the same bit of function configuration and add a few things like CloudWatch logs and an execution policy we'll expect services invoking the function to have.

Let's hookup the final piece of infrastructure we'll need for now in an API Gateway resource:

gateway.tf

resource "aws_apigatewayv2_api" "office_gateway" {
  name = "office-gateway"

  protocol_type = "HTTP"
}

resource "aws_apigatewayv2_stage" "office_gateway" {
  api_id = aws_apigatewayv2_api.office_gateway.id

  name        = "prod"
  auto_deploy = true

  access_log_settings {
    destination_arn = aws_cloudwatch_log_group.office_gateway.arn

    format = jsonencode({
      requestId               = "$context.requestId"
      sourceIp                = "$context.identity.sourceIp"
      requestTime             = "$context.requestTime"
      protocol                = "$context.protocol"
      httpMethod              = "$context.httpMethod"
      resourcePath            = "$context.resourcePath"
      routeKey                = "$context.routeKey"
      status                  = "$context.status"
      responseLength          = "$context.responseLength"
      integrationErrorMessage = "$context.integrationErrorMessage"
    }
    )
  }
}

resource "aws_apigatewayv2_integration" "get_quote" {
  api_id = aws_apigatewayv2_api.office_gateway.id

  integration_uri    = aws_lambda_function.office_quotes.invoke_arn
  integration_type   = "AWS_PROXY"
  integration_method = "POST"
}

resource "aws_apigatewayv2_route" "get_quote" {
  api_id = aws_apigatewayv2_api.office_gateway.id

  route_key = "GET /quotes"
  target    = "integrations/${aws_apigatewayv2_integration.get_quote.id}"
}

resource "aws_cloudwatch_log_group" "office_gateway" {
  name = "/aws/api-gateway/${aws_apigatewayv2_api.office_gateway.name}"

  retention_in_days = 1
}

resource "aws_lambda_permission" "office_gateway" {
  statement_id = "AllowExecutionFromAPIGateway"
  action       = "lambda:InvokeFunction"

  function_name = aws_lambda_function.office_quotes.function_name
  principal     = "apigateway.amazonaws.com"

  source_arn = "${aws_apigatewayv2_api.office_gateway.execution_arn}/*/*"
}

Our API Gateway configuration is defined as a good ole fashioned HTTP API that has a route integration under the /quotes path that will invoke our Lambda function. With the help of Terraform, we can reference bits of infrastructure created in other files without needing to hard code or manually point to things.

When we apply all of our Terraform infrastructure, we'll need a way to reference our created API Gateway without having to step into the AWS Console and click our way through to the API Gateway dashboard. Let's add an outputs.tf file that tell Terraform that once all of our infrastructure is created, output it to the console for us. The created resources are also saved in our terraform.tfstate file as well, so we can also reference them there if need be.

output "base_url" {
  description = "Base URL for API Gateway stage."

  value = aws_apigatewayv2_stage.office_gateway.invoke_url
}

output "lambda_bucket_name" {
  description = "Name of the S3 bucket used to store function code."

  value = aws_s3_bucket.lambda_bucket.id
}

While we're at it, we'll go ahead and print out our bucket name as well as everytime we'll destroy/apply infrastructure, it'll change due to the random module we're leveraging to avoid clashing buckets. Now if we apply this plan, we should see the fruits of our labor:

$ terraform apply -auto-approve
 
# A bunch of logs about planned infrastructure...
 
Apply complete! Resources: 15 added, 0 changed, 0 destroyed.
 
Outputs:
 
base_url = "https://dbyhxt543e.execute-api.us-west-1.amazonaws.com/prod"
lambda_bucket_name = "rust-lambda-choice-cricket"

And if I ping my URL (yours will be different) at the /quotes route:

$ curl -l "https://dbyhxt543e.execute-api.us-west-1.amazonaws.com/prod/quotes" | jq .
{
  "author": "Dwight Schrute",
  "quote": "Identity theft is not a joke, Jim! Millions of families suffer every year."
}

Success! Rust, running on Lambda, publicly available through an API Gateway. This is great and all, but we need a way to reliably rebuild our infrastructure and apply changes.