The sad truth about AWS Lambda cold starts in .NET

When building a web application using a serverless micro-service architecture on AWS, you end up using lambdas extensively. Lambdas are very useful and simple but they suffer from the cold start problem. As described in details by Serhat Can in this article, a cold start is defined as:

A latency experienced when you trigger a function.

A cold start only happens if there is no idle container available waiting to run the code. This is all invisible to the user and AWS has full control over when to kill containers.

So when a user of a serverless web application is the victim of a cold start, he may get impatient because he has to wait longer for the page to load.

After some investigation, I came to the conclusion that you should expect a minimal cold start time of roughly 3 seconds (yes 3 seconds!) for an ASP.NET Core Web API hosted by an AWS lambda that has 256 MB of RAM. It doesn’t look to me like you can get significantly below 3 seconds without increasing the amount of RAM granted to the lambda (thus throwing money on the problem).

The current article explains the steps that I took to come to that conclusion.

Two different types of lambdas

The most simple kind of lambdas (which I will call the “simple” lambdas) is one that looks like this:

Lambdas like this one are typically used to do something in reaction to an AWS events (SNS, CloudWatch, SQS, etc) or to run some code in the background (StepFunction). Being in reaction to something and running in the background, they are affected by the lambda cold start problem but it doesn’t usually directly affect the user. He does not have to wait for something synchronously.

The second kind of lambdas are those that are used to expose a REST API using ASP.NET Core lambdas. Such lambdas are typically invoked by an AWS API Gateway or an AWS Application Load Balancer. Since the REST API of such lambdas are typically consumed by a web application, the users of the web application are directly affected by the lambda cold start problem. So we want the cold start of these lambdas to be as small as possible.

The current article will measure the cold start of these two types of lambdas.

First benchmarks

To do my measurements, I wanted to create these two types of lambdas while keeping them as simple of possible. They should contain virtually no business logic and they should have a minimal set of dependencies. To achieve this, I used the templates provided by the AWS Toolkit for Visual Studio. These templates setup the minimal infrastructure that you would typically require. So here are the template that I used:

• Simple Lambda: Empty Function
• REST API: ASP.NET Core Web API (I cleaned a little the generated code to remove the dependency on DynamoDb)

I then created a very simple benchmark project using BenchmarkDotNet. This project has two benchmarks:

• SimpleLambda: invokes the lambda by using the InvokeAsync method of the AWSSDK.
• AspnetCoreLambda: invokes the lambda by using an HTTP client.

The resulting code is available here. Here are the results of the benchmark:

Since BenchmarkDotNet has a warm-up phase, these numbers don’t show the effect of the cold start. So we basically measure the performance of our lambdas when they are warm. This is not what we want to measure. But still, we can see that my SimpleLambda is a little faster to invoke than the AspnetCoreLambda. This is not really surprising since in one case I go straight to the lambda and in the other case I have to go through the API Gateway.

Let’s now force a cold start on every invocation of the lambda. I do this by changing an environment variable on my lambda before each invocation. It’s a little hackish but it does the job. Here’s the commit that does this and here are the results:

The first sad conclusion is that ASP.NET Core lambdas are significantly slower to start than the simple lambdas. 1.6 seconds slower! Our users will get impatient for sure!

Let’s see if we can improve that.

Tiered compilation to the rescue?

Tiered compilation is a new opt-in feature of .NET core 2.1 that attempts to provide two improvements:

• Faster application startup time
• Faster steady-state performance

The first point may help us to improve our lambda cold starts. Let’s give it a shot. I enabled it and ran the tests again.

So what do we see here?

• The SimpleLambda starts 184 ms faster on average with tiered compilation enabled while the AspnetCoreLambda starts 104 ms faster. So our application does start faster with tiered compilation but it is not a game changer. 2.8 seconds is still very slow.
• For some reason that I can’t explain, the standard deviation is significantly higher with tiered compilation.

Given how small my code base is, it may explain why I don’t see a significant difference. With a larger real life code base, the impact may be more significant.

Lambda package size?

When talking about optimizing cold start time, one advise that always comes up is to minimize the size of the lambda package. So, would it be possible that my ASP.NET Core lambda starts slowly just because the zip package uploaded to AWS is significantly bigger than the package of my SimpleLambda? To verify this, I downloaded the zip of my two lambdas. I got this:

• SimpleLambda: 203 KB
• AspnetCoreLambda: 922 KB

Here’s the actual content of these zip files (SimpleLambda on the right):

They are therefore pretty small. I guess that my extra 1.6 seconds cold start delay in AspnetCoreLambda is not caused simply by the small extra 719 KB. For now, I will give the benefit of the doubt to AWS.

So tiered compilation does not really help and my packages are already pretty small. Could pre-jitting help?

Pre-jitted ASP.NET Core Assemblies

When developing a docker based micro-service on top of ASP.NET Core, it is good practice to base your docker images on top of the image microsoft/dotnet:2.1-aspnetcore-runtime because it has pre-jitted ASP.NET Core assemblies. Is there something similar with Lambdas that could be done to improve our cold start?

As we’ve seen when looking inside of the zip package of my ASP.NET Core lambda, the zip does not contain any ASP.NET core related assemblies. It only contain the Amazon related assemblies and my own assemblies. As explained here, this is because the ASP.NET Core assemblies are already stored in the “Runtime package store”. According to the same article, they are pre-jitted.

These packages have also been pre-jitted, meaning they’re already compiled from .NET’s intermediate language (IL) to machine instructions. This improves startup time when you use these packages. The store also reduces your deployment package size, further improving the cold startup time.

We can therefore rule out something else from the equation. ASP.NET Core is already pre-jitted.

Being a little out of ideas, I will now dig in the logs of the ASP.NET Core lambda to see if they reveal something interesting.

Digging in the ASP.NET Core lambda logs

I now want to see what’s going on inside of the lambda. This can be done with the logs that end up in CloudWatch automatically (everything written on stdout ends in CloudWatch). Unfortunately, the logs don’t include a precise timestamp by default. I went ahead and created a custom logger that displays more accurate timestamps. After I deployed this modified version, I called my lambda twice to cause both a cold start and a warm start. Here are the results.

In the cold start logs, we see that:

• It takes about 1 second to hit the START row. In this 1 second, we did everything that is done only once in the lifetime of the lambda.
• After the START row, it takes about 1.5 seconds to execute the ASP.NET Core pipeline and return
• The time between the Request starting and Route matched with is 1.28 seconds.

And in the warm start logs:

• The time between the Request starting and Route matched with is 0.0011 seconds. This is much faster than on the first call!

So, it looks like ASP.NET Core is responsible for this extra warm-up latency. Since the ASP.NET Core pipeline is not being executed when using a simple lambda, it explains why these lambdas have a much better startup time than the ASP.NET Core lambdas.

What could be so long between Request starting and Route matched with? My guess would be that on the first call, ASP.NET has to use reflection to find the different controllers and routes so that it can know what can be matched to the current request. If my guess is a good one, then there isn’t much that an application developer can do to improve this (at least to my knowledge). Improvements have to come from ASP.NET to reduce this delay.

Sad conclusion

At this point, I am out of ideas on what I could do to improve the cold start of my lambdas:

• I have a very minimal lambda: it has no business logic and a very small deployment package.
• Tiered compilation does not significantly help.
• The ASP.NET Core assemblies are pre-jitted.
• The first call to ASP.NET Core introduces a delay of about 1.3 seconds to dispatch the call to the good route.

I therefore come to the conclusion that, when allocating only 256 MB of RAM to an ASP.NET Core Lambda, you should expect a cold start time of roughly 3 seconds.

If you have any insights on what could be done, then please share them! I would be more than happy to be proved wrong 😉