AWS lambda function not updating

[sebastian]

hi.
I have a bamboo build plan with two steps:
"source code checkout"
"aws lambda function"

everything seems to work fine, I can run it successfully.

I go to the aws console and it says the lambda function has been updated.

but then I run the function and the code has not been updated.

what could be causing this problem?

[Steffen Opel _Utoolity_ · Accepted solution]

Hi Sebastian,

The perception of updated AWS Lambda function code not being used is most commonly caused by the client referencing and invoking a specific function version:

You can use versions to manage the deployment of your AWS Lambda functions. For example, you can publish a new version of a function for beta testing without affecting users of the stable production version.

The system creates a new version of your Lambda function each time that you publish the function. [...]

As outlined in Using versions, clients can reference one of two ARNs for a Lambda function:

• Qualified ARN – The function ARN with the version suffix.

  arn:aws:lambda:aws-region:acct-id:function:helloworld:$LATEST

• Unqualified ARN – The function ARN without the version suffix.

  arn:aws:lambda:aws-region:acct-id:function:helloworld

The special '$LATEST' version is updated automatically each time you update the function code, whereas publishing a new version is an optional separate step if you need more control over rolling out function updates into different environments (version numbers start with 1, and there is a limit of 5 versions currently, so your sixth published version will implicitly delete version 1).

Function versions can also be indirectly referenced via an alias in turn:

You can create one or more aliases for your AWS Lambda function. A Lambda alias is like a pointer to a specific Lambda function version. Users can access the function version using the alias ARN.

Aliasing the '$LATEST' version rather than a dedicated version number means that the alias is automatically tracking the latest code update.

Now, in many contexts, clients are simply referencing an unqualified function version, which the API translates to '$LATEST'. In this scenario, any code updates should be picked up automatically.

However, in other contexts, clients default to or even require referencing specific version numbers via a qualified ARN like 'arn:aws:lambda:aws-region:acct-id:function:helloworld:42'. In this scenario, your code updates would not be picked up until you also update the qualified ARN, either by explicitly referencing a new version number, or by shifting the alias to the new version, depending on which reference type is used.

I hope this helps to identify the issue at hand - if not, please add a comment with details where/how you are referencing the Lambda function.

Cheers,
Steffen

[sebastian]

The bamboo steps for updating or creating a new lambda seem to work fine.
In aws it says "function updated" or "function created".
The problem is that the code is not the code I modified in Bitbucket.
It's an older version.
The strange thing is that if in Bamboo I add a step to create an artifact.... the code of the artifact is the latest.
So the code bamboo puts in the artifact is the latest (from bitbucket) but the code bamboo sends to aws is not the latest (this is using the "aws lambda function" task that bamboo provides).

Either the code sent to AWS is not the latest or AWS is not using the code from the lambda but the code in the S3 bucket.

[Steffen Opel _Utoolity_]

Glad to hear that you already verified the code as such being modified by creating an artifact, good idea. I propose to verify each subsequent step one by one in a similar fashion until we have identified the culprit:

1. Process wise I'm assuming you are uploading that very code as a deployment package aka ZIP archive to the S3 bucket with the Amazon S3 Object task's Upload File(s) action so that you can subsequently reference it from the AWS Lambda task's Update Function Code action, is that correct?
2. If 1 is correct, you should be able to manually download the deployment package from S3 again to verify that it's content is identical with what you uploaded, similar to how you verified this via the Bamboo artifact.
3. Now that we have established the code being readily available in S3, you can go to the Lambda console, select the function in question, and then manually update the Function code via the Upload a file from Amazon S3 feature.
4. Once 3 is done your function is guaranteed to have your code as $LATEST so that you can verify your integration actually invokes this version of the code.

As outlined in my preceding comment, I would expect the issue being rooted in an unexpected version or alias being referenced by the integration that invokes the function, which should in theory be revealed by the outcome of 4:

• If 4 works as expected, this would mean that the configuration of the Bamboo tasks do not yield the same outcome and would need to be adjusted to match what you did manually.
• If 4 does not work, this would mean that the wiring between your integration and the Lambda function does not match your expectation and would need to be adjusted to match your goals.

If neither of this applies, I would need more details to craft an appropriate reproduction scenario. If you are unable to share those publicly here, we can also continue this investigation via a private support request.

[sebastian]

Just a little update, in case you are interested in the outcome of this...

1- I was able to integrate everything.
I can do a git push and Bamboo will build the project and put it in the S3 bucket.
Then from the AWS console I go to the lambda and tell it to update the code with the S3 from the bucket.
Then I can run the lambda and everything is ok.
To do this upload to AWS I am using a task in Bamboo of type "Amazon S3 Object".

2- So that works well but there is still one thing I'm not sure how to use.
There is another type of task in Bamboo called "AWS Lambda Function".
If I replace the "Amazon S3 Object" task with a "AWS Lambda Function" task then it seems to work as well..... in the AWS console I see the "last modified" for the lambda change.
So it seems to be doing something.
But then I run the lambda and it didn't change the code.

So with "Amazon S3 Object" it works, but with "AWS Lambda Function" not. because it doesn't seem to be updating the code.

[Steffen Opel _Utoolity_]

Thanks for those details, which likely explain what's happening:

By default, we use a 'follow the API' approach for the task design, and the Lambda API expects you to upload the code via the S3 API first, so that the resulting S3 object can then be referenced from the Lambda API when Creating or updating a function.

That's exactly what you achieved manually already, and that's also what you need to do when using Tasks for AWS, i.e. you need both the S3 Object and the Lambda Function tasks, in that order:

1. Use the Amazon S3 Object task's Upload File(s) action to upload the ZIP archive to the S3 bucket, which creates or updates an S3 object
2. Use the AWS Lambda task's Update Function Code action to reference the uploaded S3 object from step 1 to update the function code

I realize that from usability perspective (UX) it would be reasonable to expect step 1 to be implied in step 2, but while we aim to provide UX shortcuts where possible, integrating the many optional parameters for uploading an S3 object right in the Lambda task seems to actually reduce the overall UX, so similar to the AWS CLI we opted to retain this API induced separation of concerns (see aws . lambda).

I hope this resolves the mystery at hand :)

[sebastian]

Yes now it works perfectly, thanks.