How to verify the Secrets Manager credential rotation is successful

问题

I enabled the rotation and checked lambda funcion logs, there is no error, everything looks fine. But I retrieved the secret in Secrets Manager console, it's still the old password. I don't know what's going on.

In AWS CLI:

aws secretsmanager list-secret-version-ids --secret-id xxx

Output:
{
    "Versions": [
        {
            "VersionId": "4********2f",
            "VersionStages": [
                "AWSPREVIOUS"
            ],
            "LastAccessedDate": 1580428800.0,
            "CreatedDate": 1580484370.988
        },
        {
            "VersionId": "9********69",
            "VersionStages": [
                "AWSPENDING"
            ],
            "LastAccessedDate": 1580428800.0,
            "CreatedDate": 1580484483.303
        },
        {
            "VersionId": "b5*********c5",
            "VersionStages": [
                "AWSCURRENT"
            ],
            "LastAccessedDate": 1580428800.0,
            "CreatedDate": 1580484482.179
        }
    ],
    "ARN": "arn:aws:secretsmanager:us-east-1:xxxxxxxxxxxxxx",
    "Name": "xxxxxxxxxxxxxx"
}

I see a AWSPENDINGabove, what does it mean? I notice the version ID changed, does it mean the rotation is successful? Many thanks.

Additional info: In cloudwatch logs:

[INFO]  2020-02-01T19:30:47.741Z    3d6XXX81    Found credentials in environment variables.
[INFO]  2020-02-01T19:30:48.416Z    3dXXX81 createSecret: Successfully retrieved secret for arn:XXX
END RequestId: 3dXXX81
REPORT RequestId: 3dXXX81
START RequestId: adXXX52 Version: $LATEST
END RequestId: adXXX52
REPORT RequestId: adXXX52
2020-02-01T19:30:51.485Z adXXX52 Task timed out after 3.00 seconds

I started a new rotation manually, and it gave me a new error:

[ERROR] ValueError: Unable to log into database with previous, current, or pending secret of secret xxxxxx

When I tried to access to MySQL db I couldn't login by using the original password, does this mean the password has been rotated to a random value and it's 'PENDING' now? Not sure what to do now, is there a way to reset the password?

Retriggered another rotation, it gave me unable to log into db and [ERROR] ClientError: An error occurred (AccessDeniedException) when calling the GetSecretValue operation:

I really don't understand why I kept getting new errors, I'm pretty sure I added 'GetSecretValue' in the Lambda exec role, anyone knows what the hell is going on?

来源：https://stackoverflow.com/questions/60007180/how-to-verify-the-secrets-manager-credential-rotation-is-successful

标签

amazon-web-services

aws-lambda

credentials

aws-kms

aws-secrets-manager