AWS secrets manager, 'A previous rotation isn’t complete' when rotating secrets

Question

I\'ve created a secret and updated it to have a lambda rotation function

My secret looks like

aws secretsmanager list-secret-version-ids --secret-id envi         


        

Answer 1

The problem was that the lambda function was failing and the retry was happening w/o my control (there is currently no way to limit the retries on a lambda function).

Answer 2

For anyone still having this issue what you can try doing is clearing the pending version and reattempting the rotation

For example with a secret with secret id thefrog, call

aws secretsmanager get-secret-value \
    --secret-id thefrog \
    --version-stage AWSPENDING

to get the version id of the version with the pending label. The result would look like

{                                                                      
    "CreatedDate": 1541540242.561,                         
    "Name": "thefrog",                
    "VersionStages": [                               
        "AWSPENDING"                                        
    ],                                                    
    "SecretString": "TOP-SECRET",                                                    
    "ARN": "arn:aws:secretsmanager:xxxxxxxxxxxxxxxxxxxxxxxxxxxx",
    "VersionId": "2a27cecb-23c7-4320-b168-78661c24612f"   
} 

Then call

aws secretsmanager update-secret-version-stage \
    --secret-id thefrog \
    --version-stage AWSPENDING \
    --remove-from-version-id 2a27cecb-23c7-4320-b168-78661c24612f

to remove the version of secret that has the pending label.

From here you can retry the rotation

Answer 3

I had a similar problem. For my documentdb I have a cloudformation template which got the following template content:

  MyDocumentDbSecret:
    Type: AWS::SecretsManager::Secret
    Properties:
      Name: "/secrets/documentdb/root"
      Description: 'DocDB root secret'
      GenerateSecretString:
        SecretStringTemplate: !Sub '{"username": "${DefaultDocDbUser}"}'
        GenerateStringKey: "password"
        PasswordLength: 16
        ExcludeCharacters: '"@/\'

But with this cloudformation template I always get a connection timeout in the lambda function (Which try to change my password for the user).

But when I change my cloudformation template to this with ssl = true in the SecretStringTemplate attribute:

  DocDBClusterRotationSecret:
    Type: AWS::SecretsManager::Secret
    Properties:
      Name: "/secrets/documentdb/root"
      Description: 'DocDB root secret'
      GenerateSecretString:
        SecretStringTemplate: !Sub '{"username": "${DefaultDocDbUser}", "ssl": "true"}'
        GenerateStringKey: "password"
        PasswordLength: 16
        ExcludeCharacters: '"@/\'

then it works properly. For my cloudformation type: AWS::SecretsManager::SecretTargetAttachment does not provide my secret with the attributes ssl=true, therefore I need to add it manually in my cloudformation template. Now it works perfectly with no errors.

My secret string looks like this nowadays:

{
 "password": "My PW",
 "engine": "mongo",
 "port": 27017,
 "host": "My Host",
 "ssl": "true",
 "username": "My User"
}

Answer 4

Just a note for people in future who might get the same error...

If you are using the AWS Secrets Manager to rotate an Amazon RDS password, the Secrets Manager will automatically create a Lambda function. This function requires:

• Access to the Internet (to call the Secrets Manager) OR VPC endpoint for Secrets Manager service in subnet/subnets associated with the lambda function
• Access to the RDS instance (to login and change the password)

As such, the following combinations work:

• Publicly accessible database (bad for security) with a Lambda function that is not attached to a VPC, OR
• The Lambda function in a private subnet with a NAT Gateway in the public subnet (so the Lambda function can access the Internet) OR an Elastic IP Address attached to the Lambda function's ENI

Also, the Security Group attached to the database needs to permit inbound access from the Lambda function. By default, the Lambda function is assigned the same security group as used by the database, so either:

• Edit the database security group to permit inbound connections from itself (that is, from Lambda to the database via the same security group), OR
• Change the security group that is used by the Lambda function to one that is currently permitted to access the database security group

Answer 5

For anyone who thinks that the link at https://forums.aws.amazon.com/thread.jspa?threadID=280093&tstart=0 doesn't apply, make sure to check the output of both aws secretsmanager list-secret-version-ids and aws secretsmanager list-secrets to make sure they are in sync with each other. I just had one secret I could not rotate, kept getting the "A previous rotation isn’t complete. That rotation will be reattempted" error message. I had a support case with AWS open on it, and while I was waiting on hold to speak to a support rep, I decided to check the output of list-secrets, and lo and behold I found an AWSPENDING label on the secret I could not rotate (that label did NOT show up on the output of list-secret-version-ids for that secret). Once I cleared that label, I could then successfully rotate the secret I was having problems with.