How to delete or purge old files on S3?

Question

Are there existing solutions to delete any files older than x days?

Answer 1

Amazon has introduced object expiration recently.

Amazon S3 Announces Object Expiration

Amazon S3 announced a new feature, Object Expiration that allows you to schedule the deletion of your objects after a pre-defined time period. Using Object Expiration to schedule periodic removal of objects eliminates the need for you to identify objects for deletion and submit delete requests to Amazon S3.

You can define Object Expiration rules for a set of objects in your bucket. Each Object Expiration rule allows you to specify a prefix and an expiration period in days. The prefix field (e.g. logs/) identifies the object(s) subject to the expiration rule, and the expiration period specifies the number of days from creation date (i.e. age) after which object(s) should be removed. Once the objects are past their expiration date, they will be queued for deletion. You will not be billed for storage for objects on or after their expiration date.

Answer 2

You can use AWS S3 Life cycle rules to expire the files and delete them. All you have to do is select the bucket, click on "Add lifecycle rules" button and configure it and AWS will take care of them for you.

You can refer the below blog post from Joe for step by step instructions. It's quite simple actually:

https://www.joe0.com/2017/05/24/amazon-s3-how-to-delete-files-older-than-x-days/

Hope it helps!

Answer 3

Here is how to implement it using a CloudFormation template:

  JenkinsArtifactsBucket:
    Type: "AWS::S3::Bucket"
    Properties:
      BucketName: !Sub "jenkins-artifacts"
      LifecycleConfiguration:
        Rules:
          - Id: "remove-old-artifacts"
            ExpirationInDays: 3
            NoncurrentVersionExpirationInDays: 3
            Status: Enabled

This creates a lifecycle rule as explained by @Ravi Bhatt

Read more on that: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-s3-bucket-lifecycleconfig-rule.html

How object lifecycle management works: https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lifecycle-mgmt.html

Answer 4

You can use the following Powershell script to delete object expired after x days.

[CmdletBinding()]
Param(  
  [Parameter(Mandatory=$True)]
  [string]$BUCKET_NAME,             #Name of the Bucket

  [Parameter(Mandatory=$True)]
  [string]$OBJ_PATH,                #Key prefix of s3 object (directory path)

  [Parameter(Mandatory=$True)]
  [string]$EXPIRY_DAYS             #Number of days to expire
)

$CURRENT_DATE = Get-Date
$OBJECTS = Get-S3Object $BUCKET_NAME -KeyPrefix $OBJ_PATH
Foreach($OBJ in $OBJECTS){
    IF($OBJ.key -ne $OBJ_PATH){
        IF(($CURRENT_DATE - $OBJ.LastModified).Days -le $EXPIRY_DAYS){
            Write-Host "Deleting Object= " $OBJ.key
            Remove-S3Object -BucketName $BUCKET_NAME -Key $OBJ.Key -Force
        }
    }
}

Answer 5

Here is some info on how to do it...

http://docs.amazonwebservices.com/AmazonS3/latest/dev/ObjectExpiration.html

Hope this helps.