Aws S3 Filter by Tags. Search by tags

Question

We have our bucket with new Aws SDK API on AWS S3. We uploaded and tagged lots of files and folders with tags.

How can we filter on key-value tag, or only one of them?

Answer 1

AFAIK - Resource Groups don't support tags on an S3 Object level only on a bucket level.

Source: https://aws.amazon.com/blogs/aws/new-aws-resource-tagging-api/ (scroll down the page to the table).

Answer 2

There's no way to filter/search by tags. But you can implement this yourself using S3.

You can create a special prefix in a bucket, e.g. /tags/. Then for each actual object you add and want to assign a tag (e.g. Department=67), you add a new object in /tags/, e.g: /tags/XXXXXXXXX_YYYYYYYYY_ZZZZZZZZZ, where

XXXXXXXXX = hash('Department')
YYYYYYYYY = hash('67')
ZZZZZZZZZ = actualObjectKey

Then when you want to get all objects that have a particular tag assigned (e.g. Department), you have to execute the ListObjectsV2 S3 API for prefix /tags/XXXXXXXXX_. If you want objects that have particular tag value (e.g. Department=67), you have to execute the ListObjectsV2 S3 API for prefix /tags/XXXXXXXXX_YYYYYYYYY_

https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html

It's not that fast but still does the job.

Obvious downside is that you have to remove the tags yourself. For example, you can do all of this above with a S3 triggers and lambda.

Answer 3

You should be able to query tags and values that you added using resource-groups/query resource:

https://${region}.console.aws.amazon.com/resource-groups/resources

Answer 4

I also hoped that AWS will eventually support "search files by tags" because that would open up possibilities like e.g. having a photo storage with the names, descriptions, location stored in tags so I wouldn't need a separate database.

But, apparently AWS explicitly is not supporting this, and will probably never do so. Quoting from their storage service white paper:

Amazon S3 doesn’t suit all storage situations. [...] some storage needs for which you should consider other AWS storage options [...]

Amazon S3 doesn’t offer query capabilities to retrieve specific objects. When you use Amazon S3 you need to know the exact bucket name and key for the files you want to retrieve from the service. Amazon S3 can’t be used as a database or search engine by itself.

Instead, you can pair Amazon S3 with Amazon DynamoDB, Amazon CloudSearch, or Amazon Relational Database Service (Amazon RDS) to index and query metadata about Amazon S3 buckets and objects.

AWS suggests using DynamoDB, RDS or CloudSearch instead.

Answer 5

There seems to be one way to achieve what you're looking for, although it's not ideal, or particularly user-friendly.

The AWS S3 tagging documentation says that you can grant accounts permissions for objects with a given tag. If you created a new account with the right permissions then you could probably get the filtered list.

Not particularly useful on an ongoing basis, though.

Answer 6

AWS now supports tagging of S3 images.

They have APIs to add/remove tags.

Amazon S3 Select, Amazon Athena can be used to search for S3 resources with TAGS.

Currently the max number of tags per resource is 10 (thanks Kyle Bridenstine for pointing out the correct number).