Cloudfront/S3 - Limit the content that users can access for static site

Question

I have a static site hosted on a private S3 bucket with cloudfront configured (+ domain/routing). One problem I am noticing is that any user could access all the files in the S3