Emrfs file sync with s3 not working

Question

After running a spark job on an Amazon EMR cluster, I deleted the output files directly from s3 and tried to rerun the job again. I received the following error upon trying to w

Answer 1

I arrived at this page because I was getting the error "key is marked as directory in metadata but is file in s3" and was very puzzled. I think what happened is that I accidentally created both a file and directory by the same name. By deleting the file it solved my issue:

aws s3 rm s3://bucket/directory_name_without_trailing_slash