Why does file uploaded to S3 have content type application/octet-stream unless i name the file .html

Question

Even though I set content type to text/html it ends up as application/octet-stream on S3.

ByteArrayInputStream contentsAsStream = new ByteArrayInputStream(contentAsBytes);
ObjectMetadata md = new ObjectMetadata();
md.setContentLength(contentAsBytes.length);
md.setContentType("text/html");
s3.putObject(new PutObjectRequest(ARTIST_BUCKET_NAME, artistId, contentsAsStream, md));

If however I name the file so that it ends up with .html

s3.putObject(new PutObjectRequest(ARTIST_BUCKET_NAME, artistId + ".html", contentsAsStream, md));

then it works.

Is my md object just being ignored ? How can I get round this programmatically as over time I need to upload thousands of files so cannot just go into S3 UI and manually fix the contentType.

Answer 1

You must be doing something else in your code. I just tried your code example using the 1.9.6 S3 SDK and the file gets the "text/html" content type.

Here's the exact (Groovy) code:

class S3Test {
    static void main(String[] args) {

        def s3 = new AmazonS3Client()

        def random = new Random()
        def bucketName = "raniz-playground"
        def keyName = "content-type-test"

        byte[] contentAsBytes = new byte[1024]
        random.nextBytes(contentAsBytes)

        ByteArrayInputStream contentsAsStream = new ByteArrayInputStream(contentAsBytes);
        ObjectMetadata md = new ObjectMetadata();
        md.setContentLength(contentAsBytes.length);
        md.setContentType("text/html");
        s3.putObject(new PutObjectRequest(bucketName, keyName, contentsAsStream, md))

        def object = s3.getObject(bucketName, keyName)
        println(object.objectMetadata.contentType)
        object.close()
    }
}

The program prints

text/html

And the S3 metadata says the same:

Here are the communication sent over the net (courtesy of Apache HTTP Commons debug logging):

>> PUT /content-type-test HTTP/1.1
>> Host: raniz-playground.s3.amazonaws.com
>> Authorization: AWS <nope>
>> User-Agent: aws-sdk-java/1.9.6 Linux/3.2.0-84-generic Java_HotSpot(TM)_64-Bit_Server_VM/25.45-b02/1.8.0_45
>> Date: Fri, 12 Jun 2015 02:11:16 GMT
>> Content-Type: text/html
>> Content-Length: 1024
>> Connection: Keep-Alive
>> Expect: 100-continue
<< HTTP/1.1 200 OK
<< x-amz-id-2: mOsmhYGkW+SxipF6S2+CnmiqOhwJ62WfWUkmZk4zU3rzkWCEH9P/bT1hUz27apmO
<< x-amz-request-id: 8706AE3BE8597644
<< Date: Fri, 12 Jun 2015 02:11:23 GMT
<< ETag: "6c53debeb28f1d12f7ad388b27c9036d"
<< Content-Length: 0
<< Server: AmazonS3

>> GET /content-type-test HTTP/1.1
>> Host: raniz-playground.s3.amazonaws.com
>> Authorization: AWS <nope>
>> User-Agent: aws-sdk-java/1.9.6 Linux/3.2.0-84-generic Java_HotSpot(TM)_64-Bit_Server_VM/25.45-b02/1.8.0_45
>> Date: Fri, 12 Jun 2015 02:11:23 GMT
>> Content-Type: application/x-www-form-urlencoded; charset=utf-8
>> Connection: Keep-Alive
<< HTTP/1.1 200 OK
<< x-amz-id-2: 9U1CQ8yIYBKYyadKi4syaAsr+7BV76Q+5UAGj2w1zDiPC2qZN0NzUCQNv6pWGu7n
<< x-amz-request-id: 6777433366DB6436
<< Date: Fri, 12 Jun 2015 02:11:24 GMT
<< Last-Modified: Fri, 12 Jun 2015 02:11:23 GMT
<< ETag: "6c53debeb28f1d12f7ad388b27c9036d"
<< Accept-Ranges: bytes
<< Content-Type: text/html
<< Content-Length: 1024
<< Server: AmazonS3

And this is also the behaviour that looking at the source code shows us - if you set the content type the SDK won't override it.

Answer 2

Because you have to set content type at the end just before sending, using the putObject method;

        ObjectMetadata md = new ObjectMetadata();

        InputStream myInputStream = new ByteArrayInputStream(bFile); 
        md.setContentLength(bFile.length);
        md.setContentType("text/html");
        md.setContentEncoding("UTF-8");

        s3client.putObject(new PutObjectRequest(bucketName, keyName, myInputStream, md));

And after upload, content type is set as "text/html"

Here is a working dummy code, check that out, I've just tried and it's working;

public class TestAWS {

    //TEST
    private static String bucketName = "whateverBucket";

    public static void main(String[] args) throws Exception {
        BasicAWSCredentials awsCreds = new BasicAWSCredentials("whatever", "whatever");

        AmazonS3 s3client = new AmazonS3Client(awsCreds);
        try
        {
            String uploadFileName = "D:\\try.txt";
            String keyName = "newFile.txt";

            System.out.println("Uploading a new object to S3 from a file\n");
            File file = new File(uploadFileName);

            //bFile will be the placeholder of file bytes
            byte[] bFile = new byte[(int) file.length()];
            FileInputStream fileInputStream=null;

            //convert file into array of bytes  
            fileInputStream = new FileInputStream(file);
            fileInputStream.read(bFile);
            fileInputStream.close();

            ObjectMetadata md = new ObjectMetadata();

            InputStream myInputStream = new ByteArrayInputStream(bFile); 
            md.setContentLength(bFile.length);
            md.setContentType("text/html");
            md.setContentEncoding("UTF-8");

            s3client.putObject(new PutObjectRequest(bucketName, keyName, myInputStream, md));
        } catch (AmazonServiceException ase)
        {
            System.out.println("Caught an AmazonServiceException, which "
                    + "means your request made it "
                    + "to Amazon S3, but was rejected with an error response"
                    + " for some reason.");
            System.out.println("Error Message:    " + ase.getMessage());
            System.out.println("HTTP Status Code: " + ase.getStatusCode());
            System.out.println("AWS Error Code:   " + ase.getErrorCode());
            System.out.println("Error Type:       " + ase.getErrorType());
            System.out.println("Request ID:       " + ase.getRequestId());
        } catch (AmazonClientException ace)
        {
            System.out.println("Caught an AmazonClientException, which "
                    + "means the client encountered "
                    + "an internal error while trying to "
                    + "communicate with S3, "
                    + "such as not being able to access the network.");
            System.out.println("Error Message: " + ace.getMessage());
        }

    }

}

Hope that it helps.

Answer 3

It seems that

When uploading files, the AWS S3 Java client will attempt to determine the correct content type if one hasn't been set yet. Users are responsible for ensuring a suitable content type is set when uploading streams. If no content type is provided and cannot be determined by the filename, the default content type, "application/octet-stream", will be used.

Giving the file a .html extension provides a way to set the correct type.

According to the examples I've been looking at, the code you show should be doing what you want to do. :/

Answer 4

Do you have any Override on the default mime content on your S3 account? Look at this link to see how to check it: How to override default Content Types.

Anyway, it looks like your S3 client fails to determine the correct mime-type by the content of the file, so it relies on the extension. octet-stream is the widely used default content mime type when a browser/servlet can't determine the mimetype: Is there any default mime type?