Upload to s3 with curl using pre-signed URL (getting 403)

Question

I\'m using curl to call into a Java ReST API to retrieve a URL. Java then generates a pre-signed URL for S3 upload using my S3 credentials, and returns that in the ReST reply. C

Answer 1

I've been able to generate a pre-signed URL via C# and upload it thereafter via curl as expected. Given my tests I suspect you are indeed not using curl correctly - I've been able to upload a file like so:

curl -v --upload-file ${fileName} ${location}

The parameter -v dumps both request and response headers (as well as the SSL handshake) for debugging and illustration purposes:

> PUT [...] HTTP/1.1
> User-Agent: curl/7.21.0 [...]
> Host: [...]
> Accept: */*
> Content-Length: 12
> Expect: 100-continue

Please note, that --upload-file (or -T) facilitates PUTas expected, but adds more headers as appropriate, yielding a proper response in return:

< HTTP/1.1 100 Continue
< HTTP/1.1 200 OK
< x-amz-id-2: [...]
< x-amz-request-id:  [...]
< Date: Tue, 31 Jan 2012 18:34:56 GMT
< ETag: "253801c0d260f076b0d5db5b62c54824"
< Content-Length: 0
< Server: AmazonS3

Answer 2

when doing this with curl, you need to place the url in single quotes or else half of the query string gets chopped off (the part with the key/signature).

Answer 3

The way to generate the URL:

private static URL generateRUL(String objectKey, String ACCESS_KEY, String SECRET_KEY, String BUCKET_NAME) {
    AmazonS3 s3Client = new AmazonS3Client(new BasicAWSCredentials(ACCESS_KEY, SECRET_KEY));
    URL url = null;

    try {
        GeneratePresignedUrlRequest request  = new GeneratePresignedUrlRequest(BUCKET_NAME, objectKey);
        request.setMethod(com.amazonaws.HttpMethod.PUT);
        request.setExpiration(new Date( System.currentTimeMillis() + (60 * 60 * 1000)));

        // Very important ! It won't work without adding this! 
        // And request.addRequestParameter("Content-Type", "application/octet-stream") won't work neither
        request.setContentType("application/octet-stream");

        url = s3Client.generatePresignedUrl(request ); 
    } catch (AmazonServiceException exception) { 
    } catch (AmazonClientException ace) { }

    return url;
}

The way to upload the file:

public int upload(byte[] fileBytes, URL url) {
    HttpURLConnection connection = (HttpURLConnection) url.openConnection();
    connection.setDoOutput(true);
    connection.setRequestMethod("PUT");
    connection.setRequestProperty("Content-Type", "application/octet-stream"); // Very important ! It won't work without adding this!
    OutputStream output = connection.getOutputStream();

    InputStream input = new ByteArrayInputStream(fileBytes);
    byte[] buffer = new byte[4096];
    int length;
    while ((length = input.read(buffer)) > 0) {
        output.write(buffer, 0, length);
    }
    output.flush();

    return connection.getResponseCode();
}

Answer 4

Despite the fact that GeneratePresignedUrlRequest accepts an http method argument (and has a setMethod function), it appears to be unusable for anything but GET.

http://wiki.nercomp.org/wiki/images/0/05/AmazonWebServices.pdf states "The practice of signing a request and giving it to a third-party for execution is suitable only for simple object GET requests." Perhaps setting another method can be used for something, but apparently not this.

So, instead, I had to follow the instructions here:

http://aws.amazon.com/articles/1434?_encoding=UTF8&jiveRedirect=1

This is more complex, because the client is required to post a complete form, rather than just using a URL, and also means all that post info has to be communicated to the client separately, but it does seem to work.