How to do a Pre-signed POST upload to AWS S3 in Go?

Question

I would like to do a pre-signed POST to upload files to an AWS S3 bucket - how would this be done in Go?

Please note that this is not the same as Pre-signed upload w

Answer 1

So in order to help others I will answer the question myself and provide some code to help others who might have the same problem.

Example web app for Google App Engine rendering a pre-signed POST form can be found here.

And a small library I created doing the pre-signed POST in Go.

In short, doing a presigned POST to a public-read Amazon S3 bucket you need to:

1. Configure the S3 bucket to only allow public download.

Example bucket policy that allow only public read.

{
    "Version": "2012-10-17",
    "Id": "akjsdhakshfjlashdf",
    "Statement": [
        {
            "Sid": "kjahsdkajhsdkjasda",
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            },
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::BUCKETNAMEHERE/*"
        }
    ]
}

2. Create a policy for the HTTP POST that allows the upload.

AWS S3 docs

Example POST policy template with expiration to upload a specific key, into a specific bucket and allow public-read access.

{ "expiration": "%s",
    "conditions": [
        {"bucket": "%s"},
        ["starts-with", "$key", "%s"],
        {"acl": "public-read"},

        {"x-amz-credential": "%s"},
        {"x-amz-algorithm": "AWS4-HMAC-SHA256"},
        {"x-amz-date": "%s" }
    ]
}

3. Generate and sign the policy using the S3 bucket owner's credentials.

AWS docs

• Fill in the correct values for expiration, bucket, key, credentials and date.
• base64 encode the policy.
• HMAC-SHA256 the policy to get a signature.
• hex encode the signature.

4. Construct and POST the multipart form data

AWS S3 docs

Now either you would generate an HTML form and automatically get the correct multipart form data request like described in the above link.

I wanted to do this by hand in Go so here's how to do that.

Either way you need to provide all the parts that are specified in the POST policy you created in steps 2 and 3. You can also not have additional fields in the request except for the mandatory ones (not in the policy).

The order of the fields is also specified and all of them are multipart fields in the HTTP POST request.

func Upload(url string, fields Fields) error {
    var b bytes.Buffer
    w := multipart.NewWriter(&b)
    for _, f := range fields {
            fw, err := w.CreateFormField(f.Key)
            if err != nil {
                    return err
            }
            if _, err := fw.Write([]byte(f.Value)); err != nil {
                    return err
            }
    }
    w.Close()

    req, err := http.NewRequest("POST", url, &b)
    if err != nil {
            return err
    }
    req.Header.Set("Content-Type", w.FormDataContentType())

    client := &http.Client{}
    res, err := client.Do(req)
    if err != nil {
            return err
    }
    if res.StatusCode != http.StatusOK {
            err = fmt.Errorf("bad status: %s", res.Status)
    }
    return nil
}

Answer 2

Here is an alternative approach from https://github.com/minio/minio-go that you might like for a full programmatic way of generating presigned post policy.

package main

import (
    "fmt"
    "log"
    "time"

    "github.com/minio/minio-go"
)

func main() {
    policy := minio.NewPostPolicy()
    policy.SetKey("myobject")
    policy.SetBucket("mybucket")
    policy.SetExpires(time.Now().UTC().AddDate(0, 0, 10)) // expires in 10 days
    config := minio.Config{
        AccessKeyID:     "YOUR-ACCESS-KEY-HERE",
        SecretAccessKey: "YOUR-PASSWORD-HERE",
        Endpoint:        "https://s3.amazonaws.com",
    }
    s3Client, err := minio.New(config)
    if err != nil {
        log.Fatalln(err)
    }
    m, err := s3Client.PresignedPostPolicy(policy)
    if err != nil {
        fmt.Println(err)
        return
    }
    fmt.Printf("curl ")
    for k, v := range m {
        fmt.Printf("-F %s=%s ", k, v)
    }
    fmt.Printf("-F file=@/etc/bashrc ")
    fmt.Printf(config.Endpoint + "/mybucket\n")
}

---

Step 1:

    policy := minio.NewPostPolicy()
    policy.SetKey("myobject")
    policy.SetBucket("mybucket")
    policy.SetExpires(time.Now().UTC().AddDate(0, 0, 10)) // expires in 10 days

Instantiate a new policy structure, this policy structure implements following methods.

func NewPostPolicy() *PostPolicy
func (p *PostPolicy) SetBucket(bucket string) error
func (p *PostPolicy) SetContentLength(min, max int) error
func (p *PostPolicy) SetContentType(contentType string) error
func (p *PostPolicy) SetExpires(t time.Time) error
func (p *PostPolicy) SetKey(key string) error
func (p *PostPolicy) SetKeyStartsWith(keyStartsWith string) error
func (p PostPolicy) String() string

---

Step 2:

    m, err := s3Client.PresignedPostPolicy(policy)
    if err != nil {
        fmt.Println(err)
        return
    }

Now PresignedPostPolicy() takes the PostPolicy structure and returns back a map of "key/values" which can be used in your HTML form or curl command to upload data to s3.

Answer 3

At a glance it looks like POST works with an attached policy and signature -- designed for browser based uploads. See the AWS Docs for details.

Specifically, you need to generate a policy and sign that -- then include them in the HTML form, and thereby the POST request -- along with the rest of the required information. Or let the browser do it for you.

In the case of HTML form POST uploads you are only signing the policy string. The final URL to be posted to can vary based on the form contents: https://bucket.s3.amazonaws.com/<depends-on-form-content>. So you can't presign that URL because you don't know what it is.

This is different than a signed URL to which you PUT a file. You can sign that because you know the full URL: https://bucket.s3.amazonaws.com/known-key

You could build a POST request with the appropriate policy and parameters and upload via POST that way. However, you would need to know the contents of the form to know the URL beforehand. In which case you may as well use a presigned PUT URL.

At least that is how it appears at a glance...