How to keep files in memory - transferring files between servers without storing them locally

Question

I am in the process of writing a simple deployment tool which needs to take tar files from s3, extract them and then upload them to our staging server. I would like to do th

Answer 1

This is mentioned in the docs for Download:

The w io.WriterAt can be satisfied by an os.File to do multipart concurrent downloads, or in memory []byte wrapper using aws.WriteAtBuffer.

So use an aws.WriteAtBuffer instead of an *os.File:

buf := new(aws.WriteAtBuffer)

numBytes, err := downloader.Download(buf, &s3.GetObjectInput{
    Bucket: aws.String(s3bucket),
    Key:    aws.String(item),
})

tr := tar.NewReader(bytes.NewReader(buf.Bytes()))
// ...