How to save a file downloaded from S3 with Rusoto to my hard drive?

Question

I am trying to download a file from a bucket with Rusoto and I am getting the file content:

fn get_object(client: &TestClient, bucket: &str, filename         


        

Answer 1

You're almost there. Your code will put the object in body, which is a Vec<u8>.

To write the contents of body to a file:

use std::io::Write;
use std::fs::File;

let mut file = File::create("/path/to/my-object").expect("create failed");
file.write_all(&body).expect("failed to write body");

Answer 2

Rusoto now uses the standard library futures and no longer offers the sync method, so the previous answer is no longer valid.

Reading to memory

use futures::stream::TryStreamExt;
use rusoto_core::Region;
use rusoto_s3::{GetObjectRequest, S3Client, S3};

type Error = Box<dyn std::error::Error>;
type Result<T, E = Error> = std::result::Result<T, E>;

const BUCKET_NAME: &str = "my very own bucket name";

#[tokio::main]
async fn main() -> Result<()> {
    let client = S3Client::new(Region::UsEast2);

    let mut object = client
        .get_object(GetObjectRequest {
            bucket: BUCKET_NAME.into(),
            ..Default::default()
        })
        .await?;

    let body = object.body.take().expect("The object has no body");

    let body = body.map_ok(|b| b.to_vec()).try_concat().await?;
    println!("body length: {}", body.len());

    Ok(())
}

AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY need to be specified. I chose to set environment variables outside of the code.

Streaming to a file

use rusoto_core::Region;
use rusoto_s3::{GetObjectRequest, S3Client, S3};
use tokio::{fs::File, io};

type Error = Box<dyn std::error::Error>;
type Result<T, E = Error> = std::result::Result<T, E>;

const BUCKET_NAME: &str = "my very own bucket name";

#[tokio::main]
async fn main() -> Result<()> {
    let client = S3Client::new(Region::UsEast2);

    let mut object = client
        .get_object(GetObjectRequest {
            bucket: BUCKET_NAME.into(),
            ..Default::default()
        })
        .await?;

    let body = object.body.take().expect("The object has no body");

    let mut body = body.into_async_read();
    let mut file = File::create("/tmp/a-place-to-write").await?;
    io::copy(&mut body, &mut file).await?;

    Ok(())
}

While ByteStream has an alluring into_blocking_read method, I do not recommend using it. If you attempt to use it inside of an async context, you get a panic because it starts a nested Tokio executor. If you use it outside of an async context, it will truncate the data unless you take great care to have the async runtime around but not to be within it.

See also:

• Writing a chunk stream to a file asynchronously using hyper
• How do I write a futures::Stream to disk without storing it entirely in memory first?

Dependency versions

[dependencies]
rusoto_s3 = "0.43.0"
rusoto_core = "0.43.0"
tokio = { version = "0.2.21", features = ["macros"] }
futures = "0.3.5"