Is there a way to stream data directly from python request to minio bucket

Question

I am trying to make a GET request to a server to retrieve a tiff image. I then want to stream it directly to MinIO using the put_object method in the MinIO python SDK.

Answer 1

Reading documentation on MinIO about put_object, there are examples how to add a new object to the object storage server. Those examples only explain how to add a file.

This is definition of put_object function:

put_object(bucket_name, object_name, data, length, content_type='application/octet-stream', metadata=None, progress=None, part_size=510241024)

We are interested in data parameter. It states:

Any python object implementing io.RawIOBase.

RawIOBase is base class for raw binary I/O. It also defines method read.

If we were to use dir() built-in function to attempt to return a list of valid attributes for r.content, we could then check if read is there:

'read' in dir(r.content) -> return False

That's the reason why you get AttributeError: 'bytes' object has no attribute 'read'. It's because type(r.content) is bytes class.

---

You can convert r.content into class that inherits from RawIOBase. That is, using io.BytesIO class. To get size of an object in bytes, we could use io.BytesIO(r.content).getbuffer().nbytes.

So if you want to stream raw bytes of data to your bucket, convert bytes class to io.BytesIO class:

import io
import requests

r = requests.get(url_to_download, stream=True)
raw_img = io.BytesIO(r.content)
raw_img_size = raw_img.getbuffer().nbytes

Minio_client.put_object("bucket_name", "stream_test.tiff", raw_img, raw_img_size)

---

NOTE: Examples show reading binary data from file and getting its size by reading st_size attribute from stat_result which is returned by using os.stat() function.

st_size is equivalent of to io.BytesIO(r.content).getbuffer().nbytes.