Storing large blob with Objectify Appengine

Question

I have this class which I want to persist using Objectify, this class will represent a data larger than 1MB so there's a List of Blob objects which represents a fragment of the byte array stored that is less than 1MB in size:

@Entity
public class BigBlob {

    @Id
    private Long id;
    public static final int FRAGMENT_LIMIT = 777 * 1024;
    @Serialized
    private List<Blob> fragments = new ArrayList<Blob>();

    ...

}

Yet, the the "fragments" is @Serialized, which will render the size of this BigBlob class/object larger than 1MB.

Causing this error:

com.google.apphosting.api.ApiProxy$RequestTooLargeException: The request to API call datastore_v3.Put() was too large.

If I use @Embedded annotation I get this error:

Cannot place array or collection properties inside @Embedded arrays or collections

How do I make sure that the "fragments" are stored as a separate entity?

BTW, I already have the byte chunking logic that chops the whole byte array and put the fragments into a List of Blob so this question does not pertain as to how to chop bytes.

Mostly what I want to know is more on the persisting side.

Answer 1

Rick's answer is really the best - store blobs in the blobstore, especially if you are new to GAE and having conceptual issues with the datastore.

On the other hand, there are some good reasons to use split entities for storing blobs, especially if you are storing data that is close to the 1M edge. You wouldn't want to do this with 100MB blobs, but 2MB blobs can make sense.

First of all, you don't want serialized or embedded. Those are simply ways to structure data inside a single entity.

Also, there's no magic annotation that lets you split blobs across entities. You have to do it all by hand. You don't need to actually create a 'master' or root entity; just create all the entity fragments with a parent defined by an id (but no actual entity) and use an ancestor() query to fetch all the pieces.

Answer 2

You should store it in the Blobstore and just save the Blobkey in Objectify. Objectify works on top of the datastore, not the blobstore.