How to avoid from always loading cached app data from Google Drive

Question

Currently, I'm using Google Drive Android API, to store my Android app data, to Google Drive App Folder.

This is what I'm doing when saving my application data

1. Generate a checksum for the current local zip file.
2. Search in Google Drive App Folder, to see whether there is an existing App Folder zip file.
3. If there is, overwrite the content of existing App Folder zip file, with current local zip files. Also, we will rename existing App Folder zip filename, with the latest checksum.
4. If there isn't existing App Folder zip file, generate a new App Folder zip file, with local zip file's content. We will use the latest checksum as App Folder zip filename.

Here's the code which performs the above-mentioned operations.

Generate new App Folder zip file, or update existing App Folder zip file

public static boolean saveToGoogleDrive(GoogleApiClient googleApiClient, File file, HandleStatusable h, PublishProgressable p) {
    // Should we new or replace?

    GoogleCloudFile googleCloudFile = searchFromGoogleDrive(googleApiClient, h, p);

    try {
        p.publishProgress(JStockApplication.instance().getString(R.string.uploading));

        final long checksum = org.yccheok.jstock.gui.Utils.getChecksum(file);
        final long date = new Date().getTime();
        final int version = org.yccheok.jstock.gui.Utils.getCloudFileVersionID();
        final String title = getGoogleDriveTitle(checksum, date, version);

        DriveContents driveContents;
        DriveFile driveFile = null;

        if (googleCloudFile == null) {
            DriveApi.DriveContentsResult driveContentsResult = Drive.DriveApi.newDriveContents(googleApiClient).await();

            if (driveContentsResult == null) {
                return false;
            }

            Status status = driveContentsResult.getStatus();
            if (!status.isSuccess()) {
                h.handleStatus(status);
                return false;
            }

            driveContents = driveContentsResult.getDriveContents();

        } else {
            driveFile = googleCloudFile.metadata.getDriveId().asDriveFile();
            DriveApi.DriveContentsResult driveContentsResult = driveFile.open(googleApiClient, DriveFile.MODE_WRITE_ONLY, null).await();

            if (driveContentsResult == null) {
                return false;
            }

            Status status = driveContentsResult.getStatus();
            if (!status.isSuccess()) {
                h.handleStatus(status);
                return false;
            }

            driveContents = driveContentsResult.getDriveContents();
        }

        OutputStream outputStream = driveContents.getOutputStream();
        InputStream inputStream = null;

        byte[] buf = new byte[8192];

        try {
            inputStream = new FileInputStream(file);
            int c;

            while ((c = inputStream.read(buf, 0, buf.length)) > 0) {
                outputStream.write(buf, 0, c);
            }

        } catch (IOException e) {
            Log.e(TAG, "", e);
            return false;
        } finally {
            org.yccheok.jstock.file.Utils.close(outputStream);
            org.yccheok.jstock.file.Utils.close(inputStream);
        }

        if (googleCloudFile == null) {
            // Create the metadata for the new file including title and MIME
            // type.
            MetadataChangeSet metadataChangeSet = new MetadataChangeSet.Builder()
                    .setTitle(title)
                    .setMimeType("application/zip").build();

            DriveFolder driveFolder = Drive.DriveApi.getAppFolder(googleApiClient);
            DriveFolder.DriveFileResult driveFileResult = driveFolder.createFile(googleApiClient, metadataChangeSet, driveContents).await();

            if (driveFileResult == null) {
                return false;
            }

            Status status = driveFileResult.getStatus();
            if (!status.isSuccess()) {
                h.handleStatus(status);
                return false;
            }
        } else {
            MetadataChangeSet metadataChangeSet = new MetadataChangeSet.Builder()
                    .setTitle(title).build();

            DriveResource.MetadataResult metadataResult = driveFile.updateMetadata(googleApiClient, metadataChangeSet).await();
            Status status = metadataResult.getStatus();
            if (!status.isSuccess()) {
                h.handleStatus(status);
                return false;
            }
        }

        Status status;
        try {
            status = driveContents.commit(googleApiClient, null).await();
        } catch (java.lang.IllegalStateException e) {
            // java.lang.IllegalStateException: DriveContents already closed.
            Log.e(TAG, "", e);
            return false;
        }

        if (!status.isSuccess()) {
            h.handleStatus(status);
            return false;
        }

        status = Drive.DriveApi.requestSync(googleApiClient).await();
        if (!status.isSuccess()) {
            // Sync request rate limit exceeded.
            //
            //h.handleStatus(status);
            //return false;
        }

        return true;
    } finally {
        if (googleCloudFile != null) {
            googleCloudFile.metadataBuffer.release();
        }
    }
}

Search for existing App Folder zip file

private static String getGoogleDriveTitle(long checksum, long date, int version) {
    return "jstock-" + org.yccheok.jstock.gui.Utils.getJStockUUID() + "-checksum=" + checksum + "-date=" + date + "-version=" + version + ".zip";
}

// https://stackoverflow.com/questions/1360113/is-java-regex-thread-safe
private static final Pattern googleDocTitlePattern = Pattern.compile("jstock-" + org.yccheok.jstock.gui.Utils.getJStockUUID() + "-checksum=([0-9]+)-date=([0-9]+)-version=([0-9]+)\\.zip", Pattern.CASE_INSENSITIVE);

private static GoogleCloudFile searchFromGoogleDrive(GoogleApiClient googleApiClient, HandleStatusable h, PublishProgressable p) {
    DriveFolder driveFolder = Drive.DriveApi.getAppFolder(googleApiClient);

    // https://stackoverflow.com/questions/34705929/filters-ownedbyme-doesnt-work-in-drive-api-for-android-but-works-correctly-i
    final String titleName = ("jstock-" + org.yccheok.jstock.gui.Utils.getJStockUUID() + "-checksum=");
    Query query = new Query.Builder()
            .addFilter(Filters.and(
                Filters.contains(SearchableField.TITLE, titleName),
                Filters.eq(SearchableField.TRASHED, false)
            ))
            .build();

    DriveApi.MetadataBufferResult metadataBufferResult = driveFolder.queryChildren(googleApiClient, query).await();

    if (metadataBufferResult == null) {
        return null;
    }

    Status status = metadataBufferResult.getStatus();

    if (!status.isSuccess()) {
        h.handleStatus(status);
        return null;
    }

    MetadataBuffer metadataBuffer = null;
    boolean needToReleaseMetadataBuffer = true;

    try {
        metadataBuffer = metadataBufferResult.getMetadataBuffer();
        if (metadataBuffer != null ) {
            long checksum = 0;
            long date = 0;
            int version = 0;
            Metadata metadata = null;

            for (Metadata md : metadataBuffer) {
                if (p.isCancelled()) {
                    return null;
                }

                if (md == null || !md.isDataValid()) {
                    continue;
                }

                final String title = md.getTitle();

                // Retrieve checksum, date and version information from filename.
                final Matcher matcher = googleDocTitlePattern.matcher(title);
                String _checksum = null;
                String _date = null;
                String _version = null;
                if (matcher.find()){
                    if (matcher.groupCount() == 3) {
                        _checksum = matcher.group(1);
                        _date = matcher.group(2);
                        _version = matcher.group(3);
                    }
                }
                if (_checksum == null || _date == null || _version == null) {
                    continue;
                }

                try {
                    checksum = Long.parseLong(_checksum);
                    date = Long.parseLong(_date);
                    version = Integer.parseInt(_version);
                } catch (NumberFormatException ex) {
                    Log.e(TAG, "", ex);
                    continue;
                }

                metadata = md;

                break;

            }   // for

            if (metadata != null) {
                // Caller will be responsible to release the resource. If release too early,
                // metadata will not readable.
                needToReleaseMetadataBuffer = false;
                return GoogleCloudFile.newInstance(metadataBuffer, metadata, checksum, date, version);
            }
        }   // if
    } finally {
        if (needToReleaseMetadataBuffer) {
            if (metadataBuffer != null) {
                metadataBuffer.release();
            }
        }
    }

    return null;
}

---

The problem occurs, during loading application data. Imagine the following operations

1. Upload zip data to Google Drive App Folder for the first time. The checksum is 12345. The filename being used is ...checksum=12345...zip
2. Search for zip data from Google Drive App Folder. Able to find the file with filename ...checksum=12345...zip. Download the content. Verify the checksum of content is 12345 too.
3. Overwrite new zip data to existing Google Drive App Folder file. New zip data checksum is 67890. The existing app folder zip file is renamed to ...checksum=67890...zip
4. Search for zip data from Google Drive App Folder. Able to find the file with filename ...checksum=67890...zip. However, after downloading the content, the checksum of the content is still old 12345!

Download App Folder zip file

public static CloudFile loadFromGoogleDrive(GoogleApiClient googleApiClient, HandleStatusable h, PublishProgressable p) {
    final java.io.File directory = JStockApplication.instance().getExternalCacheDir();
    if (directory == null) {
        org.yccheok.jstock.gui.Utils.showLongToast(R.string.unable_to_access_external_storage);
        return null;
    }

    Status status = Drive.DriveApi.requestSync(googleApiClient).await();
    if (!status.isSuccess()) {
        // Sync request rate limit exceeded.
        //
        //h.handleStatus(status);
        //return null;
    }

    GoogleCloudFile googleCloudFile = searchFromGoogleDrive(googleApiClient, h, p);

    if (googleCloudFile == null) {
        return null;
    }

    try {
        DriveFile driveFile = googleCloudFile.metadata.getDriveId().asDriveFile();
        DriveApi.DriveContentsResult driveContentsResult = driveFile.open(googleApiClient, DriveFile.MODE_READ_ONLY, null).await();

        if (driveContentsResult == null) {
            return null;
        }

        status = driveContentsResult.getStatus();
        if (!status.isSuccess()) {
            h.handleStatus(status);
            return null;
        }

        final long checksum = googleCloudFile.checksum;
        final long date = googleCloudFile.date;
        final int version = googleCloudFile.version;

        p.publishProgress(JStockApplication.instance().getString(R.string.downloading));

        final DriveContents driveContents = driveContentsResult.getDriveContents();

        InputStream inputStream = null;
        java.io.File outputFile = null;
        OutputStream outputStream = null;

        try {
            inputStream = driveContents.getInputStream();
            outputFile = java.io.File.createTempFile(org.yccheok.jstock.gui.Utils.getJStockUUID(), ".zip", directory);
            outputFile.deleteOnExit();
            outputStream = new FileOutputStream(outputFile);

            int read = 0;
            byte[] bytes = new byte[1024];

            while ((read = inputStream.read(bytes)) != -1) {
                outputStream.write(bytes, 0, read);
            }
        } catch (IOException ex) {
            Log.e(TAG, "", ex);
        } finally {
            org.yccheok.jstock.file.Utils.close(outputStream);
            org.yccheok.jstock.file.Utils.close(inputStream);
            driveContents.discard(googleApiClient);
        }

        if (outputFile == null) {
            return null;
        }

        return CloudFile.newInstance(outputFile, checksum, date, version);
    } finally {
        googleCloudFile.metadataBuffer.release();
    }
}

First, I thought

Status status = Drive.DriveApi.requestSync(googleApiClient).await()

doesn't do the job well. It fails in most of the situation, with error message Sync request rate limit exceeded. In fact, the hard limit imposed in requestSync, make that API not particularly useful - Android Google Play / Drive Api

---

However, even when requestSync success, loadFromGoogleDrive still can only get the latest filename, but outdated checksum content.

I'm 100% sure loadFromGoogleDrive is returning me a cached data content, with the following observations.

1. I install a DownloadProgressListener in driveFile.open, bytesDownloaded is 0 and bytesExpected is -1.
2. If I use Google Drive Rest API, with the following desktop code, I can find the latest filename with correct checksum content.
3. If I uninstall my Android app and re-install again, loadFromGoogleDrive will able to get the latest filename with correct checksum content.

Is there any robust way, to avoid from always loading cached app data from Google Drive?

---

I manage to produce a demo. Here are the steps to reproduce this problem.

Step 1: Download source code

https://github.com/yccheok/google-drive-bug

Step 2 : Setup in API console

Step 3: Press button SAVE "123.TXT" WITH CONTENT "123"

A file with filename "123.TXT", content "123" will create in the app folder.

Step 4: Press button SAVE "456.TXT" WITH CONTENT "456"

The previous file will be renamed to "456.TXT", with content updated to "456"

Step 5: Press button LOAD LAST SAVED FILE

File with filename "456.TXT" was found, but the previous cached content "123" is read. I was expecting content "456".

Take note that, if we

1. Uninstall demo app.
2. Re-install demo app.
3. Press button LOAD LAST SAVED FILE, file with filename "456.TXT" and content "456" is found.

I had submitted issues report officially - https://code.google.com/a/google.com/p/apps-api-issues/issues/detail?id=4727

---

Other info

This is how it looks like under my device - http://youtu.be/kuIHoi4A1c0

I realise, not all users will hit with this problem. For instance, I had tested with another Nexus 6, Google Play Services 9.4.52 (440-127739847). The problem doesn't appear.

I had compiled an APK for testing purpose - https://github.com/yccheok/google-drive-bug/releases/download/1.0/demo.apk

Answer 1

1. Search on Google Drive is slow. Why not use properties of the base folder to store id of the zip file? https://developers.google.com/drive/v2/web/properties
2. File names on Google Drive are not unique, you can upload multiple files with same names. The File ID returned by Google, however, is unique.