Google cloud storage API with Android API key not working (however browser API key works)

Question

I am trying to access a list of objects from google cloud storage from my android application. The call I am using to access google cloud storage is:

GET https://www.googleapis.com/storage/v1/b/{BUCKET_NAME}/o?key={YOUR_API_KEY}

When I set up a Browser API key with no referrers, I get the JSON results I expect. The files in the bucket are publicly accessible and I am simply trying to read them. However, I would like to use an Android API key so that not anyone can access the files. When I use the Android API key, the result I get is:

{
   "error":{
      "errors":[
         {
            "domain":"usageLimits",
            "reason":"ipRefererBlocked",
            "message":"There is a per-IP or per-Referer restriction configured on your API key and the request does not match these restrictions. Please use the Google Developers Console to update your API key configuration if request from this IP or referer should be allowed.",
            "extendedHelp":"https://console.developers.google.com"
         }
      ],
      "code":403,
      "message":"There is a per-IP or per-Referer restriction configured on your API key and the request does not match these restrictions. Please use the Google Developers Console to update your API key configuration if request from this IP or referer should be allowed."
   }
}

I have set up the Android API key with the correct package name and fingerprints for both my debug keystore and my release build keystore. My app is on the play store using the release keystore, so the package name is unique and the keystore has been used multiple times to update the app. I get the same error returned for both debug builds and release builds.

I have seen people having this issue before, and everyone tells them to use the browser API key, but I would prefer if any access to the cloud storage was only from my app (to keep potential rates low). Am I overlooking something? Has anyone managed to access the cloud storage api with an Android API key?