How to fetch device location using API AI?

Question

I am using GUI tools provided by API AI to create Actions. Is it possible to fetch device location? I have heard that this is possible by requesting permissions. Is this documented anywhere? An example/code snippet will be very useful.

Answer 1

The documentation is a bit unclear. I hope this can help someone.

All you have to do is create a child fallback intent for the intent you are requesting permissions from.

1. To do so you need to click on the "Add follow-up intent" link on your intent. Note that the link will only appear when you hover it. Select the fallback option from the dropdown list and a child fallback intent will be created for you.

Screen shot for how to create a child fallback intent

2. Click into the child fallback intent and enable "Use webhook".

Screen shot for how to configure the child fallback intent

That's it. Now once the permission is requested, the user response will be post back to you with the action which is configured in your child fallback intent.

See below for the webhook sample code.

'use strict';

const express = require('express')();
const router = require('express').Router();
const bodyParser = require('body-parser');
const ActionsSdkApp = require('actions-on-google').ActionsSdkApp;
const ApiAiApp = require('actions-on-google').ApiAiApp;

express.use(bodyParser.json({type: 'application/json'}));

// In aip.ai console, under Fulfillment set webhook url to
// https://[YOUR DOMAIN]/example/location
// don't forget to select "Enable webhook for all domains" for the DOMAIN field
router.post('/location', (req, res) => {
	const app = new ApiAiApp({request: req, response: res});
	const intent = app.getIntent();
	
	switch(intent){
		case 'input.welcome':
			// you are able to request for multiple permissions at once
			const permissions = [
				app.SupportedPermissions.NAME,
				app.SupportedPermissions.DEVICE_PRECISE_LOCATION
			];
			app.askForPermissions('Your own reason', permissions);
		break;
		case 'DefaultWelcomeIntent.DefaultWelcomeIntent-fallback':
			if (app.isPermissionGranted()) {
				// permissions granted.
				let displayName = app.getUserName().displayName;
				
				//NOTE: app.getDeviceLocation().address always return undefined for me. not sure if it is a bug.
				// 			app.getDeviceLocation().coordinates seems to return a correct values
				//			so i have to use node-geocoder to get the address out of the coordinates
				let coordinates = app.getDeviceLocation().address;
				
				app.tell('Hi ' + app.getUserName().givenName + '! Your address is ' + address);
			}else{
				// permissions are not granted. ask them one by one manually
				app.ask('Alright. Can you tell me you address please?');
			}
		break;
	}
});

express.use('/example', router);

express.listen('8081', function () {
  console.log('Example app is running')
})

Answer 2

There are three parts to this:

1. Asking for permission
2. Telling API.AI to handle the response
3. Handling the information forwarded by API.AI

Asking for permission

A lot of people aren't mentioning the JSON body that needs to be made to request permissions in the first place. This is (poorly) documented at actions > assistant > helpers > User information since it doesn't really cover what needs to be sent if you're using API.AI.

In short, your JSON needs to look something like this:

{
    "speech": "PLACEHOLDER_FOR_PERMISSION",
    "data": {
        "google": {
            "expectUserResponse": true,
            "isSsml": false,
            "noInputPrompts": [],
            "systemIntent": {
                "intent": "actions.intent.PERMISSION",
                "data": {
                    "@type": "type.googleapis.com/google.actions.v2.PermissionValueSpec",
                    "optContext": "To pick you up",
                    "permissions": [
                        "NAME",
                        "DEVICE_PRECISE_LOCATION"
                    ]
                }
            }
        }
    }
}

Telling API.AI to handle the response

API.AI needs to know which Intent handles permission information. You do this by setting the Event to actions_intent_PERMISSION.

This will look something like this:

You can set the Action to whatever makes sense for your webhook, and be sure to enable webhook fulfillment for the Intent as well.

If you need to keep track of where the permission request was initiated from, and handle it through a different Action, you can set a Context and have different handling Intents based on different Context settings.

The Fallback Intent method works because there is no better match at that point since you hadn't specified a regular Intent with actions_intent_PERMISSION. It isn't the best choice, however, since it could match other situations from your user.

Get the info in your webhook

Your webhook will get a JSON object in the body.

The user's name and related information will be at the object path originalRequest.data.user.profile.

Their location is provided at originalRequest.data.device.location.

See Overview > Actions on Google lets you extend the functionality for references.

Answer 3

Yes, it's possible to fetch device location and some other bits of info (name) after the user gives permission.

It's already documented well by google, so I don't think it's necessary for me to repeat it here.

If the question is whether it's possible to get the location information without a custom webhook implementation that's called through API.AI's fulfillment, then the answer is no: API.AI itself does not natively provide this capability, you need to use API.AI's extension mechanism, that is, a webhook.

Answer 4

As you can see here Api.Ai doc in the response there is a data field. You need to create a json response exactly Google expects the webhook response (see Google docs) but if you are using Api.Ai you need to nest the Google json element in under the data field.

Answer 5

I was struggling with this for so long. I found this forum and something clicked. Turns out it's really not that hard.

In your php webhook, your original intent should return this as json -- or json_encode this array:

array(
'data' => 
 array(
 'google' => 
  array(
   'expectUserResponse' => true,
   'systemIntent' => 
   array(
     'intent' => 'actions.intent.PERMISSION',
     'data' => 
   array(
       '@type' => 
      'type.googleapis.com/google.actions.v2.PermissionValueSpec',
       'optContext' => 'To find the closest site ',
       'permissions' => 

        'DEVICE_PRECISE_LOCATION'
      ,
       ),
     ),
   ),
 ),
)

then create another intent with the event: actions_intent_PERMISSION

In your php webhook for the newly created intent, you will now have the location returned. It is in the response: ["originalRequest"]['data']['device']['location']['coordinates']

Easy Peasy...