Using ADB to access a particular UI control on the screen

Question

Is it possible for adb shell console to access an Android app\'s specific button using its id? or text?

I\'m trying to automate the button click on the dev

Answer 1

if me, i use python to do that, combine cmd command with subprocess.Popen, the script would be:

from subprocess import Popen, PIPE

import os

adbpath = r'YOUR ADB DIR'
adbfile = 'adb.exe'
adb = os.path.join(adbpath, adbfile)

def tap(a):
    cor = a
    t = Popen([adb, 'shell', 'input', 'tap', str(a[0]), str(a[1])])

#i give you extra script

def swipe(a, b):
     first = a
     last = b
     sab = Popen([adb, 'shell', 'input', 'swipe', str(first[0]), str(first[1]), str(last[0]), str(last[1]), '200'])

then when you have a button coord to tap just do this:

ok_coord = 122,137
tap(ok_coord)

when you want to swipe:

coord1 = 122,373
coord2 = 122,26
swipe(coord1, coord2)

to get the coordinate of button or a point, use 'uiautomatorviewer.bat' that exists in folder 'sdk\tools\bin'

or you can use apricot, erm3nda or Vinicius Dantas method,

Answer 2

Yes you can, although it's not pretty.

You can get the view hierarchy by using this command:

adb exec-out uiautomator dump /dev/tty, which prints the output to the screen as an XML file. Unfortunately it's not valid XML, as there's some text appended. So let's filter that out:

adb exec-out uiautomator dump /dev/tty | awk '{gsub("UI hierchary dumped to: /dev/tty", "");print}'

Now we've got a valid XML. Let's run this through XPath:

adb exec-out uiautomator dump /dev/tty | awk '{gsub("UI hierchary dumped to: /dev/tty", "");print}' | xpath '//node[@resource-id="com.example:id/btnDone"]' 2> /dev/null, which will search for all nodes with the specific ID. Now you got the node, and you can do some more stuff on this.

If you were to get only the viewbounds, you'd have to do this:

adb exec-out uiautomator dump /dev/tty | awk '{gsub("UI hierchary dumped to: /dev/tty", "");print}' | xpath '//node[@resource-id="com.example:id/btnDone"]/@bounds' 2> /dev/null | awk '{$1=$1};1' | awk '{gsub("bounds=", "");print}' | awk '{gsub("\"", "");print}', which cleans up the string afterwards and just outputs [294,1877][785,1981].

The source of the specific node is:

<node index="1" text="Let’s get started!" resource-id="com.example:id/btnDone" class="android.widget.Button" package="com.example" content-desc="" checkable="false" checked="false" clickable="true" enabled="true" focusable="true" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[294,1877][785,1981]" />

Answer 3

UI automator gives resource IDs, text, and the bounds of the UI element. An XML viewer or Chrome browser can be used to get a better view of the file.

adb pull $(adb shell uiautomator dump | grep -oP '[^ ]+.xml') /tmp/view.xml

The UI element's bounds can be be extracted and the mid point can be calculated. Replace text= with resource-id= or content-desc= as needed.

coords=$(perl -ne 'printf "%d %d\n", ($1+$3)/2, ($2+$4)/2 if /text="MY_BUTTON_TEXT"[^>]*bounds="\[(\d+),(\d+)\]\[(\d+),(\d+)\]"/' /tmp/view.xml)

Now we have the coordinates of the UI element's center in $coords and we just need to send a tap event.

adb shell input tap $coords

Answer 4

I think the best you can do is to inject touch based on a coordinate.

Please see send touch event from ADB to a device and simulating touch using ADB

You might get a coordinate of a button from dumpsys window or activity.

You could also check out Monkeyrunner.

Answer 5

You won't be able to use the button id, but you can get the button coordinates and simulate a tap event.

Android comes with an input command-line tool that can simulate miscellaneous input events. To simulate tapping, use:

input tap x y

You can use the adb shell to run the command remotely:

adb shell input tap x y

Other options are:

shell@m0:/ $ input
input
usage: input ...
       input text <string>
       input keyevent <key code number or name>
       input [touchscreen|touchpad|touchnavigation] tap <x> <y>
       input [touchscreen|touchpad|touchnavigation] swipe <x1> <y1> <x2> <y2> [duration(ms)]
       input trackball press
       input trackball roll <dx> <dy>