Make your program USE a gui

Question

I\'d like to write a program able to \"use\" other programs by taking control of the mouse/keyboard and being able to \"see\" what\'s on the screen.

I used AutoIt to

Answer 1

I've had some luck with similar tasks using PyWinAuto.

pywinauto is a set of python modules to automate the Microsoft Windows GUI. At it's simplest it allows you to send mouse and keyboard actions to windows dialogs and controls.

It also has some support for capturing images of dialogs and such using the Python Imaging Library PIL.

Answer 2

If you are comfortable with pascal, a really powerful keyboard/mouse/screen-reading program is SCAR: http://freddy1990.com/index.php?page=product&name=scar It can do OCR, bitmap finding, color finding, etc. It's often used for automating online games, but it can be used for any situation where you want to simulate a human reading the screen and giving input.

Answer 3

AutoIt is completely capable of doing everything you mentioned. When I'm wanting to do some automation but use the features of Python, I find it easiest to use AutoItX which is a DLL/COM control.

Taken from this answer of mine:

import win32com.client
oAutoItX = win32com.client.Dispatch( "AutoItX3.Control" )

oAutoItX.Opt("WinTitleMatchMode", 2) #Match text anywhere in a window title

width = oAutoItX.WinGetClientSizeWidth("Firefox")
height = oAutoItX.WinGetClientSizeHeight("Firefox")

print width, height

Answer 4

I've used the Windows (only) Input API to write a VNC-like remote-control application in the past. It lets you fake keyboard and mouse input nicely at a system level (ie not just posting events to a single application).

If you're trying to do any sort of automated testing of whole systems at the GUI level, this excellent USENIX paper describing automated responsiveness testing is a must-read.

Answer 5

You can use WATSUP under Windows.