Automated testing for OpenGL application

Question

I have a Java Application that uses JOGL to provide a large part of the GUI.

Is there any tool which you know of, or have used which can automate the testing of OpenGL a

Answer 1

I've been thinking of using a picture-diff tool such as PDiff to test OpenGL code, by taking snapshots, saving them to disk and comparing with previous regression output. That way, really bad stuff (missing textures) pop up but humanly unnoticable things (such as the above mentioned small diffs between implementation) go through fine.

Also, for automating user interaction, either the GUI classes should be sufficiently open for you to send events or call 'clicked' on a button, or you have to inject OS events manually to your apps. This is possible, but much more troublesome. Might be easier to open up the GUI layer, if it's Open Source.

Answer 2

I have written unit-tests for C++ (Qt on Linux) & OpenGL before. I don't know any reason it shouldn't work for Java too.

The things which worked for me are:

• Abstract your OpenGL context provider so the rest of your code is independent of it. In my case the main app used Qt's QGLWidget, but the unittests used a pbuffer-based one which I could create with no windowing infrastructure at all (other than a designated X11 DISPLAY). Later I added an "offscreen Mesa" (pure software OpenGL implementation) so they'd even work on a headless build machine with no GPU at all.

• Keep your OpenGL code independent of your GUI code. In my case the OpenGL "rendering engine" didn't know anything about Qt classes (e.g mouse events). Define your own testable API which isn't tied to any specific GUI concepts, and write tests for it.

• In the unittests, read the contents back from the framebuffer using glReadPixels and either hit them with some assertions about which pixels should be particular values, or go down the regression-testing route and compare the framebuffer capture with a stored image you know is good (either from manual verification, or because it's produced from some other reference model).

• Allow a bit of fuzziness in any image regression testing; most OpenGL implementations produce slightly different output.

• (I didn't do this, but...) Ideally you want to be able to test the GUI layer by asserting that it makes the expected sequence of calls to your rendering engine in response to GUI activity. If it does that, and you're confident of your render layer because of the above tests... well, no need to actually do the rendering. So create a suitable mock object of your rendering layer for use when GUI testing (I'd imagine tests like "mouse drag from here to there results in a call to the rendering layer to set a particular transform matrix"... stuff like that).

Answer 3

You can test your JOGL based application using Sikuli which performs UI automation via image recognition techonolgy on screen-shots.

I am currently using Sikuli to functional-test a Java app that is predominantly based on the NASA Worldwind Java SDK (which is based on JOGL). Using the Sikuli Java API my test suite can recognise icons within the OpenGL canvas, click on them and also drag them around. Sikuli can also recognise and extract text from the canvas via OCR, however the performance of this seems to be a little hit-and-miss (depending on language, font, size and background colours behind the text).

I have done a lot of automated UI-Testing using other tools that work by introspecting the windowing toolkit (e.g. Swing, SWT, native Windows) and found that Sikuli runs much slower than those, however that is understandable given the amount of image-processing it needs to do behind the scenes. Also note that Sikuli currently requires your application to run in a Window (not in full screen mode).

Sikuli runs on both Windows and Linux. I'd recommend you try it out. I couldn't find any other tool capable of doing that level of functional-testing of an OpenGL based application.