iOS Automated Tests - XCTest vs Appium

Question

I am looking for opinions from test engineers and iOS developers regarding a specific matter.

I have been using Appium for over a year now, and I have come across variou

Answer 1

As you've already discovered, one of the greatest disadvantages of Appium is that it's a third party framework and it breaks pretty much every time Apple makes an Xcode release. This is unlikely to change since Apple now maintains its own UI testing framework.

There are some other advantages to using Appium - you can pick from quite a few languages (Java, Ruby, Python, C#...), which makes it accessible to anyone who can program in one of those very popular languages, and you can conceivably share code between tests for the same app on iOS and Android.

In my experience, while there are a lot of people out there using Appium, the level of community support doesn't make up for the disappointing level of maintenance; aside from the fragility of Appium's compatibility with Xcode, I've found that some key functions have remained unimplemented in some bindings, e.g. scrolling in Python.

With UI tests, reliability is the most important aspect of your framework. Without reliability, you can't trust the tests to flag up problems, and without trust, your tests provide little to no value to you and your team. This is why I recommend XCTest over any third-party framework.

With XCTest, you never have to worry about not being able to update your version of Xcode, and the framework is maintained to Apple's release standards. As with all the iOS UI testing frameworks, there are some bugs, (particularly around pickers) but I find that the stability of the framework and the fact that it's owned by Apple outweighs the disadvantages of the odd bug.

Being endorsed by Apple is a significant pro for using XCTest, since Apple could remove access to the APIs which Appium depends on and Appium could stop working forever overnight. Historically, Apple do not simply remove support for their own frameworks without at least a year of notice.

To use XCTest directly, you need to use Swift (recommended) or Objective-C. There isn't as much language choice as Appium gives, but support for both languages is consistent as they both use the same implementation. Swift is a strong choice of language, especially for larger projects, because its type-safety allows you to notice many programming errors before runtime. Both languages also give you great intellisense (autocomplete) support in Xcode, which is something that is not offered out of the box by 'dynamic' languages like Python or Ruby.

The community around XCTest is growing as more information becomes available about using it with UI tests and more people feel able to adopt it. Many parts of the framework used for UI testing have been being used for many years for unit tests, so in many ways, there was already a lot of information available about using it, before the UI testing support was added.

Both frameworks use similar concepts - XCUIApplication is similar to Appium's Driver, which gives you access to what's on the screen. The level of functionality offered by both frameworks is arguably very similar, so it depends where your priorities lie - with reliability (XCTest) or reusability across other platforms and language accessibility (Appium).

Answer 2

Some pros & cons of Appium and XCTest:

In favour of XCTest:

• Apple's own product so less likely to have compatibility issues emerge when new versions of Xcode/iOS are launched
• Tests run more efficiently (faster and less random errors)
• Record/play back feature arguably makes object identification easier for some of those more tricky to identify objects
• Far easier to setup than Appium (especially after iOS 10+)

In favour of Appium:

• Allows more code re-use across Apple and Android automation suites

• Greater choice of compatible languages

• Compatible with Cucumber frontend code

There are other pros & cons of both tools, but having used both I feel these are the main ones (in my personal opinion)