Any QA engineer in any project sooner or later reaches the point when it is necessary to automate the testing, no matter if it is a web or mobile application, Android or iOS. For example, we have faced the need to implement a complex and hard to understand workflow. In the course of development we discussed with our customer all the wishes and requirements, noted and documented everything. However, in a couple of weeks, it is not easy to test it and recall all the fine points. That is why it would be great to get it automated.

Which automated mobile testing tool to choose?

Our project Hey Ya! is an application for Android and iOS with a web server on Ruby on Rails. The tools provided by the Internet are: Calabash Android, MonkeyTalk, Robotium, Selendroid, UIAutomator for Android and Calabash iOS, Frank, UIAutomation, KeepItFunctional and ios-driver for iOS.

All of these tools work well but do not all of them meet our criteria, which are:

• Ruby on Rails server.
• Cross-platform support (we have Android and iOS apps).
• Regular updating of the framework.
• Running tests on real devices - our application uses camera but it is impossible to work with the camera in iOS simulator.

The tool that was chosen is Calabash.

Advantages of Calabash mobile app automation testing:

• Calabash is available for Android and iOS platforms.
• It is written in Ruby.
• Calabash uses Gherkin Syntax (Cucumber style),
• The latest update of Calabash was a month ago (at the moment of writing the article), i.e. it is updated quite frequently.
• It supports testing on the real devices.

The following is a guide, based on what we have learnt about mobile app automation testing with Calabash.

Installation and running tests

First of all we need xcode. We install gem and generate calabash-ios-setup command in our project. The calabash-ios-gem command generates skeleton which is almost the same as the Cucumber's one, aside from the fact that it has its own launchers. In the screen you can see the launcher for iOS application.

The calabash-ios-setup command generates the Calabash schema for iOS project right in xcode and connects Calabash framework to the project. It is executed very easy - with just a single command.

In order to run tests on actual mobile devices, we require the bundle-kit of our application, which can be accessed via the xcode and device endpoint. The device endpoint refers to the IP of the device that is connected to Wi-Fi. Automated tests for iOS mobile applications are executed using Cucumber as the launch command with Calabash. This launch command is simply Cucumber, which initiates the tests. The environment variable can be set within the command or specified in the launcher.

The installation process for Android is similar, with the only difference being the launchers used in the skeleton.

The tests on Android are run by the calabash-android run command with the specified path to apk-file. For iOS, it is necessary to assemble (build) the application on the device beforehand. Also, it is essential to build it with Calabash scheme, because the test server is installed directly in the app.

How to avoid duplication?

The given state of things implies that we have two projects (iOS and Android). Let's take an ordinary feature, for instance sign up. The structure of Cucumber is well-known: we have a file-feature, then the file-steps and we use page_object_pattern, after which the implementation of the functions invoked by steps is performed. It turns out that till a certain phase we write the same code: sign_up_features and sign_up_steps are similar in iOS and Android. Why? Because the design of our applications is almost identical. "Almost" because some things are not possible in Android but feasible in iOS and vice versa.

What should we do to avoid the duplication? A special gratitude here goes to Dmitriy Dordovskiy and Andrey Kutejko for their valuable assistance in delivery of the following solution.

The remedy is trivial and easy: to install gems on the server side, i.e. extract Calabash out of our mobile projects, split page_object into modules and glue together calabash_iOS and calabash_Android launchers.

Now, if we specify the iOS client platform in the environment variable, the Cucumber will be activated. If we run tests on the Android client platform, it will go with Calabash Android path and will display corresponding files and methods.

One of essential things here is running the test server. It means that our app doesn't use staging/production but goes to the local server. The tests are stored together with the server on the server side. We have access to the server and it does not require efforts to run it. We run it on the host 0.0.0.0 and in the application we set the host which we will use - IP.

The next step is splitting page_object into modules. The features folder contains page_object_pattern and now it also holds Android and iOS. Here is an example of one module pages general - it stores code that executes our iOS-actions. In the Android folder there is everything associated with Android: pages and all the necessary methods.

What do we get with this approach? We have a single signup_feature file instead of two ones and one signup_steps_rb file. At this stage, we do not make any separation because we have the same steps for iOS and Android. After the steps file we follow iOS or Android path and use iOS- or Android-modules. The launch commands look like this: the platform is specified, run by cucumber or calabash-android run commands and checked.

Conclusion

Let's sum up all pros and cons of the given way of dealing with Calabash test automation for mobile apps. The advantage of such implementation is that we do not have a duplicated code in feature and steps_rb files - we significantly reduce efforts on writing and duplication. It is possible to access objects via ActiveRecord and emulate work of the second client on the server side. In particular, our application does not have the logout function and sometimes it is necessary to test the cases where the first client has done something and we use the second one to check this. Thus, on the server side, we emulate the work and we use the client to make the check.

Obviously, there are some disadvantages of this approach. However, they are purely visual, i.e. different launch commands (cucumber, calabash-android run). Also, it is not possible to work in the debug mode, provided by calabash (the remedy is binding.pry). We have to stop in the required place, check it in the debug mode, verify all what we want, find the needed element and then continue to work. So, the disadvantages exist but they are not crucial.