The Robot Framework(RF) was my main testing tool for several years. During my participation in a large-scale projects, I experienced two different approaches to testing automation with this tool:
- writing tests on a pure DSL Robot Framework,
- working in conjunction with Python.
If the first path has a low entry barrier, then the second, in my opinion, is more convenient for supporting large projects. Although there is no fundamental difference between the approaches. One way or another, it all comes down to finding libraries. However, it’s worth talking about the features of the approaches
What is the Robot Framework
At first, I have to introduce this powerful tool.
Robot Framework is a keyword-driven testing framework. It is used to automate acceptance testing and ATDD (development approach through acceptance testing). This system has easy-to-use syntax for test data and allows you to automate tests using keywords. Besides, the Robot Framework has excellent built-in and third-party libraries that allow you to quickly start and integrate automation into work processes without wheel reinventing.
The basic structure of the Robot Framework is implemented using Python and also works on Jython (JVM) and IronPython (.NET).
Examples of use, as well as full documentation on the framework and internal libraries, can be found on the official website of the project: http://robotframework.org/.
First approach. Pure DSL and Java
First time I encountered the Robot Framework a year ago, after changing my job. Before that, I used Java and C# to write automated tests.
Best IDE
To choose the proper testing tool for the existing project I interviewed new colleagues about their preferences. They did not agree on the best IDE for the Robot Framework. Mostly they used various IDEs plugins to deal with the Robot Framework. All recommendations I collected were split 50/50 between Atom and PyCharm. Of course, there is a RIDE, but this is not a panacea. At that time (a year ago) I did not find normal RIDE documentation, and in the one that I found, I did not find any big advantages for my task. Therefore, for starters, I decided to try the Atom with RF plugins.
Jython junction
I’ve entered the project when it was almost finished. Conjunction of Jython and Robot Framework already worked there, a huge test solution (written on the DSL Robot Framework itself) was assembled from more than 1000 tests and several thousand lines of code of auxiliary libraries in Java. The solution was architected in the way so that it was easy to attract third-party experts who know only the DSL Robot Framework and don’t know Java.
Assembly and running
Since the Jython + RF bundle was used, everything was assembled by maven, and the robot files were simply copied to the target directory for later execution. Tests were run by scripts (.bat or .sh files) with the path to either a separate test case (a separate .robot file) or a test plan (a file with a list of relative paths to test cases) as inputs.
Report format
The standard report (in the screenshot above) consists of the report.html and log.html files:
- the report contains a general summary of the past run, where you can see the surface results of all tests (Passed or Failed);
- in the log-file, you can see more detailed information – the execution of each test step by step. There you can display everything you need to debug tests.
Honestly, from the first glance at the Robot Framework report, the eye begins to twitch a bit: a huge amount of information is displayed and it takes some time to understand the end-to-end structure of the tests and develop the skill to read such a log. Within a couple of months, I could quote The Matrix: “You get used to it. I don’t even see the code. All I see is blonde, brunette, redhead. ” So I saw all the necessary information in a file without additional tools.
The advantage is that the output can be controlled: different levels of logging determine which information will be displayed and which will not. The level can be adjusted even for each line individually through the built-in library method
Results
We worked for about six months with the DSL Robot Framework as our main tool. During this time, I switched from Atom to VSCode, but this did not change the essence of the approach.
The project was developing. In the final iteration, the auxiliary library totaled 6,700 lines of code on a pure Robot Framework. With such an amount of code, it has become difficult to maintain the system. Refactoring required resources that were not allocated. After a short discussion with the customer, the approach was changed.
Second Approach. Python conjunction
We’ve decided to develop Python tests in conjunction with the Robot Framework instead of creating everything on the DSL Robot Framework. At first, we’ve written helper libraries and other low-level interactions with the product in Python. The Robot Framework became just a “runner”.
Since Python is a pure high-level language, not DSL, there are more structuring options. At a minimum, you can use the Python IDE to help you find the same methods (they do the same thing, but are called differently) or even write a piece of code for you. Some data could be wrapped in generators, hang decorators on functions, etc. I was happy to return to a higher-level language. Working with this approach is more like an ordinary development.
During the time that I worked with a new approach, I felt that it gives more opportunities. However, the writing of the tests has not changed much. In this particular project, inside the Python code, the Robot Framework built-in library methods were still called. This was due to the peculiarity of the customer’s requirements for test development. We simply separated the executable part from the test case algorithm.
Which approach is better?
I prefer the second approach. However, choosing the path of your project, it is worth starting from the task – who writes tests and how.
As I said above, Python (the second approach) provides more opportunities, but in this case, we need to hire people who are familiar with this language. Robot Framework itself (and the first approach) is less demanding – you can approach it by reading the official documentation on its DSL. After studying the user guide, you can create any tests – they will look pretty neat.
As a result, the Robot Framework by itself is more suitable for yesterday’s manual testers without any explicit programming experience in high-level languages. Even a person who is not involved in programming can write a test, simply by calling up the necessary keywords.
However, if you want to keep the executable part separate and have a friendly environment for refactoring, the second approach is for you.