Posts Tagged ‘ctest’

Preventing graphical automated tests from interrupting you


You may have found that running graphical automated tests (like unit tests for view subdirectories in ktutorial-library and ktutorial-editor) can be very annoying. Ideally, you would launch the tests and then go on working on something else while they run. The problem is that, while you are trying to work on something else, the windows created by the unit tests are constantly appearing and disappearing.

If you don’t want to use the time spent running the tests giving a rest to your eyes far away from your monitor, and you want to use your computer while the tests are running, there is an easy solution: run the tests in a different display than the one you are using.

To do this, you must start that other display before running the tests. Xnest can be used to start another display inside a window in your current display. You should also start a window manager for that display to ensure that the tests will run in a “real” environment. Both things can be done in a single command (and as a user, there is no need to become root for this):
(Xnest :1 2>/dev/null & ) && (kwin -display :1 2>/dev/null & )

That command assumes that you are using just one display, and that you want to start a second display :1. If you are already using several displays, adjust the number accordingly. The redirection of error output (2>/dev/null) is not mandatory; it is just used to prevent the console to be flooded with error messages.

Once the second display is started, you can execute the tests on it with:
DISPLAY=:1 ctest

Of course, you may also export the DISPLAY variable, or run a specific test executable, instead of using ctest.

Finally, once you no longer need the auxiliar display, just close its window. That will end the kwin and Xnest processes.


Unit testing a KDE 4 library


Here it is, my first English post!

I have still some things to finish (unrelated to KTutorial) before I can get a steady development pace, but until I finish them I’m going to make some sporadic work, mostly fixing problems (not necessarily bugs) to have a better base. For example, improving the unit tests.

I’m an advocate of testing before coding, but I can’t understand how or why in this project I had only 1 class and 4 methods tested. That had to change! So I entered in a test frenzy state and covered most of the code (KDevelop4 coverage plugin reports a healthy 91% coverage) with tests, which also helped to unveil some lurking bugs.

However, I’m not going to talk about the tests themselves, but about the infrastructure for them.

Executing tests without installing the library

When checking the first and (then) only test case I found that it passed when run using the shell wrapper, but not when run calling the executable directly.

Shell wrapper? What are you taking about? Ok, ok, let me explain. When you develop a library, you’re likely developing a dynamic library. That is, the applications using it will link to it when executed, not when compiled, and the library will remain as a separate file in the system.

In GNU/Linux the linker usually looks for the libraries in system directories. So if your test executable is linked against your dynamic library, the linker won’t find it until you install it. Apart from using system wide configuration, you can tell the linker other paths to look for the libraries with LD_LIBRARY_PATH environment variable.

And here is where the shell wrapper appears. The KDE 4 macros for CMake add, for each executable, a script which sets the needed paths in LD_LIBRARY_PATH for the executable to run and then calls it. This way, you can run your executable linked with dynamic libraries that are in the build directory but not installed yet.

But I needed the test executable to work even without the shell wrapper so I can run all the tests through CTest (more on this later). Unlike dynamic libraries, static libraries are linked at compiling time and added to the executable, so they aren’t needed separately.

So I thought “well, let’s create a ktutorial-static library to be used only by test executables and not even to be installed”. However, I wasn’t satisfied with that because the size of each test executable is much larger, specially in full debug mode, as each of them now contain the library. And I also felt that it was just a dirty workaround and there had to be a better solution.

And yes, there was. Reading the KDE 4 Macros documentation I found that KDE4_ADD_UNIT_TEST “adds a unit test […] (that) will be built with RPATH pointing to the build dir”. Mmm, what was that RPATH thing? CMake wiki gave me the answer: it was just what I was looking for. It’s like hardcoding LD_LIBRARY_PATH in the executable.

But, if the macro sets it, why didn’t it work? After some diving in KDE 4 Macros I concluded that it was a bug and opened RPATH isn’t set for tests added with KDE4_ADD_UNIT_TEST report. The bug is now fixed, so all the things explained here are superfluous, as now KDE4_ADD_UNIT_TEST works out of the box with dynamic libraries. Yes, you have wasted your time reading all this, sorry 😛

Running the tests

But, why do I need to execute the tests not using the shell wrapper? In KDE 4, the preferred unit testing library is QtTestLib. In QtTestLib each test case is an executable on its own, unlike other libraries where you can execute a test suite that groups several test cases. So you need a way to run all the tests without needing to run each test executable alone.

CMake provides its own test system, CTest. It has several fancy features, like sending the results of the tests to testing dashboards. But right now I just want to execute all the tests and see whether they pass or not. Fortunately, with CMake and CTest that’s a piece of cake.

CTest uses a CTestTestfile.cmake (or DartTestfile.txt, for legacy purposes) where it is specified the tests for its directory and subdirectories to look for other CTestTestfile.cmake. A CTestTestfile.cmake is created and filled by CMake in each build directory if ENABLE_TESTING command was run, which it is when including KDE4Defaults. CMake macro ADD_TEST adds a new test in the CTestTestfile.cmake of its directory. This macro is called from KDE4_ADD_UNIT_TEST, so using it effectively prepares CTest to run all the tests, even those in subdirectories of the base test directory. However, the tests are executed directly, not through their shell wrapper.

Well, I thought that. But… in fact, the generated CTestTestfile.cmake do use the shell wrapper. Since January 2008… But I was sooo smart to forget to update my KDE environment since then. Don’t ask… (Note, however, that this was written some months ago, although I didn’t publish it until now 😉 )

So now CTest works out of the box with KDE 4 Macros. We only have to build the tests (calling CMake with KDE4_BUILD_TESTS=ON variable they will be built with the rest of the project) and run CTest in our build/test directory. It will execute all the tests in that directory and, recursively, in all the needed subdirectories. If we need a more detailed output than the default summary, run ctest with -V or even -VV parameters to see the full test output.

Running QTest test from KDevelop4

A very interesting new feature of KDevelop4 is the support for unit test frameworks (although, as far as I know, only QTest works at this moment). With QTest plugin, you can enable and disable QTest tests to execute and see their results.

With some build systems, you must configure the plugin specifying the executables to be taken into account. However, if your project uses CMake, they are automatically discovered using CTestTestfile.cmake files (there is no problem even if they specify non-QTest tests, as these would be ignored).

There is only a little requeriment: there must be a CTestTestfile.cmake file in the root build directory. So just by using ENABLE_TESTING command in the root CMakeLists.txt file the CTestTestfile.cmake file is created and filled to recursively search in its subdirectories.

The only drawback with this approach is that CTestTestfile.cmake files will be created even in build directories where they aren’t needed (for example, src). However, although it is not the best solution, it is good enough 😉

Macro for adding several tests at once in CMakeLists.txt

In order to add a test, you need two or three CMake macros: KDE4_ADD_UNIT_TEST, TARGET_LINK_LIBRARIES and, optionally, SET (to set a variable with the list of files of the tests). As unit tests usually need just one file, you can avoid creating a variable and set the file in the KDE4_ADD_UNIT_TEST macro without losing legibility.

Anyway, the two calls are almost identical for every test, just changing the name of the test. It will be better to tell CMake just the name of the tests, instead of copy-pasting the calls for each test.

Of course, this can be easily done. As I have no idea about CMake programming, I took a look at KDECORE_UNIT_TESTS macro in CMakeLists.txt for kdecore and tweaked it for my own needs, as seen in KTutorial commit #24.