Here it is, my first English post!
I have still some things to finish (unrelated to KTutorial) before I can get a steady development pace, but until I finish them I’m going to make some sporadic work, mostly fixing problems (not necessarily bugs) to have a better base. For example, improving the unit tests.
I’m an advocate of testing before coding, but I can’t understand how or why in this project I had only 1 class and 4 methods tested. That had to change! So I entered in a test frenzy state and covered most of the code (KDevelop4 coverage plugin reports a healthy 91% coverage) with tests, which also helped to unveil some lurking bugs.
However, I’m not going to talk about the tests themselves, but about the infrastructure for them.
Executing tests without installing the library
When checking the first and (then) only test case I found that it passed when run using the shell wrapper, but not when run calling the executable directly.
Shell wrapper? What are you taking about? Ok, ok, let me explain. When you develop a library, you’re likely developing a dynamic library. That is, the applications using it will link to it when executed, not when compiled, and the library will remain as a separate file in the system.
In GNU/Linux the linker usually looks for the libraries in system directories. So if your test executable is linked against your dynamic library, the linker won’t find it until you install it. Apart from using system wide configuration, you can tell the linker other paths to look for the libraries with LD_LIBRARY_PATH environment variable.
And here is where the shell wrapper appears. The KDE 4 macros for CMake add, for each executable, a script which sets the needed paths in LD_LIBRARY_PATH for the executable to run and then calls it. This way, you can run your executable linked with dynamic libraries that are in the build directory but not installed yet.
But I needed the test executable to work even without the shell wrapper so I can run all the tests through CTest (more on this later). Unlike dynamic libraries, static libraries are linked at compiling time and added to the executable, so they aren’t needed separately.
So I thought “well, let’s create a ktutorial-static library to be used only by test executables and not even to be installed”. However, I wasn’t satisfied with that because the size of each test executable is much larger, specially in full debug mode, as each of them now contain the library. And I also felt that it was just a dirty workaround and there had to be a better solution.
And yes, there was. Reading the KDE 4 Macros documentation I found that KDE4_ADD_UNIT_TEST “adds a unit test [...] (that) will be built with RPATH pointing to the build dir”. Mmm, what was that RPATH thing? CMake wiki gave me the answer: it was just what I was looking for. It’s like hardcoding LD_LIBRARY_PATH in the executable.
But, if the macro sets it, why didn’t it work? After some diving in KDE 4 Macros I concluded that it was a bug and opened RPATH isn’t set for tests added with KDE4_ADD_UNIT_TEST report. The bug is now fixed, so all the things explained here are superfluous, as now KDE4_ADD_UNIT_TEST works out of the box with dynamic libraries. Yes, you have wasted your time reading all this, sorry :P
Running the tests
But, why do I need to execute the tests not using the shell wrapper? In KDE 4, the preferred unit testing library is QtTestLib. In QtTestLib each test case is an executable on its own, unlike other libraries where you can execute a test suite that groups several test cases. So you need a way to run all the tests without needing to run each test executable alone.
CMake provides its own test system, CTest. It has several fancy features, like sending the results of the tests to testing dashboards. But right now I just want to execute all the tests and see whether they pass or not. Fortunately, with CMake and CTest that’s a piece of cake.
CTest uses a CTestTestfile.cmake (or DartTestfile.txt, for legacy purposes) where it is specified the tests for its directory and subdirectories to look for other CTestTestfile.cmake. A CTestTestfile.cmake is created and filled by CMake in each build directory if ENABLE_TESTING command was run, which it is when including KDE4Defaults. CMake macro ADD_TEST adds a new test in the CTestTestfile.cmake of its directory. This macro is called from KDE4_ADD_UNIT_TEST, so using it effectively prepares CTest to run all the tests, even those in subdirectories of the base test directory. However, the tests are executed directly, not through their shell wrapper.
Well, I thought that. But… in fact, the generated CTestTestfile.cmake do use the shell wrapper. Since January 2008… But I was sooo smart to forget to update my KDE environment since then. Don’t ask… (Note, however, that this was written some months ago, although I didn’t publish it until now ;) )
So now CTest works out of the box with KDE 4 Macros. We only have to build the tests (calling CMake with KDE4_BUILD_TESTS=ON variable they will be built with the rest of the project) and run CTest in our build/test directory. It will execute all the tests in that directory and, recursively, in all the needed subdirectories. If we need a more detailed output than the default summary, run ctest with -V or even -VV parameters to see the full test output.
Running QTest test from KDevelop4
A very interesting new feature of KDevelop4 is the support for unit test frameworks (although, as far as I know, only QTest works at this moment). With QTest plugin, you can enable and disable QTest tests to execute and see their results.
With some build systems, you must configure the plugin specifying the executables to be taken into account. However, if your project uses CMake, they are automatically discovered using CTestTestfile.cmake files (there is no problem even if they specify non-QTest tests, as these would be ignored).
There is only a little requeriment: there must be a CTestTestfile.cmake file in the root build directory. So just by using ENABLE_TESTING command in the root CMakeLists.txt file the CTestTestfile.cmake file is created and filled to recursively search in its subdirectories.
The only drawback with this approach is that CTestTestfile.cmake files will be created even in build directories where they aren’t needed (for example, src). However, although it is not the best solution, it is good enough ;)
Macro for adding several tests at once in CMakeLists.txt
In order to add a test, you need two or three CMake macros: KDE4_ADD_UNIT_TEST, TARGET_LINK_LIBRARIES and, optionally, SET (to set a variable with the list of files of the tests). As unit tests usually need just one file, you can avoid creating a variable and set the file in the KDE4_ADD_UNIT_TEST macro without losing legibility.
Anyway, the two calls are almost identical for every test, just changing the name of the test. It will be better to tell CMake just the name of the tests, instead of copy-pasting the calls for each test.
Of course, this can be easily done. As I have no idea about CMake programming, I took a look at KDECORE_UNIT_TESTS macro in CMakeLists.txt for kdecore and tweaked it for my own needs, as seen in KTutorial commit #24.