Posts Tagged ‘kross’

Internationalization in Kross scripts

2010/01/04

As I said in the previous post, internationalization in Kross scripts is a subject that deserves its own post. Again, this is based on my own experiences, and I’m by no means an expert in script languages or Kross, so don’t treat this information as a universal truth 😉

Also note that internationalization (from now on, i18n) has a very broad scope. Here we’ll talk only about string i18n (allowing the messages to be translated to other languages). If you are interested in other i18n areas you’ll have to keep looking 😉

So let’s see if I can give some useful information after all 😉

Translation systems

First of all, what systems do we have to translate strings? The de facto standard in free software world is GNU gettext. It provides tools to extract the messages to be translated from the source code, a file format to store the translations, a library to translate the strings at runtime… Most popular programming languages support gettext, being it through its standard API (like in Python) or from 3rd party projects (like in Ruby).

But it is not the only system available. For example, Qt has its own translation system as part of its i18n support. Ruby also has an i18n module that contain a translation system (although it can use gettext files switching the backend).

What does KDE use? Although KDE uses a lot of Qt infrastructure, in the case of string translation it uses gettext instead. Well, an enhanced version of gettext with some very interesting features. For example, the semantic markup helps translators to better understand how a string should be translated, but it also benefits the users as it provides a richer and more consistent appearance for messages. Another interesting feature is Transcript, which aids in the correct translation of case based languages (among others).

Moreover, KDE libraries take care of gettext initialization, catalogs and so on, so you just have to mark the strings to be translated without worrying about setting everything up.

What translation systems can be used from Kross scripts? Thanks to the Translation Module, the KDE translation system can be used in Kross scripts, no matter what programming language the script is written in. But, as Kross only acts as a bridge between the script interpreter and C++ code, the translation systems available for the programming language of the script can be used from Kross. So the i18n module in Ruby or gettext in Python could be used. And if Qt bindings are available for the language, even the Qt translation system could be used!

Anyway, in my humble opinion, the best choice is to use Kross Translation Module (if we know for sure that the script will be executed from Kross). The Translation Module is available no matter which programming language was used to write the script, and it is provided by Kross, so it doesn’t depend on external libraries. Moreover, as it just forwards the function calls to the KDE translation system, all the fancy features given by it are available for free in the scripts. And, finally, the messages can be extracted from the scripts like they are extracted from C++ sources, storing them in the same file (so translators don’t have to care where the strings came from) and accessing them at runtime without any special configuration.

Message extraction

Translators need to know what strings they have to translate, and in order to accomplish this the translatable strings must be extracted from the source code. To do this, an extraction utility analyzes the source code looking for those strings. The programmer must mark the translatable strings so the extraction utility knows which strings have to be extracted and which not.

To mark the string usually just means to wrap the string with a function/method call that will also translate that string to the appropriate language when the application is running.

To extract the strings, gettext provides xgettext, which understands source code in a lot of programming languages, including Python. However, it doesn’t support yet Javascript nor Ruby explicitly. When a programming language is unknown, xgettext tries (as far as I know) with C like strings and functions. So, although neither Javascript nor Ruby are explicitly supported, double quote C like strings can be extracted apparently without problems. But it doesn’t support single quote strings, custom delimited strings or function calls without parenthesis.

See the following Ruby code as an example (note that mark is just an example function name):

mark("Hello world double quotes") # Extracted by xgettext
object.mark("Hello world method") # Extracted by xgettext

mark('Hello world single quotes') # Not extracted by xgettext
mark(%/Hello world custom quotes/) # Not extracted by xgettext
mark "Hello world without parenthesis" # Not extracted by xgettext

xgettext has also a very useful feature: it let’s you specify how the strings were marked in the source code. So it supports the default gettext function names, but can also extract custom functions. For example, you can say “wherever you find a function called i18nc with 2 arguments, treat the first item as string context and the second argument as the string to be translated”. This is used by KDE, as KDE i18n functions have a different name than gettext canonical functions.

Note that the file created by xgettext when the messages are extracted uses a format understood by gettext (although other translation systems may be able to read those files). Anyway, it is not a problem using Kross Translation Module, because as it was already said, KDE uses (an enhanced) gettext as its translation system.

In fact, the extraction of strings in KDE is done with xgettext, and it is automatic if the application is part of KDE’s subversion repository, although it can also be simplified a lot with an script for 3rd party applications, as shown in i18n Build System. However, the Messages.sh for subversion and 3rd party applications given in that article only support C/C++ files. They must be adjusted to also look for messages in Kross scripts. Take a look to the changes in KTutorial commit #65 as an example.

The gettext project for Ruby provides rgettext, which is a xgettext tailored for Ruby source code. However, it just works for Ruby source code, and as far as I can tell it doesn’t allow you to specify how the strings are marked (maybe with -r argument you can extend the behavior of rgettext, but I haven’t tested it), so you can just use the canonical gettext function names. That is, it can’t extract messages from a Ruby script that uses Kross Translation Module (as the function names are the same used in KDE libraries).

So, in the end, in order to be able to extract messages from Kross scripts using the translation module you just have to include the scripts in the files to be looked for strings, and keep a C/C++ style in Javascript and Ruby strings and translation module calls.

Runtime translation, catalog location…

Another section that starts with bold letters… It means that there is a lot to say about this subject, right? Wrong. That’s one of the nicest things about using the translation module: once the messages from the scripts are extracted along with the messages from C++ code you are done. They will be treated like the C++ internationalized strings when translated, merged, installed and used at runtime. Do the string translation already work in C++? So it will for Kross scripts 🙂

Well, I lied. There is one case where this is not so easy: when the scripts aren’t part of the application. If an script is provided as an add-on for the application by 3rd parties, the messages from that script weren’t probably extracted with the rest of messages of the application. So the script must provide also its own translation files like additional data (for example, packaging the script and the translations in a tar.gz file or something like that). The translation files would need to be loaded at runtime, which would require some infrastructure.

Anyway, I haven’t explored this scenario as it seems very strange, at least for tutorials. If someone makes a nice scripted tutorial for an application that tutorial is likely going to end as part of the application itself. It would be very strange if the tutorial was kept as a separate entity, and even more strange if it got its own translations (getting localized seems like a sign that it is good enough to become part of the application 😉 ).

Some notes about Kross

2009/09/27

While experimenting with Kross in order to implement scripted tutorials I collected a series of notes about what can and what can not be done with Kross. Although they are based on my own experiences (testing what happened if I changed things, reading Kross code and debugging it), I consider them to be pretty accurate (at the time of this writing, in the future it may change). But if you use this information for your own code and your computer explodes, don’t blame me 😛

Also take into account that I have tested this with Javascript and Python backends. I have no idea about how mature are other backends.

And now, let’s see those notes:

  • Before Qt 4.5, only slots could be invoked from scripts. From Qt 4.5 and on, slots and methods marked with Q_INVOKABLE macro (which was introduced in Qt 4.5) can be invoked. Kross probably uses Qt Meta-Object System internally to make its magic.
  • Qt does not support Q_OBJECT classes that use templates. So, as you could expect, you can’t use templated classes or templated methods from scripts. Maybe playing with inheritance and method overriding it can be done… but I haven’t tested it.
  • It seems that in C++ classes you can’t have two slots (or invokable methods) with the same name and the same number of arguments (of different types). That is, it seems that Kross doesn’t call the appropriate slot based on the data types passed as arguments, but the first defined slot is always called. However, it only happens then the type of arguments is different, but their number is the same. If the name of the slots is the same but the number of arguments is different, Kross calls the slot with the matching number of arguments. So it supports a partial overloading of slots and invokable methods in C++ classes. Yes, even calling them from Python!
  • In Qt 4.5 the method newInstance was added to QMetaObject, which as you can expect creates a new instance of the QObject subclass associated to the QMetaObject (if the constructor to use is marked as Q_INVOKABLE). However, Kross doesn’t seem to be able (at least yet) to create new objects using invokable constructors. Something similar can be got if, in the C++ code, a method is implemented that given the name of a class returns an object of it. For this, the class names have to be previously associated with their QMetaObject, as Qt doesn’t seem to have a way to get a QMetaObject for the given class name. However, like invokeMethod, newInstance uses Q_ARG for its arguments, and being a macro it receives its values hardcoded, so I don’t see a way to pass parameters from scripts. Nevertheless, a constructor without arguments can be made, finishing the proper object initialization using set methods, or an init method, or something like that. It is not the best solution, but it is good enough. An example of all this can be seen in KTutorial commit #50, where (among other things) creating WaitFors from scripts was added to ScriptModule.
  • Apart from simple types and objects from QObject and child classes there are other data types that can be used in scripts: those supported by QMetaType. These data types are normal classes, not inheriting from QObject. In fact, it seems that data types supported by QMetaType aren’t designed to be classes inheriting from QObject, as classes inheriting from QObject can’t implement a copy constructor, while QMetaTypes need a copy constructor to be registered using qRegisterMetaType. Not being QObjects (thus lacking properties or slots), how they can be used from a script depends on the backend being used. For example, in Javascript backend “new QColor(255, 255, 255)” can be done, and also calling methods like red(), while in Python backend the QColor are just plain strings like “#ffffff”. A good example of what can be done and what can’t be done with each language can be seen looking in the unit tests of each Kross backend.
  • When passing lists and dictionaries to C++ methods, the arguments have to be declared as QList and QMap (QHash doesn’t seem to be supported) storing QVariant (QVariantList and QVariantMap). If, for example, QMap<QString, int> is used, the method call is done successfully, and in the C++ method even the number of values stored in the map can be got. But when the values are tried to be got, it crashes, even if the type of the data matches those declared in the template. May it be a bug?
  • The pointer to simple types (like int*) fail when used as C++ method arguments. Instead of passing a pointer to the integer, the pointer itself contains the value of the integer. It seems that only pointers to QObject (and derived classes) are implemented.
  • Naturally, SIGNAL and SLOT macros don’t work in Kross scripts (as they are a C++ preprocessor matter). So Kross provides a special connect function (or method, depending on the backend) which just uses the names of the signals and slots, and it is not necessary to wrap them with the macros. However, what does it happen when you need to use the macros in other places that are not the typical connect (for example, because you do a connect internally in a method using the name of the signal or slot passed to it)? Looking through Qt code I saw that those macros just add “1” in the case of SLOT and “2” in the case of SIGNAL to the front of the string. So to solve this, just add internally in the method to which the names of signals or slots are passed the appropriate number at the front of the signature and you are done. Take a look to KTutorial commit #55 as an example. Also take into account that if the name of the signal or slot to connect to is passed from the script to a C++ method, the method can’t be declared as receiving a const char* argument, but a QString.
  • Objects from classes created in scripts doesn’t seem to be compatible in any way with C++ code. I hoped that the interpreter made some kind of magic and “translated” them to QObject inherited classes, allowing them to be used in C++ methods receiving QObjects, but it’s not the case. As QObject class is neither available in scripts, classes can’t be made to inherit from it. However, this can be workarounded to some extent playing with signals. It can be done with a base class that emits signals with the object itself as argument in those methods that need to be overriden in child classes. The signals are connected to a function wrote in the script, and that function does the things that would have been done in the overriden method. See, for example, the test method stepWithCustomSetup in ScriptingTest and the ScriptedTutorial class.
  • Both modules and QObjects added to Kross actions (using Kross::Manager::self().addObject() the objects can be added to all the Kross actions) can be used to feed the scripts with C++ objects. In the scripts, modules will be loaded using Kross.module(“moduleName”), whereas QObjects can be used directly with the name they were gave (in the examples of the tutorials the objects are imported when Python is used, but at least based on the tests I made it isn’t necessary).  Personally, I find adding the QObjects to use way easier than using modules (at least, with KTutorial current design).
  • In the case of Python, the methods added by Kross (and which naturally can be called) can be seen in kdebindings/python/krosspython/pythonextension.cpp, in PythonExtension constructor. Those methods are, among others, className() or slotNames().
  • Kross provides a translation module that makes possible internationalize scripts and localize them using .po files like the rest of KDE. The module is Kross::TranslationModule and offers the typical functions i18n, i18nc, etcetera. However, I still have to check how string extraction works in this case, and how to localize a script added a posteriori, for example, downloaded via “Get Hot New Stuff!”. All this suggests that this matter will be worth its own post, and here it is: Internationalization in Kross scripts.
  • In the classes of the C++ objects used in scripts, the arguments of the signals and the return values of the slots that are not declared as pure QObject* but as QObject subclasses don’t work. On the other hand, the arguments of the slots can be QObject subclasses without any problem.
  • WrapperInterface class makes possible to use in scripts objects of classes that aren’t QObject*, nor QWidget*, nor any of the data types supported by QVariant. Unfortunately, although WrapperInterface is something at a generic level in Kross, it isn’t assured that it will work, as it depends on the backend (and thus, the language) used. The Javascript backend doesn’t seem to support this system, although the Python one does.
  • If we know that the scripts will only be written in languages which backend support WrapperInterface, it can be done the arguments of the signals and the return value of the slots of the C++ classes not to need to be declared as QObject*, but being able to be declared as the concrete subclass they belong to. For this, a handler for each class should be added, as shown below:
    QVariant SomeQObjectSubclassHandler(void* ptr) {
    QVariant r;
    r.setValue( static_cast(ptr) );
    return r;
    }
    void Scripting::initialization() {
    ...
    Kross::Manager::self().registerMetaTypeHandler("SomeQObjectSubclass*", SomeQObjectSubclassHandler);
    ...
    }
  • At first, I happily believed that the functions defined in scripts would be magically translated to slots of some object and that in C++ code a signal could be connected to a function in the script using that object. That, unfortunately, doesn’t happen (take into account that signals can be connected to functions, but in the script, not in C++ code as I needed). However, as script functions can be called from C++ code, it can be workarounded to some extent calling the corresponding function when a signal is emitted. An example can be seen in KTutorial commit #53 (although what I needed was under a pretty controlled situation, where signals included the emitter as their only argument).