Text Input Method Support in Wayland

Over the last years, we at Openismus have worked on input methods for mobile devices (see for example the Maliit input method framework). During that work we noticed that there are several problems caused by the toolkit-level interface between the input method system and the applications (beside having to implement interfaces for every toolkit, like GtkIMContext and QInputContext). One of this problems is synchronizing events happening to the focus window and the virtual keyboard window. Therefore good platform integration of input methods such as virtual keyboards requires additional window management policies, as the window manager has to gain knowledge about input methods. The compositing window manager on Nokia’s N9 serves as an example for the amount of required integration. It also taught us about what can go wrong and that Xorg induced latency is a hard problem to solve (see Compositing in Maliit). There is also the XIM X extension which integrated input methods on the X level, but its limitations and complexity puts it as a second choice compared to the toolkit interfaces mentioned above (especially since it does not manage to solve there problems).

The move from Xorg to Wayland offers the possibility to move the input method system from the toolkit level down to the display server, as discussed in our Wayland Input Method System Proposal. Last week, Openismus let me work on a small prototype technical demo for Weston (the Wayland reference compositor) which implements some of the ideas in that proposal. This should be useful for testing the idea of having the input method system integrated in Wayland.

New Wayland interfaces

Standalone input method system with text and input method protocols

The prototype defines three interfaces. The first is the text_model, which is used to communicate from the application/toolkit side with the input method system. It provides the input method system with all required information about the active text, such as selection, surrounding text and cursor index. The application receives events over this interface, such as preedit and commit, from the input method system. For the prototype, I implemented an example editor client, which uses the text model to send the surrounding text to the input method system and receives commit events.

The second interface is the input_method interface which is used to communicate between the input method and Wayland. For the prototype we use a really simple keyboard client which just sends commit events when pressing one of the keys.

The third interface is used to register the keyboard surface as an input panel on the shell. I added the input_panel interface to the desktop shell protocol. This allows the shell to stack the keyboard surface into the right layer (for the prototype I used the panel layer). Additionally, the prototype adds support to the shell to show the keyboard only when a text model is active.

Demos


The screen cast shows the prototype in action, but for those with a working Wayland installation, it should also be possible to just compile and run the prototype.

Outlook

The prototype is the outcome of less than a week of coding, but it already uses some of the concepts we learned while working on Maliit. It should serve as a baseline for further Wayland input method system integration, even if many basic items are missing. For instance, there is no integration with hardware keyboard events, nor keyboard focus (which is handled by wl_seat in Wayland). Nevertheless I will polish my patches a bit and send them to the Wayland mailing list for integration into Weston, so that they can be used for more work on input methods in Wayland/Weston. My colleague Michael will present this work at aKademy in two weeks, which will be a good opportunity to discuss the possibilities of such an input method system in Wayland.

8 thoughts on “Text Input Method Support in Wayland

  1. csslayer

    Interesting, but how can we suffer from protocol limitation, or shall we use a more extensive way?

    Having a general protocol is great, thus some toolkit that don’t using gtk or qt can use this general protocol, but still I see some limitation.

    Custom protocol mean input method can run a custom module inside the application, which will give input method a lot of opportunity to implement some very custom thing.

    Maybe some “key-value” like interface can be added thus input method can ask for some more information and application can decide what info to support.

    Reply
  2. csslayer

    Interesting, but I hope it can be more extensive.

    Custom protocol have some benefit, it run inside the application, thus it can provide quite a lot info which a fixed protocol cannot, and the control is under the IMF developer. For example, maybe input method want to know the window position, or the process info (which is not quite common, but can be useful).

    Maybe a key-value like query should be added? For example, input method tells what it would like to know, and the client send the info via a packed-key-value-map. Thus it would more extensive from my point of view.

    Reply
  3. Pingback: Murray’s Blog » Blog Archive » Openismus Status

  4. mikhas

    csslayer, the protocols here are heavily stripped down, the bare minimum really to show a VKB and get some text input into the application. If you check out the full proposal, you will notice that the actual protocols we have in mind are more sophisticated. Since you have experience with Chinese input methods, now would be a good time to check out the proposal and point out where the protocols are lacking.

    Reply
  5. Pingback: Text Input Method Support in Wayland ‒ Update | Jan Arne Petersen's Blog

  6. Pingback: EFL integration for Wayland input methods | Jan Arne Petersen's Blog

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong> <pre lang="" line="" escaped="" cssfile="">