There is no EFL support in Maliit (and no Maliit support in EFL) but since there is wayland input method support for EFL available one can just use Maliit with EFL applications under Wayland. This shows one of the advantages of the Wayland input methods, that input method frameworks do not need explicit support for all kind of UI toolkits anymore.

The current plan is to get input method support into Wayland 1.2. Adding support for Wayland input method to Maliit allowed us to discover some missing features which we will work on next together with some more improvement suggestions to get the input methods really ready for Wayland 1.2.

To show the input methods working we have an editor and keyboard example client in Weston. But we also want to demo input methods in real world applications. As a start we implemented an input method module for the EFL (Enlightenment Foundation Libraries) toolkit.

The EFL input method module is available at github.

The move from Xorg to Wayland offers the possibility to move the input method system from the toolkit level down to the display server, as discussed in our Wayland Input Method System Proposal. Last week, Openismus let me work on a small prototype technical demo for Weston (the Wayland reference compositor) which implements some of the ideas in that proposal. This should be useful for testing the idea of having the input method system integrated in Wayland.

New Wayland interfaces

Standalone input method system with text and input method protocols

The prototype defines three interfaces. The first is the text_model, which is used to communicate from the application/toolkit side with the input method system. It provides the input method system with all required information about the active text, such as selection, surrounding text and cursor index. The application receives events over this interface, such as preedit and commit, from the input method system. For the prototype, I implemented an example editor client, which uses the text model to send the surrounding text to the input method system and receives commit events.

The second interface is the input_method interface which is used to communicate between the input method and Wayland. For the prototype we use a really simple keyboard client which just sends commit events when pressing one of the keys.

The third interface is used to register the keyboard surface as an input panel on the shell. I added the input_panel interface to the desktop shell protocol. This allows the shell to stack the keyboard surface into the right layer (for the prototype I used the panel layer). Additionally, the prototype adds support to the shell to show the keyboard only when a text model is active.

Demos

The screen cast shows the prototype in action, but for those with a working Wayland installation, it should also be possible to just compile and run the prototype.

Outlook

The prototype is the outcome of less than a week of coding, but it already uses some of the concepts we learned while working on Maliit. It should serve as a baseline for further Wayland input method system integration, even if many basic items are missing. For instance, there is no integration with hardware keyboard events, nor keyboard focus (which is handled by wl_seat in Wayland). Nevertheless I will polish my patches a bit and send them to the Wayland mailing list for integration into Weston, so that they can be used for more work on input methods in Wayland/Weston. My colleague Michael will present this work at aKademy in two weeks, which will be a good opportunity to discuss the possibilities of such an input method system in Wayland.

The development for Maliit, a cross-platform text input method framework for mobile devices, until 0.8 was mainly based on requirements for MeeGo Harmattan, the operating system on the Nokia N9. Harmattan applications are fullscreen. Animation of orientation changes is, together with other animations, done by the application. This is in contrast to Maemo Fremantle on the Nokia N900, were such animations were done by the X window manager.
When the virtual keyboard is up, the application remains fully interactive. Instead of using a proxy text entry, the text input goes directly to the application’s focused text entry. Showing the virtual keyboard should not change the layout, nor can it affect the application’s stability. The virtual keyboard runs in a separate process (maliit-server). But it requires animations showing, hiding and switching between different language layouts. In addition it should be possible to display overlays for word candidates or extended keys.
To satisfy these requirements the virtual keyboard window is shown in fullscreen mode. The visible parts of the virtual keyboard cover the bottom area of the application and screen, whereas the remaining area of the toplevel window is seemingly translucent. This keeps most of the application visible, including the focused text entry, which will render any text input immediately. It is also possible to pan the application’s page contents and to switch focus to another text entry, or remove focus altogether.

Shaped input region

To pass input events through to the application we use the shape extension of the X server and XFixesSetWindowShapeRegion to set the input shape of the MPassThruWindow to the region which the virtual keyboard and overlays really occupy. The X server will deliver events inside this region to the input method, and the rest to the application window.

void MPassThruWindow::updateInputRegion()
{
	XRectangle * const xrects = convertQRectanglesToXRectangles(region.rects());
	const XserverRegion shapeRegion =
		XFixesCreateRegion(display, rects, region.rects().size());
    	XFixesSetWindowShapeRegion(display, winid, ShapeInput, 0, 0, shapeRegion); 	XFixesDestroyRegion(display, shapeRegion);
	delete[] xrects;
}

Compositing Window Manager

RGBA window with translucent background

When the window manager supports compositing, it is possible to use 32 bit translucent RGBA windows for the overlay. Mcompositor, Harmattan’s window manager, supports this feature. That is why we turned the background of the top level virtual keyboard window translucent. In Qt this can be done like:

MPassThruWindow::MPassthruWindow()
{
	setAttribute(Qt::WA_TranslucentBackground);
}

Since compositing window managers are widely available for Linux desktops and the performance is acceptable it has been the default for Maliit 0.8.

Direct rendering

Compositing Window Manager

When the active application window is fullscreen and the application features a non-translucent background (which is the case on Harmattan), then the redirection of the active window into an off-screen window and additional compositing is unnecessary. Under circumstances, this redirection cuts the frame rate in half, which was unacceptable for animations and scrolling. For that mcompositor supports direct rendering of the active window into the frame buffer.
Since in this mode the keyboard window is transparent mcompositor cannot use the direct rendering optimization. Additional the context switches between the three processes, application, keyboard and compositor results in increased latency for text input, which affects the user experience. To reduce latency we wanted to further optimize performance for Harmattan and the N9 by using direct rendering with a 16 bit window for the keyboard.

Compositing using shaped output region

Our first approach used the same technique for the in- also for the output, with the help of XFixesSetWindowShapeRegion:

void MPassThruWindow::updateInputRegion()
{
	...
    	XFixesSetWindowShapeRegion(display, winid, ShapeBounding, 0, 0, shapeRegion);
	...
}

This solution does not require a compositing window manager but made it difficult to use shadows and overlays. It also was not performing as well as expected, since there is no support for direct rendering of shaped window. A better solution had to be found.

Compositing in Maliit

Compositing in Maliit

We were able to try another approach for using direct rendering. A 16bit RGB window together with the composite and damage X extensions is used to compose the content of the application window into the background of the virtual keyboard window. Instead of using the window manager for the compositing, Maliit handles the compositing itself.
The application window is represented through MImRemoteWindow in Maliit. XCompositeNameWindowPixmap is used to get a pixmap of the application’s window content:

void MImRemoteWindow::setupPixmap()
{
	xpixmap = XCompositeNameWindowPixmap(display, winId);
	pixmap = QPixmap::fromX11Pixmap(xpixmap, QPixmap::ExplicitlyShared);
}

This pixmap is blitted into the background of a QGraphicsView:

void GraphicsView::drawBackground(QPainter *painter, const QRectF &rect)
{
        painter->drawPixmap(rect.toRect(), remoteWindow->pixmap(), rect.toRect());
}

To get notified about updates in the application window the damage extension is used:

void MImRemoteWindow::setupDamage()
{
    damage = XDamageCreate(display, winId, XDamageReportNonEmpty); 

}

On damage events there is a signal emitted, which is used to trigger a repaint.

void MImRemoteWindow::handleDamageEvent(XDamageNotifyEvent *e)
{
    XserverRegion parts = XFixesCreateRegion(display, 0, 0);
    XDamageSubtract(display, e->damage, None, parts); 

    QRegion region(convertXRegionToQRegion(parts)); 

    XFixesDestroyRegion(display, parts); 

    Q_EMIT contentUpdated(region);
}

RGB window with application content composited into the background

When handing over compositing from the the window manager to the input method process brief flicker can occur. To suppress it, mcompositor had to be patched to only map the virtual keyboard window after the application window has gotten redirected for compositing in Maliit. Additional QWidget attributes need to be set for all QWidgets in the virtual keyboard:

MPassThruWindow::MPassthruWindow()
{
	setAttribute(Qt::WA_OpaquePaintEvent);
	setAttribute(Qt::WA_NoSystemBackground);
}

In addition to improving the performance this approach also allowed to synchronize the rotation animation of the application and keyboard.
This mode is used on Harmattan and the Nokia N9 and can be activated by starting maliit-server with the -use-self-composition flag.

Plugins

Since plugins up until Maliit 0.8 are using fullscreen sized children widgets of the MPassThruWindow, their sizing and positioning depends on the toplevel window being a fullscreen window.
For compositing in Maliit to work, those widgets have to blit the application window content into their background. A QGraphicsView for example can use MAbstractInputMethodHost::background to paint the application window content as its background in QGraphicsView::drawBackground.

Conclusion

The fullscreen keyboard window approach was the proper choice for the Nokia N9 device. But on other devices, where orientation change animations are done by the compositor, or on a desktop system without a compositing window manager a non-fullscreen keyboard window could be a better choice. Unfortunately input method plugins based on Maliit 0.8 releases are tied to the fullscreen keyboard window concept and even need to take care for implementation details like the compositing in Maliit. The Maliit 0.9 series, which provide the basis for a first stable 1.0 release, should be used to find a better solution for plugins which allows different window modes and makes it easier for plugin developers by not burden them with any implementation details.

Luckily there is already a simple platform input context for the Maliit input method framework in the Qt code base available. There are some smaller bugs open, but they should be easy to fix. So it should be already possible to use Maliit with Qt 5 applications.

Additionally we would of course like to compile Maliit itself against Qt 5 too. Based on the previous work eliminating the X requirement in Maliit it was a quite easy task to compile it with Qt 5 and required just fixing some smaller bugs and updating some build system files.

After that I tried to run Maliit with the XCB platform. This showed an ugly bug, the virtual keyboard stole the focus from the application. On Qt 4 we used the WA_X11DoNotAcceptFocus Qt::WidgetAttribute which does not work for the XCB (and Xlib) platforms in Qt 5. I filed a bug about this issue and created a patch (which is sadly only visible when one has a gerrit account registered). With that patch applied to Qt 5 Maliit was basically working.

Still our target is to support also non X platforms like Wayland, which is currently planned to be used in Meego in future (see Wayland in Meego 1.3). As a first step in this direction we made X optional in Maliit. Than as a second step we implemented basic support for Qt 4.8 Lighthouse. Qt Lighthouse provides support for different backends like for example Android, xcb and Wayland. Compiling Maliit with that Qt version allows it now to run Maliit on Wayland. Next step is to implement better support for Maliit in Wayland like a window management policy.

To install the packages:

1. Download the .repo file and save it as /etc/yum.repos.d/maliit.repo.
2. Install the packages meego-inputmethodkeyboard and meego-inputmethodbridges with the Add/Remove Software application (or with yum)

To try it out, run as a user

1. meego-im-uiserver
2. GTK_IM_MODULE=meego-im gedit

This packages contains the vanilla versions of Maliit and the default theme (Meego). The custom GNOME 3 theme (see Customizing the MeeGo Keyboard) as it was shown by Jon Nordby on a tablet needs to be still packaged.

For more information see the Maliit Wiki.