Binaries for POT 5.5.0-beta1 on Qt 5.8.0-rc1 and Experimental Accelerated QtWebKit

A new version of Qt is out and some interesting changes are being developed. Interesting work was done on WebEngine. This package contains an experimental build of Qt 5.8.0-rc1 with the POT driver to provide hardware acceleration to video decoding and rendering. POT was updated to build on the new plugin architecture of Qt and includes ffmpeg 3.2.2. For info on the content of the package refer to this article. The youtube player still runs on WebKit 1 and so is still hardly useful.

Please report issues on github. Have fun ;-)

Download POT 5.5.0-beta1 for Raspbian Jessie Lite Pi2 (also tested on Pi3) here (md5: 7ca1b961ab8c70176f6aeb13d0bc4f9a).

Binaries for POT 5.3.0-beta1 on Qt 5.6.0 and Experimental Accelerated QtWebKit

This is an experimental version of POT that includes a few new features. The driver now includes an experimental version of playback of remote content including proper buffering of the media and the QtWebKit library is able to play videos using proper hardware acceleration.

Features

The driver can now be used through the regular Qt/QML interface to play content over protocols different from file://. I only tested HTTP but others should be available.
This new feature allowed me to also implement a new player in QtWebKit using the POT driver to stream content to the regular Qt QWebView. Unfortunately I could only find the time to implement this in the WebKit 1 branch, not in WebKit 2 which requires more work. This makes the implementation hardly useful, but it is a start. If anyone wanted to contribute this is where to start. This video shows the new implementation in action:

As QtWebKit now uses the GPU properly, I could also implement a YouTube player app using the same QWebView with just a few lines of code of regular Qt code:

In the video you can also see an application I wrote to implement a YouTube fullscreen player using the YouTube iframe API. Such a sample app is also included in the package.

Of course the real target would be to implement this in QtWebEngine. I was never able to find a version of QtWebEngine with video acceleration for Pi. Anyone who knows one?

Package

The package now includes:

• build_valgrind.tar: a build of valgrind to analyse your code;
• libpiomxtextures_qmlutils.so: QML plugin to provide a video probe in QML;
• piomxtextures_browser_we: an application that tests the QtWebEngine module (run passing a URL, ./piomxtextures_browser_we http://www.youtube.com);
• piomxtextures_browser_wk: an application that tests the QtWebKit module:
• you can run on WebKit 1 like ./piomxtextures_browser_wk http://www.youtube.com;
• or you can run on WebKit 2 like ./piomxtextures_browser_wk --wk2 http://www.youtube.com;
• piomxtextures_pocplayer: a sample QML player;
• piomxtextures_pocplayer_yt: a youtube sample player (run passing a video ID like ./piomxtextures_pocplayer_yt 71UvXMzVgx4);
• qtdeps.tar: the usual libs needed by Qt and POT;
• Qt-rasp2-5.6.0.tar: a build of Qt 5.6.0 stable including:
• regular Qt 5.6.0 modules;
• untested bluetooth module including BLE support;
• untested MySQL plugin;
• untested QtFtp module;
• untested Qt hat tools module;
• QtWebEngine;
• QtWebKit.

Please report any bug you find on github. Don't use in production. Sources will be available shortly. Have fun! ;-)

Download POT 5.3.0-beta1 for Raspbian Jessie Lite Pi2 (also tested on Pi3) here (md5: edbe0ad2a552a5c8280e3876d16b237d).

Hardware Acceleration on WebKit 1, WebKit 2 and QtWebEngine with Qt on Raspberry Pi 2

There are a few solutions to get a browser with good performance on Raspberry Pi 2, but I wanted to test solutions that provide this to a Qt app on OpenGL without X11. The two main options I know are the "old" QtWebKit and the new QtWebEngine. I know that there are a couple of ways to use QtWebKit, one is with the first implementation, WebKit 1, the other is through WebKit 2. Also, it is possible to use WebKit 1 using the regular software raster paint engine and the OpenGL paint engine, which is the one I used in the previous article to implement the HTML5 video feature.
I therefore wanted to have a look at the performance of the animations (I'm not talking about WebGL here), so here it is; the video shows three samples (thanks to the authors!) run on each of these alternatives: WebKit 1 on raster, WebKit 1 on OpenGL, WebKit 2 and then QtWebEngine.

As you can see WebKit 1 on raster is completely unusable and WebKit 1 on OpenGL is clearly better, but still not very good. Also you can see how the pointer moves: the main thread is too busy and user interaction is difficult. WebKit 2 and QtWebEngine solve this issue and improve performance.

Hands-on WebKit and HTML5 Media Elements on Raspberry Pi

I tried to do this a few months ago actually, but I got lost in the huge amount of code and architecture variants of WebKit. Now I convinced myself it would be too interesting to see it working. I don't have much spare time, so I wanted to start with the path of least-resistance and then, depending on the time available, continue climbing. This activity is very time consuming, so I do not know where I'll get.

This is however the first stage: using POT to implement 1080p hardware accelerated zero-copy video playback in the Qt port of WebKit1. No QML here. Of no use, but it somehow works. I read there are far better results out there, but I learned much by doing this, and I had fun :-)
Of course, as you can see, there is much much work being done in the UI thread. JavaScript is mostly run in the main thread, making the framerate pretty bad on heavy websites. That is why WebKit2 is there after all :-)

Demo

Bye! ;-)

Bring up Qt 5 on Raspberry Pi with Wayland

Ok, I've been waiting to do this for quite some time but never had the time to actually do it. I tried this quickly twice but without success because of many issues. Now I invested some hours and made it to the end of the journey :-)
I therefore try to describe here the steps to make Qt 5.0.1 (the current version in the Qt git) on the new wheezy image with Wayland support.

Building the Qt Fundamental Modules

Of course the procedure is almost identical to the one used for Qt 5.0 that I described here. I only did a couple of things to speed up the process, you choose how to do it. I briefly describe here some of the steps.

1. Download the latest available wheezy image from the Raspberry Pi website: http://www.raspberrypi.org/downloads.
2. Uncompress the image and flash it to your SD card.
3. Boot the new image on your board.
4. Install some libs that we will need (I won't compile the xcb platform plugin here, so in this case I decided not to install those xcb libs):
   
   $ sudo apt-get install libdbus-1-dev libudev-dev libssl-dev
   $ sudo apt-get install libasound2-dev
   $ sudo apt-get install libgstreamer0.10-dev libgstreamer-plugins-base0.10-dev libgstreamer-plugins-bad0.10-dev
   $ sudo apt-get install libffi-dev libpixman-1-dev
   $ sudo apt-get install libsqlite3-dev libicu-dev libfontconfig1-dev
   
   libdbus-1-dev is used to get the QtDBus module compiled from qtbase, libudev-dev to get udev support, libssl-dev for OpenSSL and libasound2-dev will provide Qt what it needs for ALSA support.
   GStreamer libs instead are mainly used in the qtmultimedia and qtwebkit modules. If the environment is setup correctly for gstreamer support, then the configure script will report the success.
   libffi-dev libpixman-1-dev are needed to compile the qtwayland module or its dependencies. libsqlite3-dev libicu-dev and libfontconfig1-dev instead are needed only if you intend to use QtWebKit.
5. Instead of the loopback mount of the image on your system to get a correct sysroot, I quickly scp'ed the needed binaries from my board to a newly created sysroot. In particular I copied:
   • /lib
   • /usr/lib
   • /usr/include
   • /opt
   I'll refer to the directory containing all of this as rasp_sysroot. Quick and dirty. You might also consider using rsync though.
   As a final note on this I have to say that scp has the somehow pleasant collateral effect of following the symlinks in libs.
6. Clone the qt5 git repo and start building the qtbase module:
   
   $ git clone git://gitorious.org/qt/qt5.git
   $ cd qt5
   $ ./init-repository
   $ cd qt5/qtbase
   $ ./configure -prefix your_qt_prefix -release -device linux-rasp-pi-g++ \
   -make libs -device-option CROSS_COMPILE=your_toolchain_path/bin/arm-linux-gnueabihf- \
   -device-option DISTRO=wheezy -sysroot your_sysroot_path -opensource \
   -confirm-license -no-pch -make examples -nomake tests
   $ make -jn
   $ sudo make install
   
   In my case it was done automatically by the configure script, but you might need to set the pkgconfig path before running the configuration script:
   
   $ export PKG_CONFIG_PATH=your_sysroot_path/usr/lib/arm-linux-gnueabihf/pkgconfig
   $ export PKG_CONFIG_LIBDIR=yout_sysroot_path/usr/lib/pkgconfig:your_sysroot_path/usr/lib/arm-linux-gnueabihf/pkgconfig
   $ export PKG_CONFIG_SYSROOT_DIR=yout_sysroot_path
   
   
7. During compilation I got an error indicating that it was impossible to find the header "vchost_config.h". I solved by editing the file in rasp_sysroot/opt/vc/include/interface/vmcs_host/vcgencmd.h:
   33c33
   
   < #include "vchost_config.h"
   ---
   > #include "linux/vchost_config.h"
   
   This is not very elegant maybe... anyway it is sufficient. You might add an include path in the qmake.conf or similar, but it seemed good that way :-)
8. At this point qtbase should have been successfully compiled. Now at least you should compile the qtscript, qtjsbackend and the qtdeclarative module:
   
   $ cd ..
   $ cd qtscript
   $ your_qt_prefix/bin/qmake
   $ make -jn
   $ sudo make install
   
   Repeat the same for the other two modules.

Building the QtWayland Dependencies

The xkbcommon and the wayland libraries must be cross-compiled before trying to build QtWayland. To do that first setup the environment; I used this script to do it:

export RPI_SYSROOT=your_sysroot_path
export TOOLCHAIN=your_toolchain_path
export QTDIR=your_qt_sources_dir/qtbase
export PATH=$QTDIR/bin:$TOOLCHAIN/bin:$PATH
export PREFIX=your_qt_prefix
export PKG_CONFIG_PATH="$RPI_SYSROOT/usr/lib/pkgconfig:$RPI_SYSROOT/$PREFIX/lib/pkgconfig:$RPI_SYSROOT/$PREFIX/share/pkgconfig"
export PKG_CONFIG_SYSROOT_DIR="$RPI_SYSROOT"
export PKG_CONFIG_ALLOW_SYSTEM_LIBS=1
export PKG_CONFIG_ALLOW_SYSTEM_CFLAGS=1
export CPP=$TOOLCHAIN/bin/arm-linux-gnueabihf-cpp
export CC=$TOOLCHAIN/bin/arm-linux-gnueabihf-gcc
export CXX=$TOOLCHAIN/bin/arm-linux-gnueabihf-g++
export CFLAGS="--sysroot=$RPI_SYSROOT"
export CXXFLAGS="--sysroot=$RPI_SYSROOT"
export CPPFLAGS="--sysroot=$RPI_SYSROOT"
export LD=$TOOLCHAIN/bin/arm-linux-gnueabihf-ld
export LDFLAGS="--sysroot=$RPI_SYSROOT"
export AS=$TOOLCHAIN/bin/arm-linux-gnueabihf-as
export STRIP=$TOOLCHAIN/bin/arm-linux-gnueabihf-strip
export AR=$TOOLCHAIN/bin/arm-linux-gnueabihf-ar

Start the script and download from git the xkbcommon lib:

$ source env_setup.sh
$ git clone git://people.freedesktop.org/xorg/lib/libxkbcommon.git
$ cd libxkbcommon/
$ ./autogen.sh --prefix=some_prefix --host=arm-linux-gnueabihf
$ make && make install

Copy libs and headers resulting to your_sysroot_path.
Now before compiling the wayland library the wayland scanner is needed for the generation of C code from Wayland protocols. To compile this, open a new environment for standard compilation and start compiling the wayland-scanner and place it in the PATH:

$ git clone git://anongit.freedesktop.org/wayland/wayland
$ cd wayland
$ ./autogen.sh --disable-documentation
$ make
$ cp src/wayland-scanner $QTDIR/bin

Now get back to the cross-compilation environment and compile the wayland library itself:

$ cd wayland_dir
$ git clean -dxf
$ ./autogen.sh --host=arm-linux-gnueabihf --prefix=$RPI_SYSROOT$PREFIX --disable-scanner --disable-documentation
$ make
$ sudo make install

Then, if necessary, copy the libraries into your sysroot.
Now let's build QtWayland:

$ export QT_WAYLAND_GL_CONFIG=brcm_egl
$ cd your_qt_sources_dir
$ git clone http://qt.gitorious.org/qt/qtwayland
$ cd qtwayland
$ your_qt_prefix/qmake CONFIG+=wayland-compositor
$ make
$ sudo make install

At this point I have to say I had issues during the execution of the qmake binary. Unfortunately I couldn't track down all the reasons, but it seems that libxkbcommon couldn't be found. According to the .pro file, the qtCompileTest function is used to check if config.test/xkbcommon can be built. Appearantly, the inclusion of X11/keysym.h couldn't be satisfied, and also it shouldn't be needed... but although the file can be compiled, the qtwayland.pro file still was failing, so I simlply removed the checks for xkbcommon and the rest of the build procedure succeeded.

load(configure)
qtCompileTest(wayland)
#qtCompileTest(xkbcommon)
qtCompileTest(wayland_scanner)
qtCompileTest(wayland_egl)
qtCompileTest(egl)
qtCompileTest(brcm_egl)
qtCompileTest(glx)
qtCompileTest(xcomposite)

CONFIG += config_xkbcommon

load(qt_parts)

!config_wayland {
error(QtWayland requires Wayland 1.0.0 or higher)
}

#!config_xkbcommon {
error(QtWayland requires xkbcommon 0.2.0 or higher)
}

!config_wayland_scanner {
error(QtWayland requires wayland-scanner)
}

!config_wayland_egl {
message("no wayland-egl support detected, cross-toolkit compatibility disabled");
}

Running Applications Using the QtWayland Platform Plugin

Now copy the result of the build which should now be in your sysroot into your Pi and try to run the Wayland example compositor from QtWayland:

cd your_qt_prefix/examples/qtwayland/qml-compositor
export XDG_RUNTIME_DIR=/tmp
./qml-compositor -platform eglfs

At this point the server should be running. Now open another shell and try to run any Qt application using the wayland-brcm platform plugin:

$ cd your_app_path
$ ./your_app_bin -platform wayland-brcm &

Now you should see the window on the screen.
s It is possible however that some EGL/OpenGL error occurs, like eglCreatePixmapSurface failed: 3003, global image id: 0 0, then consider increasing the memory reserved to the GPU, that is a bad_allocation error. Simply add gpu_mem=n, where n is the number of MBs to assign to the GPU in the /boot/config.txt file. Read here for more information: http://elinux.org/RPiconfig.

Building QtWebKit

For more details refer to this. It seems Qt guys have done a good work on QtWebKit. Making it work simply requires to build and run. Compile as said the qtwebkit module, then copy back the libraries to the device and load a WebView element.
The only thing that still seems to be missing is the 16bit color depth support: if you try tu run you might see a mess on the scren, that is because the QtWebProcess is writing 24bit image on 16bit mode. More details on this here.
Anyway, it seems now it is sufficient to set the framebuffer to 24bits to make it work:

$ fbset -depth 24

No need to modify the eglfs plugin anymore. The EGL configuration seems to correctly reflect the framebuffer color depth.

Building QtMultimedia

QtMultimedia is the module responsible for the multimedia content handling. For Linux, it is based on gstreamer, which is available, ad already said, for Raspberry Pi. Anyway, gstreamer relies on plugins to decode/render multimedia content, but most of those are clearly not hardware accelerated, which makes it nearly useless on an embedded platform for video playback.

Anyway, there is a plugin that is supposed to use the RPi accelerated OpenMAX libraries, gst-omx. I have to say I still have never seen it work well, so I'm not sure whether this is working or not on Pi. It is an interesting subject, but I don't if or when I'll get my hands on that.
I tried the QtMultimedia module a couple of times though, and I could play a couple of videos, but the result was clearly useless. Something like this should play the video (no audio):

import QtQuick 2.0
import QtMultimedia 5.0

Rectangle {
   width:  1920
   height: 1080

   color: "black"
 
   MediaPlayer {
      id: player
      source: "file://..."
      autoPlay: true
   }

   VideoOutput {
      id: videoOutput
      source: player
      anchors.fill: parent
   }
}

Bye!

Making QtWebKit with WebKit2 in Qt 5.0 work on Raspberry Pi

After compiling Qt 5.0 for Raspberry Pi, I wanted to try all the modules. I compiled and tested the simplest modules first, like:

1. qtbase
2. qtdeclarative
3. qtjsbackend
4. qtquick1
5. qtscript
6. qtxmlpatterns

Everything worked quite smoothly, without any change, except the patch to the v8 engine in qtjsbackend module for ARM hardfp (read here).

I then turned my attention to the QtWebKit module: I know that many changes have been applied to this module for in Qt 5.0, so it is very interesting to try it. Therefore, I started to compile it the usual way:

cd your_qt_sources/qtwebkit
your_qt_prefix/bin/qmake
make -jn
make install

Unfortunately, at the moment of writing, it didn't work for me: the module compiled and linked successfully, but crashed almost as soon as the page load procedure started. I therefore had to start searching around and asking Qt guys about this issue, and during this, I was frequently directed here. Turned out this is exactly the issue I encountered: I applied the latest patch (consider reading the entire report and check that a more stable fix is available):

cd your_qt_sources/qtwebkit
patch -p1 < ~/webkit_patch

where webkit_patch is the linked diff. So, now you can compile the module to include the differences and try again. Consider that the patch seems to be near land, so it might not be necessary for you in the future. As a first test, this is the code I tried:

#include
#include

int main(int argc, char* argv[])
{
    QApplication app(argc, argv);
    QWebView webView;
    webView.setUrl(QUrl("http://www.repubblica.it"));
    webView.show();
    return app.exec();
}

At this point, it should work. As you can see, performance is unacceptable. This is not the way an embedded browser is supposed to work indeed, this is more like classes that can be used for a desktop browser.
Next step was testing WebKit2 with tiling and all that nice stuff. So I tried to create a simple application loading a WebView QML element, like this:

main.cpp:
#include

int main(int argc, char *argv[])
{
    QGuiApplication app(argc, argv);
    QQuickView viewer;
    viewer.setSource(QUrl("/home/pi/main.qml"));
    viewer.show();
    return app.exec();
}

main.qml:
import QtQuick 2.0
import QtWebKit 3.0

WebView {
    url: "http://www.cnn.com"
    x: 100
    y: 100  
    width:  1500
    height: 900
}

When I tried this the first time, it closed with an anonymous error on QtWebProcess:

Failed to start " QtWebProcess 14"

After looking into the sources, I found this is simply a failure to find the QtWebProcess binary. It is needed because WebKit2 uses a separate process to draw the content. Placing it in /usr/bin is sufficient:

cd /usr/bin
sudo ln -s your_qt_prefix/bin/QtWebProcess QtWebProcess

Or better use a relative path. Now what you might get is... nothing at all... At least this is what I got. After some seconds a crash occurred and often nothing was drawn.
It took me some time to guess what was happening: it seems that QtWebKit is trying to draw with 32 bit color depth, while instead the rest of Qt is rendering using 16 bit color depth (especially if you're using the eglfs platform plugin) on a RGB565 EGL configuration.
According to the Qt WebKit code, it seems that the entire code is assuming 32 bit color depth is being used, and this is hardcoded instead of negotiated with the main process. As a quick fix, I made the entire EGL configuration to be 32 bit (read this).
By setting this, it is possible to make QtWebKit work in 1080p with 32 bit color depth. If I'll find the time I'll do the needed modifications to have the same configuration with the correct color depth.

Consider that, at the moment of writing, the configuration of the wheezy image does not assign sufficient memory to the GPU.You'll see therefore pages where many tiles are black: that is the result of insufficient GPU memory when trying to allocate for the texture. Just assign some more memory to the GPU editing the /boot/config.txt file of your Pi.

Sample Video

NOTE: For updated information read here.