Binaries for POT 5.4.0 on Qt 5.7.0 and Experimental Accelerated QtWebKit

So far, no real complains about POT 5.4.0-beta1 so I suppose it is sufficiently safe to consider it POT 5.4.0. Refer to this article for more info.

Binaries for POT 5.4.0-beta1 on Qt 5.7.0 and Experimental Accelerated QtWebKit

This is a new binary package including the same content of POT 5.3.0-beta1 package, with some updates:

• new improvements in POT taken from omxplayer code;
• bump to ffmpeg 3.1;
• based on Qt 5.7.0.

The build includes support for network protocols and WebKit 1 supporting video html5 elements (including youtube).

This version is still experimental. Please report bugs on github.

Have fun! Bye! ;-)

Download POT 5.4.0-beta1 for Raspbian Jessie Lite Pi2 (also tested on Pi3) here (md5: e6b6a1f92b71ac2bdd3d9756efd7669c).

Fully Optimised Raspbian Jessie Lite with Accelerated Qt Multimedia Backend (Pi2)

This is a follow-up to the image provided in this article. The concept is identical, but in this case this firmware is smaller, it is based on Raspbian Jessie Lite and includes updated software. The firmware was 1.9GB while now is 759MB. This is the current setup (but I encourage to update to the latest possible firmware):

pi@raspberrypi:~ $ vcgencmd version
Feb 25 2016 14:25:47 
Copyright (c) 2012 Broadcom
version dea971b793dd6cf89133ede5a8362eb77e4f4ade (clean) (release)
pi@raspberrypi:~ $ uname -a
Linux raspberrypi 4.1.18-v7+ #846 SMP Thu Feb 25 14:22:53 GMT 2016 armv7l GNU/Linux

Download the image here (torrent) (md5: 06d75d03f63674350be6cb807850bf09).
(Please be patient while downloading and decompressing)

The image includes:

• Qt 5.6.0-rc1: built for armv7 with neon optimisations, including support for touch screens, gstreamer, libinput, tslib, and Bluetooth/Bluetooth BLE using bluez (complete configuration below).
• QtWebKit 5.6.0: includes WebKit 1 and WebKit 2.
• QtWebEngine 5.6.0-rc1: multimedia only based on ffmpeg.
• Dependant libs: evdev, icu (not the one provided by raspbian), mtdev, libts.
• Samples: samples from the repo to test POT.
• POT: POT version 5.2.1 including ffmpeg 2.8.6.

Samples

To have a quick look at multimedia in Qt/QML test with this commands:

• /usr/local/Qt-rasp2-5.6.0-rc1/bin/qmlscene "some_video_file" samples/video_simple.qml
• /usr/local/Qt-rasp2-5.6.0-rc1/bin/qmlscene "some_url" samples/webkit_simple.qml
• /usr/local/Qt-rasp2-5.6.0-rc1/bin/qmlscene "some_url" samples/webengine_simple.qml
• /home/pi/piomxtextures_pocplayer --multipleanimtest "720p_video_1" "720p_video_2"

Please let me know if you find something wrong and file a bugreport on https://github.com/carlonluca/pi.

Have fun! ;-)

Hardware Acceleration on WebKit 1, WebKit 2 and QtWebEngine with Qt on Raspberry Pi 2

There are a few solutions to get a browser with good performance on Raspberry Pi 2, but I wanted to test solutions that provide this to a Qt app on OpenGL without X11. The two main options I know are the "old" QtWebKit and the new QtWebEngine. I know that there are a couple of ways to use QtWebKit, one is with the first implementation, WebKit 1, the other is through WebKit 2. Also, it is possible to use WebKit 1 using the regular software raster paint engine and the OpenGL paint engine, which is the one I used in the previous article to implement the HTML5 video feature.
I therefore wanted to have a look at the performance of the animations (I'm not talking about WebGL here), so here it is; the video shows three samples (thanks to the authors!) run on each of these alternatives: WebKit 1 on raster, WebKit 1 on OpenGL, WebKit 2 and then QtWebEngine.

As you can see WebKit 1 on raster is completely unusable and WebKit 1 on OpenGL is clearly better, but still not very good. Also you can see how the pointer moves: the main thread is too busy and user interaction is difficult. WebKit 2 and QtWebEngine solve this issue and improve performance.

Hands-on WebKit and HTML5 Media Elements on Raspberry Pi

I tried to do this a few months ago actually, but I got lost in the huge amount of code and architecture variants of WebKit. Now I convinced myself it would be too interesting to see it working. I don't have much spare time, so I wanted to start with the path of least-resistance and then, depending on the time available, continue climbing. This activity is very time consuming, so I do not know where I'll get.

This is however the first stage: using POT to implement 1080p hardware accelerated zero-copy video playback in the Qt port of WebKit1. No QML here. Of no use, but it somehow works. I read there are far better results out there, but I learned much by doing this, and I had fun :-)
Of course, as you can see, there is much much work being done in the UI thread. JavaScript is mostly run in the main thread, making the framerate pretty bad on heavy websites. That is why WebKit2 is there after all :-)

Demo

Bye! ;-)

Fully Optimised Raspbian Jessie with Accelerated Qt Multimedia Backend (Pi2)

This is an image of Raspbian Jessie ready to test (and use) the Qt Multimedia Backend POT. This image is optimised for maximum performance (see the demos below). This is the current setup:

pi@raspberrypi ~ $ sudo /opt/vc/bin/vcgencmd version
Nov 11 2015 21:31:07 
Copyright (c) 2012 Broadcom
version 54011a8ad59a9ae1c40bd07cddd9bcf90e779b66 (clean) (release)
pi@raspberrypi ~ $ uname -a
Linux raspberrypi 4.1.13-v7+ #826 SMP PREEMPT Fri Nov 13 20:19:03 GMT 2015 armv7l GNU/Linux
pi@raspberrypi ~ $ lsb_release -a
No LSB modules are available.
Distributor ID: Raspbian
Description:    Raspbian GNU/Linux 8.0 (jessie)
Release:        8.0
Codename:       jessie

Download image here.

The image includes:

• Qt 5.5.1 built for armv7 with neon optimisations, including support for touch screens, gstreamer, libinput, X11 (no multimedia), tslib, and Bluetooth/Bluetooth BLE using bluez (complete configuration below). Modules provided include QtWebKit (no multimedia support).
• Dependant libs: evdev, icu (not the one provided by raspbian), mtdev, libts.
• POCPlayer and QML samples to show the performance of the backend and test bugs.
• POT library including ffmpeg 2.7.2.

Everything is built for armv7 including neon optimisations. To quickly test the performance there are a few things you can launch:

/home/pi/piomxtextures_pocplayer --multipleanimtest /home/pi/sintel_trailer_720p.mp4 /home/pi/big_buck_bunny_720p_h264.mov

This is the result you should expect with a proper overclocking:

It will show you a continuous animation while running two 720p videos concurrently. To improve the frame rate of the animations you can also overclock your Pi2 (see below).

Other samples to test the features of the plugin are stored in /home/pi/samples:

/usr/local/Qt-rasp2-5.5.1/bin/qmlscene /home/pi/samples/video_simple.qml file:///home/pi/big_buck_bunny_1080p_h264.mov

Another interesting test is running qmlvideofx example included in the Qt sources. This will show you the perfomance of the effects applied on the video using shaders. You can find an executable in /usr/local/Qt-rasp2-5.5.1/examples/multimedia/video/qmlvideofx/qmlvideofx. This shows approximately what you should expect (the video shows the result on Pi1, Pi2 is just a little better):

Overclocking

The Pi can be overclocked by tuning parameters in /boot/config.txt. I provided a configuration that I find useful:

#force_turbo=1  # Voids Warranty!

#boot_delay=1   # Helps to avoid sdcard corruption when force_turbo is enabled.

#arm_freq=1100

#sdram_freq=450

#core_freq=550

#over_voltage=4

#temp_limit=80  # Will throttle to default clock speed if hit.

In the image this is all disabled. If you intent to enable please keep in mind I don't take responsibility for any consequence. Please refer to the documentation for this procedure.

Have fun! Bye ;-)

Qt Configuration

   Configure summary

Building on:   linux-g++ (x86_64, CPU features: mmx sse sse2)

Building for:  devices/linux-rasp-pi2-g++ (arm, CPU features: neon)

Platform notes:

            - Also available for Linux: linux-clang linux-kcc linux-icc linux-cxx

        

qmake vars .......... styles += mac fusion windows QT_CFLAGS_GLIB = -pthread -I/opt/rpi/sysroot/usr/include/glib-2.0 -I/opt/rpi/sysroot/usr/lib/arm-linux-gnueabihf/glib-2.0/include  QT_LIBS_GLIB = -L/opt/rpi/sysroot/usr/lib/arm-linux-gnueabihf -lgthread-2.0 -pthread -lglib-2.0  QT_CFLAGS_PULSEAUDIO = -D_REENTRANT -I/opt/rpi/sysroot/usr/include/glib-2.0 -I/opt/rpi/sysroot/usr/lib/arm-linux-gnueabihf/glib-2.0/include  QT_LIBS_PULSEAUDIO = -L/opt/rpi/sysroot/usr/lib/arm-linux-gnueabihf -lpulse-mainloop-glib -lpulse -lglib-2.0  QMAKE_CFLAGS_FONTCONFIG = -I/opt/rpi/sysroot/usr/include/freetype2  QMAKE_LIBS_FONTCONFIG = -L/opt/rpi/sysroot/usr/lib/arm-linux-gnueabihf -lfontconfig -lfreetype  QMAKE_INCDIR_LIBUDEV =  QMAKE_LIBS_LIBUDEV = -L/opt/rpi/sysroot/usr/lib/arm-linux-gnueabihf -ludev  QMAKE_LIBINPUT_VERSION_MAJOR = 1 QMAKE_LIBINPUT_VERSION_MINOR = 0 QMAKE_INCDIR_LIBINPUT = /opt/rpi/sysroot/home/pi/qtdeps/include  QMAKE_LIBS_LIBINPUT = -L/opt/rpi/sysroot/home/pi/qtdeps/lib -linput  QMAKE_LIBXI_VERSION_MAJOR = 1 QMAKE_LIBXI_VERSION_MINOR = 7 QMAKE_LIBXI_VERSION_PATCH = 4 QMAKE_X11_PREFIX = /usr QMAKE_XKB_CONFIG_ROOT = /usr/share/X11/xkb QMAKE_CFLAGS_XCB =  QMAKE_LIBS_XCB = -L/opt/rpi/sysroot/usr/lib/arm-linux-gnueabihf -lxcb-sync -lxcb-xfixes -lxcb-render -lxcb-randr -lxcb-image -lxcb-shm -lxcb-keysyms -lxcb-icccm -lxcb-shape -lxcb  INCLUDEPATH +=  "/opt/rpi/sysroot/home/pi/qtdeps/include" LIBS +=  -L"/opt/rpi/sysroot/home/pi/qtdeps/lib" sql-drivers =  sql-plugins =  sqlite qmake switches ......... 

Build options:

  Configuration .......... accessibility accessibility-atspi-bridge alsa audio-backend c++11 clock-gettime clock-monotonic compile_examples concurrent cross_compile cups dbus egl eglfs eglfs_brcm enable_new_dtags evdev eventfd fontconfig full-config getaddrinfo getifaddrs glib gstreamer-1.0 harfbuzz iconv icu inotify ipv6ifname large-config largefile libinput libproxy libudev linuxfb medium-config minimal-config mremap mtdev neon nis opengl opengles2 openssl pcre png posix_fallocate pulseaudio qpa qpa reduce_exports release rpath shared small-config system-freetype system-jpeg system-png system-zlib tslib use_gold_linker xcb xcb-glx xcb-plugin xcb-render xcb-xlib xinput2 xkbcommon-qt xlib xrender 

  Build parts ............  libs examples

  Mode ................... release

  Using sanitizer(s)...... none

  Using C++11 ............ yes

  Using gold linker....... yes

  Using new DTAGS ........ yes

  Using PCH .............. no

  Target compiler supports:

    Neon ................. yes

Qt modules and options:

  Qt D-Bus ............... yes (loading dbus-1 at runtime)

  Qt Concurrent .......... yes

  Qt GUI ................. yes

  Qt Widgets ............. yes

  Large File ............. yes

  QML debugging .......... yes

  Use system proxies ..... no

Support enabled for:

  Accessibility .......... yes

  ALSA ................... yes

  CUPS ................... yes

  Evdev .................. yes

  FontConfig ............. yes

  FreeType ............... yes (system library)

  Glib ................... yes

  GStreamer .............. yes (1.0)

  GTK theme .............. no

  HarfBuzz ............... yes (bundled copy)

  Iconv .................. yes

  ICU .................... yes

  Image formats: 

    GIF .................. yes (plugin, using bundled copy)

    JPEG ................. yes (plugin, using system library)

    PNG .................. yes (in QtGui, using system library)

  journald ............... no

  libinput................ yes

  mtdev .................. yes (system library)

  Networking: 

    getaddrinfo .......... yes

    getifaddrs ........... yes

    IPv6 ifname .......... yes

    libproxy.............. yes

    OpenSSL .............. yes (loading libraries at run-time)

  NIS .................... yes

  OpenGL / OpenVG: 

    EGL .................. yes

    OpenGL ............... yes (OpenGL ES 2.0+)

    OpenVG ............... no

  PCRE ................... yes (bundled copy)

  pkg-config ............. yes 

  PulseAudio ............. yes

  QPA backends: 

    DirectFB ............. no

    EGLFS ................ yes

      EGLFS i.MX6....... . no

      EGLFS KMS .......... no

      EGLFS Mali ......... no

      EGLFS Raspberry Pi . yes

      EGLFS X11 .......... no

    LinuxFB .............. yes

    XCB .................. yes (system library)

      EGL on X ........... no

      GLX ................ yes

      MIT-SHM ............ yes

      Xcb-Xlib ........... yes

      Xcursor ............ yes (loaded at runtime)

      Xfixes ............. yes (loaded at runtime)

      Xi ................. no

      Xi2 ................ yes

      Xinerama ........... yes (loaded at runtime)

      Xrandr ............. yes (loaded at runtime)

      Xrender ............ yes

      XKB ................ yes

      XShape ............. yes

      XSync .............. yes

      XVideo ............. yes

  Session management ..... yes

  SQL drivers: 

    DB2 .................. no

    InterBase ............ no

    MySQL ................ no

    OCI .................. no

    ODBC ................. no

    PostgreSQL ........... no

    SQLite 2 ............. no

    SQLite ............... yes (plugin, using bundled copy)

    TDS .................. no

  tslib .................. yes

  udev ................... yes

  xkbcommon-x11........... yes (bundled copy, XKB config root: /usr/share/X11/xkb)

  xkbcommon-evdev......... no

  zlib ................... yes (system library)

Updates on Hardware Accelerated Qt Multimedia Backend on Raspberry Pi (01.05.2015)

Many complained about an horizontal tearing issue when playing videos on the Pi using the PiOmxTextures driver. The issue was caused by a missing switch of buffers backing the textures on the output of the renderer component. I took a little time to quickly improve the rendering and the new result is interesting. I also fixed an OpenMAX setup that caused my Pi to crash when stressed (have a look at the commit logs). These were the videos showing the old implementation:

Pay attention to the fact that video was only 720p when shaders were applied. I couldn't do it on 1080p.

Consider that in these videos animations involving both 1080p graphics and 1080p source video were very difficult. In fact you can see that video is sometimes only 720p.

After recent changes I tried again the same sample apps and created a small demo of what you can achieve with Pi. Now source video, images and UI are all 1080p resolution.

Source code can be found in the usual github repo. At the moment the latest changes are all committed to the branch no_tearing. Keep in mind that those changes may break the rest of the implementation, so some work is needed if you really want to use it in your project. The modifications were only made to show what performance Pi can achieve. You're welcome to send patches to fix the branch as well if you're interested.

Binaries for Hardware Accelerated Qt Multimedia Backend on Raspberry Pi

As someone still have troubles building PiOmxTextures, I prepared a little package that contains the binaries needed to make it run and have a look. In the past I tried to provide an entire firmware, but it was large to upload and not very comfortable to update. This package only contains:

• PiOmxTextures.tar: library to provide the playback capabilities, including ffmpeg 2.5.
• Qt-rasp-5.4.0.tar: the entire tree of Qt 5.4.0 binaries built for Raspberry Pi (firmware ba43047, rpath is set to /usr/local/Qt-rasp-5.4.0).
• Qt-rasp-5.4.0_host.tar: the utility to be used on the host when cross-building (built for x86 Kubuntu 13.10).

Current version of the package is 4.3.0: download.

To have a quick look, just place Qt-rasp-5.4.0 in /usr/local into your Pi, then place PiOmxTextures somewhere and add it to the runtime library path. Now you can test running qmlvideo or qmlvideofx from the Qt Multimedia examples. Something similar to this should come up: 

Bye! ;-)

Updates on Hardware Accelerated Qt Multimedia Backend on Raspberry Pi (23.12.2014)

As I see some are still working on the Pi, and someone asked to "port" to Qt 5.4.0, this is a small update to the Qt Multimedia Backend POC.

This is what I updated so far:

• Merged code from omxplayer until a4ee074.
• Updated to work with ffmpeg 2.5.
• Updated to work on the latest Pi firmware ba43047.
• Updated some build scripts to speed up build.
• Updated to work with the new Qt 5.4.0.

The build procedure is mostly unchanged. I summarise it here with the new updates (I assume you're in the root of the repo sources):

export RPI_SYSROOT=
export COMPILER_PATH=
cd tools
./compile_ffmpeg.sh
export QMAKE_PATH=
./prepare_openmaxil_backend.sh


This should build ffmpeg, build the PiOmxTextures lib (needed to get the hardware interface to video and audio playback) and prepare the sources of the Qt multimedia backend.
NOTE: I had to modify the libssh pkgconfig file ($RPI_SYSROOT/usr/lib/arm-linux-gnueabihf/pkgconfig/libssh.pc) to:


prefix=/usr
exec_prefix=${prefix}
libdir=${prefix}/lib/arm-linux-gnueabihf
includedir=${prefix}/include

Name: libssh
Description: The SSH Library
Version: 0.5.4
Libs: -L{libdir} -lssh
Cflags: -I{includedir}

Now you have to copy the directory of the plugin to the src/plugins subdirectory of your Qt source tree. Once finished, this plugin will be built.
Unfortunately, still a patch to QtMultimedia is needed, so go into the qtmultimedia dir of your Qt source tree (I tested on the Qt 5.4.0 tree) and patch using the patch provided in tools:

cp -a /openmaxil_backend /qtmultimedia/src/plugins/openmaxil
patch -p1 < /tools/qt_multimedia_patch_5.4.0.patch
$QMAKE_PATH CONFIG+=raspberry
make -j
make install


Now you should have the new openmaxil plugin in your Qt prefix. When using, remember that libPiOmxTextures must be in the runtime library path when using the plugin. If an error like this is returned when running an application using the plugin:

luca@raspberrypi ~ $ ./POCPlayer
* failed to open vchiq instance

then set the permissions appropriately:

sudo chmod a+rw /dev/vchiq

Or better add your user to the video group. Same stands for input devices:

sudo usermod -a -G video
sudo usermod -a -G input

As usual remember this is just a POC.
If anything went wrong, file an issue on github. Someone may have a look.
Bye! ;-)

NOTE: It seems that 1080p video on 1080p output really stresses the GPU at the point that the video starts to have issues and blocks. I tried to test Pi turbo mode, that should increase GPU performance but my Pi seems not to agree on the overvolt parameter. USB ports completely crash and then the system goes bye-bye. If anyone was able to test somehow animations and 1080p playout on 1080p resolution with turbo mode and like 450MHz GPU, please let me know!

Having Fun with OpenGL, Android and Hardware Decoding

OpenGL is a good technology, and playing around with it on Android can be real fun! :-) By streaming a video to texture it is possible to create interesting effects, like it is possible to do with Qt. With a simple shader, for instance, lightning effects can be added:

#extension GL_OES_EGL_image_external : require

precision mediump float;

uniform samplerExternalOES u_Texture;

uniform vec3 u_LightPos;      // The position of the light in eye space.

varying vec2 v_TexCoordinate; // Interpolated texture coordinate per fragment.

varying vec3 v_Position;      // Interpolated position for this fragment.

varying vec4 v_Color; 

varying vec3 v_Normal;        // Interpolated normal.

void main()

{

   float distance = length(u_LightPos - v_Position);

   vec3 lightVector = normalize(u_LightPos - v_Position);

   float diffuse = max(dot(v_Normal, lightVector), 0.0);

   diffuse = diffuse*(1.0/(1.0 + (0.10*distance)));

   diffuse = diffuse + 0.3;

   gl_FragColor = v_Color*diffuse*texture2D(u_Texture, v_TexCoordinate);

}

This is the result on a couple of different devices (Nexus 7 with Qualcomm Snapdragon and an Allwinner A31s chip):