Multiple Animated Videos on Raspberry Pi 2 with Fading Audio

Someone asked if PiOmxTextures (library and Qt OpenMAX-based multimedia plugin) could be used on Raspberry Pi 2 and if it could be used to play more than one video contemporarily. I worked on this and a couple of recent patches made the following result possible.

The video shows two videos running contemporarily on Pi 2, with QML animations applied to the videos and with audio fading from one to the other. The videos are 720p h264 main and high profile. UI is written in QML in 1080p with 24 bit color depth. As you can see, load on the CPU is very low. The demo also shows the same QML file running on Mac OS X.

To get this performance you'll need to properly overclock your Pi 2, otherwise the achievable frame rate is lower.

NOTE: Code for this is not yet on github.

Binaries for Hardware Accelerated Qt Multimedia Backend on Raspberry Pi (4.4.0)

This is a package containing binaries for PiOmxTextures 4.4.0. The relevant improvements here over 4.3.0 are:

• Fixed horizontal tearing effect.
• Fixed leaks when starting/stopping and restarting video.
• Added support for 3.5mm analog audio output. It is now possible to select which one to use (hdmi, analog or both) by setting a AUDIO_OUT env variable (possible values are both, local and hdmi; default: hdmi).
• Fixed garbage at the beginning of the video.
• Merged code from omxplayer code base to revision 74aac. This includes update to ffmpeg 2.6.
• Other minor fixes and improvements.
• Port to Qt 5.4.1.

The package contains:

• piomxtextures_pocplayer: sample player which includes tests for animations, loops and commands. Have a look at the sources to understand how it works.
• Qt-rasp-5.4.1.tar: the entire tree of Qt 5.4.1 binaries built for Raspbian (firmware
     b71d7b6, rpath is set to /usr/local/Qt-rasp-5.4.1).
• Qt-rasp-5.4.1_host.tar: the utility to be used on the host when cross-building (built for x86 Kubuntu 13.10).

Current version of the package is 4.4.0: download.

To have a quick look, just place Qt-rasp-5.4.1 in /usr/local into your Pi. Now you can test running qmlvideo, qmlvideofx or piomxtextures_pocplayer from the Qt Multimedia examples. Something similar to this should come up: 

To build using the host tools:

1. Place the Qt libs provided in /usr/local in your Pi filesystem.
2. Place the sysroot of your Pi into /opt/rpi/sysroot (you should then have Qt libs into /opt/rpi/sysroot/usr/local/Qt-rasp-5.4.1).
3. Place the host tools in /usr/local.
4. Place the Linaro toolchain (https://github.com/raspberrypi/tools/tree/master/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian) in /opt/rpi (which means you should have /opt/rpi/gcc-linaro-arm-linux-gnueabihf-raspbian).
5. You should be done. 

I tested on a Ubuntu 15.04 64 bit and the procedure worked, just be sure to use the 64 bit toolchain (in the same position) and add compatibility libs (qmake is a 32 bit binary).

Bye! ;-)

NOTE: Please notice that the provided binaries contain both the gstreamer plugin (the default unaccelerated Qt multimedia plugin) and the PiOmxTexures (POT) plugin. You'll have to ensure that your app uses POT to run and not gstreamer. To avoid confusion, you can simply remove (or move away) the gstreamer plugin from the default path (/usr/local/Qt-rasp-5.4.1/plugins/mediaservice).

"Je suis Charlie"

I stand for Charlie Hebdo: people who expressed with the pen instead of cowardly expressing with Kalashnikovs.

Can't sleep tonight, thoughts with my French cartooning colleagues, their families and loved ones #CharlieHebdo pic.twitter.com/LqIMRCHPgK

— David Pope (@davpope) January 7, 2015

The civil world must not accept Sharia as the official law. Thank you France for granting freedom of self-expression!

Updates on Hardware Accelerated Qt Multimedia Backend on Raspberry Pi (01.05.2015)

Many complained about an horizontal tearing issue when playing videos on the Pi using the PiOmxTextures driver. The issue was caused by a missing switch of buffers backing the textures on the output of the renderer component. I took a little time to quickly improve the rendering and the new result is interesting. I also fixed an OpenMAX setup that caused my Pi to crash when stressed (have a look at the commit logs). These were the videos showing the old implementation:

Pay attention to the fact that video was only 720p when shaders were applied. I couldn't do it on 1080p.

Consider that in these videos animations involving both 1080p graphics and 1080p source video were very difficult. In fact you can see that video is sometimes only 720p.

After recent changes I tried again the same sample apps and created a small demo of what you can achieve with Pi. Now source video, images and UI are all 1080p resolution.

Source code can be found in the usual github repo. At the moment the latest changes are all committed to the branch no_tearing. Keep in mind that those changes may break the rest of the implementation, so some work is needed if you really want to use it in your project. The modifications were only made to show what performance Pi can achieve. You're welcome to send patches to fix the branch as well if you're interested.

Binaries for Hardware Accelerated Qt Multimedia Backend on Raspberry Pi

As someone still have troubles building PiOmxTextures, I prepared a little package that contains the binaries needed to make it run and have a look. In the past I tried to provide an entire firmware, but it was large to upload and not very comfortable to update. This package only contains:

• PiOmxTextures.tar: library to provide the playback capabilities, including ffmpeg 2.5.
• Qt-rasp-5.4.0.tar: the entire tree of Qt 5.4.0 binaries built for Raspberry Pi (firmware ba43047, rpath is set to /usr/local/Qt-rasp-5.4.0).
• Qt-rasp-5.4.0_host.tar: the utility to be used on the host when cross-building (built for x86 Kubuntu 13.10).

Current version of the package is 4.3.0: download.

To have a quick look, just place Qt-rasp-5.4.0 in /usr/local into your Pi, then place PiOmxTextures somewhere and add it to the runtime library path. Now you can test running qmlvideo or qmlvideofx from the Qt Multimedia examples. Something similar to this should come up: 

Bye! ;-)

Updates on Hardware Accelerated Qt Multimedia Backend on Raspberry Pi (23.12.2014)

As I see some are still working on the Pi, and someone asked to "port" to Qt 5.4.0, this is a small update to the Qt Multimedia Backend POC.

This is what I updated so far:

• Merged code from omxplayer until a4ee074.
• Updated to work with ffmpeg 2.5.
• Updated to work on the latest Pi firmware ba43047.
• Updated some build scripts to speed up build.
• Updated to work with the new Qt 5.4.0.

The build procedure is mostly unchanged. I summarise it here with the new updates (I assume you're in the root of the repo sources):

export RPI_SYSROOT=
export COMPILER_PATH=
cd tools
./compile_ffmpeg.sh
export QMAKE_PATH=
./prepare_openmaxil_backend.sh


This should build ffmpeg, build the PiOmxTextures lib (needed to get the hardware interface to video and audio playback) and prepare the sources of the Qt multimedia backend.
NOTE: I had to modify the libssh pkgconfig file ($RPI_SYSROOT/usr/lib/arm-linux-gnueabihf/pkgconfig/libssh.pc) to:


prefix=/usr
exec_prefix=${prefix}
libdir=${prefix}/lib/arm-linux-gnueabihf
includedir=${prefix}/include

Name: libssh
Description: The SSH Library
Version: 0.5.4
Libs: -L{libdir} -lssh
Cflags: -I{includedir}

Now you have to copy the directory of the plugin to the src/plugins subdirectory of your Qt source tree. Once finished, this plugin will be built.
Unfortunately, still a patch to QtMultimedia is needed, so go into the qtmultimedia dir of your Qt source tree (I tested on the Qt 5.4.0 tree) and patch using the patch provided in tools:

cp -a /openmaxil_backend /qtmultimedia/src/plugins/openmaxil
patch -p1 < /tools/qt_multimedia_patch_5.4.0.patch
$QMAKE_PATH CONFIG+=raspberry
make -j
make install


Now you should have the new openmaxil plugin in your Qt prefix. When using, remember that libPiOmxTextures must be in the runtime library path when using the plugin. If an error like this is returned when running an application using the plugin:

luca@raspberrypi ~ $ ./POCPlayer
* failed to open vchiq instance

then set the permissions appropriately:

sudo chmod a+rw /dev/vchiq

Or better add your user to the video group. Same stands for input devices:

sudo usermod -a -G video
sudo usermod -a -G input

As usual remember this is just a POC.
If anything went wrong, file an issue on github. Someone may have a look.
Bye! ;-)

NOTE: It seems that 1080p video on 1080p output really stresses the GPU at the point that the video starts to have issues and blocks. I tried to test Pi turbo mode, that should increase GPU performance but my Pi seems not to agree on the overvolt parameter. USB ports completely crash and then the system goes bye-bye. If anyone was able to test somehow animations and 1080p playout on 1080p resolution with turbo mode and like 450MHz GPU, please let me know!