Binaries for Hardware Accelerated Qt Multimedia Backend on Raspberry Pi

As someone still have troubles building PiOmxTextures, I prepared a little package that contains the binaries needed to make it run and have a look. In the past I tried to provide an entire firmware, but it was large to upload and not very comfortable to update. This package only contains:

• PiOmxTextures.tar: library to provide the playback capabilities, including ffmpeg 2.5.
• Qt-rasp-5.4.0.tar: the entire tree of Qt 5.4.0 binaries built for Raspberry Pi (firmware ba43047, rpath is set to /usr/local/Qt-rasp-5.4.0).
• Qt-rasp-5.4.0_host.tar: the utility to be used on the host when cross-building (built for x86 Kubuntu 13.10).

Current version of the package is 4.3.0: download.

To have a quick look, just place Qt-rasp-5.4.0 in /usr/local into your Pi, then place PiOmxTextures somewhere and add it to the runtime library path. Now you can test running qmlvideo or qmlvideofx from the Qt Multimedia examples. Something similar to this should come up: 

Bye! ;-)

Updates on Hardware Accelerated Qt Multimedia Backend on Raspberry Pi (23.12.2014)

As I see some are still working on the Pi, and someone asked to "port" to Qt 5.4.0, this is a small update to the Qt Multimedia Backend POC.

This is what I updated so far:

• Merged code from omxplayer until a4ee074.
• Updated to work with ffmpeg 2.5.
• Updated to work on the latest Pi firmware ba43047.
• Updated some build scripts to speed up build.
• Updated to work with the new Qt 5.4.0.

The build procedure is mostly unchanged. I summarise it here with the new updates (I assume you're in the root of the repo sources):

export RPI_SYSROOT=
export COMPILER_PATH=
cd tools
./compile_ffmpeg.sh
export QMAKE_PATH=
./prepare_openmaxil_backend.sh


This should build ffmpeg, build the PiOmxTextures lib (needed to get the hardware interface to video and audio playback) and prepare the sources of the Qt multimedia backend.
NOTE: I had to modify the libssh pkgconfig file ($RPI_SYSROOT/usr/lib/arm-linux-gnueabihf/pkgconfig/libssh.pc) to:


prefix=/usr
exec_prefix=${prefix}
libdir=${prefix}/lib/arm-linux-gnueabihf
includedir=${prefix}/include

Name: libssh
Description: The SSH Library
Version: 0.5.4
Libs: -L{libdir} -lssh
Cflags: -I{includedir}

Now you have to copy the directory of the plugin to the src/plugins subdirectory of your Qt source tree. Once finished, this plugin will be built.
Unfortunately, still a patch to QtMultimedia is needed, so go into the qtmultimedia dir of your Qt source tree (I tested on the Qt 5.4.0 tree) and patch using the patch provided in tools:

cp -a /openmaxil_backend /qtmultimedia/src/plugins/openmaxil
patch -p1 < /tools/qt_multimedia_patch_5.4.0.patch
$QMAKE_PATH CONFIG+=raspberry
make -j
make install


Now you should have the new openmaxil plugin in your Qt prefix. When using, remember that libPiOmxTextures must be in the runtime library path when using the plugin. If an error like this is returned when running an application using the plugin:

luca@raspberrypi ~ $ ./POCPlayer
* failed to open vchiq instance

then set the permissions appropriately:

sudo chmod a+rw /dev/vchiq

Or better add your user to the video group. Same stands for input devices:

sudo usermod -a -G video
sudo usermod -a -G input

As usual remember this is just a POC.
If anything went wrong, file an issue on github. Someone may have a look.
Bye! ;-)

NOTE: It seems that 1080p video on 1080p output really stresses the GPU at the point that the video starts to have issues and blocks. I tried to test Pi turbo mode, that should increase GPU performance but my Pi seems not to agree on the overvolt parameter. USB ports completely crash and then the system goes bye-bye. If anyone was able to test somehow animations and 1080p playout on 1080p resolution with turbo mode and like 450MHz GPU, please let me know!

Having Fun with OpenGL, Android and Hardware Decoding

OpenGL is a good technology, and playing around with it on Android can be real fun! :-) By streaming a video to texture it is possible to create interesting effects, like it is possible to do with Qt. With a simple shader, for instance, lightning effects can be added:

#extension GL_OES_EGL_image_external : require

precision mediump float;

uniform samplerExternalOES u_Texture;

uniform vec3 u_LightPos;      // The position of the light in eye space.

varying vec2 v_TexCoordinate; // Interpolated texture coordinate per fragment.

varying vec3 v_Position;      // Interpolated position for this fragment.

varying vec4 v_Color; 

varying vec3 v_Normal;        // Interpolated normal.

void main()

{

   float distance = length(u_LightPos - v_Position);

   vec3 lightVector = normalize(u_LightPos - v_Position);

   float diffuse = max(dot(v_Normal, lightVector), 0.0);

   diffuse = diffuse*(1.0/(1.0 + (0.10*distance)));

   diffuse = diffuse + 0.3;

   gl_FragColor = v_Color*diffuse*texture2D(u_Texture, v_TexCoordinate);

}

This is the result on a couple of different devices (Nexus 7 with Qualcomm Snapdragon and an Allwinner A31s chip):

Qt on Linux i.MX 6

i.MX 6 is a good device. Qt runs pretty well on it. Also a plugin for hardware accelerated and zero-copy rendering seems to work pretty well. Simply building Qt for i.MX 6 makes it possible to use the Qt Multimedia plugin. This is the result (on Linux):

I also tested a provided Android port but the result was not as good.

Video with Animations on Android using VLC for software decoding

Again, video with animations: this time video is demuxed and decoded using the awesome VLC library. VLC is provided buffers from Java that are then drawn on a view. The result is pretty good if you think all the decoding runs on the CPU. And I'm sure it can be improved.

Video with Animations on Raspberry Pi and A31s Compared

For many uses, playing video while animations are running is pretty useful and it is also considerably stressful for the GPU. I therefore tested a couple of cheap chips with the same Qt/QML sample app. I created a small demo video comparing Qt on A31s and Raspberry Pi (PiOmxTextures):

In the demo, all the media (video and images) are 720p, and the output is 720p for A31s and 1080p for Raspberry Pi (16bpp). It is pretty hard to tell from the video, but Raspberry seems to still be above A31s.
Bye! ;-)

A Tribute to Qt Portability :-)

The new ports of Qt to mobile systems like Android, iOS and WinRT are getting more and more reliable. I just wanted to have a look at the current stage of development and I therefore wrote a sample app. This is the result:

This is a quick app to test the portability of Qt on desktop, mobile and embedded.

The video shows client/server code, communicating via WebSockets (using the new WebSockets module in Qt), with a server in Qt/C++ running on Pi and with a fluid hardware accelerated interface in C++/QML running on Raspberry Pi, Mac OS X Maverick, Android Phone and Tablet, Windows 8 (Modern UI), Windows Phone 8, iOS and Linux Kubuntu. The server (on Raspberry Pi) stores information about tasks and the client provides a remote interface to manage. A client written using HTML5 canvas and WebSockets from the browser is also shown.

Unfortunately I don't own iOS and Winphone ARM devices, otherwise I'd add those as well :-) Good start anyway: this is really "write once, deploy anywhere".

Using the ports is simple: configure Qt Creator with the correct Qt build, setup the toolchain to use, qmake and build (for iOS and WinRT I preferred Xcode and Visual Studio respectively yet). Your done!
Pretty cool! Bye! ;-)

FFmpeg gstreamer plugin on Ubuntu 14.04

EDIT 02.25.2015: It seems to be missing even in Ubuntu 14.10. But probably the quickest way to install is:

sudo add-apt-repository ppa:mc3man/gstffmpeg-keep
sudo apt-get update
sudo apt-get install gstreamer0.10-ffmpeg


END EDIT

I was trying to develop an application that uses gstreamer 0.10 on Ubuntu 14.04 and I noticed this runtime error message:

"No decoder available for type 'video/x-h264, stream-format=(string)avc, alignment=(string)au, level=(string)3, profile=(string)main, codec_data=(buffer)014d401effe10015274d401ea9181b07bcde00d4040406db0ad7bdf01001000428de09c8, width=(int)854, height=(int)480, framerate=(fraction)24/1, pixel-aspect-ratio=(fraction)1/1, parsed=(boolean)true'."

This is because it seems that the gstreamer 0.10 ffmpeg plugin is missing in 14.04. This also makes my Qt code fail it seems, as it uses 0.10 for the Qt Multimedia module. It used to be sufficient to install ubuntu-restricted-extras until 13.10, but it seems something has changed in 14.04. I really couldn't find that plugin anywhere soI simply built that from the sources here. Luckily it also contains the ffmpeg sources and builds those statically it seems.
Download the package, extract and:

./configure make make install

You'll probably need to install the usual build-essentials, gstreamer0.10, libgstreamer0.10, libgstreamer-plugins-base0.10-dev etc... It may fail with an error on a macro like this:

./libavcodec/get_bits.h: In function 'skip_bits': ./libavcodec/get_bits.h:387:1: warning: variable 're_cache' set but not used [-Wunused-but-set-variable] libavcodec/x86/h264_qpel_mmx.c: Assembler messages: libavcodec/x86/h264_qpel_mmx.c:1294: Error: operand type mismatch for `cmp' libavcodec/x86/h264_qpel_mmx.c:1294: Error: operand type mismatch for `cmp' libavcodec/x86/h264_qpel_mmx.c:1298: Error: operand type mismatch for `cmp' libavcodec/x86/h264_qpel_mmx.c:1298: Error: operand type mismatch for `cmp' libavcodec/x86/h264_qpel_mmx.c:964: Error: operand type mismatch for `cmp' libavcodec/x86/h264_qpel_mmx.c:964: Error: operand type mismatch for `cmp' libavcodec/x86/h264_qpel_mmx.c:964: Error: operand type mismatch for `cmp' make[1]: *** [libavcodec/x86/dsputil_mmx.o] Error 1

If that happens just apply the patch here. Place the plugin libgstffmpeg.so into /usr/lib/x86_64-linux-gnu/gstreamer-0.10, or wherever you have your gstreamer plugins and you're done. Bye! ;-)

Updates on Hardware Accelerated Qt Multimedia Backend on Raspberry Pi (4.15.2014)

For those of you still working on this exciting subject, I did some new changes to PiOmxTextures and it still works pretty good. To have more information on PiOmxTextures refer to this older article. PiOmxTextures is the code behind the creation of a Qt Multimedia backend to provide hardware accelerated video (and software-decoded audio using ffmpeg) decoding and rendering from/in QML scenes on Raspberry Pi. PiOmxPlayer leverages the code from omxplayer. These are the new additions:

1. Imported code from omxplayer to revision 9b0793faf2c12f49b743ae3778ece03e31ed1538.
2. Now using ffmpeg 2.2.
3. Now it can be built using gcc 4.8 using the recent Raspberry Pi Linaro toolchain released in 2014.
4. Tested using Qt 5.3.0 beta.
5. Tested on the most recent Raspberry Pi firmware: it seems Broadcom has made some changes (fixes?) to the egl_render component. Use the new patch to Qt Multimedia and rebuild. The hack I was forced to add to compensate is no more needed.

Everything seems to keep working correctly, both in 1080p and 720p. Enjoy: https://github.com/carlonluca/pi. Bye! ;-)

Installing Meld on Mac OS X

I really don't like FileMerge.app: I've always been feeling that Mac OS was missing a good tool like meld. Actually this turns out to be simply false: installing meld on Mac OS is pretty simple. Here are the steps:

• Install the great macports.
• Install meld. This will take some time. Keep in mind that at the time of writing, because of a bug, it is necessary to install rarian first:
  
  sudo port install rarian
  sudo port install meld

• Start the dbus service and make it start at boot:
  
  sudo launchctl load -w /Library/LaunchDaemons/org.freedesktop.dbus-system.plist
  launchctl load -w /Library/LaunchAgents/org.freedesktop.dbus-session.plist

• Define the LC_ALL variable to your locale (i.e.):
  
  export LC_ALL=en_US

• Run meld (i.e.):
  
  /opt/local/bin/meld

Pretty simple! Bye! ;-)

Comfortable Logcat Output on Linux and Mac OS

Android's logcat is really a brilliant tool for analysis and debugging. Unfortunately I really don't like using it from Eclipse and using from the shell is not entirely comfortable, being it completely uniform. logcat-colorize is a great tool to colorize logcat output and make it really clear. Many thanks to the author! Unfortunately it was not working on Mac OS, so I forked and fixed; everything seems pretty OK: you can find the final fix in my fork at the moment.

How to Build

Follow the readme for building on Mac OS. The project depends on boost, so install it through macports and simply make. Assuming the boost libs are in system paths or default macports paths you should be good to go

Download the Binary

If you prefer, I also built a static version that you can download and use right away on Mac OS 10.9. This version does not require boost or macports to be installed.

Bye! ;-)