Showing posts with label JNI. Show all posts
Showing posts with label JNI. Show all posts

Friday, April 19, 2013

Accelerated Video Decoding and Rendering in QML with Necessitas Qt 4 for Android

Ok, I did this a couple of months ago but I just realized it might be of help to someone who is currently using Necessitas Qt4 for some project and still cannot use Qt5.
This is a sample code which shows how to create a custom QML component in the Qt4 Necessitas porting to use hardware acceleration on any Android devices with API level at least 11. The result is pretty good, you can check the demo I uploaded on youtube a couple of months ago (the third application shown is the one which is implemented over Qt 4):



The description says it all: "The third sample code uses a custom QML component written in C++ using a Qt 4.8.2 port for Android (Necessitas). Regular QML animations are then applied to the custom component. The semi-transparent image is a regular QML Image element with alpha set to 0.5."

The code is available now here on github: https://github.com/carlonluca/TextureStreaming. The project TextureStreaming can be opened with Qt Creator and run on a device (assuming API level constraint is met).

Take into consideration that Qt guys are working on the QtMultimedia backend for Android, and I think it should be available in Qt 5.2. You might want to try that also: http://qt.gitorious.org/qt/qtmultimedia/trees/dev/src/plugins/android.

How it Works

As you can see from the code, a custom QML component is used and placed in the QML scene. That component instantiates some Java classes through JNI glue code and use the Android standard Media Player to start decoding video and playing audio. The sink is set to be a SurfaceTexture instance, which provides the OpenGL texture that the custom QML component renders in the QML scene. Result is pretty good.

Sunday, February 10, 2013

Animations on a Surface Rendering Video on Android

I've been recently asked to do a little research on how to implement animations on a video surface in Android, somehow similarly to how I did in the previous posts on RPi. It seemed interesting so I tried to do some investigations.

After reading something in the Android's documentation, I took a random Android 4.0 low-cost tablet and I started analyzing the problem.

First thing I tried is creating a simple VideoView and applying some regular Android animations on it. I tried to apply a TranslateAnimation and a ScaleAnimation, but the result was that the video geometry didn't change, only a black square representing the view was animated. Seems to be more or less similar to this.

I also tried to use the 3.1 animation system, but the result was the video actually moving, but leaving a trace behind it. Both this defects might be related to how the video rendering is performed at the lower levels, so it might not be the case for other boards.

The only other thing I tried before starting to dig into the OpenGL world is to actually "create" an animation by changing the layout parameters applied to the VideoView. By interpolating the values like a damped harmonic ascillator I got the result in the video. Implementing it more accurately you might get much better results.

I therefore started to look at something different: starting from API level 11 the SurfaceTexture might be the solution to all the needs. By using this class as a surface for the MediaPlayer it is possible to stream the video to an OpenGL texture. This seems to work pretty well (see the video) and it is not difficult to implement if you know OpenGL.

Anyway, for simple tasks, OpenGL might be overkill, so I tried to look at some Android classes that could let me render the texture without needing to create the entire application in OpenGL. I have not found a way yet (if you do please add a comment!), but I started to think that, once again, Qt might be the answer :-)

The third sample application you see in the video is a custom QML item created by rendering in the Qt OpenGL context the texture provided by the SurfaceTexture class controlled using JNI. The result is very good. The QML code I used is exactly the same used in previous posts for the RPi sample application. The Qt porting for Android I used is the old Necessitas alpha 4

EDIT: If API level 14+ is available, then it is possible to render the texture provided by the SurfaceTexture in a TextureView (thanks Tim for pointing this out!): this is the fourth sample in the video.

Friday, August 17, 2012

Programming using the Android NDK with Qt Creator

I recently had to implement libraries for the Android OS using the Android NDK, or implementing JNI bindings for others to leverage C++ libraries from the Java layer. I commonly did this using simple toold like kate or similar text editors. To be sincere, it was not a completely satisfying approach.
During one of my projects, I had to port a Qt application to Android, using the excellent necessitas SDK. This excellent SDK made it possible to implement for Android using C++, the NDK and Qt Creator as the preferred IDE. I started from here to think I could use Qt Creator to more generally program in C++/JNI to implement libraries for Android, with or without the Qt libraries. This is how I did it.

Download the necessary tools

  1. Download the Qt libraries (I won't use the libraries, I only need the environment).
  2. Download Qt Creator.
  3. Download the Android NDK for your platform.

Add the Android platform specification

I took the platform specification used in the necessitas project and modified it a bit. Look for the directory containing the platform specifications (<qt_libs>/mkspecs), and place in there the platform specification file and the qmake configuration file. You'll have to define at least the ANDROID_NDK_ROOT to the root of the NDK you downloaded.

Setup Qt Creator

In Qt Creator you can specify a new toolchain to use: go to the preferences, "Build & Run", "Toolchains", "Add" and choose "GCC". Then, select the gcc compile from the Android NDK (I see it under <ndk_root>/toolchains/arm-linux-androideabi-4.4.3/prebuilt/darwin-x86/bin/arm-linux-androideabi-g++).
You'll have to create a new build configuration in the project properties: add a new specification with the default build specification and name it differently. Also, under "Build Steps", add "-spec android-g++" to the additional arguments. This will tell Qt Creator to build according to the Android specifications we added before. Also, select the Android toolchain in the "General" section.

Additional setup

Consider that I also had to remove QMAKE_INCDIR_QT in some cases because it was adding /usr/include to the include paths, thus making the build fail. You may want to add that to the qmake.conf file.

Creating native executables for Android using qmake and Qt Creator

It is sometime useful to creato native executables for Android. You can use this technique to do it very simply. For instance, this is a test I created in a minute (you can use Qt Creator or the command line directly):

nano TestProject.pro

QT              -= core gui
TEMPLATE        = app
SOURCES         = main.cpp
QMAKE_INCDIR_QT =

Now write a simple "Hello Android!" application:

nano main.cpp

#include <stdio.h>

int main(int argc, char** argv)
{
   printf("Hello Android!\n");
   return 0;
}

Now create the Makefile using qmake and the android platform specification:

qmake -spec android-g++
make

You'll get a TestProject execuable compiled for arm platform using the Android bionic C libraries. You can try uploading to an Android emulator:

luca-macbook:test_project luca$ adb remount
remount succeeded
luca-macbook:test_project luca$ adb push TestProject /data
173 KB/s (34639 bytes in 0.195s)
luca-macbook:test_project luca$ adb shell
# cd /data
# ./TestProject
Hello Android!
# rm TestProject

I used this approach on both Mac OS X and Linux. Never tried on Windows.

Wednesday, May 2, 2012

Embedded cross-platform development for Android, iOS and Linux Embedded

I recently came across a new problem: cross-platform development of libraries on iOS, Android and Linux Embedded devices. What I wanted was a cross-platform way of writing a library letting me perform as much non-GUI work as possible.

The only reasonable solution I could think of was developing the library entirely in C/C++ language. This can be quite long, but simple to recompile for all these platforms. Linux Embedded should commonly support entire C/C++ toolchains (like Codesourcery toolchains), iOS supports standard C/C++ quite well and can be used very simply from Objective-C using Objective-C++ and Android can make use of C/C++ libraries through a JNI interface (not as much quick as the others, but possible).

Ok, then C/C++ is my choice. Anyway, would it possible to increase my productivity by using some kind of general purpose C/C++ library instead of relying only on the standard library? There are many possible solutions to this, one of which is the Neptune library.

During the last months anyway, I've come to be very close to the Qt libraries. I started to love the completeness, the flexibility and the comfortable API Qt provides.

I've been looking with interest for months to the new development of the Qt porting to Android (thanks to BogDan Vatra) and iOS but I never had the chance to try. Anyway, those ports make use of the QPA build of Qt, which started with the Lighthouse project to provide Qt portability on the GUI level to other platforms. This is not exactly what I needed: I'm not interested at the moment on the GUI development. What I need is simpler. Anyway, I found this very useful to start working.

The environment I'm currently working with allows me to create a common Qt Makefile to be parsed by qmake to generate a Makefile for any platform I need: for Android the toolchain provided by the NDK and for iOS the toolchain provided by Apple.

After this introduction, I explain the steps I followed to setup the environment.

Setting up for Android development

First thing to do is to download the Android NDK from the official website. Uncompress it somewhere.

Download and compile the Android port of Qt: it might be good to compile a specification file for the build, it is quite simple, but the Android's toolchain is not a complete toolchain. This implies that some modifications to the Qt sources are needed. I started to do it, but for the moment simply downloading from the git the last sources from Necessitas is good enough.

Now you have both the Android toolchain and the Qt sources compiled for Android. You can select the new Qt platform and compile directly from Qt Creator. This is damn good, cause it makes it quite simple to implement the JNI interface also, which I've never found a way to write comfortably from Eclipse.

Once the library is compiled, you can place it into an Android project as instructed in the manual of the Android NDK and all is done. You have C/C++ sources with Qt support ready to be used in an Android project. You can also implement the JNI interface and include the sources only when building for android using the qmake scopes.

Setting up for iOS development

It might seem a paradox, but I found it more difficult to compile for iOS than for Android... What you can do is use the QPA specification for iOS that can be found in <Qt sources>/mkspecs/qpa/macx-iphone*. By using qmake from the command line you can create a Makefile for iOS and then compile it on a Mac:

qmake -spec qpa/macx-iphonesimulator-g++


you can also directly include Objective-C and Objective-C++ sources directly into the Qt project file by using the variable:

OBJECTIVE_SOURCES += source_list


Just remember that only static libraries are allowed on iOS, so include something like:


ios {
CONFIG += static
}

 

to compile statically when on iOS.

When I tried this for the first time it didn't work. It seems that for some reason there is a difference between my XCode setup and the one used as a reference for the platform specification. What I had to do is to fix the qmake.conf specification for the iOS platform so that I compiles correctly. I found this article very useful to see what needed to be changed.
In particular, for recent versions of Xcode, I had to change the variable QMAKE_IOS_DEV_PATH (in mkspecs/qpa/macx-iphonesimulator-g++/qmake.conf) from:

QMAKE_IOS_DEV_PATH = /Developer/Platforms/iPhoneSimulator.platform/Developer


to:

QMAKE_IOS_DEV_PATH = /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer

and the QMAKE_IOS_SDK_VERSION (in mkspecs/qpa/common/g++-base-macx-iphone.conf) to 5.1.

Using Qt libraries on Android and iOS

There is one other point when implementing libraries using Qt: many Qt features require an event loop to be running and that an instance of QCoreApplication to be instantiated somehow. To do this, it might be sufficient to run the event loop and instantiate QCoreApplication in a thread of the library: in the init function, just spawn a new thread and exec the event loop.
This is an example of what I tried: first I created a Java class like this:

package com.lukeshq.android.test.qt;

/**
 * Author Luca Carlon
 * Date: 05.10.2012
 */

public class QtTest {
   // Static initializer.
   static {
      System.loadLibrary("QtCore");
      System.loadLibrary("AndroidTest");
   }

   public QtTest() {
      new Thread(new Runnable() {
         @Override
         public void run() {
            while (true) {
               try {
                  Thread.sleep(1000);
               }
               catch (InterruptedException e) {
                  e.printStackTrace();
               }
               mySlot();
            }
         }
      }).start();
   }

   // Native interface.
   public native boolean mySlot();
}

then, in my library, I created a simple object like this:

class MyQObject : public QObject
{
   Q_OBJECT
public:
   explicit MyQObject(QObject* parent = 0);

public slots:
   void mySlot();
};

and this is the implementation of the bindings:

/*----------------------------------------------------------------------
|    includes
+---------------------------------------------------------------------*/
#include <QString>
#include <QtConcurrentRun>
#include <QCoreApplication>
#include <QTimer>
#include <jni.h>
#include <android/log.h>
#include "myqobject.h"

/*----------------------------------------------------------------------
|    declarations
+---------------------------------------------------------------------*/
MyQObject* myQObject;

/*----------------------------------------------------------------------
|    runQCoreApplication
+---------------------------------------------------------------------*/
void runQCoreApplication()
{
   // Instantiate QCoreApplication.
   int i = 1;
   char* c[1] = {"MyLib"};
   QCoreApplication a(i, c);

   // Start the QTimer.
   __android_log_print(ANDROID_LOG_INFO, "LibTag", "Starting QTimer!");
   QTimer* t = new QTimer();
   myQObject = new MyQObject();
   QObject::connect(t, SIGNAL(timeout()), myQObject, SLOT(mySlot()));
   t->setInterval(1000);
   t->setSingleShot(false);
   t->start();

   // Start the event loop.
   a.exec();
}

/*----------------------------------------------------------------------
|    JNI_OnLoad
+---------------------------------------------------------------------*/
extern "C" JNIEXPORT jint JNICALL
JNI_OnLoad(JavaVM* vm, void* reserved)
{
   Q_UNUSED(vm);
   Q_UNUSED(reserved);

   // Test QString.
   QString myString = QString("My QString!");
   __android_log_print(ANDROID_LOG_INFO, "LibTag", "%s", qPrintable(myString));

   // Start the QCoreApplication event loop.
   QtConcurrent::run(&runQCoreApplication);

   return JNI_VERSION_1_6;

}

/*----------------------------------------------------------------------
|    JNI_OnUnload
+---------------------------------------------------------------------*/
extern "C" JNIEXPORT void JNICALL
JNI_OnUnload(JavaVM* vm, void* reserved)
{
   Q_UNUSED(vm);
   Q_UNUSED(reserved);

   __android_log_print(ANDROID_LOG_INFO, "LibTag", "Unloading!");
}

/*----------------------------------------------------------------------
|    Java_com_lukeshq_android_test_qt_QtTest_myslot
+---------------------------------------------------------------------*/
extern "C" JNIEXPORT void JNICALL
Java_com_lukeshq_android_test_qt_QtTest_mySlot(JNIEnv* env, jobject thiz)
{
   Q_UNUSED(env);
   Q_UNUSED(thiz);

   QMetaObject::invokeMethod(myQObject, "mySlot");
}