Friday, August 17, 2012

Programming using the Android NDK with Qt Creator

I recently had to implement libraries for the Android OS using the Android NDK, or implementing JNI bindings for others to leverage C++ libraries from the Java layer. I commonly did this using simple toold like kate or similar text editors. To be sincere, it was not a completely satisfying approach.
During one of my projects, I had to port a Qt application to Android, using the excellent necessitas SDK. This excellent SDK made it possible to implement for Android using C++, the NDK and Qt Creator as the preferred IDE. I started from here to think I could use Qt Creator to more generally program in C++/JNI to implement libraries for Android, with or without the Qt libraries. This is how I did it.

Download the necessary tools

  1. Download the Qt libraries (I won't use the libraries, I only need the environment).
  2. Download Qt Creator.
  3. Download the Android NDK for your platform.

Add the Android platform specification

I took the platform specification used in the necessitas project and modified it a bit. Look for the directory containing the platform specifications (<qt_libs>/mkspecs), and place in there the platform specification file and the qmake configuration file. You'll have to define at least the ANDROID_NDK_ROOT to the root of the NDK you downloaded.

Setup Qt Creator

In Qt Creator you can specify a new toolchain to use: go to the preferences, "Build & Run", "Toolchains", "Add" and choose "GCC". Then, select the gcc compile from the Android NDK (I see it under <ndk_root>/toolchains/arm-linux-androideabi-4.4.3/prebuilt/darwin-x86/bin/arm-linux-androideabi-g++).
You'll have to create a new build configuration in the project properties: add a new specification with the default build specification and name it differently. Also, under "Build Steps", add "-spec android-g++" to the additional arguments. This will tell Qt Creator to build according to the Android specifications we added before. Also, select the Android toolchain in the "General" section.

Additional setup

Consider that I also had to remove QMAKE_INCDIR_QT in some cases because it was adding /usr/include to the include paths, thus making the build fail. You may want to add that to the qmake.conf file.

Creating native executables for Android using qmake and Qt Creator

It is sometime useful to creato native executables for Android. You can use this technique to do it very simply. For instance, this is a test I created in a minute (you can use Qt Creator or the command line directly):

nano TestProject.pro

QT              -= core gui
TEMPLATE        = app
SOURCES         = main.cpp
QMAKE_INCDIR_QT =

Now write a simple "Hello Android!" application:

nano main.cpp

#include <stdio.h>

int main(int argc, char** argv)
{
   printf("Hello Android!\n");
   return 0;
}

Now create the Makefile using qmake and the android platform specification:

qmake -spec android-g++
make

You'll get a TestProject execuable compiled for arm platform using the Android bionic C libraries. You can try uploading to an Android emulator:

luca-macbook:test_project luca$ adb remount
remount succeeded
luca-macbook:test_project luca$ adb push TestProject /data
173 KB/s (34639 bytes in 0.195s)
luca-macbook:test_project luca$ adb shell
# cd /data
# ./TestProject
Hello Android!
# rm TestProject

I used this approach on both Mac OS X and Linux. Never tried on Windows.

Saturday, August 11, 2012

Compiling SQLCipher for Windows

SQLCipher is a great extension to the SQLite library. It can be used to encrypt the database when using the SQLite API. It can be compiled for the major platforms like Linux, Mac OS X, iOS, Android and Microsoft Windows.
It is very simple to compile SQLCipher, except for the Windows case, where it can take some more time to get a working library. I spent some hours to have a static library, and I suppose it might be useful to write it down to accelerate the process for anybody who needs it.

In this thread you can find some information on how to compile SQLCipher for Windows. Instead, I preferred to take a different way: I didn't bother compiling OpenSSL (which is needed to compile SQLCipher), and used cygwin to buil the SQLCipher library linking to OpenSSL.

Download SQLCipher sources

First of all download the SQLCipher sources, with a zip archive or from the git. Uncompress the sources somewhere (I'll refer to this path using <sqlcipher_sources>).

Download the OpenSSL library

You can either compile OpenSSL using the Microsoft compiler or mingw or you can download the library from this website. I'll refer to the path containing the libraries with <openssl_binaries>.

Download and setup cygwin

I used cygwin to compile SQLCipher. Download it from the website. When requested, install at least these packages:
  1. make
  2. nano (in case you need to edit)
  3. tcl: Tool Command Language
  4. gcc-mingw-core
  5. mingw-gcc-core
This should be sufficient.

Compile SQLCipher from the cygwin command line

I used MinGW to compile the sources. First of all set the correct C compiler that make will use to compile the sources:

export CC=i686-pc-mingw32-gcc

Now go to the SQLCipher directory:

cd sqlcipher_sources

and lunch the configure script. We will need to lunch it this way:

./configure --disable-tcl CFLAGS="-DSQLITE_HAS_CODEC -Iopenssl_binaries/include" LDFLAGS="openssl_binaries/lib/libeay32.lib /usr/lib/gcc/i686-pc-mingw32/4.5.2/libgcc.a"

This will create the Makefile. Now, you can use the make command to build the library.
At a certain point, I got this message:

libtool: link: i686-pc-mingw32-gcc -DSQLITE_HAS_CODEC -I/cygdrive/c/OpenSSL-Win32/include -DSQLITE_OS_WIN=1 -I. -I./src -I./ext/rtree -D_HAVE_SQLITE_CONFIG_H -DBUILD_sqlite -DNDEBUG -DSQLITE_THREADSAFE=1 -DSQLITE_OMIT_LOAD_EXTENSION=1 /cygdrive/c/OpenSSL-Win32/lib/libeay32.lib -DHAVE_READLINE=0 -o .libs/sqlite3.exe ./src/shell.c /usr/lib/gcc/i686-pc-mingw32/4.5.2/libgcc.a ./.libs/libsqlite3.a -lpthread -L/usr/local/lib
./.libs/libsqlite3.a(sqlite3.o):sqlite3.c:(.text+0xb04): undefined reference to `_EVP_get_cipherbyname'
./.libs/libsqlite3.a(sqlite3.o):sqlite3.c:(.text+0xb0d): undefined reference to `_OPENSSL_add_all_algorithms_noconf'
./.libs/libsqlite3.a(sqlite3.o):sqlite3.c:(.text+0xbeb): undefined reference to `_RAND_bytes'
./.libs/libsqlite3.a(sqlite3.o):sqlite3.c:(.text+0x110f): undefined reference to `_EVP_get_cipherbyname'
./.libs/libsqlite3.a(sqlite3.o):sqlite3.c:(.text+0x1123): undefined reference to `_EVP_CIPHER_key_length'
./.libs/libsqlite3.a(sqlite3.o):sqlite3.c:(.text+0x113a): undefined reference to `_EVP_CIPHER_iv_length'
./.libs/libsqlite3.a(sqlite3.o):sqlite3.c:(.text+0x1151): undefined reference to `_EVP_CIPHER_block_size'
./.libs/libsqlite3.a(sqlite3.o):sqlite3.c:(.text+0x115f): undefined reference to `_EVP_sha1'
./.libs/libsqlite3.a(sqlite3.o):sqlite3.c:(.text+0x1167): undefined reference to `_EVP_MD_size'
./.libs/libsqlite3.a(sqlite3.o):sqlite3.c:(.text+0x1748): undefined reference to `_HMAC_CTX_init'
./.libs/libsqlite3.a(sqlite3.o):sqlite3.c:(.text+0x174d): undefined reference to `_EVP_sha1'
./.libs/libsqlite3.a(sqlite3.o):sqlite3.c:(.text+0x1784): undefined reference to `_HMAC_Init_ex'
./.libs/libsqlite3.a(sqlite3.o):sqlite3.c:(.text+0x17a3): undefined reference to `_HMAC_Update'
./.libs/libsqlite3.a(sqlite3.o):sqlite3.c:(.text+0x17c3): undefined reference to `_HMAC_Update'
./.libs/libsqlite3.a(sqlite3.o):sqlite3.c:(.text+0x17e3): undefined reference to `_HMAC_Final'
./.libs/libsqlite3.a(sqlite3.o):sqlite3.c:(.text+0x17f3): undefined reference to `_HMAC_CTX_cleanup'
./.libs/libsqlite3.a(sqlite3.o):sqlite3.c:(.text+0x1a1e): undefined reference to `_EVP_CipherInit'
./.libs/libsqlite3.a(sqlite3.o):sqlite3.c:(.text+0x1a34): undefined reference to `_EVP_CIPHER_CTX_set_padding'
./.libs/libsqlite3.a(sqlite3.o):sqlite3.c:(.text+0x1a65): undefined reference to `_EVP_CipherInit'
./.libs/libsqlite3.a(sqlite3.o):sqlite3.c:(.text+0x1a8f): undefined reference to `_EVP_CipherUpdate'
./.libs/libsqlite3.a(sqlite3.o):sqlite3.c:(.text+0x1ab7): undefined reference to `_EVP_CipherFinal'
./.libs/libsqlite3.a(sqlite3.o):sqlite3.c:(.text+0x1acb): undefined reference to `_EVP_CIPHER_CTX_cleanup'
./.libs/libsqlite3.a(sqlite3.o):sqlite3.c:(.text+0x1c23): undefined reference to `_PKCS5_PBKDF2_HMAC_SHA1'
./.libs/libsqlite3.a(sqlite3.o):sqlite3.c:(.text+0x1cf1): undefined reference to `_PKCS5_PBKDF2_HMAC_SHA1'
collect2: ld returned 1 exit status
Makefile:512: recipe for target `sqlite3.exe' failed
make: *** [sqlite3.exe] Error 1

I'm sure there is a more elegant way of solving this, but simply correcting the command which failed was sufficient for me (the linking order is the cause; any static library or object that depends on other library should be placed before it in the command line, so libeay32.lib should be placed after libsqlite3.a):

i686-pc-mingw32-gcc -DSQLITE_HAS_CODEC -I/cygdrive/c/OpenSSL-Win32/include -DSQLITE_OS_WIN=1 -I. -I./src -I./ext/rtree -D_HAVE_SQLITE_CONFIG_H -DBUILD_sqlite -DNDEBUG -DSQLITE_THREADSAFE=1 -DSQLITE_OMIT_LOAD_EXTENSION=1 -DHAVE_READLINE=0 -o .libs/sqlite3.exe ./src/shell.c /usr/lib/gcc/i686-pc-mingw32/4.5.2/libgcc.a ./.libs/libsqlite3.a -lpthread -L/usr/local/lib /cygdrive/c/OpenSSL-Win32/lib/libeay32.lib

You should find the static library in sqlcipher_sources.lib directory, together with the sqlite3 executable.

Building a project linking to libsqlcipher.a

When building a project using the libsqlcipher, I had to link all the necessary libraries in the correct order:
  1. libeay32.lib
  2. libsqlite3.a
  3. libgcc.a
For example, in a Qt project I added this to link to the library:

win32 {
message("Compiling for Windows!")
LIBS += \
    $$_PRO_FILE_PWD_/os_specific_windows/3rdparty/lib/libeay32.lib \
    $$_PRO_FILE_PWD_/os_specific_windows/3rdparty/lib/libsqlite3.a \
    $$_PRO_FILE_PWD_/os_specific_windows/3rdparty/lib/libgcc.a

INCLUDEPATH += 3rdparty/include
}

Windows 7 64bit

After being asked, out of curiosity, I also tried to compile for Windows 7 64bit, and I followed the exact same procedure (I just downloaded and installed the 64bit version of all the libs and cygwin packages). The error during make is:

libtool: link: x86_64-w64-mingw32-gcc -shared .libs/sqlite3.o /usr/lib/gcc/x86_64-w64-mingw32/4.5.3/libgcc.a -lpthread -o .libs/cygsqlite3-0.dll -Wl,--enable-auto-image-base -Xlinker --out-implib -Xlinker .libs/libsqlite3.dll.a
Creating library file: .libs/libsqlite3.dll.a .libs/sqlite3.o:sqlite3.c:(.text+0xc73): undefined reference to `EVP_get_cipherbyname'
.libs/sqlite3.o:sqlite3.c:(.text+0xc7d): undefined reference to `OPENSSL_add_all_algorithms_noconf'
.libs/sqlite3.o:sqlite3.c:(.text+0xd7f): undefined reference to `RAND_bytes'
.libs/sqlite3.o:sqlite3.c:(.text+0x133a): undefined reference to `EVP_get_cipherbyname'
.libs/sqlite3.o:sqlite3.c:(.text+0x1352): undefined reference to `EVP_CIPHER_key_length'
.libs/sqlite3.o:sqlite3.c:(.text+0x136c): undefined reference to `EVP_CIPHER_iv_length'
.libs/sqlite3.o:sqlite3.c:(.text+0x1386): undefined reference to `EVP_CIPHER_block_size'
.libs/sqlite3.o:sqlite3.c:(.text+0x1395): undefined reference to `EVP_sha1'
.libs/sqlite3.o:sqlite3.c:(.text+0x139d): undefined reference to `EVP_MD_size'
.libs/sqlite3.o:sqlite3.c:(.text+0x1a37): undefined reference to `HMAC_CTX_init'
.libs/sqlite3.o:sqlite3.c:(.text+0x1a3c): undefined reference to `EVP_sha1'
.libs/sqlite3.o:sqlite3.c:(.text+0x1a76): undefined reference to `HMAC_Init_ex'
.libs/sqlite3.o:sqlite3.c:(.text+0x1a96): undefined reference to `HMAC_Update'
.libs/sqlite3.o:sqlite3.c:(.text+0x1ab3): undefined reference to `HMAC_Update'
.libs/sqlite3.o:sqlite3.c:(.text+0x1ad0): undefined reference to `HMAC_Final'
.libs/sqlite3.o:sqlite3.c:(.text+0x1ae2): undefined reference to `HMAC_CTX_cleanup'
.libs/sqlite3.o:sqlite3.c:(.text+0x1d38): undefined reference to `EVP_CipherInit'
.libs/sqlite3.o:sqlite3.c:(.text+0x1d4d): undefined reference to `EVP_CIPHER_CTX_set_padding'
.libs/sqlite3.o:sqlite3.c:(.text+0x1d7b): undefined reference to `EVP_CipherInit'
.libs/sqlite3.o:sqlite3.c:(.text+0x1da4): undefined reference to `EVP_CipherUpdate'
.libs/sqlite3.o:sqlite3.c:(.text+0x1dce): undefined reference to `EVP_CipherFinal'
.libs/sqlite3.o:sqlite3.c:(.text+0x1de4): undefined reference to `EVP_CIPHER_CTX_cleanup'
.libs/sqlite3.o:sqlite3.c:(.text+0x1f50): undefined reference to `PKCS5_PBKDF2_HMAC_SHA1'
.libs/sqlite3.o:sqlite3.c:(.text+0x2025): undefined reference to `PKCS5_PBKDF2_HMAC_SHA1'
/usr/lib/gcc/x86_64-w64-mingw32/4.5.3/../../../../x86_64-w64-mingw32/bin/ld: .libs/sqlite3.o: bad reloc address 0x1c0 in section `.data'
collect2: ld returned 1 exit status
Makefile:501: recipe for target `libsqlite3.la' failed
make: *** [libsqlite3.la] Error 1


So it seems that also the library now is failing to compile: maybe something has changed in some tool or in the Makefile, I don't know.
Luckily, sqlite3 uses amalgamation; it is therefore possible to compile manually quite simply. I built the library this way:

x86_64-w64-mingw32-ar cru .libs/libsqlite3.a /usr/lib/gcc/x86_64-w64-mingw32/4.5.3/libgcc.a sqlite3.o
x86_64-w64-mingw32-ranlib .libs/libsqlite3.a


since the object file sqlite3.o was already compiled by make, and the sqlcipher executable simply like this:

x86_64-w64-mingw32-gcc -DSQLITE_HAS_CODEC -I/cygdrive/c/OpenSSL-Win64/include -DSQLITE_OS_WIN=1 -I. -I./src -I./ext/rtree -D_HAVE_SQLITE_CONFIG_H -DBUILD_sqlite -DNDEBUG -DSQLITE_THREADSAFE=1 -DSQLITE_OMIT_LOAD_EXTENSION=1 -DHAVE_READLINE=0 -o .libs/sqlite3.exe ./src/shell.c /usr/lib/gcc/x86_64-w64-mingw32/4.5.3/libgcc.a .libs/libsqlite3.a /cygdrive/c/OpenSSL-Win64/lib/libeay32.lib

I tried to create a simple encrypted database and seemed to work.
Hope this helps!