Raspberry OS Bullseye and Cross Toolchain for Linux and Mac OS 10.2.1

Raspberry OS based on Debian bullseye was recently published. The compiler provided with this new version was updated to 10.2.1. As usual, I need to cross build many large projects for the platform, and I had problems in the past with cross toolchains not precisely matching the versions in the target sysroot. I therefore typically prefer to build my own toolchains based on the precise versions of the packages provided with the OS. These are the toolchains I’m using for the past Raspberry Pi systems:

• Cross toolchain GCC 6.3.0 for Stretch: here.
• Cross toolchain GCC 8.3.0 for Buster: here.

For Bullseye, these are the new toolchains I’m currently using:

• Cross toolchain GCC 10.2.1 for Linux hosts for Bullseye: here.
• Cross toolchain GCC 10.2.1 for Mac OS hosts for Bullseye: here.

As usual, place the toolchain in /opt/rpi, and the sysroot in /opt/rpi/sysroot. I tested these toolchains to build Qt 6.2.1 and everything seems to be fine so far.

Have fun! Bye 😉 

Cross-building the Ubuntu Kernel for the Raspberry Pi 4 (arm64)

I recently had to rebuild the Ubuntu kernel for the Raspberry Pi pretty often to apply some patches. Building on the Pi is simple, but takes A LOT of hours and completely saturates the rpi4. Also, it means installing many dev packages, which I do not typically need. I also tried to do it on a development rpi3: took many hours, failed a few times because of insufficient RAM etc… It is a pain.

I therefore decided to try to cross-build it from my machine, which is not a Ubuntu machine at the moment. Quickest and simplest way I found is to use Docker. The result seems to work, so I decided to also provide the tools I created for myself here: https://github.com/carlonluca/docker-rpi-ubuntu-kernel.

Usage

The image only contains the tools and the libs needed to compile. The kernel itself and the script is provided externally. A workspace directory will contain both the kernel code to build and the output debian packages. With the following command the script is executed in the container and you should get your debs:

docker run --rm -it --name builder -v $PWD/workspace:/workspace \
    -v $PWD/build.sh:/workspace/build.sh carlonluca/docker-rpi-ubuntu-kernel:focal \
    /workspace/build.sh

The workspace directory must contain a src directory with the kernel, so you’ll have to create it yourself and then patch your kernel:

mkdir workspace
cd workspace
git clone https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux-raspi/+git/hirsute src
[apply needed patches]

The resulting packages can then be moved to the pi and installed with:

sudo dpkg -i *.deb

Hope this tool can be useful! Bye ;-)

Versioning and CI/CD on a Raspberry Pi (or other arm embedded devices)

Blog

It's been many years now since I started to use a Pi as a version control server. I recently even started to use it more heavily for my development, using it for running my unit tests and for CI/CD. I list here the alternatives I tested and the results.

Subversion

The first version control system that I used on the Pi was SVN. It does not probably make much sense to use SVN as a version control service in 2021, but I still have old SVN repos that I do not want to migrate to git. If this is the case for you as well, user krisdavison prepared a docker image that includes a SVN server with a web server to browse the repos: https://hub.docker.com/r/krisdavison/svn-server. Unfortunately, krisdavison is only providing amd64 images, so I created my own fork, including images for armv7, arm64, x86 and x64, which should cover Pi versions from 2 to 4: https://hub.docker.com/r/carlonluca/docker-svn.
If you prefer, Raspbian also offers the SVN server package and WebSVN for browsing.

Gitea

At the beginning, I simply installed git in my Raspbian to be able to push to remotes stored in my pi. This was quick, worked well and I could also browse with GitWeb https://git-scm.com/book/en/v2/Git-on-the-Server-GitWeb. Raspbian includes all you need to setup git and GitWeb on Apache.

After some time though, I started to feel this was a bit restrictive. I could benefit from an issue tracker, a better browser experience with syntax highlighting etc... Therefore, I started to think about installing GitLab on my Pi. On my first Pi 2, this was difficult and, as a matter of fact, impossible, due to hardware limitations (it was taking minutes to load the first login page). 1GB of RAM, with other services running, is not enough for GitLab, so I started to look for other options and I found Gitea: https://gitea.io. Gitea is an excellent git service written in Go, it does not provide all the features GitLab offers, but is incredibly fast and requires much much less RAM. Unfortunately, the Gitea team was not providing a docker arm image for my Pi (https://hub.docker.com/r/gitea/gitea), so I started to build my own fork for x86, amd64, armv6, armv7 and armv8. For Pi 2, the armv7 image was working perfectly fine: https://hub.docker.com/r/carlonluca/gitea.

Gitea is a great self-hosted service if you are running on a Pi. It is a git service with many features, it is fast and it is lightweight.

GitLab

Unfortunately, all good things come to an end :-( and my glorious Pi 2 died in a tragical stormy summer night. RIP. I therefore "had" to switch to a shiny new Pi 4, with 4GB of RAM. This led me to think: "hey, what about GitLab now"?!
It seems GitLab is not currently providing docker images for arm64 (or any other arm variant): https://hub.docker.com/r/gitlab/gitlab-ce. I therefore built my own image for arm64: https://hub.docker.com/r/carlonluca/gitlab. The result is awesome. GitLab is a bit heavy, but it works well. Follow the same instructions of the original image (https://docs.gitlab.com/omnibus/docker) and everything should work. It is a bit slower than Gitea, much heavier, but has more to offer. In particular, package repos and the docker registry can be very handy.

Reference repo for GitLab docker image is https://github.com/carlonluca/docker-gitlab.

Jenkins

Another interesting topic in software development is continuous integration and deployment. As I'm used to Jenkins, this is what I looked into at first. There is a good official docker image for Jenkins: https://hub.docker.com/r/jenkins/jenkins. Unfortunately, atm, the image is only provided for amd64. I therefore, again, forked and created my arm64 image: https://hub.docker.com/r/carlonluca/jenkins. On my Pi 4, Jenkins behaves fine. You can also create other docker containers in your Pi and communicate with those containers from the Jenkins container. Everything seems to be working fine.

GitLab Runner

After some time, I wanted to also try the CI/CD feature included in my GitLab arm64 image. I therefore created a few gitlab configurations and configured a GitLab runner on the same Pi 4. The configuration of the runner was a bit tricky, but seems to be possible. In particular I had troubles using my internal DNS service, but host networking for the runner seems to work so far. Luckily, the runner is already available for arm64: https://hub.docker.com/r/gitlab/gitlab-runner. I created a configuration for a node app and created a CI configuration using a mongo docker image for running my unit tests. Everything worked properly. So you can have your node + angular apps versioned and tested on your arm64 Pi.

Dind

Of course, I also wanted to create a docker image for my node app. So I tried to use the dind service included in GitLab, to see if it could be used on the Pi properly. Yes, it worked :-) So you can have your app versioned on the Pi, tested on the Pi, dockerized on your Pi and uploaded to docker hub or the GitLab registry on the Pi.

Multiarch

The image resulting from the previous step worked great, it can be installed and used properly... on arm64. Also, the images I listed above are pretty long to build and maintain. Gitea, in particular, needs a long building process on an Intel Xeon, because the simplest procedure needs emulators for cross-arch building. Also, I have a slow Internet service, which is even slower when uploading. I was wondering if it could be possible to do all this on the Pi and let it build and transfer for me, instead of keeping my main machine busy... Yes, it is technically possible :-) qemu seems to be able to emulate other arm archs and amd64 on an arm64 kernel.
To do this, I needed an image including docker and buildkit for arm64: I couldn't find one, so I created one for me: https://hub.docker.com/r/carlonluca/docker-multiarch. At this point I wrote the gitlab CI file and tested. Unfortunately, I got less encouraging results this time: I got many types of failures, probably due to several unrelated causes. I could however identify some:

• slow Internet connection: it seems that docker or buildkit have too short timeouts. Couldn't find a way to set the timeout, but reducing concurrency of builds reduced the frequency of errors (see below).
• ubuntu binfmt not properly working in some cases: I used this project this fix this issue: https://hub.docker.com/r/tonistiigi/binfmt. I use it in all my GitLab CI files.
• buildx building with high concurrency: the authors tried to make full use of the build machine by building the images concurrently. This is reasonable, but... they are not currently providing a way to reduce or remove concurrency when needed... which is less reasonable. On the Pi, concurrent complex builds are difficult/impossible. If a single build requires much RAM, 4 builds concurrently over emulators require really too much RAM. The only way I found to build sequentially is to build and push each arch separately, and then use the manifest tool to merge. Here is an example: https://github.com/carlonluca/darkmediawiki-docker/blob/master/.gitlab-ci.yml.

I'm still working on this topic, but I'm mostly able to multiarch-build on my Pi with GitLab CI/CD. All the docker images I listed above are built this way, including the docker-multiarch image, which requires itself to build itself :-) this is a handy way of keeping them up to date. The only partial exception is Gitea: the build procedure of the image also builds Gitea itself, and there seems to be an issue building for x86. The other archs seem to properly build.

This is a list of the images I'm currently using on my Pi 4 in this context:
* https://hub.docker.com/r/carlonluca/docker-svn
* https://hub.docker.com/r/carlonluca/gitea
* https://hub.docker.com/r/carlonluca/gitlab
* https://hub.docker.com/r/carlonluca/jenkins
* https://hub.docker.com/r/gitlab/gitlab-runner
* https://hub.docker.com/r/carlonluca/docker-multiarch


Have fun! ;-)

Raspbian Buster and Cross Toolchain for Linux and Mac OS 8.3.0

Raspbian Buster is out. It seems it updated gcc from version 6.3.0 (used for Stretch) to gcc version 8.3.0. This is not always needed, but I had a lot of problems in the past related to using an outdated crosscompiler, or the one provided in the raspberry repo: https://github.com/raspberrypi/tools. I therefore built my own toolchain for Stretch, which I provided here, both for Linux and Mac OS.

I did the same for Buster: I built my own toolchain and I built many projects with it, like Qt, ffmpeg etc... Just place the toolchain in /opt/rpi (probably position independent) and you should be done. This toolchain should be 100% compatible with gcc provided by Buster.

Download cross toolchain GCC 6.3.0 for Stretch here.
Download Linux x64 cross toolchain GCC 8.3.0 here.
Download Mac OS cross toolchain GCC 8.3.0 here.

Bye ;-)

POT 5.7.0 with Qt 5.10.0 built for armv8 with GCC 6.3.0 on Raspbian Stretch

This was a good test for my cross toolchain with gcc 6.3.0 I uploaded some days ago: the new Qt 5.10.0 built with -march=armv8-a -mtune=cortex-a53 -mfpu=crypto-neon-fp-armv8 -mfloat-abi=hard, including POT for video acceleration on Raspbian Stretch. The build includes most Qt modules plus Qt WebKit (the modernized QtWebKit from
Konstantin Tokarev, version 5.212, https://github.com/annulen/webkit) plus QtWebEngine 5.9.1, a bit patched to build here. I didn't test browsers much, you'll have to work on those yourself if you need them.

By using the Qt build and the cross toolchain you can build your own applications.

Download here the build for Qt 5.10.0 with POT 5.7.
Download here the cross toolchain.

Raspbian Stretch and Cross Toolchain for Linux and Mac OS

Raspbian Stretch is out for Raspberry Pi. I needed a cross toolchain with the same exact version used to build the system so I built my own. As I had my script ready, I also tried to build one for Mac OS and it seems to work for the moment. I tested these cross toolchains for my projects and seemed to be able to build everything that I tried: Qt libs, apps, omxplayer, ffmpeg etc...

Download Linux x64 cross toolchain GCC 6.3.0 for Stretch here.
Download Mac OS cross toolchain GCC 6.3.0 for Stretch here.

NOTE: The Mac OS version requires some packages to be installed from macports.
NOTE: The toolchain is built to crossbuild for armv7-a. I could also build for armv8 but not sure if it also works for armv6.

Have fun! Bye! ;-)

Binaries for POT 5.4.0 on Qt 5.7.0 and Experimental Accelerated QtWebKit

So far, no real complains about POT 5.4.0-beta1 so I suppose it is sufficiently safe to consider it POT 5.4.0. Refer to this article for more info.

Binaries for POT 5.4.0-beta1 on Qt 5.7.0 and Experimental Accelerated QtWebKit

This is a new binary package including the same content of POT 5.3.0-beta1 package, with some updates:

• new improvements in POT taken from omxplayer code;
• bump to ffmpeg 3.1;
• based on Qt 5.7.0.

The build includes support for network protocols and WebKit 1 supporting video html5 elements (including youtube).

This version is still experimental. Please report bugs on github.

Have fun! Bye! ;-)

Download POT 5.4.0-beta1 for Raspbian Jessie Lite Pi2 (also tested on Pi3) here (md5: e6b6a1f92b71ac2bdd3d9756efd7669c).

Binaries for Hardware Accelerated Qt Multimedia Backend on Raspberry Pi (4.4.0)

This is a package containing binaries for PiOmxTextures 4.4.0. The relevant improvements here over 4.3.0 are:

• Fixed horizontal tearing effect.
• Fixed leaks when starting/stopping and restarting video.
• Added support for 3.5mm analog audio output. It is now possible to select which one to use (hdmi, analog or both) by setting a AUDIO_OUT env variable (possible values are both, local and hdmi; default: hdmi).
• Fixed garbage at the beginning of the video.
• Merged code from omxplayer code base to revision 74aac. This includes update to ffmpeg 2.6.
• Other minor fixes and improvements.
• Port to Qt 5.4.1.

The package contains:

• piomxtextures_pocplayer: sample player which includes tests for animations, loops and commands. Have a look at the sources to understand how it works.
• Qt-rasp-5.4.1.tar: the entire tree of Qt 5.4.1 binaries built for Raspbian (firmware
     b71d7b6, rpath is set to /usr/local/Qt-rasp-5.4.1).
• Qt-rasp-5.4.1_host.tar: the utility to be used on the host when cross-building (built for x86 Kubuntu 13.10).

Current version of the package is 4.4.0: download.

To have a quick look, just place Qt-rasp-5.4.1 in /usr/local into your Pi. Now you can test running qmlvideo, qmlvideofx or piomxtextures_pocplayer from the Qt Multimedia examples. Something similar to this should come up: 

To build using the host tools:

1. Place the Qt libs provided in /usr/local in your Pi filesystem.
2. Place the sysroot of your Pi into /opt/rpi/sysroot (you should then have Qt libs into /opt/rpi/sysroot/usr/local/Qt-rasp-5.4.1).
3. Place the host tools in /usr/local.
4. Place the Linaro toolchain (https://github.com/raspberrypi/tools/tree/master/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian) in /opt/rpi (which means you should have /opt/rpi/gcc-linaro-arm-linux-gnueabihf-raspbian).
5. You should be done. 

I tested on a Ubuntu 15.04 64 bit and the procedure worked, just be sure to use the 64 bit toolchain (in the same position) and add compatibility libs (qmake is a 32 bit binary).

Bye! ;-)

NOTE: Please notice that the provided binaries contain both the gstreamer plugin (the default unaccelerated Qt multimedia plugin) and the PiOmxTexures (POT) plugin. You'll have to ensure that your app uses POT to run and not gstreamer. To avoid confusion, you can simply remove (or move away) the gstreamer plugin from the default path (/usr/local/Qt-rasp-5.4.1/plugins/mediaservice).

Qt on Linux i.MX 6

i.MX 6 is a good device. Qt runs pretty well on it. Also a plugin for hardware accelerated and zero-copy rendering seems to work pretty well. Simply building Qt for i.MX 6 makes it possible to use the Qt Multimedia plugin. This is the result (on Linux):

I also tested a provided Android port but the result was not as good.

Video with Animations on Raspberry Pi and A31s Compared

For many uses, playing video while animations are running is pretty useful and it is also considerably stressful for the GPU. I therefore tested a couple of cheap chips with the same Qt/QML sample app. I created a small demo video comparing Qt on A31s and Raspberry Pi (PiOmxTextures):

In the demo, all the media (video and images) are 720p, and the output is 720p for A31s and 1080p for Raspberry Pi (16bpp). It is pretty hard to tell from the video, but Raspberry seems to still be above A31s.
Bye! ;-)

A Tribute to Qt Portability :-)

The new ports of Qt to mobile systems like Android, iOS and WinRT are getting more and more reliable. I just wanted to have a look at the current stage of development and I therefore wrote a sample app. This is the result:

This is a quick app to test the portability of Qt on desktop, mobile and embedded.

The video shows client/server code, communicating via WebSockets (using the new WebSockets module in Qt), with a server in Qt/C++ running on Pi and with a fluid hardware accelerated interface in C++/QML running on Raspberry Pi, Mac OS X Maverick, Android Phone and Tablet, Windows 8 (Modern UI), Windows Phone 8, iOS and Linux Kubuntu. The server (on Raspberry Pi) stores information about tasks and the client provides a remote interface to manage. A client written using HTML5 canvas and WebSockets from the browser is also shown.

Unfortunately I don't own iOS and Winphone ARM devices, otherwise I'd add those as well :-) Good start anyway: this is really "write once, deploy anywhere".

Using the ports is simple: configure Qt Creator with the correct Qt build, setup the toolchain to use, qmake and build (for iOS and WinRT I preferred Xcode and Visual Studio respectively yet). Your done!
Pretty cool! Bye! ;-)

Light Logging Facilities for Android/iOS/Linux/MacOS/Windows

I don't like to reinvent the wheel, and there are many logging codes out there, but still I couldn't find what I wanted. Those were all either too complex or not exactly what I wanted. That is why I took my old macros and added some ideas taken from this good tutorial: http://www.drdobbs.com/cpp/logging-in-c/201804215.

The result of a few hours is this: https://github.com/carlonluca/LightLogger. I'm using it on Windows/Linux/Mac OS/iOS/Android. This is an overview:

1. Simple: include the header and build.
2. Logging to Android results in logs sent to logcat (consider having a look at logcat-colorize, pretty project: https://bitbucket.org/brunobraga/logcat-colorize).
3. Supports the usual feature of log levels.
4. Supports coloring of logs: coloring is implemented using ANSI Escape sequences on Linux and MacOS. On Windows I disabled it, but you can enable if you're using something like cygwin or similar. For iOS I use XcodeColors, a great project: https://github.com/robbiehanson/XcodeColors. So a specific implementation is reserved for that platform.
5. Selecting debug levels with a macro at build-time enables/disables logs. Compiler optimizations should, in most cases, simply strip logs entirely from the binary, resulting in no/minimal overhead.
6. It is possible to reimplement the "sink" of the logs by reimplementing an output.
7. I preferred the printf way of formatting logs to the stream implementation. Anyway I tried to provide both. The usage of a "null sink" when logs are disabled should, together with compiler optimizations, make the overhead minimal.
8. Each log is flushed to avoid issues related to buffering. This might increase the overhead, but it is simple to remove it.
9. On Windows/Linux/iOS a stack trace function is also available to show the current call stack.
10. Should be entirely thread-safe.
11. Each log can be associated to a tag; I commonly use this with grep to filter logs by module.

There a still things I don't like about this approach, like being impossible to use in C sources, needing Objective-C++ instead of simple Objective-C and so on. Still someone may find it useful, so I uploaded to github: https://github.com/carlonluca/LightLogger.

How to Use

Just include in your sources and you're done. Most useful functions are those level-based:

inline bool log_info_t_v(const char* log_tag, const char* format, va_list args);
inline bool log_info_t(const char* log_tag, const char* format, ...);
inline bool log_info_v(const char* format, va_list args);
inline bool log_info(const char* format, ...);

These functions work differently according to the platform: on Android send INFO logs to logcat, on iOS print colored text to Xcode (and thus the XcodeColors plugin is needed if you keep colors enabled), on Windows simply print text and on Mac OS/Linux print text with ANSI colors to the shell. I use the return type to do something like:

if (!condition)
   return log_warn("Ooops, something bad happened, returning failure.");

These functions are "wrappers" for the LC_Log template. You choose a delegate and call:

LC_Log(...).printf(...);
LC_Log(...) << "Some stream based " << "text.";

or use the default logger:

LC_LogDef(...).printf(...);
LC_LogDef(...) << "Some stream based " << "text.";

For specific cases I also needed to write text with specific attributes in the past, so I added something like:

inline bool log_formatted_t_v(
   const char* log_tag,
   LC_LogAttrib a,
   LC_LogColor c,
   const char* format,
   va_list args
   );
inline bool log_formatted_t(
   const char* log_tag,
   LC_LogAttrib a,
   LC_LogColor c,
   const char* format,
   ...
   );
inline bool log_formatted_v(
   const char* format,
   va_list args
   );
inline bool log_formatted(
   LC_LogAttrib a,
   LC_LogColor c,
   const char* format,
   ...
   );
inline bool log_formatted(
   LC_LogColor c,
   const char* format,
   ...
   );

There are a few macros I use to configure for each project: COLORING_ENABLED to enable colors, BUILD_LOG_LEVEL_* to set the logging verbosity and XCODE_COLORING_ENABLED to enable/disable XcodeColors support.

In github you'll find Qt, iOS/XCode and Android sample projects.

How it Works

Pretty simple: wrapper functions like log_info(...) use the template class LC_Log to print the string. The LC_Log delegates actual log call to the class of type T, which can be implemented according to the needs. Delegates I currently implemented are:

1. LC_Output2Std: outputs to standard output, adding ANSI escape codes.
2. LC_Output2FILE: write the logs to file (no escape codes here).
3. LC_OutputAndroid: implements logging to logcat.
4. LC_Output2XCodeColors: implements logging using XcodeColors format.

Hope you can find this useful. If you improve this, share your work! If you find issues, report them! Bye! ;-)