Docker Image for the Awesome MLDonkey Service

I really like this excellent piece of software. I found some docker images for it for the arm64 architecture, but they seemed not to be very up to date, so I created my own. Here you can find my version of a dockerized MLDonkey service, built for many archs: armv7, arm64, x86 and x64. The arm versions are perfect for Raspberry Pi variants.

The latest version of the image is taken from this commit, which is the latest at the time of writing. It also includes a couple of patches that I sent to add a dark theme.

The image is built on top of debian:buster. For the moment, it is difficult to use something newer because of this known issue.

• Image on Docker Hub: link;
• GitHub repo for the project: link.

To run it, you only need a single command:

$ docker run -i -t -v "`pwd`/data:/var/lib/mldonkey" carlonluca/mldonkey

For more options read the readme in the repo.

Bye! 😉

Cross-building the Ubuntu Kernel for the Raspberry Pi 4 (arm64)

I recently had to rebuild the Ubuntu kernel for the Raspberry Pi pretty often to apply some patches. Building on the Pi is simple, but takes A LOT of hours and completely saturates the rpi4. Also, it means installing many dev packages, which I do not typically need. I also tried to do it on a development rpi3: took many hours, failed a few times because of insufficient RAM etc… It is a pain.

I therefore decided to try to cross-build it from my machine, which is not a Ubuntu machine at the moment. Quickest and simplest way I found is to use Docker. The result seems to work, so I decided to also provide the tools I created for myself here: https://github.com/carlonluca/docker-rpi-ubuntu-kernel.

Usage

The image only contains the tools and the libs needed to compile. The kernel itself and the script is provided externally. A workspace directory will contain both the kernel code to build and the output debian packages. With the following command the script is executed in the container and you should get your debs:

docker run --rm -it --name builder -v $PWD/workspace:/workspace \
    -v $PWD/build.sh:/workspace/build.sh carlonluca/docker-rpi-ubuntu-kernel:focal \
    /workspace/build.sh

The workspace directory must contain a src directory with the kernel, so you’ll have to create it yourself and then patch your kernel:

mkdir workspace
cd workspace
git clone https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux-raspi/+git/hirsute src
[apply needed patches]

The resulting packages can then be moved to the pi and installed with:

sudo dpkg -i *.deb

Hope this tool can be useful! Bye ;-)

Docker Image for Qt Development and Continuous Integration on x64 and arm64

Intro

It happens from time to time that I would benefit from having a simple Docker image with all the deps needed to build and run Qt apps on specific Qt versions. For instance, I may need to build some binary on Mac OS for Linux or I may need to have an image for some continuous integration system.

Problem

I had a look around and there are quite a few projects: some do not provide an arm build, some rely on the Qt official installer, that is a bit tricky to automate and tends to change often, some rely on distro-provided packages, that do not provide every version etc… I’d prefer to have a solid build procedure to build images for any Qt version I want. Also, my CI system runs on arm64, so I’d need a multiarch image, and therefore two different Qt builds. I thought it could be simpler to just build Qt on-the-fly in a Dockerfile but… turned out it is not exactly simple.

Solution

At first I created a Dockerfile to:

1. install all the needed deps on Ubuntu focal;
2. download Qt sources;
3. configure;
4. build;
5. install;
6. cleanup.

This proved to be a simple solution, but building Qt for arm on qemu or building on arm took way too much. So I though of a different solution. I created two images:

• carlonluca/qt-builder is an image with the proper tools to build Qt for x64 and cross-build Qt for arm64;
• carlonluca/qt-dev is an image with the proper tools to build and run Qt apps.

With the first image Qt can be built and cross-built in a few hours for both x64 and arm64, directly on x64. The builds are then injected into a second multiarch image. This is way faster than building on qemu.

Only one problem left: in Qt 5, crossbuilding Qt results in some Qt tools like qmake to be built for the host arch, which is unacceptable in this case. So, in the second image, a build of the only qtbase module is also needed to build arm64 binaries. Qt 6 works differently instead.

Result

The code to build Qt and all the images are available at: https://github.com/carlonluca/docker-qt.

The image to build Qt is available at: https://hub.docker.com/repository/docker/carlonluca/qt-builder.

The image to use for app development is available at: https://hub.docker.com/repository/docker/carlonluca/qt-dev.

Continuous Integration on GitLab with Qt

An example of usage of the dev image is running continuous integration and unit tests on your code on Jenkins or GitLab. For example, I can now run my unit tests in my GitLab instance running on Raspberry Pi 4 with the following code (taken from my project https://github.com/carlonluca/lqobjectserializer):

variables:

GIT_SUBMODULE_STRATEGY: recursive

stages:
  - test_qt5
  - test_qt6

Test_qt5:
  stage: test_qt5
  image:
    name: "carlonluca/qt-dev:5.15.2"
    entrypoint: [""]
  script:
    - cd LQObjectSerializerTest
    - mkdir build
    - cd build
    - cmake ../qt5
    - make
    - ./LQObjectSerializerTest
    - ./LGithubTestCase

Test_qt6:
  stage: test_qt6
  image:
    name: "carlonluca/qt-dev:6.1.2"
    entrypoint: [""]
  script:
    - cd LQObjectSerializerTest
    - mkdir build
    - cd build
    - cmake ../qt6
    - make
    - ./LQObjectSerializerTest
    - ./LGithubTestCase

These images are still experimental though.

Have fun 😉

Bye!

Dark Theme for MediaWiki in Docker

I recently started to use MediaWiki as a way to store personal information, notes, links etc... It is comfortable to have all that info with me, properly structured, immediately editable online regardless of the operating system and versioned. Adding an instance of MediaWiki to an existent setup is straightforward, and I like to do it with docker. The official docker image of MediaWiki is already multiarch, so I could add it to my Raspberry Pi quickly. Default MediaWiki includes a light theme, but does not seem to include a dark theme. This is where this project by Martynov Maxim comes to help: https://github.com/dolfinus/DarkVector. You'll have to add it to your MediaWiki container and select it for your users.

Keeping MediaWiki up to date (and only usable through HTTPS) is important for security reasons (https://www.mediawiki.org/wiki/Manual:Security) so I created my own MediaWiki multiarch image including that theme by default. You can freely use it: https://hub.docker.com/repository/docker/carlonluca/darkmediawiki. I use it successfully on my aarch64 installation. This is the result:

Refreshing the image is almost effertless thanks to the CI/CD capabilities of GitLab.
For more info refer to the GitHub project: https://github.com/carlonluca/darkmediawiki-docker. Have fun ;-)

Versioning and CI/CD on a Raspberry Pi (or other arm embedded devices)

Blog

It's been many years now since I started to use a Pi as a version control server. I recently even started to use it more heavily for my development, using it for running my unit tests and for CI/CD. I list here the alternatives I tested and the results.

Subversion

The first version control system that I used on the Pi was SVN. It does not probably make much sense to use SVN as a version control service in 2021, but I still have old SVN repos that I do not want to migrate to git. If this is the case for you as well, user krisdavison prepared a docker image that includes a SVN server with a web server to browse the repos: https://hub.docker.com/r/krisdavison/svn-server. Unfortunately, krisdavison is only providing amd64 images, so I created my own fork, including images for armv7, arm64, x86 and x64, which should cover Pi versions from 2 to 4: https://hub.docker.com/r/carlonluca/docker-svn.
If you prefer, Raspbian also offers the SVN server package and WebSVN for browsing.

Gitea

At the beginning, I simply installed git in my Raspbian to be able to push to remotes stored in my pi. This was quick, worked well and I could also browse with GitWeb https://git-scm.com/book/en/v2/Git-on-the-Server-GitWeb. Raspbian includes all you need to setup git and GitWeb on Apache.

After some time though, I started to feel this was a bit restrictive. I could benefit from an issue tracker, a better browser experience with syntax highlighting etc... Therefore, I started to think about installing GitLab on my Pi. On my first Pi 2, this was difficult and, as a matter of fact, impossible, due to hardware limitations (it was taking minutes to load the first login page). 1GB of RAM, with other services running, is not enough for GitLab, so I started to look for other options and I found Gitea: https://gitea.io. Gitea is an excellent git service written in Go, it does not provide all the features GitLab offers, but is incredibly fast and requires much much less RAM. Unfortunately, the Gitea team was not providing a docker arm image for my Pi (https://hub.docker.com/r/gitea/gitea), so I started to build my own fork for x86, amd64, armv6, armv7 and armv8. For Pi 2, the armv7 image was working perfectly fine: https://hub.docker.com/r/carlonluca/gitea.

Gitea is a great self-hosted service if you are running on a Pi. It is a git service with many features, it is fast and it is lightweight.

GitLab

Unfortunately, all good things come to an end :-( and my glorious Pi 2 died in a tragical stormy summer night. RIP. I therefore "had" to switch to a shiny new Pi 4, with 4GB of RAM. This led me to think: "hey, what about GitLab now"?!
It seems GitLab is not currently providing docker images for arm64 (or any other arm variant): https://hub.docker.com/r/gitlab/gitlab-ce. I therefore built my own image for arm64: https://hub.docker.com/r/carlonluca/gitlab. The result is awesome. GitLab is a bit heavy, but it works well. Follow the same instructions of the original image (https://docs.gitlab.com/omnibus/docker) and everything should work. It is a bit slower than Gitea, much heavier, but has more to offer. In particular, package repos and the docker registry can be very handy.

Reference repo for GitLab docker image is https://github.com/carlonluca/docker-gitlab.

Jenkins

Another interesting topic in software development is continuous integration and deployment. As I'm used to Jenkins, this is what I looked into at first. There is a good official docker image for Jenkins: https://hub.docker.com/r/jenkins/jenkins. Unfortunately, atm, the image is only provided for amd64. I therefore, again, forked and created my arm64 image: https://hub.docker.com/r/carlonluca/jenkins. On my Pi 4, Jenkins behaves fine. You can also create other docker containers in your Pi and communicate with those containers from the Jenkins container. Everything seems to be working fine.

GitLab Runner

After some time, I wanted to also try the CI/CD feature included in my GitLab arm64 image. I therefore created a few gitlab configurations and configured a GitLab runner on the same Pi 4. The configuration of the runner was a bit tricky, but seems to be possible. In particular I had troubles using my internal DNS service, but host networking for the runner seems to work so far. Luckily, the runner is already available for arm64: https://hub.docker.com/r/gitlab/gitlab-runner. I created a configuration for a node app and created a CI configuration using a mongo docker image for running my unit tests. Everything worked properly. So you can have your node + angular apps versioned and tested on your arm64 Pi.

Dind

Of course, I also wanted to create a docker image for my node app. So I tried to use the dind service included in GitLab, to see if it could be used on the Pi properly. Yes, it worked :-) So you can have your app versioned on the Pi, tested on the Pi, dockerized on your Pi and uploaded to docker hub or the GitLab registry on the Pi.

Multiarch

The image resulting from the previous step worked great, it can be installed and used properly... on arm64. Also, the images I listed above are pretty long to build and maintain. Gitea, in particular, needs a long building process on an Intel Xeon, because the simplest procedure needs emulators for cross-arch building. Also, I have a slow Internet service, which is even slower when uploading. I was wondering if it could be possible to do all this on the Pi and let it build and transfer for me, instead of keeping my main machine busy... Yes, it is technically possible :-) qemu seems to be able to emulate other arm archs and amd64 on an arm64 kernel.
To do this, I needed an image including docker and buildkit for arm64: I couldn't find one, so I created one for me: https://hub.docker.com/r/carlonluca/docker-multiarch. At this point I wrote the gitlab CI file and tested. Unfortunately, I got less encouraging results this time: I got many types of failures, probably due to several unrelated causes. I could however identify some:

• slow Internet connection: it seems that docker or buildkit have too short timeouts. Couldn't find a way to set the timeout, but reducing concurrency of builds reduced the frequency of errors (see below).
• ubuntu binfmt not properly working in some cases: I used this project this fix this issue: https://hub.docker.com/r/tonistiigi/binfmt. I use it in all my GitLab CI files.
• buildx building with high concurrency: the authors tried to make full use of the build machine by building the images concurrently. This is reasonable, but... they are not currently providing a way to reduce or remove concurrency when needed... which is less reasonable. On the Pi, concurrent complex builds are difficult/impossible. If a single build requires much RAM, 4 builds concurrently over emulators require really too much RAM. The only way I found to build sequentially is to build and push each arch separately, and then use the manifest tool to merge. Here is an example: https://github.com/carlonluca/darkmediawiki-docker/blob/master/.gitlab-ci.yml.

I'm still working on this topic, but I'm mostly able to multiarch-build on my Pi with GitLab CI/CD. All the docker images I listed above are built this way, including the docker-multiarch image, which requires itself to build itself :-) this is a handy way of keeping them up to date. The only partial exception is Gitea: the build procedure of the image also builds Gitea itself, and there seems to be an issue building for x86. The other archs seem to properly build.

This is a list of the images I'm currently using on my Pi 4 in this context:
* https://hub.docker.com/r/carlonluca/docker-svn
* https://hub.docker.com/r/carlonluca/gitea
* https://hub.docker.com/r/carlonluca/gitlab
* https://hub.docker.com/r/carlonluca/jenkins
* https://hub.docker.com/r/gitlab/gitlab-runner
* https://hub.docker.com/r/carlonluca/docker-multiarch


Have fun! ;-)