Versioning and CI/CD on a Raspberry Pi (or other arm embedded devices)

Blog

It's been many years now since I started to use a Pi as a version control server. I recently even started to use it more heavily for my development, using it for running my unit tests and for CI/CD. I list here the alternatives I tested and the results.

Subversion

The first version control system that I used on the Pi was SVN. It does not probably make much sense to use SVN as a version control service in 2021, but I still have old SVN repos that I do not want to migrate to git. If this is the case for you as well, user krisdavison prepared a docker image that includes a SVN server with a web server to browse the repos: https://hub.docker.com/r/krisdavison/svn-server. Unfortunately, krisdavison is only providing amd64 images, so I created my own fork, including images for armv7, arm64, x86 and x64, which should cover Pi versions from 2 to 4: https://hub.docker.com/r/carlonluca/docker-svn.
If you prefer, Raspbian also offers the SVN server package and WebSVN for browsing.

Gitea

At the beginning, I simply installed git in my Raspbian to be able to push to remotes stored in my pi. This was quick, worked well and I could also browse with GitWeb https://git-scm.com/book/en/v2/Git-on-the-Server-GitWeb. Raspbian includes all you need to setup git and GitWeb on Apache.

After some time though, I started to feel this was a bit restrictive. I could benefit from an issue tracker, a better browser experience with syntax highlighting etc... Therefore, I started to think about installing GitLab on my Pi. On my first Pi 2, this was difficult and, as a matter of fact, impossible, due to hardware limitations (it was taking minutes to load the first login page). 1GB of RAM, with other services running, is not enough for GitLab, so I started to look for other options and I found Gitea: https://gitea.io. Gitea is an excellent git service written in Go, it does not provide all the features GitLab offers, but is incredibly fast and requires much much less RAM. Unfortunately, the Gitea team was not providing a docker arm image for my Pi (https://hub.docker.com/r/gitea/gitea), so I started to build my own fork for x86, amd64, armv6, armv7 and armv8. For Pi 2, the armv7 image was working perfectly fine: https://hub.docker.com/r/carlonluca/gitea.

Gitea is a great self-hosted service if you are running on a Pi. It is a git service with many features, it is fast and it is lightweight.

GitLab

Unfortunately, all good things come to an end :-( and my glorious Pi 2 died in a tragical stormy summer night. RIP. I therefore "had" to switch to a shiny new Pi 4, with 4GB of RAM. This led me to think: "hey, what about GitLab now"?!
It seems GitLab is not currently providing docker images for arm64 (or any other arm variant): https://hub.docker.com/r/gitlab/gitlab-ce. I therefore built my own image for arm64: https://hub.docker.com/r/carlonluca/gitlab. The result is awesome. GitLab is a bit heavy, but it works well. Follow the same instructions of the original image (https://docs.gitlab.com/omnibus/docker) and everything should work. It is a bit slower than Gitea, much heavier, but has more to offer. In particular, package repos and the docker registry can be very handy.

Reference repo for GitLab docker image is https://github.com/carlonluca/docker-gitlab.

Jenkins

Another interesting topic in software development is continuous integration and deployment. As I'm used to Jenkins, this is what I looked into at first. There is a good official docker image for Jenkins: https://hub.docker.com/r/jenkins/jenkins. Unfortunately, atm, the image is only provided for amd64. I therefore, again, forked and created my arm64 image: https://hub.docker.com/r/carlonluca/jenkins. On my Pi 4, Jenkins behaves fine. You can also create other docker containers in your Pi and communicate with those containers from the Jenkins container. Everything seems to be working fine.

GitLab Runner

After some time, I wanted to also try the CI/CD feature included in my GitLab arm64 image. I therefore created a few gitlab configurations and configured a GitLab runner on the same Pi 4. The configuration of the runner was a bit tricky, but seems to be possible. In particular I had troubles using my internal DNS service, but host networking for the runner seems to work so far. Luckily, the runner is already available for arm64: https://hub.docker.com/r/gitlab/gitlab-runner. I created a configuration for a node app and created a CI configuration using a mongo docker image for running my unit tests. Everything worked properly. So you can have your node + angular apps versioned and tested on your arm64 Pi.

Dind

Of course, I also wanted to create a docker image for my node app. So I tried to use the dind service included in GitLab, to see if it could be used on the Pi properly. Yes, it worked :-) So you can have your app versioned on the Pi, tested on the Pi, dockerized on your Pi and uploaded to docker hub or the GitLab registry on the Pi.

Multiarch

The image resulting from the previous step worked great, it can be installed and used properly... on arm64. Also, the images I listed above are pretty long to build and maintain. Gitea, in particular, needs a long building process on an Intel Xeon, because the simplest procedure needs emulators for cross-arch building. Also, I have a slow Internet service, which is even slower when uploading. I was wondering if it could be possible to do all this on the Pi and let it build and transfer for me, instead of keeping my main machine busy... Yes, it is technically possible :-) qemu seems to be able to emulate other arm archs and amd64 on an arm64 kernel.
To do this, I needed an image including docker and buildkit for arm64: I couldn't find one, so I created one for me: https://hub.docker.com/r/carlonluca/docker-multiarch. At this point I wrote the gitlab CI file and tested. Unfortunately, I got less encouraging results this time: I got many types of failures, probably due to several unrelated causes. I could however identify some:

• slow Internet connection: it seems that docker or buildkit have too short timeouts. Couldn't find a way to set the timeout, but reducing concurrency of builds reduced the frequency of errors (see below).
• ubuntu binfmt not properly working in some cases: I used this project this fix this issue: https://hub.docker.com/r/tonistiigi/binfmt. I use it in all my GitLab CI files.
• buildx building with high concurrency: the authors tried to make full use of the build machine by building the images concurrently. This is reasonable, but... they are not currently providing a way to reduce or remove concurrency when needed... which is less reasonable. On the Pi, concurrent complex builds are difficult/impossible. If a single build requires much RAM, 4 builds concurrently over emulators require really too much RAM. The only way I found to build sequentially is to build and push each arch separately, and then use the manifest tool to merge. Here is an example: https://github.com/carlonluca/darkmediawiki-docker/blob/master/.gitlab-ci.yml.

I'm still working on this topic, but I'm mostly able to multiarch-build on my Pi with GitLab CI/CD. All the docker images I listed above are built this way, including the docker-multiarch image, which requires itself to build itself :-) this is a handy way of keeping them up to date. The only partial exception is Gitea: the build procedure of the image also builds Gitea itself, and there seems to be an issue building for x86. The other archs seem to properly build.

This is a list of the images I'm currently using on my Pi 4 in this context:
* https://hub.docker.com/r/carlonluca/docker-svn
* https://hub.docker.com/r/carlonluca/gitea
* https://hub.docker.com/r/carlonluca/gitlab
* https://hub.docker.com/r/carlonluca/jenkins
* https://hub.docker.com/r/gitlab/gitlab-runner
* https://hub.docker.com/r/carlonluca/docker-multiarch


Have fun! ;-)