RE: Build Flow - Docker CircleCI ROS ARM7

/

The last post briefly summarized the build flow for our system. That build flow remains the same, though we did switch from TravisCI to CircleCI. In this post, I'll offer more detail on the intricacies of the systems, as well as the "gotchas" that led me to implement them. Admittedly, ours is a unique setup. Because of that, I could find none of my solutions in one place. Hopefully, this serves as a top google result for a very specific case.

To summarize the steps:

  1. Base docker image has its own GitHub repo and auto-build in DockerHub on git push
  2. git push to project repo triggers CircleCI to build code
  3. CircleCI pushes new image with built code to another DockerHub repo

 

The Base Image - Gotcha #1

We wanted to make use of Docker's automated builds (AB) for our base image because having it ever ready and up-to-date is nice. Unfortunately, Docker's AB supports neither ARM kernels nor kernel module access (that means no binfmt_misc). Some kind internet people created a workaround that catches all shell calls and then runs them in an architecture emulator (QEMU). 

To implement this solution, first compile the altered QEMU source on your ARM device, and rename it qemu-arm-static. Then make a file called cross-build-start with the contents:

#!/usr/bin/qemu-arm-static /bin/sh
mv /bin/sh /bin/sh.real
cp /usr/bin/sh-shim /bin/sh

And cross-build-end with the contents:

#!/usr/bin/qemu-arm-static /bin/sh.real
mv /bin/sh.real /bin/sh

And sh-shim with the contents:

#!/usr/bin/qemu-arm-static /bin/sh.real
set -o errexit
cp /bin/sh.real /bin/sh
/bin/sh "$@"
cp /usr/bin/sh-shim /bin/sh

Add all these files to the repo with your Dockerfile. Finally, you can ignore most of the bits in my Dockerfile, except for these important lines:

ENV QEMU_EXECVE 1
COPY dockerutils/* /usr/bin/
RUN [ "cross-build-start" ]
... <your commands here> ...
RUN [ "cross-build-end" ]
ENTRYPOINT [ "/usr/bin/qemu-arm-static", "/usr/bin/env", "QEMU_EXECVE=1" ]
CMD [ "/bin/bash" ]

Note that the dockerutils directory contains cross-build-start/end, sh-shim, and qemu-arm-static.

Resources:

The Base Image - Gotcha #2

After step one, you might have a working automated DockerHub image. We, however, ran into another issue-- DockerHub was not able to establish a connection with the official ubuntu-ports repo server for several ARMHF packages. To resolve this, change the image's sources list to point toward http://ftp.tu-chemnitz.de/pub/linux/ubuntu-ports/, or an alternative mirror (though there aren't many). See sources.sh for a handy script to call from your Dockerfile.

Circle CI 

Vs. Travis CI

We quickly made our transition from Travis CI to Circle CI because Travis continued to stall on catkin builds (even short ones). Initially, we suspected our install step was too long for Travis, which is the reason why we moved that step to DockerHub's automated service. Additionally, Circle:

  • Offers 4 simultaneous builds for open source projects
  • Allows up to 32 simultaneous processes per build (lower across simultaneous builds)
  • Separates console output per command for easy debugging
  • Does not automatically pull git submodules

Tips, Tricks

To add support for git submodules, add section

checkout:
  post:
    - git submodule sync
    - git submodule update --init

To obscure your circle.yml environment variables, add them to an empty file on your local computer in the form `VAR=value`. Then, encrypt the file with a KEY of your choosing, using the command `openssl aes-256-cbc -e -in secret-env-plain -out secret-env-cipher -k $KEY`. Set KEY in your Circle CI project settings.