Go Cross Compilation from Ubuntu amd64 to arm7l : exec user process caused “exec format error”

Question

I am having an headache with cross compilation from amd64 to arm7l

I could finally do it with Gitlab CI, so now, I compile my binary in a docker image, here is the dockerfile:

FROM golang

WORKDIR /go/src/gitlab.com/company/edge_to_bc

COPY . .
RUN dpkg --add-architecture armhf && apt update && apt-get install -y gcc-arm-linux-gnueabihf libltdl-dev:armhf

I build it as Then I will build the the new container "cross-compil" ready with the name ubuntu:cross-compil

Now, I can compile my binary with:

docker run -it -v ${EDGE_TO_BC_PATH}/release:/go/src/gitlab.com/company/edge_to_bc/release ubuntu:cross-compil  bash -c 'CC=arm-linux-gnueabihf-gcc CXX=arm-linux-gnueabihf-g++ CGO_ENABLED=1 GOOS=linux GOARCH=arm GOARM=7 go build -v -o ./release/edge_to_bc '

I can see my executable generated in ./release/edge_to_bc

Then I build my docker image:

docker build -t registry.gitlab.com/company/edge_to_bc:armhf .

And I push it.

In the Dockerfile, I just copy the executable from host:

FROM alpine:3.7

RUN apk --no-cache add ca-certificates libtool

WORKDIR /sunclient/

COPY ./release/edge_to_bc ./
EXPOSE 5555

CMD [ "./edge_to_bc" ]

But when I run it in my arm board with:

docker run --rm registry.gitlab.com/company/edge_to_bc:armhf

I get:

standard_init_linux.go:207: exec user process caused "no such file or directory"

When debugging, if I want to get list of files with

docker run --rm registry.gitlab.com/company/edge_to_bc:armhf

I get:

standard_init_linux.go:207: exec user process caused "exec format error"

Which indicate binary still not have correct format...

What did I miss ? I spent a lot of time on this topic, and don't have much more ideas.

When I check the architecture of binary, this is what I get:

 edge_to_bc git:(master) ✗ readelf -h ./release/edge_to_bc
ELF Header:
  Magic:   7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00 
  Class:                             ELF32
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              EXEC (Executable file)
  Machine:                           ARM
  Version:                           0x1
  Entry point address:               0x19209
  Start of program headers:          52 (bytes into file)
  Start of section headers:          23993360 (bytes into file)
  Flags:                             0x5000400, Version5 EABI, hard-float ABI
  Size of this header:               52 (bytes)
  Size of program headers:           32 (bytes)
  Number of program headers:         10
  Size of section headers:           40 (bytes)
  Number of section headers:         49
  Section header string table index: 48

On the target OS, this is what I get:

[root@gw-sol1 ~]# uname -a
Linux gw-sol-1 4.4.113-UNRELEASED #1 SMP PREEMPT Thu Mar 7 16:46:40 CET 2019 armv7l armv7l armv7l GNU/Linux

EDIT:

When I build the app directly on ARM device, it will work:

go build -o ./release/edge_to_bc -v -ldflags '-w -s -extldflags "-static"' ./...

the ELF:

ELF Header:
  Magic:   7f 45 4c 46 01 01 01 03 00 00 00 00 00 00 00 00 
  Class:                             ELF32
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - GNU
  ABI Version:                       0
  Type:                              EXEC (Executable file)
  Machine:                           ARM
  Version:                           0x1
  Entry point address:               0x125f1
  Start of program headers:          52 (bytes into file)
  Start of section headers:          16594072 (bytes into file)
  Flags:                             0x5000400, Version5 EABI, hard-float ABI
  Size of this header:               52 (bytes)
  Size of program headers:           32 (bytes)
  Number of program headers:         7
  Size of section headers:           40 (bytes)
  Number of section headers:         38
  Section header string table index: 37

It seems quite similar to the other one, at least in architecture.

Then build the docker image:

docker build . -t image/peer-test:armhf

Answer 1

When you run the build on the amd64 system of the arm7 image with this command:

docker build -t registry.gitlab.com/company/edge_to_bc:armhf .

It will use base images and run commands in that image for amd64. So even if your single edge_to_bc binary may be cross compiled, the rest of the image is not. Next, the binary compile command itself looks like it is linking to libraries, which quite likely are not in your final image. You can run ldd edge_to_bc to see these links, and if they are missing, you'll get the file not found error.

---

In my own cross compile test, I'm using BuildKit, Buildx, and some experimental features on 19.03.0-rc2, so there will be parts of this that are not backwards compatible, but hopefully you find helpful. I'm using a multi-stage build to avoid the compile outside of docker and then a second build. I'm also specifying the platform for the build host and using the target arch and OS variables to configure go to cross compile. In this scenario, I went with a completely statically linked binary so there were no libraries to include, and with only copy commands in my final release image, I avoid issues running the build on a different platform.

# syntax=docker/dockerfile:experimental
# ^ above line must be at the beginning of the Dockerfile for BuildKit

# --platform pulls an image for the build host rather than target OS/Arch
FROM --platform=$BUILDPLATFORM golang:1.12-alpine as dev
RUN apk add --no-cache git ca-certificates
RUN adduser -D appuser
WORKDIR /src
COPY . /src/
CMD CGO_ENABLED=0 go build -o app . && ./app

# my dev stage is separate from build to allow mounting source and rebuilding
# on developer machines    
FROM --platform=$BUILDPLATFORM dev as build
ARG TARGETPLATFORM
ARG TARGETOS
ARG TARGETARCH
# --mount is an experimental BuildKit feature
RUN --mount=type=cache,id=gomod,target=/go/pkg/mod/cache \
    --mount=type=cache,id=goroot,target=/root/.cache/go-build \
    CGO_ENABLED=0 GOOS=${TARGETOS} GOARCH=${TARGETARCH} \
    go build -ldflags '-w -extldflags -static' -o app .
USER appuser
CMD [ "./app" ]

# this stage will have the target OS/Arch and cannot have any RUN commands
FROM scratch as release
COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=build /etc/passwd /etc/group /etc/
COPY --from=build /src/app /app
USER appuser
CMD [ "/app" ]

FROM scratch as artifact
COPY --from=build /src/app /app

FROM release

To build this, I can run one-off build commands with BuildKit:

DOCKER_BUILDKIT=1 docker build --platform=linux/amd64 -f Dockerfile -t golang-app:amd64 .
DOCKER_BUILDKIT=1 docker build --platform=linux/arm64 -f Dockerfile -t golang-app:arm64 .

But even better is the Buildx version that creates a multi-arch image that will run on multiple platforms. This does require that you push to a registry server:

docker buildx build -f Dockerfile --platform linux/amd64,linux/arm64 \
  -t ${docker_hub_id}/golang-app:multi-arch --output type=registry .

For your scenario, you would swap out the references from arm64 to your own architectures. The --platform option was listed as experimental in many commands I ran, so you may need to configure the following in the /etc/docker/daemon.json file to enable:

{ "experimental": true }

I believe this required a full restart of the docker daemon ( systemctl restart docker ), not just a reload, to take effect.

To extract the artifact from the build (the compiled binary for the specific architecture), I use:

docker build -f Dockerfile --target artifact -o type=local,dest=. .

That will output the contents of the artifact stage (a single binary) to the local directory.

---

The above is option 3 in Docker's list of ways to build multi-arch images.

Option 1 is to configure qemu with binfmt_misc to build and run images for different platforms. I have not yet been able to get this to work on Linux with Buildx, but Docker has done this for Mac and you may be able to find more details on what they did in the LinuxKit project. Using qemu for your situation may be ideal if you need to run commands and install other tools as part of your build, not just cross compile a single statically linked binary.

Option 2 is to run the build on the target platform, which as you've seen works well. With Buildx , you can add multiple build nodes, up to one per platform, allowing you to run the build from a single location (CI server).