尽管有 CONFIG_COMPAT，但在 aarch64 上运行 32 位 ARM 二进制文件不起作用

Question

I've got a 64-bit ARM machine that I'd like to run 32-bit ARM binaries on. As a test case I've built a small hello world for 32-bit ARM using the arm-linux-gnueabihf-gcc toolchain. file shows it as:

root@ubuntu:/home/ubuntu# file hello
hello: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.3, for GNU/Linux 3.2.0, BuildID[sha1]=61ffe5e22117a6d4c2ae37a1f4c76617d3e5facc, not stripped

But trying to run it produces:

root@ubuntu:/home/ubuntu# ./hello
bash: ./hello: cannot execute binary file: Exec format error

Based on a prior question , I checked whether the kernel was built with the CONFIG_COMPAT option, and it was:

root@ubuntu:/home/ubuntu# grep CONFIG_COMPAT= /boot/config-$(uname -r)
CONFIG_COMPAT=y

I also verified that the armhf architecture has been added and that the armhf version of the loader is present. Note that the loader itself doesn't run either:

root@ubuntu:/home/ubuntu# dpkg --print-foreign-architectures
armhf
root@ubuntu:/home/ubuntu# dpkg -l libc6:armhf
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version       Architecture Description
+++-==============-=============-============-=================================
ii  libc6:armhf    2.30-0ubuntu2 armhf        GNU C Library: Shared libraries
root@ubuntu:/home/ubuntu# file /lib/arm-linux-gnueabihf/ld-2.30.so 
/lib/arm-linux-gnueabihf/ld-2.30.so: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, BuildID[sha1]=dff2b536287d61ddca68f3e001e14b7c235bbf68, stripped
root@ubuntu:/home/ubuntu# /lib/arm-linux-gnueabihf/ld-2.30.so
bash: /lib/arm-linux-gnueabihf/ld-2.30.so: cannot execute binary file: Exec format error

Other relevant system info:

root@ubuntu:/home/ubuntu# cat /proc/cpuinfo 
processor       : 0
BogoMIPS        : 400.00
Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics cpuid asimdrdm
CPU implementer : 0x43
CPU architecture: 8
CPU variant     : 0x1
CPU part        : 0x0af
CPU revision    : 2

root@ubuntu:/home/ubuntu# uname -a
Linux ubuntu 5.3.0-24-generic #26-Ubuntu SMP Thu Nov 14 01:14:25 UTC 2019 aarch64 aarch64 aarch64 GNU/Linux

I'm running out of things to try at this point. Any idea how to get the kernel to recognize these binaries and run them?

Answer 1

Ok, it looks like the underlying problem here is not in software, unfortunately. The CPU we're using, the Cavium ThunderX2, is one of the few 64-bit ARM chips that does not have aarch32 support. Quoting from WikiChip :

• Only the 64-bit AArch64 execution state is support. No 32-bit AArch32 support.

So, this explains why it's not able to run 32-bit ARM binaries. I'm still fairly sure that other 64-bit ARM chips, like the Cortex-A57, are able to do this.

Update: older 32-bit ARM binaries do indeed work on aarch64 with a CPU that supports it, as shown below on an AWS ARM a1.metal instance:

ubuntu@ip-172-31-12-156:~$ cat /proc/cpuinfo | tail

processor   : 15
BogoMIPS    : 166.66
Features    : fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid
CPU implementer : 0x41
CPU architecture: 8
CPU variant : 0x0
CPU part    : 0xd08
CPU revision    : 3

ubuntu@ip-172-31-12-156:~$ uname -a
Linux ip-172-31-12-156 4.15.0-1054-aws #56-Ubuntu SMP Thu Nov 7 16:18:50 UTC 2019 aarch64 aarch64 aarch64 GNU/Linux
ubuntu@ip-172-31-12-156:~$ dpkg --print-foreign-architectures
armhf
ubuntu@ip-172-31-12-156:~$ file hello_hf
hello_hf: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-, for GNU/Linux 3.2.0, BuildID[sha1]=c95f0c46dfab925db53506751d7677156e334e5c, not stripped
ubuntu@ip-172-31-12-156:~$ ./hello_hf
hello, world!

Answer 2

This question has become more relevant as 32bit ARM chips are being phased out. Me, I like to see old 32 bit binaries supported on new 64 bit hardware. Not so important, but convenient. I took out my old Raspberry Pi Zero (32 bit) and Raspberry Pi 3B+ (64 bit) and for the first time installed a 64 bit OS on the 3B+ despite warnings of bugs and such and did some compiling. No special make parameters. Just plain ordinary 32 bit compiling. Runs fine on 32 bit and 64 bit. Controllers and VMs will be running 32 bit architecture for many more years. 4GB memory limit is a pain yes. But for small apps its not a problem.