I got Linux running in big endian mode on Altra

More difficult than I anticipated due to ACPI… Very few people need to run in big endian mode (we do it just to test for big endian cleanliness of our drivers an software), but the combination of nvme slots, great performance, and lots of cores is fantastic.

Thought this might be mildly amusing/interesting for others.
Booted

6 Likes

Hi. This might be useful to me. Do you have a wr tie up somewhere of what you did?

Thanks,
Dan

Hi Dan,

It has been a while since I did that work; this reply is from memory and may have “bit rot”… I never published this work because it would never be accepted into the opensource projects it modified… it was not a “clean” solution and I had no way to test it on other machines… my basic strategy was to attack each issue I encountered until the kernel came up.

Preparatory work: I used a cross build environment in Gentoo to create a big endian “stage 3” Gentoo tarball, and then I installed the Gentoo Linux stage 3 base image to a disk following the standard Gentoo installation guide. This stage 3 big endian build is publicly available.on Google drive…

The next step was to cross compile the kernel for big endian which was trivial…

The first real problem was GRUB. The Ampere wants to boot with EFI… but EFI is little endian. Normally GRUB wants to call into the EFI stub within the kernel to set things up before calling the kernel entry point… but GRUB is all little endian with no provisions for big endian that I could find. My solution was to rebuild the kernel in little endian, extract the EFI boot stub from the kernel and directly link the EFI boot stub into grub. Now I had a little endian grub with all the EFI setup code that could load the kernel and pass all the EFI tables to my big endian kernel. The entry to the kernel is always little endian, but VERY early in initialization the switch is made to big endian,

The next problem was the EFI tables set up by the AMI ROM in the Ampere. All the data is little endian. This was straight forward but ugly - it is all documented. So I modified the kernel EFI initialization code to walk all the EFI tables and swap the data to big endian. This was easier than finding all the places in the kernel that access EFI data and wrapping the accesses in leXX_to_cpu() macros, which is arguably a better solution.

The biggest problem (for me anyway) was that some of the machine description data is actually interpreted at run time, so I had to modify the interpreter. This is not for the faint of heart. I got just enough of this done to support the data for the Mt. Jade system… no guarantees it would work on other systems and no way for me to test it.

So the basic flow is a little endian GRUB loads the big endian kernel into memory, then calls its own private little endian EFI stub passing in a pointer to the EFI system table. The EFI stub performs all the basic EFI setup and returns. The kernel entry point is called from little endian and the kernel switches to big endian mode and starts initializing… during initialization the kernel walks all the EFI tables and byte swaps them, then it sets up hardware resources by processing the (now big endian) system tables and uses a modified interpreter to create the necessary hardware description data structures. From that point forward it is all unmodified: devices are enumerated and claimed by drivers.

Obvious restrictions: EFI runtime services are disabled. There is no calling back into the ROM environment once the switch is made to big endian.

Less obvious restrictions: SInce so few people run big endian these days, some device drivers (looking at you Qlogic) are not big endian clean.

If you are running a Mt. Jade, I can easily supply binaries for everything. If you are not in a rush I could probably create patch files.

As it stands today I have a GRUB installation that supports all the standard GRUB features, allowing me to create boot menus with multiple choices for booting into either little endian or big endian mode.

-Rory