[SOLVED] Can't install/boot RouterOS 7 on ALTRAD8UD-1L2T with Q80-30

General Discussion
1

I’m doing a bunch of experiments with networking on my various ARM64 systems, and part of those experiments involves getting RouterOS to work, either on the bare metal or using their CHR image. (I’m an ISP and have a ton of MikroTik gear running the network.)

I recently built a box with a Q80-30 and ASRock Rack board, with 64GB of RAM, a SATA card, an Intel 520 NIC, a Mellanox 40G NIC, and am having a heck of a time getting it to just boot into the installer. I have Debian working on the box, with the Proxmox port the Chinese guy made (it’s in a PVE cluster with my RPi CM5), and RouterOS CHR ARM64 works fine in that environment.

When I boot into the RouterOS installer (on bare metal), the EFI stub reports it’s handing off an empty DTB, then it sits there at a blank screen for two minutes. Eventually a bunch of kernel tasks time out and it tries to restart itself:

[  244.481978][  T331] Kernel panic - not syncing: hung_task: blocked tasks
[  244.488675][  T331] SMP: stopping secondary CPUs
[  245.526632][  T331] SMP: failed to stop secondary CPUs 1,3
[  245.532105][  T331] Kernel Offset: disabled
[  245.536275][  T331] CPU features: 0x0000a,28002a38
[  245.541052][  T331] Memory Limit: none
[  245.544791][  T331] Rebooting in 5 seconds..
[  250.600750][  T331] SMP: stopping secondary CPUs
[  251.638698][  T331] SMP: failed to stop secondary CPUs 1,3

I’m using the 3.06 beta of ASRock’s BIOS and BMC. I’ve tried with all cards and the NVMe SSD removed; with the core count set to 2, 16, and 80 (default); with SR-IOV on and off; and either the other settings aren’t related to booting or are obscure enough that I don’t know what effect they would have on boot up.

MikroTik support says their lab Q64-22 works fine (I don’t know which motherboard), and suggests disabling SMP, if that’s an option (likely because the kernel is complaining about cores 1,3). Under CPU settings, all I can do is set the number of active cores, turn on/off NUMA splits (ANC mode), plus other slightly more nerdy bits that I have yet to research and tinker with.

Any suggestions?

2 Likes
2

Welcome to the forum.
Mikrotik are likely using a sgl socket board from Supermicro (ARS Mega DC range) or an ADLINK AADP aadp-docs – I-Pi SMARC.
I’ve not tried RouterOS under Proxmox - how does RouterOS boot if it’s the sole OS?
The SMP won’t be the issue. ASRR’s firmware version for Altra will auto detect 2P.
I would also try the stable firmware 2.06 - https://www.asrockrack.com/general/productdetail.asp?Model=ALTRAD8UD-1L2T#Download

@bexcran any other suggestions?

3

RouterOS boots and works fine in Proxmox/KVM.

When I try to boot it on the bare metal, the kernel seems to load just fine, but hangs on something (not sure what yet).

There’s discussion on the MikroTik forums and this Github repo (GitHub - tikoci/fat-chr: Builder for CHR images with UEFI support (testing)) that explain that at least the CHR version of the image for arm64 uses ext2 for the EFI partition instead of FAT, and that some UEFI implementations can’t handle that. I’ve tried some of those images. They don’t natively boot, but if I drop to the EFI shell and manually start the BOOTxxx.EFI file, it loads the stub and the kernel, and subsequently stalls the same.

So my guess at this point is that the kernel can’t find the filesystem.

4

Through a lot of reboots and debugging I fiigured it out.

Once it’s booted, the installer’s text doesn’t show up on the VGA console. It only showed up on the serial console (UEFI/BMC defaults). I had SSH’d into port 2200 to watch things and as I selected the RouterOS USB boot, from across the room I could see my SSH session showing the installation options.

The kernel panics are likely from the ASMedia-based SATA controller I had plugged in. I had tried with it and without the card, but wasn’t monitoring the serial console initially, so I had missed the install prompts.

Once it’s installed, RouterOS only sees 64 out of the 80 cores. It also doesn’t recognize the onboard 10G X550 NICs, just the i210 NIC on this board. It did see the X710’s on a separate card I had installed.

With the 64-core limitation and the fact that it doesn’t see the onboard NICs, I think it’s safe to say this machine will run Proxmox, and any routing it does will be via CHR VM’s.

2 Likes