I recently received an interesting inquiry from a developer looking to set up a nested hypervisor environment on Ampere ARM64 (Ubuntu/KVM). They are particularly interested in:
Best practices for KVM nested configuration on Ampere Altra/AmpereOne.
Performance benchmarks or whitepapers related to nested virtualization latency (L0 → L1 → L2).
While we know Ampere’s high core count makes it a beast for virtualization, I’d love to hear from this community:
Who is currently running nested KVM?
What are your L1/L2 performance stats like?
Any specific quirks in Ubuntu/Linux kernel to look out for?
As it happens, I was recently playing with nested virt on AmpereOne! I ran into a few “gotchas” that are important to know - especially on AmpereOne.
First, it is very sensitive to toolchains - you need a recent kernel (6.15 or later, more recent is better) to get FEAT_NV2 support in the kernel, and you also need a recent version of qemu-kvm (10.x) to allow VMs to be hypervisors in Arm64. That will limit which OSes you can use to very recent ones. Right now there is a known issue with nested VMs on AmpereOne for kernels 6.19 and 7.0 - still unresolved, but people are working on it. So you’ll have to pin your kernel to 6.18 or earlier until that is cleared up.
Once you have that, you need to pass in a kernel command line option to GRUB (kvm_arm.nested=on), and you should be good to go. It’s a little tricky to pass in the right options into qemu if you are using libvirt to enable virtualization in the L1 hypervisor, but once you’ve done that, starting a VM in that hypervisor is trivial.
I did some testing, and found that running sysbench on bare metal (pinned to 8 cores) vs running it inside an 8 core L1 VM was almost identical in performance - passthrough is really efficient! There was a small hit on tail latencies (which you would expect). And when running the same benchmark in an 8 core L2 VM, I see a 10% drop-off in performance, and a slightly larger hit in tail latencies. I was very impressed!
All in all, I would say it’s still a very recent feature in the kernel & userspace world, and is not really ready for prime time for serious usage yet.
