Hi, I have been banging my head against the wall trying to get a working configuration for enabling Hyper-V on a 13700K intel host.

I've referenced various CPU flags & kernel versions (5.15-6.2) to try as well as Windows 10/11/Server2019/Server2022 as OS options with no joy so far. With any configuration I use the boot hangs after enabling Hyper-V on the guest.

What are the best practices when trying a setup like this? I used a 10700K as a proof of concept of this idea an was able to enable Hyper-V and utilize GPU-P to split a passed through NVIDIA GPU across the guest and nested guest within. Now on 13th gen I do not get the same result with the same VM details and BIOS options.

I only need to pass the whole GPU via VFIO and don't have issues with code 43 on the guest, so I am just trying to get nested virtualization to work on a Windows guest with hyper-v. How can I troubleshoot why the guest hangs on the boot logo in console?

Thanks! And please let me know if I can provide any more info!

As a follow up, I was able to get this to work on the 13700k intel host with the following arguments in the vm conf file:

args: -cpu SandyBridge,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time,+vmx

Sorry for the necroposting, but I looked so hard to find a solution to this problem affecting also 12th gen that I really want to thank @brewdamaster and confirm it's working.

Performance hit anyway is huge (near 45%).
Here a Geekbench 6 comparison between host and SandyBridge options:

Using a more recent CPU model and enabling all the 12th gen supported features like below

args: -cpu Cascadelake-Server-noTSX,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time,+vmx,+pdpe1gb,+md-clear,+mds-no,+taa-no,+tsx-ctrl,+spec-ctrl,+stibp,+ssbd,+pcid

mitigate the performance hit (near 10%):

Thanks @fakezeta ! I was able to get almost double single core score on 13700k with your flags. Not quite as improved on the multicore score (about 30% improved) but still a nice performance bump!

THANK YOU. fixed my issue just want to give my thanks!

I think this resolved our issue, we just got 5th generation Xeon R760 servers and to my surprise we couldn't boot with host CPU type, which is required to run WSL/nested virtualization. Besides the fact virtualized CPUs have awful performance in some areas.

After using the suggested -cpu host,hv_pass... suggestion we can boot and "wsl --install -d Ubuntu-20.04" completes post reboot whereas before it would error about BIOS needing virtualization support.

Thank you

I don't understand why Windows is so weird about drivers.

Well, this was a huge find. I ignorantly installed WSL on Win11 yesterday without doing research, rebooted, and found myself in recovery mode. Attempted to search but so many false paths and old information. I managed to find this this morning and the "Win11 Args" booted me right in!

Thanks to everyone who contributed to this thread.

To get Windows 11 Pro started, it was necessary to add the -waitpkg flag (same link as above) and use the following arguments:

Code:

args: -cpu host,hv_passthrough,level=30,-waitpkg

It was also essential that MSRs are ignored according to the nested docs. Without the two adjustments, the test machine did not boot or ended up with a BSOD.

Are there driver updates for it as well? It looks like my i7-13700K is showing up with a basic Microsoft driver from 4/21/2009 version "10.0.22621.3374", and I can only find drivers online for the igpu.

Are there driver updates for it as well? It looks like my i7-13700K is showing up with a basic Microsoft driver from 4/21/2009 version "10.0.22621.3374", and I can only find drivers online for the igpu.

This is normal. CPU's don't have drivers. My 13th gen shows up with the same 2009 driver in Windows.

Just wanted to give a huge thanks to @brewdamaster and the others here. A few months ago, I migrated my Proxmox cluster from some old Xeon servers to more power/noise/size friendly i9 and i7 boxes. Everything was smooth EXCEPT my Proxmox VMs running Hyper-V servers. On my old Xeon servers, I didn't do anything special, just set the VM CPU to host and Hyper-V worked just fine.

I was having a heck of a time trying dozens of different flags and CPU models. Nothing I did was able to solve the issue and I'd nearly given up on running Hyper-V servers on the new cluster.

The Proxmox wiki page on nested virtualization was useless, since none of the instructions in there were able to make a working Hyper-V server.

This thread was the first information found, that works. After dozens of searches, this thread finally got me sorted out. I now have a fully functional AzureStack HCI cluster running on Intel i9 processors.

Thank you!

Thanks!

Just a note for those with snapshots: if you edit the config file, it wasn't enough just to add `args` to the current snapshot; I had to add to all of the snapshots.

Anyone getting iGPU passthorough to work with WSL2 enabled? No matter which flags I set, Intel 13900T UHD 770 iGPU will not work correctly (error code 43) with virtual platform enabled.

Anyone getting iGPU passthorough to work with WSL2 enabled? No matter which flags I set, Intel 13900T UHD 770 iGPU will not work correctly (error code 43) with virtual platform enabled.

Are you really trying to passthrough the whole iGPU or are you trying to use SR-IOV? The latter will never work since enabling WSL2 will nest your Windows VM inside an additional Hyper-V layer. But if you really passthough the whole iGPU, it should/could work. But it might require using an extratced ROM file to work correctly. Did you already try that?

Good point. As you noticed I am trying to passthrough SR-IOV. So there is no way to make SR-IOV passthrough work? The Hyper-V layer would have to be configured to passthrough the iGPU too, right?
Have not tried the complete passthrough yet as other VMs need the iGPU as well (SR-IOV).

In the meantime I am using WSL1 and deactivated all virtual platform features but WSL in Windows. Now iGPU is back working with SR-IOV.

For those on older Westmere hardware, I finagled out the following custom CPU to use with the W11 VMs we have that need WSL2 and it seems to be working so far. 1. It tells Windows 11 that it is on a supported CPU by model name, 2. I can use virtualization for WSL2. Feedback welcome.

```ini
cpu-model: Icelake-Westmere-Nested
flags +pcid;+flush-l1d;+pdcm;+pdpe1gb;+rdtscp;+umip;+vmx;+aes;+hv-evmcs;+spec-ctrl;+ssbd;+sse4.1;+sse4.2;+popcnt;+hv-vapic;-avx;-avx2;-avx512f;-avx512dq;-avx512cd;-avx512bw;-avx512vl;-fma;-fma4;-bmi1;-bmi2;-fsgsbase;-smep;-smap;-erms;-invpcid;-rdrand;-rdseed;-sha-ni;-movbe;-x2apic;-f16c;-xsave;-md-clear;-intel-pt;-clflushopt;-clwb;-adx;-pku;-umip;-stibp;-la57;-waitpkg;-gfni;-vaes;-vpclmulqdq;-abm;-3dnowprefetch;-wbnoinvd;-hle;-rtm;-avx512vbmi;-avx512vbmi2;-avx512vnni;-avx512bitalg;-avx512-vpopcntdq;-xsaveopt;-xsavec;-xgetbv1;-vmx-apicv-register;-vmx-apicv-vid;-vmx-vmfunc;-vmx-posted-intr;-vmx-eptad;-waitpkg
hidden 0
hv-vendor-id GenuineIntel
reported-model Icelake-Server
phys-bits 40
```

I was searching for days in order to get my WIndows 10 Pro Desktop VM back online again (while keeping the host cpu) after enabling Core Isolation (aka Virtualization-based Security) without doing a snapshot in advance. And then I luckily found this link. It turned out that getting the nested Hyper-V and WSL running on my Intel i7-13700T Proxmox VE 8.1 host was just a matter of adding

in the VM config file.

Afterwards the VM instantly booted again but still felt a bit slow and caused its assigned vcpus to constantly run at high load. After carefully reading the QEMU documentation for Hyper-V Enlightenments for the hv_* flags, I enabled everything that promised some performance gain:
View attachment 64367

From this I would recommend to use somthing like the following custom arguments for the VM config for such a processor:

Code:

args: -cpu host,level=30,hv_relaxed,hv_reset,hv_runtime,hv_time,hv_spinlocks=0x1fff,hv_vapic,hv_vpindex,hv_ipi,hv_synic,hv_stimer,hv_apicv,hv_xmm_input,hv_stimer_direct,hv_frequencies,hv_reenlightenment,hv_evmcs,hv_emsr_bitmap,hv_tlbflush,hv_tlbflush_ext,hv_tlbflush_direct

Please note that the hv_evmcs flag is Intel-only and some of the others might only be supported on AlderLake/RaptorLake processors and newer.

But using passthrough also seems to work fine and is much simpler and less error prone:

Code:

args: -cpu host,hv_passthrough,level=30

Guess this would also allow to use Credential Guard with WIndows 10 Enterprise or Windows 11 (sorry for the German screenshots):
View attachment 64368

Windows Subsystem for Linux (WSL) also works without any issues:
View attachment 64369
View attachment 64370
View attachment 64371

To get Windows 11 Pro started, it was necessary to add the -waitpkg flag (same link as above) and use the following arguments:

Code:

args: -cpu host,hv_passthrough,level=30,-waitpkg

It was also essential that MSRs are ignored according to the nested docs. Without the two adjustments, the test machine did not boot or ended up with a BSOD.

For some reason, Windows 11 works much faster than Windows 10 and gives close to bare metal results (also in WSL) with Hyper-V and VBS enabled:
View attachment 64399
View attachment 64400

For the benchmarks I was only using 6 P-cores (x2) while the other 2 P-cores were dedicated to the PVE host.
All 8 E-cores have been used exclusively for 6 background VMs (nextcloud, omv, dc, ...) that were more or less idling during the tests.

Thanks to @brewdamaster and @fakezeta for this thread for search inspirations!

Hope this is helpful.

This has been running for some time now, however for me this solution no longer seems to work. It seems somewhere it broke. I recently updated Proxmox to 9.2.3 and now my windows 11 vm that used to boot with host and hv_passtrough enabled no longer boots.

I now get a blue screen every time i try to boot with this configuration. Nothing has changed in hardware or config. It's been running fine for years now, but decided to do a proxmox update and now the vm no longer boots. Enabling an "emulated" cpu causes it to boot again, but i really want host with hv passthough. Any advice on what changed would be greatly appreciated.