To install Windows 11 on ESXi 8, the minimum requrements should meet.

Summary is:

1. 1+ GPU
2. 4+ GB Memory
3. 64+ GB Disk
4. UEFI, Secure Boot capable
5. TPM
6. Graphics card (Optional, can install on ESXi 8 without this)
7. Display (Optional, can install on ESXi 8 without this)

The main reason causes this issue is TPM

vSphere support adding vTPM with a Native Key Provider, but I don’t know how to get one and where to config on ESXi 8.

The solution is skipped TPM checking while installing Windows 11 on ESXi 8.

1. Setup windows 11 ISO for Virtual Machine.
2. Boot the Virtual Machine from CD/DVD drive.
3. While on following screen, press Shift + F10 to Windows Terminal.
   
4. Type regedit enter the Registry Editor.
5. Navigate to HKEY_LOCAL_MACHINE\SYSTEM\Setup and add a new key name LabConfig.
6. In LabConfig create a value name ByPassTPMCheck with type DWORD (32-bit) and data 1.
   
7. Then close Registry Editor and cmd dialogs, came back to installation steps.
8. Now you can install.

References:

1. How to fix: This PC can’t run windows 11 vmware esxi
2. This PC can’t run Windows 11 on ESXi 7