marstr.online

Virtualizing your operating systems with full GPU access

Linux

I am using a fully KVM'd setup now which consists of a number of virtual machines, all of which have access to my Nvidia GPU. I primarily use a Endeavour OS VM as my main daily driver, but for some entertainment - namely games and VR - I do require a Windows 10 install.

Setting up a KVM environment, including a Windows 10 machine, gives me full control over what an operating system is allowed to access, and what is hidden from it. If I don't grant access through Virt Manager, the VM cannot access that particular component.

You will only need a few steps so that you can pass the GPU into a VM, I will just describe the needed steps without going into too much detail.

Set up IOMMU for either Intel or AMD respectively. Open your /etc/default/grub and add the follwing into the GRUB_CMDLINE_LINUX_DEFAULT section:

intel_iommu=on
amd_iommu=on

Only either one of those please, depending on the processor you have.

Next

update-grub

and reboot the machine.

Off to blacklisting the graphic cards drivers so that they won't be accessed and much less loaded on boot.

Open /etc/modprobe.d/blacklist.conf and add the following entries:

(NVidia users)
blacklist nvidia
blacklist nouveau

(AMD users)
blacklist radeon
blacklist amdgpu

Initramfs update is now needed:

update-initramfs -u -k all

Next, we need to find the actual hardware IDs of the GPU you want to pass into your VMs. This will then prevent the Linux kernel to access these hardware components, thus reserving this for VM access.

Perform a

lspci -vnn

to see the hardware IDs. Write down all IDs of your Nvidia or AMD GPU - you will need them in the next step.

Open /etc/modprobe.d/vfio.conf and put in

options vfio-pci ids=IDS_OF_PREVIOUS_STEP (separated with a comma)

It is important to note that Nvidia GPUs may show two or even four separate devices in this list - add all IDs as all these single devices make up the GPU, and thus any OS using this GPU needs access to all components.

Open /etc/modules and load the appropriate modules for VM passthrough:

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

Install the appropriate drivers, libraries, and software to get going:

apt install qemu-system libvirt-clients libvirt-daemon-system virt-manager

Add yourself to the libvirt group so that you can create, modify and control your VMs:

usermod -a -G libvirt username

Reboot your machine and you should now be ready to go!

If you want to be able to remotely control your machines, you will need to install ssh and start the ssh daemon for remote access.

From here on in you should be able to create VMs and pass the GPU into it for normal operation. The setup is different for every installation so I won't go further than this - the setup and configuration you decide for yourself.