前置条件
显卡直通依赖于一些虚拟化技术, 可能在你的机器上不可用, 必须先检查以下条件:
-
首先CPU必须支持硬件虚拟化(for KVM)和IOMMU(for 直通)
-
-
兼容的 Intel CPUs 列表 (Intel VT-x and Intel VT-d). 可以理解为 Intel VT-x 是CPU的虚拟化，VT-d 是 I/O 设备的虚拟化，两个东西不一样。
-
Bulldozer 及更高版本（包括 Zen）的所有 AMD CPU 都应该兼容。
-
-
主板必须支持 IOMMU
-
-
芯片组和 BIOS 必须都支持 IOMMU。很难直接看出来是否支持，但是有一个比较完整的列表供查询：Xen Wiki 和 Wikipedia:List of IOMMU-supporting hardware
-
-
GPU ROM 必须支持 UEFI。2012 年以后的 GPU 几乎都支持。
比较方便的检查方法就是在BIOS设置中找到并打开 VT-d (Intel) 或者 IOMMU (AMD)。如果没有这些选项，那就是不支持。
在KVM主机上启动IOMMU
# 首先编辑文件
$ vim /etc/default/grub
# 第6行: 添加 intel_iommu=on (如果是AMD CPU, 添加 [amd_iommu=on])
# 还应该添加参数 iommu=pt, 阻止 Linux 接触不能透传的设备
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet intel_iommu=on iommu=pt"
GRUB_DISABLE_RECOVERY="true"
$ grub2-mkconfig -o /boot/grub2/grub.cfg
$ reboot # 重启重启之后验证IOMMU是否启用：
$ dmesg | grep -E "DMAR|IOMMU"
[ 0.000000] ACPI: DMAR 00000000bf79e0d0 00118 (v01 AMI OEMDMAR 00000001 MSFT 00000097)
[ 0.000000] DMAR: IOMMU enabled
[ 0.069846] DMAR: Host address width 40
[ 0.069848] DMAR: DRHD base: 0x000000fbffe000 flags: 0x1
[ 0.069856] DMAR: dmar0: reg_base_addr fbffe000 ver 1:0 cap c90780106f0462 ecap f020fe
[ 0.069858] DMAR: RMRR base: 0x000000000ec000 end: 0x000000000effff
[ 0.069859] DMAR: RMRR base: 0x000000bf7ec000 end: 0x000000bf7fffff
[ 0.069861] DMAR: ATSR flags: 0x0
[ 0.069863] DMAR-IR: IOAPIC id 6 under DRHD base 0xfbffe000 IOMMU 0
[ 0.070128] DMAR-IR: Enabled IRQ remapping in xapic mode
[ 0.960840] DMAR: dmar0: Using Queued invalidation
[ 0.960860] DMAR: Setting RMRR:
[ 0.960890] DMAR: Setting identity map for device 0000:00:1a.0 [0xbf7ec000 - 0xbf7fffff]
[ 0.960925] DMAR: Setting identity map for device 0000:00:1a.1 [0xbf7ec000 - 0xbf7fffff]
[ 0.960958] DMAR: Setting identity map for device 0000:00:1a.2 [0xbf7ec000 - 0xbf7fffff]
[ 0.960994] DMAR: Setting identity map for device 0000:00:1a.7 [0xbf7ec000 - 0xbf7fffff]
[ 0.961031] DMAR: Setting identity map for device 0000:00:1d.0 [0xbf7ec000 - 0xbf7fffff]
[ 0.961064] DMAR: Setting identity map for device 0000:00:1d.1 [0xbf7ec000 - 0xbf7fffff]
[ 0.961100] DMAR: Setting identity map for device 0000:00:1d.2 [0xbf7ec000 - 0xbf7fffff]
[ 0.961132] DMAR: Setting identity map for device 0000:00:1d.7 [0xbf7ec000 - 0xbf7fffff]
[ 0.961153] DMAR: Setting identity map for device 0000:00:1a.0 [0xec000 - 0xeffff]
[ 0.961167] DMAR: Setting identity map for device 0000:00:1a.1 [0xec000 - 0xeffff]
[ 0.961182] DMAR: Setting identity map for device 0000:00:1a.2 [0xec000 - 0xeffff]
[ 0.961196] DMAR: Setting identity map for device 0000:00:1a.7 [0xec000 - 0xeffff]
[ 0.961210] DMAR: Setting identity map for device 0000:00:1d.0 [0xec000 - 0xeffff]
[ 0.961224] DMAR: Setting identity map for device 0000:00:1d.1 [0xec000 - 0xeffff]
[ 0.961238] DMAR: Setting identity map for device 0000:00:1d.2 [0xec000 - 0xeffff]
[ 0.961255] DMAR: Setting identity map for device 0000:00:1d.7 [0xec000 - 0xeffff]
[ 0.961270] DMAR: Prepare 0-16MiB unity mapping for LPC
[ 0.961287] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[ 0.961424] DMAR: Intel(R) Virtualization Technology for Directed I/O启动 vfio-pci 内核模块
保留 RHEL/CentOS 官方提供的通用内核即可
$ uname -a
# 显示显卡的PCI认证数字和[供应商ID:设备ID]
$ lspci -nn | grep -i nvidia
3b:00.0 3D controller [0302]: NVIDIA Corporation GP104GL [Tesla P4] [10de:1bb3] (rev a1)
86:00.0 3D controller [0302]: NVIDIA Corporation GP104GL [Tesla P4] [10de:1bb3] (rev a1)
# 编辑vfio配置文件
$ vim /etc/modprobe.d/vfio.conf
# 创建一个新行，指定ids=供应商ID:设备ID
options vfio-pci ids=10de:1bb3,10de:1bb3
# 创建一个新文件，写入 vfio-pci
$ echo 'vfio-pci' > /etc/modules-load.d/vfio-pci.conf
$ reboot # 重启重启之后检查 vfio 是否加载
扫描二维码关注公众号，回复：
14320722 查看本文章
$ dmesg | grep -i vfio
[ 6.441900] VFIO - User Level meta-driver version: 0.3
[ 6.497125] vfio-pci: probe of 0000:3b:00.0 failed with error -22
[ 6.497163] vfio-pci: probe of 0000:86:00.0 failed with error -22
[ 6.497181] vfio_pci: add [10de:1bb3[ffff:ffff]] class 0x000000/00000000
[ 6.497233] vfio-pci: probe of 0000:3b:00.0 failed with error -22
[ 6.497249] vfio-pci: probe of 0000:86:00.0 failed with error -22
[ 6.497258] vfio_pci: add [10de:1bb3[ffff:ffff]] class 0x000000/00000000更新QEMU
# 查看QEMU版本
$ /usr/libexec/qemu-kvm -version
QEMU emulator version 1.5.3 (qemu-kvm-1.5.3-175.el7_9.3), Copyright (c) 2003-2008 Fabrice Bellard
# 升级新版
$ yum -y install centos-release-qemu-ev
Running transaction
Installing : centos-release-virt-common-1-1.el7.centos.noarch 1/2
Installing : centos-release-qemu-ev-1.0-4.el7.centos.noarch 2/2
Verifying : centos-release-qemu-ev-1.0-4.el7.centos.noarch 1/2
Verifying : centos-release-virt-common-1-1.el7.centos.noarch 2/2
Installed:
centos-release-qemu-ev.noarch 0:1.0-4.el7.centos
Dependency Installed:
centos-release-virt-common.noarch 0:1-1.el7.centos
# 关闭 CentOS-QEMU-EV
$ sed -i -e "s/enabled=1/enabled=0/g" /etc/yum.repos.d/CentOS-QEMU-EV.repo
# 将 qemu-kvm 替换成 qemu-kvm-ev
$ yum --enablerepo=centos-qemu-ev -y install qemu-kvm-ev
$ systemctl restart libvirtd
# 查看 KVM 版本
$ /usr/libexec/qemu-kvm -version
QEMU emulator version 2.12.0 (qemu-kvm-ev-2.12.0-44.1.el7_8.1)
Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers
# 验证是否包含了 q35
$ /usr/libexec/qemu-kvm -machine help | grep q35
q35 RHEL-7.6.0 PC (Q35 + ICH9, 2009) (alias of pc-q35-rhel7.6.0)
pc-q35-rhel7.6.0 RHEL-7.6.0 PC (Q35 + ICH9, 2009)
pc-q35-rhel7.5.0 RHEL-7.5.0 PC (Q35 + ICH9, 2009)
pc-q35-rhel7.4.0 RHEL-7.4.0 PC (Q35 + ICH9, 2009)
pc-q35-rhel7.3.0 RHEL-7.3.0 PC (Q35 + ICH9, 2009)创建新虚拟机
$ virt-install \
--name centos7 \
--ram 8192 \
--disk path=ubuntu16.img,size=30 \
--vcpus 4 \
--os-type linux \
--os-variant ubuntu16.04 \
--network bridge=br0 \
--graphics none \
--console pty,target_type=serial \
--location ubuntu-16.04.7-server-amd64.iso \
--extra-args 'console=ttyS0,115200n8 serial' \
--host-device 3b:00.0 \
--features kvm_hidden=on \
--machine q35
参数解释：
name 虚拟机名称
ram 内存大小
disk 指定存储设备及类型
vcpus 虚拟 CPU 个数
os-type 操作系统类型
os-variant 具体操作系统变体
network 连接宿主机的网桥或网络
graphics 虚拟机显示器访问接口，默认是 vnc
console ??
location 安装源, 支持 FTP、HTTP 及 NFS 等
extra-args 当执行从 –location 选项指定位置的客户机安装时，附加内核命令行参数到安装程序
host-device 附加物理设备到客户机
features kvm_hidden=on 从宿主机隐藏虚拟机状态
machine QEMU模拟的CPU架构