This I mean when I said you made my day 🙂
A positive and helpful approach from other users to find a solution.
Uefi problem:
I will put in guide's OP, I didn't think was an important step because in modern system Uefi are always advised .
In my case system boots fine also in CSM and 4g (on/off) but it could be different for others
Usb slowness for high capacity drive
I do not have this behaviour so we have to investigate also with other user
Thunderbolt part:
I have an old Asus Thunderbolt Ex 3 and I didn't find any clue it could work without a thunderbolt header. So here I can't say any interesting things.
vfio-pci:
I did a fresh install from scratch on a clean ssd drive I have noticed a weird thing.
All I need to have system booting fine in Proxmox / OSX is to complete module file in /etc/modprobe.d path to have modules loaded from kernel
No blacklist.conf, no vfio.conf, no other things found on traditional way to set Proxmox ( I have asked in private to some users to test this, a x570 user confirmed these but I can't be sure for different configurations)
Internal audio stuff and USB controller (related?) problems:
Above screenshot is the best I can achieve
It seems perfect but it isn't so..realtek USB2.0 Audio (internal audio on motherboard) is bad, scratchy and it looses synch during play audio or video viewing
DP audio from Nvidia is perfect.
I am investigating with @tomnic (similar problem for him on a X570 rig) and with @Gengik84with his skills to produce solutions for weird situations 🙂 )
In my try I have noticed the same things you write above:
An impossibility to pass two USB controllers in may case for USB:
04:00.3
23:00.3
And in my rig these "unpassed" stuff produces 4 USB 3.x rear ports non working
I have no tested internal USB 2.0 ports for now
ASmedia USB 3.x ports work well (connected to my Thermaltake X9 case)
I have passed fine 23.00.4 (matisse audio controller) but in my opinion it conflicts with a device passed in automatic way from VM basic parameters (I think, not sure, investigating on this)
I have discovered an important thing in my rig yesterday
with all things I can pass on, with audio output connected on green jack of rear motherboard backplate, booting from scratch produce this:
in this situation only DP audio is working perfectly
if I disconnect green jack from back, reboot linux and VM I see:
Also in this situation I have a perfect DP audio from Nvidia and a bad and scratchy audio from internal matisse audio controller.
in following box you can see my passed parts:
00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 61)
Subsystem: Micro-Star International Co., Ltd. [MSI] FCH SMBus Controller [1462:7c60]
Kernel driver in use: vfio-pci
Kernel modules: i2c_piix4, sp5100_tco
01:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961 [144d:a804]
Subsystem: Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961 [144d:a801]
Kernel driver in use: vfio-pci
02:00.0 Non-Volatile memory controller [0108]: Phison Electronics Corporation E12 NVMe Controller [1987:5012] (rev 01)
Subsystem: Phison Electronics Corporation E12 NVMe Controller [1987:5012]
Kernel driver in use: vfio-pci
21:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP102 [TITAN Xp] [10de:1b02] (rev a1)
Subsystem: NVIDIA Corporation GP102 [TITAN Xp] [10de:123e]
Kernel driver in use: vfio-pci
Kernel modules: nvidiafb, nouveau
21:00.1 Audio device [0403]: NVIDIA Corporation GP102 HDMI Audio Controller [10de:10ef] (rev a1)
Subsystem: NVIDIA Corporation GP102 HDMI Audio Controller [10de:123e]
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel
23:00.4 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller [1022:1487]
Subsystem: Micro-Star International Co., Ltd. [MSI] Starship/Matisse HD Audio Controller [1462:cb60]
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel
44:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03)
Subsystem: Micro-Star International Co., Ltd. [MSI] I211 Gigabit Network Connection [1462:7c60]
Kernel driver in use: vfio-pci
Kernel modules: igb
45:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03)
Subsystem: Micro-Star International Co., Ltd. [MSI] I211 Gigabit Network Connection [1462:7c60]
Kernel driver in use: vfio-pci
Kernel modules: igb
46:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
Kernel driver in use: vfio-pci
46:00.1 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
Subsystem: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:1486]
Kernel driver in use: vfio-pci
46:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
Subsystem: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:148c]
Kernel driver in use: vfio-pci
47:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
Subsystem: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901]
Kernel driver in use: vfio-pci
Kernel modules: ahci
48:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
Subsystem: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901]
Kernel driver in use: vfio-pci
Kernel modules: ahci
49:00.0 Ethernet controller [0200]: Aquantia Corp. AQC107 NBase-T/IEEE 802.3bz Ethernet Controller [AQtion] [1d6a:07b1] (rev 02)
Subsystem: Micro-Star International Co., Ltd. [MSI] AQC107 NBase-T/IEEE 802.3bz Ethernet Controller [AQtion] [1462:b912]
Kernel driver in use: vfio-pci
Kernel modules: atlantic
in this box you can see my PCI Device in VM conf:
hostpci0: 21:00,pcie=1,x-vga=1
hostpci1: 02:00.0 #nvme controller
hostpci10: 00:14.0 #smbus controller
hostpci11: 49:00 #Aquantia ethernet controller
hostpci12: 43:00 #Asmedia USB controller
hostpci2: 44:00 #I211 ethernet controller
hostpci3: 46:00.1 #USB controller
hostpci4: 46:00.3 #USB controller
hostpci5: 47:00 #Sata controller
hostpci6: 48:00 #Satacontroller
hostpci7: 01:00.0 #Nvme controller
hostpci8: 45:00 #I211 ethernet controller
hostpci9: 23:00.4 #matisse audio controller
Actual /etc/modules:
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
actual /etc/modprobe.d/blacklist.conf
#blacklist nouveau
#blacklist radeon
#blacklist snd_hda_codec_hdmi
#blacklist snd_hda_intel
#blacklist snd_hda_codec
#blacklist snd_hda_core
#blacklist xhci_hcd
#blacklist i2c_piix4
#blacklist sp5100_tco
all commentedl (I try stuff..not useful in my testing..maybe for others it could be?)
vfio.conf
#options vfio-pci ids=1022:148c
#options vfio-pci ids=1462:cb60
#options vfio-pci ids=1022:1487
all commented (obviously tested but no useful result for my rig)