Rox67er

i’m not 100% sure but think port 17 and 16 are on 49:00, 14 and 15 are on 48:00. I use an external ssd for proxmox on one of the usb ports next to the ps2 connector.

good catch i will update this! I had the GTX980 in slot 1 and Radeon VII in slot 3. After this I removed the GTX980 and put the Radeon in slot 1 so indeed this is le dump. 🤦🏼‍♂️ almost, I DO pass the usb-c through.

I came across some interesting stuff on USB ports / mapping for different TRX40 boards, maybe something we can use for our USB challenges: https://forums.unraid.net/topic/85811-trx40-motherboards/ Also, I can confirm that IF I pass any of the two (or both) USB controllers (47:00.0 / 47:00.1 / 47:00.3) my system will freeze after shutdown in MacOS and then restarting the VM (even with the GPU reset hook script) I have now removed the passthrough of the USB controllers completely (only use the USB-C rear) and everything works prefect. Also the GPU reset script now works fine so stop-start VM works fine.

Will try to investigate further, I’m pretty sure I didn’t have the issue when only GPU and nvme passed through. Did you see changes when passing things in different order?

I tried it and also irq setting but both did not help with audio issue.

I have the Sapphire Radeon VII so that gives the same result unfortunately. I switched my config over to CPU host and removed wifi / bluetooth and am quite happy now, boot reliable apart from the shutdown - start issue when booting windows, shutting down and then starting Proxmox without power off I get the USB 7-5 error and that leads to the error 99. Somehow low level settings which don't get reset in the bios. It might be related to BIOS and agesa version. I added my external U-Phoria UMC404HD audio interface and also got the stuttering audio. (didn't add the IRQ setting to VM yet, will try later) It works fine however if I plug it into my USB hub which is connected to the rear USB-C dedicated controller. So it is definitely something USB related. Do we need any SSDT or kext for the USB since we are passing them directly?

SSDT in deed. 😎

Good remark, I will look into that. I allocate 120GB (120*1024 = 122880, thought leaving 8GB for the VM would be sufficient.) and everything seems to be stable and running well. The huge pages suppose to give somewhat better performance, since the page table where ram allocation is kept is much smaller if you allocate huge chunks. (https://forum.proxmox.com/threads/huge-pages-support.15849/) I did change my 100.conf to use the "CPU host" config and tweaked my opencore config with insight from @Driftwood this gives 8-10% better performance in GeekBench than before! @Driftwood, is there a reason for the DSDT in your opencore ? I think they can both be removed. (I removed both and don't see a difference, plus the DSDT-ALS is I think for Ambient Light Sensor which is often a laptop thing) Anyhow, boot is now reliable although I did see the 99 error again so I have not gotten to the bottom of that one. I will try to remove some HW passthrough (maybe USB?) and see if I can narrow down the issue. @fabiosun I saw you also pass SMBus controller to MacOS, what is the benefit of this?

If you use the hook script it should work. I also noticed in your EFI you use the MacPro7,1 setting. Any reason for that? Benefits?

Well it seems the intel bluetooth was screwing up. I removed the kext and now everything is fine, no network issue anymore, reboots work, sleep works, shutdown and re-start VM works with the hook script. So pretty happy. Also the 99 error is now gone...? Looks like these low level hacks (injecting bluetooth firmware can have some unexpected behaviour)

@fabiosun @Driftwood I Just posted my config files and the IOMMU / USB pdf file on GitHub: https://github.com/iroger/ASRock-TRX40-proxmox I will try to keep it up to date with my latest config so others can use and learn as well. Things planned: * Swap out my intel wifi/USB for a fenvi BCM94360 NGFF version (It is in the mail now and should be a drop in replacement) since the intel one gives headaches. * Diagnose an USB 7-5 error I get occasionally when booting proxmox, when I get this error and then restart I get the Dr debug 99 error and have to switch off the power supply to get it booting again. (USB 7-5 is the internal LED controller) * Complete the USB overview, which port goes where * Go to a cleaner OpenCore config, probably with passing "CPU host" Any remarks and tips are welcome!

I notice you pass the cpu as host, can you share your EFI folder so I can compare with mine? I have to completed the usb overview but will post what I have tomorrow. It is possible to pass front USB and the two rear without passing entire controller.

I have to come back on my statement. It does make a difference in some occasions. Sometimes during boot I got an error on the bluetooth bios not loaded or something. After starting macos in deed no bluetooth. (with intel bluetooth kexts) after adding the bluetooth to vfio.conf proxmox leaves bluetooth alone. I still have issue though that when rebooting macos I loose the aquantia network. seems to work but only gives an self assigned ip.

@Driftwood I’m wondering, since you have been trying a lot with PCIe passthrough as I have and are on the same Hw, did you ever run into the issue where the system freezes at boot and the Dr Bios shows 99 on the motherboard? I have had this regularly and am thinking that I may have a hardware issue in my motherboard. The only thing that helps is to switch off the power supply and after that it boots normally...???

Can confirm that vfio.conf without ids=xxxx:yyyy is working just fine on my TRX40. I just removed all ids and no change in functionality.

Below my lspci dump translated to "Group / device / id / description" in deed my 03:00.3 / 25:00.3 and 25:00.4 give issues when passing through to VM. Also 148c / 1487 ID's. GROUP DEVICE ID DESCRIPTION 0 0000:00:01.0 1022:1482 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482] 1 0000:00:01.1 1022:1483 Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483] 2 0000:00:02.0 1022:1482 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482] 3 0000:00:03.0 1022:1482 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482] 4 0000:00:04.0 1022:1482 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482] 5 0000:00:05.0 1022:1482 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482] 6 0000:00:07.0 1022:1482 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482] 7 0000:00:07.1 1022:1484 Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484] 8 0000:00:08.0 1022:1482 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482] 9 0000:00:08.1 1022:1484 Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484] 10 0000:00:14.0 1022:790b Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 61) 10 0000:00:14.3 1022:790e Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51) 11 0000:00:18.0 1022:1490 Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 0 [1022:1490] 11 0000:00:18.1 1022:1491 Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 1 [1022:1491] 11 0000:00:18.2 1022:1492 Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 2 [1022:1492] 11 0000:00:18.3 1022:1493 Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 3 [1022:1493] 11 0000:00:18.4 1022:1494 Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 4 [1022:1494] 11 0000:00:18.5 1022:1495 Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 5 [1022:1495] 11 0000:00:18.6 1022:1496 Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 6 [1022:1496] 11 0000:00:18.7 1022:1497 Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 7 [1022:1497] 12 0000:01:00.0 10de:13c0 NVIDIA Corporation GM204 [GeForce GTX 980] [10de:13c0] (rev a1) 12 0000:01:00.1 10de:0fbb NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1) 13 0000:02:00.0 1022:148a Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a] 14 0000:03:00.0 1022:1485 Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485] 15 0000:03:00.3 1022:148c Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller [1022:148c] 16 0000:20:01.0 1022:1482 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482] 17 0000:20:02.0 1022:1482 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482] 18 0000:20:03.0 1022:1482 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482] 19 0000:20:03.1 1022:1483 Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483] 20 0000:20:04.0 1022:1482 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482] 21 0000:20:05.0 1022:1482 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482] 22 0000:20:07.0 1022:1482 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482] 23 0000:20:07.1 1022:1484 Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484] 24 0000:20:08.0 1022:1482 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482] 25 0000:20:08.1 1022:1484 Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484] 26 0000:21:00.0 1002:14a0 Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:14a0] (rev c1) 27 0000:22:00.0 1002:14a1 Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:14a1] 28 0000:23:00.0 1002:66af Advanced Micro Devices, Inc. [AMD/ATI] Vega 20 [Radeon VII] [1002:66af] (rev c1) 29 0000:23:00.1 1002:ab20 Advanced Micro Devices, Inc. [AMD/ATI] Vega 20 HDMI Audio [Radeon VII] [1002:ab20] 30 0000:24:00.0 1022:148a Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a] 31 0000:25:00.0 1022:1485 Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485] 32 0000:25:00.1 1022:1486 Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP [1022:1486] 33 0000:25:00.3 1022:148c Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller [1022:148c] 34 0000:25:00.4 1022:1487 Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller [1022:1487] 35 0000:40:01.0 1022:1482 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482] 36 0000:40:01.1 1022:1483 Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483] 37 0000:40:01.3 1022:1483 Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483] 38 0000:40:02.0 1022:1482 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482] 39 0000:40:03.0 1022:1482 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482] 40 0000:40:04.0 1022:1482 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482] 41 0000:40:05.0 1022:1482 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482] 42 0000:40:07.0 1022:1482 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482] 43 0000:40:07.1 1022:1484 Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484] 44 0000:40:08.0 1022:1482 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482] 45 0000:40:08.1 1022:1484 Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484] 46 0000:41:00.0 1022:57ad Advanced Micro Devices, Inc. [AMD] Device [1022:57ad] 47 0000:42:02.0 1022:57a3 Advanced Micro Devices, Inc. [AMD] Device [1022:57a3] 48 0000:42:03.0 1022:57a3 Advanced Micro Devices, Inc. [AMD] Device [1022:57a3] 49 0000:42:04.0 1022:57a3 Advanced Micro Devices, Inc. [AMD] Device [1022:57a3] 50 0000:42:05.0 1022:57a3 Advanced Micro Devices, Inc. [AMD] Device [1022:57a3] 51 0000:42:08.0 1022:57a4 Advanced Micro Devices, Inc. [AMD] Device [1022:57a4] 51 0000:47:00.0 1022:1485 Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485] 51 0000:47:00.1 1022:149c Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c] 51 0000:47:00.3 1022:149c Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c] 52 0000:42:09.0 1022:57a4 Advanced Micro Devices, Inc. [AMD] Device [1022:57a4] 52 0000:48:00.0 1022:7901 Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51) 53 0000:42:0a.0 1022:57a4 Advanced Micro Devices, Inc. [AMD] Device [1022:57a4] 53 0000:49:00.0 1022:7901 Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51) 54 0000:43:00.0 1b21:3242 ASMedia Technology Inc. Device [1b21:3242] 55 0000:44:00.0 1d6a:07b1 Aquantia Corp. AQC107 NBase-T/IEEE 802.3bz Ethernet Controller [AQtion] [1d6a:07b1] (rev 02) 56 0000:45:00.0 8086:2723 Intel Corporation Device [8086:2723] (rev 1a) 57 0000:46:00.0 10ec:8125 Realtek Semiconductor Co., Ltd. Device [10ec:8125] (rev 01) 58 0000:4a:00.0 1987:5016 Phison Electronics Corporation Device [1987:5016] (rev 01) 59 0000:4b:00.0 1022:148a Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a] 60 0000:4c:00.0 1022:1485 Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485] 61 0000:60:01.0 1022:1482 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482] 62 0000:60:02.0 1022:1482 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482] 63 0000:60:03.0 1022:1482 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482] 64 0000:60:04.0 1022:1482 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482] 65 0000:60:05.0 1022:1482 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482] 66 0000:60:07.0 1022:1482 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482] 67 0000:60:07.1 1022:1484 Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484] 68 0000:60:08.0 1022:1482 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482] 69 0000:60:08.1 1022:1484 Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484] 70 0000:61:00.0 1022:148a Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a] 71 0000:62:00.0 1022:1485 Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]

Could you explain a little bit more? Which USB controller did you pass to fix audio? 1) passthrough 46:00 (so all functions) 2) Do not pass through 23:00.4 ? Thanks! I think VFIO.conf is not needed. I put video card in there and none of the other PCIE. All passthrough OK, I will try to remove the video card as well and report back if that still works.

I took it one step further, I can now boot windows via F11 but ALSO run it from the same install in a VM. In this way I can run both in parallel, wanting to hook up two monitors and use https://github.com/debauchee/barrier to share one keyboard / mouse on both systems. 🙂 (barrier is still theory only but it should work)

USB table is still work in process, I will post my in between results mid next week. I just edited in nano and paste the content inside, make sure the file is executable. (chmod +x)

I also get a ipv6 but don’t have an issue with it. Maybe disable ipv6 on your router? I have not use install 6.2 yet, just upgraded from my installed 6.1 version. I did see some issues when passing through hardware to the guest which is also using resources in the host (e.g. Proxmox drive) it then messed up my pci in such way that a reboot wouldn’t help. Keep getting an “99” error on the DrBios on the motherboard. Bios defaults seemed to work but ass soon as I enabled the xmp profile for memory it froze again on boot. Had to switch of my power supply and restart to get it resolved. I also have seen that leaving the Bluetooth module on auto gives me a bios rom error for Bluetooth when booting Proxmox. If I set it to enabled it does work.

Last night first time had macOS and Windows running in parallel. 😃 I just completed my first test result with: 1) VM running Catalina on Radeon VII, 48 cores, 96 Gb memory Geekbench 5 single core ~1150 / multicore ~22000 2) VM running Windows 10 on GTX980, 16 cores, 24 Gb memory Geekbench 5 single core ~1200 / multicore ~10500 Both running in parallel with dedicated GPU / monitor and dedicated disk. 🙂 Now figuring out how to split resources (USB / SATA) I’m working on a table showing which USB port goes to which physical port and is on which USB-hub / pci will share once completed. 👍🏻

For those on ASRock TRX40 creator running into network issues, I locked myself out at one point with an autostating VM with aquantia passthrough. I had to use a linux recovery disk (USB stick) and mount the Proxmox root partition to access the network file and change it to the other NIC to get into Proxmox again. More info here: https://pve.proxmox.com/wiki/Network_Configuration

Good tip! Interesting things happen when you passthrough the SATA controller which has the Proxmox disk hooked up. 🙂 He driftwood, I had the same issue, you need to update Proxmox to support the Realtek. It was added recently. Look here: https://forum.proxmox.com/threads/linux-kernel-5-4-for-proxmox-ve.66854/ I think you can now do a normal update - upgrade step since the kernel is released now.

It is a so called hook script, save it in the location: /var/lib/vz/snippets and add it to your conf file with: hookscript: local:snippets/reset-gpu.pl

So you have two Radeon VII in your rig? I guess one in slot 1 and one in slot 3. Do you have the GPU reset issue if you stop the VM and start it again that the GPU doesn't init anymore until you reboot the host?