Proxmox VE OSX Guide discussion

[tsongz]

I'm attaching my pre-compiled PVE kernels. It likely won't work for you unless you have a Radeon VII or Vega FE (since those are the device IDs I selected, but I'll write the instructions below)

 

This is my modified patch script for Vega

 

Where 0x6863 is device ID of Vega FE and 0x66af is Device ID of Radeon VII. You can find yours specifically either through TechPowerUp or through your lspci (more accurate)

 

 

Then replace the part after 0x with the last 4 of the device ID. Ideally, it would have all the affected Vega devices, but I have kept it at 2 since the original script was 2.

 

diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
index 44c4ae1abd00..27840129e4b0 100644
--- a/drivers/pci/quirks.c
+++ b/drivers/pci/quirks.c
@@ -3433,6 +3433,14 @@ DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_ATHEROS, 0x0034, quirk_no_bus_reset);
*/
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_CAVIUM, 0xa100, quirk_no_bus_reset);

+/*
+ * Radeon RX Vega and Navi devices break on bus reset. Oi...
+ * This is *not a real workaround* - disabling bus reset
+ * for your GPU may have unintended consequences.
+ */
+DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_ATI, 0x6863, quirk_no_bus_reset); <- Modify & remove this comment (Vega FE)
+DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_ATI, 0x66af, quirk_no_bus_reset); <- Modify & remove this comment (Radeon VII)
+
static void quirk_no_pm_reset(struct pci_dev *dev)
{
/*

 

For good measure, what I did was create a new root user in Proxmox for compiling specifically, so that things don't get mixed up.

 

Assuming you already have pve-kernel git cloned and the submodules already updated, then this should take care of patching

 

# Or whichever Linux kernel you're using for the path below.
cd ~/pve-kernel/submodules/ubuntu-focal
# The path below expects fix-vega-reset.patch above to be the path below, modify it to match where you saved the file above.
patch -p1 < /home/pvecompiler/fix-vega-reset.patch 
# You can check the git diff to see that it has applied
git diff
# Now go back to the pve-kernel folder and make it
cd /home/pvecompiler/pve-kernel #or cd - or cd ../../
make

 

Then I downloaded the Reset-test app from the original thread in PVE, chmod +x the application, and set it in my script. 

 

Link to PVE Kernels here. It was 70mb compressed, so not able to upload here. I installed all 3 of the kernels, since it's within the same PVE version.

 

[Driftwood]

10 hours ago, asynic said:

Hey, im having issues with a 3900x and a vega vii with passthrough. The machine works but freezes a lot for like 1-2 seconds and i cant play videos off of youtube for example neither, it stutters a lot. Any Ideas?

Sounds like you havent pass thru'd the USBs correctly or something. Shows us your VFIO and vm config please 

[meina222]

@tsongz - thank you. Yes, I believe this works only on Vega cards. Fabian from Proxmox has Navi reset 5.8.3 kernel, which I suspect involves the power switch - I built that myself on a lower 5.7.x kernel a couple of months back, and didn't quite work for my 5700XT, but I can retry with the 5.8.3.

[meina222]

Question on Proxmox and USB - does any of you use custom SSDT? No matter what I inject Hackintool always shows no or very few ports. Do any SSDT's help with port capabilities when passing through PCI USB controller from Proxmox?

[tsongz]

@meina222 - I currently do with these

 

 

My PCIe Configuration

 

**** In-Use Passthrough Devices ****

PCIe ID: 23:00.0 | VFIO ID: 1002:6863 | IOMMU Group: 29 | Used By: hostpci1      | Device: VGA compatible controller - Advanced Micro Devices, Inc.
PCIe ID: 23:00.1 | VFIO ID: 1002:aaf8 | IOMMU Group: 30 | Used By: hostpci1      | Device: Audio device - Advanced Micro Devices, Inc.
PCIe ID: 25:00.3 | VFIO ID: 1022:148c | IOMMU Group: 34 | Used By: hostpci3      | Device: USB controller - Advanced Micro Devices, Inc.
PCIe ID: 45:00.0 | VFIO ID: 1d6a:07b1 | IOMMU Group: 60 | Used By: hostpci2      | Device: Ethernet controller - Aquantia Corp. AQC107 NBase-T/IEEE 802.3bz Ethernet Controller
PCIe ID: 4b:00.0 | VFIO ID: 1987:5012 | IOMMU Group: 68 | Used By: hostpci0      | Device: Non-Volatile memory controller - Phison Electronics Corporation E12 NVMe Controller
********************************************************************************************************************************************

 

 

This allows the full passthrough of my MB top USB slots. The downside with this is that the keyboard/USB slots doesn't initialize until after the OC Boot screen, unless a USB device is passed through explicitly from proxmox, so I have a secondary keyboard I passthrough as a USB Device in case I need to do anything. I don't pass through my case USB devices.

 

[meina222]

Thank you @tsongz. What does SSDT-ALS do?

 

Also, in Hackintool USB section - does it show any USB ports? Can you set port capability via SSDT or it's not needed?

[tsongz]

They were the included SSDTs from the original PVE-OC iso, and I haven't removed it (probably should). ALS is for Ambient Light Sensors (not needed).

 

I think the main one should be SSDT-EC.

 

This is what's showing up for me. I have not modified it unlike for Bare Metal.

 

[meina222]

I have the same. No matter what kexts I use, the injection is as above. Probably I should disable anything related to USB other than EC.

Edited August 28, 2020 by meina222

[23d1]

Gonna give this a go on a 3990x this weekend. The plan is to have a triple-boot — Linux / Windows / macOS. Will report back.

Edited September 23, 2020 by 23d1

[fabiosun]

hi @23d1welcome here 

glad to see your triple 1080ti GPU rig

I think you will try macOS High Sierra to have a full GPU driver support

In Proxmox should be more simple than bare metal because in bare metal trx40 and HighSierra have some additional problem to solve

 

[23d1]

Hey @fabiosun, thanks!

Yeah, I've been holding on to the 1080Ti cards for the purpose of using in macOS, but will likely mix it up with the new AMD Big Navi coming out (the rumored 6900xt, if it pans out) to be able to bump up to Big Sur. Definitely gonna do Proxmox. I think my main concern at the moment is the triple-VM setup and how I'm gonna sort out allocating hardware resources. I do a lot of GPU accelerated rendering, and Redshift is coming to Apple Metal (which I'm beta testing at the moment), so if I can get out from the Nvidia strangle-hold, I'll be a happy camper. Worst case I can probably do an Arch Linux setup for the GPU related work, and then KVM into macOS for Adobe and so forth. Gonna be a Frankenstein setup. Hah. 🙂

 

By the way, I haven't seen anyone have any luck with stable baremetal on the 3990x yet—has there been many reports of success? I'll scour the forum(s) and see as well.

Edited September 23, 2020 by 23d1

[fabiosun]

Check @meina222he had success in both tasks👍👍

He has a gigabyte ex and a 3990x

[meina222]

@23d1

 

3990x works in bare metal and VM passthrough in both Catalina and Big Sur. Never tried lower such as High Sierra, where @fabiosun has lots of experience. Don't know if the 1080ti is supported. 

 

On the 3990x, you need to disable SMT for bare metal (which would make sleep impossible if you care about that) and you can't pass through more than 64 logical cores to MacOS in Proxmox (where I would strongly recommend to enable SMT, especially if you want to try running VMs in parallel, something I don't do).  For  optimal setup on both VM and bare metal you may need to switch BIOS settings if you change boots.

Edited September 23, 2020 by meina222

[23d1]

Thanks so much, @fabiosun and @meina222 — it seems Proxmox is the way to go with this massive core count. Hopefully macOS will support more cores in the near future, but since Apple will transition to ARM, it's going to be interesting to see everything unfold in the coming couple of years.

[meina222]

It's a personal preference - 64 cores (no SMT) performs very close to what 128 logical cores would for pretty much all applications. In your case the only challenge will be the GPU if you go Proxmox. You would have to stick with High Sierra. Purchasing an AMD card and upgrading above High Sierra will cause you headaches with reset and you will find the VM experience a bit frustrating. If the reset issues get fixed with Navi 21/22/23, VM remains one of the best options as performance is excellent.

[meina222]

Has anyone succeeded installing Windows 10 on Proxmox VM guest? I can't seem to apply the virtIO drivers and as a result my guest doesn't see the installation target disk. I followed all directions I could find including watching this https://www.proxmox.com/en/training/video-tutorials/item/install-windows-2016-server-on-proxmox-ve which according to Proxmox should be the same for Windows 10. I also got latest virtio drivers (1.189) and Windows 10 2004 iso directly from Microsoft. Tried also Windows 10 1909 and drivers 1.185 and 1.173. Same issue.

[fabiosun]

@meina222 I did in the past and I do not remember any particular problem

I did with latest virtio drivers iso available, but to install I need in the start only of windows.iso downloaded from microsoft and a vm.conf created as usual

 

[meina222]

@fabiosun

 

This thread described the problem.

 

https://forum.proxmox.com/threads/can-not-install-windows-2019.75479/

 

I verified the .iso (downloaded from Microsoft). The only thing I can think of that makes my setup "unusual" is that the target drive is the 4x4x4x4 NVMe AIC. Works great for my MacOS VMs. Finally for the 1st time since getting the TRX40 I decided to install Windows and to my surprise this is proving more difficult than MacOS so far on VM 😂

[fabiosun]

That thread is newer than when I have tried last time...so maybe some differences or in proxmox kernel or other thing 

[meina222]

I managed to install Windows with sata and 1909 version. I then installed the virtio drivers from within Windows. Definitely not an easy install, took me days to figure out. I/O performance on the zfs array is blistering under CrystalMark with write back (unsafe) - shockingly good sequential read speeds, GPU performance is on par or slightly better than MacOS With latest Adrenalin, but CPU is seriously lagging to MacOS. I’ll post some results a bit later.

[cj750]

Hello,

 

wow this group has grown. Well I dee most have transition over to Big Slur.. 

 

Im using 10.15.7. But I need help with getting thunderbolt 2 or 3 to work. I used my USB C port for a USB multiport. Like in the past post here. It works great. But I have one USB C port in the front also. But i need help with getting thunderbolt 2 to work. Do you know of any card that will support it on HachOSX. I have 2 5 bay drives i need to connect to the system. 

 

Thank you 

[Driftwood]

3 hours ago, cj750 said:

Im using 10.15.7. But I need help with getting thunderbolt 2 or 3 to work. I used my USB C port for a USB multiport. Like in the past post here. It works great. But I have one USB C port in the front also. But i need help with getting thunderbolt 2 to work. Do you know of any card that will support it on HachOSX. I have 2 5 bay drives i need to connect to the system. 

Asrock have told me that our motherboard wont support Thunderbolt and I requote:

"Our TRX40 models have no header for a Thunderbolt add-in card. No way to add Thunderbolt support to these boards, sorry".

 

So unless @fabiosun and @iGPU have furthered their quest using different Thunderbolt cards I don't think its possible or reliable. Guys?

 

Edited October 12, 2020 by Driftwood

[Driftwood]

On 9/24/2020 at 6:07 PM, meina222 said:

Has anyone succeeded installing Windows 10 on Proxmox VM guest? I can't seem to apply the virtIO drivers and as a result my guest doesn't see the installation target disk. I followed all directions I could find including watching this https://www.proxmox.com/en/training/video-tutorials/item/install-windows-2016-server-on-proxmox-ve which according to Proxmox should be the same for Windows 10. I also got latest virtio drivers (1.189) and Windows 10 2004 iso directly from Microsoft. Tried also Windows 10 1909 and drivers 1.185 and 1.173. Same issue

Maybe install vanilla and let Windows try and reset its drivers on the fly using your Proxmox setup with a slightly differing boot setup to OSX. Try Host or Penryn.

[cj750]

2 hours ago, Driftwood said:

Asrock have told me that our motherboard wont support Thunderbolt and I requote:

"Our TRX40 models have no header for a Thunderbolt add-in card. No way to add Thunderbolt support to these boards, sorry".

 

So unless @fabiosun and @iGPU have furthered their quest using different Thunderbolt cards I don't think its possible or reliable. Guys?

 

OK Ty,  well i guess i will have to invest in a 10gb bay and just use the 10gb port to stream music samples. 

 

Ty again

 

[meina222]

@cj750,

 

If you're asking about Thunderbolt in MacOS VM - I'm not aware of anyone having passed through the TB card setup. You might be able to passthrough Thunderbolt leaf devices, but I have not tried that yet. On the other hand, you should be able to make Thunderbolt work in bare metal. Please, contact @iGPU and @fabiosun for help on that.