Proxmox VE OSX Guide discussion

[fabiosun]

@Driftwoodin Catalina no rendering support for Nvidia

in high Sierra performance in adobe or davinci resolve app are similar and maybe little better than a Radeon VII

 

[Driftwood]

ASRock techie guy sent me his 1.63 IOMMU Groups spreadsheet which looks correct. He is interested to see how we get on with Linux and Proxmox. 

Also attached is my latest IOMMU groups with my firewire and decklink cards swapped into pcie slots 2 and 4 for those who need to see what is going on.

 

 

IOMMU TRX40 Creator BIOS 1.60 & 1.63.ods.zip

IOMMU groups Driftwood.xlsx.zip

Edited June 10, 2020 by Driftwood

[cj750]

OK,

 

This is nail bitting for me. I feel im so close but so far off. 

 

So the display cable didn't make a difference. If i don't use the PCI GPU for the video card it boot fine. But as soon as i boot with my GPU i get a black screen. Iv tried every one  tisp and config changes and no luck. Maybe it because Im new to Proxmox. I made multiple templates so i could not mess up the boot original. 

 

Is there another error log i can see that will help me find why it won't boot with my card? I know there a reset bug. But i not sure if my screen goes black after the promos loader begins and i have to use another computer to start the VMs. 

[fabiosun]

@cj750 what do you mean when younsay it boots fine? From webinterface console window you see osx ?

could you post vm conf file, blacklist.conf, modules, vfio.conf, kvm.conf?

also grub file would be fine

 

[Rox67er]

18 hours ago, Driftwood said:

ASRock techie guy sent me his 1.63 IOMMU Groups spreadsheet which looks correct. He is interested to see how we get on with Linux and Proxmox. 

Also attached is my latest IOMMU groups with my firewire and decklink cards swapped into pcie slots 2 and 4 for those who need to see what is going on.

 

 

IOMMU TRX40 Creator BIOS 1.60 & 1.63.ods.zip 7.34 kB · 5 downloads

IOMMU groups Driftwood.xlsx.zip 9.54 kB · 0 downloads

So I tried the new bios and it doesn’t help for me. Without my hook script I get the stuck in D3 error on the GPU and with the hook script I get the 99 error after starting vm the second time as soon as I also pass the USB controllers. I do see some improvement when using pcie_acs_override setting in grub (system doesn’t freeze with 99 error but reboots)

[cj750]

Hello,

 

sorry i was a little tired when i made that post. 

 

You requested post vm conf file, blacklist.conf, modules, vfio.conf, kvm.conf and grub. Iv added it to the post. 

 

What i was trying to say was. When i don't add the PCI device:GPU to the VM and just use standard VGA. The MACOS boot , but every time i use PCI device my GPU  and display-none. It just goes to a black screen. i didnt use the new bios driftwood got yet. wanted to see if i could get it working on latest one first. 

Myconfig.zip

Edited June 11, 2020 by cj750

[fabiosun]

hi

have you ever seen proxmox logo?

Could you try to comment in this way blacklist.conf?:

#blacklist snd_hda_intel

#blacklist snd_hda_codec_hdmi

#blacklist i915

#blacklist radeon

#blacklist nouveau

#blacklist nvidia

#blacklist i2c-nvidia-gpu

#blacklist amdgpu

in this way you can operate directly from your AMD and in the shell you can execute:

qm start 101

and see

in grub line I 'd put only:

GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on video=efifb:off"

and remember to run update command:

update-initramfs -u -k all

update-grub

and then:

reboot

 

 

also if you can you can use FileZilla to copy your files..

 

[Rox67er]

My new WiFi / BT card has arrived! 😁

 

and no, I don’t have very thick fingers... 

 

[Driftwood]

12 minutes ago, Rox67er said:

new WiFi / BT card has arrived! 😁

Interested to see how u get on with this. If it works, I buy!

[cj750]

3 hours ago, fabiosun said:

hi

have you ever seen proxmox logo?

Could you try to comment in this way blacklist.conf?:

#blacklist snd_hda_intel

#blacklist snd_hda_codec_hdmi

#blacklist i915

#blacklist radeon

#blacklist nouveau

#blacklist nvidia

#blacklist i2c-nvidia-gpu

#blacklist amdgpu

in this way you can operate directly from your AMD and in the shell you can execute:

qm start 101

and see

in grub line I 'd put only:

GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on video=efifb:off"

and remember to run update command:

update-initramfs -u -k all

update-grub

and then:

reboot

 

 

also if you can you can use FileZilla to copy your files..

 

I will try this when I get back home.

 

Thank you

[fabiosun]

@Rox67erhave you a proper slot for that wifi?

or you will use a pcie adapter for it?

 

[Rox67er]

1 hour ago, fabiosun said:

@Rox67erhave you a proper slot for that wifi?

or you will use a pcie adapter for it?

 

@Driftwood @fabiosun

 

This is one happy guy. 😁

I swapped it out with the intel AX200 which was on-board on the TRX40 Creator, It is a drop in replacement if you buy the BCM94360NG version.

 

Initially showed the wifi as EN2 (which I'm not sure handoff / continuity likes) so after resetting the Networkinterfaces in MacOs and rebooting 

sudo rm /Library/Preferences/SystemConfiguration/NetworkInterfaces.plist 
sudo rm /Library/Preferences/SystemConfiguration/preferences.plist

It now shows as EN1, and adding the built-in flag in OpenCore I got this:

 

Now wifi and bluetooth were working, airdrop was working but continuity and Apple Watch unlock not yet. Next I signed out of apple-id on my hack and signed in again and....

wooohooo everything works, BT / Wifi / Airdrop / Continuity / Apple Watch unlock / iMessage

 

The swap is relatively easy (considering you have to take the motherboard out to reach it) remove these two screws (holding the metal cover for the wifi card)

 

Then you can lift the entire metal case up (internal wifi card will disconnect from the motherboard connector) if you bring it under an angle you can remove it to the rear of the motherboard without removing the heatsink.

 

It leaves you with this:

 

And after removing the one Philips screw you can open the metal case by tilting the lid. (I used a small screw driver to open it)

 

Swap out the intel board for the Broadcom one, reconnect the antenna's (make sure they stay connected as when closing the lid it may push against the antenna cables)

Make also sure that you don't mount the new wifi board too high on the internal Philips screw as it will have a poor connection to the motherboard then. (Also don't push it down as far as possible as you will find there is tension when mounting the metal frame back on the motherboard.

 

Below image should illustrate what I mean (same space to the sides as to the bottom of the screw):

 

Mount everything back in reverse order and enjoy your fully working BT / WiFi.😁

 

Proxmox settings:

Off course you have to add the cards pci  to you vfio with "options vfio-pci ids=14e4:43a0" and add it to the VM passthrough. I also added "blacklist bcma" to make sure linux leaves the wifi card alone. (checked, MacOs wil hang during boot if this is omitted)

Edited June 11, 2020 by Rox67er
Added Proxmox settings

[fabiosun]

@Rox67ermy motherboard has an empty slot but I thought it was a pciex1 slot

thank you for your detailed tutorial 🙂

[Rox67er]

21 minutes ago, fabiosun said:

@Rox67ermy motherboard has an empty slot but I thought it is a pciex1 slot

thank you for your detailed tutorial 🙂

From what I can see in the picture it is exactly the same connector, I think it is called an NGFF intended for WiFi. So if you can buy the same card with the metal casing you can add wifi too without offering a PCIE slot. (Some nice info here: https://www.delock.com/infothek/M.2/M.2_e.html)

 

Or use some internal antenna like this:

Edited June 11, 2020 by Rox67er

[Driftwood]

2 hours ago, Rox67er said:

The swap is relatively easy (considering you have to take the motherboard out to reach it) remove these two screws (holding the metal cover for the wifi card)

Nice work. Yeah I had to take the mobo apart and unscrew to remove the old one. Now I will buy Broadcom, just ordered off ebay... totally forgot about buying this.

 

Thanks

 

 

2 hours ago, fabiosun said:

motherboard has an empty slot but I thought it was a pciex1 slot

Yeah, its actually a m.2 slot but they dont advise using m.2s in it 🙂

Edited June 11, 2020 by Driftwood

[cj750]

6 hours ago, fabiosun said:

hi

have you ever seen proxmox logo?

Could you try to comment in this way blacklist.conf?:

#blacklist snd_hda_intel

#blacklist snd_hda_codec_hdmi

#blacklist i915

#blacklist radeon

#blacklist nouveau

#blacklist nvidia

#blacklist i2c-nvidia-gpu

#blacklist amdgpu

in this way you can operate directly from your AMD and in the shell you can execute:

qm start 101

and see

in grub line I 'd put only:

GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on video=efifb:off"

and remember to run update command:

update-initramfs -u -k all

update-grub

and then:

reboot

 

 

also if you can you can use FileZilla to copy your files..

 

When i made the changes this is what i get below and still black screen. 

 

qm start 101

kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]

kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]

kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]

kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]

 

Jun 11 10:44:58 pmce kernel: vfio-pci 0000:03:00.0: vfio_ecap_init: hiding ecap 0x19@0x270

Jun 11 10:44:58 pmce kernel: vfio-pci 0000:03:00.0: vfio_ecap_init: hiding ecap 0x1b@0x2d0

[cj750]

1 hour ago, Rox67er said:

@Driftwood @fabiosun

 

This is one happy guy. 😁

I swapped it out with the intel AX200 which was on-board on the TRX40 Creator, It is a drop in replacement if you buy the BCM94360NG version.

 

Initially showed the wifi as EN2 (which I'm not sure handoff / continuity likes) so after resetting the Networkinterfaces in MacOs and rebooting 

sudo rm /Library/Preferences/SystemConfiguration/NetworkInterfaces.plist 
sudo rm /Library/Preferences/SystemConfiguration/preferences.plist

It now shows as EN1, and adding the built-in flag in OpenCore I got this:

 

Now wifi and bluetooth were working, airdrop was working but continuity and Apple Watch unlock not yet. Next I signed out of apple-id on my hack and signed in again and....

wooohooo everything works, BT / Wifi / Airdrop / Continuity / Apple Watch unlock / iMessage

 

The swap is relatively easy (considering you have to take the motherboard out to reach it) remove these two screws (holding the metal cover for the wifi card)

 

Then you can lift the entire metal case up (internal wifi card will disconnect from the motherboard connector) if you bring it under an angle you can remove it to the rear of the motherboard without removing the heatsink.

 

It leaves you with this:

 

And after removing the one Philips screw you can open the metal case by tilting the lid. (I used a small screw driver to open it)

 

Swap out the intel board for the Broadcom one, reconnect the antenna's (make sure they stay connected as when closing the lid it may push against the antenna cables)

Make also sure that you don't mount the new wifi board too high on the internal Philips screw as it will have a poor connection to the motherboard then. (Also don't push it down as far as possible as you will find there is tension when mounting the metal frame back on the motherboard.

 

Below image should illustrate what I mean (same space to the sides as to the bottom of the screw):

 

Mount everything back in reverse order and enjoy your fully working BT / WiFi.😁

 

Proxmox settings:

Off course you have to add the cards pci  to you vfio with "options vfio-pci ids=14e4:43a0" and add it to the VM passthrough. I also added "blacklist bcma" to make sure linux leaves the wifi card alone. (checked, MacOs wil hang during boot if this is omitted)

Nice Work!

[fabiosun]

1 hour ago, cj750 said:

When i made the changes this is what i get below and still black screen. 

 

qm start 101

kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]

kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]

kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]

kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]

 

Jun 11 10:44:58 pmce kernel: vfio-pci 0000:03:00.0: vfio_ecap_init: hiding ecap 0x19@0x270

Jun 11 10:44:58 pmce kernel: vfio-pci 0000:03:00.0: vfio_ecap_init: hiding ecap 0x1b@0x2d0

Your vga is on 03:00 group?

 

[cj750]

2 minutes ago, fabiosun said:

Your vga is on 03:00 group?

 

No my FE GPU is on 03:30

[fabiosun]

Post output of

lspci -nnk

 

3:30 seems weird

[Rox67er]

1 hour ago, cj750 said:

When i made the changes this is what i get below and still black screen. 

 

qm start 101

kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]

kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]

kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]

kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]

 

Jun 11 10:44:58 pmce kernel: vfio-pci 0000:03:00.0: vfio_ecap_init: hiding ecap 0x19@0x270

Jun 11 10:44:58 pmce kernel: vfio-pci 0000:03:00.0: vfio_ecap_init: hiding ecap 0x1b@0x2d0

The warnings come from the +pcid option in the VM cpu line. It is safe to remove that.

 

Are you sure your settings as posted are correct? I see this:

rgs: -device isa-applesmc,osk="ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc" -smbios type=2 -cpu Penryn,kvm=on,vendor=GenuineIntel,+kvm_pv_unhalt,+kvm_pv_eoi,+invtsc,vmware-cpuid-freq=on,+pcid,+ssse3,+sse4.2,+popcnt,+avx,+aes,+xsave,+xsaveopt,check -device usb-kbd,bus=ehci.0,port=2
Penryn,kvm=on,vendor=GenuineIntel,+kvm_pv_unha$
balloon: 0
bios: ovmf

The first line should start with:

args: -device .....

 

The second line is a half copy of something from the first line and should not be there.

 

I don't understand your GRUB video=amdgpu setting is that because of an internal GPU ?

 

To try and release the GPU from Proxmox you could try this command to start the VM: (It basically does the same as my hook script)

echo 0 > /sys/class/vtconsole/vtcon0/bind; echo 0 > /sys/class/vtconsole/vtcon1/bind; echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind; qm start 101

 

[cj750]

13 minutes ago, fabiosun said:

Post output of

lspci -nnk

 

3:30 seems weird

First sorry i meant 03:00 was GPU. My bad.

 

Going to look at Rox67er post and check my setting again,

[Driftwood]

So here's my back panel layout for the ASRock Creator TRX40:-

 

 

Motherboard Layout

 

 

 

* Its important you keep a tab on your IOMMU Groups & ID addresses for VFIO.conf because if you move your cards around address will move around for the PCIe cards.

Edited June 11, 2020 by Driftwood

[cj750]

7 hours ago, Rox67er said:

 

 

 

OK ,

 

So I with the word the screen was just smaller that caused the wrap around and I did a bad job of copying and pasting. 

 

I took my time and check all the setting as you suggested. And Also was able to run the VM unbind command . 

 

Below is the syslog: It keep running "Started Proxmox VE replication runner"  This make me feel that the mac is running in the background but i just cant see it on my screen. Which is a LG GX 55' TV.. I tried to use HDMI and a 4K Displayport to HDMi cable. When the computer boot i see the BIOS for the MB and then the Proxmox to choose the version. But after that the screen goes black. 

 

Now when i login from another computer. I can start the VM. If i don't use the GPU i can VNC into  the VM and the mac os boot fine. But when i add the GPU. I get a black screen and nothing after that. I quoted this post because these are the reason you seen those random entries like the +pcid option. I did remove it. I also remove the GRUB video=amdgpu.

 

When i tried to run the unbind 

 

root@pmce:~# /sys/bus/platform/drivers/efi-framebuffer/unbind
-bash: /sys/bus/platform/drivers/efi-framebuffer/unbind: Permission denied

 

Also i added screen shot of what the syslog said after running the qm start 101.  Im trying my best to add as much detail at the same time with my learning proxm. Again ty for everyone help ahead of time. 

 

O i also tried to plug HDMI cable and Display at same time no luck. Seen you did it on a post also. 

 

 

Jun 11 18:56:00 pmce systemd[1]: Started Proxmox VE replication runner.
Jun 11 18:57:00 pmce systemd[1]: Starting Proxmox VE replication runner...
Jun 11 18:57:00 pmce systemd[1]: pvesr.service: Succeeded.
Jun 11 18:57:00 pmce systemd[1]: Started Proxmox VE replication runner.
Jun 11 18:57:32 pmce sshd[12831]: Accepted password for root from 192.168.165.169 port 52195 ssh2
Jun 11 18:57:32 pmce sshd[12831]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 11 18:57:32 pmce systemd[1]: Created slice User Slice of UID 0.
Jun 11 18:57:32 pmce systemd[1]: Starting User Runtime Directory /run/user/0...
Jun 11 18:57:32 pmce systemd-logind[3810]: New session 2 of user root.
Jun 11 18:57:32 pmce systemd[1]: Started User Runtime Directory /run/user/0.
Jun 11 18:57:32 pmce systemd[1]: Starting User Manager for UID 0...
Jun 11 18:57:32 pmce systemd[12940]: pam_unix(systemd-user:session): session opened for user root by (uid=0)
Jun 11 18:57:32 pmce systemd[12940]: Listening on GnuPG cryptographic agent (ssh-agent emulation).
Jun 11 18:57:32 pmce systemd[12940]: Reached target Timers.
Jun 11 18:57:32 pmce systemd[12940]: Listening on GnuPG cryptographic agent and passphrase cache.
Jun 11 18:57:32 pmce systemd[12940]: Listening on GnuPG cryptographic agent and passphrase cache (restricted).
Jun 11 18:57:32 pmce systemd[12940]: Listening on GnuPG cryptographic agent and passphrase cache (access for web browsers).
Jun 11 18:57:32 pmce systemd[12940]: Reached target Paths.
Jun 11 18:57:32 pmce systemd[12940]: Listening on GnuPG network certificate management daemon.
Jun 11 18:57:32 pmce systemd[12940]: Reached target Sockets.
Jun 11 18:57:32 pmce systemd[12940]: Reached target Basic System.
Jun 11 18:57:32 pmce systemd[12940]: Reached target Default.
Jun 11 18:57:32 pmce systemd[12940]: Startup finished in 49ms.
Jun 11 18:57:32 pmce systemd[1]: Started User Manager for UID 0.
Jun 11 18:57:32 pmce systemd[1]: Started Session 2 of user root.
Jun 11 18:58:00 pmce systemd[1]: Starting Proxmox VE replication runner...
Jun 11 18:58:00 pmce systemd[1]: pvesr.service: Succeeded.
Jun 11 18:58:00 pmce systemd[1]: Started Proxmox VE replication runner.
Jun 11 18:59:00 pmce systemd[1]: Starting Proxmox VE replication runner...
Jun 11 18:59:00 pmce systemd[1]: pvesr.service: Succeeded.
Jun 11 18:59:00 pmce systemd[1]: Started Proxmox VE replication runner

 

 

 

 

Logs

Edited June 12, 2020 by cj750

[Driftwood]

1 hour ago, cj750 said:

Im trying my best to add as much detail at the same time with my learning proxm. Again ty for everyone help ahead of time. 

 

@cj750 What is your vfio.conf looking like? You are passing the addresses: for video and HDMI audio: -

ie you have the following for the Frontier card in your vfio.conf

options vfio-pci ids=1002:6863 ,1002:aaf8 disable_vga=1?

 

After every modprobe vfio change your updating the initramfs, update grub, pve-efiboot-tool refresh and shutdown now? switch off, then power back up.

 

Here's a few VFIO guide tips: https://github.com/SMeatBoy/vfio/blob/master/README.md

Edited June 12, 2020 by Driftwood