Proxmox VE OSX Guide discussion

[meina222]

3 hours ago, dtek said:

OMG I can't wait to try this.  GPU passthrough  was removed from my VM ever since I had to reinstall everything from scratch.  It refused to boot after multiple forceful resets 😣

On your host (Proxmox) try:

 

git clone https://github.com/gnif/vendor-reset.git
cd vendor-reset
dkms install .
echo "vendor-reset" >> /etc/modules
update-initramfs -u

reboot

 

If all works well, your GPU should start resetting.

 

You may need to install the dkms package if not there by default.

Edited November 27, 2020 by meina222

[dtek]

On 11/27/2020 at 4:13 PM, meina222 said:

On your host (Proxmox) try:

 

git clone https://github.com/gnif/vendor-reset.git
cd vendor-reset
dkms install .
echo "vendor-reset" >> /etc/modules
update-initramfs -u

reboot

 

If all works well, your GPU should start resetting.

 

You may need to install the dkms package if not there by default.

Is there an updated hookscript or should I remove it completely?

Edited November 29, 2020 by dtek

[meina222]

I re-enabled the re-binding of the framebuffer and vtconsole - the highlighted commands were previously commented out - this is needed since otherwise you can't unbind in the pre-start phase when you restart.

 

When you shutdown your VM, you'd need an external device to restart it as main display won't come back to host (still not sure how to achieve that) but you can re-start VM from web console on a different device and display should come back. Similarly, rebooting a VM should just work and GPU should reset properly (for me DisplayPort sound stops working but this is WIP by developer).

 

#!/bin/bash

vmid="$1"
phase="$2"

if [[ "$phase" == "pre-start" ]]; then
    clear
    echo "Starting VM $vmid - please wait..."
    IFILE=/var/lib/vz/snippets/interfaces.$vmid
    if [[ -f "$IFILE" ]]; then
        cp /var/lib/vz/snippets/interfaces.$vmid /etc/network/interfaces
        systemctl restart networking
    fi          
    echo 0 > /sys/class/vtconsole/vtcon0/bind
    echo 0 > /sys/class/vtconsole/vtcon1/bind
    echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind
elif [[ "$phase" == "post-start" ]]; then
    main_pid="$(< /run/qemu-server/$vmid.pid)"

    cpuset="0-63"
    #cpuset="$(< /etc/pve/qemu-server/$vmid.cpuset)"

    taskset --cpu-list  --all-tasks --pid "$cpuset" "$main_pid"
elif [[ "$phase" == "post-stop" ]]; then
    echo "Post-stop VM $vmid"
    #reboot
    #shutdown -h now

    sleep 1 
    echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/bind
    echo 1 > /sys/class/vtconsole/vtcon0/bind
    echo 1 > /sys/class/vtconsole/vtcon1/bind
fi
vmhook.sh (END)

 

 

 

Alternatively, you can get a cheap GPU for the host and not worry about the framebuffer unbind/rebind in the hook and manipulate the display output selector.

Edited November 30, 2020 by meina222

[23d1]

Hey guys,

 

Been working a lot lately, so haven't had time to mess around for the past couple of months, but getting back into it yesterday, I updated my Proxmox install (just the classic apt update stuff, which shouldn't break anything as it's all the stable Debian branch of Linux anyway). For some reason, when trying to start any VM with PCIe passthrough, I get an error saying "Cannot find vfio-pci module", and so it will not start. I pass through all my GPUs to VMs, and it was working great until I booted up and updated the system. Very strange.

 

Anyone run into this?

[dtek]

On 11/30/2020 at 12:49 AM, meina222 said:

I re-enabled the re-binding of the framebuffer and vtconsole - the highlighted commands were previously commented out - this is needed since otherwise you can't unbind in the pre-start phase when you restart.

 

When you shutdown your VM, you'd need an external device to restart it as main display won't come back to host (still not sure how to achieve that) but you can re-start VM from web console on a different device and display should come back. Similarly, rebooting a VM should just work and GPU should reset properly (for me DisplayPort sound stops working but this is WIP by developer).

 

#!/bin/bash

vmid="$1"
phase="$2"

if [[ "$phase" == "pre-start" ]]; then
    clear
    echo "Starting VM $vmid - please wait..."
    IFILE=/var/lib/vz/snippets/interfaces.$vmid
    if [[ -f "$IFILE" ]]; then
        cp /var/lib/vz/snippets/interfaces.$vmid /etc/network/interfaces
        systemctl restart networking
    fi          
    echo 0 > /sys/class/vtconsole/vtcon0/bind
    echo 0 > /sys/class/vtconsole/vtcon1/bind
    echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind
elif [[ "$phase" == "post-start" ]]; then
    main_pid="$(< /run/qemu-server/$vmid.pid)"

    cpuset="0-63"
    #cpuset="$(< /etc/pve/qemu-server/$vmid.cpuset)"

    taskset --cpu-list  --all-tasks --pid "$cpuset" "$main_pid"
elif [[ "$phase" == "post-stop" ]]; then
    echo "Post-stop VM $vmid"
    #reboot
    #shutdown -h now

    sleep 1 
    echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/bind
    echo 1 > /sys/class/vtconsole/vtcon0/bind
    echo 1 > /sys/class/vtconsole/vtcon1/bind
fi
vmhook.sh (END)

 

 

 

Alternatively, you can get a cheap GPU for the host and not worry about the framebuffer unbind/rebind in the hook and manipulate the display output selector.

After a  successful  GPU passthrough attempt,  my machine is acting weird.  It won't run both VMs simultaneously.   When I start one VM, the other would crash.  Have you experience this before? 

Edited December 2, 2020 by dtek

[dtek]

On 11/27/2020 at 4:13 PM, meina222 said:

On your host (Proxmox) try:

 

git clone https://github.com/gnif/vendor-reset.git
cd vendor-reset
dkms install .
echo "vendor-reset" >> /etc/modules
update-initramfs -u

reboot

 

If all works well, your GPU should start resetting.

 

You may need to install the dkms package if not there by default.

 

 

root@dtk:~/vendor-reset# dkms install .

Creating symlink /var/lib/dkms/vendor-reset/0.0.18/source ->
                 /usr/src/vendor-reset-0.0.18

DKMS: add completed.
Error! Your kernel headers for kernel 5.4.34-1-pve cannot be found.
Please install the linux-headers-5.4.34-1-pve package,
or use the --kernelsourcedir option to tell DKMS where it's located

 

I got this error when installing dkms
 

[meina222]

hi @23d1 - can you "update-initramfs -u" and reboot and try? This I think is a Proxmox script error message that indicates that your VFIO kernel module is not loaded properly.

 

@dtekyou need to install the kernel headers 1st. I believe I already had them as I've done kernel builds on the host.

 

apt install pve-headers

Edited December 2, 2020 by meina222

[meina222]

I don't usually run 2 VMs at the same time as I have 1 GPU only and always passthru, but I can test later and let you know.

Btw, Proxmox kernel is available all the way to 5.4.78. It's better to update to latest stable. Also, Proxmox 6.3 is out.

Edited December 2, 2020 by meina222

[dtek]

17 minutes ago, meina222 said:

hi @23d1 - can you "update-initramfs -u" and reboot and try? This I think is a Proxmox script error message that indicates that your VFIO kernel module is not loaded properly.

 

@dtekyou need to install the kernel headers 1st. I believe I already had them as I've done kernel builds on the host.

 

apt install pve-headers

Now I got this error after installing pve-headers

 

root@dtk:~/vendor-reset# dkms install .
Error! DKMS tree already contains: vendor-reset-0.0.18
You cannot add the same module/version combo more than once.
 

[meina222]

It's already installed. You can try  "dkms uninstall vendor-reset-0.0.18"

 

and reinstall it to be sure.

[dtek]

3 minutes ago, meina222 said:

It's already installed. You can try  "dkms uninstall vendor-reset-0.0.18"

 

and reinstall it to be sure.

root@dtk:~# dkms uninstall vendor-reset-0.0.18
Error! Invalid number of arguments passed.
Usage: uninstall <module>/<module-version> or
       uninstall -m <module>/<module-version> or
       uninstall -m <module> -v <module-version>

I tried dkms -m uninstall vendor-reset-0.0.18 and dkms -m uninstall vendor-reset -v 0.0.18, both have the same error

[meina222]

Try "dkms status"

 

dkms uninstall vendor-reset/0.0.18

[dtek]

4 minutes ago, meina222 said:

Try "dkms status"

 

dkms uninstall vendor-reset/0.0.18

root@dtk:~# dkms status
vendor-reset, 0.0.18: added

root@dtk:~# dkms uninstall vendor-reset/0.0.18
Error! The module vendor-reset 0.0.18 is not currently installed.
This module is not currently ACTIVE for kernel 5.4.78-1-pve (x86_64).

 

 

This command worked

 dkms remove vendor-reset/0.0.18 --all

Edited December 2, 2020 by dtek

[meina222]

How about "dkms remove -m vendor-reset -v 0.0.18 --all" ?

[dtek]

7 minutes ago, meina222 said:

Try "dkms status"

 

dkms uninstall vendor-reset/0.0.18

root@dtk:~/vendor-reset# update-initramfs -u
update-initramfs: Generating /boot/initrd.img-5.4.78-1-pve
Running hook script 'zz-pve-efiboot'..
Re-executing '/etc/kernel/postinst.d/zz-pve-efiboot' in new private mount namespace..
No /etc/kernel/pve-efiboot-uuids found, skipping ESP sync.
 

Is this normal?

[meina222]

Yeah, it seems just an info and not an error. Not sure what it means but I wouldn't sweat it.

 

I don't think you need to remove it but try " "dkms remove -m vendor-reset -v 0.0.18 --all"  

 

Then try installing again and update-initramfs -u

 

reboot

Edited December 2, 2020 by meina222

[dtek]

1 hour ago, meina222 said:

Yeah, it seems just an info and not an error. Not sure what it means but I wouldn't sweat it.

 

I don't think you need to remove it but try " "dkms remove -m vendor-reset -v 0.0.18 --all"  

 

Then try installing again and update-initramfs -u

 

reboot

Reinstalled and rebooted.  I forgot to tell you I bought a 6800XT and it's currently installed on my machine.  The reset bug seems to be gone and a passthrough was a success.  There's no support for this GPU yet, so there's no video acceleration.  The only issue it has now is it can't run both Windows 10 and Big Sur simultaneously.  I'm only able to test with one GPU at the moment due to my water cooling setup  allowing space for one GPU(6800xt).  I'm hoping once both GPUs are successfully passthrough,  then both VMs can run without crashing.

Edited December 3, 2020 by dtek

[meina222]

Where did you get that 6800XT? I wish I can find one without being gouged.

 

I just landed an Asus 3090 Strix OC from a retailer at $1899 (only 100 above MSRP). I got very lucky as this is not Newegg or Best Buy and I don't think it's targeted by bots and I snagged the only one in stock. I really wanted a 6900XT but I doubt I can get it given how bad the 6800XT launch is and that only 4% of all 6000 cores are for 6900XT bins. Will get the Asus in a few days. Don't know what to do with it, I think I will start ML practice.

The multi VM issue could be some problem with AMD PCIE arbitration. I vaguely remember that Linus Sebstian had an issue in one of his "1 PC, 2 gaming VMs" YouTube videos where he tried simultaneous 1v1 w 2 cards and AMD gave me issues and he switched to Intel. I only use 1 VM at time right now but with the 3090 that can change. We'll see.

Edited December 3, 2020 by meina222

[dtek]

47 minutes ago, meina222 said:

Where did you get that 6800XT? I wish I can find one without being gouged.

 

I just landed an Asus 3090 Strix OC from a retailer at $1899 (only 100 above MSRP). I got very lucky as this is not Newegg or Best Buy and I don't think it's targeted by bots and I snagged the only one in stock. I really wanted a 6900XT but I doubt I can get it given how bad the 6800XT launch is and that only 4% of all 6000 cores are for 6900XT bins. Will get the Asus in a few days. Don't know what to do with it, I think I will start ML practice.

The multi VM issue could be some problem with AMD PCIE arbitration. I vaguely remember that Linus Sebstian had an issue in one of his "1 PC, 2 gaming VMs" YouTube videos where he tried simultaneous 1v1 w 2 cards and AMD gave me issues and he switched to Intel. I only use 1 VM at time right now but with the 3090 that can change. We'll see.

Wow that's super lucky.   I'd just sell the 3090 and use that profit to pay for the 6900 markup price.  I got mine on ebay for $300 above MSRP =(

[meina222]

I would if I can find the 6900 at MSRP. I think it will be 2K at least by scalpers - not willing to pay that much. I think I could resell the 3090 for 2400 based on how much these Asus boards go for on ebay (they are top overclockers) but I don't want to be a scalper. Let's see if I even get it as I got the confirmation but it's another story if it will ship. I found it on TigerDirect - saw at least a couple of 3090s in stock there the past week and they were in stock for hours - both EVGA models. Worthwhile checking this site every few hours. Fingers crossed - feel like I won the lottery lol.

 

Edited December 3, 2020 by meina222

[23d1]

@meina222

 

I have indeed tried updating the intramfs and triple checked BIOS settings and everything is as it was, just simply stopped working. Very very strange.

[23d1]

On 12/3/2020 at 12:17 PM, 23d1 said:

@meina222

 

I have indeed tried updating the intramfs and triple checked BIOS settings and everything is as it was, just simply stopped working. Very very strange.

 

@meina222 

Quick update; I had to move the contents of the /etc/modules file to /etc/intramfs-tools/modules in order to get it back to working. However, I have the remaining issue that the macOS vm resets randomly to the OpenCore bootloader. This happens both in Proxmox and when launching in ArchLinux. Super frustrating. Thinking it might be the OpenCore bootloader itself, so I'm investigating. Any idea, or ever run into this issue?

[meina222]

@23d1,

 

No issues here. I run 4 VMs - 2 MacOS, 1 Windows 10 and 1 Linux. Had a problem w GPU reset on 5700XT but now this is largely solved (with the exception of DisplayPort). I did multiple Proxmox kernel upgrades. I find 5.4.xx series to be the most performant for my VMs (compared to 5.7/5.8/5.9) by a significant margin so I'll stay on it (latest 5.4.78 kernel and Proxmox 6.3) until the next LTS.

 

What changes did you make between the time you had your VMs working fine and now?

[23d1]

18 hours ago, meina222 said:

@23d1,

 

No issues here. I run 4 VMs - 2 MacOS, 1 Windows 10 and 1 Linux. Had a problem w GPU reset on 5700XT but now this is largely solved (with the exception of DisplayPort). I did multiple Proxmox kernel upgrades. I find 5.4.xx series to be the most performant for my VMs (compared to 5.7/5.8/5.9) by a significant margin so I'll stay on it (latest 5.4.78 kernel and Proxmox 6.3) until the next LTS.

 

What changes did you make between the time you had your VMs working fine and now?

 

I updated some packages, is all. And the latest dist upgrade as well. I have since tried a fresh install and so on. I got everything working again, but can't get past the reset to OpenCore bug for some reason. I wonder if I need to load some kind of NVMe kext or anything else that might be causing this issue. Any chance you could share your EFI for macOS (I run High Sierra until I can get my hands on an AMD card, preferably the 6800xt so I can bump up to whichever version of Big Sur supports that)?

[meina222]

2 hours ago, 23d1 said:

 

I updated some packages, is all. And the latest dist upgrade as well. I have since tried a fresh install and so on. I got everything working again, but can't get past the reset to OpenCore bug for some reason. I wonder if I need to load some kind of NVMe kext or anything else that might be causing this issue. Any chance you could share your EFI for macOS (I run High Sierra until I can get my hands on an AMD card, preferably the 6800xt so I can bump up to whichever version of Big Sur supports that)?

 

Here is my Big Sur OC 0.6.4 VM EFI

 

I have removed the  "Resources" sub-folder which takes a lot of space and you can just grab from the OC release sample folder. For the rest you need serials and your Mac address if you plan to use iMessage.

 

p.s.

I commented out 1st 3 PCI devices in DeviceProperties as they may differ from yours.

 

The VM config file corresponding to this is:

 

args: -device isa-applesmc,osk="ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc" -smbios type=2 -device usb-kbd,bus=ehci.0,port=2 -cpu host,+invtsc,vendor=GenuineIntel
balloon: 0
bios: ovmf
boot: cdn
bootdisk: virtio0
cores: 64
cpu: Penryn
efidisk0: aorus:vm-101-disk-1,size=1M
hookscript: local:snippets/vmhook.sh
hostpci0: 43:00,pcie=1,x-vga=1,romfile=vbios.bin
hostpci1: 86:00,pcie=1
hostpci2: 85:00,pcie=1
hostpci3: 88:00,pcie=1
hostpci4: 02:00,pcie=1
hugepages: 1024
ide2: local:iso/OpenCoreBeta.iso,size=150M
machine: q35
memory: 196608
name: bigsur
numa: 1
ostype: other
scsihw: virtio-scsi-pci
smbios1: uuid=4b5493a6-6a73-48b7-8ce5-2be70a66a383
sockets: 1
vga: none
virtio0: aorus:vm-101-disk-0,cache=unsafe,discard=on,size=250G
vmgenid: 18d68c27-3a62-4059-9280-7f86a572af59

 

 

EFI.zip

Edited December 9, 2020 by meina222