Proxmox VE OSX Guide discussion

[fabiosun]

Only to say that @iGPU is the king in this subject and he is studying how to have also usb working on this card

[cj750]

13 hours ago, meina222 said:

@cj750,

 

If you're asking about Thunderbolt in MacOS VM - I'm not aware of anyone having passed through the TB card setup. You might be able to passthrough Thunderbolt leaf devices, but I have not tried that yet. On the other hand, you should be able to make Thunderbolt work in bare metal. Please, contact @iGPU and @fabiosun for help on that.

 

Not sure when you mean make it work thru bare metal. Mostly eveything working. I have the back UBC C port working thru a UCB C hub. But my front top is not working with a Thunbder bolt adapter. I even tried my startec hub that has thunberbold and proxmox can see the device but can see it when the mac boots. I cant get a card because the MB does not have the connection for TB even thou they said it would come with the MB when released but never did. 

[meina222]

What I mean by 'works' is as follows - as far I understand, TB is a PCI extension device. USB-C and TB are different. So when you plugin a PCI TB controller card in your motherboard PCIE slot, and you get TB ports you use PCI lanes and can attain higher transfer rates. What I mean by "works" in bare metal is that with appropriate firmware, MacOS can recognize Titan and Alpine ridge based controllers and you can plug in your TB devices and get the max data rate (40gb/s in TB 3.0) and if lucky have hot plug work as well. In order for your Startec TB hub to work, you probably need to plug in a TB port connected to your PCI lanes, not in USB-C. You can get a card, and @iGPU can help you on how to bypass the TB header, but even then you may need custom firmware and SSDT. @iGPU has had some success with Titan Ridge based controllers. I have my Titan Ridge ver 1.0 card working too (came with my Gigabyte Designare TRX40 board), and I do have header, but that header is not needed for PCI functionality - but you need to fool the controller into not needing a header by jumping the pins.

 

You posted in Proxmox section - and what I tried to explain is that no-one knows how to have VM see the Thunderbolt controller and PCI hub, by passing it thru to the VM. Your Startec might be recognized as a USB device but not as TB one, however you may have some luck passing an eGPU or external TB SSD as if is PCI already by having Proxmox itself (and not MacOS) arbitrate the TB protocol. This should work in theory. For that you only need a Linux supported TB card and no custom firmware - you still need to jump the pins to bypass the header.

 

p.s.

The pin jumping trick is something I have never done but people do it and claim to have result. Best to PM iGPU. People use soldering, I think there could be jumpers you can buy, but you need to figure this on your own.

 

https://egpu.io/forums/builds/thunderbolt-3-with-threadripper-on-a-asrock-x399-fatality-motherboard-help-needed/

Edited October 14, 2020 by meina222

[cj750]

4 hours ago, meina222 said:

What I mean by 'works' is as follows - as far I understand, TB is a PCI extension device. USB-C and TB are different. So when you plugin a PCI TB controller card in your motherboard PCIE slot, and you get TB ports you use PCI lanes and can attain higher transfer rates. What I mean by "works" in bare metal is that with appropriate firmware, MacOS can recognize Titan and Alpine ridge based controllers and you can plug in your TB devices and get the max data rate (40gb/s in TB 3.0) and if lucky have hot plug work as well. In order for your Startec TB hub to work, you probably need to plug in a TB port connected to your PCI lanes, not in USB-C. You can get a card, and @iGPU can help you on how to bypass the TB header, but even then you may need custom firmware and SSDT. @iGPU has had some success with Titan Ridge based controllers. I have my Titan Ridge ver 1.0 card working too (came with my Gigabyte Designare TRX40 board), and I do have header, but that header is not needed for PCI functionality - but you need to fool the controller into not needing a header by jumping the pins.

 

You posted in Proxmox section - and what I tried to explain is that no-one knows how to have VM see the Thunderbolt controller and PCI hub, by passing it thru to the VM. Your Startec might be recognized as a USB device but not as TB one, however you may have some luck passing an eGPU or external TB SSD as if is PCI already by having Proxmox itself (and not MacOS) arbitrate the TB protocol. This should work in theory. For that you only need a Linux supported TB card and no custom firmware - you still need to jump the pins to bypass the header.

 

p.s.

The pin jumping trick is something I have never done but people do it and claim to have result. Best to PM iGPU. People use soldering, I think there could be jumpers you can buy, but you need to figure this on your own.

 

https://egpu.io/forums/builds/thunderbolt-3-with-threadripper-on-a-asrock-x399-fatality-motherboard-help-needed/

Thank you so much for explaning it. Ok I wil do a little more research. Thank you again. I will reach out to @iGPU and see what they say. 

[dtek]

On 4/4/2020 at 8:31 AM, fabiosun said:

 

In this thread we can talk about any subject related to AMD 3970x Proxmox VE guide published here:

 

 Hi,

 

I was able to get Opencore and Catalina installed on proxmox thanks to your guide.  I attempted a passthrough for my 5700 xt GPU but failed.  Now every time I boot my machine,  I can't see the ip and login:  like how it used to display before the  attempted passthrough.   Proxmox web ui and ssh still works but rebooting just hangs.  The power needs to be turned off and back on, otherwise  there's no access to the web ui.  All the settings were reverted but no luck with getting the old login screen back.  This is what I see every time I boot proxmox

Edited November 3, 2020 by dtek

[fabiosun]

I think this could be the famigerate AMD reset bug and it could be unresolved by now

if you follow @Imanol steps you could solve, but always system could hang if you reboot it

last user i know active here to use proxmox with an AMD gpu is @meina222

maybe it could share his experience with you

i have now an old AMD card but i have not tested in proxmox for now

sorry

[dtek]

12 hours ago, fabiosun said:

I think this could be the famigerate AMD reset bug and it could be unresolved by now

if you follow @Imanol steps you could solve, but always system could hang if you reboot it

last user i know active here to use proxmox with an AMD gpu is @meina222

maybe it could share his experience with you

i have now an old AMD card but i have not tested in proxmox for now

sorry

 

I got my login screen to show again so I'm giving the passthrough another shot.  I'm confused with this part of the guide.

 

Create a file in etc/modprobe.d/ named vfio.conf:

echo "options vfio-pci ids=10de:1b02,10de:10ef" > /etc/modprobe.d/vfio.conf and add on it:

 

options vfio-pci ids=10de:1b02,10de:10ef disable_vga=1

 

My vfio.conf should display both of these lines below?

options vfio-pci ids=10de:1b02,10de:10ef 

options vfio-pci ids=10de:1b02,10de:10ef disable_vga=1

[fabiosun]

Today I have tested on proxmox for the first time with my new amd Vega 64 gpu @dtek

I have had only changed my gpu address from 21:00 to 23:00 and my usb controller from 23:00 .3/4 to 25:00.3/4

end

system has booted fine with proper gpu acceleration without touching nothing else

same efi I have used before

 

all this text lines to say tha t our motherboard does not need any vfio stuff and blocking things..

If you want to use vfio as in your message above you have to put inside correct ids Above seems from a nvidia card and not of your amd

try to te read and concentrate about gpu rom declaration if your gpu has no uefi bios.. and stay more simple you can about Linux configuration part, msi trx40 pro needs few adjustment also in gpu passthrough

 

[dtek]

25 minutes ago, fabiosun said:

Today I have tested on proxmox for the first time with my new amd Vega 64 gpu @dtek

I have had only changed my gpu address from 21:00 to 23:00 and my usb controller from 23:00 .3/4 to 25:00.3/4

end

system has booted fine with proper gpu acceleration without touching nothing else

some efi I have used before

 

all this text lines to say tha t our motherboard does not need any vfio stuff and blocking things..

If you want to use vfio as in your message above you have to put inside correct ids Above seems from a nvidia card and not of your amd

try to te read and concentrate about gpu rom declaration if your gpu has no uefi bios.. and stay more simple you can about Linux configuration part, msi trx40 pro needs few adjustment also in gpu passthrough

 

I changed the address to 05:00 but when adding  pci device, my GPU doesn't show up on the list so I added manually in vm.conf.  Also downloaded the rom for the 5700 and renamed it to 5700xtref.rom but I got the spinner icon when starting my vm.

Edited November 3, 2020 by dtek

[fabiosun]

@dtek
Have you loaded rom file in the proper location?

in vm config have you put display to none?

 

[dtek]

3 minutes ago, fabiosun said:

@dtek
Have you loaded rom file in the proper location?

in vm config have you put display to none?

 

The rom file is in /usr/share/kvm.  How do I change display to none?  My vm.conf looks like this

 

args: -device isa-applesmc,osk="ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc" -smbios type=2 -device usb-kbd,bus=ehci.0,port=2 -cpu host,+invtsc,vendor=GenuineIntel
hostpci1: 05:00,pcie=1,x-vga=1,romfile=5700xtref.rom
balloon: 0
bios: ovmf
bootdisk: ide2
cores: 32
cpu: Penryn
efidisk0: local-lvm:vm-101-disk-1,size=4M
ide0: local:iso/Catalina-installer.iso,cache=unsafe,size=2096528K
ide2: local:iso/OpenCore-v9.iso,cache=unsafe,size=150M
machine: q35
memory: 131072
name: macOs
net0: vmxnet3=5E:57:E7:CD:64:BC,bridge=vmbr0,firewall=1
numa: 0
ostype: other
sata0: local-lvm:vm-101-disk-0,cache=unsafe,discard=on,size=300G,ssd=1
scsihw: virtio-scsi-pci
smbios1: uuid=1754abd0-bbbc-4e0f-969b-a9d87ca43e14
sockets: 1
vga: vmware
vmgenid: c8335d64-7052-44fa-a1f2-4f46432c56f7
 

[fabiosun]

Sorry

vga: none

You are using vmware

it is not good for your  passthrough task

[fabiosun]

I am on my mobile but you have many not correct parameter in your vm

1)boot arg 

2) number of cores 64 it is fine for 3970x

3)if you have 128gb you have to use less ram in vm config

 

[dtek]

14 minutes ago, fabiosun said:

I am on my mobile but you have many not correct parameter in your vm

1)boot arg 

2) number of cores 64 it is fine for 3970x

3)if you have 128gb you have to use less ram in vm config

The other 32 cores are reserved for windows.  I updated my profile to 256gb ram but I don't know why it doesn't show.  This was my original boot arg but I copied @meina222 boot arg since he said the new one has better performance.  Should I revert boot arg back to the original one?  updated vga: none with the same result

 

 -device isa-applesmc,osk="ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc" -smbios type=2 -device usb-kbd,bus=ehci.0,port=2 -cpu Penryn,kvm=on,vendor=GenuineIntel,+kvm_pv_unhalt,+kvm_pv_eoi,+hypervisor,+invtsc,+pcid,+ssse3,+sse4.2,+popcnt,+avx,+avx2,+aes,+fma,+fma4,+bmi1,+bmi2,+xsave,+xsaveopt,check

Edited November 3, 2020 by dtek

[fabiosun]

Tomorrow I will upload my config

however if you use passthrough of a single gpu you can assign all the resource of your system to a vm

you can’t run 2 vm in this case in the same time

i think problem is on Linux part

also you can use host as cpu and a minimal boot args in vm

it also depends to kernel patches you use in your efi config.plist

if I do not misunderstood you said system works if you do not passthrough gpu?

isn’t it?

 

[dtek]

32 minutes ago, fabiosun said:

Tomorrow I will upload my config

however if you use passthrough of a single gpu you can assign all the resource of your system to a vm

you can’t run 2 vm in this case in the same time

i think problem is on Linux part

also you can use host as cpu and a minimal boot args in vm

it also depends to kernel patches you use in your efi config.plist

if I do not misunderstood you said system works if you do not passthrough gpu?

isn’t it?

 

Ok looking forward to your config.  I kinda need 2 vms running at the same time so that will not work for me.  The system works 100% without gpu passthrough.  Also there's one part in the guide that I skipped since I don't know how to change "RadeonDeinit" setting in opencore.  My GPU is currently in slot 2,  I'll try to move it to slot 1 and see if that makes a difference.

 

 

32 minutes ago, fabiosun said:

 

 

[meina222]

Hi @dtek,

 

As @fabiosun mentioned, I am an active Proxmox user and am typing this from my Proxmox VM, so I hope this helps. I suppose you're trying to run a single GPU as I see you try to shadow the vbios. Here's one of my MacOS VM configs where I pass my 5700XT

 

args: -device isa-applesmc,osk="ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc" -smbios type=2 -device usb-kbd,bus=ehci.0,port=2 -cpu host,+invtsc,vendor=GenuineIntel
balloon: 0
bios: ovmf
boot: cdn
bootdisk: virtio0
cores: 64
cpu: Penryn
efidisk0: aorus:vm-101-disk-1,size=1M
hookscript: local:snippets/vmhook.sh
hostpci0: 43:00,pcie=1,x-vga=1,romfile=vbios.bin
hostpci1: 86:00,pcie=1
hostpci2: 85:00,pcie=1
hostpci3: 88:00,pcie=1
hostpci4: 02:00,pcie=1
hugepages: 1024
ide2: local:iso/OpenCoreBeta.iso,size=150M
machine: q35
memory: 196608
name: bigsur
numa: 1
ostype: other
scsihw: virtio-scsi-pci
smbios1: uuid=4b5493a6-6a73-48b7-8ce5-2be70a66a383
sockets: 1
vga: none
virtio0: aorus:vm-101-disk-0,cache=unsafe,discard=on,size=250G
vmgenid: 18d68c27-3a62-4059-9280-7f86a572af59
vmgenid: 0c7cc702-74ba-4d8e-ba4b-d52d5fe53847

 

 

As you probably know, you need to make sure your vbios is matching your card, and is located in  /usr/share/kvm/ (in my case it is called /usr/share/kvm/vbios.bin).

 

Also, in the vmhook.sh I run (located in /var/lib/vz/snippets/ ) you need to unbind the framebuffer. I do some other stuff as well (can elaborate), but here's the full hook with highlighted relevant section. Note that I also reboot my host if I shutdown the machine - this is sadly needed due to the reset bug and hopefully will be resolved with the new AMD 6000 cards so I can disable this section - otherwise your GPU will not come back most of the time and you'd be left with a blank screen - hence I comment out re-enabling the framebuffer.

 

#!/bin/bash

vmid="$1"
phase="$2"

if [[ "$phase" == "pre-start" ]]; then
    clear
    echo "Starting VM $vmid - please wait..."
    IFILE=/var/lib/vz/snippets/interfaces.$vmid
    if [[ -f "$IFILE" ]]; then
        cp /var/lib/vz/snippets/interfaces.$vmid /etc/network/interfaces
        systemctl restart networking
    fi          
    echo 0 > /sys/class/vtconsole/vtcon0/bind
    echo 0 > /sys/class/vtconsole/vtcon1/bind
    echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind
elif [[ "$phase" == "post-start" ]]; then
    main_pid="$(< /run/qemu-server/$vmid.pid)"

    cpuset="0-63"
    #cpuset="$(< /etc/pve/qemu-server/$vmid.cpuset)"

    taskset --cpu-list  --all-tasks --pid "$cpuset" "$main_pid"
elif [[ "$phase" == "post-stop" ]]; then
    echo "Post-stop VM $vmid"
    reboot
    #shutdown -h now

    #sleep 5
    # Attempt rebind to EFI-Framebuffer
    #echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/bind
    # Attempt rebind to virtual consoles
    #echo 1 > /sys/class/vtconsole/vtcon0/bind
    #echo 1 > /sys/class/vtconsole/vtcon1/bind
    #sleep 5
    #shutdown -h now
fi

 

 

Edited November 3, 2020 by meina222

[dtek]

1 hour ago, meina222 said:

Hi @dtek,

 

As @fabiosun mentioned, I am an active Proxmox user and am typing this from my Proxmox VM, so I hope this helps. I suppose you're trying to run a single GPU as I see you try to shadow the vbios. Here's one of my MacOS VM configs where I pass my 5700XT

 

args: -device isa-applesmc,osk="ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc" -smbios type=2 -device usb-kbd,bus=ehci.0,port=2 -cpu host,+invtsc,vendor=GenuineIntel
balloon: 0
bios: ovmf
boot: cdn
bootdisk: virtio0
cores: 64
cpu: Penryn
efidisk0: aorus:vm-101-disk-1,size=1M
hookscript: local:snippets/vmhook.sh
hostpci0: 43:00,pcie=1,x-vga=1,romfile=vbios.bin
hostpci1: 86:00,pcie=1
hostpci2: 85:00,pcie=1
hostpci3: 88:00,pcie=1
hostpci4: 02:00,pcie=1
hugepages: 1024
ide2: local:iso/OpenCoreBeta.iso,size=150M
machine: q35
memory: 196608
name: bigsur
numa: 1
ostype: other
scsihw: virtio-scsi-pci
smbios1: uuid=4b5493a6-6a73-48b7-8ce5-2be70a66a383
sockets: 1
vga: none
virtio0: aorus:vm-101-disk-0,cache=unsafe,discard=on,size=250G
vmgenid: 18d68c27-3a62-4059-9280-7f86a572af59
vmgenid: 0c7cc702-74ba-4d8e-ba4b-d52d5fe53847

 

 

As you probably know, you need to make sure your vbios is matching your card, and is located in  /usr/share/kvm/ (in my case it is called /usr/share/kvm/vbios.bin).

 

Also, in the vmhook.sh I run (located in /var/lib/vz/snippets/ ) you need to unbind the framebuffer. I do some other stuff as well (can elaborate), but here's the full hook with highlighted relevant section. Note that I also reboot my host if I shutdown the machine - this is sadly needed due to the reset bug and hopefully will be resolved with the new AMD 6000 cards so I can disable this section - otherwise your GPU will not come back most of the time and you'd be left with a blank screen - hence I comment our re-enabling the framebuffer.

 

#!/bin/bash

vmid="$1"
phase="$2"

if [[ "$phase" == "pre-start" ]]; then
    clear
    echo "Starting VM $vmid - please wait..."
    IFILE=/var/lib/vz/snippets/interfaces.$vmid
    if [[ -f "$IFILE" ]]; then
        cp /var/lib/vz/snippets/interfaces.$vmid /etc/network/interfaces
        systemctl restart networking
    fi          
    echo 0 > /sys/class/vtconsole/vtcon0/bind
    echo 0 > /sys/class/vtconsole/vtcon1/bind
    echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind
elif [[ "$phase" == "post-start" ]]; then
    main_pid="$(< /run/qemu-server/$vmid.pid)"

    cpuset="0-63"
    #cpuset="$(< /etc/pve/qemu-server/$vmid.cpuset)"

    taskset --cpu-list  --all-tasks --pid "$cpuset" "$main_pid"
elif [[ "$phase" == "post-stop" ]]; then
    echo "Post-stop VM $vmid"
    reboot
    #shutdown -h now

    #sleep 5
    # Attempt rebind to EFI-Framebuffer
    #echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/bind
    # Attempt rebind to virtual consoles
    #echo 1 > /sys/class/vtconsole/vtcon0/bind
    #echo 1 > /sys/class/vtconsole/vtcon1/bind
    #sleep 5
    #shutdown -h now
fi

 

 

@meina222

 

/usr/share/kvm/vbios.bin doesn't exist.  Where can I download this file?  There's no snipplets folder in /var/lib/vz, should I create snipplets folder with vmhook.sh in it?

Edited November 3, 2020 by dtek

[dtek]

32 minutes ago, dtek said:

@meina222

 

/usr/share/kvm/vbios.bin doesn't exist.  Where can I download this file?  There's no snipplets folder in /var/lib/vz, should I create snipplets folder with vmhook.sh in it?

https://www.techpowerup.com/vgabios/212169/sapphire-rx5700xt-8192-190616

 

I found this but it's not in .bin format.

[meina222]

vbios.bin is just how I named the file. You can rename that.

[dtek]

33 minutes ago, meina222 said:

vbios.bin is just how I named the file. You can rename that.

I got this error

 

TERM environment variable not set.
Starting VM 101 - please wait...
/var/lib/vz/snippets/vmhook.sh: line 15: echo: write error: No such device
TASK ERROR: hookscript error for 101 on pre-start: command '/var/lib/vz/snippets/vmhook.sh 101 pre-start' failed: exit code 1

[meina222]

@dtek which version of Proxmox do you use? Yes, you can create snippets.

what devices do you have listed under

 

/sys/class/vtconsole/

[dtek]

1 minute ago, meina222 said:

@dtek which version of Proxmox do you use? Yes, you can create snippets.

 6.2-4

[dtek]

3 minutes ago, meina222 said:

@dtek which version of Proxmox do you use? Yes, you can create snippets.

My gpu is currently in slot 2.  Do I need to make any changes to the script or vm.conf ?

[meina222]

Yeah - you may. The script is just a sample. You need to check how your devices are named.

 

Could you list what you have under /sys/class/vtconsole