Proxmox VE OSX Guide discussion

[dtek]

2 minutes ago, meina222 said:

Yeah - you may. The script is just a sample. You need to check how your devices are named.

 

Could you list what you have under /sys/class/vtconsole

 

 

vtcon0  vtcon1

[meina222]

Also, make sure you blacklist your radeon driver under /etc/modprobe.d/

 

e.g. here is is described how

 

https://gist.github.com/SimonTod/72945457c4d7e741a382ba9ce9c4731b

[meina222]

Are you able to connect to the Proxmox host remotely? Can you open a console via the web interface, so when you disable the GPU, you can recover?

 

If so, can you try manually running each of the lines, and let me know which fails.  Do not copy my hook verbatim - I have stuff there that requires 128 CPUs, you have a 3970x. Just use the gpu section. But for now run manually:

 

 echo 0 > /sys/class/vtconsole/vtcon0/bind

 echo 0 > /sys/class/vtconsole/vtcon1/bind
 echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind

 

You can reboot from the web console to recover display or try to manually rebind.

 

Need to step away for 30 min. Will check back in a bit.

 

Edited November 4, 2020 by meina222

[dtek]

17 minutes ago, meina222 said:

Are you able to connect to the Proxmox host remotely? Can you open a console via the web interface, so when you disable the GPU, you can recover?

 

If so, can you try manually running each of the lines, and let me know which fails.  Do not copy my hook verbatim - I have stuff there that requires 128 CPUs, you have a 3970x. Just use the gpu section. But for now run manually:

 

 echo 0 > /sys/class/vtconsole/vtcon0/bind

 echo 0 > /sys/class/vtconsole/vtcon1/bind
 echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind

 

You can reboot from the web console to recover display or try to manually rebind.

 

Need to step away for 30 min. Will check back in a bit.

 

I can connect remotely and via console after blacklisting.  Rebooting proxmox requires cutting off the power  and restarting pc.

root@dtk:~# echo 0 > /sys/class/vtconsole/vtcon0/bind
root@dtk:~# echo 0 > /sys/class/vtconsole/vtcon1/bind
root@dtk:~# echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind
-bash: echo: write error: No such device

[dtek]

28 minutes ago, meina222 said:

Are you able to connect to the Proxmox host remotely? Can you open a console via the web interface, so when you disable the GPU, you can recover?

 

If so, can you try manually running each of the lines, and let me know which fails.  Do not copy my hook verbatim - I have stuff there that requires 128 CPUs, you have a 3970x. Just use the gpu section. But for now run manually:

 

 echo 0 > /sys/class/vtconsole/vtcon0/bind

 echo 0 > /sys/class/vtconsole/vtcon1/bind
 echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind

 

You can reboot from the web console to recover display or try to manually rebind.

 

Need to step away for 30 min. Will check back in a bit.

 

I got the spinner again with no log output when starting the vm

[meina222]

 

When you click on your host under 'Datacenter' in Proxmox web gui, you can go under 'System' in the tab and check the log of the Proxmox host under 'Syslog'.

 

But basic things 1st - your VM config is located in /etc/pve/qemu-server/  file will be called [vmid].conf where [vmid] is the name of your VM

 

Comment out the GPU passthru for now by putting a # in front of the hostpci line that has  your GPU id.

 

Also replace vga: none with vga: vmware

 

Are you able to start your VM then?

 

Hint: you can manually start the VM by "qm start [vmid]" or just use the web interface.

 

also disable the hook for now (comment with same # in front of line). We want to make sure your VM starts fine w/out GPU passthru.

Edited November 4, 2020 by meina222

[meina222]

Also could you share the output of 

 

less /etc/default/grub

 

Want to make sure you're not disabling efi framebuffer. Some guides online tell you to do so and that might explain the error you get.

[dtek]

7 minutes ago, meina222 said:

 

When you click on your host under 'Datacenter' in Proxmox web gui, you can go under 'System' in the tab and check the log of the Proxmox host under 'Syslog'.

 

But basic things 1st - your VM config is located in /etc/pve/qemu-server/  file will be called [vmid].conf where [vmid] is the name of your VM

 

Comment out the GPU passthru for now by putting a # in front of the hostpci line that has  your GPU id.

 

Also replace vga: none with vga: vmware

 

Are you able to start your VM then?

 

Hint: you can manually start the VM by "qm start [vmid]" or just use the web interface.

 

also disable the hook for now (comment with same # in front of line). We want to make sure your VM starts fine w/out GPU passthru.

I can start just fine without the GPU passthru

[dtek]

18 minutes ago, meina222 said:

Also could you share the output of 

 

less /etc/default/grub

 

Want to make sure you're not disabling efi framebuffer. Some guides online tell you to do so and that might explain the error you get.

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="Proxmox Virtual Environment"
GRUB_CMDLINE_LINUX_DEFAULT="quiet"
GRUB_CMDLINE_LINUX=""

# Disable os-prober, it might add menu entries for each guest
GRUB_DISABLE_OS_PROBER=true

# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

# Uncomment to disable graphical terminal (grub-pc only)
#GRUB_TERMINAL=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=640x480

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true

# Disable generation of recovery mode menu entries
GRUB_DISABLE_RECOVERY="true"

# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"

26 minutes ago, meina222 said:

 

When you click on your host under 'Datacenter' in Proxmox web gui, you can go under 'System' in the tab and check the log of the Proxmox host under 'Syslog'.

 

But basic things 1st - your VM config is located in /etc/pve/qemu-server/  file will be called [vmid].conf where [vmid] is the name of your VM

 

Comment out the GPU passthru for now by putting a # in front of the hostpci line that has  your GPU id.

 

Also replace vga: none with vga: vmware

 

Are you able to start your VM then?

 

Hint: you can manually start the VM by "qm start [vmid]" or just use the web interface.

 

also disable the hook for now (comment with same # in front of line). We want to make sure your VM starts fine w/out GPU passthru.

I couldn't find the bios for Sapphire Pulse 5700 XT so I extracted with this software  https://www.techpowerup.com/download/ati-atiflash/ .  Do you think that might prevent it from starting the vm?  I found this https://www.techpowerup.com/vgabios/212169/sapphire-rx5700xt-8192-190616 but it doesn't exactly match my GPU so I extracted my own.

Edited November 4, 2020 by dtek

[meina222]

Yeah - you need the real vbios - so if you can't find it you need to dump it (which is recommended anyways).

 

Retry with new file.

 

But I am more concerned with the inability to unbind the framebuffer - as if it is already unbound. If you can't successfully do that, the guest cannot take over the GPU.

Edited November 4, 2020 by meina222

[dtek]

4 minutes ago, meina222 said:

Yeah - you need the real vbios - so if you can't find it you need to dump it (which is recommended anyways).

 

Retry with new file.

 

But I am more concerned with the inability to unbind the framebuffer - as if it is already unbound. If you can't successfully do that, the guest cannot take over the GPU.

My GPU is in slot 2.  Is this the right setting for it?  Do I need to change hostpci0 to hostpci1 or hostpci2 ?

 

hostpci0: 05:00,pcie=1,x-vga=1,romfile=vbios.bin
 

 

[meina222]

As long as the id 05:00 is correct, and the vbios.bin file is the real rom and located in /usr/share/kvm/ then it should not matter what pci slot you specify. In one of my VMs I use hostpci3 even though my GPU is in Slot 1.

 

You need to be able to unload the framebuffer though (unless you have a 2nd GPU lying around).

Do you blacklist your radeon driver?

Edited November 4, 2020 by meina222

[meina222]

Here's my /etc/modprobe.d content

 

-rw-r--r--  1 root root  205 Jul 18 17:54 blacklist.conf
-rw-r--r--  1 root root   26 Jun 21 22:54 kvm.conf
-rw-r--r--  1 root root  171 May 10 15:06 pve-blacklist.conf
-rw-r--r--  1 root root  148 Jul  5 01:15 vfio.conf

 

less blacklist.conf

 

blacklist radeon
#blacklist nouveau
#blacklist nvidia
blacklist amdgpu
blacklist snd_hda_codec_hdmi
blacklist snd_hda_codec
blacklist snd_hda_core
blacklist snd_hda_intel
blacklist iwlwifi
blacklist btusb

 

less vfio.conf

 

options vfio-pci ids=1002:731f,1002:ab38 disable_vga=1
options vfio-pci ids=1022:148c
options vfio-pci ids=1022:149c
options vfio-pci ids=8086:1533

 

For vfio.conf, follow this guide as your id's will be different:

 

https://pve.proxmox.com/wiki/PCI(e)_Passthrough

My theory is that you don't disable the host taking over your GPU and cannot unbind the efi framebuffer as a result.

Edited November 4, 2020 by meina222

[meina222]

Here's my grub

 

less /etc/default/grub

 

 

# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
#   info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="Proxmox Virtual Environment"
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt default_hugepagesz=1G hugepagesz=1G hugepages=192"
GRUB_CMDLINE_LINUX="fbcon=rotate:0"

# Disable os-prober, it might add menu entries for each guest
GRUB_DISABLE_OS_PROBER=true

# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

# Uncomment to disable graphical terminal (grub-pc only)
#GRUB_TERMINAL=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=640x480

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true

# Disable generation of recovery mode menu entries
GRUB_DISABLE_RECOVERY="true"

# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"

 

 

 

Try updating your grub with 

GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt"

Edited November 4, 2020 by meina222

[dtek]

2 minutes ago, meina222 said:

Here's my /etc/modprobe.d content

 

-rw-r--r--  1 root root  205 Jul 18 17:54 blacklist.conf
-rw-r--r--  1 root root   26 Jun 21 22:54 kvm.conf
-rw-r--r--  1 root root  171 May 10 15:06 pve-blacklist.conf
-rw-r--r--  1 root root  148 Jul  5 01:15 vfio.conf

 

less blacklist.conf

 

blacklist radeon
#blacklist nouveau
#blacklist nvidia
blacklist amdgpu
blacklist snd_hda_codec_hdmi
blacklist snd_hda_codec
blacklist snd_hda_core
blacklist snd_hda_intel
blacklist iwlwifi
blacklist btusb

 

less vfio.conf

 

options vfio-pci ids=1002:731f,1002:ab38 disable_vga=1
options vfio-pci ids=1022:148c
options vfio-pci ids=1022:149c
options vfio-pci ids=8086:1533

 

For vfio.conf, follow this guide as your id's will be different:

 

https://pve.proxmox.com/wiki/PCI(e)_Passthrough

My theory is that you don't disable the host taking over your GPU and cannot unbind the efi framebuffer as a result.

root@dtk:/etc/modprobe.d# lspci | grep VGA
05:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 (rev c1)
 

root@dtk:/etc/modprobe.d# cat blacklist.conf
blacklist radeon
blacklist nouveau
blacklist nvidia
 

root@dtk:/etc/modprobe.d# cat kvm.conf
options kvm ignore_msrs=1
 

root@dtk:/etc/modprobe.d# cat vfio.conf
options vfio-pci ids=1002:731f,1002:ab38 disable_vga=1

 

root@dtk:/etc/modprobe.d# lspci -n -s 05:00
05:00.0 0300: 1002:731f (rev c1)
05:00.1 0403: 1002:ab38
 

 

Am I missing anything?

 

[meina222]

Not that I see yet - other than the GRUB cmd line I posted above. It's been a while since I put this and I don't recall the significance, but noticed the difference.

 

And throw in your blacklist

 

blacklist amdgpu 

 

for good measure.

Edited November 4, 2020 by meina222

[meina222]

Basically when you're all done and reboot you need to be able to succeed from your primary display to do this

 

echo 0 > /sys/class/vtconsole/vtcon0/bind  &  echo 0 > /sys/class/vtconsole/vtcon1/bind &  echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind

 

This will block your main display but then you can use the web console to start the VM and the card should switch over to the VM bios after 20-30 seconds and you should see the VM bringing your display back alive. When you verify this, you can use the hook again (but don't use other stuff in this hook as it is specific to core count and network setup).

 

Basically the way I operate is:

 

1. Start Proxmox

2. Login

3. run "qm start [vmid]" (whichever VM I like to start)

4. Screen blocks as the hook unbinds the buffer but comes back alive when the VM takes it over and I see the VM booting and GPU being taken over host.

5. Since AMD reset bug makes the GPU unclaimable by the host, you cannot restart the VM until you reboot the host, but this is a 2-ary issue.

Edited November 4, 2020 by meina222

[dtek]

2 minutes ago, meina222 said:

Basically when you're all done and reboot you need to be able to succeed from your primary display to do this

 

echo 0 > /sys/class/vtconsole/vtcon0/bind  &  echo 0 > /sys/class/vtconsole/vtcon1/bind &  echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind

 

This will block your main display but then you can use the web console to start the VM and the card should switch over to the VM bios after 20-30 seconds and you should see the VM bringing your display back alive.

Updated grub, blacklisted  amdgpu and   pve-efiboot-tool refresh

I think the only thing left to try is  swapping out the rom file and moving GPU to slot 1

[meina222]

Yes, you need the true rom file. I would not move the GPU yet. In fact the GPU being in slot 2 may be an advantage - try even w/out the rom file as I think shadowing the rom may only be required in slot 1.

 

Also if the VM fails to start - check the host log Syslog as I described above. You can  do it from the web GUI in realtime.

Edited November 4, 2020 by meina222

[dtek]

1 hour ago, meina222 said:

Yes, you need the true rom file. I would not move the GPU yet. In fact the GPU being in slot 2 may be an advantage - try even w/out the rom file as I think shadowing the rom may only be required in slot 1.

 

Also if the VM fails to start - check the host log Syslog as I described above. You can  do it from the web GUI in realtime.

I finally got it working.  I was sharing a monitor with an all in one pc  with hdmi input and output and that caused a lot of issues. Switched  to an old monitor and almost everything works except for the hookscript.   My host screen and console is blank right now.  I can only connect with teamviewer.  I made some changes to the hookscript.  Can you check to see if this will work?

 

#!/bin/bash
vmid="$1"
phase="$2"

if [[ "$phase" == "pre-start" ]]; then
    clear
    echo "Starting VM $vmid - please wait..."
    IFILE=/var/lib/vz/snippets/interfaces.$vmid
    if [[ -f "$IFILE" ]]; then
        cp /var/lib/vz/snippets/interfaces.$vmid /etc/network/interfaces
        systemctl restart networking
    fi
    echo 0 > /sys/class/vtconsole/vtcon0/bind
    echo 0 > /sys/class/vtconsole/vtcon1/bind
    echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind
elif [[ "$phase" == "post-start" ]]; then
    main_pid="$(< /run/qemu-server/$vmid.pid)"

   # cpuset="0-63"
    #cpuset="$(< /etc/pve/qemu-server/$vmid.cpuset)"

    #taskset --cpu-list  --all-tasks --pid "$cpuset" "$main_pid"
#elif [[ "$phase" == "post-stop" ]]; then
 #   echo "Post-stop VM $vmid"
    reboot
    #shutdown -h now

    #sleep 5
    # Attempt rebind to EFI-Framebuffer
    #echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/bind
    # Attempt rebind to virtual consoles
    #echo 1 > /sys/class/vtconsole/vtcon0/bind
    #echo 1 > /sys/class/vtconsole/vtcon1/bind
    #sleep 5
    #shutdown -h now
fi
 

Edited November 4, 2020 by dtek

[meina222]

For now you can remove everything from the hook script except these (even though vmid is not used it may come handy later):

 

The critical one is the frame buffer unbinding and then reboot is optional - you normally don't want to reboot but I do it as the GPU can't reset.

 

#!/bin/bash
vmid="$1"
phase="$2"

if [[ "$phase" == "pre-start" ]]; then
    echo 0 > /sys/class/vtconsole/vtcon0/bind
    echo 0 > /sys/class/vtconsole/vtcon1/bind
    echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind
#elif [[ "$phase" == "post-stop" ]]; then
    echo "Post-stop VM $vmid"    

    reboot
fi

[dtek]

2 hours ago, meina222 said:

For now you can remove everything from the hook script except these (even though vmid is not used it may come handy later):

 

The critical one is the frame buffer unbinding and then reboot is optional - you normally don't want to reboot but I do it as the GPU can't reset.

 

#!/bin/bash
vmid="$1"
phase="$2"

if [[ "$phase" == "pre-start" ]]; then
    echo 0 > /sys/class/vtconsole/vtcon0/bind
    echo 0 > /sys/class/vtconsole/vtcon1/bind
    echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind
#elif [[ "$phase" == "post-stop" ]]; then
    echo "Post-stop VM $vmid"    

    reboot
fi

I'm so happy everything works ☺️.  Thanks for that.  I'm experiencing the reset but now and it's quite annoying lol.  Since black friday is near,  I'll be on the hunt for a new GPU.  Is there any decent GPU that you can recommend without the reset bug?

[meina222]

Your best bet is to wait for the new Radeon 6000 and hope they fixed it there. There are some older AMD GPUs that reset but nothing newer than a few years at this point (except maybe a few RX 580 Sapphire Pulse models, but have only read anecdotes of that, not verified it and would not trust to buy based on that alone). New radeon 6000 will be out by end of Nov and we'll know very soon if it resets.

[dtek]

33 minutes ago, meina222 said:

Your best bet is to wait for the new Radeon 6000 and hope they fixed it there. There are some older AMD GPUs that reset but nothing newer than a few years at this point (except maybe a few RX 580 Sapphire Pulse models, but have only read anecdotes of that, not verified it and would not trust to buy based on that alone). New radeon 6000 will be out by end of Nov and we'll know very soon if it resets.

Ok I can wait another month.  I'm having some issues after the passthru.   Every time I start both Windows10 and Catalina, one or both will stop.  Any idea what the problem might be?

[dtek]

6 minutes ago, dtek said:

Ok I can wait another month.  I'm having some issues after the passthru.   Every time I start both Windows10 and Catalina, one or both will stop.  Any idea what the problem might be?

I figured out what the problem was.  I followed the other gpu passthrough guide and set a crontab on every reboot.  That has conflicts with both machines somehow,  removed it and all is good now.  Thanks again