dtek

4 minutes ago, meina222 said:

Yeah - you need the real vbios - so if you can't find it you need to dump it (which is recommended anyways).

 

Retry with new file.

 

But I am more concerned with the inability to unbind the framebuffer - as if it is already unbound. If you can't successfully do that, the guest cannot take over the GPU.

My GPU is in slot 2.  Is this the right setting for it?  Do I need to change hostpci0 to hostpci1 or hostpci2 ?

hostpci0: 05:00,pcie=1,x-vga=1,romfile=vbios.bin
 

18 minutes ago, meina222 said:

Also could you share the output of 

 

less /etc/default/grub

 

Want to make sure you're not disabling efi framebuffer. Some guides online tell you to do so and that might explain the error you get.

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="Proxmox Virtual Environment"
GRUB_CMDLINE_LINUX_DEFAULT="quiet"
GRUB_CMDLINE_LINUX=""

# Disable os-prober, it might add menu entries for each guest
GRUB_DISABLE_OS_PROBER=true

# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

# Uncomment to disable graphical terminal (grub-pc only)
#GRUB_TERMINAL=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=640x480

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true

# Disable generation of recovery mode menu entries
GRUB_DISABLE_RECOVERY="true"

# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"

26 minutes ago, meina222 said:

 

When you click on your host under 'Datacenter' in Proxmox web gui, you can go under 'System' in the tab and check the log of the Proxmox host under 'Syslog'.

 

But basic things 1st - your VM config is located in /etc/pve/qemu-server/  file will be called [vmid].conf where [vmid] is the name of your VM

 

Comment out the GPU passthru for now by putting a # in front of the hostpci line that has  your GPU id.

 

Also replace vga: none with vga: vmware

 

Are you able to start your VM then?

 

Hint: you can manually start the VM by "qm start [vmid]" or just use the web interface.

 

also disable the hook for now (comment with same # in front of line). We want to make sure your VM starts fine w/out GPU passthru.

I couldn't find the bios for Sapphire Pulse 5700 XT so I extracted with this software  https://www.techpowerup.com/download/ati-atiflash/ .  Do you think that might prevent it from starting the vm?  I found this https://www.techpowerup.com/vgabios/212169/sapphire-rx5700xt-8192-190616 but it doesn't exactly match my GPU so I extracted my own.

7 minutes ago, meina222 said:

 

When you click on your host under 'Datacenter' in Proxmox web gui, you can go under 'System' in the tab and check the log of the Proxmox host under 'Syslog'.

 

But basic things 1st - your VM config is located in /etc/pve/qemu-server/  file will be called [vmid].conf where [vmid] is the name of your VM

 

Comment out the GPU passthru for now by putting a # in front of the hostpci line that has  your GPU id.

 

Also replace vga: none with vga: vmware

 

Are you able to start your VM then?

 

Hint: you can manually start the VM by "qm start [vmid]" or just use the web interface.

 

also disable the hook for now (comment with same # in front of line). We want to make sure your VM starts fine w/out GPU passthru.

I can start just fine without the GPU passthru

28 minutes ago, meina222 said:

Are you able to connect to the Proxmox host remotely? Can you open a console via the web interface, so when you disable the GPU, you can recover?

 

If so, can you try manually running each of the lines, and let me know which fails.  Do not copy my hook verbatim - I have stuff there that requires 128 CPUs, you have a 3970x. Just use the gpu section. But for now run manually:

 

 echo 0 > /sys/class/vtconsole/vtcon0/bind

 echo 0 > /sys/class/vtconsole/vtcon1/bind
 echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind

 

You can reboot from the web console to recover display or try to manually rebind.

 

Need to step away for 30 min. Will check back in a bit.

 

I got the spinner again with no log output when starting the vm

17 minutes ago, meina222 said:

Are you able to connect to the Proxmox host remotely? Can you open a console via the web interface, so when you disable the GPU, you can recover?

 

If so, can you try manually running each of the lines, and let me know which fails.  Do not copy my hook verbatim - I have stuff there that requires 128 CPUs, you have a 3970x. Just use the gpu section. But for now run manually:

 

 echo 0 > /sys/class/vtconsole/vtcon0/bind

 echo 0 > /sys/class/vtconsole/vtcon1/bind
 echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind

 

You can reboot from the web console to recover display or try to manually rebind.

 

Need to step away for 30 min. Will check back in a bit.

 

I can connect remotely and via console after blacklisting.  Rebooting proxmox requires cutting off the power  and restarting pc.

root@dtk:~# echo 0 > /sys/class/vtconsole/vtcon0/bind
root@dtk:~# echo 0 > /sys/class/vtconsole/vtcon1/bind
root@dtk:~# echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind
-bash: echo: write error: No such device

2 minutes ago, meina222 said:

Yeah - you may. The script is just a sample. You need to check how your devices are named.

 

Could you list what you have under /sys/class/vtconsole

 

 

vtcon0  vtcon1

3 minutes ago, meina222 said:

@dtek which version of Proxmox do you use? Yes, you can create snippets.

My gpu is currently in slot 2.  Do I need to make any changes to the script or vm.conf ?

1 minute ago, meina222 said:

@dtek which version of Proxmox do you use? Yes, you can create snippets.

 6.2-4

33 minutes ago, meina222 said:

vbios.bin is just how I named the file. You can rename that.

I got this error

TERM environment variable not set.
Starting VM 101 - please wait...
/var/lib/vz/snippets/vmhook.sh: line 15: echo: write error: No such device
TASK ERROR: hookscript error for 101 on pre-start: command '/var/lib/vz/snippets/vmhook.sh 101 pre-start' failed: exit code 1

32 minutes ago, dtek said:

@meina222

 

/usr/share/kvm/vbios.bin doesn't exist.  Where can I download this file?  There's no snipplets folder in /var/lib/vz, should I create snipplets folder with vmhook.sh in it?

https://www.techpowerup.com/vgabios/212169/sapphire-rx5700xt-8192-190616

I found this but it's not in .bin format.

1 hour ago, meina222 said:

Hi @dtek,

 

As @fabiosun mentioned, I am an active Proxmox user and am typing this from my Proxmox VM, so I hope this helps. I suppose you're trying to run a single GPU as I see you try to shadow the vbios. Here's one of my MacOS VM configs where I pass my 5700XT

 

args: -device isa-applesmc,osk="ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc" -smbios type=2 -device usb-kbd,bus=ehci.0,port=2 -cpu host,+invtsc,vendor=GenuineIntel
balloon: 0
bios: ovmf
boot: cdn
bootdisk: virtio0
cores: 64
cpu: Penryn
efidisk0: aorus:vm-101-disk-1,size=1M
hookscript: local:snippets/vmhook.sh
hostpci0: 43:00,pcie=1,x-vga=1,romfile=vbios.bin
hostpci1: 86:00,pcie=1
hostpci2: 85:00,pcie=1
hostpci3: 88:00,pcie=1
hostpci4: 02:00,pcie=1
hugepages: 1024
ide2: local:iso/OpenCoreBeta.iso,size=150M
machine: q35
memory: 196608
name: bigsur
numa: 1
ostype: other
scsihw: virtio-scsi-pci
smbios1: uuid=4b5493a6-6a73-48b7-8ce5-2be70a66a383
sockets: 1
vga: none
virtio0: aorus:vm-101-disk-0,cache=unsafe,discard=on,size=250G
vmgenid: 18d68c27-3a62-4059-9280-7f86a572af59
vmgenid: 0c7cc702-74ba-4d8e-ba4b-d52d5fe53847

 

 

As you probably know, you need to make sure your vbios is matching your card, and is located in  /usr/share/kvm/ (in my case it is called /usr/share/kvm/vbios.bin).

 

Also, in the vmhook.sh I run (located in /var/lib/vz/snippets/ ) you need to unbind the framebuffer. I do some other stuff as well (can elaborate), but here's the full hook with highlighted relevant section. Note that I also reboot my host if I shutdown the machine - this is sadly needed due to the reset bug and hopefully will be resolved with the new AMD 6000 cards so I can disable this section - otherwise your GPU will not come back most of the time and you'd be left with a blank screen - hence I comment our re-enabling the framebuffer.

 

#!/bin/bash

vmid="$1"
phase="$2"

if [[ "$phase" == "pre-start" ]]; then
    clear
    echo "Starting VM $vmid - please wait..."
    IFILE=/var/lib/vz/snippets/interfaces.$vmid
    if [[ -f "$IFILE" ]]; then
        cp /var/lib/vz/snippets/interfaces.$vmid /etc/network/interfaces
        systemctl restart networking
    fi          
    echo 0 > /sys/class/vtconsole/vtcon0/bind
    echo 0 > /sys/class/vtconsole/vtcon1/bind
    echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind
elif [[ "$phase" == "post-start" ]]; then
    main_pid="$(< /run/qemu-server/$vmid.pid)"

    cpuset="0-63"
    #cpuset="$(< /etc/pve/qemu-server/$vmid.cpuset)"

    taskset --cpu-list  --all-tasks --pid "$cpuset" "$main_pid"
elif [[ "$phase" == "post-stop" ]]; then
    echo "Post-stop VM $vmid"
    reboot
    #shutdown -h now

    #sleep 5
    # Attempt rebind to EFI-Framebuffer
    #echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/bind
    # Attempt rebind to virtual consoles
    #echo 1 > /sys/class/vtconsole/vtcon0/bind
    #echo 1 > /sys/class/vtconsole/vtcon1/bind
    #sleep 5
    #shutdown -h now
fi

 

 

@meina222

/usr/share/kvm/vbios.bin doesn't exist.  Where can I download this file?  There's no snipplets folder in /var/lib/vz, should I create snipplets folder with vmhook.sh in it?

32 minutes ago, fabiosun said:

Tomorrow I will upload my config

however if you use passthrough of a single gpu you can assign all the resource of your system to a vm

you can’t run 2 vm in this case in the same time

i think problem is on Linux part

also you can use host as cpu and a minimal boot args in vm

it also depends to kernel patches you use in your efi config.plist

if I do not misunderstood you said system works if you do not passthrough gpu?

isn’t it?

Ok looking forward to your config.  I kinda need 2 vms running at the same time so that will not work for me.  The system works 100% without gpu passthrough.  Also there's one part in the guide that I skipped since I don't know how to change "RadeonDeinit" setting in opencore.  My GPU is currently in slot 2,  I'll try to move it to slot 1 and see if that makes a difference.

32 minutes ago, fabiosun said:

 

14 minutes ago, fabiosun said:

I am on my mobile but you have many not correct parameter in your vm

1)boot arg 

2) number of cores 64 it is fine for 3970x

3)if you have 128gb you have to use less ram in vm config

The other 32 cores are reserved for windows.  I updated my profile to 256gb ram but I don't know why it doesn't show.  This was my original boot arg but I copied @meina222 boot arg since he said the new one has better performance.  Should I revert boot arg back to the original one?  updated vga: none with the same result

 -device isa-applesmc,osk="ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc" -smbios type=2 -device usb-kbd,bus=ehci.0,port=2 -cpu Penryn,kvm=on,vendor=GenuineIntel,+kvm_pv_unhalt,+kvm_pv_eoi,+hypervisor,+invtsc,+pcid,+ssse3,+sse4.2,+popcnt,+avx,+avx2,+aes,+fma,+fma4,+bmi1,+bmi2,+xsave,+xsaveopt,check

3 minutes ago, fabiosun said:

@dtek
Have you loaded rom file in the proper location?

in vm config have you put display to none?

 

The rom file is in /usr/share/kvm.  How do I change display to none?  My vm.conf looks like this

args: -device isa-applesmc,osk="ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc" -smbios type=2 -device usb-kbd,bus=ehci.0,port=2 -cpu host,+invtsc,vendor=GenuineIntel
hostpci1: 05:00,pcie=1,x-vga=1,romfile=5700xtref.rom
balloon: 0
bios: ovmf
bootdisk: ide2
cores: 32
cpu: Penryn
efidisk0: local-lvm:vm-101-disk-1,size=4M
ide0: local:iso/Catalina-installer.iso,cache=unsafe,size=2096528K
ide2: local:iso/OpenCore-v9.iso,cache=unsafe,size=150M
machine: q35
memory: 131072
name: macOs
net0: vmxnet3=5E:57:E7:CD:64:BC,bridge=vmbr0,firewall=1
numa: 0
ostype: other
sata0: local-lvm:vm-101-disk-0,cache=unsafe,discard=on,size=300G,ssd=1
scsihw: virtio-scsi-pci
smbios1: uuid=1754abd0-bbbc-4e0f-969b-a9d87ca43e14
sockets: 1
vga: vmware
vmgenid: c8335d64-7052-44fa-a1f2-4f46432c56f7
 

25 minutes ago, fabiosun said:

Today I have tested on proxmox for the first time with my new amd Vega 64 gpu @dtek

I have had only changed my gpu address from 21:00 to 23:00 and my usb controller from 23:00 .3/4 to 25:00.3/4

end

system has booted fine with proper gpu acceleration without touching nothing else

some efi I have used before

 

all this text lines to say tha t our motherboard does not need any vfio stuff and blocking things..

If you want to use vfio as in your message above you have to put inside correct ids Above seems from a nvidia card and not of your amd

try to te read and concentrate about gpu rom declaration if your gpu has no uefi bios.. and stay more simple you can about Linux configuration part, msi trx40 pro needs few adjustment also in gpu passthrough

 

I changed the address to 05:00 but when adding  pci device, my GPU doesn't show up on the list so I added manually in vm.conf.  Also downloaded the rom for the 5700 and renamed it to 5700xtref.rom but I got the spinner icon when starting my vm.

12 hours ago, fabiosun said:

I think this could be the famigerate AMD reset bug and it could be unresolved by now

if you follow @Imanol steps you could solve, but always system could hang if you reboot it

last user i know active here to use proxmox with an AMD gpu is @meina222

maybe it could share his experience with you

i have now an old AMD card but i have not tested in proxmox for now

sorry

I got my login screen to show again so I'm giving the passthrough another shot.  I'm confused with this part of the guide.

Create a file in etc/modprobe.d/ named vfio.conf:

echo "options vfio-pci ids=10de:1b02,10de:10ef" > /etc/modprobe.d/vfio.conf and add on it:

options vfio-pci ids=10de:1b02,10de:10ef disable_vga=1

My vfio.conf should display both of these lines below?

options vfio-pci ids=10de:1b02,10de:10ef 

options vfio-pci ids=10de:1b02,10de:10ef disable_vga=1

On 4/4/2020 at 8:31 AM, fabiosun said:

 

In this thread we can talk about any subject related to AMD 3970x Proxmox VE guide published here:

 

 Hi,

I was able to get Opencore and Catalina installed on proxmox thanks to your guide.  I attempted a passthrough for my 5700 xt GPU but failed.  Now every time I boot my machine,  I can't see the ip and login:  like how it used to display before the  attempted passthrough.   Proxmox web ui and ssh still works but rebooting just hangs.  The power needs to be turned off and back on, otherwise  there's no access to the web ui.  All the settings were reverted but no luck with getting the old login screen back.  This is what I see every time I boot proxmox

21 minutes ago, fabiosun said:

yes I know..but if you never answer to my question is difficult to help you better 🙂

 

however search in your config allowfullnvramreset or similar and flag it if it is not

 

if you do not see again in boot menu press space and see

 

Sorry I might have missed your questions.  What do I need to do to get old settings back?  My build is from a fresh Catalina install.

21 minutes ago, fabiosun said:

I have my GPU on first slot

acquantia card in slot 3

Thunderbolt card in slot 4

Uefi On

4g on

MMIO list said before,,but you can try with nothing in MMIO list only use DevirtualizeMMIO on

 

To get it working again I had to make a few changes. 

1.Added bootarg ncpi=2000 and the original ssdt files,

2.disabled 4g 

To my surpise, the  acquantia card, wifi and bluetooth all work now but it has to rely on those settings and files.   I can't remove them anymore.  Also do you know why my boot menu doesn't have "clear nvram" selection?

1 hour ago, fabiosun said:

@dtek in big Sur you mean?

It is a clean installation or you have ported it from Proxmox?

 

you can delete your network on SystemPreferences/Network, apply this changement and then add again your aquantia card (in may case it is named en5 o similar

I can't say if you need patches I see floating around because in HS I do not need of it (People using Big Sur said here also is useful)

So I moved my GPU and 10g lan card to a different slot to make room for a wifi/bluetooth card but now I can't get it to boot anymore.  I'm stuck at lost battery power and [PCI configuration begin].  Any idea how to fix this?

16 hours ago, Driftwood said:

@dtek Aquantia shouldn't need anything. Its supported by Apple. Shouldn't need to add anything for it.

 

 

The Intel 10G should need a kext I believe, I replaced my wifi pic card like others here to Broadcom Mac compatible.

13 hours ago, iGPU said:

 

Aquantia is supported natively through Mojave (and I thought ok in Catalina), but Big Sur requires a kernel patch. The inactivation has nothing to do with SSDT or MmioWhitelist.

 

Pavo has provided this on another forum. Here it is below (as written this will work on HS thru BS; if you only want for Catalina-BS, change 17.0.0 to 19.0.0):

 

The Aquantia card used to work for me but I don't know what changes I made and now it doesn't work anymore.  My other 2 integrated lan cards work fine.  When I plugged the ethernet cable in, there's activity light but no internet.  I started from scratch and it still doesn't work, then I tested in windows and it works fine.

1 hour ago, meina222 said:

@dtek

 

There is hardly any way to get all quirks worked out without spending time 😂. There’s an added benefit of being able to adapt to BIOS and hardware upgrades much more easily on your own. I think I burnt hundreds of hours on this - more than the cost of a brand new 28 core Mac Pro for sure in man-hours.

 

Yeah for real I spent more time on this build than at my actual job lol. I agree it's a never ending learning curve.  All good tho, with everything going on, I take this opportunity to gain extra knowledge.  Also that's a crazy nice setup you got there.  I'm so jealous hah

4 hours ago, fabiosun said:

I see different stuff in your boot arg

npci=0x2000 shouldn't be there for our motherboard

and if use it 4g have to be off

I would try without that arg and with 4g enabled

also I do not use any SSDT, only one for NVRAM..but it is not mandatory to boot

 

By the way..I do not use intensively Big Sur so some your args could be mandatory for it...but IDK

@dtek

Do you by any chance use the 10g lan card that comes with the board?  Mine suddenly stopped working.  I don't know if it's  related to MMIO whitelist editing I just made and/or the removal of SSDT.  I have to revert the changes to see if it works again.

2 hours ago, iGPU said:

 

The linked instruction is about as lucid as you'll find on such an esoteric build. But this makes me wonder did you actually create the debug file to generate your own hex values? 

 

Do you understand that OC comes in RELEASE and DEBUG versions, and you need to use the latter? (The link within the link, here, again, tells you how to make this file in steps A1 a + b.)

 

Look at the pop-up that has green/red numbers (4th spoiler) in first link (here again). Those are hex. The hex values are given in your debug text file. The white numbers in the right column are decimal. You calculate the decimals equivalent, from the hex values, derived from your debug file. (Even the built-in Mac calculator can convert; Google for other on-line calculators if you don't like that one.)

 

The left green/red values should be similar to those in your debug file. Look for a pattern. Your list created in the debug file should match up (as described).

 

Building a Hackintosh, and esp a TRX40, means having to sit and think about things on your own. Problem solving is part of building one, and will be for the life of the computer.  Develop those problem solving skills now.

 

The file that I attached has my own hex values in it.  I used Fabiosun's mmiowhitelist because we have the same board.   Your instructions are very clear,  it's just that I'm very new at this.  It's literally my first build.  I definitely spent way more time on this than I'd like,.  Thanks for the tip tho,  appreciate your work

3 hours ago, Driftwood said:

@dtek

Do u have this in BIOS? Or equivalent?

 

I enabled S5 Soft off for USB wakeup and it seems it guarantees it in BS BM. 

 

Havent tried the others yet...

I was able to change wakeup to OS/BIOS and it's working perfectly now. thanks

44 minutes ago, fabiosun said:

I see different stuff in your boot arg

npci=0x2000 shouldn't be there for our motherboard

and if use it 4g have to be off

I would try without that arg and with 4g enabled

also I do not use any SSDT, only one for NVRAM..but it is not mandatory to boot

 

By the way..I do not use intensively Big Sur so some your args could be mandatory for it...but IDK

@dtek

My settings  and files are now identical to yours with the exception of my video card boot args.  Everything works 99% now.  The only thing that doesn't work is waking up when pc is in sleep mode but that's ok I rarely put my pc to sleep anyway.   I've learned so much from this forum and really appreciate everyone trying to help.  ❤️