Proxmox VE OSX Guide discussion

[fabiosun]

11 hours ago, iGPU said:

I was monitoring Proxmox on a Mac laptop: when running Cinebench 20, there is an expected increase in CPU usage over that during idle stand-by.

 

Idle:

  Hide contents

 

Cinebench 20:

  Reveal hidden contents

 

I did the same, I have monitored it directly on my AMD OSX PC

Spoiler

 

 

 

[fabiosun]

about Thunderbolt study:

https://www.geeknchic.com/2019/07/making-a-thunderbolt-3-akitio-drive-enclosure-work-with-proxmox-on-a-nuc/

maybe it is premature but I could be useful in a near future

 

[fabiosun]

Short video

This hack beats the AMD hack fastest ever you can find in the you tube 🙂 🙂 🙂 🙂

I understand too because she leaves her case open 🙂

 

 

 

 

Some additional Datas

I can handle about 600 tracks in play, CPU temperatures become prohibitive... (in this video 580 Tracks)

Audio in DP works well in this audio stress test 🙂

 

[Driftwood]

Nice!

[Driftwood]

Been a bit slow to get back to this project... Right.. errors >

I'm getting a communication failure then host reboot ust trying to get Radeon VII passthrough.

 

Did all the stuff in the guide about rerading in address of pcie for Radeon exactly.

 

Getting log warning: STOPPED UNEXPECTED STATUS QMSTART error  kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
kvm: warning: host doesn't support requested feature: CPUID.80000001H:ECX.fma4 [bit 16]

 

followed by an Unable to Read Tail (got 0 bytes) status.

 

Log shows

May 09 01:10:51 pve systemd[1]: Startup finished in 34.590s (firmware) + 24.060s (loader) + 4.985s (kernel) + 9.133s (userspace) = 1min 12.769s.
May 09 01:10:51 pve kernel: atlantic: link change old 0 new 1000
May 09 01:10:51 pve kernel: vmbr0: port 1(enp69s0) entered blocking state
May 09 01:10:51 pve kernel: vmbr0: port 1(enp69s0) entered forwarding state
May 09 01:10:51 pve kernel: IPv6: ADDRCONF(NETDEV_CHANGE): vmbr0: link becomes ready
 

So cant seem to get into  Catalina to see if the passthrough is working.

 

Any ideas? 

 

If I remove the h'Hardware' passthrough pcie raadeon items I can boot back into Catalina so its a pcie radeon vii fault. ..summit I havent done right... do I need a ROM? Becaus ein Hardware it was saying NoRoM

 

 

 

 

 

 

Edited May 9, 2020 by Driftwood

[fabiosun]

@Driftwood if you want to debug post here following files:

/etc/modprobe.d

blacklist.conf

vfio.conf

also grub.cfg

and modules

then your vm conf and the output of lspci -nnk

in my opinion the only problem you should have with radeon is the famous reset bug. It happens when restrt your vm not before

 

 

[fabiosun]

again....benchmarking:

 

Final Cut X 10.4.6 (works in High Sierra)

Bruce X benchmark test

about 18s

 

I have followed method published here:

https://blog.alex4d.com/2013/10/30/brucex-a-new-fcpx-benchmark/

 

I think Nvidia GFX is not appropriate in FCPX..but results are not so bad

Maybe it improves in a newer Metal OSX version (No web drivers so no test for me)

 

[fabiosun]

 

[Rocket88]

After a week of messing around, I finally got my Threadripper 3970x, TRX40 Creator, MSI Radeon Vega 64 to work. I only have one video card.

Here are the changes I had to make:

/etc/default/grub:

GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on textonly video=amdgpu video=efifb:off"    <<< I need efifb:off or I get the BAR 0 error.

*** Note: My monitor will not show anything past the Ramdisk message with video=efifb:off" set

 

/etc/modprob.d/vlio:

# Make vfio-pci a pre-dependency of the usual video modules
softdep amdgpu pre: vfio-pci           <<< Not sure if this is still needed

options vfio-pci ids=1002:687f,1002:aaf8 disable_vga=1
options kvm_amd avic=1                <<< Not sure if this is still needed

 

/etc/pve/qemu-server/101.conf

hostpci0: 03:00,pcie=1,x-vga=on,romfile=MSI.RXVega64.8176.170719.rom    <<< romfile goes in   /usr/share/kvm/

*** Set the boot flag. Without the boot flag set, the display hangs up after the Ramdisk message.

 

I also made the following BIOS changes, and I have not checked yet to see if they are still needed:

 

1.     SR-IOV Support. 

2.     In bios under Advanced\AMD CBS\NBIO Common Options

   a.     Enable AER Cap

   b.     Enable ACS Enable.  (not visible until a. is done)

   c.      Turn on ARI

 

 

Now when I power up the machine, It jumps to the PROXMOX screen, then I can click on the icon of my MacOS, and then I can boot up my new AMD Macintosh. 

Things I need to do:

1. Create a hook script for powering down my hardware. For example something that periodically checks to see if the vm is running and if not, send "shutdown -h now" command. 

2. Everything else. I have days (weeks) of tinkering to do.

 

Once again I would like to thank all of you for your posts!!!!

 

 

[fabiosun]

hi @Rocket88thank you for your message above

 

for to do list

1) is a common research because OSX misses Qemu Agent so if you shutdown OSX linux will be on

 

I think you can avoid something you used..maybe in your blacklist.conf file to pass "hangs after Ramdisk message" (it could mean you have blacklisted your GPU driver so you need to run VM from another pc/device

Only you can verify this

 

When you have time post here benchmark for your system, It could be useful for other users

thank you again for posting your experience!

 

[fabiosun]

@here 

USB audio partially solved....

verify few things then I will expose it how I did.

I repeat USB audio not internal Realtek USB 2.0 motherboard audio for now 

 

Not usable for production.

 

 

 

[Driftwood]

1 hour ago, Rocket88 said:

After a week of messing around, I finally got my Threadripper 3970x, TRX40 Creator, MSI Radeon Vega 64 to work. I only have one video card.

Here are the changes I had to make:

±Nice work Rocket88 :-)

Im having strange problems with my first network output. Its gone awol and I can no longer log in to the host root. I think somewhere along the line after a crash it went missing (or was renamed) sometime after by a really slow boot into Mac, followed by  out of space error on my Proxmox lvm drive, so Im starting out all over again.

Has anyone else found their first nic going missing? ie its not able to be ping'd by the Web interface, and the host doesn't see the web guest either.
I used to log in with 198.192.1.134:8006/ on it, now when I reinstall proxmox the install autos to 198.162.100.2 - which is completely unuseable.
So my only route into / from web interface is to install and connect to the 2nd nic (the Aquina one) now which luckily auto finds in Proxmox install the correct home network (198.162.1.134:8006/).
I'm wondering how to write back the correct nic to the non-working port?
I tried reflash my ASRock BIOS but its still the same!
It al started when I was messing with passthrough of the Radeon VII card.... then not sure what happened!
So wondering if through Proxmox I can bring my first nic back to its default etho0. Incidentally, the nic stiull lights up as though its communicating but that weird address can no longer join my home network ping to/back in any way.

Edited May 10, 2020 by Driftwood

[fabiosun]

@Driftwoodpassing radeon should be unrelated to your problem

Have you passed some network device? I mean the one you have started proxmox when you install it first time?

if so you should do a bridge with it ( I am not confident to how to this and you have to manage at /hosts/interface or similar

simple way is to stay with initial default VMware ethernet controller disabling passed network controller in your VM config

 

[Driftwood]

8 hours ago, fabiosun said:

@Driftwoodpassing radeon should be unrelated to your problem

Have you passed some network device? I mean the one you have started proxmox when you install it first time?

if so you should do a bridge with it ( I am not confident to how to this and you have to manage at /hosts/interface or similar

simple way is to stay with initial default VMware ethernet controller disabling passed network controller in your VM config

 

Its weird as I did note the Aquantia is showing up in Proxmox 'Network' under pve as vmbr0 'enp69s0' (Linux Bridge) so where has my Intel Nic gone?! Hardware doesn't seem to be able to see it.

±Okay after much re-reading and lots of stuff about nic commands being deprecated in Proxmox, I see you can show your nics by typing;-
'ip link show'  in proxmox shell

Whats 'UP' is live and active, what's 'DOWN' is not active. So it appears no. 3 option is DOWN and is the rogue corrupted nic.

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp69s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP mode DEFAULT group default qlen 1000
    link/ether a8:a1:59:16:33:87 brd ff:ff:ff:ff:ff:ff
3: wlp70s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 50:eb:71:77:e7:fa brd ff:ff:ff:ff:ff:ff
4: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether a8:a1:59:16:33:87 brd ff:ff:ff:ff:ff:ff

 

Now just gotta work out how to change/rename/edit it....

Edited May 11, 2020 by Driftwood
more info

[Driftwood]

On 5/6/2020 at 2:45 PM, fabiosun said:

about (my) internal audio problem:

 

23:00.4 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller [1022:1487]

passed as hostpciX: 23:00.4 and dpcimanager app see it as below:

you can also see here a weird audio device:

08086293E (intel related??)

 

if I do not blacklist  blacklist snd_hda_intel  when I boot on proxmox I see a message snd_hda_intel codec not found.

this message seems related to  starship/matisse HD Audio controller (famous 23:00.4)

So or Proxmox Ve associated internal ALC audio to intel codec..or we should think how to eliminate that weird audio device you can see in pictures above

 

If you have some ideas to share it is welcomed as usual 🙂

 

Did you have your HD Audio Enabled in BIOS? or AUTO? ENABLED better ?

[fabiosun]

2 hours ago, Driftwood said:

Did you have your HD Audio Enabled in BIOS? or AUTO? ENABLED better ?

No differences

i think problem is Proxmox/osx related

in a proxmox windows environment it works as it should be

[AllubzV]

@Driftwood Note I had a similar problem. I would seemingly randomly lose connection to the host. I have two NICs, one setup to passthrough, one dedicated to ProxMox. What happened? They changed device names. This maybe happened because I added a third (Thunderbolt) NIC which also took a name. In the end enp68s0 jumped to 69, 69 to 70, and there was trouble.  This seemed fixable by adding in GRUB_CMDLINE_LINUX_DEFAULT "net.ifnames=0 biosdevname=0", or a variation of it.

 

Anyhow, Predicatable Network Naming is a thing: https://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/ - I'm not sure if it at any point helped me, because after a while of using said GRUB line and having them renamed to Eth0, and Eth1, I manage to break the thing again by messing around with other network settings. After that I reinstalled and let normal names be used, and just make sure I don't add a third NIC.

 

Also note I seem to need to use a RadeonVII ROM in order to boot past the last stage and enter desktop, but I'm using Whatevergreen.kext and no boot-arg (OpenCore) that could allow losing the ROM file. There may be a way.

 

@iGPU It's great to see you're so headstrong on getting TB passthrough to work! If I can do a test, let me know. I'm currently a bit overburdened, so I can't invest a lot of time in this whole project anymore. I do read up, like now.

 

@fabiosunI see you're trying to get the audio fixed. What I read in the links you supplied is that the audio device on TRX is not a typical device, but a two audio chips bridged over USB, which makes things overly complicated for VM passthrough.

 

The FLR patch doesn't seem to work for my TRX40 Designare.  Now I note that this is a snippet of the patch:

+/* FLR causes Ryzen 3000s built-in HD Audio & USB Controllers to hang on VFIO passthrough */
+DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_AMD, 0x149c, quirk_intel_no_flr);
+DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_AMD, 0x1487, quirk_intel_no_flr);

 

But, my ID's are 148c(!), 149c, and 1487 (audio). 149c was already doing fine, and 148c seems to remain problematic:

 

03:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller [1022:148c]
        Subsystem: Gigabyte Technology Co., Ltd Starship USB 3.0 Host Controller [1458:5007]
        Kernel driver in use: xhci_hcd
..
25:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller [1022:148c]
        Subsystem: Gigabyte Technology Co., Ltd Starship USB 3.0 Host Controller [1458:5007]
        Kernel driver in use: xhci_hcd
25:00.4 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller [1022:1487]
        Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller [1022:d102]
        Kernel modules: snd_hda_intel
..
48:00.1 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
        Subsystem: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:1486]
        Kernel driver in use: xhci_hcd
48:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
        Subsystem: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:148c]
        Kernel driver in use: xhci_hcd

 

149c is passed through succesfully, also before the FLR patch I believe. 148c seems to be bound to that USB Audio Bridge.

 

Can someone hint me on how I can apply a modified FLR patch? I don't really understand whether I need to recompile the kernel or I can just "patch-in-place"? I can perhaps help solving the passthrough audio mystery 😉

Edited May 11, 2020 by AllubzV

[iGPU]

A few days ago, in an attempt to allow better TB3 functioning (due to lane-sharing issues between PCIe and M.2 slots), I moved the macOS NVMe drive from slot closest to the CPU, along with the Proxmox MVMe drive from the next slot, placing them both into the two M2M slots farthest away from CPU. (The GB TRX40 Designare has 4 M.2 slots.) These latter 2 slots go through the TRX40 chip rather than the CPU, and supposedly should eliminate lane-sharing issues. Instead of helping TB problems, it created many other problems, and left TB3 working as poorly as ever.

 

So, I took a well-working system and converted it into a broken one. I immediately lost control both on the host machine and my laptop SSH/Console to control Proxmox. Even moving the Proxmox drive back to its original location didn't restore control. One thing I did find, after a suggestion by fabiosun, was to look at my blacklist. I found that blacklisting amdgpu prevented seeing the command line on the host monitor. If I'd had access to the host, I could have salvaged the original build. I've since permanently removed amdgpu from the blacklist.conf file.

 

I've gave up trying to fix the mess and re-installed Proxmox. Yet, nothing was working as well as it had. Besides many re-installs, I've even re-flashed BIOS. Some problems, I now track back to using parts of PAVOs GitHub that somehow gave me a green screen when booting both Proxmox and Apple screens; even different GPUs gave same green screens. I don't understand why. Only by using the very basic instructions by fabiosun was I able to get a bootable build.

 

But all is not as it was. While I can boot via Console into macOS, if I pass-thru the Radeon VII (that was working perfectly last week), the boot only makes it to the Apple logo with no progress bar: frozen. When verbose mode is turned on, it stops at slightly different locations on each boot attempt. If I revert back to Console, macOS fully boots; back again to GPU pass-thru: macOS again stops at the Apple logo. So pass-thru works (the main monitor is on and showing the start of the boot), but Mojave won't fully boot. BTW, other things pass-thru okay; strange. While I've still not sorted this out, the next step in trouble shooting will be to remove all pass-thru devices except GPU and slowly re-introduce them.

 

***

 

On a positive note: I found that if you boot using "boot: ide2" (where, ide2: local:iso/opencore.iso,cache=unsafe,size=512M), once you're in macOS, you can load the EFI from the QEMU drive. This contains the de-compressed opencore.iso. Since it's booting from the iso file, you can alter and save it, effectively making changes to the iso file. In other words, if you make changes to this EFI folder (update opencore to v058, add SSDTs, turn kexts on or off, etc.), the changes will be written back to the opencore.iso file under Proxmox: this is like an automatic ISO conversion. You can then copy this updated opencore.iso file back to another computer using the command from the other computer (I used Terminal from my laptop). I did this so I could re-load this updated opencore.iso between my many re-installs.

scp root@IP_addr:/var/lib/vz/template/iso/opencore.iso /Users/userName/Downloads/opencore.iso
(where IP_addr is the VM's log-in IP)

If the ISO file is several GB is size (like a Win 10 iso), the built-in Proxmox Upload feature will fail, so you'll need to use the following for large files:

scp /Users/userName/Downloads/Window_10.iso root@IP_addr:/var/lib/vz/template/iso/Window_10.iso

 

Similarly, it is easy to upload the GPU.rom file from an external computer using the same command to the folder that is for GPU rom files (different from iso location):

scp /Users/userName/Downloads/myGPU.rom root@IP_addr:/usr/share/kvm/myGPU.rom

If, as has happened to me with the various re-installs of Proxmox, there are 'host key' restrictions preventing you from reading or writing using the above commands, enter the following into Terminal on Mac and this will re-set permissions, allowing you to re-try one of the above commands:

rm .ssh/known_hosts

 

***

EDIT:

With pass-thru using Radeon VII, I can now get 2/3 of Apple progress bar (earlier today was zero progression). The change: disabling 4G.

 

The whole problem is curious. Before I moved the NVMe drives, 4G was enabled and the Radeon VII was easily pass-thru and good booting was seen. Those slots share lanes with the CPU. Now that I've moved both drives to M2M slots, which instead share lanes with the TRX40 chip, the 4G must be disabled for graphics. This is an interesting issue. When I was working on the X570, I had many issues with lane-sharing and it took me many days to sort it out as I'd not seen it discussed (briefly covered on my GitHub for X570 here and throughout a thread at AMD-OSX co-worked with CaseySJ).

 

So here it is again with the TRX40 mobo: when different NVMe slots and different PCIe slots are used, different BIOS settings and different PCIe addresses are required.

 

I will probably move the NVMe drives back to the original positions as the current setup is more difficult to successfully boot.

Edited May 12, 2020 by iGPU

[Driftwood]

Finally got the Radeon VII working on the host and using DP and HDMI together....Big sigh, next... all the other passthroughs....

 

 

 

 

[fabiosun]

@AllubzV take a look and compare with yours (I would like also others TRX40 users put their similar datas to compare)

 

04:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller [1022:148c]
        Subsystem: Micro-Star International Co., Ltd. [MSI] Starship USB 3.0 Host Controller [1462:7c60]

23:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller [1022:148c]
        Subsystem: Micro-Star International Co., Ltd. [MSI] Starship USB 3.0 Host Controller [1462:7c60]
        Kernel driver in use: xhci_hcd

23:00.4 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller [1022:1487]
        Subsystem: Micro-Star International Co., Ltd. [MSI] Starship/Matisse HD Audio Controller [1462:cb60]
        Kernel driver in use: vfio-pci

46:00.1 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
        Subsystem: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:1486]
        Kernel driver in use: vfio-pci
46:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
        Subsystem: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:148c]
        Kernel driver in use: vfio-pci

And focalise also on Subsystem data (check also your BIOS AGESA, TRX40 ones are not updated to the last patch).

Now, as you have seen FLR patch is not perfectly calibrated for TRX40 users and if you follow below link I have inserted on Proxmox forum, Stefan_R (a proxmox forum staff guy)  says we have to do some others step because "we are experiment on the joy to use bleeding edge new hardware" 🙂

 

FLR patch was done some months ago and follow leveltech1 comment in some cases is possible to insert via kernel command (in my case it not was working)

@pavo compiled that patch in a kernel (which is not the one suggested by Fabian) and it is available on his GitHub 

 

In my case is not working and for some x570 users is working partially 

 

So our goal should be how to compile new Kernel you can find in Proxmox git and then to understand if it possible to insert on a kernel line command FLR patch

 

Proxmox Stefan_R advice

 

level1techs FLR patch discussion

 

Proxmox git for latest Kernel

 

As last information I would like to share, I can pass also 23:00.4 as you can see in my above box (it uses vfio-pci kernel )but audio is problematic

I can have a working internal audio via matisse audio controller or with a simple usb audio adapter bought to test this problem.

Audio works..but it is not perfect (I loose it, some time is scratchy an so on) some time is perfect but usually if I change audio source from ie a video to a mp3 file ..I loose it

For this I said before no ready for a professional use

Nvidia Dp/hdmi is perfect.

 

My idea is a negative idea in this subject (I hope to be very wrong on this).

problem for me is due Audio osx system driver and how it is interact with trx40 CPU.

In Proxmox Windows 10 64 bit VM, same linux configuration, internal audio  (USB 2.0 Realtek) is perfect

 

As last last ( 🙂 ) information I would like to advice to test audio problem (it is valid also for other problem you can have) to test with a minimal set of passed controllers..

I pass for this task only NVME controllers where I have my OSX disk and then I pass mouse/keyboard as single USB ports.

To have proxmox distro installed on an USB disk could be useful to pass fine all your Sata  or NVME controllers to OSX (I use back type C port of my motherboard) and doing this I loose possibility to pass my ASMedia USB controller to OSX, but both ports on it are working (simple USB port passthrough)

 

 

[fabiosun]

5 hours ago, Driftwood said:

Finally got the Radeon VII working on the host and using DP and HDMI together....Big sigh, next... all the other passthroughs....

 

 

Good, If you think it could help other people and if you want , describe your procedure to have it working 🙂

 

I have had  private conversation with many AMD radeon VII and seems some Radeon VII card has not  a UEFI BIOS

It is not confirmed directly by me, but it is a sort of "rumours", and this should be the cause of many AMD GPU card users.

 

[fabiosun]

@iGPU

scp command are in the guide and then are also obsolete in some case from an advice @Rocket88gives about using FileZilla ftp client(if you search you can find easily)

link here

A remainder as you did is good too 🙂

 

about lane sharing..this could be the big task for all of us togheters multiple "bridge" connection we have..

I think it is also an unexplored territory also for Proxmox guy and maybe in many case Windows VM is not affected from the same our problems because in there more supported stuff is (virtio drivers, spice and so on)

I am convincing myself  ( 🙂 ) that audio problem is not related to my poor knowledge but it is more a "system" problem

I can have it perfectly working but it is not stable at all as it should be and also how Nvidia DP/Hdmi is.

 

I would like to share an example:

with many tries I have "a working" Realtek audio 

I launch DaVinci Resolve to produce some video you can find in macOS86 you tube channel 

edit video cutting, scribbling audio perfectly..

I put a cross dissolve ( a simple cross dissolve to insert macOS86 logo as a super on main video ) audio scratch during cross dissolve effect...

Our system (mine in particular) can work with multiple 8k layer without any problems)

but using this problematic audio it fails (I repeat..with DP audio is perfect)

so it should be OSX audio driver not working well with VM previously configured part ..

Searching on the net many talking about problematic audio in VM but no one gives a solution to try 

 

 

[fabiosun]

to all:

Maybe it is not well shown in the guide but you have to remember a thing:

Clover/OpenCore iso image we upload on Proxmox appropriate folder if associated in our VM config it become a normal bootable EFI as we have in a "real" hackintosh

So, if you modify it you automatically modify uploaded iso file in proxmox.

ie my clover.iso I have in proxmox now inside has Opencore 0.59 boot loader..

So pay attention on this. In my experience is better to have uploaded a clean and simple working iso you can use only for repair bad  things gone wrong, also in my case I have a commented line in VM with opencore/clover working iso that I never modify

Usually I boot from EFI in my NVME drive, when things go bad (for osx I mean)  I uncomment that line and from OVMF Proxmox bios I choose that iso to boot fine.

 

[fabiosun]

@AllubzV

this could be the command (boot parameter) for me to test...adding where? they say boot argument.. I have tested in grub.cfg with no success for now

 

pcie_no_flr=1022:148c,1462:cb60

[fabiosun]

https://forum.proxmox.com/threads/proxmox-ve-6-2-released.69647/