Proxmox VE OSX Guide discussion

cj750 · June 12, 2020

21 minutes ago, Driftwood said:

@cj750 What is your vfio.conf looking like? You are passing the addresses: for video and HDMI audio: -

ie you have the following for the Frontier card in your vfio.conf

options vfio-pci ids=1002:6863 ,1002:aaf8 disable_vga=1?

After every modprobe vfio change your updating the initramfs, update grub, pve-efiboot-tool refresh and shutdown now? switch off, then power back up.

Here's a few VFIO guide tips: https://github.com/SMeatBoy/vfio/blob/master/README.md

Here is

options vfio-pci ids=1002:6863,1002:aaf8 disable_vga=1

looks the same

fabiosun · June 12, 2020

@cj750 I have no answer for your problem

obviusly is gpu related but if you have done all things proposed here from amd gpu user... I have no idea😢

fabiosun · June 12, 2020

16 hours ago, Rox67er said:

From what I can see in the picture it is exactly the same connector, I think it is called an NGFF intended for WiFi. So if you can buy the same card with the metal casing you can add wifi too without offering a PCIE slot. (Some nice info here: https://www.delock.com/infothek/M.2/M.2_e.html)

Or use some internal antenna like this:

----

Have you a link to buy also metal case because I have not it and it seems safer to mount a Wi-Fi bt in an original metal case

Rox67er · June 12, 2020

2 hours ago, fabiosun said:

Have you a link to buy also metal case because I have not it and it seems safer to mount a Wi-Fi bt in an original metal case

I searched a bit for it yesterday but didn't find anything. Sorry.

Rox67er · June 12, 2020

5 hours ago, cj750 said:

Here is

options vfio-pci ids=1002:6863,1002:aaf8 disable_vga=1

looks the same

Have you tried the 5700X special settings from the original guide? It uses a slightly different unbind code. Please check over there.

#!/bin/bash
echo "0000:03:00.0" > /sys/bus/pci/drivers/vfio-pci/unbind
echo 0 > /sys/class/vtconsole/vtcon0/bind
echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind

Rox67er · June 12, 2020

@fabiosun @Driftwood

I noticed something strange regarding my USB controller issue, in my IOMMU overview the group for the two USB Controllers looks like this:

IOMMU Group 49 42:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a4]
IOMMU Group 49 47:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
IOMMU Group 49 47:00.1 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
IOMMU Group 49 47:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]

But for the excel file shared by @Driftwoodfrom the ASRock techie it is like this:

IOMMU Group 55 42:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4]
IOMMU Group 55 48:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
IOMMU Group 55 48:00.1 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
IOMMU Group 55 48:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]

I noticed that my 42:08.0 only is called "Device" while the same controller in the other dump is called "Matisse PCIe GPP Bridge".

Can you check how this looks like on your side?

Edited June 12, 2020 by Rox67er

Rox67er · June 12, 2020

And maybe more interesting which kernel driver is loaded (lspci -nnk):

42:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a4]
        Kernel driver in use: pcieport

Is this the same with you @fabiosun ?

fabiosun · June 12, 2020

11 minutes ago, Rox67er said:
And maybe more interesting which kernel driver is loaded (lspci -nnk):
42:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a4]
        Kernel driver in use: pcieport
Is this the same with you @fabiosun ?

I have this:

IOMMU Group 49:
        42:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a4]
        46:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
        46:00.1 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
        46:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]

with:

42:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a4]
        Kernel driver in use: pcieport
46:00.1 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
        Subsystem: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:1486]
        Kernel driver in use: vfio-pci
        Kernel modules: xhci_pci
46:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
        Subsystem: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:148c]
        Kernel driver in use: vfio-pci
        Kernel modules: xhci_pci

these results without adding lines on vfio.conf or in blacklist.conf

Rox67er · June 12, 2020

34 minutes ago, fabiosun said:

I have this:


IOMMU Group 49:
        42:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a4]
        46:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
        46:00.1 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
        46:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]

with:


42:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a4]
        Kernel driver in use: pcieport
46:00.1 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
        Subsystem: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:1486]
        Kernel driver in use: vfio-pci
        Kernel modules: xhci_pci
46:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
        Subsystem: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:148c]
        Kernel driver in use: vfio-pci
        Kernel modules: xhci_pci

these results without adding lines on vfio.conf or in blacklist.conf

ok so that also doesn’t seem to be the cause...

Driftwood · June 12, 2020

1 hour ago, Rox67er said:

noticed something strange regarding my USB controller issue, in my IOMMU overview the group for the two USB Controllers

Im asking ASRock these very questions and to fill in all the addresses being employed on the motherboard for the two SATAs (which is controlling which ports?) and the USB 2.0 device, front panel addresses etc...

56 minutes ago, Rox67er said:

I have this:


IOMMU Group 49:
        42:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a4]
        46:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
        46:00.1 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
        46:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]

with:


42:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a4]
        Kernel driver in use: pcieport
46:00.1 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
        Subsystem: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:1486]
        Kernel driver in use: vfio-pci
        Kernel modules: xhci_pci
46:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
        Subsystem: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:148c]
        Kernel driver in use: vfio-pci
        Kernel modules: xhci_pci

these results without adding lines on vfio.conf or in blacklist.conf

These will change addresses depending on what is in the PCIe slots/m.2 slots occupied. etc. Where you have 46:00.1 etc I have them at 47:00.1 etc...

Edited June 12, 2020 by Driftwood

Driftwood · June 12, 2020

Type in command shell:-

for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*};n=${n%%/*}; printf 'IOMMU Group %s ' " $n";lspci -nns "${d##*/}"; done;

and copy it into a excel spreadsheet. Could you show me your complete spreadsheet?

Edited June 12, 2020 by Driftwood

fabiosun · June 12, 2020

2 hours ago, Driftwood said:

Type in command shell:-

for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*};n=${n%%/*}; printf 'IOMMU Group %s ' " $n";lspci -nns "${d##*/}"; done;

and copy it into a excel spreadsheet. Could you show me your complete spreadsheet?

this is mine

MSI TRX40 PRO 10G IOMMU.xls.zip

Edited June 12, 2020 by fabiosun
edited

Driftwood · June 12, 2020

I posted mine in yesterday's post after the ASRock Techie's. but here it is again.

https://www.macos86.it/applications/core/interface/file/attachment.php?id=19337

fabiosun · June 12, 2020

@DriftwoodI thought you want to see other result.. 🙂

Driftwood · June 12, 2020

ASRock Creator TRX40 board owners : I believe that the following should be set in BIOS. A little tip from ASRock that may or may not help us...

Advanced>Onboard Devices Configuration>SR-IOV Support>Enabled

Advanced>AMD CBS>NBIO Common Options>IOMMU>Enabled

Advanced>AMD CBS>NBIO Common option>Enable AER Cap>Enable

15 minutes ago, fabiosun said:

@DriftwoodI thought you want to see other result.. 🙂

Yours is very nice @fabiosun nice and neat 🙂 It seems easier to config MSI!

@Rox67er

ASRock techie said the report for IOMMU groups was done on the following (not using Proxmox)

It was just a fresh Ubuntu desktop 20.04 LTS installation: https://ubuntu.com/download/desktop

So that should be kernel 5.4.

Edited June 12, 2020 by Driftwood

fabiosun · June 12, 2020

1 minute ago, Driftwood said:

ASRock Creator TRX40 board owners : I believe that the following should be set in BIOS. A little tip from ASRock that may or may not help us...

Advanced>Onboard Devices Configuration>SR-IOV Support>Enabled

Advanced>AMD CBS>NBIO Common Options>IOMMU>Enabled

Advanced>AMD CBS>NBIO Common option>Enable AER Cap>Enable

Yours is very nice @fabiosun nice and neat 🙂 It seems easier to config MSI!

I don't know if it is simpler or not than other motherboards, but I always wondered why other users need to block things in vfio when in my case you just have to pass them in the virtual machine configuration to be automatically passed to vfio. ... I started this journey by doing things as recommended by the proxmox wiki..to cross out all the useless things for me

Rox67er · June 12, 2020

My IOMMU attached.

@Driftwood I already assumed he didn't use Proxmox I think it is kernel related how the devices are recognised. (Naming and drivers)

ASRock TRX40 IOMMU.xlsx.zip

cj750 · June 12, 2020

9 hours ago, Rox67er said:
Have you tried the 5700X special settings from the original guide? It uses a slightly different unbind code. Please check over there.
#!/bin/bash
echo "0000:03:00.0" > /sys/bus/pci/drivers/vfio-pci/unbind
echo 0 > /sys/class/vtconsole/vtcon0/bind
echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind

Rox67er,

I was missing the line below. But nothing changed. Question when proxmox loads. And it choses the version screen. Should i see anything on the screen right after?. As soon as ProxM loads it goes to black screen. Before i started the mac os build it would log to a login root screen. Now i can only get to the GUI login from a different computer. But the only thing that has change when i did this before was my card was in slot 3 not slot 1. But still black screen. Also i change the Bios setting

echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind

3 hours later:

OMG......... Speechless ................

So checked everything over and over. read 25 pages of post twice. And guide i rather not say how many times. It load to the proxmox screen like it does 1,000 of time on my TV, but then goes black when it loads. This time Im like what if i choose Proxmox environment 5.4.34-1-pve instead of 5.4.41-1-pve. It goes to black and then i see the proxmox loading file screen and there it was the root login window on my screen.

I was in shock. I didnt know what to do. But i rebooted the pc again to see if it was just luck. Nope it worked on older version. So when i used web to start the VM with GPU it loaded...

I cant say Thank You enough for everyone help an not giving up on me. But now i have to maybe figure out why the new version of pro does not load my card. But im so happy im pouring me glass of scotch. I know i have allot to do more. But for me this was a big step.

Edited June 12, 2020 by cj750

fabiosun · June 12, 2020

@cj750 send us a glass of scotch 😂

and yes it is a big step for you

now you have the hope it can work also for you👍

Rox67er · June 12, 2020

4 hours ago, Driftwood said:

ASRock Creator TRX40 board owners : I believe that the following should be set in BIOS. A little tip from ASRock that may or may not help us...

Advanced>Onboard Devices Configuration>SR-IOV Support>Enabled

Advanced>AMD CBS>NBIO Common Options>IOMMU>Enabled

Advanced>AMD CBS>NBIO Common option>Enable AER Cap>Enable

Yours is very nice @fabiosun nice and neat 🙂 It seems easier to config MSI!

@Rox67er

ASRock techie said the report for IOMMU groups was done on the following (not using Proxmox)

It was just a fresh Ubuntu desktop 20.04 LTS installation: https://ubuntu.com/download/desktop

So that should be kernel 5.4.

@Driftwood @fabiosun

Well close but not completely. After reading your post I jumped in and started tweaking the bios again....

Using the above settings crashes Proxmox on boot but I noticed that when enabling AER Cap a new option showed up in bios which was disabled. After also enabling that one I can stop-start-reboot and get no 99 error. Tried it now for like 8 times... 😁

So my current bios settings for ASRock TRX40 Creator:

(Still using Bios 1.63 but expect no impact from that to be honest)

* Start with default UEFI settings

* Set XMP profile for your RAM

* Tweaked FAN settings

* Advanced>Onboard Devices Configuration>SR-IOV Support>Enabled

* Advanced>AMD CBS>NBIO Common Options>IOMMU>Enabled

* Advanced>AMD CBS>NBIO Common option>Enable AER Cap>Enable

* Advanced>AMD CBS>NBIO Common option>ACS Enable>Enable

I did not need to set above 4G or anything else. Let's see if this is stable but looks like we nailed it !!! 😎

cj750 · June 12, 2020

2 minutes ago, fabiosun said:

@cj750 send us a glass of scotch 😂

and yes it is a big step for you

now you have the hope it can work also for you👍

I will send!

Well i can move the mouse and keyboard and login. So im going to take a break with my drink and play some hockey for 30mins. To clear the mind. And start the next hurdle.

Driftwood · June 12, 2020

The ASRock TrX40 Creator has two SATA controllers at 48:00.0 and 49:00.0 respectively.

I got ports 5-8 working on the SATA controller at address 48:00.0 now and passedthru:

VFIO

options vfio=pci ids=1022:7901

VM 100.CONF

hostpci6: 48:00.0,pcie=1,rombar=1

This means I can now use four internal drives on those ports (indeed, Im looking at my windows boot drive on port 5 right now!), whilst keeping the Proxmox SSD boot drive on 49:00.0, SATA port 1 (1-4 is free) from outside the hypervisor. 🙂

Here's v2 of my chart updated. Now lets get these USB ports figured out! @Rox67er

Edited June 13, 2020 by Driftwood

Driftwood · June 12, 2020

4 hours ago, Rox67er said:

* Start with default UEFI settings

* Set XMP profile for your RAM

* Tweaked FAN settings

* Advanced>Onboard Devices Configuration>SR-IOV Support>Enabled

* Advanced>AMD CBS>NBIO Common Options>IOMMU>Enabled

* Advanced>AMD CBS>NBIO Common option>Enable AER Cap>Enable

* Advanced>AMD CBS>NBIO Common option>ACS Enable>Enable

I did not need to set above 4G or anything else. Let's see if this is stable but looks like we nailed it !!! 😎

This BIOS looks good @Rox67er... Im reporting back to ASRock. I can now run my 64 cores stable too with these BIOS settings.

I'm running Above 4G too and you get slightly better benchmarks.

Edited June 12, 2020 by Driftwood

fabiosun · June 13, 2020

@DriftwoodDo you use Davinci resolve for your job/hobby?

if so can you test Candle scene benchmark if you have two radeon VII installed?

I would like to see your results 🙂

and of all people who want to try

Rox67er · June 13, 2020

15 hours ago, Driftwood said:

This BIOS looks good @Rox67er... Im reporting back to ASRock. I can now run my 64 cores stable too with these BIOS settings.

I'm running Above 4G too and you get slightly better benchmarks.

Well I did run into the 99 error again today, I didn't change anything though. Still needs more testing I guess.

I tried performance run with Above 4G on / off results below (Geekbench 5 CPU and GPU Metal score)

Above 4G OFF

1282 / 23881 / 61463 (average of 4 runs)

Above 4G ON

1282 / 23714 / 60924 (average of 3 runs)

I would say no difference in my setup.

Sign In

Proxmox VE OSX Guide discussion

Recommended Posts

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Create an account or sign in to comment

Create an account

Sign in

Recently Browsing 0 members

Important Information