Rox67er

34 minutes ago, fabiosun said:

 

I have this:

IOMMU Group 49:
        42:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a4]
        46:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
        46:00.1 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
        46:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]

with:

42:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a4]
        Kernel driver in use: pcieport
46:00.1 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
        Subsystem: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:1486]
        Kernel driver in use: vfio-pci
        Kernel modules: xhci_pci
46:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
        Subsystem: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:148c]
        Kernel driver in use: vfio-pci
        Kernel modules: xhci_pci

these results without adding lines on vfio.conf or in blacklist.conf

 

ok so that also doesn’t seem to be the cause...

And maybe more interesting which kernel driver is loaded (lspci -nnk):

42:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a4]
        Kernel driver in use: pcieport

Is this the same with you @fabiosun ?

@fabiosun @Driftwood

I noticed something strange regarding my USB controller issue, in my IOMMU overview the group for the two USB Controllers looks like this:

IOMMU Group 49 42:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a4]
IOMMU Group 49 47:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
IOMMU Group 49 47:00.1 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
IOMMU Group 49 47:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]

But for the excel file shared by @Driftwoodfrom the ASRock techie it is like this:

IOMMU Group 55 42:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4]
IOMMU Group 55 48:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
IOMMU Group 55 48:00.1 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
IOMMU Group 55 48:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]

I noticed that my 42:08.0 only is called "Device" while the same controller in the other dump is called "Matisse PCIe GPP Bridge".

Can you check how this looks like on your side?

5 hours ago, cj750 said:

Here is 

 

options vfio-pci ids=1002:6863,1002:aaf8 disable_vga=1

 

looks the same

Have you tried the 5700X special settings from the original guide? It uses a slightly different unbind code. Please check over there.

#!/bin/bash
echo "0000:03:00.0" > /sys/bus/pci/drivers/vfio-pci/unbind
echo 0 > /sys/class/vtconsole/vtcon0/bind
echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind

2 hours ago, fabiosun said:

Have you a link to buy also metal case because I have not it and it seems safer to mount a Wi-Fi bt in an original metal case

I searched a bit for it yesterday but didn't find anything. Sorry.

1 hour ago, cj750 said:

When i made the changes this is what i get below and still black screen. 

 

qm start 101

kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]

kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]

kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]

kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]

 

Jun 11 10:44:58 pmce kernel: vfio-pci 0000:03:00.0: vfio_ecap_init: hiding ecap 0x19@0x270

Jun 11 10:44:58 pmce kernel: vfio-pci 0000:03:00.0: vfio_ecap_init: hiding ecap 0x1b@0x2d0

The warnings come from the +pcid option in the VM cpu line. It is safe to remove that.

Are you sure your settings as posted are correct? I see this:

rgs: -device isa-applesmc,osk="ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc" -smbios type=2 -cpu Penryn,kvm=on,vendor=GenuineIntel,+kvm_pv_unhalt,+kvm_pv_eoi,+invtsc,vmware-cpuid-freq=on,+pcid,+ssse3,+sse4.2,+popcnt,+avx,+aes,+xsave,+xsaveopt,check -device usb-kbd,bus=ehci.0,port=2
Penryn,kvm=on,vendor=GenuineIntel,+kvm_pv_unha$
balloon: 0
bios: ovmf

The first line should start with:

args: -device .....

The second line is a half copy of something from the first line and should not be there.

I don't understand your GRUB video=amdgpu setting is that because of an internal GPU ?

To try and release the GPU from Proxmox you could try this command to start the VM: (It basically does the same as my hook script)

echo 0 > /sys/class/vtconsole/vtcon0/bind; echo 0 > /sys/class/vtconsole/vtcon1/bind; echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind; qm start 101

21 minutes ago, fabiosun said:

@Rox67ermy motherboard has an empty slot but I thought it is a pciex1 slot

thank you for your detailed tutorial 🙂

From what I can see in the picture it is exactly the same connector, I think it is called an NGFF intended for WiFi. So if you can buy the same card with the metal casing you can add wifi too without offering a PCIE slot. (Some nice info here: https://www.delock.com/infothek/M.2/M.2_e.html)

Or use some internal antenna like this:

1 hour ago, fabiosun said:

@Rox67erhave you a proper slot for that wifi?

or you will use a pcie adapter for it?

 

@Driftwood @fabiosun

This is one happy guy. 😁

I swapped it out with the intel AX200 which was on-board on the TRX40 Creator, It is a drop in replacement if you buy the BCM94360NG version.

Initially showed the wifi as EN2 (which I'm not sure handoff / continuity likes) so after resetting the Networkinterfaces in MacOs and rebooting 

sudo rm /Library/Preferences/SystemConfiguration/NetworkInterfaces.plist 
sudo rm /Library/Preferences/SystemConfiguration/preferences.plist

It now shows as EN1, and adding the built-in flag in OpenCore I got this:

Now wifi and bluetooth were working, airdrop was working but continuity and Apple Watch unlock not yet. Next I signed out of apple-id on my hack and signed in again and....

wooohooo everything works, BT / Wifi / Airdrop / Continuity / Apple Watch unlock / iMessage

The swap is relatively easy (considering you have to take the motherboard out to reach it) remove these two screws (holding the metal cover for the wifi card)

Then you can lift the entire metal case up (internal wifi card will disconnect from the motherboard connector) if you bring it under an angle you can remove it to the rear of the motherboard without removing the heatsink.

It leaves you with this:

And after removing the one Philips screw you can open the metal case by tilting the lid. (I used a small screw driver to open it)

Swap out the intel board for the Broadcom one, reconnect the antenna's (make sure they stay connected as when closing the lid it may push against the antenna cables)

Make also sure that you don't mount the new wifi board too high on the internal Philips screw as it will have a poor connection to the motherboard then. (Also don't push it down as far as possible as you will find there is tension when mounting the metal frame back on the motherboard.

Below image should illustrate what I mean (same space to the sides as to the bottom of the screw):

Mount everything back in reverse order and enjoy your fully working BT / WiFi.😁

Proxmox settings:

Off course you have to add the cards pci  to you vfio with "options vfio-pci ids=14e4:43a0" and add it to the VM passthrough. I also added "blacklist bcma" to make sure linux leaves the wifi card alone. (checked, MacOs wil hang during boot if this is omitted)

My new WiFi / BT card has arrived! 😁

and no, I don’t have very thick fingers... 

18 hours ago, Driftwood said:

ASRock techie guy sent me his 1.63 IOMMU Groups spreadsheet which looks correct. He is interested to see how we get on with Linux and Proxmox. 

Also attached is my latest IOMMU groups with my firewire and decklink cards swapped into pcie slots 2 and 4 for those who need to see what is going on.

 

 

IOMMU TRX40 Creator BIOS 1.60 & 1.63.ods.zip 7.34 kB · 5 downloads

IOMMU groups Driftwood.xlsx.zip 9.54 kB · 0 downloads

So I tried the new bios and it doesn’t help for me. Without my hook script I get the stuck in D3 error on the GPU and with the hook script I get the 99 error after starting vm the second time as soon as I also pass the USB controllers. I do see some improvement when using pcie_acs_override setting in grub (system doesn’t freeze with 99 error but reboots)

7 hours ago, TheDaemonBarber said:

That one is for AM4 and all the other smaller chips, but they do make a bigger one for Threadripper:

 https://www.coolermaster.com/catalog/coolers/cpu-liquid-coolers/masterliquid-ml360-rgb-tr4/

I use that one, so far works fine and satisfied with it. Tweaked down the pump and fans profiles to minimise noise.

Which file system did you guys install Proxmox on? Do you use zfs? I installed on default but notice when I disable csm the grub blue screen builds up really slow. I think it is not on UEFI while the boot menu selection states it is...?

10 minutes ago, Rocket88 said:

I am using a single MSI Vega64 card in the first slot. I have lilu.kext and whatevergreen.kext installed and I am running OpenCore 0.58

I had to add this line to grub to make the video pass thru:

GRUB_CMDLINE_LINUX_DEFAULT="quiet iommu=pt amd_iommu=on video=efifb:off"

 

To start my VM, I either need another computer or I set onboot: 1 in my 100.conf file

You can probably remove the video=efifb:off if you use my hook script at the start of the VM, it effectively does the same as the grub setting but at least gives you normal boot with capability to start the VM.

Look here: https://github.com/iroger/ASRock-TRX40-proxmox/tree/master/var/lib/vz/snippets

Add to your VM 100.conf like:

hookscript: local:snippets/reset-gpu.pl

If you leave out the last command in the reset script:

system("rtcwake -m mem -s 3");

You don't get the 3 second system stand-by (To reset GPU from D3 power state) but effectively get the same as video=efifb:off

24 minutes ago, iGPU said:

 

I'm using a swapped BT/Wfi card. I had all sorts of trouble with the Aquantia, so I ended up leaving it for the host and passing the Intel I211 and using the usual SmallTree kext. I do pass 48:00.1 and 48:00.3.

Interesting, I have a 94360NG on order. Should be a swap in replacement for the onboard version. If I don't use WIFI/BT my aquantia plays perfect. I patched it to internal and even iMessage works.

Regarding my 99 error, are you able to shutdown VM and start it again while passing through the USB controllers? Since you also run the Radeon VII this would rule out the Reset Bug as a cause but point at differences in bios. Maybe should have gone for the MSI board... 🤬

58 minutes ago, Driftwood said:

ASRock Creator owners please report your problem here and we can get some action - you dont have to mention Mac!

 

https://event.asrock.com/tsd.asp

I can boot on Displayport or/and with HDMI no problem.

I just filed the 99 error, hope they pick this up since MSI boards don't seem to suffer the issue but HW wise are very similar.

4 hours ago, Rocket88 said:

I am getting the 99 error on my Asrock TRX40 Creator also. I thought it was just me. I get this error approximately every 3rd time I power up. This includes booting with Windows by hitting F11 on startup. To make the error go away, I have to unplug (or power off) the power supply, start the computer, which drains the capacitors, plug in the power supply, and then start again. Everything works great after that. The other issue that I can't seem to solve is Proxmox will only start on HDMI, not DisplayPort.  

Which video card do you use? Do you load lilu.kext and whatevergreen.kext in OpenCore? Should solve most HDMI / Displayport output issues.

Thanks for confirming the 99 error, at a point I thought I had a bad mother board. Hope ASRock can improve in the biosfirmware and next Proxmox version should include new kernel with a lot of AMD improvements. 

I checked running Proxmox disk on the USB-C port and passing the 47:00.1 and 47:00.3 USB Controllers. (As long as I block bluetooth / wifi MacOs works fine, as soon as I try to load the intelUSB kext my Aquantia controller gets messed up. My wifi/BT upgrade has not arrived yet...)

However, in this config I can only boot the VM once. If I shut down and then start the VM again, it locks up with the 99 error on the motherboard. It may be related to older AGESA / BIOS since we are still on 1.0.0.3 B if I understand correctly, Fabiosun has been on 1.0.0.3 C and is now on 1.0.0.4.

Could also be the reset bug for the Radeon VII.

3 hours ago, Driftwood said:

A bit of discussion here: How many cores are you all passing and why?

 

At only 64gb RAM I've found stability (whilst waiting for more memory to come in the post) using only 32 cores and leaving at least 4gb ram for proxmox host . The reason mainly is I understand from reading Proxmox site that you have to leave something in the way of at least 1 core (even though we push through 64 cores to our VMs or share the 64 cores amongst more if you have more than 1 vm running). And then you read the experts that suggest you should follow the power of 2 rule (offer only 2, 4, 16, 32 cores) and hold the rest back for the Host and others VMs.  

However if you have this rule then it appears we should be only running a max of 32 cores for the 3970x correct me if Im wrong?

 

I have extremely stable operation when I set Mac OS to use 56Gb Ram and only 32 cores. Has anyone else with 64gb RAM experienced this?

 

PS Ive now ordered 256Gb G.Skills AMD QL cert. DIMMs and realise I could be waiting a few weeks as they've only just launched them 😞

I run 120Gb hugepages of 1Gb. (out of 128Gb) and pass all cores.

Proxmox allocates the full memory at startup that is why it uses all memory. (In my node view it shows 121.9 GiB in use (So some margin left, but in the VM it doesn't use memory since all the hugepages are pre-allocated)

On 6/3/2020 at 2:19 AM, Driftwood said:

 

I nearly bought those, now ordered the G.Skill 3600s. 18 timings XMP 2 should work!

Will you be able to drive 256Gb with those as well? When I looked the 3200 was the highest to support 256Gb

2 hours ago, Driftwood said:

RAM Tips for TRX40 Boards

 

So Ive been using 64Gb HypeX 3600 4x16 DIMMS in my setup (HyperX HX436C17PB3K4/64 Predator DDR4 64 GB (Kit 4 x 16 GB), 3600 MHz CL17 DIMM XMP) which are great at 3600, and work with XMP 1, but Im thinking of upgrading to 128Gb and it doesnt look like HyperX have anything compatible.

I've been looking around for 'compatible' options and so far here's the links (please add yours!);-

 

https://linustechtips.com/main/topic/1133083-best-ram-and-configuration-tipps-on-ryzen-thredripper-3970x-build/

 

3200

https://uk.pcpartpicker.com/product/rJkgXL/gskill-ripjaws-v-128-gb-4-x-32-gb-ddr4-3200-memory-f4-3200c16q-128gvk

 

I run G.Skill DDR4 Ripjaws-V 4x32GB 3200Mhz [F4-3200C16Q-128GVK]

run fine on ASRock TRX40 and leave option to 256GB open. 😋

37 minutes ago, fabiosun said:

@Rox67erif port 8 is an ASMEDIA Port it seems pretty identical to my situation

Connecting  there Proxmox disk you can pass all others controllers (SATA,USB,VGA and so on) easily

This DO NOT MEAN you do not have lag problem If you passthrough all controllers (also problematic one like Audio /usb controllers), but you can pass both sata controllers and use well all disk connected to them as I do

 

 

@fabiosun Correct, port 8 is ASMedia.

The difference to my opinion is that port 10a / 10b / 15a / 15b in my case are NOT connected to 47:00.1 / 47:00.3 but to one of the Starship USB Controllers. If I pass 47:00.1 and 47:00.3 to MacOS these ports remain connected to the host. (tested with mouse which still shows under "Node / Hardware / Add / USB device"

@Driftwood regarding Radeon VII reset bug, I found this post on a huge number of AMD pulls to kernel 5.5 (Proxmox is now on 5.4) so we may see some improvement to the GPU reset issue on next Proxmox version. https://www.phoronix.com/scan.php?page=news_item&px=Linux-5.5-AMDGPU-More-Features

2 hours ago, fabiosun said:

Agree!

i use ASMedia usb for proxmox with a Samsung 840 pro ssd. And this is the easy way to have the chance to pass trough all other controllers I have in my msi trx40 pro 10g

Addition:

On the ASRock TRX40 Creator the USB allocation seems to be different. I have a USB with Samsung SSD running on one of the USB ports next to the PS2 connector. (Port 15a / 15b)

On ASRock this is connected to one of the problematic starship USB Controllers either 05:00.3 or 22:00.3 so I can still use the other USB.

(But now that I'm thinking of this, what if passing the other USB still causes some issue with this one which in the end causes instability and freezes the host... If I find some time I will test moving the Proxmox disc to ASMedia and retry passing the USB ports.) 

On 5/31/2020 at 5:22 PM, Driftwood said:

 

So Unraid isn't going to be specifically any better than Proxmox on these trx40 Mobo chips too?

 

My system is stable but am probably going to use a USB stick to boot it now.

 

@All is there a quick way to back up my working  4gb sized ssd proxmox boot to a bootable USB?

 

If so, please type FULL & EXACT details below.

 

This will help anyone following this thread 

Thanks

 

 

I have read somewhere that running from normal USB stick is a bad idea because of write leveling. Something of wearing out your USB stick in a matter of months to a year. Related to saving of log files etc, tip there was to always use ssd storage.

29 minutes ago, iGPU said:

Following comments are FYI and have no immediate direct bearing on this build.

 

I'm researching various Linux distributions and came across this news:

Linus Torvalds has switched to using an MSI TRX40 Creator + 3970X, where the build is described here. Linus describes here using Linux 5.7-rc7.

This all would suggest that the TRX40 platform will have increasingly better Linux support.

 

I left Windows 10 running over night and zero issues, even after running many different stress tests. That led me to look for issues affecting Linux but not Windows:  Core 6 issues can do this. But after making those changes described here in BIOS (changing 'Typical current idle', from 'Auto' or 'Low current idle'), the same error messages and kernel panel with shutdown happened under Proxmox. So no fix for my CPU.

 

 

nice, in the long list of patches I see several iommu / amd patches... 👍🏼

9 hours ago, fabiosun said:

for me Catalina is working well as high sierra if I consider USB job

I pass all controllers less audio controller (xx:00.4)

In Catalina usb is threated differently and USB drivers work has they do in a window VM (as bare metal), in High Sierra driver is old and for now I can use only USB 2.0 device.

for both no lag at all

Pay attention (for lag stuff) if you have some WIFI/BT which need usb power)..I haven't ..but it is problematic if you have

 

Do you run any kext or other files for usb in opencore?