Proxmox VE OSX Guide discussion

[Driftwood]

17 hours ago, iGPU said:

What I still don't understand is why my particular CPU was originally working on Proxmox but then gradually became unstable to the point that I couldn't run Proxmox (and I've tried their latest v6.2).

 

I had similar problems with Proxmox, relating to Ethernet (and wifi before pulling the BT wifi) and getting Pavo to build a proper Dragon / Realtek 8125 driver for linux.  We noticed  the install of Proxmox 6.2 sometimes had issues and going back to 6.1 then updating to 6.2 later worked.

 

BIOS should be set to pretty much default UEFI settings on install. Later you can add things like disable CSM and Enable Above 4G.

Edited June 1, 2020 by Driftwood

[iGPU]

I'm still perplexed by the computer behavior.

 

After running ArcoLinux (I tried to get a VM going, but not so easy), I went back and did another install of Proxmox 6.2. It is now running and stable. What changed?

 

As I mentioned above, during install of ArcoLinux, it had option to install AMD microcode. I assumed that this was a software add-on, but now I wondering if ArcoLinux injected the CPU with microcode, and that updated microcode is allowing the CPU to work again with Prxomox. I ran Proxmox overnight, no errors were on host screen, and macOS still running (having timed-out and presented me with re-login screen). Meanwhile, MSI has not yet updated BIOS for TRX40 Creator mobo as they did for PRO 10G.

 

So while the CPU is working better with Proxmox, macOS is not stable, but I think another issue. If any apps are run (Safari, Hackintool, etc), the OS freezes after a few minutes; and this is running under console, no GPU pass-thru. I will do a fresh install of Catalina and see how that behaves.

 

[fabiosun]

@iGPU it is very weird all things have been happened with 2 different motherboard and different linux distros

I can't say what you can do more to debug..

I can only say that in my experiences (with all bios available on my motherboard (I have tested all during this experience)) I have no difference behaviour

IOMMU group are always the same as it is always the same group separation..

so it is difficult for me to understand

[Driftwood]

21 minutes ago, iGPU said:

So while the CPU is working better with Proxmox, macOS is not stable, but I think another issue. If any apps are run (Safari, Hackintool, etc), the OS freezes after a few minutes; and this is running under console, no GPU pass-thru. I will do a fresh install of Catalina and see how that behaves.

 

 

Ensure GPU (Radeon VII) is on PCIE lane 1 (16x), try NOT passing thru the (47:00.1/ .3s USB or whatever your USB/address with audio shared on - Starship/Matisse), my Mac crackles and is unstable a little after passing that one thru. Just use Asmedia and the other Starship USB.

 

Then tell me if you have improvement. 

[iGPU]

I did a fresh install of Catalina (no data migration) on the Sabrent Rocket drive. It is now working well.

 

So the macOS problem was due to having recently did a fresh install, but then a migration of data. The migration can bring instability of macOS, so not a good idea if there is a question of instability. But now I'm at work (my schedule this week returns to usual hours since lock-down), so no more testing until later tonightl.

 

In summary there are at least two separate problems:

1) macOS - solved with fresh installation without migration.

2) CPU issues

   a) seemingly resolved since doing a separate drive install of ArcoLinux, which possibly flashed AMD microcode to CPU.

   b) maybe mobo problems related to microcode (if true, then GB Designare mobo was prob okay, but now returned).

   c) no more errors reported in Proxmox since ArcoLinux install.

 

All testing with new Proxmox install/new Catalina install is done as mentioned in my previous posts using console and not passing GPU. Other items passed are two NVMe drives, BT/Wifi, Aquantia, ASMedia at 44:00 and Matisse USB at 48:00.

 

Tonight I'll pass both GPUs. Instructions on most TRX40 mobos are slots 1 & 3 (as these are the x16 pcie slots) and then monitor for stability.

 

Edited June 1, 2020 by iGPU

[Driftwood]

1 hour ago, iGPU said:

ArcoLinux, which possibly flashed AMD microcode

Interesting stuff this microcoding the cpu. Did you modprobe cpuid to check if your cpu was?

 

Detecting available microcode update

It is possible to find out if the intel-ucode.img contains a microcode image for the running CPU with iucode-tool.

1. Install intel-ucode (changing initrd is not required for detection)
2. Install iucode-tool
3. Load the cpuid kernel module:

   # modprobe cpuid

4. Extract microcode image and search it for your cpuid:

   # bsdtar -Oxf /boot/intel-ucode.img | iucode_tool -tb -lS -

5. If an update is available, it should show up below selected microcodes
6. The microcode might already be in your vendor bios and not show up loading in dmesg. Compare to the current microcode running grep microcode /proc/cpuinfo

 

[Driftwood]

1 hour ago, iGPU said:

Tonight I'll pass both GPUs. Instructions on most TRX40 mobos are slots 1 & 3 (as these are 16, not 8, slots) and then monitor for stability.

 

Thats correct. I get no problems with dual GPUs on 1 and 3 with x16 speeds. Everything is stable if I remove the Matisse 47:00 USB/Audio from passthru. Interestingly 47:00.1 is just the rear USB ports and a LED controller (which I don't use). The Matisse USBs on 47:00.1 very nearly work - but I get the occasional crackle in audio so its definitely not happy with firewire co-existence.

 

The firewire is best on lane 2 and Decklink on lane 4 (all SDI ports working) which only require x8 speeds. And this is what I liked about the ASRock Creator - the PCI lanes are evenly spaced apart with no 'touching' of the firerwire/Decklink to the GPUs, whereas other boards (inc expensive ones) lane 2 & 4 seem too tight to lanes 1 & 3.

Edited June 1, 2020 by Driftwood

[fabiosun]

@Driftwoodare you in 10.15.5?

[Driftwood]

Heres my config table for the ASRock Creator TRX40 running Catalina 10.15.5 @fabiosun

The only BIOS setup I use is CSM = Enabled for Proxmox install, and then Disabled with Above 4G checked when Im installed in Catalina.

 

 

* Note: You only need to passthru  one copy of the Starship USB 3 address (1022:148c) in your VFIO.conf

** I'm using two MP600 m2 drives on the Phison controllers

 

Im getting ready to pass SATAs tonight. There appears to be two sata controllers: So I need to discover which controller/address set handles port 1-4 & port 5-8. If I can pass the secondary controller I'll be happy to use the first as boot Proxmox.

Edited June 1, 2020 by Driftwood

[fabiosun]

@Driftwood47:00.3? Don’t use it?

[Driftwood]

4 hours ago, fabiosun said:

Driftwood47:00.3? Don’t use it?

 

No thats the USB Audio ports. I dont need it - when I do the mouse drag and crackle is worse for my desktop. I have tons of pro audio inputs/outputs on my RME Fireface 800 card connected to the firewire card. Plus 64 I/Os with the Decklink. 2with my DELL HDMI/DP monitor... so Im spoilt!

 

By the way @iGPU I did a Migration Assistant on my build after install and didn't migrate any of the network stuff (left uinchecked) which would screw the build.

 

ODDITY
What's weird is, my mobo wouldn't even display sleep on fresh install, now post Migration Assistant  it sleeps fine - going from screensaver to display sleep with wake on mouse/keyboard movement without problem.

Edited June 2, 2020 by Driftwood

[fabiosun]

Are you sure about this?

23:00.4 in my case is Matisse audio

46:00.1 and 46:00.3 are many usb 

I think internal 3.2 slot and a bunch of rear usb

[Driftwood]

18 minutes ago, fabiosun said:

Are you sure about this?

Yep. I went thru a complete USB/passthru check last night. I even had to correct some Pavo vfio addresses so I was glad I did. 

I'm 100% stable without 47:00.1 and 90% with it enabled (the odd audible crackle even though audio is not addressed in that space). If I enable 47:00.3 (with audio ports) I'm 60% stable and with mouse lag - even though its not on that port!

 

I'll upload my rear panel diagram shortly.  

Edited June 1, 2020 by Driftwood

[fabiosun]

47:00.3 interferes with audio but I think also without you loose some usb3 ports and maybe 4 internal usb2
in 10.15.5 passing all usb controllerS does not affect audio for me but I have understood that for you is different

[Rox67er]

On 5/31/2020 at 5:22 PM, Driftwood said:

 

So Unraid isn't going to be specifically any better than Proxmox on these trx40 Mobo chips too?

 

My system is stable but am probably going to use a USB stick to boot it now.

 

@All is there a quick way to back up my working  4gb sized ssd proxmox boot to a bootable USB?

 

If so, please type FULL & EXACT details below.

 

This will help anyone following this thread 

Thanks

 

 

I have read somewhere that running from normal USB stick is a bad idea because of write leveling. Something of wearing out your USB stick in a matter of months to a year. Related to saving of log files etc, tip there was to always use ssd storage.

[Driftwood]

 

sata port recognition is a joke. I plug in another drive on port 5 or 6 and the ata numbers change - including the proxmox boot on port 1. Random.... left pic with the extra drive, shutdown, pull it off and restart dmesg and the proxmox boot has change ata number (right pic). Guess Ill have to use disk-by-device...

Edited June 2, 2020 by Driftwood

[Driftwood]

1022                7901                AMD SATA Controller (derived from the x370 chipset)

1022                43B5                AMD SATA Controller (derived from the Ryzen CPU)

 

The above is the sata chipsets for the Taichi. Does anyone know what the controllers are doing in the Creator TRX40? 

Edited June 2, 2020 by Driftwood

[fabiosun]

8 hours ago, Rox67er said:

I have read somewhere that running from normal USB stick is a bad idea because of write leveling. Something of wearing out your USB stick in a matter of months to a year. Related to saving of log files etc, tip there was to always use ssd storage.

Agree!

i use ASMedia usb for proxmox with a Samsung 840 pro ssd. And this is the easy way to have the chance to pass trough all other controllers I have in my msi trx40 pro 10g

[Rox67er]

2 hours ago, fabiosun said:

Agree!

i use ASMedia usb for proxmox with a Samsung 840 pro ssd. And this is the easy way to have the chance to pass trough all other controllers I have in my msi trx40 pro 10g

Addition:

 

On the ASRock TRX40 Creator the USB allocation seems to be different. I have a USB with Samsung SSD running on one of the USB ports next to the PS2 connector. (Port 15a / 15b)

On ASRock this is connected to one of the problematic starship USB Controllers either 05:00.3 or 22:00.3 so I can still use the other USB.

(But now that I'm thinking of this, what if passing the other USB still causes some issue with this one which in the end causes instability and freezes the host... If I find some time I will test moving the Proxmox disc to ASMedia and retry passing the USB ports.) 

[fabiosun]

@Rox67erif port 8 is an ASMEDIA Port it seems pretty identical to my situation

Connecting  there Proxmox disk you can pass all others controllers (SATA,USB,VGA and so on) easily

This DO NOT MEAN you do not have lag problem If you passthrough all controllers (also problematic one like Audio /usb controllers), but you can pass both sata controllers and use well all disk connected to them as I do

 

 

[Rox67er]

37 minutes ago, fabiosun said:

@Rox67erif port 8 is an ASMEDIA Port it seems pretty identical to my situation

Connecting  there Proxmox disk you can pass all others controllers (SATA,USB,VGA and so on) easily

This DO NOT MEAN you do not have lag problem If you passthrough all controllers (also problematic one like Audio /usb controllers), but you can pass both sata controllers and use well all disk connected to them as I do

 

 

@fabiosun Correct, port 8 is ASMedia.

The difference to my opinion is that port 10a / 10b / 15a / 15b in my case are NOT connected to 47:00.1 / 47:00.3 but to one of the Starship USB Controllers. If I pass 47:00.1 and 47:00.3 to MacOS these ports remain connected to the host. (tested with mouse which still shows under "Node / Hardware / Add / USB device"

 

@Driftwood regarding Radeon VII reset bug, I found this post on a huge number of AMD pulls to kernel 5.5 (Proxmox is now on 5.4) so we may see some improvement to the GPU reset issue on next Proxmox version. https://www.phoronix.com/scan.php?page=news_item&px=Linux-5.5-AMDGPU-More-Features

[fabiosun]

ciao @Rox67erin my case ..and sorry I am not so precise in it but I remember this you can see here

[Driftwood]

2 hours ago, Rox67er said:

regarding Radeon VII reset bug, I found this post on a huge number of AMD pulls to kernel 5.5 (Proxmox is now on 5.4) so we may see some improvement to the GPU reset issue on next Proxmox version. https://www.phoronix.com/scan.php?page=news_item&px=Linux-5.5-AMDGPU-More-Features

Sounds promising. We wait in anticipation.

[cj750]

From follow the post. It best to install proxm. On to a SSD and to also leave the iso on them. And leave my M2 drive as scratch drives to run other programs? Am I correct in this logic of setting this up. To use my 2tb ssd as the installer drive and iso hub?

[fabiosun]

my advise is to install proxmox on a minimal SSD disk connected on USB type c (ASMedia)

this escamotage will allow you to pass easily all others controllers you have like NVME, SATA, USB (USB with all caution you must have also reading entire this thread)

In my particular case I have all sata disk connected and two NVME passed happily..

I think also for your creator could be the same

 

then everyone must adopt best way for his pipeline...and for now..if you follow what I have written here and in the guide..with latest 10.15.5 you have also a perfect working internal audio..

Only with latest 10.15.5 because with previous version scratch could appear some time...but it is a driver problem with our "weird" usb audio (two chip for this task)