[GUIDE] TRX40 OSX Bare metal Proxmox setup(6.2-1) Updated 25-07-2020

[fabiosun]

 

Preface:

 

Inspired by the lack of patches for this new ThreadRipper CPU series , I have started to find a way to have "the fast Amd hack ever".

Hard task was to setup a  pretty bare machine inside an OSX linux vm.

Many solutions were evaluated to reach this important goal.

Ubuntu, CentOS, UnRaid and so on

Tested many VM and found many interesting stuff on the net.

Kholia's or foxlet's GitHub guide for example or nicksherlock's guide more Proxmox like.

 

With these guide is possible to start our job and to have a working OSX..but I need more! 😉

 

I want to use my single Nvidia Gpu on it and for it I need a different approach.

Single GPU usually is a pain because when we passthrough* it to the guest PC..we will loose it.

So, we have to use another PC to connect in it or a simple Android or iOS tablet (now I am use my iPhone for this task 🙂 )

So, after some days of brainstorming with @tomnic we choose to start with Proxmox VE.

It seemed the most flexible linux distro to reach my goal.

macOS High Sierra and Nvidia web driver installed with no compromise.

Proxmox Ve allows to build a bare metal system with Gpu passthrough, and, most important thing it has a web interface to control guest PC

 

* Passthrough is a technique to "pass" hardware to a virtual machine loosing only a minimal part of hardware performance

More difficult and critical task is passthrough our hardware to VM in a way which will allow me to have a pretty similar performance as a real hack and overall to have GPU at its best in this system.

 

Prequisites:
 

• A minimal linux experience (I have had no idea before about linux and this was the main difficulty for me)
• To know what to search usefull for my need!
• A OSX, linux or windows system to collect some files (I have used used linux and OSX, with latest one preferred.

 

System configured is in my signature, summarized here:

MSI Trx40 10G Pro
AMD 3970x Threadripper 32 cores / 64 threads (this guide is also useful for 3960x and maybe for 3990x (capping cores not supported by OSX kernel)
1Tb Nvme Sabrent
and so on...

 

Let's start!

 

Mandatory Elements:

 

0) Proxmox VE (Proxmox Ve's wiki link)

1) a working OSX. Linux,  Windows ( in this guide we use OSX High Sierra 10.13.6

2) download Vm_assistant's script, an exclusive script by macOS86.it  and @Gengik84

 

VM assistant

From this link a simple guide:

 

 

Promox VE installation procedure:

 

In this guide I will not do a detailed guide to install this linux release, you can find useful information on:

link to Proxmox

link to iso installer

 

Use your favorite Iso installer to produce a bootable USB pen drive  (8gb or more is fine)

In this guide I have used Balena Etcher:

link Balena Etcher

 

Warning for initial installation part! *

 

Due a problem with new TRX40 motherboard , ProxMox 6.1 and lower will hang on early booting stage.

To avoid this you have to modify grub.cfg file in this way:

find this line:

linux   /boot/vmlinuz-5.3.10-1-pve root=/dev/mapper/pve-root ro  quiet

with this

linux   /boot/vmlinuz-5.3.10-1-pve root=/dev/mapper/pve-root ro  quiet mce=off

 

adding mce=off allows us to start this exciting journey....

 

Alternatively you can modify this line directly on boot menu selection before selecting your choice (press E and add mce=off as shown before) thanks @KTC for pointing me in the right direction

* this problem could be resolved with a future Proxmox update SOLVED in Proxmox 6.1-3 and greater

 

step 1:

boot from your ProxMox Ve USB pen drive, you will have:

 

select Install Proxmox VE and then follow indication you can see below.

Note

Fill your network data with your (setup should do it for you except for Domain and your email)

 

 

 

 

 

 

 

Installation is done! 😛

 

Reboot and choose your usb pen drive and boot from it.

 

In a few second you will see:

 

Now pay attention..if you have not modified grub.cfg...you MUST do it now pressing E on keyboard and adding manually mce=off as previously shown.

You have also to remember to do it permanently when we start in Proxmox VE first time.

 

 

 

At login user is root

password (your previous choice during Proxmox installation)

 

now it is possible to be a bit worried..if you are (as I was) not confident with linux command)..but I will try to stay more simple I can.

 

Start to find googling how to read or to modify some files in linux (text configuration file I mean).

I have used nano command..you can also use vim or similar command.

 

 

Now, take note of link you can see in this picture: https://192.16.1.92:8006  (yours could be different) and the magic starts : )

 

[fabiosun]

Now, with our Threadripper PC on we have to use another pc or tablet (also phone) connected in the same network, cable or wi-fi is fine

Open your browser and digit the address noted above and you will see Proxmox web interface login mask:

 

 

For the VM configuration part we will spend some time here

Login with user name root and password your previously inserted password and you will see:

 

 

Proxmox has a main subscription way to have a more business support, you can click simply on ok to go on.

 

 

This is the main page.

You can see I have configured 3 VMs and now we start to configure new one.

 

 

you should have this page empty only pve node and local and local-lvm voices 

 

See on the storage 'local' pn node "pve" the upload button.

This is the place where we upload things useful like OSX installer or clover iso to use in installation process.

 

We need of new shining script made by @Gengik84 for macOs86.it users to simplify some process, you can reach similar result using other script but this is very simple to use.

So download VM assistant from

and execute it:

 

 

choose system you want to install (you need an internet connection for all this process and also for the installation)

In my case HighSierra.

choose 0

 

VM Assistant terminal options:

-------------------------------------------------------------
Usage:

Show this help page:

/Users/fabio/Desktop/VM_Assistant-2 -h
-------------------------------------------------------------

Download latest clover for VM:

/Users/fabio/Desktop/VM_Assistant-2 -dc 
or 
/Users/fabio/Desktop/VM_Assistant-2 --downloadclover

-------------------------------------------------------------
Convert Installer.app to ISO:

/Users/fabio/Desktop/VM_Assistant-2 -cti 
or 
/Users/fabio/Desktop/VM_Assistant-2 --apptoiso 

-------------------------------------------------------------
Download Catalina full Installer:

/Users/fabio/Desktop/VM_Assistant-2 -dcata 
or 
/Users/fabio/Desktop/VM_Assistant-2 --downloadcatalina

-------------------------------------------------------------
Download Mojave full Installer:

/Users/fabio/Desktop/VM_Assistant-2 -dmoja 
or 
/Users/fabio/Desktop/VM_Assistant-2 --downloadmojave

-------------------------------------------------------------
Download HighSierra full Installer:

/Users/fabio/Desktop/VM_Assistant-2 -dhs 
or 
/Users/fabio/Desktop/VM_Assistant-2 --downloadhighsierra

-------------------------------------------------------------

 

when all is done you can find on your desktop a new folder named Apple_DMG with inside:

10_13 folder, hs.iso and clover1.iso files (their name could be different).

You can now upload them via web interface pressing upload button.

In this place you have to upload all iso useful to create a VM (also windows.iso or other linux iso if your goal is to instal others system)

Osx needs of clover1.iso and hs.iso (boot loader and Osx installer), nothing else!

 

 

[fabiosun]

and now....Virtual machine creation.

 

 

Create button, start to fill all data in the same way I did.

Put a flag on Advanced, name your VM in my case macOS86 and go on

 

 

choose local in Storage tab, clover.iso in ISO image and Other in Guest OS type, next.

Now important page (system) .

We come back here to the last part of this guide to configure GPU passthrough , for now in Graphic card us VMware compatible, in BIOS OVMF (UEFI) in Machine Q35 in storage local-lvm, then next Hard disk section.

 

Here I have configured in a simple way, pretty similar to other guide, but it is possible to apply many solutions based on your skill to configure this beast...

Bus/device SATA cache write back Unsafe, Storage local-lvm, size 32 (as you want), flag on SSD emulation.

 

 

press next to go to CPU part.

In a VM cores and thread are the same (not sure but..hey..I am not an expert 😛 )

so fill as following image, Sockets 1, Cores 64, Type Penryn, flag on enable NUMA, leave default all other parts.

 

 

You can also try others CPU type but no different results. In a  windows VM you can use Host as Processor type.

In OSX no! (not really true..we discover it later)

next to the memory part.

Here you define memory passed to th VM.

It is not possible to pass all memory to VM, in my case I can pass about 125 Gb*, in this VM I will use 64 GB (65536 Mb)

*less is better, now I am using 118 Gb

 

 

Next network part.

This is very important because it will be mandatory to complete Osx installation (if you use BaseSystem.dmg ReinstallOS method)

I use for now VMware vmxnet3 and leave the rest as is.

 

 

next step review all your selections in confirm tab and If all is ok finish it.

 

 

Now we have to modify some stuff in a terminal shell so click on node and to shell as shown in this following pictures:

 

 

now we have to passthrough our keyboard and mouse (I have it connected via USB port)

In hardware tab select add and see following pictures:

 

 

 

do the same for usb mouse.

Important note:

disable USB 3 flag otherwise usb are not working for this

 

now..the linux part of VM creation.

[fabiosun]

We have to modify manually some part of our VM configuration files and this is stored in /etc/pve/qemu-server/<VM id>.conf file.

We have to add as boot arg OSK key and other cpu features not present in old Penryn CPU.

So, open a shell:

 

 

I repeat myself, node name for me is pve, VM id is 103..for your system could be different.

you wìll have:

 

 

now insert this: (Nicholas Sherlock's site)

args: -device isa-applesmc,osk="ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc" -smbios type=2 -cpu Penryn,kvm=on,vendor=GenuineIntel,+kvm_pv_unhalt,+kvm_pv_eoi,+invtsc,vmware-cpuid-freq=on,+pcid,+ssse3,+sse4.2,+popcnt,+avx,+aes,+xsave,+xsaveopt,check -device usb-kbd,bus=ehci.0,port=2

 

then for a base system reinstall installation:

ide0: local:iso/hs.iso,cache=unsafe
ide2: local:iso/clover1.iso,cache=unsafe,size=256M

or to a full dowloaded iso installer (with the help of VM_Assistant:

 

ide0: local:iso/MacOS_Installer.iso,media=cdrom,size=9G
ide2: local:iso/clover1.iso,cache=unsafe,size=256M

to have:

 

 

or:

 

to save press ctrl x / y return

now you can select your vm and select Start.

 

 

 

Now we have to do some basic stuff to have the chance to start an OSX VM.

Follow instructions here to do it simply  (Nicholas Sherlock's site):

link to OVMF patch **

** In my system is possible to boot in the same way with clover or OpenCore, without patching this deb file

reverting to a not patched file seems to solve also my problem of maintaininga correct boot order in bios settings 

I leave this step on this guide but you have to verify by yourself if you need or less of this patch (think no...but..it is up to you!)

 

*NOTE*

there is a bug , it produces on cold boot weird  video result . 
Proxmox dev have solved this in a new commit!
https://forum.proxmox.com/threads/ovmf-screen-resolution-bug.55418/

 

after that, you can press start on Proxmox web interface to launch the OSX setup process.

 

In this part pay attention to screen resolution (we learn how to modify it if you have scrambling multiple resolution on your display )

You have to match config.plist resolution with OVMF bios resolution pressing F2 to key when you see Proxmox Ve logo on screen*

 

Added on April 9, 2020:

In opencore bootloader scrambling graphics is not present at all..so maybe it is a clover bootloader problem

 

 

press F2 here

then click on Device manager and choose:

 

 

Then verify if Preferred resolution at Next Boot and changed Preferred match.

 

 

If they do not match open selection and choose the same resolution:

 

 

Then press F10 (save settings)

and commit changes.

Reset, now you should see clover boot menu

If you see scrambled test or resolution restart computer (clover button) and enter again the procedure above (Opencore solves this problem)

 

If all went fine you can start to install OSX.

 

 

choose usual bootmacOS install icon and...Gooooooo...

Note: if you want to see verbose mode press O and insert -v option
also pressing spacebar you can choose -v flag

your choice 😉

 

 

Ok after few minutes you should see:

 

 

used here reinstall method..

with latest VM assistant script you can convert your previously downloaded install macOS app  and  use a full installer to do installation (more faster and useful if you have a no fast internet connection)*

To download a full installer you can use this app:

runme (a script based on Greg Neagle job)

*now it is possible to use new version of VM Assistant for this task

So select DiskUtility and format VM disk in usual way:

 

 

 

then close Disk Utility and start to instal/re-instal as usual

Here it is mandatory to have a working ethernet connection if you use reinstall option (base system.dmg based)

 

After some time system reboot and second steps of installation start.

After some reboot (the same steps you have in a real hack) you should see login initial option configuration phase.

 

next..some important part for the main goal of this guide..

GPU passthrough!

 

[fabiosun]

Useful link.

 

Proxmox wiki

 

In this site you can find a tons of useful informations.

So read it to clear doubts you can have.

 

Now some useful informations to start thinking to PCI Passthrough.

 

from Proxmod wiki site:

Modify grub for your cpu:

first step enable the IOMMU.

to do this open a shell from Your Proxmox's web interface:

type nano /etc/default/grub

insert in it "amd_iommu=on" without quote as you see here:

 

 

ctrl x then Y to save

then do not launch upgrade-grub but pve-efiboot-tool refresh

then reboot, to verify all is fine, open shell again and type:

dmesg | grep -e DMAR -e IOMMU

you have a pretty similar output if all is fine..otherwise repeat step above:

 

 

you can add also iommu=pt to improve some performances in PCI device not in use by your VM.

 

Now another very important part. We need some modules to load, by default they are not loaded:

 

type: nano /etc/modules and add these modules:

 

 

"Verify also IOMMU isolation (follow Proxmox wiki just in case) (follow my situation which may differ from your)"

 

"To have separate IOMMU groups, your processor needs to have support for a feature called ACS (Access Control Services)."

 

now the tricky thing:

Determinate your GPU address:

 

in shell type:

lspci

 

and check for your graphic card address:

 

 

this is my GFX address, then type:

lspci -n -s 21:00

you will see:

root@pve:~# lspci -n -s 21:00
21:00.0 0300: 10de:1b02 (rev a1)
21:00.1 0403: 10de:10ef (rev a1)
root@pve:~# 

Create a file in etc/modprobe.d/ named vfio.conf:

echo "options vfio-pci ids=10de:1b02,10de:10ef" > /etc/modprobe.d/vfio.conf and add on it:

 

options vfio-pci ids=10de:1b02,10de:10ef disable_vga=1

(ctrl x then y)

 

then you have to blacklist the drivers.. PAY ATTENTION and save it , when you reboot after this step it could be possible to have to operate exclusively from Web Pc interface:

so

type:

echo "blacklist radeon" >> /etc/modprobe.d/blacklist.conf 
echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf 
echo "blacklist nvidia" >> /etc/modprobe.d/blacklist.conf

reboot your pc

Note.. remember..we have set our VM bios in this way:

bios: ovmf

GFX need to be UEFI supported visit this link to understand other important stuff just in case:
http://vfio.blogspot.fr/2014/08/does-my-graphics-card-rom-support-efi.html

 

then to avoid some crashes add this line in kvm.conf:

echo "options kvm ignore_msrs=1" > /etc/modprobe.d/kvm.conf

also useful to avoid warning messages in your system log this:

echo "options kvm ignore_msrs=1 report_ignored_msrs=0" > /etc/modprobe.d/kvm.conf

as advice check always that command went fine in this way ie:

 

for now this is an important step, now we have to access to our previously vm created and add "The passthrough"

 

add PCI Device and choose your GFX:

 

 

in my case it adds this:

 

and this is enough 😉

to summarize..before rebooting check this files:

grub.cfg
blacklist.conf
kvm.conf
vfio.conf

 

and see if all stuff is there.

After reboot in your Threadripper you will see initial Proxmox VE Boot menu (blue one page)

and the only two lines (loading stuff)

If not check again all steps above.

Remember some Mandatory parameter:

 

1) mce=off in grub

2) all conf files set properly for your system/GPU

and so on 😛

 

If you see only those two lines..launch your vm in your Web interface or via terminal command :

qm start 100

 

and see the log:

 

 

this is good 🙂 (warning message disappear if you do no t use +pcid in VM conf boot args)

 

this no 😛

 

 

Any different case could be possible...

remember to check Proxmox wiki if some different problems appear!

For now is enough..I will check this guide and if you find some incongruence fill free to declare.

 

added 24-04-2020

The above GPU configuration is perfectly calibrate for my system and my Nvidia TitanXp GPU

I have asked @Imanol to describe procedure for a different GPU (AMD 5700XT).

He had success following this guide and now he is helping me to show you how he did with his Navi card.
this is working in his system, maybe you have to adapt it for your system.

 

-------

I'll share some tips to get the 5700XT GPU passthrough working. This can be helpful for other people using AMD GPUS. I had to do some extra things:

 

The default GPU BIOS doesn't have UEFI enabled. Instead of updating the firmware, which is risky (or non existent tools) , you can download a ROM with UEFI support and let proxmox load/expose it when starting the VM. In my case I downloaded this rom (https://www.techpowerup.com/vgabios/212439/gigabyte-rx5700xt-8192-190616). You have to copy it to:

/usr/share/kvm/5700xtcustom.rom

and specify the ROM file in the GPU pcie passthrough line vm conf:

 

hostpci1: 03:00,pcie=1,x-vga=1,romfile=5700xtref.rom

 

Additionally, I had to enable some extra flags in clover or it didn't boot correctly.

 

Make sure that you pass the agpmod=pikera boot argument. In graphics settings enable RadeonDeinit. Make sure that InjectATI is correctly disabled. If you are using clover configurator press the checkbox twice, otherwise it's enabled by default.

 

 

 

I forgot to mention that to make my 5700XT passthrough work I needed to unbind some system drivers and framebuffers before launching the GPU. Just blacklisting the GPU drivers, enabling nomodeset in Grub or setting GRUB_GFXMODE_LINUX=text in grub didn't work for me. Those unbinds need to be done each time you restart the computer, before launching the VM. I automated it using the linux crontab utility.

 

First open crontab config using this command in a proxmox terminal:

crontab -e

 

Add this line into the document and save it:

 

@reboot /root/unbind.sh

Create a script in the specified path:

 

nano /root/unbind.sh

Add those unbind command (Change 03:00 with the PCI address of your GPU):

 

#!/bin/bash
echo "0000:03:00.0" > /sys/bus/pci/drivers/vfio-pci/unbind
echo 0 > /sys/class/vtconsole/vtcon0/bind
echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind

Don't forget to set the correct permissions for the script after saving:

 

chmod 755 /root/unbind.sh

Now, each time you restart the computer proxmox will run this script and the guest mac VM will launch without any conflict.

-------

 

 

Important stuff:

macOS86.it is a free site and people here are helpful if tone remain in a calm way..Any different approach will not be tolerate .

 

Now some kudos:

 

to algrey and @Shaneee for their work to understand if there is a way to have this new chip booting in a real hack (for now not possible at all)

 

best people here @Gengik84 to host this and solving many problem for us

@Ciro82, @carlo_67 @iCanarofor their invaluable passion and help

 @tomnic who is corresponsible of this "success"..we can define this hack..."The first AMD Hack fast then ever"

 

Have a nice life

 

 

 

[fabiosun]

 

Optimizing our VM in Proxmox VE (Proxmox wiki reference):

 

Add an existing physical disk to OSX's VM

 

Open Web interface and open a shell window. If you do not have lshw terminal command type:

 

apt-get install lshw

 

to install it, then type:

 

lshw -class disk -class storage

 

to see disks attacched to our Threadripper PC

You will see a list of useful data for your disk:

*-disk:1
          description: ATA Disk
          product: ST10000VX0004-1Z
          physical id: 1
          bus info: scsi@5:0.0.0
          logical name: /dev/sdd
          version: AV01
          serial: ZA26YT4A
          size: 9314GiB (10TB)
          capabilities: gpt-1.00 partitioned partitioned:gpt
          configuration: ansiversion=5 guid=52a25657-f7d6-45a2-b0b4-07337d277d16 logicalsectorsize=512 sectorsize=4096

I would like to attach this disk, it is a windows disk formatted in exfat.

type this command using serial number you see above:

 

ls -l /dev/disk/by-id | grep ZA26YT4A

 

output in my case is:

root@pve:~# ls -l /dev/disk/by-id | grep ZA26YT4A
lrwxrwxrwx 1 root root  9 Apr  5 14:58 ata-ST10000VX0004-1ZE101_ZA26YT4A -> ../../sdd
lrwxrwxrwx 1 root root 10 Apr  5 14:58 ata-ST10000VX0004-1ZE101_ZA26YT4A-part1 -> ../../sdd1
lrwxrwxrwx 1 root root 10 Apr  5 14:58 ata-ST10000VX0004-1ZE101_ZA26YT4A-part2 -> ../../sdd2

 

now we can add it to our vm in this way, my VMid is 100, change the command for your id and disk:

qm set  100  -scsi2 /dev/disk/by-id/ata-ST10000VX0004-1ZE101_Z852YT4A

if all went as it have to be output will be:

update VM 100: -scsi2 /dev/disk/by-id/ata-ST10000VX0004-1ZE101_Z852YT4A

open from shell your VMID.conf, in my case 100.conf and you should see two pending operation (I renamed scsi2 to sata x.

Shutdown VM (no restart) and then start it again

 

In disk utility you have now your new attached disk.

 

[fabiosun]

 

[fabiosun]

Passing completely NVME Samsung pro controller:

01:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961 [144d:a804]

 

After rebooting we will have:

 

 

In my case this one is a NTFS windows 10 drive..noew is it possible to use it also in OSX as usual

[fabiosun]

Now it is possible to boot also with opencore 0.57 bootloader! 🙂

With opencore is now possible to boot fine in every condition without weird graphics if OVMF resolution mismatch of bootloader resolution.

With clover bootloader problem was there

 

 

Added April 14 2020

 

now also using with a graphical interface

 

thanks @carlo_67

 

 

[fabiosun]

Cold boot with Proxmox, OC 0507

Use OC to avoid graphics problem without any workaround!

 

https://youtu.be/SribJXSeAaU

[fabiosun]

Now a tricky part if you are not confident as I am  not with linux..

 

If you have followed this guide you have (maybe ) a working VM and you are experimenting things..

One of the most important think in an OSX installation is bootloader EFI folder and its content.

 

Now when you upload bootloader iso files in Proxmox you have an iso extension but it is a sort of img file converted in iso, not accessible in a simple way, so, if you need to modify it or change something inside it is hard to achieve  (rebuild a proper Efi structure with @Kholia script or similar method with all things you need (ssdt, kext, efi drivers, config and so on adapted for your particular pc)

We learn here how to transfer a file from guest to host and vice versa.

Thanks @tomnic to point me in right direction with very useful example.

 

first part:

Copy a file from OSX folder to host linux Host root via terminal windows in osx:

 

ie 4428.iso file I have in /Users/fabio/Desktop/fab_iso path

 

I would like to copy in:

host root

 

my host address is:

192.168.1.97

 

open terminal windows and type:

scp /Users/fabio/Desktop/fab_iso/4428.iso root@192.168.1.97:/

insert your root linux password and you will have this:

root@192.168.1.97's password:*******
4428.iso                                                                                                         100%  256MB 212.7MB/s   00:01    
iMac-Pro:~ fabio$ scp /Users/fabio/Desktop/fab_iso/4428.iso root@192.168.1.97:/root

Open a shell in linux via Proxmox web interface and in it type dir:

file is there !

 

try to do the same by yourself from host to VM guest.

 

just in case.... 😉

 

iMac-Pro:~ fabio$ scp root@192.168.1.97:102.conf /Users/fabio/Desktop/fab_iso/
root@192.168.1.97's password:
102.conf                                                                               100%  858   227.8KB/s   00:00    
iMac-Pro:~ fabio$

 

 

[fabiosun]

 

[fabiosun]

ehmmmmmm

I discover a thing...

take a look 🙂

[fabiosun]

ehmmmmm two 🙂

I hope I do not fry it 🙂

[fabiosun]

+pcid as args in Penryn cpu type produce this:

 

without it will be:

 

In both cases OSX VM starts fine with all its functionality

[fabiosun]

 

OpenCore 0.57 graphical boot menu in VM environment 🙂

thanks @carlo_67 for pointing me in the right direction!

 

[fabiosun]

args: -device isa-applesmc,osk="ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc" -smbios type=2 -cpu Penryn,kvm=on,vendor=GenuineIntel,+kvm_pv_unhalt,+kvm_pv_eoi,+invtsc,vmware-cpuid-freq=on,+ssse3,+sse4.2,+popcnt,+avx,+adx,+avx2,+bmi1,+bmi2,+fma,+f16c,+tsc_adjust,+tm,+tm2,+aes,+xsave,+xsaveopt,check

Boot args I am using now..I loose a little performance then before in Catalina..but I am not concentrated on it because my goal is High Sierra

[fabiosun]

some words about PCI,USB and SATA passthrough.

 

Open a shell window in your host pc ant type lspci, you will have a similar list of useful data:

 

04:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller
21:00.0 VGA compatible controller: NVIDIA Corporation GP102 [TITAN Xp] (rev a1)
21:00.1 Audio device: NVIDIA Corporation GP102 HDMI Audio Controller (rev a1)
23:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller
23:00.4 Audio device: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller
43:00.0 USB controller: ASMedia Technology Inc. Device 3242
44:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
45:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
46:00.1 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
46:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
47:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
48:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
49:00.0 Ethernet controller: Aquantia Corp. AQC107 NBase-T/IEEE 802.3bz Ethernet Controller [AQtion] (rev 02)

this is a short selection of groups and data you should see in your shell.

You can see USB controllers, audio controller, via controllers and so on

With web interface we have a graphical way to insert this important stuff to pass controller completely.

Web interface has a limitation..it is possible to pass only a limited number of  PCI controllersf.

It is simple to by pass this limit using shell editing.

Web interface add:

 

 

 

or to avoid 4 PCI limit also you can type in your linux shell:

root@pve:~# nano /etc/pve/qemu-server/100.conf

and you should see:

 

You can compare easily with previous data in my linux shell (lspci) and see my "passed" stuff for now...

This can work easily if groups are separated and well defined (search on Proxmox wiki for this), our new TRX40 shouldn't have many problems in this

 

[fabiosun]

another step reached:

Passed all except one USB controllers (ASmedia and two AMD one)

ù

 

Also passed Intel lan and Aquantia, deleted in config net0:

vmxnet3=12:83:7D:28:45:26,bridge=vmbr0,firewall=1

 

 

to solve/adjust:

Starting VM now null the chance to reach my host linux..I can navigate and all other functionality but..maybe I have to bridge something 🙂

 

 

 

[fabiosun]

00:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex [1022:1480]
00:00.2 IOMMU [0806]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU [1022:1481]
00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
00:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 61)
00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 0 [1022:1490]
00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 1 [1022:1491]
00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 2 [1022:1492]
00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 3 [1022:1493]
00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 4 [1022:1494]
00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 5 [1022:1495]
00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 6 [1022:1496]
00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 7 [1022:1497]
01:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961 [144d:a804]
02:00.0 Non-Volatile memory controller [0108]: Phison Electronics Corporation E12 NVMe Controller [1987:5012] (rev 01)
03:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
04:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
04:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller [1022:148c]
20:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex [1022:1480]
20:00.2 IOMMU [0806]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU [1022:1481]
20:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
20:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
20:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
20:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
20:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
20:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
20:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
20:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
20:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
20:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
21:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP102 [TITAN Xp] [10de:1b02] (rev a1)
21:00.1 Audio device [0403]: NVIDIA Corporation GP102 HDMI Audio Controller [10de:10ef] (rev a1)
22:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
23:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
23:00.1 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP [1022:1486]
23:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller [1022:148c]
23:00.4 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller [1022:1487]
40:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex [1022:1480]
40:00.2 IOMMU [0806]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU [1022:1481]
40:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
40:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
40:01.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
40:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
40:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
40:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
40:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
40:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
40:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
40:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
40:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
41:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57ad]
42:02.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a3]
42:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a3]
42:05.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a3]
42:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a4]
42:09.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a4]
42:0a.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a4]
43:00.0 USB controller [0c03]: ASMedia Technology Inc. Device [1b21:3242]
44:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03)
45:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03)
46:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
46:00.1 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
46:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
47:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
48:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
49:00.0 Ethernet controller [0200]: Aquantia Corp. AQC107 NBase-T/IEEE 802.3bz Ethernet Controller [AQtion] [1d6a:07b1] (rev 02)
4a:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
4b:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
60:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex [1022:1480]
60:00.2 IOMMU [0806]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU [1022:1481]
60:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
60:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
60:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
60:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
60:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
60:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
60:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
60:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
60:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
61:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
62:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]

 

To your thoughts 😉

 

[fabiosun]

Discussion, benchmark, help request and improvements for this guide could be posted here:  https://www.macos86.it/topic/2675-proxmox-ve-osx-guide-discussion/

 

 

[fabiosun]

In this part we will deepen the passage of some peripherals integrated on motherboard such as the AQC107 NBase-T/IEEE 802.3bz Ethernet card and the two Intel Corporation I211 Gigabit network cards. The first one is directly supported by macOS HighSierra onwards. Instead for the Intel i211 a kext downloadable in the download section of our site will be used. Also download the dcpimanager application to verify that the ID of your network card is included in the same kext

Our initial friends as usual is Web interface shell.

Open it and type:

lspci

in the following part I will put only a useful selection for this part of the guide so I have to find network card useful data:

 

root@pve:~# lspci
............................
44:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
45:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
............................
49:00.0 Ethernet controller: Aquantia Corp. AQC107 NBase-T/IEEE 802.3bz Ethernet Controller [AQtion] (rev 02)
............................
root@pve:~# 

 

Now we can edit our VM config file and insert on it passthrough data:

 

hostpci6 49:00
#Aquantia
hostpci7 44:00
#I211 (first one)
hostpci7 45:00
#I211 (second one)

Save your config as usual (ctrl x--->y--->send)

 

Before rebooting your System and relaunching modified VM, download i211 ethernet kext from here:

Insert it in your kext folder and, if you use OpenCore boot loader you have to declare it in config .plist

Close your VM and restart it again.

You should see in your system report all your Ethernet card available in your system.

 

 

[fabiosun]

I think in this Covid 19 time many people have to be in their home to respect some restrictions all we have to hope in a happy end

In my case I have spent my time to this task..The 3970x AMD guide.

I have started using my NVME Sabrent disk as container of all VMs (VN files) and also for Proxmox installation.

Yesterday I have decided to do it better

Using a different disk for Proxmox and basic iso upload and passing completely to Vm my Nvme Sabrent to have a "vanilla" OSX High Sierra on it

 

So, I have formatted it and read again my guide to proceed in this task.

 

I have discovered many things , ie many parts of this guide could be skipped but I have decided to leave in place because maybe in different situations it could be useful to approach it in this way.

I noticed that I can boot easily without blocking and blacklisting many stuff..but it is advised in many "regular" guide to follow previous guideline so do as you think is better for your system.😂

 

So now I have a 840 pro ssd disk (256 Gb) for proxmox and clover/opencore iso

Sabrent NVme 1 Tb drive for a vanilla OSX installation.

 

why?

I do not believe this system could work better in a near future of this, if Apple does not insert in its kernel a proper AMD support..but if they will do I will have OSX as a real Mac has and I will update it to the next generation AMD OSX macOS 🙂

now some datas:

02:00.0 Non-Volatile memory controller: Phison Electronics Corporation E12 NVMe Controller (rev 01)

 

[fabiosun]

Others little optimisation for my rig

 

 

 

Proxmox disk connected to USB 3.2 (2x2 typeC)

 

[fabiosun]

Stimolated from this @pavo user message here:

 

I did further investigation on my problem (boot order)

Now it is working.

 

Problem was on patched OVMF bios in my case (I am investigating so I can't be definitive in this aspect, you can try by yourself)

I reverted to a non patched one and now I have correct boot order.

If someone else could try I will update my guide consequently 🙂

 

now I have this :

 

 

so no more qemu-drive (virtual disk on file present 😉

 

in my case no need to point to a bootx64.efi file in OVMF bios