Catalina Installation problem on TRX40

[jasin755]

Hello.

I can't install macOS Catalina. I have tried two tutorials. This:

 and this: 

https://www.nicksherlock.com/2019/10/installing-macos-catalina-10-15-on-proxmox-6/

 

I success boot to Clover and select install macOS. But next screen always freeze with this error:

 

Here are my VM settings:

 

and config: 

 

args: -device isa-applesmc,osk="ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc" -smbios type=2 -cpu Penryn,kvm=on,vendor=GenuineIntel,+kvm_pv_unhalt,+kvm_pv_eoi,+invtsc,vmware-cpuid-freq=on,+pcid,+ssse3,+sse4.2,+popcnt,+avx,+aes,+xsave,+xsaveopt,check -device usb-kbd,bus=ehci.0,port=2
balloon: 0
bios: ovmf
boot: cdn
bootdisk: ide2
cores: 48
cpu: Penryn
efidisk0: local-lvm:vm-100-disk-1,size=4M
ide0: local:iso/Catalina-installer.iso,cache=unsafe
ide2: local:iso/clover-r5107.iso,cache=unsafe,size=256M
machine: q35
memory: 40960
name: macOS86
net0: vmxnet3=56:DF:B4:F2:1A:D5,bridge=vmbr0,firewall=1
numa: 0
ostype: other
sata0: local-lvm:vm-100-disk-0,cache=unsafe,size=32G,ssd=1
scsihw: virtio-scsi-pci
smbios1: uuid=9eb56a09-214e-461b-b033-47dab50757e6
sockets: 1
unused0: local-lvm:vm-100-disk-2
usb0: host=05ac:0250
usb1: host=046d:c07e
vga: vmware
vmgenid: b703524d-08cf-4404-9292-9fb395e6879

 

Where can be problem? 

 

Thanks for help.

 

Edited April 23, 2020 by jasin755

[fabiosun]

it seems you have no patched OVMF deb file

 

also you have to protect it for future upgrade of it

[jasin755]

I patched OVMF by this tutorial https://www.nicksherlock.com/2018/04/patch-ovmf-to-support-macos-in-proxmox-5-1/ for 6.1 of course 😞 

 

PS: I have tried Hight Sierra and Mojave same problem.

 

Edited April 23, 2020 by jasin755

[fabiosun]

hi @jasin755

so situation is a little better than you have pm me.

now we have a system booting in clover boot menu

then it hangs in ++++++++ stage

If you are 100% sure you patched your OVMF like nick's site teaches to us the only thing to try is:

during clover boot menu

press spacebar

then select disable Nvidia (nv_disable=1)

and see if something different happens

[jasin755]

Hi.

With nv_disable=1 I got same error 😞 

 

About OVMF patch. I tried again and got message:

Unpacking pve-edk2-firmware (2.202002-1) over (2.202002-1) ...
Setting up pve-edk2-firmware (2.202002-1) ...

 

I thing it confirms that patch has already installed.

 

[jasin755]

Ok I found little clue. When I create new VM with 4 cores and 4GB RAM it was started working. When I change params to 48 cores and 40GB RAM. It will freeze.

 

Edit:

Problem is in number of cores. What it exactly mean? Number of physicals cores on number of threads?

 

Edit2:

I tried set to 24 (its my count of physical cores and it is freeze again.

 

Edit3:

16 cores i maximum for me. I don't know why.

 

Edit4:

3 x 16cores working

Edited April 24, 2020 by jasin755

[fabiosun]

Hi @jasin755

For Proxmox VM cores and thread are the same

in my case I put 64 and system starts fine

You could also add sockets in VM creation till a max of 4

in my case 4 sockets, 16 cores mean 64 in total

I prefer 1 socket solution

 

If you can test also with 48 (as your cpu is) and less memory and see

Maybe your rigs allocates more memory or cores for proxmox (weird but possible)

 

also please put your exact vm.conf file in text mode here

I would like to see

 

[jasin755]

Is it possible to use VM directly without VNC? 

Edited April 24, 2020 by jasin755

[fabiosun]

I use in this way

after I have passthrough my GPU I start my vm on my AMD Threadripper in linux environment after login

qm start 100

in my case

and my Opencore/OSX starts

have you seen video on macos86 you tube channel?

 

I do not know what happens when you pass your unsupported GPU in OSX

but if it passes maybe you can do the same without acceleration (obviously)

 

[jasin755]

I have new problem with GPU passthrough. I always stuck on boot screen.

 

 

Output from start:

kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
TASK OK

 

dmesg | grep -e DMAR -e IOMMU
[    2.263728] pci 0000:60:00.2: AMD-Vi: IOMMU performance counters supported
[    2.263761] pci 0000:40:00.2: AMD-Vi: IOMMU performance counters supported
[    2.263777] pci 0000:20:00.2: AMD-Vi: IOMMU performance counters supported
[    2.263794] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[    2.294370] pci 0000:60:00.2: AMD-Vi: Found IOMMU cap 0x40
[    2.294373] pci 0000:40:00.2: AMD-Vi: Found IOMMU cap 0x40
[    2.294375] pci 0000:20:00.2: AMD-Vi: Found IOMMU cap 0x40
[    2.294377] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[    2.297843] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).
[    2.297854] perf/amd_iommu: Detected AMD IOMMU #1 (2 banks, 4 counters/bank).
[    2.297865] perf/amd_iommu: Detected AMD IOMMU #2 (2 banks, 4 counters/bank).
[    2.297878] perf/amd_iommu: Detected AMD IOMMU #3 (2 banks, 4 counters/bank).
[    5.736528] AMD-Vi: AMD IOMMUv2 driver by Joerg Roedel <jroedel@suse.de>

 

lspci
...
4a:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Vega 10 XL/XT [Radeon RX Vega 56/64] (rev c3)
4a:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Vega 10 HDMI Audio [Radeon Vega 56/64]
....

 

 

One think I don't know. If my Vega56 supports EFI. I tried by this tutorial http://vfio.blogspot.com/2014/08/does-my-graphics-card-rom-support-efi.html

 

but when I type 

echo 1 > rom

I got:

-bash: rom: Permission denied

 

in

/sys/bus/pci/devices/0000:01:00.0/

folder.

[fabiosun]

13 minutes ago, jasin755 said:

I have new problem with GPU passthrough. I always stuck on boot screen.

 

 

Output from start:

kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
TASK OK

 

dmesg | grep -e DMAR -e IOMMU
[    2.263728] pci 0000:60:00.2: AMD-Vi: IOMMU performance counters supported
[    2.263761] pci 0000:40:00.2: AMD-Vi: IOMMU performance counters supported
[    2.263777] pci 0000:20:00.2: AMD-Vi: IOMMU performance counters supported
[    2.263794] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[    2.294370] pci 0000:60:00.2: AMD-Vi: Found IOMMU cap 0x40
[    2.294373] pci 0000:40:00.2: AMD-Vi: Found IOMMU cap 0x40
[    2.294375] pci 0000:20:00.2: AMD-Vi: Found IOMMU cap 0x40
[    2.294377] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[    2.297843] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).
[    2.297854] perf/amd_iommu: Detected AMD IOMMU #1 (2 banks, 4 counters/bank).
[    2.297865] perf/amd_iommu: Detected AMD IOMMU #2 (2 banks, 4 counters/bank).
[    2.297878] perf/amd_iommu: Detected AMD IOMMU #3 (2 banks, 4 counters/bank).
[    5.736528] AMD-Vi: AMD IOMMUv2 driver by Joerg Roedel <jroedel@suse.de>

 

lspci
...
4a:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Vega 10 XL/XT [Radeon RX Vega 56/64] (rev c3)
4a:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Vega 10 HDMI Audio [Radeon Vega 56/64]
....

 

 

One think I don't know. If my Vega56 supports EFI. I tried by this tutorial http://vfio.blogspot.com/2014/08/does-my-graphics-card-rom-support-efi.html

 

but when I type 

echo 1 > rom

I got:

-bash: rom: Permission denied

 

in

/sys/bus/pci/devices/0000:01:00.0/

folder.

your gpu passthrough is not correct 

you have to pass only first one checking all four flag you have in advanced mode

second one is hdmi feature and it is passed with first address 4a:00.0

you see proxmox logo on your AMD pc?

if so passthrough is working and (after you do this change in vm conf file) if hangs there you can try to press f2 before it hangs and force to boot from correct disk

then..

Post your blacklist.conf, vfio.conf, modules file in txt format

(if you want) 🙂

 

 

 

 

ah

by the way 

to solve host doesn't support bla bla bla message remove if you have +pcid on your boot args

 

[jasin755]

Yes I see Proxmox logo (my first picture in previous post). I deleted second HDMI feature and check all boxes in PCI device. But doesn't help.

 

Blacklist.conf:

blacklist radeon
blacklist nouveau
blacklist nvidia

 

vfio.conf

options vfio-pci ids=1002:687f,1002:aaf8 disable_vga=1

/etc/modules:

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

 

/etc/default/grub

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="Proxmox Virtual Environment"
GRUB_CMDLINE_LINUX_DEFAULT="quiet"
GRUB_CMDLINE_LINUX=""
amd_iommu=on
GRUB_DISABLE_OS_PROBER=true
GRUB_DISABLE_RECOVERY="true"

kvm.conf

options kvm ignore_msrs=Y

 

[fabiosun]

@jasin755

GRUB_CMDLINE_LINUX_DEFAULT="quiet"

check the guide

iommu stuff and so on 😉

 

check also /boot/grub/grub.cfg

also is very nice your trx40 starts without mce=off...

maybe aa different proxmoxVe version? (newer I mean)

 

 

if identical to previous modify it and run pve refresh command as you can find in the main guide

 

[jasin755]

Ok it's working now. Problem si that I don't see clover but when I press ENTER and wait few seconds macos will boot.

 

[fabiosun]

9 minutes ago, jasin755 said:

Ok it's working now. Problem si that I don't see clover but when I press ENTER and wait few seconds macos will boot.

 

if possible describe what you did to have it working so it can be helpful for others 🙂

thank you

[jasin755]

Nothing. I only fixed grub and press "RETURN" on frozen screen. 

[jasin755]

I have started using macos and seems to be little laggy. Eg. when I switching apps etc. I thing it's connected with HDD. I tried Blackmagic Design speed test for HD and I got strange results (write speed about 3000MB/s and read speed 7000MB/s) that is not possible. I have common sata SSD with 500/500MB/s speed.

[fabiosun]

Hi these as real data from a NVME disk passed completely.

Probably you are using for now a vm file..if I remember well your config

So cache do those weird results

 

Lags are there because you have an unsupported gpu (if you have had a 1080 ti or similar you will have a normal OSX interface)

Remember AMD hack (this method or patched method )has some less performance in some case respect windows in GPU part.

 

[jasin755]

I have Vega56. It should be supported.  And it isn't visual lag.  Is it possible use disk directly/passthrough on SATA? 

PS: I mean disk speed test not RAW speed test.

Edited April 24, 2020 by jasin755

[fabiosun]

sorry I misunderstood because I see in signature an unsupported nvidia

Have you graphics acceleration?

could you test some gfx benchmark and see if you have usual value for your GFX?

I mean cinebench 15 graphic benchmark or other.

for other part read the guide

you can pass entire sata controller, usb or nvme controller and I did..but you have to study in relationship your original proxmox installation to avoid some conflict...

 

 

[fabiosun]

I have had different and powerful rigs with intel CPU and different GPU

I can say in this way is better..*

if you see some weir lag or other you have to figure out

 

* I have had a dual xeon 2696v4 or an Intel 7980xe..

3970x is better..and not only obviously for number of cores...

I hope you appreciate this guide... 😛

 

[fabiosun]

 

this is my sabrent NVME, if you search in this site you will find also my benchmark on my old and not virtual intel 7980xe identical result

I repeat myself, vm’s files produce some caching and abmnormal results, this one shown here are real and correct 🙂

 

[jasin755]

I tried Heaven benchmark and I have stable cca 60FPS on Extreme settings. I am pretty sure that small lagging creates IO delay on HDD. I will buy PCI-e NVME and try to install macos directly to NVME using PCIe passthrough.

[jasin755]

Ok. I found reason of micro lags. I have 3960x it has 48threads and I set 48 cores for my VM. I found on proxmox forum that is not recommended use maximum cores for your VM because Proxmox needs use little bit of CPU time for ourselves. Recommended count is -1 from your max. In my case it's little problem. Because when I switch from 3x16cores to 3x15 cores I am not able to boot. The most cores which I can use is 2x16 or 4x8 (32). I don't understand why I can't set e.g. 1x45 😞 

[fabiosun]

Hi @jasin755

take a look on this picture:

 

It says ordered on 3 December 2019

I received it on 4 December togheter my motherboard.

Other components were there.

On 4 December (about lunch time) I have had first bad news...NO OSX for it..no way

OSX it is not my main system but I wanted it 🙂

from this day I have dedicated some time to have OSX working in my system.

 

This preface to say..

If I have had to loose with this method only 1 core/1thread..for me this it not was an option. I want all 64 available!!

So problem of micro lags (in my opinion) are in other part of your setup.

 

The only thing you can't use for all is memory..in my case, now, I am testing 117 Gb and it is working well from 2 days

I can't know exactly where your problem is because I have not your hardware, but, with hardware in signature there are no lags.

I will try to put cpu : host (as advice for you) and leave Penryn in args..but only you can test and try

 

In this time I have tested Proxmox installed on a NVME drive with OSX in a virtual disk, on a SSD, on a SSD for proxmox and a "passed" nvme for OSX..never had lags

OSX on virtual disk (file) produces same speeddisk of yours (inaccurate and good only to catch some "patreon" on the net 😂)

Passed SSD or Nvme disks produce correct benchmark as I have on a non virtual system (windows)

 

all this to say..you have to investigate for yourself. and you are lucky because now, hack community has a guide (a free guide) and this guide do not hide anything...all is there without creating hype..because it is not our goal..our goal is passion for hack 😉

 

Many guide are on the net, often also better detailed..but no one did one so accurate for our TRX40 family..

 

Have a nice hack..if you dedicate time to it ..you will have it as you want. it is possible 🙂