tsongz

This might be a dumb question, but I haven't been able to find a clear guide on it over the past few months.

What are the steps to properly installing a new OS update for MacOS on the same disk? The reason I ask is that the first time I had tried to do it, I bricked the OS due to SIP and having the "seal" broken on my volume snapshot, and was never able to figure it out. Since then, I've only basically just done a fresh install for major upgrades on a new disk, definitely not practical.

EDIT: Figured it out. It was being caused by having my Intel NVME drive that wasn't APFS formatted (Proxmox files) trying to be loaded - which I finally guessed from "@filesystems.msdosfs" being the last started kext to load. I removed the NVME drive from the motherboard and it works normally now.

One issue I'm running into that I'm not able to figure out and isolate that wasn't present in Proxintosh, is this one Kernel Panic when trying to launch Blizzard Games (Heroes, SC2). I've tried dozens of different things, but haven't been able to figure it out.

The behavior is whenever the game is launched, it loads for a bit, then completely bricks (requiring hard restart/shut down). 

Config attached and Kernel Panic Log below:

Panic(CPU 24, time 812901390954): NMIPI for unresponsive processor: TLB flush timeout, TLB state:0x0
RAX: 0xffffff876302baf8, RBX: 0xffffff8762394e58, RCX: 0x00000000000003aa, RDX: 0x000002e000000000
RSP: 0xffffffa27efe3dd0, RBP: 0xffffffa27efe3e10, RSI: 0xffffff87661d4000, RDI: 0xffffff8762394e58
R8:  0xffffff800adc4240, R9:  0x00000000000052d3, R10: 0x0000000000000000, R11: 0x0000000000000000
R12: 0xffffff87661d4000, R13: 0x0000000000000000, R14: 0x000002e16ce9a975, R15: 0xffffff876302baf8
RFL: 0x0000000000000017, RIP: 0xffffff800aca0f07, CS:  0x0000000000000008, SS:  0x0000000000000010
Backtrace (CPU 24), Frame : Return Address
0xffffffa2158dcf80 : 0xffffff800adc467b mach_kernel : _interrupt + 0x12b
0xffffffa2158dcfd0 : 0xffffff800ac32bdd mach_kernel : _hndl_allintrs + 0x11d
0xffffffa27efe3e10 : 0xffffff800aca1024 mach_kernel : _lck_attr_free + 0x204
0xffffffa27efe3e40 : 0xffffff800ace2720 mach_kernel : _waitq_lock + 0x20
0xffffffa27efe3e60 : 0xffffff800acbd52b mach_kernel : _semaphore_wait + 0x5b
0xffffffa27efe3ec0 : 0xffffff800acbd79d mach_kernel : _semaphore_wait_trap + 0x3d
0xffffffa27efe3ef0 : 0xffffff800ada91fd mach_kernel : _mach_call_munger64 + 0x29d
0xffffffa27efe3fa0 : 0xffffff800ac33216 mach_kernel : _hndl_mach_scall64 + 0x16

Process name corresponding to current thread: Heroes
Boot args: -v keepsyms=1 debug=0x100 

Mac OS version:
20E5229a

Kernel version:
Darwin Kernel Version 20.4.0: Thu Mar 25 07:05:22 PDT 2021; root:xnu-7195.101.1~26/RELEASE_X86_64
Kernel UUID: 762DC741-B83E-3A82-847D-5AC153AC3E4A
KernelCache slide: 0x000000000aa00000
KernelCache base:  0xffffff800ac00000
Kernel slide:      0x000000000aa10000
Kernel text base:  0xffffff800ac10000
__HIB  text base: 0xffffff800ab00000
System model name: iMacPro1,1 (Mac-7BA5B2D9E42DDD94)
System shutdown begun: NO
Panic diags file available: NO (0xe00002bc)
Hibernation exit count: 0

System uptime in nanoseconds: 812901578727
Last Sleep:           absolute           base_tsc          base_nano
  Uptime  : 0x000000bd44b5ba5d
  Sleep   : 0x0000000000000000 0x0000000000000000 0x0000000000000000
  Wake    : 0x0000000000000000 0x00000024aca7dd00 0x0000000000000000
last started kext at 25516855985: @filesystems.msdosfs	1.10 (addr 0xffffff7fa4c19000, size 57344)
last stopped kext at 319680967784: >IOPlatformPluginLegacy	1.0.0 (addr 0xffffff7fa4304000, size 36864)
loaded kexts:
com.zxystd.!I!BFirmware	1.1.2
com.insanelymac.LucyRTL8125Ethernet	1.0.0d6
com.zxystd.itlwm	1.2.0
wtf.spinach.SMCAMDProcessor	1
as.vit9696.VirtualSMC	1.2.2
wtf.spinach.AMDRyzenCPUPowerManagement	0.6.5
org.acidanthera.NVMeFix	1.0.6
as.acidanthera.BrcmFirmwareStore	2.5.8
as.lvs1974.AirportBrcmFixup	2.1.3
as.vit9696.!AALC	1.5.9
as.vit9696.WhateverGreen	1.4.9
as.vit9696.Lilu	1.5.2
@filesystems.msdosfs	1.10
>AudioAUUC	1.70
>!AUpstreamUserClient	3.6.8
>!AMCCSControl	1.14
@kext.AMDFramebuffer	4.0.4
>X86PlatformShim	1.0.0
>AGPM	122
>!APlatformEnabler	2.7.0d0
@filesystems.autofs	3.0
@fileutil	20.036.15
@kext.AMDRadeonX5000	4.0.4
@kext.AMDRadeonServiceManager	4.0.4
@AGDCPluginDisplayMetrics	6.2.9
>pmtelemetry	1
|IOUserEthernet	1.0.1
>usb.!UUserHCI	1
|IO!BSerialManager	8.0.4d18
@Dont_Steal_Mac_OS_X	7.0.0
>!AHV	1
>!ADiskImages2	1
>!AGFXHDA	100.1.433
>!A!IMCEReporter	115
>!A!ISlowAdaptiveClocking	4.0.0
@kext.AMD10000!C	4.0.4
|IO!BUSBDFU	8.0.4d18
|SCSITaskUserClient	436.100.4
@private.KextAudit	1.0
@filesystems.apfs	1677.100.114
>!AAHCIPort	346.100.2
>AirPort.BrcmNIC	1400.1.1
>!AFileSystemDriver	3.0.1
@filesystems.tmpfs	1
@filesystems.hfs.kext	556.100.11
@BootCache	40
@!AFSCompression.!AFSCompressionTypeZlib	1.0.0
@!AFSCompression.!AFSCompressionTypeDataless	1.0.0d1
>!AHPET	1.8
>!AACPIButtons	6.1
>!ARTC	2.0
>!ASMBIOS	2.1
>!AAPIC	1.7
>!A!ICPUPowerManagementClient	222.0.0
@!ASystemPolicy	2.0.0
@nke.applicationfirewall	311
|IOKitRegistryCompatibility	1
|EndpointSecurity	1
>!A!ICPUPowerManagement	222.0.0
>!UAudio	405.39
|IOAVB!F	940.4
@plugin.IOgPTPPlugin	980.4
|IOEthernetAVB!C	1.1.0
>!ASMBus!C	1.0.18d1
|IOSMBus!F	1.1
@kext.triggers	1.0
@kext.AMDRadeonX5000HWLibs	1.0
|IOAccelerator!F2	442.9
@kext.AMDRadeonX5000HWServices	4.0.4
|IOSerial!F	11
@!AGPUWrangler	6.2.9
|IONDRVSupport	585.1
|IOAudio!F	300.6.1
@vecLib.kext	1.2.0
|IOSlowAdaptiveClocking!F	1.0.0
@kext.AMDSupport	4.0.4
@!AGraphicsDeviceControl	6.2.9
|IOGraphics!F	585.1
>X86PlatformPlugin	1.0.0
>IOPlatformPlugin!F	6.0.0d8
>usb.!UHub	1.2
>usb.IOUSBHostHIDDevice	1.2
|Broadcom!BHost!CUSBTransport	8.0.4d18
|IO!BHost!CUSBTransport	8.0.4d18
|IO!BHost!CTransport	8.0.4d18
>usb.cdc	5.0.0
>usb.networking	5.0.0
>usb.!UHostCompositeDevice	1.2
|IOAHCIBlock!S	332
>!AXsanScheme	3
|IOAHCI!F	294.100.1
>!AEthernetAquantiaAqtion	1.0.64
|IO80211!F	1200.12.2b1
>corecapture	1.0.4
|IONVMe!F	2.1.0
>usb.!UXHCIPCI	1.2
>usb.!UXHCI	1.2
>!ABSDKextStarter	3
|IOSurface	290.7
|IOSkywalk!F	1
>mDNSOffloadUserClient	1.0.1b8
@filesystems.hfs.encodings.kext	1
>usb.!UHostPacketFilter	1.0
|IOUSB!F	900.4.2
>!AEFINVRAM	2.1
>!AEFIRuntime	2.1
|IOHID!F	2.0.0
$!AImage4	3.0.0
|IOTimeSync!F	980.4
|IONetworking!F	3.4
>DiskImages	493.0.0
|IO!B!F	8.0.4d18
|IOReport!F	47
|IO!BPacketLogger	8.0.4d18
$quarantine	4
$sandbox	300.0
@kext.!AMatch	1.0.0d1
|CoreAnalytics!F	1
>!ASSE	1.0
>!AKeyStore	2
>!UTDM	511.100.9
|IOUSBMass!SDriver	184.101.1
|IOSCSIBlockCommandsDevice	436.100.4
|IO!S!F	2.1
|IOSCSIArchitectureModel!F	436.100.4
>!AMobileFileIntegrity	1.0.5
@kext.CoreTrust	1
>!AFDEKeyStore	28.30
>!AEffaceable!S	1.0
>!ACredentialManager	1.0
>KernelRelayHost	1
|IOUSBHost!F	1.2
>!UHostMergeProperties	1.2
>usb.!UCommon	1.0
>!ABusPower!C	1.0
>!ASEPManager	1.0.1
>IOSlaveProcessor	1
>!AACPIPlatform	6.1
>!ASMC	3.1.9
|IOPCI!F	2.9
|IOACPI!F	1.4
>watchdog	1
@kec.pthread	1
@kec.corecrypto	11.1
@kec.Libm	1
panic(cpu 32 caller 0xffffff800ad9e081): "IPI timeout, unresponsive CPU bitmap: 0x1000000, NMIPI acks: 0x0, now: 0x1, deadline: 812901388638, pre-NMIPI time: 0xbd44b2d236, current: 0xbd44b6075a, global: 0"@/System/Volumes/Data/SWE/macOS/BuildRoots/a9f5d55ca2/Library/Caches/com.apple.xbs/Sources/xnu/xnu-7195.101.1/osfmk/x86_64/pmap.c:2892
Backtrace (CPU 32), Frame : Return Address
0xffffffa27f01b840 : 0xffffff800ac8e02d mach_kernel : _handle_debugger_trap + 0x3fd
0xffffffa27f01b890 : 0xffffff800add48e3 mach_kernel : _kdp_i386_trap + 0x143
0xffffffa27f01b8d0 : 0xffffff800adc4eda mach_kernel : _kernel_trap + 0x55a
0xffffffa27f01b920 : 0xffffff800ac32a2f mach_kernel : _return_from_trap + 0xff
0xffffffa27f01b940 : 0xffffff800ac8d84d mach_kernel : _DebuggerTrapWithState + 0xad
0xffffffa27f01ba60 : 0xffffff800ac8db43 mach_kernel : _panic_trap_to_debugger + 0x273
0xffffffa27f01bad0 : 0xffffff800b49d68a mach_kernel : _panic + 0x54
0xffffffa27f01bb40 : 0xffffff800ad9e081 mach_kernel : _pmap_flush_tlbs + 0x7d1
0xffffffa27f01bc00 : 0xffffff800ada40cc mach_kernel : _pmap_remove_range + 0x37c
0xffffffa27f01bce0 : 0xffffff800ada4eb6 mach_kernel : _pmap_remove_options + 0x1f6
0xffffffa27f01bd40 : 0xffffff800ad33145 mach_kernel : _vm_map_remove + 0x725
0xffffffa27f01be90 : 0xffffff800ad32a93 mach_kernel : _vm_map_remove + 0x73
0xffffffa27f01bec0 : 0xffffff800ac7edde mach_kernel : __kernelrpc_mach_vm_deallocate_trap + 0x7e
0xffffffa27f01bef0 : 0xffffff800ada91fd mach_kernel : _mach_call_munger64 + 0x29d
0xffffffa27f01bfa0 : 0xffffff800ac33216 mach_kernel : _hndl_mach_scall64 + 0x16

Process name corresponding to current thread: Heroes

I've found nothing on Blizz forums, and have also tried with my other installs (Catalina) and the issue is the same, so I'm not thinking it's the OS. 

Anyone else run into something like this? 

config.plist.zip

After almost a whole year stuck on Proxmox BS 11.0, I've finally made the switch over to Bare Metal, since I got tired of not having access to Bluetooth/USB ports.

Going in, I probably spent more time researching, planning, backing up (in case of total failure since I've been using it for work), than the actual installation process actually took. (Probably 20 tabs from various pages in this thread)

Currently in the process of restoring the data from my old BS install, otherwise I'd share screen shots (will follow up with edits).

- Used Pavo's OSX Generator to create the base config

- Installed on fresh SSD

- Upgraded Asrock Motherboard BIOS to 1.7.0

- Used Driftwood's BIOS settings, though I had to enable Above 4G encoding - caused install to hang a few times because I forgot about this

- Patched MMIO using Driftwood's config

- Added additional kexts: IntelBluetoothFirmware, IntelBlueoothInjector, itlwm, NVMeFix (was getting errors during boot, so had to add this). 

- No DevProps, No ACPI

- Moved PCIe devices

-- PCIe 0: Vega Frontier

-- PCIe 1: Broadcom Network Card (originally wasn't working)

Shockingly, everything works so far. Bluetooth, WiFi, USB devices. Bluetooth and WiFi are being picked up by the Broadcom card (I think). So I may end up disabling/removing IntelBluetooth related stuff. 

Thanks everyone for all your hard work, contributions, and dedication - it's incredible to see how much progress things have come since a year ago, and how 'it just sorta works' it's become now. 

EDIT to include screen shots:

-- Further updated Plist with MMIO / Patches / DSDTs from Rox67er as well as BIOS

-- Added back in DevProps for GPU

-- Added RyzenCPUPower, SMCAMDProcessor, and AH_USB-Map found in Rox67er's Kexts. 

--> This update fixed Audio ports which weren't working properly prior, couldn't detect microphone in front case panel

1st is the latest, 2nd was the original Bare Metal, 3rd is Proxintosh

Oddly enough, Single Thread performance is down, whereas MT is up on the CPU side.

On the Compute side, Metal is way down. (1st 3 results are the latest Bare Metal)

Though honestly, the fact that I can use all the USB ports and BT is worth the trade off. 

Off-Topic, but if anyone here lives in an area that has painfully internet speeds, and have both the Ethernet port and WiFi working on your Hack, you'll likely benefit from this.

Quick backstory - I used to have 1gbps internet speeds, but the new place I'm at unfortunately only offers 50mbps. I searched everywhere, and unfortunately, there were zero other options. (*cough* ATT *cough*)

Anyways, this is a hack I use to increase my download speeds. It uses internet bonding, basically combining your two separate network connections into a single one. It works on my normal iMac, and wanted to verify that the WiFi from AX200 was working, I tested it on here as well. YMMV.

No Internet Bonding:

With Internet Bonding from same server:

Doesn't do much for Uploads unfortunately.

# install npm from brew

brew install npm



# install dispatch-proxy

npm install -g dispatch-proxy



# start dispatch

dispatch start

@Ploddles - I'm using itlwmx from this repo.

You can check compatibility in this list.

Once I download the kext, I modify the Info.plist in the kext with my WiFi network info at wifi1, and then add the device properties in my EFI config.plist.

@meina222 - I added in the intelwmx kext and injected the device properties and AX200 WiFi is now working and showing up on my PVE Big Sur. I haven't attached the Bluetooth USB yet so it doesn't show up.

@meina222 is your WiFi / BT working in Big Sur with BCM4360?

Also, I noticed an update in Dortania's OC Guide regarding TRX40 - not sure if this is correlated but I imagine it has a lot to do with your work @iGPU.

Additionally - there's a new update with intelwifi kext that does officially support AX200 - I have the kext loaded and will test it on my next reboot on my PVE Big Sur (mainly because I almost bricked my BS by trying to fix the BCM4360 kext)

They were the included SSDTs from the original PVE-OC iso, and I haven't removed it (probably should). ALS is for Ambient Light Sensors (not needed).

I think the main one should be SSDT-EC.

This is what's showing up for me. I have not modified it unlike for Bare Metal.

@meina222 - I currently do with these

My PCIe Configuration

**** In-Use Passthrough Devices ****

PCIe ID: 23:00.0 | VFIO ID: 1002:6863 | IOMMU Group: 29 | Used By: hostpci1      | Device: VGA compatible controller - Advanced Micro Devices, Inc.
PCIe ID: 23:00.1 | VFIO ID: 1002:aaf8 | IOMMU Group: 30 | Used By: hostpci1      | Device: Audio device - Advanced Micro Devices, Inc.
PCIe ID: 25:00.3 | VFIO ID: 1022:148c | IOMMU Group: 34 | Used By: hostpci3      | Device: USB controller - Advanced Micro Devices, Inc.
PCIe ID: 45:00.0 | VFIO ID: 1d6a:07b1 | IOMMU Group: 60 | Used By: hostpci2      | Device: Ethernet controller - Aquantia Corp. AQC107 NBase-T/IEEE 802.3bz Ethernet Controller
PCIe ID: 4b:00.0 | VFIO ID: 1987:5012 | IOMMU Group: 68 | Used By: hostpci0      | Device: Non-Volatile memory controller - Phison Electronics Corporation E12 NVMe Controller
********************************************************************************************************************************************

This allows the full passthrough of my MB top USB slots. The downside with this is that the keyboard/USB slots doesn't initialize until after the OC Boot screen, unless a USB device is passed through explicitly from proxmox, so I have a secondary keyboard I passthrough as a USB Device in case I need to do anything. I don't pass through my case USB devices.

I'm attaching my pre-compiled PVE kernels. It likely won't work for you unless you have a Radeon VII or Vega FE (since those are the device IDs I selected, but I'll write the instructions below)

This is my modified patch script for Vega

Where 0x6863 is device ID of Vega FE and 0x66af is Device ID of Radeon VII. You can find yours specifically either through TechPowerUp or through your lspci (more accurate)

Then replace the part after 0x with the last 4 of the device ID. Ideally, it would have all the affected Vega devices, but I have kept it at 2 since the original script was 2.

diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
index 44c4ae1abd00..27840129e4b0 100644
--- a/drivers/pci/quirks.c
+++ b/drivers/pci/quirks.c
@@ -3433,6 +3433,14 @@ DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_ATHEROS, 0x0034, quirk_no_bus_reset);
*/
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_CAVIUM, 0xa100, quirk_no_bus_reset);

+/*
+ * Radeon RX Vega and Navi devices break on bus reset. Oi...
+ * This is *not a real workaround* - disabling bus reset
+ * for your GPU may have unintended consequences.
+ */
+DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_ATI, 0x6863, quirk_no_bus_reset); <- Modify & remove this comment (Vega FE)
+DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_ATI, 0x66af, quirk_no_bus_reset); <- Modify & remove this comment (Radeon VII)
+
static void quirk_no_pm_reset(struct pci_dev *dev)
{
/*

For good measure, what I did was create a new root user in Proxmox for compiling specifically, so that things don't get mixed up.

Assuming you already have pve-kernel git cloned and the submodules already updated, then this should take care of patching

# Or whichever Linux kernel you're using for the path below.
cd ~/pve-kernel/submodules/ubuntu-focal
# The path below expects fix-vega-reset.patch above to be the path below, modify it to match where you saved the file above.
patch -p1 < /home/pvecompiler/fix-vega-reset.patch 
# You can check the git diff to see that it has applied
git diff
# Now go back to the pve-kernel folder and make it
cd /home/pvecompiler/pve-kernel #or cd - or cd ../../
make

Then I downloaded the Reset-test app from the original thread in PVE, chmod +x the application, and set it in my script. 

Link to PVE Kernels here. It was 70mb compressed, so not able to upload here. I installed all 3 of the kernels, since it's within the same PVE version.

Sure, here's my hook. Just note that this is in conjunction with the Vega 10 patch, with specifying my Vega FE device ID in the patch (I'm not sure why they don't explicitly say that part, and just let people patch it with two of the many Vega cards), and also having a different GPU in PCIe Slot 1, and passing the Vega FE through in PCIe Slot 3. 

That said - I connect my single monitor to both GPUs and switch the inputs, and it does still work, despite that first part of the text from PVE on the Vega FE card. I've passed both GPUs through to 2 separate VMs with no issues.

It doesn't make any audible "click" sound from the RTC reset as it used to. 

#!/usr/bin/env bash

export VGA_DEVICE=0000:23:00.0 < - Change this

if [ "$2" == "release" ] ; then
if [ "$3" == "end" ] ; then
echo "unbinding, resetting and rebinding $VGA_DEVICE" > /dev/kmsg 2>&1
echo $VGA_DEVICE > /sys/bus/pci/devices/$VGA_DEVICE/driver/unbind
# Change the path to reset-test below #
/root/cmds/reset-test $VGA_DEVICE
echo $VGA_DEVICE > /sys/bus/pci/drivers/vfio-pci/bind
echo "sucessfully unbound, reset and rebound $VGA_DEVICE" > /dev/kmsg 2>&1
fi
fi

@asynic Do you have any custom configuration for your Vega VII? And which SMBIOS model are you currently using?

I personally use additional device properties to tweak power settings for my Vega FE, and while the fans ramp up a bit faster than with bare metal (with same config, minus the addition of the 1070ti, so that could also be a cause), there's been no issues so far for me. I think one of the previous posts I had shared my gpu configuration.

You could download VGTab, and use the stock settings for it, and replace the power tables to see if that makes a difference.

Also @meina222 - with the patched PVE kernels installed, I no longer experience any reset issues with the Vega FE. I can start up and shut down the VM with PCIe passthrough multiple times without any problems, although sometimes a non-clean shutdown of the VM will require a full host reset. 

@fabiosun came across something interesting regarding Nvidia Drivers within Big Sur, while digging through kexts to fix BT. Not sure if you've seen these yet. Attaching Zip.

The interesting part about it is in the kext file. It seems like with adjustments to the NVArch device match, you should be able to load the GTX drivers, albeit not as optimized, or loading a custom driver it seems like.

NVGPU.zip

Back on the Proxmox Train now with Big Sur.

- Can't use my existing bare metal Catalina install because it's installed on the Intel 660p which has issues with PCIe passthrough. 

- Added 1070ti to Slot 1, moved Vega Frontier to Slot 3

- Compiled PVE Kernel 5.4.55 with Vega 10 Patch (modified device ID in the patch with Vega FE's Device ID) and installed all the compiled debs

- Modified hook script for PVE MacOS (this now fixes the need to use RTC, and passes through cleanly. Although after VM shutdown, I still need to reboot the host)

- Have macOS running with Vega FE, and have ubuntu/Windows running with the 1070ti at the same time (has no issues with multiple resets)

- Biggest headache is the constant fixing of the Self-Assigned IP that macOS keeps popping up with. Since Big Sur doesn't place nice with the BCM adapter, any time eth breaks, keeps setting me back. I haven't found a reliable way of fixing this problem. 

Other than that, everything works perfectly for my work flow. Have CUDA 11.0 on the 1070ti, and can ssh into the second VM from the macOS VM. Will likely try to fix the BCM adapter issue when I get the chance and stay in Big Sur rather than Catalina, since it seems like applications that are getting ported into Big Sur have to play nicer with arm, with the trade off of less existing capabilities right now. 

Curious @fabiosun, something I was planning on trying with Proxmox version was to create a small virtual disk that would store the EFI partition for separate OSX installations and attach them to the VM, rather than directly modifying the volume/installation EFI partition. Then setting the boot order to load the virtual EFI disk first. My reasoning being that it would allow for interchangeability of EFIs / OSX versions, and being able to modify broken EFIs from a working one. Would be similar to booting first from a USB with OC on it. 

Is there a flaw in my thinking that I'm not considering that would make this not work?

@meina222 your motherboard should have a Clear CMOS button on the back. It's basically my go-to whenever I have to do a hard reboot or anything goes wrong with NVRAM.

Don't know if you  do this already, but check your POST [little 2 digit display] monitor on your motherboard when you boot up. More often than not, if macOS caused some issue, POST will either fail to show at all at boot, or sit at 99. (at least for me). Clearing CMOS generally has done the trick.

@fabiosun - I'm in agreement with you. My benchmark scores are amazing, even beating out the more modern Pro Vegas with my Vega Frontier  - but, the compatibility and the 'it just works' nature of Apple products is anything but. AMD =/= Intel CPUs is definitely a big hurdle in software development, since all macOS libraries/kernels are compiled only for Intel. On top of that, RocM, AMD's CUDA equivalent has no intention of porting compatibility to Unix/macOS, which makes the Vega FE almost useless for Machine Learning in macOS. (There's a drop in the bucket of support for Metal API in ML - none of which is usable in my current workflow)

And the thing that was the deciding factor for me against bare metal going forward - is even if 80% of things work well, lack of modern virtualization (due to lack of Intel's VT-d) support on macOS hypervisor API, I can't take advantage of the hardware (GPU specifically) outside of macOS, as there's not really PCIe passthrough support because of ^.

What I am planning on doing now for production is going back to Proxmox, and splitting CPU/RAM allocation for an ubuntu VM + macOS VM (Intel Pendryn CPU), and adding a second GPU so I can use both concurrently. It sort of sucks to not have full hardware access all in one OS, but that's the risk we take. I am still planning on figuring out these nuances as time permits for bare metal but we're trying to break a concrete wall with a spoon, as we have little documentation and historical references.

@fabiosun - I spent a good majority of yesterday investigating software compatibility, including compiling several libraries such as Tensorflow (if you want to really benchmark your CPU - try compiling Tensorflow. It'll run your CPU at 100% for a good 20-30 mins. Peaked at around 178F for me.) 

I think I may have come close to having core AMD compatibility with macOS for applications.

Most macos applications (from what I researched) use Intel's MKL library for CPU acceleration for graphics. This includes Adobe suite software. Previously, MKL was an Intel-only feature, but because of some trouble/pressure, they've since open-sourced it. Now it's called oneAPI. It can be compiled with amd64 CPU architecture, which I was able to do, and installed it successfully on my bare metal Catalina. (compiled folder attached via link - it's over 50mb)

Some more information related to MKL and AMD/Intel CPU architectures by Puget Systems - MKL was basically a feature that intentionally crippled AMD CPUs that most developers just ended up using.

It's likely safe to run

Big Sur/Catalina

echo -n 'export MKL_DEBUG_CPU_TYPE=5' >> ~/.zshrc

Mojave/Previous

echo -n 'export MKL_DEBUG_CPU_TYPE=5' >> ~/.profile

Now, what I'm currently unsure of, as I haven't yet tested it is:

a) whether or not this new MKL library actually replaces the existing MKL

b) whether the MKL libraries are universally linked to the host or have existing MKL libraries within their own app

c) whether this new MKL is compatible with macOS apps

d) do we need the other MKL libraries as well?

Installation Logs

Install the project...
-- Install configuration: "Release"
-- Installing: /usr/local/lib/libdnnl.1.6.dylib
-- Installing: /usr/local/lib/libdnnl.1.dylib
-- Installing: /usr/local/lib/libdnnl.dylib
-- Installing: /usr/local/include/dnnl_config.h
-- Installing: /usr/local/include/dnnl_version.h
-- Installing: /usr/local/include/dnnl.h
-- Installing: /usr/local/include/dnnl.hpp
-- Installing: /usr/local/include/dnnl_debug.h
-- Installing: /usr/local/include/dnnl_threadpool_iface.hpp
-- Installing: /usr/local/include/dnnl_types.h
-- Installing: /usr/local/include/mkldnn.h
-- Installing: /usr/local/include/mkldnn.hpp
-- Installing: /usr/local/include/mkldnn_config.h
-- Installing: /usr/local/include/mkldnn_debug.h
-- Installing: /usr/local/include/mkldnn_dnnl_mangling.h
-- Installing: /usr/local/include/mkldnn_types.h
-- Installing: /usr/local/include/mkldnn_version.h
-- Installing: /usr/local/lib/cmake/dnnl/dnnl-config.cmake
-- Installing: /usr/local/lib/cmake/dnnl/dnnl-config-version.cmake
-- Installing: /usr/local/lib/cmake/dnnl/dnnl-targets.cmake
-- Installing: /usr/local/lib/cmake/dnnl/dnnl-targets-release.cmake
-- Installing: /usr/local/lib/libmkldnn.dylib
-- Installing: /usr/local/lib/libmkldnn.1.dylib
-- Installing: /usr/local/lib/libmkldnn.1.6.dylib
-- Installing: /usr/local/share/doc/dnnl/LICENSE
-- Installing: /usr/local/share/doc/dnnl/THIRD-PARTY-PROGRAMS
-- Installing: /usr/local/share/doc/dnnl/README

Download Link to precompiled MKL-Dnn 

NVRAM works for me only in certain conditions. (I think)

If it's a shutdown while I'm still in the OC Picker screen (manual, power off from PC case), it seems to hold. However, any shut down/reboot while within MacOS, whether software or hardware, causes NVRAM issues. I had previously thought this was due to RTC, but NVRAM would make more sense, and not be too far off from the real root cause. 

Another thing I found, CFG-Lock doesn't seem to be supported, at least on the AsRock Creator MB, and I've read for AMD CPUs, it was something that was recommended for boot issues as well. I went through the BIOS using UEFITool.

The method I was planning on experimenting with Big Sur and Catalina, was through loading the UEFI from Catalina, and then booting through selecting the Big Sur volume, since I had done a full BS install on NVME and not a qemu disk, but I haven't tried it yet. I do agree that Big Sur through Proxmox has functioned a lot more stable than Catalina so far, at least in terms of application compatibility goes. A lot of my development tools tend to work with their command line counterparts vs breaking in the GUI versions, which include Docker, VirtualBox, etc. Parallels 13.1.0 works with Catalina, but not the latest, due to using VirtualBox and not Apple's Hypervisor, and picks up the host CPU/GPU. I still need to figure out how to fully virtualize (or likely Dual boot Ubuntu) because AMD's ROCm drivers aren't supported on MacOS.

@Ploddles - so during my time trying to tweak RTC settings, I ran into almost all the standard, default, issues, even when trying to boot off of a working EFI (USB). What the root cause of the issue was for me, was the BIOS settings (which gets reset when I have to clear CMOS due to hard shutdowns). And this was after having a working Installation.

Here's a few things to try:

- Clear your CMOS (if you have the button option on your mobo, save your BIOS settings on a USB, or as profile, unplug from wall, clear it with the button, then load it or manually check them again)

- CSM Disabled/4G Encoding Enabled (4G Encoding was disabled which caused my PCI error)

- Clear/Reset NVRAM - reboot, before attempting reinstall. Some cache of it may be left behind from previous installs? I can't verify this.

- If you haven't already, do a full disk erase before installing

See if those things all done prior to the install can help you clear more of the installer. Also for the sake of ruling out other potential issues, make sure you don't have any disks/drives/nvme attached to your mobo that aren't MacOS (Windows, etc) plugged in. Ideally just a single drive that you intend on installing first, and then afterwards, you can add them back in.

@iGPU - I'm noticing a few interesting compatibility quirks that weren't present in the Proxmox Version. A few more apps that I had used fine on BS are now crashing on Catalina. My guess is that it's due to exposing the CPU as AMD vs masking as Intel, and those application drivers aren't loading, as there's no native MacOS AMD drivers. What's your take on this? Another thought around the reset/shutdown issue would be something to do with the USB Power. 

Adding some additional useful things below

for file in MMXCore FastCore TextModel libiomp5.dylib; do
    find /Applications/Adobe* -type f -name $file | while read -r FILE; do
        sudo -v
        echo "found $FILE"
        [[ ! -f ${FILE}.back ]] && sudo cp -f $FILE ${FILE}.back || sudo cp -f ${FILE}.back $FILE
        echo $FILE | grep libiomp5 >/dev/null
        if [[ $? == 0 ]]; then
            dir=$(dirname "$FILE")
            [[ ! -f ${HOME}/libiomp5.dylib ]] && cd $HOME && curl -sO https://excellmedia.dl.sourceforge.net/project/badgui2/libs/mac64/libiomp5.dylib
            echo -n "replacing " && sudo cp -vf ${HOME}/libiomp5.dylib $dir && echo
            rm -f ${HOME}/libiomp5.dylib
            continue
        fi
        echo $FILE | grep TextModel >/dev/null
        [[ $? == 0 ]] && echo "emptying $FILE" && sudo echo -n >$FILE && continue
        echo "patching $FILE \n"
        sudo perl -i -pe 's|\x90\x90\x90\x90\x56\xE8\x6A\x00|\x90\x90\x90\x90\x56\xE8\x3A\x00|sg' $FILE
        sudo perl -i -pe 's|\x90\x90\x90\x90\x56\xE8\x4A\x00|\x90\x90\x90\x90\x56\xE8\x1A\x00|sg' $FILE
    done
done

[ ! -d $HOME/Library/LaunchAgents ] && mkdir $HOME/Library/LaunchAgents
AGENT=$HOME/Library/LaunchAgents/environment.plist
sysctl -n machdep.cpu.brand_string | grep Ryzen >/dev/null 2>&1
x=$(echo $(($? != 0 ? 5 : 4)))
cat >$AGENT <<EOF
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>mkl-debug</string>
    <key>ProgramArguments</key>
    <array>
    <string>sh</string>
    <string>-c</string>
        <string>launchctl setenv MKL_DEBUG_CPU_TYPE $x;</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
</dict>
</plist>
EOF
launchctl load ${AGENT} >/dev/null 2>&1
launchctl start ${AGENT} >/dev/null 2>&1

for file in MMXCore FastCore TextModel libiomp5.dylib; do
    find /Applications/Adobe* -type f -name $file | while read -r FILE; do
        sudo -v
        [[ -f ${FILE}.back ]] && echo "found backup $FILE" && sudo mv -f ${FILE}.back $FILE
    done
done

AGENT=$HOME/Library/LaunchAgents/environment.plist
if [[ -f $AGENT ]]; then
    launchctl unload ${AGENT} >/dev/null 2>&1
    launchctl stop ${AGENT} >/dev/null 2>&1
    rm -rf $AGENT
fi

#!/bin/bash

locale=`defaults read -g AppleLocale | cut -c 1-2`
cpu_name=`sysctl -n machdep.cpu.brand_string | sed -e 's/ *$//'`
target="/System/Library/PrivateFrameworks/AppleSystemInfo.framework/Versions/A/Resources/${locale}.lproj/AppleSystemInfo.strings"

if [ $# -ge 1 ]; then
  cpu_name=$1
fi

echo "Locale: $locale"
echo "New Name: $cpu_name"
read -p "ok? (y/n): " yn

case $yn in
  [yY])
    sudo mount -uw / &&
    sudo cp $target $target.`date +%Y%m%d%H%M` &&
    sudo plutil -replace IntelSpeedAndTypeFormat -string "%1\$@ $cpu_name" $target
    echo "Finished."
    ;;
  *) echo "Aborted."
esac

Have confirmed that virtualization is enabled (I.e. Docker) even though the CPU flags aren't reported. Based on reports, they look at the BIOS rather than what's reported by the OS. I wasn't able to get it working in Proxmox.

Followed this guide, although with a few modifications.

ln -s "/Applications/Docker.app/Contents//Resources/bin/docker-compose" /usr/local/bin/docker-compose
ln -s "/Applications/Docker.app/Contents//Resources/bin/docker-credential-desktop" /usr/local/bin/docker-credential-desktop
ln -s "/Applications/Docker.app/Contents//Resources/bin/docker-credential-osxkeychain" /usr/local/bin/docker-credential-osxkeychain

docker-machine create --virtualbox-no-vtx-check --driver virtualbox default
docker-machine env default

Was able to fix my Bluetooth issues by using the aml file provided by @iGPU, and changing the removed HS0x to the one that was associated with my Asrock Creator, which was HS06. Did a nvram clean and booted into MacOS and no more AX200.

Now the only thing that's left is figuring out why MacOS won't shut down/reboot properly. I feel like it has something to do with RTC settings/flags, but haven't validated that yet. Figuring out that last step, this is a full fledged bare metal TRXintosh! Everyone here is awesome for being so helpful and responsive, and major kudos and credits to everyone involved.

Edit: Adding my IORE for a Asrock Creator data point

TS-iMacPro-AsrockCreator.zip

@meina222 running into the same boot/reboot issues, but it's slightly different. 

For some reason, whenever MacOS boots, or restarts, it seemingly disables the motherboard CMOS. What has happened like 10 times, is I do either restart/shutdown (from MacOS), and when it goes into the power cycle:

a) shutdown is basically a restart. Not sure why - and this is 50/50 whether b happens

b) restart causes blank screen (no BIOS POST, which can be seen from the lack of Dr CMOS lights). Fix is to unplug all, clear CMOS, and reboot. Nothing really gets affected on the MacOS volume itself. As it goes into MacOS (as I'm typing this) - I just noticed that DrCMOS is disabled. 

It may be worth noting that I have no SSDs or HDDs attached, just two NVME drives, one with BS (from Proxmox) and one with Catalina (metal), and there is a note during boot log that no cache is created because it's detecting NVME drives.

Interestingly, CPU performance benchmarks are noticeably lower, while after adding my Vega FE patches, Metal performance is significantly higher. Not only that, when measuring device bandwidth with Cling, it's gone up a lot as well. OpenCL is hit or miss. Results are also consistent with GB5 Results. The metal API on iMacPro for Vega drivers seems to be much better than for MacPro. 

Highlighted is results from PVE Big Sur (MacPro7,1 - same mods, - I'm certain that's my benchmark bc  nobody else has a Vega FE on these charts lol)

Previous for comparison, was only 8 GB/s for both Host to Device and Device to Host, and 148 GB/s for Device to Device. About 50% increase in bandwidth.

8 minutes ago, Ploddles said:

Getting further than yesterday thanks to @meina222 but still no joy. Stops at

 

[Configuring PCI]

 

I ran into this issue once during the process. It's generally caused when you're mid-way through the install (first reboot), and then there's changes in your devices, whether that's from USB positions, moving disks around, devices around, etc. So it would be best to restart the install process from scratch on the disk, so it gets remapped during that first stage of the installer.

Was able to install Catalina 10.15.6 (iMacPro1,1) bare metal with a new NVMe drive, and the stock EFI folder. One minor issue that some might run into if you're not writing over Proxmox is that after the first stage of the installer that goes back into the "Install OSX" reboot, if your config isn't mapped to your other storage devices, the installer will fail. In my case, I ended up just unplugging them.

What works now in Bare Metal:

WiFi from the PCIe Adapter (BRCM4360 - which confirms it was a Big Sur issue)

RAM Modules set and registered properly.

What doesn't work now:

Ironically the Bluetooth, as I also have AX200 showing up, despite disabling it in BIOS.

Minor benchmarking shows a slight dip in performance, but I'm willing to take that performance dip in exchange for system stability.

@meina222 - not sure if you have the option to, but in the Asrock Creator, there's an option in the BIOS to turn off the Bluetooth, which will disable the AX200. It doesn't get picked up in Proxmox either when I set that.

Would someone be able to share with me what the cpu features that are detected by the OS are?

Basically the outputs of

"sysctl -a | grep machdep.cpu"