Rox67er
-
Posts
181 -
Joined
-
Last visited
-
Days Won
1
Content Type
Profiles
Forums
Events
Downloads
Posts posted by Rox67er
-
-
10 hours ago, Rocket88 said:
I have never been able to get sleep to work. As far a shutdown goes, with the correct MMIO whitelist, my shutdown works great. Make sure Energy Saver is set to "Prevent computer from sleeping automatically..."
My main problem now is that I get a code 99 on my Asrock Creator when I power on my machine, even when I use windows.
@Rocket88 I have this exact problem also in my VM proxmox OSX. Switch off your power supply (mains switch) for 10 seconds and then try to boot again. I was not able to solve this but I think it is somehow related to the onboard RGB lighting on the asrock motherboard.
-
6 hours ago, fabiosun said:
@Rox67erto pass your sata controllers there is some trick to know?
if you have all nvme controller used I mean
asrock trx40 creator share those controllers in some way?
@Driftwood what did you do to pass it?
I didn't run into any issue yet. My lspci -nnk below. I only use 2 of the three nvme (one for osx other for Proxmox) and pass one sata controller. I hoped the short audio interrupts would reduce with fast NVMe storage for Proxmox (thought it might be latency) but it didn't help.
43:00.0 Non-Volatile memory controller [0108]: Phison Electronics Corporation E12 NVMe Controller [1987:5012] (rev 01) Subsystem: Phison Electronics Corporation E12 NVMe Controller [1987:5012] Kernel driver in use: nvme 44:00.0 USB controller [0c03]: ASMedia Technology Inc. Device [1b21:3242] Subsystem: ASMedia Technology Inc. Device [1b21:3242] Kernel driver in use: vfio-pci Kernel modules: xhci_pci 45:00.0 Ethernet controller [0200]: Aquantia Corp. AQC107 NBase-T/IEEE 802.3bz Ethernet Controller [AQtion] [1d6a:07b1] (rev 02) Subsystem: ASRock Incorporation AQC107 NBase-T/IEEE 802.3bz Ethernet Controller [AQtion] [1849:d107] Kernel driver in use: vfio-pci Kernel modules: atlantic 46:00.0 Network controller [0280]: Broadcom Limited BCM4360 802.11ac Wireless Network Adapter [14e4:43a0] (rev 03) Subsystem: Apple Inc. BCM4360 802.11ac Wireless Network Adapter [106b:0117] Kernel driver in use: vfio-pci Kernel modules: bcma 47:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. Device [10ec:8125] (rev 01) Subsystem: ASRock Incorporation Device [1849:8125] Kernel driver in use: r8169 Kernel modules: r8169 48:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485] Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485] Kernel driver in use: vfio-pci 48:00.1 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c] Subsystem: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:1486] Kernel driver in use: vfio-pci Kernel modules: xhci_pci 48:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c] Subsystem: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:148c] Kernel driver in use: vfio-pci Kernel modules: xhci_pci 49:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51) Subsystem: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] Kernel driver in use: vfio-pci Kernel modules: ahci 4a:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51) Subsystem: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] Kernel driver in use: ahci Kernel modules: ahci 4b:00.0 Non-Volatile memory controller [0108]: Phison Electronics Corporation Device [1987:5016] (rev 01) Subsystem: Phison Electronics Corporation Device [1987:5016] Kernel driver in use: vfio-pci-
1
-
-
16 hours ago, fabiosun said:
beta3 also here 🙂
What was the issue?
-
3 hours ago, iGPU said:
Enabling those 2 patches, still does not allow me to boot in BS as 'host'. I can boot into BS with both enabled or disabled as 'Penryn'.
Using the same settings, and having those 2 patches enabled and using 'host', I can boot into Catalina (but, again, not BS).
I didn't experiment with Big Sur yet but are the patches still the same for the new kernel?
Found this on Nick Sherlock's Big Sur page, seems to confirm my statement:
"The CPU is set to pretend to be Penryn, because the macOS kernel patch that would support -cpu host has not yet been ported to Big Sur. Ensure the args are all on a single line!"
-
On 7/12/2020 at 2:44 AM, iGPU said:
This post raises 2 issues: Cinebench 20 speed in Catalina vs Big Sur (trivial), and 'host' vs 'Penryn'.
***
I'm presently running Big Sur under 'Penryn' VM setting (since I cannot yet get 'host' to boot), while previously, Catalina was run under 'host' VM setting. When testing Cinebench 20, the score in Big Sur is about 500 lower than I was seeing in Catalina.
So to keep things equal, I used the same VM and EFI settings for Big Sur to boot into Catalina. When I run Cinebench 20 under these conditions, I again get a 500 greater score in Catalina (17700 to 17800) than in Big Sur (17200 to 17300). I ran these tests several times over several boots.
So while Cinebench 20 is (presently) slightly slower under Big Sur, the 'Penryn' setting does not run any slower than 'host' with Catalina.
***
The above similarity in speed in Catalina when running under either 'host' or 'Penryn', makes me wonder how significant is either setting in our VM. I've read such statements as those below, which doesn't clearly indicate one being necessarily superior to the other:
"For homogeneous clusters and single host setups, use the 'host' option. For mixed clusters, use the lowest available CPU version, so if one host is Penryn and the other Nehalem, use Penrym on both."
"The 'host' option is good, but not the most efficient because it might enable options that can be emulated but that your cpu does not support. The guest system can then be slowed down anytime it tries to use one of those features that needs emulation."
Supposedly, to see the differences between the host and the guest, one can type: grep flags /proc/cpuinfo | uniq, which gives on my system:
fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate sme ssbd mba sev ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif umip rdpid overflow_recov succor smcaBut my VM for the guest has these arguments:
-cpu Penryn,kvm=on,vendor=GenuineIntel,+kvm_pv_unhalt,+kvm_pv_eoi,+hypervisor,+invtsc,+pcid,+ssse3,+sse4.2,+popcnt,+avx,+avx2,+aes,+fma,+fma4,+bmi1,+bmi2,+xsave,+xsaveopt,checkI don't understand how the above command lists unique items, since many are in common to both host and guest, such as: aes, avx, avx2, bmi1, bmi2, popcnt, xsave, xsaveopt (assuming that the VM properly used the argument flags for the guest).
@iGPU Be aware that the kernel patches section (in OC) play an important role in being able to set "host" as cpu. I have these two active and I'm pretty sure if I leave them out also Catalina won't boot with "host" setting.
-
On 7/1/2020 at 10:37 PM, fabiosun said:
Thank you @Rox67er
have you seen in this condition max frequency reached during occt stress test?
and temperature reached in 5 minute test as you did before?
yep, I see higher clock and also higher total power. Was around 230W initial, close to 280W with default setting and up to 330W with +200Mhz PBO overclock
-
On 6/30/2020 at 9:38 PM, Rox67er said:
Just set PBO to enable but still at 3700MHz ? 😲
Will look into this if I have some time.
Well after trying all PBO / overclock settings in bios without result (steady 3.7ghz on all cores) I installed Ryzen master and after one reboot I now see switching frequencies on all cores. Ranging from 3.4 to 4.2ghz. After this also the bios settings seem to work normal now... anyhow temps stay well under 75 degree and even below 85 with aggressive overclock (330 W)
-
1
-
-
13 hours ago, fabiosun said:
@Rox67er try to put PBO in enable from auto (if you want 🙂 )
if you see my freq is @ about 4000 on all cores (i use PBO enable button, in auto it is different in my MSI (lower))
Just set PBO to enable but still at 3700MHz ? 😲
Will look into this if I have some time.
-
On 6/27/2020 at 9:06 AM, fabiosun said:
with MSI boards it is possible to check temperature using internal Q-code display
In my case I see very high temperature when I put in a playing loop 500 or more tracks in Logic Pro
I booted up Win10 and ran 5 min of OCCT. 100% load but somehow all my cores run @ 3.7Ghz , need to check that. Anyhow temps max out at 72C
I run the 360 AIO from Cooler Master: https://www.coolermaster.com/catalog/coolers/cpu-liquid-coolers/masterliquid-ml360-rgb-tr4/
Noise level is very low.
-
1
-
-
7 hours ago, fabiosun said:
Done 🙂
Great to see that this path is still possible. 😁
-
On 6/24/2020 at 4:05 PM, fabiosun said:
tested now
both Catalina amd high Sierra hang in my case
I have tested to boot in windows via opencore boot loader launched from proxmox and it starts well
not tested ( I will do in a touch) my windows vm (no real disk)
And second pictures is a vm disk (Proxmox Windows 10)..in both VM I have put 56 as cores number (1 socket)
in VM situation first boot produces a hang , no proxmox logo. I have signals on both monitors but black screen) hard reset system restart again and it booted
Intersting to see that booting with real Windows installation (I use it normally) system is seen as an intel using my vm conf file (100.conf)
Using 104.conf file (windows VM config) system is seen as an AMD with 56 cores
thx for trying. macos is picky on topology but I would say if we pretend to be xeon W it should work as that is also in 28 core version..
-
Did anybody find a way to pass 28 cores / Xeon W as the original MacPro 2019 has? I can only boot 28 core with a very odd topology:
-smp 56,sockets=7,cores=8,threads=1
Displays as 7 times 8 Core intel core i5 in about this Mac.
-
Last week I replaced my external USB Proxmox drive for an extra internal M2 drive. I bought a XLR8 CS3030 1TB and installed a fresh Proxmox 6.2 on it. Went pretty smooth just copying the configs from my "old" setup. Anyhow everything works as expected again but as expected it didn't help with the USB problem.
Some new insight in trying to trouble shoot the USB passthrough problem on 48:00.1 / 48:00.3 while doing some tests.
Fact: We are unable to pass the USB controllers but if we only pass the BT USB everything works fine. I have noticed below error (USB passthrough) sometimes when booting Proxmox which after starting the VM always leads to a freeze / 99 error. Somehow linux is trying to initialise the LED controller and USB Audio (On 48:00.1 controller) but cannot access them...
So I tried the following: Next to BT I added the USB audio AND LED controller to my VM with:
usb0: host=05ac:828d # Bluetooth
usb1: host=26ce:01a2 # LED controller
usb2: host=26ce:0a01 # USB Audio
Booting VM is OK but reboot would hang, so this gives me more or less the same behaviour as when passing the complete controller. Next step I removed the LED controller and have been able to restart the VM a few times.
I think it may be related to going from linux to MacOs, back to linux and back to MacOs again... Also the LED and Audio are not recognised in lsusb
Bus 009 Device 002: ID 26ce:0a01
Bus 007 Device 006: ID 05ac:828d Apple, Inc.
Bus 007 Device 003: ID 0a5c:4500 Broadcom Corp. BCM2046B1 USB 2.0 Hub (part of BCM2046 Bluetooth)
Bus 007 Device 002: ID 26ce:01a2
Strange thing is that I disabled the LED in USB but apart from switching off the LED, the LED controller remains visible on USB. Would be nice if we could disable it at least to try if that solves the issue. I tried blocking linux from accessing it but so far not successfull. (Even with VFIO added, linux still boots with the xhci_pci kernel driver. (When starting the VM it is released and attached to vfio)
-
1
-
-
3 hours ago, Driftwood said:
@Rox67er So messing with Hardware tab I can discover which ports at least are the USB 3s and which are the 2s. The two right hand side (on my case) are USB 3 and the two left are USB2. Then of course you have the other USB-C port. these cant be passed through properly especially if you have the BT pass thru via Hardware TAB in Proxmox.s
So it looks like we're left with trying to find stability of the ports on 48:00.1 (1/2 rear/ 1/2 fromnt usb3) and 48:00.3 (probably front usb 2). Have you found out which address is front USB-C port yet?
Most of the info is in my overview: https://github.com/iroger/ASRock-TRX40-proxmox/blob/master/Ryzen IOMMU.pdf (2nd page)
The port 11 is a USB 3.2 gen 2 which I have connected to front USB-C. It is located on 48:00.3 (It is listed on 47:00.3 in the overview but I added a NVME 1TB drive for Proxmox)
-
1 minute ago, Driftwood said:
This is the missing piece of the jigsaw perhaps as Im not passing this controller.
Yep, it can be confusing but on old Mac Pro 5.1 I remember you had to connect a separate USB cable to get bluetooth working so I knew it needed a pci port for wifi AND a USB port for bluetooth.
Mine lists as manufacturer Apple Inc.
-
1 hour ago, Driftwood said:
My wifi works out the box... Just need to black list the BT I reckon plus maybe some of the kexts... Lets try this out.
@Driftwood It should work "out of the box" I have NO Broadcom kext loaded. Only LiLu / WEG / Virtual SMC / AppleALC.
blacklist.conf added:
blacklist btusb
blacklist bcma
vfio.conf added:
# WiFi PCI broadcom
options vfio-pci ids=14e4:43a0
In your VM forward:
hostpci: 45:00.0, pcie=1. #This will give you the wifi
Bluetooth is connected on USB so make sure you either forward the USB device or completely pass the USB controller to your VM (it is connected on 47:00.1 USB controller)
lsusb gives you the BT info:Bus 007 Device 003: ID 0a5c:4500 Broadcom Corp. BCM2046B1 USB 2.0 Hub (part of BCM2046 Bluetooth)If you don't forward the USB controller use the hardware passthrough in Proxmox to select the right device, lsusb somehow gives other data?
-
-
Have been trying the suggestion of @Driftwood regarding the XHCI biosd setting, found this link also suggesting to put them on "enable" in stead of "auto". https://forums.unraid.net/topic/87557-guide-asrock-x570-taichi-vm-w-hardware-passthrough/
I do see effect, after adding the setting I now am able to boot MacOs up to the login screen, and if I then shutdown and re-start the VM, all is fine. If I however login to MacOs and THEN shutdown I still run into the 99 error after second reboot. Have been trying other options (CSM / Above 4G / ...) but did not find a combination that worked...
-
@Driftwood from your VFIO I see you are passing the 05:00.3 and 25:00.3 USB as well? Are you using a patched kernel for this?
-
6 hours ago, Driftwood said:
@Rox67er Which USB ports are you trying to pass? I only use the ones supplied in my configs.
I also use XHCI in BIOS ALL variations 'Enabled' as they are notorious for going missing in 'Auto'.
Well only the two in the same group (47:00.1 and 47:00.3 in my case, I think you have 46:00.1 and 3 because you disabled internal wifi)
I haven't tried XHCI USB settings. Maybe will give that a try.
37 minutes ago, iGPU said:I don't know if it holds true for these VM builds, but on typical, bare-metal Hackintosh builds, USB problems have been associated with faulty Power Supplies. If you have another PS (and better yet, a different brand), it might be a useful test to swap out your current PS to see how this affects your USB functionality.
Thanks for the hint iGPU, I use a Corsair HX1200i but I did have an issue with the supply making strange static noises. (as if it was sparking) It did go away though and I checked all voltages and they seem OK. I don't have a spare on hand but will think about this.
-
17 hours ago, Driftwood said:
@Driftwood well still alive. Been having some short nights sleep over this USB issue. 🥴 I went back to using the USB-C port for my USB connections. I cannot get reliable boot with the internal ports passed through. Went to MacPro 7,1 bios but no improvement, changed the order of hostpci (which does change things in the VM, I had to re-assign the built-in property) but also no improvement. Spent hours tweaking bios settings. No improvement.
I still run 1.63 beta bios which seems fine. I think the improvement I had was coming from the bios upgrade where it resets all settings. (Not the same as load defaults in bios) I did a HW bios reset (button on the back) and got a few proper boots with USB passed through then get 99 error again every 2nd reboot. It seems like something gets written in bios which doesn't clear. I can tell if the reboot will work or not by listening to the GPU fans spinning up. If they only have a short whoosh everything is fine, if they already start spinning at the E0 Dr Debug code they spin longer, will boot ok but next reboot gives 99 error. 🤯
-
I found something interesting regarding passing all cores to VM.
When passing all 64 cores to OSX VM I noticed a short audio dropout when playing Spotify and in parallel starting up Fusion 360. I also saw the Proxmox disk led blinking indicating some activity. So I tried the same when only passing 32 cores to OSX VM. I started and stopped Fusion 360 several times and don't have one audio dropout.
So there definitely is some impact of passing all cores to the VM but I can't figure out what Fusion is doing to cause this.
I tried passing 56 cores but MacOs won't start. Seems only 32 and 64 work. Did anybody get other # cores working? How?
-
15 hours ago, Driftwood said:
This BIOS looks good @Rox67er... Im reporting back to ASRock. I can now run my 64 cores stable too with these BIOS settings.
I'm running Above 4G too and you get slightly better benchmarks.
Well I did run into the 99 error again today, I didn't change anything though. Still needs more testing I guess.
I tried performance run with Above 4G on / off results below (Geekbench 5 CPU and GPU Metal score)
Above 4G OFF
1282 / 23881 / 61463 (average of 4 runs)
Above 4G ON
1282 / 23714 / 60924 (average of 3 runs)
I would say no difference in my setup.
-
1
-
-
4 hours ago, Driftwood said:
ASRock Creator TRX40 board owners : I believe that the following should be set in BIOS. A little tip from ASRock that may or may not help us...
Advanced>Onboard Devices Configuration>SR-IOV Support>Enabled
Advanced>AMD CBS>NBIO Common Options>IOMMU>Enabled
Advanced>AMD CBS>NBIO Common option>Enable AER Cap>Enable
Yours is very nice @fabiosun nice and neat 🙂 It seems easier to config MSI!
ASRock techie said the report for IOMMU groups was done on the following (not using Proxmox)
It was just a fresh Ubuntu desktop 20.04 LTS installation: https://ubuntu.com/download/desktop
So that should be kernel 5.4.
Well close but not completely. After reading your post I jumped in and started tweaking the bios again....
Using the above settings crashes Proxmox on boot but I noticed that when enabling AER Cap a new option showed up in bios which was disabled. After also enabling that one I can stop-start-reboot and get no 99 error. Tried it now for like 8 times... 😁
So my current bios settings for ASRock TRX40 Creator:
(Still using Bios 1.63 but expect no impact from that to be honest)
* Start with default UEFI settings
* Set XMP profile for your RAM
* Tweaked FAN settings
* Advanced>Onboard Devices Configuration>SR-IOV Support>Enabled
* Advanced>AMD CBS>NBIO Common Options>IOMMU>Enabled
* Advanced>AMD CBS>NBIO Common option>Enable AER Cap>Enable
* Advanced>AMD CBS>NBIO Common option>ACS Enable>Enable
I did not need to set above 4G or anything else. Let's see if this is stable but looks like we nailed it !!! 😎
-
1
-
1
-
-
My IOMMU attached.
@Driftwood I already assumed he didn't use Proxmox I think it is kernel related how the devices are recognised. (Naming and drivers)
[Discussion] - TRX40 Bare Metal - Vanilla Patches
in General
Posted
Well just took the plunge and tried to get my Proxmox Catalina install booting in bare metal. Used the @iGPU EFI and it almost immediately worked. Had to tweak above 4G to Disabled in bios, I also removed all the BRCM kexts as I replaced the internal wifi with an Apple compatible one. Seems I need to add MMIO whitelist as the system won't shutdown (Getting sleep to work would be awesome) my bootlog is attached below. (Above 4G disabled)
05:165 00:002 OCABC: MMIO devirt start 05:167 00:002 OCABC: MMIO devirt 0xB2500000 (0x81 pages, 0x8000000000000001) skip 0 05:169 00:002 OCABC: MMIO devirt 0xB3580000 (0x81 pages, 0x8000000000000001) skip 0 05:171 00:002 OCABC: MMIO devirt 0xE2100000 (0x181 pages, 0x8000000000000001) skip 0 05:173 00:002 OCABC: MMIO devirt 0xFA180000 (0x81 pages, 0x8000000000000001) skip 0 05:176 00:002 OCABC: MMIO devirt 0xFA300000 (0x100 pages, 0x8000000000000001) skip 0 05:178 00:002 OCABC: MMIO devirt 0xFEA00000 (0x100 pages, 0x8000000000000001) skip 0 05:180 00:002 OCABC: MMIO devirt 0xFEC00000 (0x1 pages, 0x8000000000000001) skip 0 05:182 00:002 OCABC: MMIO devirt 0xFEC10000 (0x1 pages, 0x8000000000000001) skip 0 05:184 00:002 OCABC: MMIO devirt 0xFED00000 (0x1 pages, 0x8000000000000001) skip 0 05:186 00:002 OCABC: MMIO devirt 0xFED40000 (0x5 pages, 0x8000000000000001) skip 0 05:188 00:002 OCABC: MMIO devirt 0xFED80000 (0x10 pages, 0x8000000000000001) skip 0 05:191 00:002 OCABC: MMIO devirt 0xFEDC2000 (0xE pages, 0x8000000000000001) skip 0 05:193 00:002 OCABC: MMIO devirt 0xFEDD4000 (0x2 pages, 0x8000000000000001) skip 0 05:195 00:002 OCABC: MMIO devirt 0xFEE00000 (0x100 pages, 0x8000000000000001) skip 0 05:197 00:002 OCABC: MMIO devirt 0xFF000000 (0x1000 pages, 0x8000000000000001) skip 0 05:199 00:002 OCABC: MMIO devirt 0x10000000000 (0x10400 pages, 0x8000000000000001) skip 0 05:201 00:002 OCABC: MMIO devirt 0x3CB90000000 (0x10400 pages, 0x8000000000000001) skip 0 05:203 00:002 OCABC: MMIO devirt 0x3CBC0000000 (0x10400 pages, 0x8000000000000001) skip 0 05:206 00:002 OCABC: MMIO devirt 0x69750000000 (0x10400 pages, 0x8000000000000001) skip 0 05:208 00:002 OCABC: MMIO devirt end, saved 1087664 KB 05:210 00:002 OCABC: Only 176/256 slide values are usable! 05:212 00:002 OCABC: Valid slides - 80-255