Proxmox VE OSX Guide discussion

[iGPU]

14 hours ago, Driftwood said:

 

You've been where I have been. I didn't want to turn on 'that' particular USB controller - the one on my 48:00.1 (not doing 48:00.3) which is probably the same as your 47:00.1 & 3 to get BT working as it requires the power from this controller to work. But without it switched on its glitch free as far as firewire goes. With it on, yes BT works (you don't need it on for wifi) but then it introduces the errors with USB NAS dropouts, the stutter of BT if you switch off wifi, and the occasional audio glitching on my firewire card/fireface 800.

 

Ive made a point of this as a moan to ASRock to see if they have any more info out of AMD or other board manus as this problem seems across the board. Especially for those who need .3 switched on for HDAudio.

 

Ive looked at similar glitches on Linux unix boxes, and Windows. Its definitely a problem with this particular USB controller. The others are fine!

 

If theres no fix coming soon I think Im going to stop passing 48:00.1 and forget about BT and the USB ports it offers.

 

UPDATE: Just going to see if @Rox67er Add USB Device method is any better without passing the controller on 48:00.1 

 

 

I've repeated tests with ArcoLinux (and re-verified in Proxmox): both ASMedia as well as 48:00.0, 48:00.1 and 48:00.3, can be pass-through, and still have proper USB audio with no  audio dropout, as long as WiFi is disabled.

 

I think I'd not initially  noticed this phenomenon (if one looks at my original posts on this thread), as I was using the native BT/Wifi card with only the BT enabled. This is becausae WiFi on native AX200 cards has, as yet, no known kexts to activate WiFi on the macOS side. So I had unintentionally inactivated WiFi and good USB Audio with no dropouts.

 

So possibly, when one is using the TRX40 mobos with a Mac VM, one will need to make one of two choices. That is, based on what you've indicated and what I've found with WiFi, if USB ports and BT are more important and WiFi is not, then pass-through all USB devices but inactivate WiFi and the USB audio/FireWire audio should be okay. On the  other hand, if WiFi is necessary, then don't pass-through 48:00.x, and have fewer active USB ports and no BT, but functional WiFi.

 

***

 

As for PCIe AIC USB cards, I've not posted this to date on this forum, but I've not yet found a card that works on a VM with pass-through, and I've tried many (just as I've not yet gotten a working FireWire card: one more is on order!).

 

If one Googles on "PCIe AIC USB cards pass-through VM", one can find several threads discussing this issue, particularly starting in 2017. There are references to a few expensive cards, but once one then researches those cards, there are further posts saying that those cards have been modified over the past 3 years of production, and don't always work in a pass-through environment. This is the same situation I faced with FireWire: newer cards are re-designed and now contain a bridge that prevents VM pass-through.

 

But if you were seeing erratic behavior of your PCIe FireWire card when 48:00.3 is active, then a PCIe USB card will most likely suffer the same fate. This would mean, again reverting to my high-lighted paragraph above, where one must make a choice between USB ports and BT or WiFi for proper audio.

Edited June 22, 2020 by iGPU

[Driftwood]

@iGPU Interesting discoveries....

 

On mine, 48:00.3 is HDaudio (that weird USB one). 48:00.1 is rear USB, BT and LED functions. Im not sure if 48:00.3 is perhaps USB front panel also? (anyone tried?) Broadcom Wifi works out of the box without doing anything as its Apple supported.

 

I'm passing all the other USB controllers without problems.

 

I've turned passthru off again and just used @Rox67er method of passing BT using Proxmox 'Add USB Device' on the 'Hardware' section of the VM, and now no glitches. And that's a strange address for BT isnt it @Rox67er!!!?!!!

Edited June 22, 2020 by Driftwood

[fabiosun]

48:00 usb controller in my case is named 46:00 , it manages rear usb and internal usb connector used usually for front cabinet usb panel (2 internal usb 3.x connectors and maybe also the 3.2 one), not sure about 2 internal usb2 connectors

also it enables a part of our tricky audio.

it seems correct that could also be linked to WiFi problem present for audio drop or bad behaviour

i have no WiFi/bt card/device so my zero problem in 10.15.5 is explained 

high Sierra and maybe latest Catalina could have additional problems with too old usb drivers (high Sierra) or too new (10.15.6 beta x)

[iGPU]

12 hours ago, Driftwood said:

I'm not sure if 48:00.3 is perhaps USB front panel also? (anyone tried?)

 

Yes, I tried and posted rear panel once or twice before. I'll repeat here again.

 

Since we're all using TRX40 mobos and the TRX40 chips are designed by AMD, all of the mobos will have a similar pattern (the exact x-value of x.1 and x.3 will vary).

 

The  x.1 is for the audio and the bulk of the the rear panel USB ports. While x.3 is for BT and the internal USB headers. (Note the internal USB3 headers listed at top.) I chose not to pass-through any 4:00 or 25:00 devices.

 

If you feel that the audio is a problem and you do NOT want to pass x.1, then you might consider adding a part like this, connected to an internal header (x.3). Such an expansion header will provide extra USB ports on the rear panel (but occupy one PCI slot), which would otherwise be absent if x.1 were not passed.

 

 

MSI Creator Rear Panel:

[fabiosun]

@iGPUthey have an Asrock but your thoughts are fine also for them

 

 

[Driftwood]

1 hour ago, iGPU said:

Yes, I tried and posted rear panel once or twice before. I'll repeat here again

 

@iGPU Yes, but our ASRocks are different layout as @fabiosun has just mentioned. 48:00.1 is rear panel (2 USB ports) a BT power, and LED connectors. 48:00.3 is probably the four front panel USBs and definitely HD Audio! There is a also a front side USB-C which could be attributed to this controller... or the ASMedia. We have a ASMedia USB-C connector on the back (last port in my layout picture posted a few days back) at definitely 44:00!

 

The fun of these mobo layout designs! 🙂

 

I am currently going through all the ASRock board ports to identify front panel USB and revise my mobo layout diagram.

 

 

Edited June 22, 2020 by Driftwood

[fabiosun]

3 hours ago, Driftwood said:

 

----

 

I am currently going through all the ASRock board ports to identify front panel USB and revise my mobo layout diagram.

 

 

you will be surprised..

if you do want a "definitive" job in this task..try to put on each usb an usb pen 3 and an usb 2 pen..and see how system detects it (if)

 

[Driftwood]

2 hours ago, fabiosun said:

5 hours ago, Driftwood said:

Read more  

you will be surprised..

if you do want a "definitive" job in this task..try to put on each usb an usb pen 3 and an usb 2 pen..and see how system detects it (if)

Probably will be!

 

Guess we can always use Proxmox Hardware tab to learn of every device specific address you plug in and then just pass it.. like the BT problem was overcome this way!

 

@fabiosun did u try passing from Proxmox Hardware tab  the HD audio address (maybe Linux detects its as pulse audio) ?

Edited June 22, 2020 by fabiosun

[fabiosun]

@Driftwoodif you not pass 46:00 in my case or 48:00 in your you can pass audio as device in proxmox web interface

but, if you have problem this way does not solve it about audio
pulse audio is a different thing and surely is not supported by our loved OS X system

 

[Driftwood]

@Rox67er So messing with Hardware tab I can discover which ports at least are the USB 3s and which are the 2s. The two right hand side (on my case) are USB 3 and the two left are USB2. Then of course you have the other USB-C port. these cant be passed through properly especially if you have the BT pass thru via Hardware TAB in Proxmox.s

 

So it looks like we're left with trying to find stability of the ports on 48:00.1 (1/2 rear/ 1/2 fromnt usb3) and 48:00.3 (probably front usb 2). Have you found out which address is front USB-C port yet?

Edited June 23, 2020 by Driftwood

[fabiosun]

@Driftwoodif I have understood well you have discovered now a things I have said some time ago

to confirm:

Your external cabinet USB, where are they connected?

to these connector as picture below?

 

thank you

[Driftwood]

20 minutes ago, fabiosun said:

if I have understood well you have discovered now a things I have said some time ago

to confirm:

Pretty much - yes, as in my diagram some posts back. lower one is usb2 top right usb 3

[fabiosun]

those controllers are 3.1 and maybe 3.2 the right ones

Catalina sees one of them as USB2 if you insert on them also USB3 devices

In high Sierra all ports are seen as USB2 (internal, external, back and so on)

I think it is driver related and also a proxmox cfg file used for compatibility with old ports type

 

 

Edited June 23, 2020 by fabiosun
picture added

[fabiosun]

the interesting thing is that we also have the same problem in the windows virtual machine. While in the same audio always works without any problem, usb recognition has the same problems as the OSX version

 

that's why I had asked you colleagues (of chipset) to map all the USB ports by inserting into the same usb2 and 3 peripherals accordingly, to figure out together how to deal with the problem with the ProxMox support guys 🙂

Edited June 23, 2020 by fabiosun
Add usb proxmox part

[Rox67er]

3 hours ago, Driftwood said:

@Rox67er So messing with Hardware tab I can discover which ports at least are the USB 3s and which are the 2s. The two right hand side (on my case) are USB 3 and the two left are USB2. Then of course you have the other USB-C port. these cant be passed through properly especially if you have the BT pass thru via Hardware TAB in Proxmox.s

 

So it looks like we're left with trying to find stability of the ports on 48:00.1 (1/2 rear/ 1/2 fromnt usb3) and 48:00.3 (probably front usb 2). Have you found out which address is front USB-C port yet?

Most of the info is in my overview: https://github.com/iroger/ASRock-TRX40-proxmox/blob/master/Ryzen IOMMU.pdf (2nd page)

The port 11 is a USB 3.2 gen 2 which I have connected to front USB-C. It is located on 48:00.3 (It is listed on 47:00.3 in the overview but I added a NVME 1TB drive for Proxmox)

[Rox67er]

Last week I replaced my external USB Proxmox drive for an extra internal M2 drive. I bought a XLR8 CS3030 1TB and installed a fresh Proxmox 6.2 on it. Went pretty smooth just copying the configs  from my "old" setup. Anyhow everything works as expected again but as expected it didn't help with the USB problem.

 

Some new insight in trying to trouble shoot the USB passthrough problem on 48:00.1 / 48:00.3 while doing some tests.

 

Fact:  We are unable to pass the USB controllers but if we only pass the BT USB everything works fine. I have noticed below error (USB passthrough) sometimes when booting Proxmox which after starting the VM always leads to a freeze / 99 error. Somehow linux is trying to initialise the LED controller and USB Audio (On 48:00.1 controller) but cannot access them...

 

So I tried the following: Next to BT I added the USB audio AND LED controller to my VM with:

usb0: host=05ac:828d # Bluetooth

usb1: host=26ce:01a2 # LED controller 

usb2: host=26ce:0a01 # USB Audio

 

Booting VM is OK but reboot would hang, so this gives me more or less the same behaviour as when passing the complete controller. Next step I removed the LED controller and have been able to restart the VM a few times.

 

I think it may be related to going from linux to MacOs, back to linux and back to MacOs again...  Also the LED and Audio are not recognised in lsusb

Bus 009 Device 002: ID 26ce:0a01

Bus 007 Device 006: ID 05ac:828d Apple, Inc.

Bus 007 Device 003: ID 0a5c:4500 Broadcom Corp. BCM2046B1 USB 2.0 Hub (part of BCM2046 Bluetooth)

Bus 007 Device 002: ID 26ce:01a2

 

Strange thing is that I disabled the LED in USB but apart from switching off the LED,  the LED controller remains visible on USB. Would be nice if we could disable it at least to try if that solves the issue. I tried blocking linux from accessing it but so far not successfull. (Even with VFIO added, linux still boots with the xhci_pci kernel driver. (When starting the VM it is released and attached to vfio)

 

 

[Driftwood]

3 hours ago, Rox67er said:

Strange thing is that I disabled the LED in USB but apart from switching off the LED,  the LED controller remains visible on USB. Would be nice if we could disable it at least to try if that solves the issue. I tried blocking linux from accessing it but so far not successfull. (Even with VFIO added, linux still boots with the xhci_pci kernel driver. (When starting the VM it is released and attached to vfio)

 

Yeah, Im kind of at the same opinion and have emailed ASRock.

[iGPU]

fabiosun,

 

You previously asked about running the 2-Candle test in DaVinci Resolve with two Radeon VIIs. I finally got around to testing. Presently, neither WEG nor RadeonBoost is enabled, and no GPU DeviceProperties are being injected, so everything very stock. In DaVinci Resolve, the GPU Preferences were set to Auto with both GPUs active; 66 nodes were left active (the maximum). macOS was Catalina 10.15.6-ß2.

 

When one Radeon VII is used, the result was 16 fps. From my searches, this is a typical result for one Radeon VII (and the same result commonly observed when using one 2080 Ti).

 

When using two Radeon VIIs, the result was 24 fps as shown below 35 fps. I plan on converting both Radeon VIIs to water-cooled in a week or so and will then overclock them and re-test.

 

 

 

Edited June 24, 2020 by iGPU
corrected project framerate per fabiosun

[fabiosun]

@iGPU thank you for testing

have you unlocked FPS in project settings?

it seems capped at 24 FPS 

you have to put a value different from default project FPS value like 200 FPS instead default ( it is 24 by default)

You can also test with 6 Trn nodes and with 66 noise node as you did before

[iGPU]

11 minutes ago, fabiosun said:

@iGPU thank you for testing

have you unlocked FPS in project settings?

it seems capped at 24 FPS 

you have to put a value different from default project FPS value like 200 FPS instead default ( it is 24 by default)

You can also test with 6 Trn nodes and with 66 noise node as you did before

 

You're absolutely correct!

 

Project was set at 1080p @ 24 fps. I re-set to 1080p @ 120 fps, and then saw a maximum of 35 fps (typically 28-32) when running both Radeon VIIs.

Edited June 24, 2020 by iGPU

[fabiosun]

Ok now it is good as it should be

thank you for testing again

[Rox67er]

Did anybody find a way to pass 28 cores / Xeon W as the original MacPro 2019 has? I can only boot 28 core with a very odd topology:

 

-smp 56,sockets=7,cores=8,threads=1

 

Displays as 7 times 8 Core intel core i5 in about this Mac.

[fabiosun]

51 minutes ago, Rox67er said:

Did anybody find a way to pass 28 cores / Xeon W as the original MacPro 2019 has? I can only boot 28 core with a very odd topology:

 

-smp 56,sockets=7,cores=8,threads=1

 

Displays as 7 times 8 Core intel core i5 in about this Mac.

tested now

both Catalina amd high Sierra hang in my case

I have tested to boot in windows via opencore boot loader launched from proxmox and it starts well

not tested ( I will do in a touch) my windows vm (no real disk)

 

 

And second pictures is a vm disk (Proxmox Windows 10)..in both VM I have put 56 as cores number (1 socket)

in VM situation first boot produces a hang , no proxmox logo. I have signals on both monitors but black screen) hard reset system restart again and it booted

 

Intersting to see that booting with real Windows installation (I use it normally) system is seen as an intel using my vm conf file (100.conf)

Using 104.conf file (windows VM config) system is seen as an AMD with 56 cores

Edited June 24, 2020 by fabiosun
added windows VM image and informations

[fabiosun]

vm 104.conf (Windows)

 

bios: ovmf
bootdisk: sata0
cores: 56
cpu: host
efidisk0: local-lvm:vm-104-disk-1,size=4M
hostpci0: 21:00,pcie=1,x-vga=1
hostpci1: 46:00.3
hostpci2: 04:00.3
hostpci3: 23:00.3
hostpci6: 46:00.1
ide0: local:iso/virtio-win-0.1.173.iso,media=cdrom,size=384670K
machine: q35
memory: 65536
name: Windows
net0: e1000=xx:90:xx:45:xF:xx,bridge=vmbr0,firewall=1
numa: 0
ostype: win10
sata0: local-lvm:vm-104-disk-0,size=32G
scsihw: virtio-scsi-pci
smbios1: uuid=xxxxx
sockets: 1
tablet: 0
vga: none
vmgenid: xxxxx

 

 

 

vm 100.conf (OSX)

#hostpci11%3A 04%3A00.3
#hostpci7%3A 23%3A00.3
#hostpci8%3A 62%3A00.0
#hostpci8%3A 46%3A00.3
#hostpci8%3A 42%3A08.0
#usb0%3A host=04d9%3A1702
#usb1%3A host=046d%3Ac050
#usb2%3A host=0c76%3A161f
#hostpci12%3A 49%3A00
#hostpci9%3A 46%3A00.1
#hostpci13%3A 23%3A00.4
agent: 0
args: -smbios type=2 -cpu host,vendor=GenuineIntel,+invtsc
balloon: 0
bios: ovmf
boot: d
cores: 56
cpu: host
efidisk0: local-lvm:vm-100-disk-1,size=4M
hostpci0: 21:00,pcie=1,x-vga=1
hostpci1: 46:00.1
hostpci10: 46:00.3
hostpci11: 47:00
hostpci12: 48:00
hostpci2: 01:00.0
hostpci3: 02:00.0
hostpci4: 04:00.3
hostpci5: 23:00.3
hostpci6: 23:00.4
hostpci7: 44:00
hostpci8: 45:00
ide2: local:iso/r5107.iso,cache=unsafe
machine: q35
memory: 121856
name: sierra
net0: vmxnet3=xx:xx:xx:xx:xx:xx,bridge=vmbr0,firewall=1
numa: 0
ostype: other
sata0: local-lvm:vm-106-disk-0,cache=unsafe,size=48G,ssd=1
scsihw: virtio-scsi-pci
smbios1: xxxxxx
sockets: 1
startup: down=20
tablet: 0
vga: none
vmgenid: xxxxxxx

 

[fabiosun]

 

Big Sur installed..for now no boot 😂