Add alias to support Microsoft Azure RDMA device via MLX4

Bug #1838939 reported by Christian Ehrhardt 
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
rdma-core (Ubuntu)
Fix Released
Medium
Unassigned
Bionic
Fix Released
Medium
Unassigned

Bug Description

[Impact]

 * Azure RDMA devices can be a "Network Direct" version of the MLX
   InfiniBand card that is exposed to the guest and they need to be
   handled by the MLX4 driver.
   But to be enabled properly the system needs to detect them as such.
   That was added in rdma-core v20 hence being fix released in >=Disco.

 * Fix [1] is just a new alias to detect the card as what it is.

[Test Case]

 * The TL;DR is to use rdma on "Microsoft Azure RDMA device".
   There is no good way without an Azure instance to test this specific
   bug. The check is if the userspace drivers are then correctly loaded.
   Lacking such an instance myself Microsoft helped on the PPA and will
   help on SRU verification.

[Regression Potential]

 * a) The regression potential should be minimal. Existing systems either
   already match the HCA table or they don't (neither before nor after the
   fix. An issue I could think of is if there are devices that announce as
   vmbus:3daf2e8ca732094bab99bd1f1c86b501 but are NOT in fact
   this device type - they would then run into issues.
 * b) Of course the fix will "enable rdma support on some more devices" if
   someone had those devices attached but didn't use/configure them they
   might now initialize a bit further. But that isn't
   an issue and especially in cloud environments where HW config is just a
   click away no one usually pays for extra devices without using them.
 * c) I happen to know that for DPDK usage of these devices several fixes
   in later rdma-core are preferred or even needed. I'm not sure about the
   usage in this case - but enabling could expose those issues which
   formerly were hidden behind the "not supported" misdetection of the
   card.

For B) and C) I'd want Microsoft to ack and test this from a PPA and do
the verification on this - to be not only ok for this but for for Azure
in general.

[Other Info]

 * This falls under the SRU exception of "other safe cases" for "For Long
   Term Support releases we regularly want to enable new hardware". New
   modaliases are explicitly listed there as cases for this.

 * This is reported to work in 16.04, but not as a classic "this commit
   broke it regression". Back then the world of rdma/infiniband just
   worked totally different as it was before the big revamp into rdma-core
   never the less one could see that (abstract, not caring about details)
   as an update-regression when going 16.04 to 18.04.

[1]: https://github.com/linux-rdma/rdma-core/commit/b98397f3068a42e7d8e67d4ef2a90393b4c4f08a

Related branches

Changed in rdma-core (Ubuntu):
status: New → Triaged
importance: Undecided → Medium
description: updated
description: updated
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

I pinged Microsoft to add some details/statements here that only they can ...

Revision history for this message
Joshua R. Poulson (jrp) wrote :

This change only affects the "Network Direct" version of the MLX InfiniBand card that is exposed to the guest, it is not an SRIOV or DPDK case. Those appear as Mellanox cards as expected. For what it's worth this works fine in 16.04 but is broken in 18.04.

Revision history for this message
Long Li (longli) wrote :

When kernel device is probed, its vendor ID and device ID are used to ask user-mode library to find the correct device driver in user-mode. The patch will enable the correct user-mode driver is selected. There is no additional configuration or time spent on the device initialization, because at this time the kernel-mode driver is already loaded and running, regardless if user-mode driver is loaded or net. To end user, the cost and billing are not dependent on what driver is in use. If the hardware is attached to VM, the billing is already in progress.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

FYI - this can be pre-tested to work against PPA [1]. I'd be very happy if one could
a) do that test if it is sufficient
b) dump the steps to test here then so that we can complete the SRU template to get things started

[1]: https://launchpad.net/~paelzer/+archive/ubuntu/bug-1838939-rdma-azure-mlx4

description: updated
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Packaging the fix is easy, here [1] is a merge proposal for my fellow packagers to review just to be sure.

[1]: https://code.launchpad.net/~paelzer/ubuntu/+source/rdma-core/+git/rdma-core/+merge/371072

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

The merge request was reviewed and approved, to be as clear as possible - what we now need to unblock this and continue is:
- testing the PPA if it is fulfilling the needs (card detection) and works fine (detected card works properly in your tests)
- description of test steps done (so it can be used for the SRU template)

Revision history for this message
Joshua R. Poulson (jrp) wrote :

Please proceed, no regressions.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Thanks Josh for the extended checks, I have tagged it up for Bionic correctly and sponsored it for the review of the SRU [1] Team. Once they accept it into Bionic-proposed they will ask for a verification, it would be great if you again could check then that it works. It could, but doesn't necessarily be the full check, but at least if the device recognition that was planned to be fixed really works with that build (of the same source) still.

[1]: https://wiki.ubuntu.com/StableReleaseUpdates

Changed in rdma-core (Ubuntu Bionic):
status: New → Triaged
importance: Undecided → Medium
Changed in rdma-core (Ubuntu):
status: Triaged → Fix Released
description: updated
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

FYI: as I was asked what this is waiting on - this was waiting for review and acceptance [1] by the SRU Team for a while now. I have pinged the SRU team vaguards of today to get it processed rather soon.
But I beg your pardon in advance as there is a sprint going on and due to that there might still some time to pass before this is done.

[1]: https://wiki.ubuntu.com/StableReleaseUpdates#Procedure

Revision history for this message
Robie Basak (racb) wrote : Please test proposed package

Hello Christian, or anyone else affected,

Accepted rdma-core into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/rdma-core/17.1-1ubuntu0.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in rdma-core (Ubuntu Bionic):
status: Triaged → Fix Committed
tags: added: verification-needed verification-needed-bionic
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

@Joshua R. Poulson (jrp) would you mind having this tested on Azure to have it SRU-released?
Update the bug here once the fix was confirmed so it can be marked accordingly.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Still waiting for verification - I'll ask on side channels as well ...

Revision history for this message
Rakesh Ginjupalli (linuxelf001) wrote :

Test engineer reports success and no issues. For more details, please refer support case 00237568.

test-vm:~# ibv_devices
    device node GUID
    ------ ----------------
    mlx4_0 00155d33ff0f0000

tags: added: verification-done verification-done-bionic
removed: verification-needed verification-needed-bionic
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package rdma-core - 17.1-1ubuntu0.2

---------------
rdma-core (17.1-1ubuntu0.2) bionic; urgency=medium

  * d/p/u/lp-1838939-Add-Microsoft-Azure-RDMA-device-to-MLX4-HCA-table.patch:
    Detect Azure RDMA devices as MLX4 (LP: #1838939)

 -- Christian Ehrhardt <email address hidden> Mon, 05 Aug 2019 08:57:39 +0200

Changed in rdma-core (Ubuntu Bionic):
status: Fix Committed → Fix Released
Revision history for this message
Robie Basak (racb) wrote : Update Released

The verification of the Stable Release Update for rdma-core has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.