calling nvidia-smi in udev rule is a gap to the driver from Nvidia

the strace of gnome-shell of success caseB.
check line 137927
mknod("/dev/nvidiactl", S_IFCHR|0666, makedev(195, 255)) = -1 EACCES (Permission denied)
then it call /usr/bin/nvidia-modprobe to create it.

the strace of gnome-shell of failed caseA.
check line 5181
mknod("/dev/nvidiactl", S_IFCHR|0666, makedev(195, 255)) = -1 EACCES (Permission denied)

Then it can not find nvidia-modprobe , so gave up create device node
stat("/usr/bin/nvidia-modprobe", 0x7ffd5ddb3f50) = -1 ENOENT (No such file or directory)

short term workaround:
 - https://github.com/tseliot/nvidia-graphics-drivers/pull/35
 - ppa for test: https://code.launchpad.net/~alextu/+recipe/lp1839309-not-use-nvidia-smi-in-udev-rule

verified on
 - ice lake machine (BIOS ID: 097B)
 - Precision-7730 NVIDIA Corporation GP104GLM [Quadro P5200 Mobile] [10de:1bb5]

next step:
 - verify the built out packages from ppa : https://code.launchpad.net/~alextu/+recipe/lp1839309-not-use-nvidia-smi-in-udev-rule

I have began some work to try implement device_create() support into nvidia driver recently, but I soon bumped into a problem that I had to stop.

device_create() takes a `struct class` pointer, which is supposed to be created by class_create(). However, class_create() along with all other device class APIs are exposed GPL-only, which follows there is no way nvidia proprietary driver may ever access them. End story. Some other work-around must be used.

Tagging all nvidia-smi related bugs with "nvidia-smi" tag. Tracking url: https://bugs.launchpad.net/somerville/+bugs?field.searchtext=&orderby=-importance&field.status%3Alist=NEW&field.status%3Alist=CONFIRMED&field.status%3Alist=TRIAGED&field.status%3Alist=INPROGRESS&field.status%3Alist=FIXCOMMITTED&field.status%3Alist=FIXRELEASED&field.status%3Alist=INCOMPLETE_WITH_RESPONSE&field.status%3Alist=INCOMPLETE_WITHOUT_RESPONSE&assignee_option=any&field.assignee=&field.bug_reporter=&field.bug_commenter=&field.subscriber=&field.structural_subscriber=&field.tag=nvidia-smi&field.tags_combinator=ANY&field.has_cve.used=&field.omit_dupes.used=&field.omit_dupes=on&field.affects_me.used=&field.has_patch.used=&field.has_branches.used=&field.has_branches=on&field.has_no_branches.used=&field.has_no_branches=on&field.has_blueprints.used=&field.has_blueprints=on&field.has_no_blueprints.used=&field.has_no_blueprints=on&search=Search

I think that using udev and another tool to create device nodes would be a simpler solution. I had some code to do that, but I never got around to completing it. I can have another look at this.

@Alberto, we're on you now. But please don't hesitate to assign to me if you don't really have the time recently.

I have provided the tool that I promised in this PPA:

https://launchpad.net/~oem-solutions-group/+archive/ubuntu/nvidia-driver-staging

The fix is only available for nvidia-graphics-drivers-440 (440.64-0ubuntu3~0.20.04.2 for Focal, 440.64-0ubuntu0~0.19.10.4 for Eoan, 440.64-0ubuntu0~0.18.04.3 for Bionic).

Please test it and let me know if you find any issues.

To verify:

* bug 1834323, which depending on nvidia-smi being executed in udev hooks
* bug 1812784, which depending on nvidia-smi not being executed in udev hooks

Verified bug 1834323 with 440.64-0ubuntu0~0.18.04.3 for Bionic, in performance or power saving mode, and suspend/resume under DC and AC mode. So far so good, except there are two (probably unrelated) issues:

  1. X hangs when switching from power saving mode to performance mode by nvidia-settings.
  2. /sbin/ub-device-create is not stripped

No device at hand for bug 1812784 yet.

dmesg log for case 1 in comment #11. I tested suspend/resumes under power saving mode first, and then try to switch to performance mode. When it hangs X, I find nvidia modules are being loaded in the background `dmesg -w` terminal window:

  nvidia-nvlink: Nvlink Core is being initialized, major device number 237
  NVRM: loading NVIDIA UNIX x86_64 Kernel Module 440.64 Fri Feb 21 01:17:26 UTC 2020
  nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms 440.64 Fri Feb 21 00:43:19 UTC 2020
  [drm] [nvidia-drm] [GPU ID 0x00000100] Loading driver
  ...

Then I have to reboot with sysrq.

This doesn't always happen as I tried a few times now but cannot reproduce. Seen twice before comment #11.

Updated from @tseliot on IRC

- The device-create tool is in Groovy now, and it will make its way into Bionic and Focal on the next update

All the drivers (except for the 340 driver) should have the device-create tool in Bionic and Focal.