[oneiric] Keyboard & mouse not working in X - incomplete migration to /run

Log file shows no input devices. Possibly some issue with udev.

bug 807291 seems to confirm this.

Thanks for your report.

Do you know after which package update the problem started ?
Could you please attach the file /var/log/apt/history.log

Thanks in advance

Hi,
  Here are the contents of my history.log - there were quite a few things that got updated - I'd also installed the -dbgsym for the xserver just before which updated the xserver-xorg-core package:

-----------------
Start-Date: 2011-07-08 01:09:34
Commandline: apt-get install xserver-xorg-core-dbgsym
Install: xserver-xorg-core-dbgsym:i386 (1.10.2.902-1ubuntu1)
Upgrade: xserver-common:i386 (1.10.2-1ubuntu1, 1.10.2.902-1ubuntu1), xserver-xorg-core:i386 (1.10.2-1ubuntu1, 1.10.2.902-1ubuntu1)
End-Date: 2011-07-08 01:09:39

Start-Date: 2011-07-08 01:20:41
Commandline: apt-get dist-upgrade
Install: libglapi-mesa:i386 (7.11~1-0ubuntu3, automatic), thunderbird-globalmenu:i386 (5.0+build1+nobinonly-0ubuntu4, automatic), libmtp9:i386 (1.1.0-2ubuntu1, automatic), linux-headers-3.0-3:i386 (3.0-3.4, automatic), libdbusmenu-glib4:i386 (0.4.90-0ubuntu3, automatic), thunderbird:i386 (5.0+build1+nobinonly-0ubuntu4), thunderbird-gnome-support:i386 (5.0+build1+nobinonly-0ubuntu4), accountsservice:i386 (0.6.12-3, automatic), libdbusmenu-gtk3-4:i386 (0.4.90-0ubuntu3, automatic), linux-headers-3.0-3-generic:i386 (3.0-3.4, automatic), linux-image-3.0-3-generic:i386 (3.0-3.4), libgnome-desktop-3-2:i386 (3.1.3-0ubuntu1, automatic), libgbm1:i386 (7.11~1-0ubuntu3, automatic), libaccountsservice0:i386 (0.6.12-3, automatic), libdbusmenu-gtk4:i386 (0.4.90-0ubuntu3, automatic), indicator-sound-gtk2:i386 (0.7.2-0ubuntu1, automatic), libmtp-runtime:i386 (1.1.0-2ubuntu1, automatic), libopenvg1-mesa:i386 (7.11~1-0ubuntu3, automatic)

-----------

In terms of udev messages, in the dmesg I did notice:

[ 9.407077] udevd[245]: starting version 171
.....
[ 9.512717] init: udev-fallback-graphics main process (306) terminated with status 1

Hmm, from that list only the new linux-image seems likely. And that's easy enough to prove/disprove - reboot, hold down the left shift to bring up grub, and boot into a prior kernel version. If it works properly then, that points to the kernel.

The history.log is interesting, but the dpkg.log might have more. Could you attach /var/log/dpkg.log?

Neither of the udev messages look relevant to input problems, although udev could still be a suspect. Thanks for digging those messages out. That second one does need solved though. I think that's just failsafe-x, which doesn't work with lightdm yet.

The older kernels (3.0-1 and 2.6.39 something) also don't work, so not that.
One thing I've had ever since I first upgraded that vm to OO is that during boot I get the message:
    error: runtime directory '/run/udev' not writeable, for now falling back .....

that's very early on (initrd?) - thought I'd mention it if we're looking at udev stuff.

dpkg-log's below. Note I've just tried

sudo stop lightdm
sudo start gdm

and gdm has the same issue.

I've also just tried:
update-initramfs -k all -u

and rebooted, didn't help

On Fri, Jul 08, 2011 at 10:44:31PM -0000, Dave Gilbert wrote:
> The older kernels (3.0-1 and 2.6.39 something) also don't work, so not that.
> One thing I've had ever since I first upgraded that vm to OO is that during boot I get the message:
> error: runtime directory '/run/udev' not writeable, for now falling back .....

Yeah that's bug #784216; unrelated to this problem.

On Fri, Jul 08, 2011 at 10:44:31PM -0000, Dave Gilbert wrote:
> dpkg-log's below.

Thanks for that. So, attached is the tabulated list of upgraded
packages. Quite a few actually, although few that have to do with
input, and only a small number of X packages.

Still, I would suggest testing downgrading individual packages to
isolate which caused the regression. The most obvious ones to try first
would be udev, xserver-xorg-input-evdev, and xorg-server (which provides
both xserver-common and xserver-xorg-core).

If you still have the old debs in your apt cache you may be able to
just do something like:

  apt-get install xserver-xorg-input-evdev=1:2.6.0-1ubuntu13

etc.

If not, you can snag the .debs from launchpad.

Similar bugs:
bug 807538
bug 807291

Yeuch this is slow going; not hit anything that fixes it yet - so far:

dpkg -i udev_171-0ubuntu3_i386.deb libudev0_171-0ubuntu3_i386.deb libgudev-1.0-0_1%3a171-0ubuntu3_i386.deb
upgrade udev 2011-07-08 01:21:43 171-0ubuntu3 171-0ubuntu4
upgrade udev (167-0ubuntu3) 2011-07-08 01:24:28 1:171-0ubuntu3 1:171-0ubuntu4

dpkg --purge xserver-xorg-core-dbgsym
install xserver-xorg-core-dbgsym 2011-07-08 01:09:36 <none> 2:1.10.2.902-1ubuntu1

wget https://launchpad.net/ubuntu/+archive/primary/+files/xserver-xorg-input-evdev_2.6.0-1ubuntu12_i386.deb ?? right version??
dpkg -i xserver-xorg-input-evdev_2.6.0-1ubuntu12_i386.deb
upgrade xserver-xorg-input-evdev 2011-07-08 01:29:52 1:2.6.0-1ubuntu12 1:2.6.0-1ubuntu13

dpkg -i xserver-xorg-core_2%3a1.10.2-1ubuntu1_i386.deb
upgrade xserver-xorg-core 2011-07-08 01:09:35 2:1.10.2-1ubuntu1 2:1.10.2.902-1ubuntu1

dpkg -i xserver-common_2%3a1.10.2-1ubuntu1_all.deb
upgrade xserver-common 2011-07-08 01:09:34 2:1.10.2-1ubuntu1 2:1.10.2.902-1ubuntu1

dpkg -i xserver-xorg-input-synaptics_1.3.99+git20110116.0e27ce3a-0ubuntu15_i386.deb
upgrade xserver-xorg-input-synaptics 2011-07-08 01:29:57 1.3.99+git20110116.0e27ce3a-0ubuntu15 1.4.1-1ubuntu1

wget https://launchpad.net/ubuntu/+source/base-files/6.3ubuntu2/+build/2482825/+files/base-files_6.3ubuntu2_i386.deb
dpkg -i base-files_6.3ubuntu2_i386.deb
upgrade base-files 2011-07-08 01:23:13 6.3ubuntu2 6.3ubuntu4

dpkg -i evince_3.0.2-0ubuntu4_i386.deb evince-common_3.0.2-0ubuntu4_all.deb libevince3-3_3.0.2-0ubuntu4_i386.deb
upgrade evince 2011-07-08 01:27:01 3.0.2-0ubuntu4 3.1.2-0ubuntu1
upgrade evince-common 2011-07-08 01:27:03 3.0.2-0ubuntu4 3.1.2-0ubuntu1
upgrade libevince3-3 2011-07-08 01:27:02 3.0.2-0ubuntu4 3.1.2-0ubuntu1

dpkg -i binutils_2.21.52.20110606-1ubuntu1_i386.deb
upgrade binutils 2011-07-08 01:21:24 2.21.52.20110606-1ubuntu1 2.21.52.20110707-1ubuntu1

wget https://launchpad.net/ubuntu/+source/dbus/1.4.8-3ubuntu4/+build/2523636/+files/dbus_1.4.8-3ubuntu4_i386.deb
wget https://launchpad.net/ubuntu/oneiric/i386/dbus-x11/1.4.8-3ubuntu4 <!!!! bad name?
wget https://launchpad.net/ubuntu/+source/dbus/1.4.8-3ubuntu4/+build/2523636/+files/libdbus-1-3_1.4.8-3ubuntu4_i386.deb
wget http://launchpadlibrarian.net/72357660/dbus-x11_1.4.8-3ubuntu4_i386.deb
dpkg -i dbus_1.4.8-3ubuntu4_i386.deb dbus-x11_1.4.8-3ubuntu4_i386.deb libdbus-1-3_1.4.8-3ubuntu4_i386.deb
upgrade dbus 2011-07-08 01:21:43 1.4.8-3ubuntu4 1.4.12-4ubuntu1

  229 wget https://launchpad.net/ubuntu/+source/policykit-desktop-privileges/0.4/+build/2258553/+files/policykit-desktop-privileges_0.4_all.deb
  230 dpkg -i policykit-desktop-privileges_0.4_all.deb
upgrade policykit-desktop-privileges 2011-07-08 01:29:24 0.4 ...

Hey Bryce, just chiming in with a me too on this one... Let me know if I can be of any help. My system was a natty system that started showing this behavior immediately after distro upgrade yesterday.

I tried starting X with --verbose 99 - the following looks relevant:

(II) config/udev: ignoring device /dev/input/event0 without property ID_INPUT set
(II) config/udev: ignoring device /dev/sr0 without property ID_INPUT set
(II) config/udev: ignoring device /dev/bsg/1:0:0:0 without property ID_INPUT set
(II) config/udev: ignoring device /dev/sg0 without property ID_INPUT set
(II) config/udev: ignoring device /dev/bus/usb/001/001 without property ID_INPUT set
(II) config/udev: ignoring device /dev/usbmon1 without property ID_INPUT set
(II) config/udev: ignoring device /dev/vda without property ID_INPUT set
(II) config/udev: ignoring device /dev/vda1 without property ID_INPUT set
(II) config/udev: ignoring device /dev/vda2 without property ID_INPUT set
(II) config/udev: ignoring device /dev/vda5 without property ID_INPUT set
(II) config/udev: ignoring device /dev/snd/midiC0D0 without property ID_INPUT set
(II) config/udev: ignoring device /dev/snd/pcmC0D0c without property ID_INPUT set
(II) config/udev: ignoring device /dev/snd/pcmC0D0p without property ID_INPUT set
(II) config/udev: ignoring device /dev/snd/pcmC0D1p without property ID_INPUT set
(II) config/udev: ignoring device /dev/snd/controlC0 without property ID_INPUT set
(II) config/udev: ignoring device /dev/fd0 without property ID_INPUT set
(II) config/udev: ignoring device /dev/input/event1 without property ID_INPUT set
(II) config/udev: ignoring device /dev/input/event2 without property ID_INPUT set
(II) config/udev: ignoring device /dev/input/mouse0 without property ID_INPUT set
(II) config/udev: ignoring device /dev/ttyS0 without property ID_INPUT set
(II) config/udev: ignoring device /dev/ttyS1 without property ID_INPUT set
(II) config/udev: ignoring device /dev/ttyS10 without property ID_INPUT set
(II) config/udev: ignoring device /dev/ttyS11 without property ID_INPUT set
(II) config/udev: ignoring device /dev/ttyS12 without property ID_INPUT set
(II) config/udev: ignoring device /dev/ttyS13 without property ID_INPUT set
(II) config/udev: ignoring device /dev/ttyS14 without property ID_INPUT set
(II) config/udev: ignoring device /dev/ttyS15 without property ID_INPUT set
(II) config/udev: ignoring device /dev/ttyS16 without property ID_INPUT set
(II) config/udev: ignoring device /dev/ttyS17 without property ID_INPUT set
(II) config/udev: ignoring device /dev/ttyS18 without property ID_INPUT set
(II) config/udev: ignoring device /dev/ttyS19 without property ID_INPUT set
(II) config/udev: ignoring device /dev/ttyS2 without property ID_INPUT set
(II) config/udev: ignoring device /dev/ttyS20 without property ID_INPUT set
(II) config/udev: ignoring device /dev/ttyS21 without property ID_INPUT set
(II) config/udev: ignoring device /dev/ttyS22 without property ID_INPUT set
(II) config/udev: ignoring device /dev/ttyS23 without property ID_INPUT set
(II) config/udev: ignoring device /dev/ttyS24 without property ID_INPUT set
(II) config/udev: ignoring device /dev/ttyS25 without property ID_INPUT set
(II) config/udev: ignoring device /dev/ttyS26 without property ID_INPUT set
(II) config/udev: ignoring device /dev/ttyS27 without property ID_INPUT set
(II) config/udev: ignoring device /dev/...

I've also run into this bug this morning when upgrading a Natty system to this release. I can get the mouse to work again by unplugging and re-plugging it (it's a USB mouse). This is using gdm as the session manager, lightdm doesn't seem to work for me but I haven't poked at it hard either.

My keyboard is PS/2, so unplugging and re-plugging doesn't seem to do anything, possibly because udev or X isn't seeing the change.

I'm not using an xorg.conf file either, maybe it's needed again? At least until this bug is fixed?

John

This looks to be a udev bug. The udev log in the original post contains /dev/input/event* notifications, but these don't have the various IS_INPUT tags set, so X won't look at them. It's possible that these tags get set later, but by then X will have already rejected those devices as not input.

I ran into the same problem, updating from natty. Because of this bug, I downgraded the system to natty again, but the problem persists (running 2.6.38-10-generic #46-Ubuntu SMP Tue Jun 28 15:07:17 UTC 2011 x86_64).

Downgrading was performed by creating the following file, followed by running apt-get dist-upgrade:

$ cat /etc/apt/preferences.d/downgrade
Package: *
Pin: release a=natty
Pin-Priority: 1001

Package: *
Pin: release a=oneiric
Pin-Priority: 10

I'm on this boat too... There's anything I can do or provide to help?

Workaround for me was:

Copy /etc/X11/xorg.conf.failsafe to /etc/X11/xorg.conf

(Replaced gfx driver to mine)

Disabled HAL

Section "ServerFlags"
        Option "AutoAddDevices" "False"
EndSection

Added a section for keyboard explicitly (for mouse it was not needed)

Section "InputDevice"
        Identifier "Generic Keyboard"
        Driver "kbd"
        Option "XkbRules" "xorg"
        Option "XkbModel" "pc104"
        Option "XkbLayout" "se"
EndSection

Installed the kbd driver, since it was missing:

apt-get install xserver-xorg-input-kbd

udevd complains about write permissions to /run/udev but even with chmod go+w it does.
Is udev started before the fs is remounted rw?

@Dave, thanks for downgrading all those components to help isolate the change. Can you confirm that you are booting into the old X server?

$ apt-cache policy xserver-xorg-core

If it downgraded correctly it should show version 2:1.10.2-1ubuntu1.

Also post your Xorg.0.log and dmesg from after all the downgrades so we have it, then go ahead and upgrade back to latest if you want.

I notice that some of the changes in udev 171 caused boot failures in earlier versions. Has anyone tried downgrading back to version 167-0ubuntu3? That would probably be what I'd try next.

I took another look through the list of recently upgraded packages, but I don't see any others that strike me as potentially relevant. I think for now concentrate on udev (and xserver, if you find it didn't downgrade).

Gentoo too it appears:

http://forums.gentoo.org/viewtopic-p-6740252.html?sid=d7f8891c41d1e31aa462d516e25c4345

Ok, nevermind my post #22. A bit more digging has revealed the problem is this:

upgrade base-files 2011-07-08 01:23:13 6.3ubuntu2 6.3ubuntu4

There is a plan to transition from /var/run to /run, according to this plan:

  http://wiki.debian.org/ReleaseGoals/RunDirectory

However, base-files 6.3ubuntu3 got uploaded prematurely, thus beginning the transition before other packages were updated to follow suit.

Guessing the guilty commit to be this one:

base-files (6.3ubuntu3) oneiric; urgency=low

  * debian/1777-dirs: remove /var/lock (replaced by /run/lock on tmpfs)
  * debian/directory-list: remove /var/run and /var/lock, add /run
  * debian/postinst.in: replace references to /var/run with /run

 -- Scott James Remnant <email address hidden> Thu, 07 Jul 2011 13:33:06 -0400

Presumably downgrading your base-files to version 6.3ubuntu2 or other version prior to this commit will restore your system to working behavior.

See comment #6 on this bug:

  https://bugs.launchpad.net/ubuntu/+source/debootstrap/+bug/807974

That's a negative. Just downgraded to 6.3ubuntu1 and no joy.

However, I can confirm the xorg.conf workaround from comment #20.

I worked around it by doing a mv /run/udev /run/udev.old

@Bryce,

downgrading the whole system doesn't help, either. So maybe it's caused by a removed or newly installed package or configuration file.

$ dpkg -l "*udev*" | grep ^.i
ii libgudev-1.0-0 1:167-0ubuntu3 GObject-based wrapper library for libudev
ii libgudev1.0-cil 0.1-2 GObject-based wrapper library for libudev -- CLI bindings
ii libudev0 167-0ubuntu3 udev library
ii system-config-printer-udev 1.3.1+20110222-0ubuntu16.4 Printer auto-configuration facility based on udev
ii udev 167-0ubuntu3 rule-based device node and kernel event manager

$ dpkg -l "*xorg*" | grep ^.i
ii xorg 1:7.6+4ubuntu3.1 X.Org X Window System
ii xorg-docs-core 1:1.5.99.901-1ubuntu1 Core documentation for the X.org X Window System
ii xserver-xorg 1:7.6+4ubuntu3.1 the X.Org X server
ii xserver-xorg-core 2:1.10.1-1ubuntu1.1 Xorg X server - core server
ii xserver-xorg-input-all 1:7.6+4ubuntu3.1 the X.Org X server -- input driver metapackage
ii xserver-xorg-input-evdev 1:2.6.0-1ubuntu12 X.Org X server -- evdev input driver
ii xserver-xorg-input-kbd 1:1.5.0+git20110108.849f5092-0ubuntu2 X.Org X server -- keyboard input driver
ii xserver-xorg-input-mouse 1:1.6.0-1ubuntu3 X.Org X server -- mouse input driver
ii xserver-xorg-input-synaptics 1.3.99+git20110116.0e27ce3a-0ubuntu12.1 Synaptics TouchPad driver for X.Org server
ii xserver-xorg-input-vmmouse 1:12.6.99.901-1ubuntu2 X.Org X server -- VMMouse input driver to use with VMWare
ii xserver-xorg-input-wacom 1:0.10.11-0ubuntu4 X.Org X server -- Wacom input driver
ii xserver-xorg-video-all 1:7.6+4ubuntu3.1 the X.Org X server -- output driver metapackage
ii xserver-xorg-video-apm 1:1.2.3-0ubuntu5 X.Org X server -- APM display driver
ii xserver-xorg-video-ark 1:0.7.3-1ubuntu3 X.Org X server -- ark display driver
ii xserver-xorg-video-ati 1:6.14.0-0ubuntu4.1 X.Org X server -- AMD/ATI display driver wrapper
ii xserver-xorg-video-chips 1:1.2.3-2ubuntu5 X.Org X server -- Chips display driver
ii xserver-xorg-video-cirrus 1:1.3.2-2ubuntu7 X.Org X server -- Cirrus display driver
ii xserver-xorg-video-fbdev 1:0.4.2-3ubuntu6 ...

Confirming #27 with latest base-files. Likely the reason why downgrading alone didn't work.

mv /run/udev /run/udev.old worked for me, too!

at boottime I get the error:

udevd[363]: error: runtime directory '/run/udev' not writable, for now falling back to /dev/.udev'

The boottime error is a different bug, as pointed in comment #10

The tip given by #27 worked, and I believe the issue started for me after I updated udev to 172-0ubuntu1.

Marc Deslauriers' workaround in #27 did the trick for me, too. Thanks a lot!

I can also comfirm that renaming (or deleting) the directory /run/udev is a workaround.
So simply, having the directory /run/udev is problematic right now.

Actually, all of /run seems to be a broken concept. None of the files are in my dpkg database (using dpkg -S on them).

Where does /run come from?

Package base-files creates that directory.

Jason, /run is the new standard location for /var/run; one is currently a symlink to the other. Nothing in there _should_ show up in dpkg, as it's typically mounted as a tmpfs (i.e., ram only).

Ijust did fix from #27 I still get udev error but keyboard and mouse are working again. An odd thing just happened when Iapplied the fix it also fixed my sound that was not working.

Great thanks for workaround in #27, worked for me. I was particularly desperate, for I am testing Ubuntu on a laptop, and here I cannot re-plug keyboard (however, pressing alt+sysrq+R and then alt+ctrl+f1 got me to tty, from where I was able to install&launch links2 to get to this bug report), which makes this bug very significant.

It happened to me right after I have updated libudev and libgudev from 171-0ubuntu4 to 172-0ubuntu1 and rebooted my system.

I can also confirm the sound issue is related to this bug: my sound stopped working (pulseaudio was relentlessly restarting), but when I got everything back to work the sound is okay too.

Ditto the above, re laptop.
mv /run/udev worked for me.
Thanks!

According to https://launchpadlibrarian.net/74981481/udev_172-0ubuntu1.diff.gz the newly included changes are quite wide.

I didn't lose my mouse like most people, but I did lose my keyboard. My mouse freaked out, was flickering all over the screen and pressing both mouse buttons (Laptop touchpad not mouse). After finally resolving that, then my keyboard was no longer working, but the touchpad was working fine.

Then I tried the fix with mv udev, and that worked for me as well...

Don't know if it will help, but here is my history log of which upgrade made my keyboard stop working....

Start-Date: 2011-07-11 12:26:56
Commandline: apt-get upgrade
Upgrade: python-cupshelpers:amd64 (1.3.3+20110707-0ubuntu1, 1.3.3+20110709-0ubuntu1), libldap-2.4-2:amd64 (2.4.25-1ubuntu1, 2.4.25-1.1ubuntu1), gnome-session-bin:amd64 (3.1.3-0ubuntu1, 3.1.3-0ubuntu2), libgudev-1.0-0:amd64 (171-0ubuntu4, 172-0ubuntu1), system-config-printer-gnome:amd64 (1.3.3+20110707-0ubuntu1, 1.3.3+20110709-0ubuntu1), gnome-disk-utility:amd64 (3.0.0-1ubuntu2, 3.0.2-1ubuntu1), gnome-settings-daemon:amd64 (3.1.3-0ubuntu2, 3.1.3-0ubuntu3), libevince3-3:amd64 (3.1.2-0ubuntu1, 3.1.2-0ubuntu2), gnome-session:amd64 (3.1.3-0ubuntu1, 3.1.3-0ubuntu2), gnome-session-common:amd64 (3.1.3-0ubuntu1, 3.1.3-0ubuntu2), python-software-properties:amd64 (0.80.12, 0.80.13), libcryptui0a:amd64 (3.0.2-0ubuntu2, 3.1.1-0ubuntu1), update-manager:amd64 (0.152, 0.152.1), update-manager-core:amd64 (0.152, 0.152.1), linux-firmware:amd64 (1.55, 1.56), notify-osd:amd64 (0.9.30-0ubuntu4, 0.9.31-0ubuntu1), libgdu-gtk0:amd64 (3.0.0-1ubuntu2, 3.0.2-1ubuntu1), udev:amd64 (171-0ubuntu4, 172-0ubuntu1), libgck0:amd64 (3.1.1-0ubuntu1, 3.1.1-0ubuntu2), libpixman-1-0:amd64 (0.21.8-1ubuntu1, 0.22.2-1), libgdu0:amd64 (3.0.0-1ubuntu2, 3.0.2-1ubuntu1), system-config-printer-udev:amd64 (1.3.3+20110707-0ubuntu1, 1.3.3+20110709-0ubuntu1), software-properties-gtk:amd64 (0.80.12, 0.80.13), evince-common:amd64 (3.1.2-0ubuntu1, 3.1.2-0ubuntu2), libgcrypt11:amd64 (1.4.6-5ubuntu1, 1.5.0-1), evince:amd64 (3.1.2-0ubuntu1, 3.1.2-0ubuntu2), libudev0:amd64 (171-0ubuntu4, 172-0ubuntu1), system-config-printer-common:amd64 (1.3.3+20110707-0ubuntu1, 1.3.3+20110709-0ubuntu1), libpam-gnome-keyring:amd64 (3.1.1-0ubuntu1, 3.1.1-0ubuntu2)
Error: Sub-process /usr/bin/dpkg returned an error code (1)
End-Date: 2011-07-11 12:28:16

Effectively, rename /run/udev to /run/udev.old solves the issue!

Still funny that an empty directory create such an issue ...

Merge proposal for a (temporary) fix:
https://code.launchpad.net/~rsalveti/ubuntu/oneiric/base-files/run-transition/+merge/67444

I've created a PPA with the proposed fix here:
  https://launchpad.net/~bryce/+archive/lime

Can someone who is able to reproduce the issue still, verify that with the above PPA the issue is fixed?

#27 worked for me to! :)

Initial testing shows the merge proposal from rsalveti I posted above does not fix it.

I'm working on getting the package installed from the PPA, I'll let you know.

To trivially reproduce the problem:

sudo dpkg -i /var/cache/apt/archives/base-files_6.3ubuntu4_i386.deb (or amd64)
sudo dpkg-reconfigure udev

if sysvinit and initramfs-tools are going to take awhile to update (which they probably will since its a very complex merge) perhaps removing /run in the base-files postinst is the way to go? This should be affecting 100% of upgraded systems now that there is a udev upgrade today and is very critical.

On Mon, Jul 11, 2011 at 11:36:38PM -0000, Robert Hooker wrote:
> if sysvinit and initramfs-tools are going to take awhile to update
> (which they probably will since its a very complex merge) perhaps
> removing /run in the base-files postinst is the way to go? This should
> be affecting 100% of upgraded systems now that there is a udev upgrade
> today and is very critical.

I don't think *removing* /run is the right thing to do. If the mere
presence of run causes udev to misbehave, then that's a udev bug that needs
to be fixed separately; someone needs to figure out what's really going on
here, as nothing in the sysvinit merge is likely to help...

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
<email address hidden> <email address hidden>

Oneiric Alpha 2 updated as of 11 July. No keyboard, no mouse.

No /run/udev either, so mv /run/udev /run/udev.old didn't work.

So I booted in recovery mode:

mkdir /run/udev
mv /run/udev /run/udev.old

Rebooted and Alpha 2 is up with keyboard & mouse....

Jerry

Nothing to do with udev, it just causes the initramfs to be rebuilt.

What happens is that apparently due to the recently uploaded base-files there is now a /run directory, which might get partially populated during runtime. Apparently during initramfs time, udev uses /dev/.udev, but then when then initramfs is done, udev gets stopped, and restarted in the "real" system again, at which point it uses /run/udev/. But the latter does not have any of the previously detected properties like ID_INPUT_*, and thus stuff fails all over the place.

As you already figured out, "sudo rm /run/*" will temporarily fix the problem again.

We need to make up our mind here if we want to complete the base-files upload for /run, and adjust initramfs-tools etc. accordingly, or revert it, to avoid having more people fall into this trap.

I'll add some robustifications to udev while the transition is in progress, to handle this case more gracefully.

Thanks Martin.

I could work again with the command:

rm -rf /run/udev

yeah mv didn't make any success... on my Thinkpad needed completely removing. Thanks!!

This bug was fixed in the package udev - 172-0ubuntu2

---------------
udev (172-0ubuntu2) oneiric; urgency=low

  * extras/cdrom_id/60-cdrom_id.rules: Revert "hd*" hack for powerpc; this
    already is, or should be fixed in the powerpc kernel configuration to use
    the SCSI interface. (Confirmed by Luke).
  * Move rules/rules.d/78-graphics-card.rules to
    debian/local/78-graphics-card.rules and install it in debian/udev.install
    instead of patching Makefile.am
  * Move inline change in rules/rules.d/80-drivers.rules to
    debian/patches/load-fbcon.patch.
  * Move extras/rule_generator/75-persistent-net-generator.rules inline change
    to debian/patches/ignore-eucalyptus-virtual-ifaces.patch.
  * Move libudev/libudev-monitor.c inline change to
    debian/patches/libudev-revert-SOCK_NONBLOCK.patch.
  * Revert autoconf files to upstream state, as we have no further direct
    patches to the upstream build system.
  * debian/source/format: Move to 3.0 (quilt) format.
  * Add debian/patches/use_run_tmpfs: Only use /run/ if it is a tmpfs. Stolen
    from Debian's udev 167-2, thanks Marco d'Itri! (LP: #807306)
  * debian/control: Update Maintainer: field to Ubuntu Developers.
  * debian/control: Bump Standards-Version to 3.9.2.
  * debian/source/options: Ignore *.types files in documentation.
 -- Martin Pitt <email address hidden> Tue, 12 Jul 2011 08:36:12 +0200

Please correct me if I'm wrong, but I think base-files is ok now, as it ships /run. What we need now is mountall actually putting a tmpfs mount there, thus I add a mountall task.

As the udev safety belt is uploaded now, the remainder of the /run transition (mountall and initramfs-tools, AFAICS) are not as urgent any more. But still keeping targetted at oneiric, as we do want to finish this or completely roll back, so keeping priority high.

wow this bug is really nasty for not so experienced users, because they get "locked out" of their system, but they shouldn´t install oneiric in devstages either... Hope this patch gets distributed to the mirrors soon.

Discussed with Steve. I'll take care of the initramfs-tools merge. Debian's 0.99 version properly supports creating /run in the initramfs, and moving it over to the actual system.

This bug was fixed in the package initramfs-tools - 0.99ubuntu1

---------------
initramfs-tools (0.99ubuntu1) oneiric; urgency=low

  * Merge with Debian unstable. See 0.98.8ubuntu1 for remaining Ubuntu
    changes. This version creates /run in the initramfs, and moves it over to
    the running system. (LP: #807306)
  * hooks/busybox: Fix symlinking of /bin/sh to busybox, to work with current
    copy_exec() function.

initramfs-tools (0.99) unstable; urgency=low

  Release "scarpe rotte e pur bisogna andar"

  [ maximilian attems ]
  * [ce45cd0] update-initramfs: Show the mkinitramfs on error message.
  * [5b84e5b] maintainer-notes.html: Fix email to send patches to.
  * [ec6a77f] doc: s/ressources/resources/
  * [3c717fa] initramfs-tools: preinst use blkid for uuid generation if around.
    (Closes: #609964)
  * [8e2ffcd] postinst cleanup very old dpkg trigger comparison.
  * [78cdb26] mkinitramfs: Add support for xz compression. (closes: #626446)
  * [f5b8347] hook-functions: Cleanup ref to ide-disk.
  * [6cab0ec] initramfs-tools: cleanup any usplash traces. (closes: #620924)
  * [08d4895] initramfs-tools: Remove mkinitramfs-kpkg.
    (Closes: #454934, #502792)
  * [0ed6376] initramfs-tools: add hid_sunplus to listed keyboard drivers.
  * [5f53d3b] init: load netconsole after loading network drivers.
    Thanks Ferenc Wagner <email address hidden> (Closes: #596742)
  * [7ff2998] debian/copyright: Update authorlist and year attribution.
  * [e789cdd] maintainer-notes: Use git dch --multimaint-merge on examples.
  * [17296ff] dep_add_modules: Use hidden_dep_add_modules for ubifs.
  * [3d44bfb] hidden_dep_add: Use mika's variadic function from $2 on.
  * [8f8299d] mkinitramfs: copy over on build instead of using symlink tree.
    (Closes: #338405, #506540)
  * [f5afa6a] mkinitramfs: Add lib search path + run full ldconfig in
    initramfs. (Closes: #612633, #619670)
  * [259ad09] mkinitramfs: creat /run initramfs directory.
  * [5add333] initramfs-tools: init mount /run tmpfs.
  * [74109b9] init: No need to touch /dev/.initramfs.
  * [8e7620a] hook-functions: xhci-hcd got renamed. (Closes: #625224)
    Thanks to Matthew Wilcox <email address hidden>

  [ Tim Small ]
  * [1fe9f78] Add Documentation for modules=list in initramfs.conf(5).
    (Closes: #603903)

  [ Ben Hutchings ]
  * [c018886] kernel hooks: Treat missing version argument as an error.
  * [58ee42c] kernel hooks: Enable error-exit (sh -e).
  * [7866542] update-initramfs: Depend on kernel hook scripts rather than
    $ramdisk invocation.
  * [43fe8e6] update-initramfs: Remove support for 'do_bootloader' and
    specific boot loaders. (closes: #594189)

  [ Gianluigi Tiesi ]
  * [9c25269] mkinitramfs: misleading message in verbose mode.
    (Closes: #611046)

  [ Timo Juhani Lindfors ]
  * [871ffe7] initramfs-tools: Make panic message visible even if panic=
    is used.
  * [2525b00] initramfs-tools: Inform the user about reboot on panic=.

  [ Michael Prokop ]
  * [465a5f1] hidden_dep_add_modules(): make it dynamically to support more
    than 3 arguments as well.
  * [3323930] Use --check=crc32 option for xz compression.
    Thanks to Ulrich Dangel <email address hidden>
  * [bedf1e3] Use -8...

On my oneiric install /var/run is not mounted anymore, but its contents go to the root filesystem. When rebooting, on my system
* the dbus daemon does not start anymore because of the pid file from the previous start,
* X11 does not start anymore because of the missing dbus daemon, and
* on the login motd I get the info I have to restart because of a kernel upgrade.

Is this still to be fixed, or just a problem on my install?

Eduard,

What versions of initscripts and mountall do you have installed? If you have upgraded initscripts, /var/run should have already been converted to a symlink pointing at /run, and there should be no problem. Is /var/run a symlink or a real directory on your system?

If you're seeing this problem because the initscripts package was not upgraded on your system, we should check the package dependencies to make sure we're preventing this from happening in the future.

The versions are:
mountall 2.30
initscripts 2.88dsf-13.10ubuntu4

/var/run is not a symlink but a real directory on my harddisk (root filesystem).

One further info: During installation of the upgrade, it looked like that not all postinst scripts were executed properly because of the cups postinst script failing. Could it be that the /var/run symlink is setup in the postinst script?

Eduard Hasenleithner [2011-07-15 6:27 -0000]:
> One further info: During installation of the upgrade, it looked like
> that not all postinst scripts were executed properly because of the cups
> postinst script failing.

This should be fixed in cups 1.4.7-1ubuntu1.

> Could it be that the /var/run symlink is setup in the postinst
> script?

No, it's set up in an upstart job, so you need to reboot after the
upgrade. But still, you shouldn't do that with half of the packages
being unconfigured due to the aborted update. So I suggest to

  sudo apt-get update
  sudo apt-get dist-upgrade

(or using update-manager) again to clean up first.

Martin

--
Martin Pitt | http://www.piware.de
Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org)

Martin,

> No, it's set up in an upstart job, so you need to reboot after
> the upgrade.

Now that is strange, I rebooted already several times, but the situation did not change. Maybe there is a chicken-and-egg problem of upstart not triggering the symlink creation because there are still old (pid, ...) files in the directory?

> But still, you shouldn't do that with half of the
> packages being unconfigured due to the aborted update.

I'll do the update/dist-upgrade this evening and report success/failure.

Thanks,
Eduard

On Fri, Jul 15, 2011 at 08:05:25AM -0000, Eduard Hasenleithner wrote:

> > No, it's set up in an upstart job, so you need to reboot after
> > the upgrade.

> Now that is strange, I rebooted already several times, but the situation
> did not change. Maybe there is a chicken-and-egg problem of upstart not
> triggering the symlink creation because there are still old (pid, ...)
> files in the directory?

No, /etc/init.d/umountroot should unconditionally nuke /var/run on reboot.

Is your /etc/init.d/umountroot the unmodified version from the package?

Do you have any mount points *under* /var/run, that would prevent the rm -rf
/var/run in this shutdown script from succeeding?

Did /var/lock get turned successfully into a symlink, or is it still a
directory like /var/run?

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
<email address hidden> <email address hidden>

I did the update/dist-upgrade but the situation did not change.

> No, /etc/init.d/umountroot should unconditionally nuke /var/run on reboot.

So I would suspect it does not get executed at all

> Is your /etc/init.d/umountroot the unmodified version from the package?

I didn't change it, md5sum is 36d5098f8a3965dce172a1c8f75e1429.

> Do you have any mount points *under* /var/run, that would prevent the rm -rf
 /var/run in this shutdown script from succeeding?

According to the output of "mount" nothing is mounted below /var/run. (Please see also my previous attachment)

>Did /var/lock get turned successfully into a symlink, or is it still a
 directory like /var/run?

eduard@phenom:~$ ls -ld /var/lock
drwxrwxrwt 4 root root 4096 2011-07-15 19:30 /var/lock

No, still a directory. I think this is also a indication that /etc/init.d/umountroot doesn't get executed on reboot.

Just for clarification: I definitely know how to set the symlinks myself, and get the system running again. I just want to investigate this issue, so that other people don't run into the same problem.

On Fri, Jul 15, 2011 at 05:40:12PM -0000, Eduard Hasenleithner wrote:
> > Is your /etc/init.d/umountroot the unmodified version from the
> > package?

> I didn't change it, md5sum is 36d5098f8a3965dce172a1c8f75e1429.

This is the correct md5sum.

Do the symlinks /etc/rc0.d/S60umountroot and /etc/rc6.d/S60umountroot exist?

> According to the output of "mount" nothing is mounted below /var/run.
> (Please see also my previous attachment)

Right, sorry - answered my own question by looking there after sending the
previous mail.

> No, still a directory. I think this is also a indication that
> /etc/init.d/umountroot doesn't get executed on reboot.

Yep, apparently. And the system shuts down cleanly, no fsck being required
on the next boot due to the disk not having been unmounted correctly?

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
<email address hidden> <email address hidden>

> Do the symlinks /etc/rc0.d/S60umountroot and /etc/rc6.d/S60umountroot exist?

Yes they do. Question: Do I remember correctly that the symlinks with S are used for starting, and those with K for stopping? If that is the case, why is start a "No-op" in the umountroot script?

On Fri, Jul 15, 2011 at 07:30:54PM -0000, Eduard Hasenleithner wrote:
> > Do the symlinks /etc/rc0.d/S60umountroot and /etc/rc6.d/S60umountroot
> exist?

> Yes they do. Question: Do I remember correctly that the symlinks with S
> are used for starting, and those with K for stopping? If that is the
> case, why is start a "No-op" in the umountroot script?

"start" and "stop" mean something different in runlevels 0 and 6. All
scripts in these runlevels are called with "stop" as the argument, the K vs.
S is only used for ordering.

At this point I'm all out of guesses as to what's going on with your system.
Could you please edit /etc/init.d/umountroot to add these two lines as the
first non-comments in the file:

  set -x
  exec > /shutdown.log 2>&1

then reboot, and attach the resulting /shutdown.log file (if any)? If the
log doesn't get created on reboot, that's interesting to know as well, of
course.

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
<email address hidden> <email address hidden>

/shutdown.log is not created, so the umountrootfs is not executed at all. Considering how fast the system reboots, I'm not sure if any script from rc6.d is executed, but that is hard to tell since I have a SSD. I'd be happy if somebody would help me finding the root cause for this in my system. But since it apparently only affects me, it is probably not important. Anyway, I'll try to find out myself.

Considering that this has only indirectly to do with the original bug report, it might be better to discuss somewhere else (e.g. per email).

Sorry for the further interruption, I think I found the reason:
the reboot script has the name S01reboot in rc6.d, so it gets executed before S01umountroot. No clue why that could be.

Hi, I'm in the same Eduard's situation, I think from 3-4 days ago.

Do you have a clean workaround steps?

I've only to remove /var/run, link /var/run to /run and remove /var/lock ?

And what about /var/lock? link to what?

I found the the root cause for the init script reordering and reported it in #811675. Actually my system was broken several days before this bug, but the brokenness showed up only due to the umountrootfs script not being executed on reboot.

Davide,

The problem with removing /var/run and /var/lock and making them symlinks yourself is that this breaks all the various things that access sockets in either of these directories. If you do this on a desktop system, your desktop will be broken until reboot, and may not even shut down cleanly.

If you are going to make this change yourself, I recommend that you boot to single user mode, make the change, and immediately reboot. /var/run should point to /run; /var/lock should point to /run/lock.

I would still like to know why the automatic upgrade didn't work for you. If users are already running into problems with the migration at this early stage in the release cycle, I'm very concerned that this will break for many more users at the time of the beta - users who are going to be less able to help us diagnose the problem.

Hi All,
  The package update fixes my original VM that caused me to report this - thanks all!

Dave

So, it is fixed? Can i use lightdm?

I confirm, following Steve's suggestions all works great

Like Eduard, my problems began with a cups postinst failed upgrade.

#27 fixed my keyboard too, however my laptop Synaptic mouse is still unusable even with a dist-upgrade of today (which means I also have the #62 problem)... FYI.

Looks like bug #811441 might be a difference facet of the same transition problem from /var/run to /run. The fix suggested here work for that bug too.

mountall 2.28 and above include the /run mount, so marking this task as resolved.

I had the exact same issue documented here on upgrade from 11.04 to 11.10.
Removing the existing /var/run and /var/run/lock and replacing with sym links as per #76 fixed it.

The impact of this was a total lack of system (wireless) networking coming up - this gives a >! minute pause during boot, as well as a subsequent total failure to be able to log into X (KDE or GNOME sessions give a cryptic error related to DBUS).

I've just run an apt-get update/upgrade after *finally* getting back into my new 11.10 beta 2 system, and no updates to the mountall show up... therefore I assume this bug is still present.

Tom,

Please file a new bug report against the 'initscripts' package in Ubuntu using the command 'ubuntu-bug initscripts' and include a copy of /var/log/dist-upgrade/apt-term.log from the affected system, and subscribe me to the resulting bug.

After you upgraded, did you reboot? If your system crashed after upgrade, instead of being cleanly shut down, this would explain the failure to migrate /var/run.

Do you have /var/run in your /etc/fstab?

I too hit this bug after upgrade today. /var/run is not in my /etc/fstab. Can you point me to the new bug? I'll submit my relevant info.

See https://bugs.launchpad.net/ubuntu/+source/dbus/+bug/811441